CS7643 Quiz 4 questions and answers
Embedding - ANSWER-A learned map from entities to vectors that encodes similarity
Graph Embedding - ANSWER-Optimize the objective that connected nodes have more
similar embeddings than unconnected nodes.
Task: convert nodes to vectors
- effectively unsupervised learning where nearest neighbors are similar
- these learned vectors are useful for downstream tasks
Multi-layer Perceptron (MLP) pain points for NLP - ANSWER-- Cannot easily support
variable-sized sequences as inputs or outputs
- No inherent temporal structure
- No practical way of holding state
- The size of the network grows with the maximum allowed size of the input or output
sequences
Truncated Backpropagation through time - ANSWER-- Only backpropagate a RNN
through T time steps
Recurrent Neural Networks (RNN) - ANSWER-h(t) = activation(U*input + V*h(t-1) +
bias)
y(t) = activation(W*h(t) + bias)
- activation is typically the logistic function or tanh
- outputs can also simply be h(t)
- family of NN architectures for modeling sequences
Training Vanilla RNN's difficulties - ANSWER-- Vanishing gradients
- Since dx(t)/dx(t-1) = w^t
- if w > 1: exploding gradients
- if w < 1: vanishing gradients
Long Short-Term Memory Network Gates and States - ANSWER-- f(t) = forget gate
- i(t) = input gate
- u(t) = candidate update gate
- o(t) = output gate
- c(t) = cell state
- c(t) = f(t) * c(t - 1) + i(t) * u(t)
- h(t) = hidden state
- h(t) = o(t) * tanh(c(t))
, Perplexity(s) - ANSWER-= product( 1 / P(w(i) | w(i-1), ...) ) ^ (1 / N)
= b ^ (-1/N sum( log(b) (P(w(i) | w(i-1), ...) ) )
- note exponent of b is per word CE loss
- perplexity of a discrete uniform distribution over k events is k
Language Model Goal - ANSWER-- estimate the probability of sequences of words
- p(s) = p(w1, w2, ..., wn)
Masked Language Modeling - ANSWER-- pre-training task - an auxiliary task different
from the final task we're really interested in, but which can help us achieve better
performance finding good initial parameters for the model
- By pre-training on masked language modeling before training on our final task, it is
usually possible to obtain higher performance than by simply training on the final task
Knowledge Distillation to Reduce Model Sizes - ANSWER-- Have fully parameterized
teacher model
- Have a much smaller student model
- Student model attempts to minimize prediction error and distance to teacher model
simultaneously
L(dist) = CE b/w student and teacher predictions
L(student) = CE b/w predicted output and actual
L = alpha * L(dist) + beta * L(student)
Advantages:
- may work well b/c of soft predictions of teacher model
- if we don't have enough labeled text we can still train student model to align
predictions
Collobert and Weston Vector Idea - ANSWER-a word and its context is a positive
training sample; a random word in that sample context gives a negative training sample
Word2vec Overview - ANSWER-Word2vec - a framework for learning word vector
Idea:
- we have a large corpus of text
- every word in. fixed vocabulary represented by a vector
- Go through each position t in the text, which has a center word c and context words o
- Use the similarity of the word vectors for c and o to calculate the probability of o given
c (or vice versa)
- Keep adjusting the word vectors to maximize this probability
Word2vec Variants - ANSWER-Skip-Gram: Predict context words given center word
Embedding - ANSWER-A learned map from entities to vectors that encodes similarity
Graph Embedding - ANSWER-Optimize the objective that connected nodes have more
similar embeddings than unconnected nodes.
Task: convert nodes to vectors
- effectively unsupervised learning where nearest neighbors are similar
- these learned vectors are useful for downstream tasks
Multi-layer Perceptron (MLP) pain points for NLP - ANSWER-- Cannot easily support
variable-sized sequences as inputs or outputs
- No inherent temporal structure
- No practical way of holding state
- The size of the network grows with the maximum allowed size of the input or output
sequences
Truncated Backpropagation through time - ANSWER-- Only backpropagate a RNN
through T time steps
Recurrent Neural Networks (RNN) - ANSWER-h(t) = activation(U*input + V*h(t-1) +
bias)
y(t) = activation(W*h(t) + bias)
- activation is typically the logistic function or tanh
- outputs can also simply be h(t)
- family of NN architectures for modeling sequences
Training Vanilla RNN's difficulties - ANSWER-- Vanishing gradients
- Since dx(t)/dx(t-1) = w^t
- if w > 1: exploding gradients
- if w < 1: vanishing gradients
Long Short-Term Memory Network Gates and States - ANSWER-- f(t) = forget gate
- i(t) = input gate
- u(t) = candidate update gate
- o(t) = output gate
- c(t) = cell state
- c(t) = f(t) * c(t - 1) + i(t) * u(t)
- h(t) = hidden state
- h(t) = o(t) * tanh(c(t))
, Perplexity(s) - ANSWER-= product( 1 / P(w(i) | w(i-1), ...) ) ^ (1 / N)
= b ^ (-1/N sum( log(b) (P(w(i) | w(i-1), ...) ) )
- note exponent of b is per word CE loss
- perplexity of a discrete uniform distribution over k events is k
Language Model Goal - ANSWER-- estimate the probability of sequences of words
- p(s) = p(w1, w2, ..., wn)
Masked Language Modeling - ANSWER-- pre-training task - an auxiliary task different
from the final task we're really interested in, but which can help us achieve better
performance finding good initial parameters for the model
- By pre-training on masked language modeling before training on our final task, it is
usually possible to obtain higher performance than by simply training on the final task
Knowledge Distillation to Reduce Model Sizes - ANSWER-- Have fully parameterized
teacher model
- Have a much smaller student model
- Student model attempts to minimize prediction error and distance to teacher model
simultaneously
L(dist) = CE b/w student and teacher predictions
L(student) = CE b/w predicted output and actual
L = alpha * L(dist) + beta * L(student)
Advantages:
- may work well b/c of soft predictions of teacher model
- if we don't have enough labeled text we can still train student model to align
predictions
Collobert and Weston Vector Idea - ANSWER-a word and its context is a positive
training sample; a random word in that sample context gives a negative training sample
Word2vec Overview - ANSWER-Word2vec - a framework for learning word vector
Idea:
- we have a large corpus of text
- every word in. fixed vocabulary represented by a vector
- Go through each position t in the text, which has a center word c and context words o
- Use the similarity of the word vectors for c and o to calculate the probability of o given
c (or vice versa)
- Keep adjusting the word vectors to maximize this probability
Word2vec Variants - ANSWER-Skip-Gram: Predict context words given center word
