Generative models view the world through the lens of probabilities
Given a finite set of samples \(S\) generated from an underlying distribution \(p_{data}\), the goal of a generative model is to approximate \(p_{data}\) from \(S\)
Parametric vs non-parameteric models
The course will focus on parametric models.
Parametric models can scale more efficiently with large datasets
Parametric models are limited in the family of distributions they can represent
Learning
Given a dataset \(data\), the goal is to find the parameters of a generative model \(\theta\) that closes the distance between \(p_{data}\) and \(p_{\theta}\) (The model’s learnt distribution and the real distribution)
Stated mathematically
\(\min_{\theta \in M}d(p\_{data}, p_{\theta})\)
Where \(d\) is a measure of distance between the two distributions and \(M\) is the model’s family
The problem: Current datasets are way too small in size compared with the possible set of values covered by the true distribution
Course Focus (Answer the following questions)
What is the representation for the model family \(M\) ?
What is the objective function d(⋅)?
What is the optimization procedure for minimizing d(⋅)?
3 Fundamental Inference Queries for generative models
Density estimation: Given a datapoint \(x\) what is the probability assigned by the model, i.e., \(p_{\theta}(x)\)?
Sampling: How can we generate novel data from the model distribution, i.e., \(x_{new} \sim p_{\theta}(x)\)?
Unsupervised representation learning: How can we learn meaningful feature representations for a datapoint \(x\)?
Current Challenges
Quantitative evaluation of generative tasks is not easy since current metrics fail to reflect desirable qualitative metrics
Not all model families permit efficient and accurate inference on all these tasks