Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model

Annotated Paper Link: Google Drive Link

These are notes for Zhou et al. (2024).

This paper intriduces a multi-modal (text and image) model that can uses a discrete distribution loss for text (next token prediction) and a continuous distribution loss for images (diffusion loss). For text, loss is calculated per token, while for images, loss is calculated per image. The loss function is a weighted sum of the two losses.

Zhou, C., Yu, L., Babu, A., Tirumala, K., Yasunaga, M., Shamis, L., Kahn, J., Ma, X., Zettlemoyer, L., & Levy, O. (2024). Transfusion: Predict the next token and diffuse images with one multi-modal model. https://arxiv.org/abs/2408.11039