Session 4: Paradigms and applications of Generative AI¶

Date: 2026-04-13

We continue with the learning paradigms of AI and understand the learning process. We then get an overview of generative AI models and applications.

Plan¶

Homework review¶

We compare homework on paradigms.

Attention mechanisms and transformers¶

One key architectural concept behind large language models and other generative AI models are attention mechanisms and transformers. We discuss these concepts using the following video resources:

The moment we stopped understanding AI [AlexNet] by Welch Labs --> this video (~18 min) emphasizes the importance of the attention mechanism and the transformer architecture for the development of generative AI models. It also highlights the shift from traditional machine learning approaches to deep learning and how this has led to a significant increase in the capabilities of AI systems.
Transformers: The best idea in AI - a snippet from the podcast discussion between Andrej Karpathy and Lex Fridman --> this video (~9 min)gives some personal insights from Andrej Karpathy, one of the leading researchers in the field of deep learning and generative AI, about the importance of transformers and their impact on the field of AI.
But What Are Transformers? by Jia-Bin Huang --> this video (~17 min) provides a detailed explanation of the transformer architecture and its components. It explains the math behind it on a higher level and gives some insights into how it works and why it is so effective for generative AI tasks.
3Blue1Brown: Transformers, the tech behind LLMs | Deep Learning Chapter 5 --> this video (~28 min) gives a more in-depth explanation of the transformer architecture, including the attention mechanism and how it works. It also provides some visualizations to help understand the concepts better. This chapter focuses on the concept of word embeddings and how they are used in transformers.
3Blue1Brown: Attention in transformers, step-by-step | Deep Learning Chapter 6 --> this video (~27 min) continues the explanation of the transformer architecture, focusing on the attention mechanism and how it works in detail. It provides a step-by-step explanation of how attention is calculated and how it contributes to the overall functioning of the transformer model. This chapter focuses on the concept of attention and how it allows transformers to focus on different parts of the input sequence when generating output.
Eventually, we also look at 3Blue1Brown: How might LLMs store facts | Deep Learning Chapter 7
Here is an interactive visualization of the transformer that also tries to explain the inner workings of the transformer architecture in an interactive way. This has also been published as a paper: Transformer Explainer: Interactive Learning of Text-Generative Models. Aeree Cho, Grace C. Kim, Alexander Karpekov, Alec Helbling, Zijie J. Wang, Seongmin Lee, Benjamin Hoover, Duen Horng Chau. Poster, IEEE VIS 2024.

Additional resources (not shown in class)¶

The Illustrated Transformer by Jay Alammar is a highly recommended blog post explaining transformers with great illustrations. There is also a book and a course that contains an updated version and some more context.

Remainder of the term¶

Next step is theory: each participant researches one application area and presents it to the group.
Then we do a small project with a chosen tool.
Finally, we reflect on the implications of generative AI.

Choice of theory topics¶

Students will choose one application area of generative AI for deeper research, for example:

Text generation with LLMs
Text-based generation of other modalities
Image generation with foundation models
Video generation with foundation models
3D model generation with foundation models
Audio generation with foundation models
Code generation with LLMs

The entry point for the research is the curated list of resources on generative AI: awesome-generative-ai. If further resources are needed, students are encouraged to search for additional materials and get inspired by the awesome-generative-ai-guide by Aishwarya Naresh Reganti. There are some more specific resources listed in the theory section as well. You can also use this generated taxonomy to get a better impression of foundation models.

Plans for the upcoming sessions¶

Session 5 on 2026-04-20: Foundation models and their applications.
Session 6 on 2026-04-27: Presentation of theory topics.
Session 7 on 2026-05-04: Choice of tool and plan creation.
Session 8 on 2026-05-11: Working on projects.
Session 9 on 2026-05-18: Working on projects.
Session 10 on 2026-06-01: No class, work on projects at home.
Session 11 on 2026-06-08: Final presentation of projects.
Session 12 on 2026-06-15: Dealing with Agentic AI and its implications for creative processes.
Session 13 on 2026-06-22: Reflection on using AI in creative processes.
Session 14 on 2026-06-29: Consequences of using AI in creative processes.

Materials¶

Code examples¶

See task description for details and links to the code examples.

Taxonomy of generative AI models and applications¶

You can find the generated taxonomy of generative AI models and applications.

Results¶

We watched and discussed the videos on transformers and attention mechanisms until 3Blue1Brown: Transformers, the tech behind LLMs | Deep Learning Chapter 5. The plan is to watch 3Blue1Brown: Attention in transformers, step-by-step | Deep Learning Chapter 6 in the next session. We also discussed the plans for the upcoming sessions and the choice of theory topics.