I teach three courses: Speech Processing with Deep Learning, Deep Learning; Transformers, and Large Language Models. These courses align with my research interests in speech recognition and understanding, conversational AI, and deep learning.
Speech recognition and synthesis have become foundational components in the development of modern artificial intelligence, bridging the gap between human communication and machine interaction. This course aims to provide a comprehensive introduction to the core principles of speech signal processing and modeling, essential for understanding how machines perceive and generate human language.
We begin with an exploration of the fundamental nature of the speech signal, examining how humans produce speech and the elements that characterize it—such as pitch, formants, and prosody. We delve into the mechanics behind speech generation, using mathematical models to represent the speech process, notably through Linear Predictive Coding (LPC) and autoregressive models. As speech signals are often complex and variable, we explore various techniques to transform these signals into machine-interpretable formats using spectral representations like STFT, Mel-spectrum, and MFCCs.
Progressing to practical applications, the course covers keyword spotting and modern automatic speech recognition (ASR). With advancements in deep learning, we explore modern architectures such as RNNs, LSTMs, and transformers, and the Connectionist Temporal Classification (CTC) loss function, which have revolutionized speech processing by leveraging neural networks for sequence modeling and self-supervised learning.
The course culminates with state-of-the-art speech synthesis techniques, including autoregressive models like WaveNet, and GAN-based approaches, leading to more natural and human-like text-to-speech systems. Advanced topics will also cover emerging trends in speech-language models that operate without explicit text, showcasing the future trajectory of research in this field.
We will explore both the theoretical foundations and practical techniques for designing, building, and analyzing deep neural networks, with a focus on supervised learning. Topics include the behavior and convergence of gradient descent and its variants, efficient methods for automatic differentiation, and the theoretical and empirical properties of multilayer networks—such as approximation capabilities, initialization strategies, generalization, and symmetry. We will also cover convolutional networks and their extensions for visual tasks, advanced training methods and their analysis, and neural architectures for sequential data, including recurrent networks, attention mechanisms, and transformers. Additionally, the course addresses strategies to improve data efficiency (e.g., pre-training, self-supervised learning) and resource efficiency (e.g., model quantization, pruning).
This course is taught with Prof. Daniel Soudry.
There will be 12 meetings: the first 7-8 meetings will be given by the lecturer (me). The last meetings will be given by the course participants (see note about working in pairs under the grade composition). Each student will have to present a paper and possible future directions. The article described in class and the proposed continuation directions will be the basis for the project. Discussions will take place during all the meetings, so attendance in the lectures is mandatory.
The course begins with an in-depth exploration of the core ideas that gave rise to the Transformer architecture, unpacking its structure, intuition, and evolution. We’ll then dive into the most influential research in the field—both theoretical advances and practical innovations—before examining how Transformers are applied across key domains, including natural language processing, speech recognition, and computer vision.
In the second half of the course, students will take a more active role by presenting selected papers from recent literature and developing a mini-project inspired by their assigned work, gaining hands-on experience with cutting-edge developments in the field.
After approximately 7-8 introductory lectures by the lecturer, the course will shift focus to student-led presentations based on assigned projects. Each student will be assigned one or more recent papers in the field, which will serve as the foundation for their project work.
The project will consist of the following components:
Reading and understanding the assigned paper(s)
Implementing any proposed algorithms, if applicable
Investigating potential extensions or improvements to the original work
Preparing a presentation summarizing the work (the slides are subject to the lecturer’s approval)
Delivering the presentation to the class