This course is a graduate-level research course covering advanced topics in NLP, introducing the state-of-the-art methods for computational understanding, analysis, and generation of natural language text. In recent years, advances in deep learning models for NLP has transformed the ability of computers to converse with human using language, giving us multi-lingual, multi-models that are capable of answering questions, composing message, translating and summarizing documents. The development of large language models (LLMs) are built on top of neural models such as transformers, that allows for the scaling up of models and training with large amounts of data. In this course, we will focus on current state-of-the-art methods in NLP including how to do parameter efficient fine-tuning, techniques for scaling models to long sequences, etc. We will also go beyond transformers to learn alternative architectures such as state-space models.
Students are expected to have prior experience with deep learning concepts and framework (Pytorch, Tensorflow, etc), and should also be familiar with basic natural language processing.
Each week, students will read papers in a particular area of natural language processing, and discuss the contributions, limitations and interconnections between the papers. Students will also work on a research project during the course, culminating in a final presentation and written report. The course aims to provide practical experience in comprehending, analyzing and synthesizing research in natural language processing and understanding.
Note: This course is NOT an introductory course to natural language processing. If you are interested in taking a introductory course about natural language processing, please take CMPT 413/713.
There are no formal prerequisites for this class. However, you are expected to be familiar with the following:
Below is a tentative outline for the course.
R: Readings, BG: (Optional) Background material / reading for deeper understanding. Provided for reference.
| Date | Topic | Notes |
|---|---|---|
| Jan 6 | Introduction, NLP review & logistics | BG: Stanford CS324 intro to LLMs |
| Jan 8 | Transformers and LLMs | BG: Attention is all you need BG: LLaMA |
| Jan 13 | Training LLMs (pre-training, instruction tuning) | BG: Instruction tuning survey |
| Jan 15 | Paper discussion 0 | |
| Jan 20 | Training LLMs (PEFT, efficient training) | |
| Jan 22 | Ensembling and mixture-of-experts | BG: Review of mixture of experts |
| Jan 27 | Using LLMs (Prompting, generation and retrieval) Potential project topics |
BG: Prompting survey BG: ACL 2023 Text generation tutorial BG: ACL 2023 RAG tutorial |
| Jan 29 | Paper discussion 1 | |
| Feb 3 | Efficient LLMs: Pruning, quantization, distillation | BG: Inference optimization (Lillian Weng) |
| Feb 5 | Handling long context | R: UniMem |
| Feb 10 | State space models | R: Mamba |
| Feb 12 | Paper discussion 2 | |
| Feb 17 | No class - Reading break | |
| Feb 19 | No class - Reading break | |
| Feb 24 | Reasoning | |
| Feb 26 | Project proposals | Due: Project proposal |
| Mar 3 | Multimodal models | |
| Mar 5 | Paper discussion 3 | |
| Mar 10 | LLM agents | |
| Mar 12 | Paper discussion 4 | |
| Mar 17 | LLM attacks and social impact | |
| Mar 19 | Project milestone presentations | Due: Project milestone |
| Mar 24 | Guest lecturer | |
| Mar 26 | Guest lecturer | |
| Mar 31 | Interpreting LLMs: what is learned in these models? | |
| Apr 2 | Paper discussion 5 | |
| Apr 7 | Project presentations | |
| Apr 9 | Project presentations + Conclusion | |
| Apr 10 | Due: Project final report |
SFU’s Academic Integrity web site is filled with information on what is meant by academic dishonesty, where you can find resources to help with your studies and the consequences of cheating. Check out the site for more information and videos that help explain the issues in plain English.
Each student is responsible for his or her conduct as it affects the University community. Academic dishonesty, in whatever form, is ultimately destructive of the values of the University. Furthermore, it is unfair and discouraging to the majority of students who pursue their studies honestly. Scholarly integrity is required of all members of the University. Please refer to this web site.