SFU NLP class: Syllabus

Georges Artsrouni's mechanical brain, a translation device patented in 1933 in France.

Schedule

The schedule is preliminary and subject to change. Slides will be updated as the term progresses.

Week	Date	Topic	Assignments	Readings and Resources
1
	1/5	Lecture: Introduction [slides]		I'm sorry Dave, I'm afraid I can't do that. Lillian Lee. (optional) Talking to Computers in Natural Language. Percy Liang. (optional) Advances in Natural Language Processing. Julia Hirschberg and Christopher D. Manning.
	1/6	Tutorial (optional): Review of a) probability, linear algebra and calculus and b) useful python/unix commands
	1/7	Lecture: Language Modeling [LM slides]		Language Modeling with N-Grams (J+M chapter 3) (optional) Language models. Michael Collins.
	1/9	Tutorial (optional): Building language models
2
	1/12	Lecture: Text classification - Naive Bayes [NB slides] [Evaluation slides]		Naive Bayes and Sentiment Classification (J+M chapter 4)
	1/13	Tutorial (optional): Learning pytorch
	1/14	Lecture: Text classification - Logistic Regression [Logistic regression slides]	Due: HW0	Logistic Regression (J+M chapter 5)
	1/16	Tutorial (optional): Building classifiers with pytorch
3
	1/19	Lecture: NN review and word Representations [Neural network and classification review]		Neural Networks and Neural Language Models (J+M chapter 7) (optional) Feedforward neural networks. Michael Collins. (optional) A Primer on Neural Network Models for Natural Language Processing. Yoav Goldberg.
	1/20	Tutorial (optional): Building text classifiers with word embeddings
	1/21	Lecture: Word Representations and neural language models [WV slides 1]		Vector Semantics and Embeddings (J+M chapter 6) (optional) Improving Distributional Similarity with Lessons Learned from Word Embeddings. Omer Levy, Yoav Goldberg, and Ido Dagan.
	1/23	Tutorial (optional): Building text classifiers with pretrained word embeddings
4
	1/26	Lecture: Word2Vec [WV slides 2]		Vector Semantics and Embeddings (J+M chapter 6) (optional) Improving Distributional Similarity with Lessons Learned from Word Embeddings. Omer Levy, Yoav Goldberg, and Ido Dagan.
	1/27	Tutorial (optional): Neural language models with BoW/Fixed Window FFN
	1/28	Lecture: Sequence modeling (HMMs and RNNs) [slides] [slides]	Due: HW1	Hidden Markov Models (J+M Appendix A.1-A.4)
	1/30	Tutorial (optional): Using RNNs in pytorch
5
	2/2	Lecture: Neural Sequence Modeling (LSTM/GRU) [slides]		RNNs and LSTMs (J+M chapter 8) (optional) Understanding LSTM Networks. Christopher Olah.
	2/4	Lecture: Sequence-to-sequence models [slides]
6
	2/9	Lecture: Transformers and contextualized word embeddings [transformer slides] [cwe slides]		Transformers (J+M chapter 9) (optional) Attention Is All You Need. Vaswani et al. 2017. (optional) The Annotated Transformer. Alexander Rush.
	2/11	Lecture: Contextualized word embeedings and project information [project info slides] [benchmark slides]	Due: HW2	Masked Language Models (J+M chapter 11) (optional) Illustrated BERT. Jay Alammar. (optional) Contextual Word Representations: A Contextual Introduction. Noah A. Smith. (optional) BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Devlin et al. 2018. (optional) Emergent linguistic structure in artificial neural networks trained by self-supervision. Manning et al. 2020. (optional) A Primer in BERTology: What We Know About How BERT Works. Rogers, Kovaleva, and Rumshisky.
7
	2/16	No class - No class - Reading break
	2/19	No class - No class - Reading break
8
	2/23	Lecture: Pretraining and fine-tuning [pretraining slides] [fine-tuning slides]	Due: Project proposal	HuggingFace PEFT: Parameter-Efficient Fine-Tuning of Billion-Scale Models on Low-Resource Hardware. Sourab Mangrulkar, Sayak Paul". (optional) Prefix-Tuning: Optimizing Continuous Prompts for Generation. Xiang Lisa Li, Percy Liang. (optional) AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning. Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, Jianfeng Gao. (optional) LoRA: Low-Rank Adaptation of Large Language Models. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen. (optional) Adapter methods. docs.adapterhub.ml.
	2/25	Lecture: Few-shot and in-context learning [slides]	Due: HW3	Model Alignment, Prompting, and In-Context Learning (J+M chapter 12.1, 12.2) Language Models are Unsupervised Multitask Learners. Open AI. Language Models are Few-Shot Learners. Open AI. GPT-4 Technical Report. Open AI. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, Luke Zettlemoyer. In-Context Learning Learns Label Relationships but Is Not Conventional Learning. Jannik Kossen, Yarin Gal, Tom Rainforth.
9
	3/2	Lecture: Constituency Parsing [slides]		Constituency Grammars (J+M chapter 18) (optional) Statistical Constituency Parsing (J+M chapter Appendix C.1-C.4)
	3/4	Lecture: Dependency parsing [slides]		Dependency Parsing (J+M chapter 19)
10
	3/9	Lecture: Instruction tuning and reinforcement learning from human feedback [instruction-tuning slides]
	3/11	Lecture: Final project tips, model debugging and analysis [slides]	Due: HW4
11
	3/16	Lecture: Parameter efficient fine-tuning [fine-tuning slides]
	3/18	Lecture: Grounding [slides]
	3/19		Due: Project milestone
12
	3/23	Lecture: Guest Lecture
	3/25	Lecture: Overview of recent multimodal research Visual grounding in 3D (Austin Wang) Text and 3D Representation Learning (Han-Hung Lee) Text-to-Scene Generation (Austin Wang) LLMs for modeling DNA (Chuanqi Tang)
13
	3/30	Lecture: Scaling laws for LLMs [slides]
	4/1	Lecture: LLM Agents [slides]
14
	4/6	Lecture: Final project presentations
	4/8	Lecture: Conclusion [slides]
	4/9		Due: Final project report