SFU NLP class: Syllabus

Georges Artsrouni's mechanical brain, a translation device patented in 1933 in France.

Schedule

The schedule is preliminary and subject to change. Slides will be updated as the term progresses.

Week	Date	Topic	Assignments	Readings and Resources
0
	1/8	Lecture: Introduction [slides]		I'm sorry Dave, I'm afraid I can't do that. Lillian Lee. (optional) Talking to Computers in Natural Language. Percy Liang. (optional) Advances in Natural Language Processing. Julia Hirschberg and Christopher D. Manning.
	1/9	Tutorial (optional): Review of a) probability, linear algebra and calculus and b) useful python/unix commands
	1/10	Lecture: Language Modeling [LM slides]		Language Modeling with N-Grams (J+M chapter 3) (optional) Language models. Michael Collins.
	1/12	Tutorial (optional): Building language models
1
	1/15	Lecture: LM smoothing and evaluation; Text classification - Naive Bayes [NB slides]		Naive Bayes and Sentiment Classification (J+M chapter 4)
	1/16	Tutorial (optional): Learning pytorch
	1/17	Lecture: Text classification - Logistic Regression [Logistic regression slides] [Evaluation slides]	Due: HW0	Logistic Regression (J+M chapter 5)
	1/19	Tutorial (optional): Building classifiers with pytorch
2
	1/22	Lecture: NN review and word Representations [Neural network and classification review] [WV slides 1]		Neural Networks and Neural Language Models (J+M chapter 7) (optional) Feedforward neural networks. Michael Collins. (optional) A Primer on Neural Network Models for Natural Language Processing. Yoav Goldberg.
	1/23	Tutorial (optional): Building text classifiers with word embeddings
	1/24	Lecture: Word Representations and neural language models [WV slides 2]		Vector Semantics and Embeddings (J+M chapter 6) (optional) Improving Distributional Similarity with Lessons Learned from Word Embeddings. Omer Levy, Yoav Goldberg, and Ido Dagan.
	1/26	Tutorial (optional): Building text classifiers with pretrained word embeddings
3
	1/29	Lecture: Sequence modeling (HMMs) [slides]		Hidden Markov Models (J+M Appendix A.1-A.4)
	1/30	Tutorial (optional): Neural language models with BoW/Fixed Window FFN
	1/31	Lecture: Neural sequence modeling (RNNs) [slides]	Due: HW1	Sequence Labeling for Parts of Speech and Named Entities (J+M chapter 8) RNNs and LSTMs (J+M chapter 9) (optional) Understanding LSTM Networks. Christopher Olah.
	2/2	Tutorial (optional): Using RNNs in pytorch
4
	2/5	Lecture: Neural Sequence Modeling (LSTM/GRU) and sequence-to-sequence models [slides] [slides]
	2/7	Lecture: Attention in sequence-to-sequence models [slides]		Transformers and Large Language Models (J+M chapter 10) (optional) Attention Is All You Need. Vaswani et al. 2017. (optional) The Annotated Transformer. Alexander Rush.
5
	2/12	Lecture: Transformers and contextualized word embeddings [transformer slides] [cwe slides]		Fine-tuning and Masked Language Models (J+M chapter 11) (optional) Illustrated BERT. Jay Alammar. (optional) Contextual Word Representations: A Contextual Introduction. Noah A. Smith. (optional) BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Devlin et al. 2018. (optional) Emergent linguistic structure in artificial neural networks trained by self-supervision. Manning et al. 2020. (optional) A Primer in BERTology: What We Know About How BERT Works. Rogers, Kovaleva, and Rumshisky.
	2/14	Lecture: Benchmark datasets [project info slides] [benchmark slides]	Due: HW2
7
	2/19	No class - No class - Reading break
	2/21	No class - No class - Reading break
8
	2/26	Lecture: Pretraining and fine-tuning [pretraining slides] [fine-tuning slides]		HuggingFace PEFT: Parameter-Efficient Fine-Tuning of Billion-Scale Models on Low-Resource Hardware. Sourab Mangrulkar, Sayak Paul". (optional) Prefix-Tuning: Optimizing Continuous Prompts for Generation. Xiang Lisa Li, Percy Liang. (optional) AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning. Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, Jianfeng Gao. (optional) LoRA: Low-Rank Adaptation of Large Language Models. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen. (optional) Adapter methods. docs.adapterhub.ml.
	2/28	Lecture: Few-shot and in-context learning [slides]	Due: HW3	Language Models are Unsupervised Multitask Learners. Open AI. Language Models are Few-Shot Learners. Open AI. GPT-4 Technical Report. Open AI. Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, Luke Zettlemoyer. In-Context Learning Learns Label Relationships but Is Not Conventional Learning. Jannik Kossen, Yarin Gal, Tom Rainforth.
9
	3/4	Lecture: Constituency Parsing [slides]	Due: Project proposal	Constituency Grammars (J+M chapter 17) (optional) Statistical Constituency Parsing (J+M chapter Appendix C.1-C.4)
	3/6	Lecture: Dependency parsing [slides]		Dependency Parsing (J+M chapter 18)
9
	3/11	Lecture: Parameter-efficient fine-tuning and instruction tuning [fine-tuning slides] [instruction-tuning slides]
	3/13	Lecture: Final project tips, model debugging and analysis [slides]	Due: HW4
11
	3/18	Lecture: NLP applications [slides]
	3/20	Lecture: Scaling laws for LLMs [slides]
	3/21		Due: Project milestone
12
	3/25	Lecture: Grounding [slides]
	3/27	Lecture: LLM Agents [slides]
13
	4/1	No class - No class - Easter
	4/3	Guest Lecture by Nick Vincent: Societal Impacts of Generative AI: Economic Factors and More [slides]
14
	4/8	Lecture: Final project presentations
	4/10	Lecture: Conclusion [slides]
	4/11		Due: Final project report