CMPT 983 (Spring 2021): Special topics in Artificial Intelligence - Grounded Natural Language Understanding


This course is a graduate-level, seminar-oriented research course covering topics at the intersection of language, vision, graphics, and robotics. The class focuses on the grounding of language to various representations and modalities. Students are expected to have prior experience with deep learning concepts and framework (Pytorch, Tensorflow, etc), and should also have familiarity with at one of the following areas: natural language processing, vision, graphics or robotic.

Each week, students will read papers in a particular area of language grounding, and discuss the contributions, limitations and interconnections between the papers. Students will also work on a research project during the course, culminating in a final presentation and written report. The course aims to provide practical experience in comprehending, analyzing and synthesizing research in grounded natural language understanding.

Note: This course is NOT an introductory course to natural language processing. If you are interested in learning about natural language processing, CMPT 413/825 is offered in the fall.


There are no formal prerequisites for this class. However, you are expected to be familiar with the following:

For some topics that we will cover in the class, it is also helpful to be familiar with:


Quick info


Below is a tentative outline for the course.

Date Topic Notes
Jan 11 Introduction to grounding & logistics  
Jan 14 How to read papers & project overview  
Jan 18 Review of basic deep learning models  
Jan 21 Multimodal embeddings  
Jan 25 Paper discussion 1  
Jan 28 Attention for multimodal grounding  
Feb 1 Paper discussion 2  
Feb 4 Pre-training for multimodal grounding  
Feb 8 Paper discussion 3  
Feb 11 Compositional grounding and structured representations  
Feb 15 No class - Reading break  
Feb 18 No class - Reading break  
Feb 22 Paper discussion 4 Project proposal due
Feb 25 Semantic parsing for grounding  
Mar 1 Paper discussion 5  
Mar 4 Speaker-listener models  
Mar 8 Paper discussion 6  
Mar 11 Language and action (block world)  
Mar 15 Paper discussion 7  
Mar 18 Language and action (visual language navigation)  
Mar 22 Paper discussion 8 Project milestone due
Mar 25 Language and vision and robotics  
Mar 29 Paper discussion 9  
Apr 1 Interactive language learning  
Apr 5 Paper discussion 10  
Apr 8 Text conditioned content generation  
Apr 12 Project presentations  
Apr 15 Conclusion Project writeup due


General policies

Academic integrity

SFU’s Academic Integrity web site is filled with information on what is meant by academic dishonesty, where you can find resources to help with your studies and the consequences of cheating. Check out the site for more information and videos that help explain the issues in plain English.

Each student is responsible for his or her conduct as it affects the University community. Academic dishonesty, in whatever form, is ultimately destructive of the values of the University. Furthermore, it is unfair and discouraging to the majority of students who pursue their studies honestly. Scholarly integrity is required of all members of the University. Please refer to this web site.