This course is a graduate-level, seminar-oriented research course covering topics at the intersection of language, vision, graphics, and robotics. The class focuses on the grounding of language to various representations and modalities. Students are expected to have prior experience with deep learning concepts and framework (Pytorch, Tensorflow, etc), and should also have familiarity with at one of the following areas: natural language processing, vision, graphics or robotic.
Each week, students will read papers in a particular area of language grounding, and discuss the contributions, limitations and interconnections between the papers. Students will also work on a research project during the course, culminating in a final presentation and written report. The course aims to provide practical experience in comprehending, analyzing and synthesizing research in grounded natural language understanding.
Note: This course is NOT an introductory course to natural language processing. If you are interested in learning about natural language processing, CMPT 413/825 is offered in the fall.
There are no formal prerequisites for this class. However, you are expected to be familiar with the following:
For some topics that we will cover in the class, it is also helpful to be familiar with:
Below is a tentative outline for the course.
| Date | Topic | Notes |
|---|---|---|
| Jan 11 | Introduction to grounding & logistics | |
| Jan 14 | How to read papers & project overview | |
| Jan 18 | Review of basic deep learning models | |
| Jan 21 | Multimodal embeddings | |
| Jan 25 | Paper discussion 1 | |
| Jan 28 | Attention for multimodal grounding | |
| Feb 1 | Paper discussion 2 | |
| Feb 4 | Pre-training for multimodal grounding | |
| Feb 8 | Paper discussion 3 | |
| Feb 11 | Compositional grounding and structured representations | |
| Feb 15 | No class - Reading break | |
| Feb 18 | No class - Reading break | |
| Feb 22 | Paper discussion 4 | Project proposal due |
| Feb 25 | Semantic parsing for grounding | |
| Mar 1 | Paper discussion 5 | |
| Mar 4 | Speaker-listener models | |
| Mar 8 | Paper discussion 6 | |
| Mar 11 | Language and action (block world) | |
| Mar 15 | Paper discussion 7 | |
| Mar 18 | Language and action (visual language navigation) | |
| Mar 22 | Paper discussion 8 | Project milestone due |
| Mar 25 | Language and vision and robotics | |
| Mar 29 | Paper discussion 9 | |
| Apr 1 | Interactive language learning | |
| Apr 5 | Paper discussion 10 | |
| Apr 8 | Text conditioned content generation | |
| Apr 12 | Project presentations | |
| Apr 15 | Conclusion | Project writeup due |
SFU’s Academic Integrity web site is filled with information on what is meant by academic dishonesty, where you can find resources to help with your studies and the consequences of cheating. Check out the site for more information and videos that help explain the issues in plain English.
Each student is responsible for his or her conduct as it affects the University community. Academic dishonesty, in whatever form, is ultimately destructive of the values of the University. Furthermore, it is unfair and discouraging to the majority of students who pursue their studies honestly. Scholarly integrity is required of all members of the University. Please refer to this web site.