CMPT 983 (Spring 2023): Special topics in Artificial Intelligence - Grounded Natural Language Understanding

Overview

This course is a graduate-level, seminar-oriented research course covering topics at the intersection of language, vision, graphics, and robotics. The class focuses on the grounding of language to various representations and modalities. Students are expected to have prior experience with deep learning concepts and framework (Pytorch, Tensorflow, etc), and should also have familiarity with at one of the following areas: natural language processing, vision, graphics or robotic.

Each week, students will read papers in a particular area of language grounding, and discuss the contributions, limitations and interconnections between the papers. Students will also work on a research project during the course, culminating in a final presentation and written report. The course aims to provide practical experience in comprehending, analyzing and synthesizing research in grounded natural language understanding.

Note: This course is NOT an introductory course to natural language processing. If you are interested in learning about natural language processing, please take CMPT 413/713.

Background

There are no formal prerequisites for this class. However, you are expected to be familiar with the following:

• Natural language Processing
• Deep learning

For some topics that we will cover in the class, it is also helpful to be familiar with:

• 3D representations
• Reinforcement Learning

Topics

• Grounded language acquisition and understanding
• Grounding of language to machine interpretable programs (semantic parsing)
• Visual grounding of language and tasks (captioning, VQA models, referring expressions)
• Interpretation of language commands for embodied navigation and interaction
• Interactive language learning through language games and dialogue
• Grounded knowledge representations for mapping language to the 3D world
• Generative models for content creation from text

Quick info

• Instructor: Angel Chang
• TA: Han-Hung Lee
• Lectures: Mondays 10:30am to 11:50am (AQ5029) and Wednesdays 10:30am to 11:50am (AQ5019).
• Office hours:
  • Angel: Wed 3pm to 4pm
  • Han-Hung: Mon 1pm to 2pm

Syllabus

Below is a tentative outline for the course.

R: Readings, BG: (Optional) Background material / reading for deeper understanding. Provided for reference.

Date	Topic	Notes
Jan 4	Introduction to grounding & logistics [slides]	BG: The Symbol Grounding Problem BG: Six lessons from babies V: How language shapes the way we think
Jan 4	How to read papers & project overview [slides]	BG: How to read a paper
Jan 9	Review of basic deep learning models [slides]	BG: Deep learning BG: Contextual word representations
Jan 11	Multimodal embeddings [slides]	BG: Multimodal Machine Learning BG: Contrastive learning
Jan 16	Paper discussion 1	R: ViCo: Word Embeddings from Visual Co-occurrences R: CLIP
Jan 18	Attention for multimodal grounding [slides]	BG: Attention? Attention!
Jan 23	Paper discussion 2	R: Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering R: FiLM: Visual Reasoning with a General Conditioning Layer
Jan 25	Vision-and-language pre-training with transformers [slides]	BG: Attention Is All You Need BG: The Illustrated Transformer
Jan 30	Paper discussion 3	R: Vilbert R: OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-sequence Learning Framework
Feb 1	Text conditioned content generation [slides]	BG: Generative models BG: Text to image survey BG: 3D generative models
Feb 6	Paper discussion 4	Project proposal due R: Hierarchical Text-Conditional Image Generation with CLIP Latents (DALLE-2) R: DreamFusion: Ttext-to-3D using 2D Diffusion
Feb 8	Project proposal presentations
Feb 13	Compositional grounding and structured representations [slides]	BG: To Understand Language is to Understand Generalization BG: Linguistic generalization and compositionality in modern artificial neural networks BG: Relational inductive biases, deep learning, and graph networks
Feb 15	Semantic parsing for grounding [slides]	BG: Semantic parsers BG: Language to Logical Form with Neural Attention
Feb 20	No class - Reading break
Feb 22	No class - Reading break
Feb 27	Paper discussion 5	R: Learning to compose neural networks for question answering R:
Mar 1	Instruction following (intro to RL) [slides]	BG: Experience Grounds Language BG: Extending machine language models toward human-level language understanding BG: Deep RL BG: A (Long) Peek into Reinforcement Learning
Mar 6	Paper discussion 6	Project milestone due R: R:
Mar 8	Instruction following (visual language navigation) [slides]	BG: Visual language Navigation
Mar 13	Paper discussion 7	R: REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments R: Visual Language Maps for Robot Navigation
Mar 15	Instruction following (rearrangement) [slides]	BG: Rearrangement
Mar 20	Paper discussion 8	R: CLIPORT: What and Where Pathways for Robotic Manipulation
Mar 22	Instruction following (planning + common sense)
Mar 27	Paper discussion 9
Mar 29	Speaker-listener models [slides]	BG: Rational Speech Acts
Apr 3	Interactive language learning [slides]	BG: Power to the people
Apr 5	Project presentations and conclusion [slides]
Apr 10	No class - Easter	Project writeup due

Grading

• 35% paper reading and reports
• 10% paper presentations
• 15% paper discussions and participation
• 40% course project
  • 10% proposal: 5% presentation + 5% written proposal
  • 10% milestone: 5% presentation + 5% report
  • 10% final presentation
  • 10% final report

General policies

Academic integrity

SFU’s Academic Integrity web site is filled with information on what is meant by academic dishonesty, where you can find resources to help with your studies and the consequences of cheating. Check out the site for more information and videos that help explain the issues in plain English.

Each student is responsible for his or her conduct as it affects the University community. Academic dishonesty, in whatever form, is ultimately destructive of the values of the University. Furthermore, it is unfair and discouraging to the majority of students who pursue their studies honestly. Scholarly integrity is required of all members of the University. Please refer to this web site.