Reference books

• There is no official textbook for the course, but if you would like to read further about NLP, here are some good reference books on NLP:
  • Speech and Language Processing by Dan Jurafsky and James Martin.
  • Natural Language Processing by Jacob Eisenstein
  • A Primer on Neural Network Models for Natural Language Processing by Yoav Goldberg (see also Neural Network methods for Natural Language Processing).
• To learn more about deep learning
  • Deep learning by Ian Goodfellow and Yoshua Bengio and Aaron Courville
  • Dive into deep learning by Aston Zhang, Zachary C. Lipton, Mu Li, and Alexander J. Smola

Debugging and learning about neural networks

• Neural Networks: Zero to Hero from Andrej Karpathy
• General tips from Andrej Karpathy for training neural networks
• Tips from Graham Neubig (CMU CS11-747) for debugging NLP models

Podcasts

• NLP highlights with Matt Gardner, Pradeep Dasigi, and Waleed Ammar

Learning about LLMs and transformers

• Andrej Karparthy videos on LLMs
  • Introduction
  • Deep dive
  • Using LLMs
• Jay Alammar’s illustrated guides
  • Transformers
  • GPT-2
  • BERT
• Borealis tutorials
  • Transformers
  • LLMs

Tools

• Deep learning for NLP
  • TorchText
• Sequence models
  • FairSeq
• Static Word Embeddings
  • Word2Vec
  • GloVe
  • FastText
• Pretrained language models
  • HuggingFace course on transformers
    • HuggingFace transformers Docummentation for various transformer based models from HuggingFace
  • OpenAI API
  • Llama
• Using LLMs
  • Prompting Guide
  • LangChain / LangGraph for building LLM agents
• Machine Translation
  • SFUTranslate SFU NLL’s MT toolkit (designed to be modern and flexible)
  • JoeyNMT Minimalist system for understanding NMT
  • OpenNMT Comprehensive NMT framework
• NLP pipelines
  • AllenNLP [demo] Modern NLP tools using Pytorch from AI2
  • Spacy.io [demo] Popular python based NLP pipeline
  • Stanza [demo] Python NLP library from Stanford
  • Stanford CoreNLP [demo] Java NLP pipeline from Stanford
  • NLTK [demo] [book] Useful basic Python tools for NLP

Tasks and datasets

A list of shared task datasets are provided below. In some cases you can also extend your homework code to produce innovative project ideas for these tasks.

Shared Task Collections

• Torchtext
• Sebastian Ruder’s curated collection
• Datasets for Natural Language Processing
• Kaggle NLP Tasks

CoNLL Shared Tasks

• CoNLL Shared Tasks
• CoNLL 2003 Named Entity Recognition Task
• CoNLL 2000 Chunking Task
• CoNLL 2018 Multilingual Parsing

SemEval Shared Tasks

• SemEval 2018
• SemEval 2017
• SemEval 2016
• SemEval 2015
• SemEval 2014

Classification Tasks

• Toxic Comment Classification
• Fake News Challenge
• Spam / Click-bait detection

Information Extraction

• Drug-Drug Interaction Extraction
• Web named entities
• Twitter sequence prediction tasks
• WNUT Emerging and Rare entity recognition shared task
• Gun violence text data

Parsing

• Universal Dependencies
• WikiText
• Opinion mining

Machine Translation

• WMT 2018 Shared Task
• WMT 2017 Shared Task
• WMT 2016 Shared Task
• WMT 2015 Shared Task
• NMT 2018 Neural MT Shared Task
• Web Inventory of Transcribed and Translated Talks

Unlabeled Data for Clustering, Language Models, etc.

• Wikipedia XML data
• Web data
• BootCat

Sentiment and Opinion Mining

• Stanford Sentiment Treebank
• Movie reviews
• Yelp Challenge
• Sentiment and opinion mining datasets

Natural Language Understanding and Inference

• GLUE Benchmark
• RepEval 2017
• The Stanford Natural Language Inference (SNLI) Corpus
• Multi-Genre NLI
• MedNLI
• XNLI

Question Answering

• Qanta shared task
• Algebra Question Answering with Rationales
• Quora Question Pairs.
• Reverse QA: Jeopardy style QA. json and csv
• CoQA
• SQuAD
• HotpotQA