Seminar in natural language processing. Topics vary by year. This is a seminar for advanced undergrads and graduate students who have prior background in natural language processing and would like to advance their knowledge and develop research skills.
The field of NLP has been going through a revolution in the last decade, with the advent of neural network approaches. The winter 2021 iteration of the seminar will explore advances in NLP, from fine-tuning to prompt-tuning. We will read key papers in the field, starting with representation learning work on static word embeddings, through contextualized word representations, to fine-tuning large language models, and culminating with prompt-based zero-shot and few-shot settings for getting massive language models to perform various tasks.
Staff:<o:p></o:p>
Yonatan Belinkov, instructor (belinkov@technion.ac.il) <o:p></o:p>
Office hours: by appointment. <o:p></o:p>
Learning outcomes: <o:p></o:p>
By the end of this course, the student will be able to:<o:p></o:p>
- Present, discuss, and critically assess research papers in NLP. <o:p></o:p>
Pre-requisites: <o:p></o:p>
This is an advanced course that assumes background in NLP and machine learning, such as can be acquired via the following courses: 236299: Introduction to natural language processing, 236756: Introduction to machine learning, 236781: Deep learning on computation accelerators. The formal requirements are 236299 plus either 236756 or 236781, and they will be enforced. It is preferable to have taken all three, although students without the official requirements will still be considered if they can demonstrate sufficient background.
Coursework:<o:p></o:p>
- Read papers before each class. <o:p></o:p>
- Post reading responses (1-2 paragraph long) about the weekly papers. The response should pose questions and comments about the paper that would be appropriate for discussion in class. (The responses should not summarize the paper.)<o:p></o:p>
- Lead discussions of 1-2 papers throughout the course. Discussion leaders will read the responses from the entire class beforehand and synthesize them when leading the discussion. <o:p></o:p>
- Contribute to the BigScience Workshop. This is a small task that may take different forms, such as writing a new prompt for a dataset, performing some evaluation, summarizing a research area, or analyzing a dataset.<o:p></o:p>
Language of instruction:<o:p></o:p>
The course materials are in English. The language of instruction and discussion is Hebrew. <o:p></o:p>
Grading:<o:p></o:p>
The course grade will be comprised approximately as follows; the exact grade composition may change.<o:p></o:p>
- Reading responses 10%<o:p></o:p>
- Attendance & participation 10%<o:p></o:p>
- Leading a discussion 65%<o:p></o:p>
- BigScience contribution 15%<o:p></o:p>
Schedule:<o:p></o:p>
The course meets weekly on Mondays at 14:30-16:30.<o:p></o:p>