Reinforcement Learning

Spring 2024 - ECE 457C

ECE 457C - Reinforcement Learning

Offered Spring 2024 by Prof. Mark Crowley

Course Description

Introduction to Reinforcement Learning (RL) theory and algorithms for learning decision-making policies in situations with uncertainty and limited information. Topics include Markov decision processes, classic exact/approximate RL algorithms such as value/policy iteration, Q-learning, State-action-reward-state-action (SARSA), Temporal Difference (TD) methods, policy gradients, actor-critic, and Deep RL such as Deep Q-Learning (DQN), Asynchronous Advantage Actor Critic (A3C), and Deep Deterministic Policy Gradient (DDPG).

Course Outline

See the official course outline at : https://outline.uwaterloo.ca/view/n9ffkw for course location, times, staff contact and other information .

This page will have additional resources linking to previous courses, topic notes etc, which may also be duplicated on LEARN.

Course News and Announcements

These will generally be posted into LEARN so that everyone can get a notification of announcements and updates. Be sure to enable notifications for course announcements.

2024-04-26

The course will have a Midterm Exam and a Final Exam this year. The Midterm is already scheduled :

  • Midterm Exam: June 17, 11am to 12:15pm, in room E7 4043
  • Full ECE Midterm Schedule : https://uwaterloo.ca/electrical-computer-engineering/midterm-schedule-0

Welcome, Let’s Talk AI!

First Lecture : May 6,

An Erratic Planned Lecture Schedule

Due to having Monday lectures, and a large number of holidays, conferences, and other events, our “regular weekly schedule” is far from regular.

  • “Regular” week: Two 1.5 hour lectures a week on Mondays and Tuesdays.
  • ButSome weeks we have a total of 1.5 hours of lecture, some weeks we have 6 hours
  • Notably, the very first week we have
    • Monday (in person): 3 hours all at once on May 6 on the first day of class
    • Tuesday (cancelled): 1.5 hours are lecture are scheduled for May 7 but I will need to cancel it, see note below.
    • Wednesday (virtual): 1.5 hours Since we’ll be losing lecture in other weeks we will use the full hours in this first week to get off to a quick start!

This means we’ll be jumping right in with full content that first day. We will certainly have a break and some ice breaker discussions to let people chat, and discuss group partners, and ask general course questions.

So come to class Monday and be ready to talk about the big topics in Artificial Intelligence!

Regular is as Regular Does…

![[457C-S24-totalhours.png]] ![[457C-S24-hoursbyday.png]]

Additional Lecture Complications

In addition to the planned schedule, from time to I will need to cancel a lecture, or provide it virtually with as a live online lecture on Teams. This is partly because of a family health situation that requires me to be in Toronto a fair bit this term. One or two weeks I’ll have be attending a conference and will switch to some lecture being virtual-live or virtual-pre-recorded.

If you have any concerns or inputs on the how I can best do all of this in a way that supports your learning, please do contact me.

Raw Data

| Week | Date | Normal Hours | Actual Hours | Monday Hours | Tuesday Hours | Wednesday Hours | | —- | —- | —- | —- | —- | —- | —- | | 1 | May 6, 2024 | 3 | 6 | 3 | 1.5 | 1.5 | | 2 | May 13, 2024 | 3 | 3 | 1.5 | 1.5 | 0 | | 3 | May 20, 2024 | 3 | 1.5 | 0 | 1.5 | 0 | | 4 | May 27, 2024 | 3 | 6 | 3 | 1.5 | 1.5 | | 5 | June 3, 2024 | 3 | 3 | 1.5 | 1.5 | 0 | | 6 | June 10, 2024 | 3 | 3 | 1.5 | 1.5 | 0 | | 7 | June 17, 2024 | 3 | 0 | 0 | 0 | 0 | | 8 | June 24, 2024 | 3 | 3 | 1.5 | 1.5 | 0 | | 9 | July 1, 2024 | 3 | 1.5 | 0 | 1.5 | 0 | | 10 | July 8, 2024 | 3 | 4.5 | 1.5 | 1.5 | 1.5 | | 11 | July 15, 2024 | 3 | 4.5 | 3 | 1.5 | 0 | | 12 | July 22, 2024 | 3 | 3 | 1.5 | 1.5 | 0 | | 13 | July 29, 2024 | 3 | 3 | 1.5 | 1.5 | 0 |

Course Communication

Hi Everyone! Looking forward to another fun spring term talking about one of the most interesting areas of modern AI these days, Reinforcement Learning! We’ll be using Piazza and Learn for discussion and content, you know the drill by now! News and announcements will be posted on Learn and piazza, as well as initially on my website for the course (https://markcrowley.ca/rlcourse/) for people who aren’t registered yet.

Boilerplate: We’ll be conducting all class-related discussion here this term. The quicker you begin asking questions on Piazza (rather than via emails), the quicker you’ll benefit from the collective knowledge of your classmates and instructors. We encourage you to ask questions when you’re struggling to understand a concept. For course communications, you can even do so anonymously to the course staff and only the TAs and Prof will know who it is.

Important: For direct contact with the Prof or TAs on individual issues on anything personal or administrative, please use private piazza messages so that all three of us get a chance to respond. Prof. Crowley is “bad at email” (…it’s true), so it might get missed. If you really have a critical time issue, or you don’t want the TA’s to see the message (for any reason at all), then you should send Prof. Crowley a direct message on Teams chat. But please use this method only when critical.

  • Prof. Mark Crowley