Full Schedule (Expected)
Spring 2023 - ECE 457C
This is the expected schedule of topics to be covered in lecture along with the dates for various holiday’s, Assignment release and due dates. The final exam will be scheduled centrally during the exam period, all assignments will be done and handed in by then.
PART I: Fundamentals of Decision Making Under Uncertainty
| Week | Lecture Date | Title | Topics Discussed | Textbook Sections |
|---|---|---|---|---|
| 1 | Monday, May 8, 2023 | Classes Begin | ||
| 1 | Monday, May 8, 2023 | Introduction | Introduction | |
| 1 | Monday, May 8, 2023 | Learning and Decision Making | Learning and Decision Making | |
| 1 | Friday, May 12, 2023 | Multi-Armed Bandits | Multi-Armed Bandits | 1.1, 1.2, 17.6 |
| 2 | Wednesday, May 17, 2023 | Assignment 1 Released | Content up to MDP policy evaluation | |
| 2 | Monday, May 15, 2023 | Regret Minimization, UCB and Thompson Sampling | MAB and More | |
| 2 | Friday, May 19, 2023 | MDP Definitions | Definition and Motivation | 3.1 |
| 2 | Friday, May 19, 2023 | MDP Rewards and Returns | Rewards and Returns | 3.3-3.4 |
| 3 | Monday, May 22, 2023 | Victoria Day Holiday (no classes) | ||
| 3 | Tuesday, May 23, 2023 | Make-up Day for Victoria Day | ||
| 2 | Friday, May 19, 2023 | MDP Value Functions | Value Functions | 3.5-3.6 |
| 3 | Tuesday, May 23, 2023 | Dynamic programming 1 | Dynamic Programming | 4.0-4.1 |
| 3 | Friday, May 26, 2023 | Dynamic programming 2 | PI, VI | 4.2-4.4 |
| 3 | Friday, May 26, 2023 | Dynamic programming 3 | GPI, AsyncVI | 4.4- |
PART II: Iterative Approximately Optimal Methods
| Week | Lecture Date | Title | Topics Discussed | Textbook Sections |
|---|---|---|---|---|
| 4 | Monday, May 29, 2023 | MC- Monte Carlo Methods | Monte Carlo Methods | 5 |
| 4 | Friday, June 2, 2023 | TD1 Temporal Difference Learning | Temporal Difference Learning | 6.0 |
| 4 | Friday, June 2, 2023 | Assignment 2 Released | Requires up to : VI,PI, QL, SARSA, Double Qlearning, ExpSarsa | |
| 4 | Friday, June 2, 2023 | Assignment 1 Due | Content up to MDP policy evaluation | |
| 4 | Friday, June 2, 2023 | TD2 Comparison to MC on Random Walk | MC vs Random Walk | 6.0-6.5 |
| 5 | Monday, June 5, 2023 | TD3 Sarsa and Qlearning | Sarsa and Qlearning | 6.0-6.5 |
| 5 | Friday, June 9, 2023 | TD4 Expected Sarsa and Double Qlearning | Expected Sarsa and Double Qlearning | 6.5? |
| 5 | Friday, June 9, 2023 | ET1 One Step vs Direct Value Updates | One Step TD | 12.1, 12.1 |
| 5 | Friday, June 9, 2023 | ET2 N Step TD Forward View | N step TD Forward View | |
| 6 | Monday, June 12, 2023 | ET3 N step TD backward View | N step TD backward View | |
| 6 | Monday, June 12, 2023 | ET4 Eligibility Trace on Policy | Eligibility Trace on Policy | |
| 6 | Monday, June 12, 2023 | ET5 Eligibility Traces Off Policy | Eligibility Traces Off Policy | |
| 6 | Friday, June 16, 2023 | Value Function Approximation | Value Function Approximation |
PART III: Generalizing States and Policies
| Week | Lecture Date | Title | Topics Discussed | Textbook Sections |
|---|---|---|---|---|
| 7 | Monday, June 19, 2023 | Neural Networks basics and Deep RL as DQN | Deep RL (DQN) | |
| 7 | Friday, June 16, 2023 | Deep Learning Fundamentals | Neural Networks, activation fucntions, loss functions, training, gradient optimizers, regularization methods, CNNs, LSTMs | |
| 7 | Monday, June 19, 2023 | Midterm Week (no midterm, regular classes) | None | |
| 7 | Friday, June 23, 2023 | Assignment 2 Due | Content up to Monday Week 6 | None |
| 7 | Friday, June 23, 2023 | Assignment 3 Released | Content up to Deep RL basics plus review of some papers | None |
| 7 | Friday, June 23, 2023 | Assignment 4 Released | Content up to Monday Week 10 | None |
| 8 | Friday, June 30, 2023 | Policy Gradient Algorithm | Policy Gradient Algorithm, REINFORCE | 13.1,13.2,13.65 |
| 9 | Monday, July 3, 2023 | Canada Day Holiday (no classes) | None | |
| 9 | Friday, July 7, 2023 | REINFORCE, Actor-Critic Algorithm, A3C,A2C | ||
| 10 | Monday, July 10, 2023 | PPO | TRPO, PPO | |
| 10 | Friday, July 14, 2023 | Evaluating RL Algorithms and Double DQN | Evaluation Methods, Double DQN | |
| 10 | Friday, July 14, 2023 | Prioritized Experience Replay and other New Deep RL algorithms | ||
| 10 | Sunday, July 16, 2023 | Assignment 3 Due | Reviewing a Deep RL paper | None |
| 11 | Monday, July 17, 2023 | DPG, DDPG, SAC and Prioritized Experience Replay | DPG, DDPG and SAC | |
| 11 | Monday, July 17, 2023 | AlphaGo and MonteCarlo Tree Search (MCTS) | AlphaGo and MonteCarlo Tree Search (MCTS) | |
| 12 | Monday, July 24, 2023 | RL Next Steps (pre-recorded) | MARL, Hierarchical RL, Supervised and Curriculum Learning, RL competitions, Free Energy, Big Ideas about AI | https://gingkoapp.com/rl-next-steps |
| 12 | Friday, July 28, 2023 | Live from ICML! RLHF and Ongoing Research | (Pre-recorded or streamed) Reinforcement Learning with Human Feedback, other uses of RL for creative solutions. | |
| 13 | Monday, July 31, 2023 | Assignment 4 Due | Comparison of multiple Deep RL implementation on Gymnasium environments. | |
| 13 | Monday, July 31, 2023 | Review Lecture | All topics (TA will lead review) |
End of Classes
| Week | Date | Title | |
|---|---|---|---|
| 13 | Wednesday, August 2, 2023 | Pre-exam Study Day | |
| 14 | Monday, August 7, 2023 | Civic Holiday | |
| 15 | Wednesday, August 9, 2023 at 7:30pm in MC4045 | Final Exam (see LEARN for info) |