Indian Institute of Technology Madras
BS Degree Programme · Examination & Study Paper
Reinforcement Learning
MDPs, returns & Bellman equations
Examination Instructions & Guidelines
- Solve on notebook/paper first before checking solutions.
- MCQ: Exactly 1 option is correct. Negative marking applies.
- MSQ: One or more options correct. Select all to score.
- NAT: Numerical response. No negative marking.
- Scientific calculator allowed for all calculations.
- Official answer key is sealed at the end of this sheet.
Read the passage below, then answer Q1–Q2
Based on the above data, answer the given subquestions.
Q1NAT2 Marks·Write the numberWhat is the total reward if the robot collects both treasures and then exits,taking the shortest possible path and never hitting a wall?
Your answer:- Q2MSQ3 Marks·Choose all correct options
- A
- B
- C
- D
- A
Read the passage below, then answer Q3–Q4
Consider a pole-balancing task where the goal is to apply forces to a cart moving along a track to keep a hinged pole from falling. A failure event occurs when the pole falls past a certain angle or the cart moves off the track, after which the pole resets to vertical.
The problem can be formulated either as an episodic task (with episodes ending at failure) or as a continuing task (after each failure, pole gets back into its initial position,
potentially infinite time horizon with discounting).Rewards can be defined as:
• Episodic: Reward +1 for each timestamp without failure, so the return is the number of steps until failure (possibly infinite if balanced forever).
• Continuing: Reward 0 for each timestamp except at each failure, where the reward is −1. Returns are discounted sums of future rewards.
Based on the above data, answer the given subquestions.
Q3MCQ4 Marks·Choose one option
- A
- B
- C
- D
- A
- Q4MCQ4 Marks·Choose one option
How does the return in the episodic, discounted pole-balancing task differ from the continuing formulation with the same reward structure?
- AEpisodic return is a single discounted reward at failure; continuing return is an infinite sum over all future failures.
- BBoth returns are always zero
- CEpisodic return accumulates rewards at every time step; continuing return only at failure.
- DThere is no difference between the two formulations.