What are reinforcement learning and Q-learning?

Reinforcement learning is learning by trial and reward. An agent chooses an action, that action changes the environment, and the environment gives back a reward and a new state. Then the loop repeats. Nobody hands the agent an answer key. It just tries things, notices what pays off, and adjusts.

This is Rex, a robot dog in a four-tile hallway. He starts on the second tile. Left is a bath, which costs him five points. Right leads to a treat, worth ten. Each turn, Rex can only move Left or Right, and right now he knows nothing about any of it.

Rex keeps a cheat sheet called a Q-table. Every tile and move gets a score for how good that move is from there. Everything starts at zero.

Say Rex is standing on Middle and happens to move Right. The hallway answers with the treat, plus ten. Rex then writes that into the Q-table for Middle and Right. But the score becomes five, not ten. The learning rate is one half, so Rex only moves halfway toward what it just saw. One step: act, get a reward, update.

Each update moves the old score partway toward reward plus the best next score. That last part lets Rex value tiles that merely lead to treats.

Look at Start-Right. It sits at 2.25, even though stepping from Start to Middle paid nothing. Middle looks promising, so some of that value flows backward. Start-Left sits at minus 2.5, the bath. Rex tries, gets a reward, updates, and the treat's value spreads.

If the treat were worth twenty instead of ten, what would Rex's first Middle-Right score become?

Ten. The update moves the score only halfway toward the target, because the learning rate is one half. Halfway from zero to twenty is ten. It does not jump straight to the full reward in one try.

Once Rex finds a decent path, he faces a dilemma. Exploit it forever, and he might never discover a bigger treat somewhere else. So sometimes he picks a random move instead. That's called epsilon-greedy. Early on he explores a lot, then less and less as his scores become trustworthy.

Reinforcement learning is a loop. Try a move, get a treat or an ouch, then nudge that move's score. Repeat until Rex heads Right without thinking. A table works in a tiny hallway, but in huge worlds, neural networks replace it.