In reinforcement learning, the purpose or goal of the agent is formalized in
terms of a special reward signal passing from the environment to the
agent. At each time step, the reward is a simple number,
. Informally, the agent's goal is to maximize the total amount
of reward it receives. This means maximizing not immediate reward, but
cumulative reward in the long run.
The use of a reward signal to formalize the idea of a goal is one of
the most distinctive features of reinforcement learning.
Although this way of formulating goals might at first appear limiting, in
practice it has proved to be flexible and widely applicable. The best way to
see this is to consider examples of how it has been, or could be, used.
For example, to make a robot learn to walk, researchers have provided reward on each
time step proportional to the robot's forward motion. In making a robot learn how to
escape from a maze, the reward is often zero until it escapes,
when it becomes 
. Another common approach in maze learning is to give a reward of
for every time step that passes prior to escape; this encourages the
agent to escape as quickly as possible. To make a robot learn to find and
collect empty soda cans for recycling, one might give it a reward of zero most of
the time, and then a reward of 
for each can collected (and confirmed as empty). One
might also want to give the robot negative rewards when it bumps into things or
when somebody yells at it. For an agent to learn to play checkers or chess, the
natural rewards are 
for winning, 
for losing, and 0 for drawing and for
all nonterminal positions.
You can see what is happening in all of these examples. The agent always learns to maximize its reward. If we want it to do something for us, we must provide rewards to it in such a way that in maximizing them the agent will also achieve our goals. It is thus critical that the rewards we set up truly indicate what we want accomplished. In particular, the reward signal is not the place to impart to the agent prior knowledge about how to achieve what we want it to do.3.4For example, a chess-playing agent should be rewarded only for actually winning, not for achieving subgoals such taking its opponent's pieces or gaining control of the center of the board. If achieving these sorts of subgoals were rewarded, then the agent might find a way to achieve them without achieving the real goal. For example, it might find a way to take the opponent's pieces even at the cost of losing the game. The reward signal is your way of communicating to the robot what you want it to achieve, not how you want it achieved.
Newcomers to reinforcement learning are sometimes surprised that the rewards--which define of the goal of learning--are computed in the environment rather than in the agent. Certainly most ultimate goals for animals are recognized by computations occurring inside their bodies, for example, by sensors for recognizing food, hunger, pain, and pleasure. Nevertheless, as we discussed in the previous section, one can redraw the agent-environment interface in such a way that these parts of the body are considered to be outside of the agent (and thus part of the agent's environment). For example, if the goal concerns a robot's internal energy reservoirs, then these are considered to be part of the environment; if the goal concerns the positions of the robot's limbs, then these too are considered to be part of the environment--that is, the agent's boundary is drawn at the interface between the limbs and their control systems. These things are considered internal to the robot but external to the learning agent. For our purposes, it is convenient to place the boundary of the learning agent not at the limit of its physical body, but at the limit of its control.
The reason we do this is that the agent's ultimate goal should be something over which it has imperfect control: it should not be able, for example, to simply decree that the reward has been received in the same way that it might arbitrarily change its actions. Therefore, we place the reward source outside of the agent. This does not preclude the agent from defining for itself a kind of internal reward, or a sequence of internal rewards. Indeed, this is exactly what many reinforcement learning methods do.