11/15/04

Today we had a two-hour meeting on intrinsic motivation with the primary question appearing to be "Is there more to intrinsic motivation than merely the desire to maximize long-term reward?" A number of ideas for motivating the agent to explore, to learn to predict, to learn to control its sensation, etc., etc. have been brain-stormed. Rich made a list of behaviors/tasks such as "learn to predict and control salient events", "learn about surprising or novel events", etc.

I would like to discuss (i) whether the question "Is there really intrinsic reward/motivation and if so then what is it?" makes sense and (ii) whether it actually matters at all. I think the fundamental question which we can never avoid is "What is the grand objective of an AI agent?" This seems to be a very important and foundational question in AI. RL answers this question as "The objective is to maximize the long-term scalar reward signal supplied by the environment". Thus, if the reward signal is provided then it is pretty cut and dry and it is exactly maximizing it that measures everything up. For instance: Does learning feel good? Well, only as long as it helps maximizing returns. Does gaining control over agent sensations feels good? Well, only if it helps maximizing the returns, and so on. Maximizing long-term reward and only it makes anything useful.

Thus, it seems to me that in the RL framework, the questions of existence of the intrinsic reward and its definition are really in the eye of the beholder. It appears similar to Rich's answer to "What is the goal of an agent?" --- "It is whatever helps us (the observers) to predict the agent's behavior". We, the observers, attribute a goal to an agent. By the same token, I would like to propose that intrinsic motivation is something that we, the observers, can attribute to the agent if it helps us explain/predict its behavior. Thus, the question of whether the agent is intrinsically motivated is somewhat meaningless.

Summary:
1. if we are observing an agent then _we_ attribute intrinsic motivation to it to help us explain/predict its behavior;
2. if we are building an RL agent and already have a reward function for it then introducing intrinsic motivation is similar to giving it shaping rewards, providing an initial Q function, creating a state abstraction, engineering options, etc. All of these are _means_ to make maximizing the "real" long-term reward faster and more efficiently;
3. so, perhaps, instead of asking whether intrinsic motivation "really" exists and what it is, we should ask what we can add to the agent's learning mechanism to make the agent maximize its reward faster and more efficiently. That is the only goal an RL agent pursues. Everything else is a means for it.

Thoughts?

-Vadim 

I like Vadim's summary above.  I think Point 2 is the most important---intrinsic motivation is a kind of shaping reward that we need to put into our RL systems so that they can learn in extremely difficult domains.  I think Point 1 is valid, although I believe it is secondary.  In particular, saying that "intrinsic motivation is in the eye of the beholder" is akin to saying "gravity is in the eye of the beholder".  In both cases, it is a theoretical construct that we use to explain our observations of the natural world.  But, heck, at some point an explanation is sufficiently compelling that it useful to say that it is in the world.  I'm not saying that intrinsic motivation is necessarily in this category, but I think one could make a strong case that it is.

I've been thinking a bit about how we could formulate an experiment to actually quantify some intrinsic rewards.  For example, there are behavioral experiments that show people giving up money for the opportunity to punish someone who isn't being fair.  I think revenge is an intrinsic reward---sometimes people get an urge to get back at someone even when it goes against other more "rational" rewards.  The other intrinsic motivations mentioned above probably have the same properties---people would be willing to satisfy these motivations, even if it costs money (up to a point).  Putting the various motivations on a scale (money is one, but there could be others), could be extremely valuable in getting a handle on what they are and how they work.

-Michael (11/16/04)

I would like to discuss a particular point Michael made above. Namely, when Michael says:

"I think revenge is an intrinsic reward---sometimes people get an urge to get back at someone even when it goes against other more 'rational' rewards."

he seems to imply that 'rational rewards' are the _extrinsic_ / _real_ rewards in human life. This brings us to one of the most important questions in the 'RL for Human Intelligence' story:

What are the real/external RL rewards in human life?

It appears important to define these for otherwise, we have no basis to talk about intrinsic (i.e., internal) rewards. But could it be that pleasure from revenge is just as external/real to us as pleasure from quenching hunger or thirst?

More generally:

Is there a reward signal whose discounted sum humans try to maximize?

But then, perhaps, this existential question is really meaningless because rewards are in the eye of the beholder. Namely, if it helps us _predict_ what humans do by _ascribing_ them a reward signal (whether extrinsic or intrinsic) and then _conjecturing_ that humans are RL agents, then so be it.

In such a case, it is purely up to us, the observers, to decide which parts of this _ascribed_ reinforcement signal we call extrinsic and which parts we call intrinsic. But if it is only a terminological question then why is it important and why cannot we just resolve it arbitrarily?

Vadim (11/22/04) 

Vadim: By "rational" rewards, I suppose I meant "rewards correlated with survival/success".  It makes sense to say "all rewards are intrinsic" in that o rewards come directly from the outside world.  Maybe the issue we're getting at is whether or not there are rewards that exist primarily to support learning/development.  That is, whereas pain appears to remain a motivator throughout our lives, some people appear to become less motivated by the drive to learn.

So, maybe the distinction is between stationary rewards ("extrinsic") and developmentally non-stationary rewards ("intrinsic").

-Michael (11/25/04)

---

Vadim, i would be interested in your position on the reward hypothesis.

Actually, i think we should all consider that the reward hypothesis in the light of intrinsic motivation. Where one goes from here seems to depend strongly on whether or not one accepts that hypothesis.

-Rich 

Michael:

I like your example of people getting less motivated to learn as they become older. However, do we really need to introduce additional rewards to explain this phenomenon? How about the following intrinsic-reward-free explanation:

People are less eager to learn as they become older simply because of the diminishing returns effect: more learning results in smaller gains in the cumulative "real" reward (i.e., related to their "success/survival"). In fact, even negative gains are possible since exploring already well-known areas of the state-action space is inferior to exploiting them.

This mechanism is similar to decreasing temperature in simulated annealing search and does not require an introduction of additional (intrinsic) rewards.

Vadim (11/25/04).