RLAI open page

	Reinforcement Learning and Artificial Intelligence (RLAI)
	Versions of RL Toolkit and Individual Tools

This page describes the changes between different versions of the RL toolkit and individual tools. Each new version was saved in a zip or tarball, and may be redownloaded from this page if necessary.
To figure out which version you have, check the readme.txt file in the RLtoolkit folder. Newer versions of the toolkit will have a date and version number in that file. If there is no date or version number, check the date on the file itself and compare it to the change dates listed below to decide which version you have.

November 8, 2011:

Uploaded a version of the toolkit obtained frome Anna Koop
Fixed a small bug that prevented the lines between cells in the gridworld from displaying properly
Updated the installation and usage instructions, and the ambition statements

-rich

January 27, 2005: (RLtoolkit 1.0 b6) located in RLtoolkit1.0b6.tgz

New versions of tools Tiles and G and Quickgraph (as below)
The demo folder has been changed to examples
There are new ways of importing the whole toolkit and specific tools within it. See the Usage section on the toolkit page (here) for new instructions.

January 25, 2005: Quickgraph code updated, located in Quickgraph050125.tgz

New version of Quickgraph put on the Quickgraph webpage - the new version will be included in the next RLtoolkit

There is a new function graphPointsOnly which allows a user to have a graph with the points from the data displayed, rather than lines through the points.
Quickgraph is set up to use the RLtoolkit version of g. If not available, it uses its own. This prevents problems with having two versions of g available (and possibly loaded).

January 25, 2005: G code updated, located in G050125.tgz

New version of G put on the G webpage - the new version will be included in the next RLtoolkit

The coordinate and viewport functions are now more robust and consistent.
The function gMainloop has been changed to gStartEventLoop (for backward compatibility, gMainloop still works though)
A set of testing files has been started. These can serve as progression/regression tests as well as user examples.
The code has been cleaned up to make it easier to maintain and more consistent and robust in action.

January 25, 2005: Tiles code updated, located in Tiles050125.tgz and CTiles050125.tgz

New version of tiles put on the Tiles webpage - the new version will be included in the next RLtoolkit

The tile function names have changed: getTiles is now tiles, loadTiles is now loadtiles, getTilesWrap is now tileswrap, and loadTilesWrap is now loadtileswrap. Arguments to these functions have not changed, nor has CollisionTable.
Some improvements have been made to speed up the Python version of tiles
A Python calling c version of tiles has been added.
There is now a "fancytiles" module consisting of functions which will give you sizes and shapes of tiles other than the basic square tile.
There is a tiles demo which is used for testing tiles as well as for illustrating the different tile shapes.
Timing code is available as well.

November 20, 2004 (RLtoolkit 1.0 b5): located in RLtoolkit1.0b5.tgz
                                   Tile coding: tiles.py, tiles.html
                                   G: g.py
                                   Quickgraph: quickgraph.tgz, graph .html

The gridworld has undergone a major revision.

The filenames have all changed and some new modules added.
The methods used for redrawing the gridworld squares have been broken up into several so that you can easily rewrite one (or make a new object and add that one method) to change some action without having to do the whole mess - the methods are squareDrawColor, squareDrawStartGoal, squareDrawArrows, squareDrawWalls, and squareDrawMore - the last one is so you can easily add your own drawing stuff (I use that on the Object gridworld). Many functions have been turned in to methods, so if you have code based on the older gridworld, you may have to make some changes. But hopefully this will be easier to work with in the long run.
Graphical objects used in displaying the gridworld are saved and deleted before being redrawn, which makes it faster.
There is a new type of gridworld being handled - with permanent and consumable objects or rewards. With the gui interface you can add or remove these objects and set their values (positive or negative). When starting the demos, use demos('gwog','run') to get the object gridworld, and demos('gwg','run') to get the standard gridworld.

Objects can now added by clicking on the gridworld. There is a button at the bottom which toggles clicks on the gridworld from add/remove barrier (blue square), to add/remove object.
The current object value is displayed at the bottom of the display. The + and - buttons on either side change the object value. The permanent/consumable button next to the + toggles between adding consumable objects (pie shaped - they disappear when the agent lands on them), and permanent objects (circles).
To change the increment, use the Object menu. You can also change the range of values for the objects. This only affects how green or red they will get - if the range is between -1 and 1 (the default), an object with a reward value of -1.0 will be the darkest red available. If the range is between -10 and 10, it will be a quite light shade of red.
Object gridworlds may be saved just like regular ones, and the objects are saved too. Use File->Save As or Save and File->Open to read and write them.
The Gridworld menu applies to object/reward gridworlds as well as the regular ones. Whichever one you start up with, you will stay with even if you read or make new ones.

Gridworld agents with traces still aren't working yet, so don't use the sarsa lambda or q lambda agents. The others seem to be working fine (dyna, sarsa, Q).

GUI demos now exit as expected when all windows are closed and can be run again with the demos command. Even the function approximation demo.
"from RLtoolkit import *" still imports all the main tools, but you must now prefix them with the module names. This is so that a single module can be imported (e.g. "from RLtoolkit.g import *" without getting everything). To compensate for having to put the module name in front, there is a new module called setup. If you do "from RLtoolkit.setup import *" you should get the tools loaded and be able to use them with no module names in front.
There is a problem with running the nonGUI demos with the demos function if you use "from RLtoolkit.setup import *". The specific commands needed to run the demo are still printed, but can only be used if you prefix RLtoolkit.demoload in front of them. Not very convenient. Try opening the RLtoolkit/demoload file in IDLE and pressing F5 to get around this problem until we get it fixed.

October 20, 2004 (RLtoolkit 1.0 b4): located in RLtoolkit1.0b4.tgz

The demos have been reworked to separate agents and environments, and the files have all been renamed. The demo folder is now demo instead of demos. The gridworld demo is now in folder gridworld instead of Gridworld. If you have been working with one of the demos, be warned that all the filenames are different in this new version.
The GUI version of the gridworld demo has been sped up. You can also change the exploration bonus and initial values from the Agent Details menu. Note that the change agent learning method does not seem to be working properly - I am going to make it build a new agent with the learning method instead. Look for that in the next update.
There is a new demoload file (automatically loaded if you do from RLtoolkit import *, otherwise do from RLtoolkit.demoload import *). This implements a simple demo loading program which lists available demos, gives information on them, and loads them. For GUI demos, this will start them automatically. For nonGUI demos, help information telling how to run the demo is automatically printed. The new function is called demos. If you call it with no arguments demos() it lists available demos. If you call it with a demo name only demos('gwn') it prints more information on the demo. If you call it with a demo name and "run" it will load the demo and run it if it has a GUI interface, or print help information on running it for a non GUI demo (e.g. demos('gwg', 'run') demos('maint','run')

October 12, 2004 (RLtoolkit 1.0 b3): located in RLtoolkit1.0b3.tgz

Started work on separating GUI and non GUI versions of toolkit, and sections of toolkit that may
   be run independently (like quickgraph).
Added initialization so that
            from RLtoolkit import *
   loads the utilities and tools.

September 28, 2004 (RLtoolkit 1.0 b2): located in RLtoolkit1.0b2a.zip
    The graphical utilities g, graph, and graph3d have been added to the toolkit instead of being in a
        separate package
    A new version of naive traces has been implemented, using dictionaries instead of arrays. It is much faster
        than the old one.
    The wrap around version of tiles has been added.
    Some minor changes have been made to graph (mostly bug fixes):
       GDEVICE will be used to keep track of graphs rather than _graphs
       You can now close a graph window and start a new one without an error
       Fixed problem with too many decimal places appearing in y axis labels
       Updated documentation

September 14, 2004 (RLtoolkit 1.0 b1): located in RLtoolkit1.0b1.zip

Extend this Page How to edit Style Subscribe Notify Suggest Help This open web page hosted at the University of Alberta. Terms of use 1955/0