I'm using PPO and I looked at the code, however I need a confirmation about what I understood about it.
The agent has a memory size (at least max_timestep * batch_size) which will hold all the episodes trajectory (state, action, reward).
After update_frequency episodes the agent's weights are updated. By default update_frequencyis the same as batch_size. Each update will do multi_steps where each step is clipped by likely_ratio_clipping and the trajectory is got from the memory based on the subsampling_fraction. So for a batch_size = 16, subsampling_fraction = 0.3 and multi_steps = 10: Each update will be 10 steps with each using 16*0.3 = 4.8 episode from the memory. Is that right?
I'm asking this because I want to improve the train speed. And the only place I can think of is the learning_rate, batch_size, multi_steps.
Regarding training speed: An obvious point to start is to reduce the
update_frequency. While this technically means that data will be slightly off-policy (e.g. update after every episode, but still use a batch_size of 16), in practice this is often no problem up to a point (also, PPOs objective accounts for importance sampling). Moreover, I think subsampling_fraction=1.0 may do the job in most cases, but it may be interesting to play around with it (also, recently you can also specify fixed-size subsampled batch-sizes by using integers (256) instead of floats (0.5), if that's desired). I would recommend starting with update_frequency=1 and subsampling_fraction=1.0, and then play around with batch_size (starting with ~10) and multi_steps (starting with 5-10). And of course learning_rate is also important.
I have another question about PPO: If the
update_frequency = 1 and I have 2 parallel environments. When one of these envs terminate the episode, for example the pole dropped in cartpole, the other env will have its actions based on the new updated network? Or it will wait to finish both envs and then update?
There is the
sync_episodes option for the Runner, and if it were desired, this behavior could be changed such that, when using sync_episodes, episodes are terminated at the same time. But I would be surprised if this is very impactful.
After each update_frequency, is the memory cleared?
For example batch_size=8, update_frequency=1, subsampling=1.0. The first episode will be updated without problems, but will the second update use the trajectory from the first in its update? Or the memory is cleared after each update_frequency.
If it is not cleared it means that after the seventh episode the first episode will still be used even tho it is from a policy so far away from the current policy. Is this correct?
(So to be clear: it's not cleared, otherwise frequency and batch size would be equivalent)
hi, is there a way to enable tensorforce use of FP16 precision rather than default FP32 to run faster? along the lines of this: https://www.tensorflow.org/guide/mixed_precision
Hey @nasrashvilg1, there is a way of changing the TF dtype used within Tensorforce, as currently illustrated in the "precision unittest", but as the comment there says, the TF optimizers seem to expect float32 or float64 and not work with float16. Currently, Tensorforce just re-uses the TF versions for the typical first-order optimizers, but obviously one could re-implement them without this constraint. If you only need lower precision for deployment, I would expect that the SavedModel format makes it possible to convert/export to a float16 version of the model.
@AlexKuhnle thanks for your response on this! - qq after 3rd episode my agent which is based on custom gym environment starts executing the same action all (sequence of actions is the same for every episode after the third-fourth episode onwards to the end of model training) what could be the issue if you or anyone else has run into this before?
@chinen93 agent = Agent.create(
agent='tensorforce',
environment=environment, # alternatively: states, actions, (max_episode_timesteps)
memory=10000,
update=dict(unit='timesteps', batch_size=64),
optimizer=dict(type='adam', learning_rate=3e-4),
policy=dict(network='auto'),
objective='policy_gradient',
reward_estimation=dict(horizon=20)
) - can you please guide me into how I can add the entropy regularization and exploration? The agent starts consistently taking the same action after few episodes - I think it might be prematurely converging to optimal policy when more learning needs to be done
agent='tensorforce',
environment=environment, # alternatively: states, actions, (max_episode_timesteps)
memory=10000,
update=dict(unit='timesteps', batch_size=64),
optimizer=dict(type='adam', learning_rate=3e-4),
policy=dict(network='auto'),
objective='policy_gradient',
reward_estimation=dict(horizon=20)
) - can you please guide me into how I can add the entropy regularization and exploration? The agent starts consistently taking the same action after few episodes - I think it might be prematurely converging to optimal policy when more learning needs to be done
Take a look in this doc link about the "tensorforce" agente: https://tensorforce.readthedocs.io/en/latest/agents/tensorforce.html
You can add entropy_regularization, l2_regularization, exploration and other hyperparameters to try to improve your agent. However it depends on your action space
@nasrashvilg1 so try to create your agent as:
Agent.create(
agent='tensorforce',
environment=environment, # alternatively: states, actions, (max_episode_timesteps)
memory=10000,
update=dict(unit='timesteps', batch_size=64),
optimizer=dict(type='adam', learning_rate=3e-4),
policy=dict(network='auto'),
objective='policy_gradient',
reward_estimation=dict(horizon=20),
exploration=3e-4,
entropy_regularization=1e-4,
l2_regularization=1e-4
)
And change the last configurations as needed.
Agent.create(
agent='tensorforce',
environment=environment, # alternatively: states, actions, (max_episode_timesteps)
memory=10000,
update=dict(unit='timesteps', batch_size=64),
optimizer=dict(type='adam', learning_rate=3e-4),
policy=dict(network='auto'),
objective='policy_gradient',
reward_estimation=dict(horizon=20),
exploration=3e-4,
entropy_regularization=1e-4,
l2_regularization=1e-4
)
And change the last configurations as needed.
in the last configuration you mean exploraiton, entropy_regularization and l2_regularization
You can either use the pretrain function (if you have data in the right format, see in the example), or more manually use the experience and update functions. A supervised learning environment probably won't work as well.
@GANdalf2357, you can use the https://github.com/tensorforce/tensorforce/blob/master/examples/save_load_agent.py to see some examples of how to save/load a model. The runner part can be explicit with a while-loop if you need more control.
Hi @GANdalf2357, in addition to what @chinen93 said: inference-only can be done e.g. by using
evaluation=True when using Runner, or act with independent=True and deterministic=True (see here).
@chinen93, the combination of experience and update is a bit like supervised learning, so train the policy distribution to output the corresponding action per state according to the data. This consideration is more or less ignoring the theory around policy gradient and on-policy etc, just looking at the problem from a supervised angle, and this can work (but can also not lead anywhere). The current Tensorforce interface is not ideal, bit too generic and hence may be used wrongly, but I also don't have too much experience with pretraining, behavioral cloning, etc.
The pretrain function is a bit specific and assumes interaction traces including reward, so basically the recorded data of another agent, as in the pretrain example. Experience and update gives more flexibility, but might not be obvious. What data do you have? Individual data points of "expert" state->action decisions? Demonstration traces of state-action pairs? Or full demo traces including reward? Or potentially even more "random" trajectory data, not "expert"/"demonstration"? Depending on which case, applies, there are different possibilities.
is there a lot of value in having discretized space for PPO agent or any other agent in general? I have a continues state space with 5 readings - which can range from 0.0 to 50. 0(float numbers) I can also have all the 5 readings discretized - just trying to decide if it's worth the effort?
Hi all, I'm having a problem with nested tensor specifications. I specified a nested block called "MID_1_bid_0", with a float element "qty", but when I try to run the network I get this error:
ValueError: 'MID_1_bid_0/qty_preprocessing' is not a valid module name. Module names must be valid Python identifiers (e.g. a valid class name).
This is in the state space of the agent.
From looking at the source it appears that the naming convention is indeed to separate parent and child objects with a '/', but then that name containing a '/' gets passed as a module name to tf.Module, which is illegal.
I'm getting this warning when I start a runner, and it's taking 2-3 minutes for the preprocessing to complete before running. Is that normal, and is there a known misuse that would cause this?
/site-packages/tensorflow/python/framework/indexed_slices.py:433: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
@AlexKuhnle I think I have another bug for you. In Agent.py, the first thing fn_act() does is to separate any variable
name + '_mask' from the state spec where name is any name found in the action_spec. It does this recursively using fmap(), calling this function: # Separate auxiliaries
def function(name, spec):
auxiliary = ArrayDict()
if self.config.enable_int_action_masking and spec.type == 'int' and \
spec.num_values is not None:
if name is None:
name = 'action'
# Mask, either part of states or default all true
auxiliary['mask'] = states.pop(name + '_mask', np.ones(
shape=(num_parallel,) + spec.shape + (spec.num_values,), dtype=spec.np_type()
))
return auxiliaryHowever, when I run it I get a KeyError exception, where the key is a root name from action space. I instrumented the line from nested_dict.pop() that threw the error, like so:
3 elif '/' in key:
2 key, subkey = key.split('/', 1)
1 if not key in self:
371 print(f"pop {key} {subkey}")
1 import pprint
2 pprint.pprint(self)
3 value = super().__getitem__(key)
4 assert isinstance(value, self.__class__)
5 return value.pop(subkey, default)This is what it printed:pop MID_1_counter_0 promise_date_mask
{'MID_1_bid_0/price': array([1.15]),
'MID_1_bid_0/promise_date': array([4]),
'MID_1_bid_0/qty': array([44135.2]),
'MID_1_bid_0/supplier_tier': array([0]),
...
Now, in the top level to pop() there is a default value, however, when a nested variable is encountered, the handling for that case doesn't consult the default value, it just says
value = super().__getitem__(key)
Hey, first, the warning
Converting sparse IndexedSlices to a dense Tensor of unknown shape. comes up if you use embeddings (used by "auto" network if state is int), and maybe in other situations as well. I've read a bit about it a while ago, and it doesn't seem to be critical, if e.g. the number of embeddings (num_values of int state) is reasonable. Model initialization may take a while if the network is bigger -- is this the case for you?
And regarding the second issue: I will look into it soon.
@AlexKuhnle Thanks for the tip. Meanwhile, I modified the pop() code in nested_dict to return the default if the super key is not found, (instead of printing debug info), but it led to an invalid-shape error. I think the problem is that if the action is nested then there won't be a shape argument for parent nodes (only leaf nodes have a shape). Given that, what I would like to do is to set
self.config.enable_int_action_masking to False, but I don't see a way to do that... The config object explicitly overrides __set_attr__ and I wasn't able to pass it as a constructor arg to the agent. So, what's the right way to do that?
Setting
enable_int_action_masking can be done via the config argument of any agent (docs here). That should hopefully work.
I've also made the change you suggested to
NestedDict -- would you mind posting the shape exception, since I don't know why that would come up?
Here is the stack trace:
File "train_rl_agent.py", line 77, in run_agent
runner.run(num_episodes=sim_config["train_episodes"])
File "/home/hinrichs/build/tensorforce/tensorforce/execution/runner.py", line 545, in run
self.handle_act(parallel=n)
File "/home/hinrichs/build/tensorforce/tensorforce/execution/runner.py", line 579, in handle_act
actions = self.agent.act(states=self.states[parallel], parallel=parallel)
File "/home/hinrichs/build/tensorforce/tensorforce/agents/agent.py", line 388, in act
deterministic=deterministic
File "/home/hinrichs/build/tensorforce/tensorforce/agents/recorder.py", line 267, in act
num_parallel=num_parallel
File "/home/hinrichs/build/tensorforce/tensorforce/agents/agent.py", line 415, in fn_act
states = self.states_spec.to_tensor(value=states, batched=True, name='Agent.act states')
File "/home/hinrichs/build/tensorforce/tensorforce/core/utils/tensors_spec.py", line 57, in to_tensor
value=value[name], batched=batched, recover_empty=recover_empty
File "/home/hinrichs/build/tensorforce/tensorforce/core/utils/tensors_spec.py", line 57, in to_tensor
value=value[name], batched=batched, recover_empty=recover_empty
File "/home/hinrichs/build/tensorforce/tensorforce/core/utils/tensor_spec.py", line 149, in to_tensor
raise TensorforceError.value(name=name, argument='value', value=value, hint='shape')
tensorforce.exception.TensorforceError: Invalid value for TensorSpec.to_tensor argument value: 0 shape.