9 replies
agent.close() and environment.close() after the agent was trained and initialized as a new agent via Agent.create. Surprisingly the new "untrained" agent behaves like a trained agent. @AlexKuhnle Let me know if you need a sample notebook for demonstration.
1 reply
I created my own environment. I tried already in parallel execution and serial without success.
Do I miss something extra I need to set before it starts writing out the best model?
evaluation=True for runner.run(...). If you have more than one parallel environment, that should hopefully work. Not clear from the docs though.
To put differently: evaltution=True is what you do when you put the algorithm into production?
Thanks already for your knowledge sharing!
- PG: discounted-sum(r_t, r_t+1, ...)
- AC: r_t + disc * Critic(s_t+1) (for horizon=1, otherwise unroll discounted sum more)
- A2C: (r_t + disc * Critic(s_t+1)) - Critic(s_t)
I'm not entirely sure whether this is how AC vs A2C is usually defined.
"tensorforce" agent, and then modify it to make it "on-policy" instead of "off-policy". But that's certainly getting into Tensorforce internals. Otherwise, any reason why not to use PPO?
Hi, everybody! I have a problem with DDPG agent. If I run it with default critic (i. e. without critic) parameters it works OK, but if I try to run it with critic specification (network and optimizer), the error is raised:
...
File "/home/mikhail/RL_10_21/venv/lib/python3.8/site-packages/tensorforce/core/optimizers/tf_optimizer.py", line 173, in step
assert len(gradients) > 0
AssertionError
Does anybody have any ideas?
InvalidArgumentError: Cannot assign a device for operation agent/VerifyFinite/CheckNumerics: Could not satisfy explicit device specification '' because the node {{colocation_node agent/VerifyFinite/CheckNumerics}} was colocated with a group of nodes that required incompatible device '/job:localhost/replica:0/task:0/device:GPU:0'. All available devices [/job:localhost/replica:0/task:0/device:CPU:0, /job:localhost/replica:0/task:0/device:GPU:0].. Full code and error message are here: https://pastebin.com/ktqYYSSv is anyone able to help?
1 reply
Hi all,
I have a C++ code environment that I want to couple with Tensorforce. I want to load my agent model directly within C++. This way, after each action I don't have to leave the C++ code and go to Python and back again. Therefore, I can run a full episode inside the C++ program and record everything because the C++ has access to the model directly.
So far, I have successfully loaded the agent model inside my C++ code and verified its performance. The code runs an entire episode and records all the required data (states, actions, rewards, terminals). Then, I load the data in Python to update the model. My understanding is that I should use an "act-experience-update" interface. However, I do not really have any "act" function as all the actions have already been carried out inside the C++ code. So, calling the experience function using the recorded data as:
agent.experience(states=states, actions=actions, reward=rewards, terminal=terminals)throws:
Exception has occurred: TensorforceError
Invalid value for SINGLETON argument value shape: != (1,).
File "/home/saeed/learning/training.py", line 55, in main
agent.experience(states=states, actions=actions, reward=rewards, terminal=terminals)
File "/home/saeed/learning/training.py", line 89, in <module>
main()The same error is seen if the pretrain function is employed.
Any idea? Has anyone done such coupling between C++ and Tensorforce successfully?
Thanks,
Saeed
1 reply
Hi all,
I am using python 3.8 and trying to build the A2C agent network as below using the AutoNetwork function:
network_spec = dict(type='auto', size=256, depth=3, final_size=256, final_depth=2, rnn=256, inputs_spec=tf.TensorSpec(shape=(276,49), dtype=float))
However, it gives me the error:
TensorforceError: Spec mismatch for argument inputs_spec: TensorSpec(shape=(276, 49), dtype=tf.float32, name=None) != TensorsSpec(SINGLETON=TensorSpec(type=float, shape=(276, 49))).
Does anyone have ideas on how to fix that? Is there anything wrong on my inputs_spec value? many thanks in advance.
Hi all,
I have a dataset of optimal control behavior. I would like to first train the agent on this dataset, and then start interacting with the actual system to continue its learning pocess. I think offline reinforcement learning or imitation learning can do it, but I am not really familiar with any of them. So I would appreciate it if someone can help me with the following questions:
1- Should I use offline RL or imitation learning to pre-train the agent on the dataset?
2- Is it possible to do pre-training on the dataset with tensorforce? any examples?
3- I need to deal with continuous action spaces, so I would like to use SAC agent. There is AC agent in tensorforce, is it the same as SAC?
Thanks
Pretraining is possible, just check the documentation here: https://tensorforce.readthedocs.io/en/latest/agents/agent.html#tensorforce.agents.TensorforceAgent.pretrain
There are also code examples that are quite easy to follow.
actions = dict(type='float', shape=(3,), min_value=-1.0, max_value=1.0)
tensorforce.exception.TensorforceError: Dense input mismatch for argument rank: 2 != 1.? I would highly appreciate any help. I get the error when I try to use a custom network: e.g.: network = [dict(type="dense", size=32, activation="relu"), dict(type="dense", size=32, activation="relu")] for the PPO Agent. Full code: https://github.com/elianderlohr/kniffel/blob/main/src/tensorforce_rl/optuna_rl.py#L609