differences:
sizeofnp.random.unifomincludes the1axis- transposing the state should not be necessary (?)
memoryhas to be explicitly set now (new change, since the old default was a bit random)'- environment created via
Environment.create, and max episode length set there (right now necessary, but i'm planning to look into that soon) conv1d_transposedoesn't work properly in the new TensorFlow, some weird gradient exception, so I removed it
I think that's all.
(presumably that's where the transpose came from, but you can just swap it everywhere.)
Right, thanks. Another question. Is there any documentation on how multiple states are fed to the neural net? Assuming I have eg. an image which I want to feed to the convolutional net and at the same time some other features (eg. the size of the image in bytes or the time of the last edit) and I want to include them but not run trough the conv layers, but feed them eg. to the dense layer(s) at the end of the network?
On the one hand, there is the
'auto' network, which provides a "default" of combining multiple inputs. The way generally such networks can be specified is via the Register and Retrieve layers here, and by specifying the network not as a list of layers, but as a list of lists where each list generally starts with a retrieve layer and ends with a register layer, so constitute a sequential "segment" of the overall network.
There used to be examples in the unittests, but good point, they are all gone by now due to the auto network. Would be worth adding this to the documentation, will look into it.
One thing I can't figure out how to do is how to build the model using tensorflow or tensorflow-lite after training and saving it. Could anyone please help me understand whether that's possible?
Hi. Thanks for the feedback. :-) Regarding your question: Saving and loading an agent can be done via
agent.save(...) and Agent.load(...) (see docs). Is that what you're looking for, or what more specifically do you mean by "building the model using tf or tf-lite after training"?
Thanks for the reply. Now I know that tensorforce is built on top of tensorflow, correct? In my case I'd like to deploy / test my model on a low powered IOT device after training. Now I know that tf-lite would perform best in such a case, and that typically, it's possible to convert tensorflow models to tf-lite, so I'm wondering whether the trained model would be tensorflow or tensorflow-lite compatible, whether it'd be possible to translate it to tf-lite. I'm a beginner when it comes to machine learning, so I hope what I'm asking about makes sense.
I haven't worked with tf-lite or similar myself, but it should be possible and I would be interested in working through it. Do you know what format is typically used to load models in tf-lite / how loading in tf-lite generally looks like? Is it SavedModel and specialized loading functionality, or can you just create and load it regularly in Python code? Tensorforce is currently still using the older TensorFlow Saver feature, so it may require upgrading to the new SavedModel stuff (shouldn't be too difficult, can look into that).
I haven't worked with TF-Lite yet as I'm still working on building and refining my model, so I'll hopefully get back to you with any issues that might come along the way so we can hopefully work through them. Reading the tensorflow documentation, it seems like TF-Lite uses a proprietary .tflite format, but there is a tool to convert tensorflow's SavedModel to TF-Lite's .tflite format. I guess I'll have to look into upgrading from the saver feature to SavedModel for it to work. Thank you for your help and for pointing me in the right direction.
Hi there, another question. I finally got tensorflow to get running with GPU support and tensorforce as well. However, I am seeing zero benefit of using a GPU. One episode takes in my case 2 seconds with 66% of the time spent in
observe, no matter if I run on GPU or CPU. I am positive GPU is really run on GPU as can be seen nvidia-smi. GPUs are 2x Tesla K80 on a cloud machine. Any hints what the problem might be?
(Btw, the more complex network setup worked very nicely).
Only one GPU utilized ~20% during running. Can I tune some params to get more utility out of the GPUs?
@nneubauer 66% of the time spent in observe sounds a lot, are you sure? What agent are you using, and could you post the spec? Usually, I would expect the GPU to not be used very much because the interactions are dominated by many batchsize-1 act calls where a GPU is not particularly helpful (different from supervised learning, where this problem doesn't exist). But in your case this sounds different, however, depending on your agent spec, there might be a simple improvement
environment = Environment.create( environment='RLenv1.Env1', max_episode_timesteps=10)
and I still got an error "ModuleNotFoundError: No module named 'RLenv1'". Do you have any idea where the problem might be? Is there some complete example of custom environment file structure that I can just emulate and modify bit by bit?
@nneubauer Thanks for your suggestion --- I'm a complete newbie, how does one create and pass an instance in place of a custom environment file?
and I still got an error "ModuleNotFoundError: No module named 'RLenv1'". Do you have any idea where the problem might be? Is there some complete example of custom environment file structure that I can just emulate and modify bit by bit?
@nneubauer Thanks for your suggestion --- I'm a complete newbie, how does one create and pass an instance in place of a custom environment file?
@AlexKuhnle I tried to figure why model saving depends on the call to tf.compat.v1.disable_eager_execution() during initialisation, but failed. Do you think you might have a chance to take a look at it ? Avoiding this call would be a big help to reintegrate tensorforce in our package. Thanks a lot in advance.
Hello,
i wonder, is this also the right place to ask questions about tf-agents? If not, please ignore the following:
I am trying to run a simple dqn agent over an environment that has a collection of different specs as observation_spec. This means that my QNetwork wants a preprocessing_combiner, so i followed its documentation and gave it tf.keras.layers.Concatenate(). Concatenate does not like to concatenate tuples, so the observation_spec has to be a list. Next, i want to implement a replay_buffer, specifically a TFUniformReplayBuffer. Converting it to a dataset (as done in https://github.com/tensorflow/agents/blob/master/docs/tutorials/1_dqn_tutorial.ipynb), i get an error and the explicit instruction to change my specs from lists to tuples. However, then i run into the problem that Concatenate doesn't like it's input. Am i doing something conceptually wrong here?
^^ sorry, autocorrect, hi Niklas... this channel is for Tensorforce, which is an independent framework and unrelated to tf-agents.
Also @/all , FYI: parallelizing environments can now easily be done via the
remote argument of Environment.create (either based on multiprocessing or socket) and the extended features of Runner / run.py, which was basically merged with the functionality of ParallelRunner (which itself was removed).
@AlexKuhnle Hi, I am reading the source code and got some questions. Does the PPO agent use the entire episode to train or batch_size of timesteps in one or more episodes? and there is a parameter subsampling fraction, the code shows that this will randomly sample a fraction from one batch? Is this because it is a modular design and PPO belongs to the policy gradient method, which use all timesteps in one episode to train? Thanks for answering my questions first
Hi @qZhang88, hope the following explanation clarifies your question: PPO, as many other standard policy gradient algorithms, uses complete rollouts (episodes) for reward estimation. In Tensorforce this means that
batch_size defines the number of episodes (each consisting of many timesteps) per update batch. Moreover, the way the PPO update works according to the paper is that it actually performs multiple updates based on randomly subsampled timestep-minibatches (the entire batch of n episodes is quite big). So the subsampling_fraction specifies what fraction of the full batch is subsampled for each minibatch, and optimization_steps specifies how often these mini-updates should happen.
Currently, the only way to save the agent is to use TensorFlow's saver format, unfortunately. However, it's planned to be updated and extended soon, including support for just extracting weight variables in numpy format or similar -- is this what you're looking for? (The recent JSON-encoding problem was about agent config saving.)
(Re TensorFlow's saver format: I don't know exactly what the format looks like, but it contains weights as well as meta information about the graph itself, so that the entire model can be reconstructed from the information.)
When evaluating, one only calls agent.act with the independent/evaluation flag set. In this case, one is now additionally required to provide an internals argument (and in return gets an additional result, the next internal states). On the first call in an episode, this is supposed to be the result of initial_internals, afterwards just the return of the previous act call. See for instance the evaluation example in the docs under getting started. I will add a bit more documentation as well.
(this can be ignored if no internal states are used)