Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • Jul 29 23:31

    dependabot[bot] on pip

    Bump mistune from 0.8.4 to 2.0.… (compare)

  • May 24 17:30

    dependabot[bot] on pip

    Bump tensorflow from 2.8.0 to 2… (compare)

  • Feb 10 08:43

    dependabot[bot] on pip

    (compare)

  • Feb 10 08:43

    AlexKuhnle on master

    Bump tensorflow from 2.7.0 to 2… Merge pull request #855 from te… (compare)

  • Feb 09 23:35

    dependabot[bot] on pip

    (compare)

  • Feb 09 23:35

    dependabot[bot] on pip

    Bump tensorflow from 2.7.0 to 2… (compare)

  • Feb 09 23:28

    dependabot[bot] on pip

    Bump tensorflow from 2.7.0 to 2… (compare)

  • Jan 08 21:53

    AlexKuhnle on master

    Correct type (compare)

  • Jan 08 21:41

    AlexKuhnle on master

    Add missing box2d dependency (compare)

  • Jan 08 16:56

    AlexKuhnle on master

    Downgrade numpy version for Py3… (compare)

  • Jan 08 16:51

    AlexKuhnle on master

    Update to TF 2.7, update depend… (compare)

  • Jan 03 16:15

    AlexKuhnle on master

    Update setup and travis config (compare)

  • Dec 29 2021 14:54

    AlexKuhnle on master

    make states ArrayDict to pass a… Merge pull request #849 from dx… (compare)

  • Nov 10 2021 20:00

    dependabot[bot] on pip

    (compare)

  • Nov 10 2021 20:00

    AlexKuhnle on master

    Bump tensorflow from 2.6.0 to 2… Merge pull request #840 from te… (compare)

  • Nov 10 2021 19:45

    dependabot[bot] on pip

    Bump tensorflow from 2.6.0 to 2… (compare)

  • Oct 20 2021 20:50

    AlexKuhnle on master

    Update gym version requirement (compare)

  • Oct 20 2021 20:48

    AlexKuhnle on master

    fix ZeroDivisionError in the pr… Merge pull request #836 from hr… (compare)

  • Oct 02 2021 15:21

    AlexKuhnle on master

    Fix environments unittest (compare)

  • Oct 02 2021 13:46

    AlexKuhnle on master

    Update gym dependency and fix e… (compare)

Alexander Kuhnle
@AlexKuhnle
@chinen93, the combination of experience and update is a bit like supervised learning, so train the policy distribution to output the corresponding action per state according to the data. This consideration is more or less ignoring the theory around policy gradient and on-policy etc, just looking at the problem from a supervised angle, and this can work (but can also not lead anywhere). The current Tensorforce interface is not ideal, bit too generic and hence may be used wrongly, but I also don't have too much experience with pretraining, behavioral cloning, etc.
Alexander Kuhnle
@AlexKuhnle
The pretrain function is a bit specific and assumes interaction traces including reward, so basically the recorded data of another agent, as in the pretrain example. Experience and update gives more flexibility, but might not be obvious. What data do you have? Individual data points of "expert" state->action decisions? Demonstration traces of state-action pairs? Or full demo traces including reward? Or potentially even more "random" trajectory data, not "expert"/"demonstration"? Depending on which case, applies, there are different possibilities.
Benno Geißelmann
@GANdalf2357
@chinen93 @AlexKuhnle thanks for your help! this is what I was looking for.
nasrashvilg1
@nasrashvilg1
is there a lot of value in having discretized space for PPO agent or any other agent in general? I have a continues state space with 5 readings - which can range from 0.0 to 50. 0(float numbers) I can also have all the 5 readings discretized - just trying to decide if it's worth the effort?
Alexander Kuhnle
@AlexKuhnle
Hey, I don't think you need to discretize these values -- in fact, I would expect it to work less well (but depends on the problem characteristics of course)
Chris Hinrichs
@chris405_gitlab
Hi all, I'm having a problem with nested tensor specifications. I specified a nested block called "MID_1_bid_0", with a float element "qty", but when I try to run the network I get this error:
ValueError: 'MID_1_bid_0/qty_preprocessing' is not a valid module name. Module names must be valid Python identifiers (e.g. a valid class name).
This is in the state space of the agent.
From looking at the source it appears that the naming convention is indeed to separate parent and child objects with a '/', but then that name containing a '/' gets passed as a module name to tf.Module, which is illegal.
Chris Hinrichs
@chris405_gitlab
The tf error is in
site-packages/tensorflow/python/module/module.py", line 113, in __init__ "identifiers (e.g. a valid class name)." % name)
Alexander Kuhnle
@AlexKuhnle
Hi @chris405_gitlab, that looks indeed like a bug, I will check that.
Alexander Kuhnle
@AlexKuhnle
@chris405_gitlab , can you again check it with the latest Github master version? I think the handling should be improved now.
Chris Hinrichs
@chris405_gitlab
@AlexKuhnle Thanks! I'll take a look.
Chris Hinrichs
@chris405_gitlab
@AlexKuhnle That does appear to have fixed it - I'm continuing debugging where I left off, so I'm not running end-to-end yet, but it does look like the next error is mine. Thanks again.
Chris Hinrichs
@chris405_gitlab
I'm getting this warning when I start a runner, and it's taking 2-3 minutes for the preprocessing to complete before running. Is that normal, and is there a known misuse that would cause this?
/site-packages/tensorflow/python/framework/indexed_slices.py:433: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
  "Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
Chris Hinrichs
@chris405_gitlab
@AlexKuhnle I think I have another bug for you. In Agent.py, the first thing fn_act() does is to separate any variable name + '_mask' from the state spec where name is any name found in the action_spec. It does this recursively using fmap(), calling this function:
        # Separate auxiliaries
        def function(name, spec):
            auxiliary = ArrayDict()
            if self.config.enable_int_action_masking and spec.type == 'int' and \
                    spec.num_values is not None:
                if name is None:
                    name = 'action'
                # Mask, either part of states or default all true
                auxiliary['mask'] = states.pop(name + '_mask', np.ones(
                    shape=(num_parallel,) + spec.shape + (spec.num_values,), dtype=spec.np_type()
                ))
            return auxiliary

However, when I run it I get a KeyError exception, where the key is a root name from action space. I instrumented the line from nested_dict.pop() that threw the error, like so:

  3         elif '/' in key:
  2             key, subkey = key.split('/', 1)
  1             if not key in self:
371                 print(f"pop {key} {subkey}")
  1                 import pprint
  2                 pprint.pprint(self)
  3             value = super().__getitem__(key)
  4             assert isinstance(value, self.__class__)
  5             return value.pop(subkey, default)

This is what it printed:
pop MID_1_counter_0 promise_date_mask {'MID_1_bid_0/price': array([1.15]), 'MID_1_bid_0/promise_date': array([4]), 'MID_1_bid_0/qty': array([44135.2]), 'MID_1_bid_0/supplier_tier': array([0]), ...

Chris Hinrichs
@chris405_gitlab
Note that in my scenario, names with _bid_ in them are state variables, and names with _counter_ in them are action variables.
Now, in the top level to pop() there is a default value, however, when a nested variable is encountered, the handling for that case doesn't consult the default value, it just says value = super().__getitem__(key)
Alexander Kuhnle
@AlexKuhnle
Hey, first, the warning Converting sparse IndexedSlices to a dense Tensor of unknown shape. comes up if you use embeddings (used by "auto" network if state is int), and maybe in other situations as well. I've read a bit about it a while ago, and it doesn't seem to be critical, if e.g. the number of embeddings (num_values of int state) is reasonable. Model initialization may take a while if the network is bigger -- is this the case for you?
And regarding the second issue: I will look into it soon.
Chris Hinrichs
@chris405_gitlab
@AlexKuhnle Thanks for the tip. Meanwhile, I modified the pop() code in nested_dict to return the default if the super key is not found, (instead of printing debug info), but it led to an invalid-shape error. I think the problem is that if the action is nested then there won't be a shape argument for parent nodes (only leaf nodes have a shape). Given that, what I would like to do is to set self.config.enable_int_action_masking to False, but I don't see a way to do that... The config object explicitly overrides __set_attr__ and I wasn't able to pass it as a constructor arg to the agent. So, what's the right way to do that?
Alexander Kuhnle
@AlexKuhnle
Setting enable_int_action_masking can be done via the config argument of any agent (docs here). That should hopefully work.
I've also made the change you suggested to NestedDict -- would you mind posting the shape exception, since I don't know why that would come up?
Chris Hinrichs
@chris405_gitlab
Here is the stack trace:
  File "train_rl_agent.py", line 77, in run_agent
    runner.run(num_episodes=sim_config["train_episodes"])
  File "/home/hinrichs/build/tensorforce/tensorforce/execution/runner.py", line 545, in run
    self.handle_act(parallel=n)
  File "/home/hinrichs/build/tensorforce/tensorforce/execution/runner.py", line 579, in handle_act
    actions = self.agent.act(states=self.states[parallel], parallel=parallel)
  File "/home/hinrichs/build/tensorforce/tensorforce/agents/agent.py", line 388, in act
    deterministic=deterministic
  File "/home/hinrichs/build/tensorforce/tensorforce/agents/recorder.py", line 267, in act
    num_parallel=num_parallel
  File "/home/hinrichs/build/tensorforce/tensorforce/agents/agent.py", line 415, in fn_act
    states = self.states_spec.to_tensor(value=states, batched=True, name='Agent.act states')
  File "/home/hinrichs/build/tensorforce/tensorforce/core/utils/tensors_spec.py", line 57, in to_tensor
    value=value[name], batched=batched, recover_empty=recover_empty
  File "/home/hinrichs/build/tensorforce/tensorforce/core/utils/tensors_spec.py", line 57, in to_tensor
    value=value[name], batched=batched, recover_empty=recover_empty
  File "/home/hinrichs/build/tensorforce/tensorforce/core/utils/tensor_spec.py", line 149, in to_tensor
    raise TensorforceError.value(name=name, argument='value', value=value, hint='shape')
tensorforce.exception.TensorforceError: Invalid value for TensorSpec.to_tensor argument value: 0 shape.
Chris Hinrichs
@chris405_gitlab
This message was deleted
Chris Hinrichs
@chris405_gitlab
I disabled enable_int_action_masking and I'm still getting that error, so it's not related to the issue with pop() not defaulting. Thanks for the link showing how to do that.
Chris Hinrichs
@chris405_gitlab

@AlexKuhnle I've figured out what's happening, but I don't fully understand the cause. I instrumented the code where the exception is raised like so:

  3         # Check whether shape matches
  2         if value.shape[int(batched):] != self.shape:
  1             print(f"\nvalue {value} type {type(value)} batched {batched}")
150             print(f"value shape {value.shape[int(batched):]} self shape {self.shape}")
  1             import pprint
  2             pprint.pprint(self)
  3             pprint.pprint(value)
  4             raise TensorforceError.value(name=name, argument='value', value=value, hint='shape')

and this is what it prints:

value [0] type <class 'numpy.ndarray'>
value shape () self shape (1,)
TensorSpec(type=int, shape=(1,), num_values=4)
array([0])

The reason is that batched is True, but the value shape doesn't have a batch dimension.

Alexander Kuhnle
@AlexKuhnle
I realise there is something missing which makes the exception message less useful/specific -- will fix that. But it looks like a subtle shape problem of some inputs, as if the value returned by the environment is not perfectly matching the shape of the states specification.
Ah, was just writing :-)
Chris Hinrichs
@chris405_gitlab
wow
I was just writing too - I tried removing batch_size from the agent params, but it tells me that one is required.
Alexander Kuhnle
@AlexKuhnle
That's what I thought: it seems your environment specifies the shape as (1,), whereas what it actually returns is of shape (). That could be the case, for instance, if the state value is returning a primitive Python type (which are of shape ()).
Tensorforce is very strict about these shapes, since TensorFlow and the computation graph are, too (but unlike e.g. NumPy, which is often very forgiving).
Chris Hinrichs
@chris405_gitlab
Ah, ok so leave scalar types as ()
I put in a shape of (1,) for all scalars earlier to remove sources of variation while debugging another problem. I can try it now without that.
Alexander Kuhnle
@AlexKuhnle
Yes, that would be the fix.
Chris Hinrichs
@chris405_gitlab
Yes, it does seem to have fixed it. Thanks again.
Chris Hinrichs
@chris405_gitlab
I'm trying to run several environments in parallel, and as soon as I go from one agent to two, I get OSError: [Errno 12] Cannot allocate memory. I also note that when I first instantiate the agent, it takes about 3 minutes to complete the Agent.create() process. So far I've been using network='auto'. I printed out the state and action spec, and my state-space has a total of 59 scalar components, and the action space has a total of 80.
As discussed before, there was a potential that the integer space embeddings could be a cause. In the state space, there are 10 variables with 4 values, and another 10 with 10 values. In the action space, there are 10 with 4, 10 with 10, and 10 with 20. I tried setting the 20-valued variable to use only 10 in the action space with the same result.
Chris Hinrichs
@chris405_gitlab
Does anyone have any experience with a model blowing up unexpectedly? Is 10 an unreasonable number of values for an int to have?
Is there a simple way to print out the auto-generated network specification?
Chris Hinrichs
@chris405_gitlab
I've created a pastebin of my state and action space in case that helps
https://pastebin.com/V4xTdYWv
Chris Hinrichs
@chris405_gitlab
Update:
I refactored the code to use a vector of N variables instead of having N nested sub-variables, and that is a LOT faster.
https://pastebin.com/3pwfjS6Y
Initializing the agent is about 10x faster, and training is also about 10x faster (running in 2 threads vs. 1 before).
Alexander Kuhnle
@AlexKuhnle
It's a good idea to provide some information about the network specification.
But generally each state gets its own "input head", which means that if you have 10 values as a single vector, there is only one head, if you split it up to 10 scalars, there will be 10 heads. While these could obvs combined internally, it's unclear when one should do this and when one shouldn't, so it's left to the user to design the space.
Another helpful addition would probably be a simple warning.
Chris Hinrichs
@chris405_gitlab

In execution.Runner, the constructor takes an argument evaluation, and the comment says that if it there are multiple environments it will run the last one only, but in Runner.run() it raises an exception like so -

  1         if evaluation and (self.evaluation or len(self.environments) > 1):
419             raise TensorforceError.unexpected()

.. and the comment says that it is an error to pass evaluation = True with multiple environments.
Which behavior is intended to be the standard? As it is, run() controls because you have to call it. Also, why would it throw an error if evaluation and self.evaluation are both True?

Alexander Kuhnle
@AlexKuhnle
The documentation should probably be improved, but if I remember correctly, the reasoning is as follows: on the one hand, Runner.run() can be used multiple times in the "standard" use case of a single environment, in particular trainingrunner.run(num_episodes=???)and subsequent evaluationrunner.run(num_episodes=???, evaluation=True)(that's therun()`evaluation argument); on the other hand, it provides an interface to parallel execution, but in that case you can't just switch from training run() to evaluation run() -- however, you can specify that one of the parallel environments is used for evaluation throughout (that's the constructor evaluation argument).
However, all this constructor vs run() arguments have not been separated in a principled way for a long time, and really a Runner should probably be a one-off specification of a run which cannot be re-used. Maybe make Runner.run() a static method and put all arguments there.