0 and 1 are both predictions, 2 is actions
So greedy actions I mean
New release for AOgmaNeo/PyAOgmaNeo is now available (1.5.0). It includes a new function setAExplore(index, true/false) as suggested by @llucid-97 , many many run-time checks in the Python bindings, the ability to set IO layers to type "none" again, and a new ImageEncoder that can be optionally hierarchical itself
sorry that it took so long!
Most tests are in Python though. Car_Racing is still C++
Also, the concept has changed a little but since last time: Instead of "goal states", it has been re-conceptualized as a "program state". A "Program" can be a sequence of 1 or more program states. In order to perform goal-state matching as before, I have included a StateAdapter alongside the RLAdapter. These adapters generate program state sequences for the UBL hierarchy
StateAdapter: Go to a goal state
RLAdapter: Perform regular reinforcement learning
RLAdapter: Perform regular reinforcement learning
Clockwork vaes are the closest deep learning based parallel to this, and comparing them on the moving mnist dataset, I can't get Ogma to achieve similar performance
I suspect it's because the deep learning based methods are being given orders of magnitude more compute power, but it got me thinking about scaling Ogma to GPUs. I notice this is something you'd tried before with a previous version. Why did you discontinue that and switch to CPU only?
@llucid-97 we used to have a GPU version, but we stopped using it mostly because experimentation was annoying when using it. I would be interested in revisiting the moving MNIST dataset again (we used it a long time ago). It should work, if you like I can take a look at your code to see what might improve performance!
Ok I thought there would be some hidden caveats to trying to scale things up on GPUs; like maybe since OgmaNeo is not designed to run it batches like deep learning, I thought there'd be some growing pains in trying to get good GPU utilisation of training throughput or something of the sort
I will try the moving MNIST digits with the latest version, I am interested to see how it does nowadays (with 1.8.1). I'll let you know of the result in an hour or so!
will share after a few more tweaks
The image encoding is the main bottleneck I think
@llucid-97 here is a paste of the code: https://pastebin.com/tPL3ZSfs It indeed seems to be mostly the ImageEncoder that is limiting it, as unless a very large (and slow) ImageEncoder is made, the digits do not have their identities preserved, only their motions.
This is with a small image encoder. The image encoder doesn't benefit from being sparse itself, as it is the dense->sparse converter basically. This makes it quite slow at sizes > 16x16x16
Guess we need to work on a GPU version again after all :)
That's Interesting. I appreciate you taking the time looking into this. So if I wanted to test that out before diving into trying a full GPU port of this, I could try setting that up with a larger encoder then just running it for a few days or weeks on CPU only just to see if that works
Looking through the code, I don't understand what the sparsity issue would be in running these on GPUs; I mean the sparsity is in activation space only and you're essentially then treating it as a dense array of integers no?
@llucid-97 the lookup pattern of the weight values is sparse, as they are accessed at out-of-order indices. Hurts cache hits on the CPU, but is still very much worth it there. Now on GPUs I don't know - in the past the GPU versions always felt way slower than they should be. It might be time to try again, though!
I would likely not write it in C++ this time however. Probably going to use Odin and bind OpenCL to it. Python can by bound to that with Cython then.
Odin is a C-replacement language that is very nice to work with. It won't affect the user experience from python anyways, which is how most people use it.