Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • 12:35
    AlexDBlack commented #8298
  • 11:56
    AlexDBlack labeled #8298
  • 11:56
    AlexDBlack labeled #8298
  • 11:56
    AlexDBlack commented #8298
  • 11:00
    AlexDBlack opened #8308
  • 10:19
    AlexDBlack closed #8275
  • 10:19
    AlexDBlack closed #8237
  • 10:19
    AlexDBlack closed #8286
  • 09:25
    AlexDBlack closed #8066
  • 09:25
    AlexDBlack commented #8066
  • 09:16
    AlexDBlack closed #8294
  • 09:16
    AlexDBlack synchronize #8294
  • 08:25
    guigautier commented #8298
  • 08:12
    cqiaoYc commented #8253
  • 03:26
    AlexDBlack synchronize #8294
  • 00:36
    saudet assigned #8307
  • Oct 22 23:28
    willishf commented #8307
  • Oct 22 23:25
    rafmontano commented #8217
  • Oct 22 22:29
    LyesSp commented #8295
  • Oct 22 21:46
    eraly commented #8298
Stefano Maestri
@maeste
it works fine in java mode, but compiling it to native I'm stuck with errors like this one:
Caused by: java.lang.IllegalArgumentException: Invalid Structure field in class com.sun.jna.platform.win32.Variant$VARIANT$_VARIANT, field name 'variant' (class com.sun.jna.platform.win32.Variant$VARIANT$_VARIANT$VARIANT): Exception thrown while instantiating an instance of class com.sun.jna.platform.win32.Variant$VARIANT$_VARIANT$__VARIANT
But I'm puzzled from the error, because I'm running on linx-x86_64 (fedora)
and other similar errors for solaris, android and so on...
Stefano Maestri
@maeste
Any hint would be more than welcome
jmmp99
@jmmp99
Hello, where can I find an example for stocks, preferably using RNN, LSTM, GRU, thanks!
Robert Altena
@RobAltena
@jmmp99 We do not have a stock price prediction example in the dl4j-examples.
nimishjain
@nimishjain
There is no example to implement custom MDP in rl4j
gitterBot
@raver120
@nimishjain Welcome! Here's a link to Deeplearning4j's Gitter Guidelines, our documentation and other DeepLearning resources online. Please explore these and enjoy! https://github.com/deeplearning4j/deeplearning4j/blob/master/deeplearning4j/GITTER_GUIDELINES.md
nimishjain
@nimishjain
Can someone help me with an example implementation of MDP ? I don't want to use gym
f
@SidneyLann
It seems gym had not update for 3 years
nimishjain
@nimishjain
I don't want to use gym but a simple example to use MDP in RL4J.
DataHorizon
@DataHorizon2_twitter
Are there DL4j resources for loading time-series data stored in HDFS and constructing an iterator out of it to fit a model?
Alex Black
@AlexDBlack

@DataHorizon2_twitter

Are there DL4j resources for loading time-series data stored in HDFS and constructing an iterator out of it to fit a model?

As in, locally fitting from remotely stored data?
Yes, it's similar to a local data pipeline (for like CSV or whatever) but you should use the following InputSplit instead of say FileInputSplit:
https://github.com/eclipse/deeplearning4j/blob/master/datavec/datavec-api/src/main/java/org/datavec/api/split/StreamInputSplit.java

Yann-Gaël Guéhéneuc
@yann-gael_gitlab

I must not look at the right place: could someone explain me/point to me how to stop the thread(s) performing the training. I'm running this code in a SwingWorker:

final QLearningDiscreteDense<State> qldd = new QLearningDiscreteDense<State>(mdp, Configuration.NETWORK, Configuration.Q_LEARNING, Configuration.DATA_MANAGER);
mdp.setFetchable(qldd);
qldd.train();
mdp.close();

and would like to stop the training when cancelling the worker (using "this.worker.cancel(true)" or some other means).

Yann-Gaël Guéhéneuc
@yann-gael_gitlab

@nimishjain I'm building such a simple example using Pavlov's dog as "problem", you can find it here: https://gitlab.com/wimp.today/Code/tree/master/RL4J%20Tests%20%28From%20Maven%29

It has a CLI version and a UI to visually display the training (and soon the using) of the model. It doesn't require anything special (no GYM or Web server...) only RL4J (through Maven) and Java.

Yann-Gaël Guéhéneuc
@yann-gael_gitlab
Stefano Maestri
@maeste
newbie question: I've imported a tensorflow pb and trying to execute it with data read by NativeImageLoader but I'm getting this exception: java.lang.IllegalStateException: Invalid array shape: cannot associate an array with shape [1, 3, 128, 128] with a placeholder of shape [-1, 128, 128, 3]:shape is wrong rank or does not match on one or more dimensions
it seems dimensions are in the wrong order...can you point me to an example on how to sort dimension in an INDArray?
(BTW -1 means any dimension is valid, right?)
Hem Chand
@versatile-hem

Hi Guys,

could you please help for selecting ?

one of the below:

ActiveMQ
vs
Kafka
vs
RabbitMQ

gitterBot
@raver120
@versatile-hem Welcome! Here's a link to Deeplearning4j's Gitter Guidelines, our documentation and other DeepLearning resources online. Please explore these and enjoy! https://github.com/deeplearning4j/deeplearning4j/blob/master/deeplearning4j/GITTER_GUIDELINES.md
raguenets
@montardon
Hi, I'm having trouble with a .h5 model I imported with KerasModelImport.importKerasModelAndWeights. I was expecting to get the same results with the same inputs, compared to Python version. I double-checked image normalization, I even took fully black ones as input. My previous models were giving the exact same results so I guess there is something different in this one. I'm using latest version beta-5. I'm opening a ticket to provide model with test code.
Samuel Audet
@saudet
@yann-gael_gitlab there are some changes for that in this PR: eclipse/deeplearning4j#8072
Polly13
@Polly1358941334_twitter

Hello. I m importing keras functional model and as far as my model has 2 Lambda layers I registered them by «KerasLayer.registerLambdaLayer». lambda_1 layer is a simple function that squares values. The layer is located between convolutional and dense layers. To make the model work, I have to set in «InputType getOutputType» the value InputType.recurrent(16) only, and nothing else works. (for details see code and model in link below).

1) Why recurrent? It's not logical. The previous layer to lambda is convolutional, the next one is dense. The model doesn't contain recurrent layers at all.
2) For what «InputType getOutputType» is responsible for? For input or for output?
3) As for 2-nd lambda layer: InputType in lambda_2 layer is set to InputType.feedforward(2). The argument can be set equal to any integer and final result doesn't change. Why there is no change? The parameter has no meaning?

I attached gist with:
1) code;
2) model (h5);
3) summary of origin keras model (png).
You can run code with model and check. Set your path when importing model. Model also shows summary and allows you to put values (in range 0-39) to check how result changed.

Code+Model+Summary

raguenets
@montardon
Alex Black
@AlexDBlack
@maeste that's the difference between "channels first" (NCHW) and "channels last (NHWC) data
after loading your image, do image.permute(0,2,3,1) to convert from NCHW to NHWC format
f
@SidneyLann
TF said java can't be used for Graph Program Extraction because java can't do static analysis, DL4J has this problem? https://github.com/tensorflow/swift/blob/master/docs/WhySwiftForTensorFlow.md
Emmanuel Keskes
@mankeskes
@AlexDBlack I am trying to upgrade to beta5 - but I am getting the following maven error: Failure to find org.nd4j:nd4j-cuda-10.1:jar:macosx-x86_64:1.0.0-beta5 - Any idea?
Yann-Gaël Guéhéneuc
@yann-gael_gitlab
@saudet Thanks: should I use an AsyncLearning rather than a SyncLearning subclass then?
Alex Black
@AlexDBlack
@mankeskes unfortunately we dropped CUDA support on Mac in 1.0.0-beta4
https://deeplearning4j.org/release-notes.html
given Apple hasn't supported NVIDIA cards in their machines for many years, we decided to drop support to minimize development and release overheads
we're not the only library to do this, tensorflow doesn't release an OSX CUDA version either, for example
Oliver Rausch
@orausch
Is there any way to manually deallocate INDArrays? In my training loop, I have a producer-consumer architecture , where multiple threads place INDArrays into a queue, and a single thread that propagates these through samediff. Or is there maybe some way to use Workspaces for this?
Alex Black
@AlexDBlack
@SidneyLann graph extraction is only relevant for "define by run" (i.e., tensorflow eager mode)
DL4J, SameDiff and non-eager TF are all considered "define then run", which doesn't have that issue, as you create your graph then execute it

@orausch yes, you have two options
1) use workspaces - best performance (for cyclical workloads), but a little more complex
https://deeplearning4j.org/docs/latest/deeplearning4j-config-workspaces
https://github.com/eclipse/deeplearning4j-examples/blob/master/nd4j-examples/src/main/java/org/nd4j/examples/Nd4jEx15_Workspaces.java

2) Deallocate manually using INDArray.close() (which is fine, but has some performance overhead associated with it)

DataHorizon
@DataHorizon2_twitter
@AlexDBlack Thanks for the reply. But I was referring to data in HDFS and using within the cluster on something like Zeppelin. Would the information you provided still be the same?
Alex Black
@AlexDBlack
@DataHorizon2_twitter right, it should be the same in that case
there's basically 2 cases here
(a) cluster execution - apache spark etc
(b) single node execution - accessing remote data
I described how to do (b)
Samuel Audet
@saudet
@yann-gael_gitlab it depends on the algorithm you're looking to use/implement
Emmanuel Keskes
@mankeskes
@AlexDBlack Thanks vm
agatheLB
@AgatheLB
Hello, new to DL4j, I'm trying to normalize text and for that implementing feature hashing, in order to create a model afterwards. I've been looking for doc/examples/tutorials, but didn't find anything on the feature hashing (unless I'm mistaken).
Explanation: applying the feature hashing principle, I have a Hashmap<Integer, Integer> containing in index the hash%N and in value the number of occurrences. Then, I'd like to know how to feed a model with these type of data, do I need to go through a TransformProcess?
gitterBot
@raver120
@AgatheLB Welcome! Here's a link to Deeplearning4j's Gitter Guidelines, our documentation and other DeepLearning resources online. Please explore these and enjoy! https://github.com/deeplearning4j/deeplearning4j/blob/master/deeplearning4j/GITTER_GUIDELINES.md
Stefano Maestri
@maeste
another newbie question: what does it mean getting this exception importing a .pb: "ND4JIllegalStateException: No tensorflow op found for Substr possibly missing operation class?"
thanks in advance
Megil Gallant
@MegilGallant_twitter
Hi all, what kind of dataset iterator do I use when my output is continuous. All the examples on dl4j seem to focus on classification problems (where I'm trying to predict a value, not classify something)
gitterBot
@raver120
@MegilGallant_twitter Welcome! Here's a link to Deeplearning4j's Gitter Guidelines, our documentation and other DeepLearning resources online. Please explore these and enjoy! https://github.com/deeplearning4j/deeplearning4j/blob/master/deeplearning4j/GITTER_GUIDELINES.md
ChrisYohann
@ChrisYohann

Hi everyone, I'm currently facing an issue using Spark and DL4J.
I'm trying to fit a neural network for timeseries forecasting. I have done all the data preparation steps relative to Spark : Load my data as a RDD of some objects to get at the end an RDD<DataSet> .
Then I create my SparkDl4jMultiLayer to fit my network. I use Gradient Sharing Implementation.
The problem is I encounter a NullPointerException coming from SharedTrainingWrapper.java, Here is a more detailed stacktrace :

19/10/02 22:18:13 WARN SharedTrainingWrapper: Exception encountered during fit operation
java.lang.NullPointerException
    at org.deeplearning4j.spark.parameterserver.pw.SharedTrainingWrapper.run(SharedTrainingWrapper.java:475)
    at org.deeplearning4j.spark.parameterserver.functions.SharedFlatMapPathsAdapter.call(SharedFlatMapPaths.java:94)
    at org.deeplearning4j.spark.parameterserver.functions.SharedFlatMapPathsAdapter.call(SharedFlatMapPaths.java:62)
    at org.datavec.spark.transform.BaseFlatMapFunctionAdaptee.call(BaseFlatMapFunctionAdaptee.java:40)
    at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$4$1.apply(JavaRDDLike.scala:153)
    at org.apache.spark.api.java.JavaRDDLike$$anonfun$fn$4$1.apply(JavaRDDLike.scala:153)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:800)
    at org.apache.spark.rdd.RDD$$anonfun$mapPartitions$1$$anonfun$apply$23.apply(RDD.scala:800)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
    at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
    at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
    at org.apache.spark.scheduler.Task.run(Task.scala:109)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

After running the debugger, it looks like that the SharedTrainingWrapper instance has an attribute consumer that is initialized to null and gets never updated. The only place where it seems to be updated is at line 310 :

                    // if we're running in spark localhost mode - we don't want double initialization
                    if (!ModelParameterServer.getInstance().isInitialized())

But since i'm running Spark in localhost mode I don't pass the if statement.

My project is in scala, built with sbt. I'm using spark 2.3.1 and deeplearning4j 1.0.0_beta4.

Could someone help me with this ?

jmmp99
@jmmp99
@RobAltena Thanks ! Any thoughts or links to start out?
Robert Altena
@RobAltena
@jmmp99 We are leaving the realm of dl4j (where this channel should be limited to.) But have a look at this: https://twitter.com/fchollet/status/1177633367472259072
Alex Black
@AlexDBlack
@maeste that means we don't yet support that operation for import
our ops coverage is pretty good, but there's a few we don't get have
open an !issue and we'll take a look