Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • 11:47
    AlexDBlack edited #8755
  • 11:29

    AlexDBlack on master

    Fix limits on flaky test to avo… tf.keras model import (#258) *… Backpropagation implementation … and 5 more (compare)

  • 11:29
    AlexDBlack closed #8814
  • 11:28
    AlexDBlack milestoned #8813
  • 11:28
    AlexDBlack labeled #8813
  • 11:25
    AlexDBlack milestoned #7319
  • 11:21
    AlexDBlack opened #8814
  • 11:11
    AlexDBlack assigned #8739
  • 09:10
    saudet commented #3022
  • 07:52
    liweigu opened #8813
  • Mar 27 12:33
    treo labeled #7319
  • Mar 27 12:33
    treo commented #7319
  • Mar 27 08:15
    AlexDBlack edited #8812
  • Mar 27 08:14
    AlexDBlack labeled #8812
  • Mar 27 08:14
    AlexDBlack labeled #8812
  • Mar 27 08:14
    AlexDBlack labeled #8812
  • Mar 27 08:14
    AlexDBlack opened #8812
  • Mar 26 19:42
    linuxciscoarnaud opened #8811
  • Mar 26 18:26
    sshepel commented #8750
  • Mar 26 13:33
    AlexDBlack closed #8765
acon10
@acon10
Can you help me to fix it?
Sascha4j
@Sascha4j
Hello everyone, i try to get my application running with cuda 10.2 , i tried beta6 and snaphot with the cuda redist dependency . gpu usage is very low. 0%-4%. here is the output when start learning [main] INFO org.nd4j.linalg.factory.Nd4jBackend - Loaded [JCublasBackend] backend
[main] INFO org.nd4j.nativeblas.NativeOpsHolder - Number of threads used for linear algebra: 32
[main] INFO org.nd4j.nativeblas.Nd4jBlas - Number of threads used for OpenMP BLAS: 0
[main] INFO org.nd4j.linalg.api.ops.executioner.DefaultOpExecutioner - Backend used: [CUDA]; OS: [Windows 10]
[main] INFO org.nd4j.linalg.api.ops.executioner.DefaultOpExecutioner - Cores: [4]; Memory: [14,2GB];
[main] INFO org.nd4j.linalg.api.ops.executioner.DefaultOpExecutioner - Blas vendor: [CUBLAS]
[main] INFO org.nd4j.linalg.jcublas.JCublasBackend - ND4J CUDA build version: 10.2.89
[main] INFO org.nd4j.linalg.jcublas.JCublasBackend - CUDA device 0: [GeForce 840M]; cc: [5.0]; Total memory: [2147483648]
[main] INFO org.deeplearning4j.models.sequencevectors.SequenceVectors - Starting vocabulary building...

and stacktrace 2020-02-28 16:35:54
Full thread dump OpenJDK 64-Bit Server VM (25.242-b08 mixed mode):

"VectorCalculationsThread 0" #21 prio=5 os_prio=0 tid=0x00000253b5a3c000 nid=0x1a38 runnable [0x000000c0accfe000]
java.lang.Thread.State: RUNNABLE
at org.nd4j.nativeblas.Nd4jCuda.execCustomOp2(Native Method)
at org.nd4j.linalg.jcublas.ops.executioner.CudaExecutioner.exec(CudaExecutioner.java:2498)
at org.nd4j.linalg.jcublas.ops.executioner.CudaExecutioner.exec(CudaExecutioner.java:2306)
at org.deeplearning4j.models.embeddings.learning.impl.elements.SkipGram.iterateSample(SkipGram.java:534)
at org.deeplearning4j.models.sequencevectors.SequenceVectors$VectorCalculationsThread.run(SequenceVectors.java:1317)

Locked ownable synchronizers:

- None

"AsyncSequencer thread" #20 daemon prio=5 os_prio=0 tid=0x00000253b5a3b000 nid=0x1848 runnable [0x000000c0acbff000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:338)
at org.deeplearning4j.util.ThreadUtils.uncheckedSleep(ThreadUtils.java:26)
at org.deeplearning4j.models.sequencevectors.SequenceVectors$AsyncSequencer.run(SequenceVectors.java:1170)

Locked ownable synchronizers:

- None

"DeallocatorServiceThread_1" #19 daemon prio=5 os_prio=0 tid=0x00000253b5a33800 nid=0x1b50 in Object.wait() [0x000000c0acaff000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)

- locked <0x00000006b7f205a0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
at org.nd4j.linalg.memory.deallocation.DeallocatorService$DeallocatorServiceThread.run(DeallocatorService.java:123)

Locked ownable synchronizers:

- None

"DeallocatorServiceThread_0" #18 daemon prio=5 os_prio=0 tid=0x00000253b5a32000 nid=0x1504 in Object.wait() [0x000000c0ac9ff000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)

- locked <0x00000006b7f20758> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
at org.nd4j.linalg.memory.deallocation.DeallocatorService$DeallocatorServiceThread.run(DeallocatorService.java:123)

Locked ownable synchronizers:

- None

"Threadly clock updater" #13 daemon prio=5 os_prio=0 tid=0x0000025338537000 nid=0x2534 in Object.wait() [0x000000c0ac7fe000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at org.threadly.util.Clock$ClockUpdater.run(Clock.java:250)

- locked <0x00000003c0bfe020> (a java.lang.Object)
at java.lang.Thread.run(Thread.java:748)

Locked ownable synchronizers:

- None

"JavaCPP Deallocator" #11 daemon prio=10 os_prio=2 tid=0x000002537c5f7000 nid=0x2498 in Object.wait() [0x000000c0ac5fe000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)

- locked <0x00000003c0098080> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
at org.bytedeco.javacpp.Pointer$DeallocatorThread.run(Pointer.java:375)

Locked ownable synchronizers:

- None

"Service Thread" #10 daemon prio=9 os_prio=0 tid=0x000002537bc8a800 nid=0x9bc runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:

- None

"C1 CompilerThread2" #9 daemon prio=9 os_prio=2 tid=0x000002537bc83000 nid=0x638 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:

- None

"C2 CompilerThread1" #8 daemon prio=9 os_prio=2 tid=0x000002537bc13800 nid=0x17e8 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

Locked ownable synchronizers:

- None

"C2 CompilerThread0" #7 daemon prio=9 os_prio=2 tid=0x000002537bc13000 nid=0x5bc waiting on condition

onil-innarin
@onil-innarin
Hello, does anyone happen to know a way to get an INDArray of one of the images from an ImageRecordReader?
AgathiyaTam
@AgathiyaTam
Hi,
I am getting same result whatever feed any images to Tflite interpreter in Android Studio(Java)
Please let me know if anyone have an idea about that
ethiel
@ethiel
Hi, people.
ethiel
@ethiel
Hi, people. I'm trying to load the google vectors model into DL4j, to use it later for sentiment analysis. I'm able to do it but with a big cost in RAM memory. I read about workspaces and how the memory is hold off-heap, so I tried to load it using loadStaticModel inside a workspace,just to see if my RAM consumption was lower. However, I'm not able to do it if I set my available amount of RAM to a number less of 3.4 GB (the size of Google vectors model). So, does it mean the workspaces are only for certain operations?. It's that a bug? . the code is quite simple: final WorkspaceConfiguration mmap = WorkspaceConfiguration.builder()
.initialSize(9500000000L)
.policyLocation(LocationPolicy.MMAP)
.policyAllocation(AllocationPolicy.STRICT)
.policyReset(ResetPolicy.ENDOFBUFFER_REACHED)
.tempFilePath("/Users/Downloads/temporalFile.temp")
.build();
final File wordVectorsFile = new File("/sentiment/GoogleNews-vectors-negative3002.bin");
try (final MemoryWorkspace ws = Nd4j.getWorkspaceManager().getAndActivateWorkspace(mmap, "M2")) {
final WordVectors wordVectors = WordVectorSerializer.loadStaticModel(wordVectorsFile);
final ComputationGraph computationGraph =
}
@AlexDBlack, thanks.
Karan Shah
@kshah1
How do I use dl4j within my own projects?
onil-innarin
@onil-innarin
Is there a way to manually download the dl4j jars?
jverheul
@jverheul
Customer of mine is manually processing word documents with configuration data. Those are legal contracts between an insurer and doctors (PCP primary care provider) where the PCP agrees to deliver certain type of care starting certain date. Those documents must be converted into a structural definition in our claims processing system. The contract may state that the PCP will deliver the services for Commercial PPO, but not for Medicare PPO for example. So basically the problem is to map a document to a set of tables L1, L2 .. LN. What is the best technique to use for this problem. Which example of DL4J is the best starting point?
Complicating factor: a label L may actually be record: not only the code is relevant, but perhaps also a start date and perhaps some amounts.
urmay shah
@urmay
@treo :point_up: February 13, 2018 6:00 PM
I have tried to load pretrained glove model .
WordVectors wordVectors = WordVectorSerializer.loadTxtVectors(new File("D:\Data\glove.6B.50d.txt"));
ParagraphVectors vectors = new ParagraphVectors.Builder()
.useExistingWordVectors(wordVectors)
.iterate(iterator)
.resetModel(false)
.build();
But it is giving me : Model being passed as existing has no syn1/syn1Neg available
clownkalai
@clownkalai
How to get wife router access
kingpaky
@kingpaky
hi sorry something could drive me to have support for my java application please.
RobertJSharp
@RobertJSharp
Has anyone had issues where the Datavec LocalTransformExecutor does not release it's memory? I'm not sure if I'm missing something, but a simple test is if you loop a transform, the memory just goes up and up. I've tried putting in a GC but it does nothing, and it's not heap memory, it's non-heap so I think is Ndl4j.
sengler97
@sengler97
I do have the following problem with the UI component. Im following this guide (https://deeplearning4j.org/docs/latest/deeplearning4j-nn-visualization) to use it, but there seem to be problems with some javascript libraries.
Results look like the following: https://imgur.com/a/kRjX32j.
When opening the website we get following errors in Java. "WARN 2020-03-09 09:54:46.935 [ctor.default-dispatcher-7][ ] - Could not find requested UI asset: webjars/bootstrap/2.3.2/css/bootstrap.min.css (Assets.java:57)
java.io.FileNotFoundException: META-INF/resources/webjars/bootstrap/2.3.2/css/bootstrap.min.css cannot be opened because it does not exist"
Same error for jquery.
Im starting dl4j out of Eclipse using Gradle as built tool and the latest version. Im not sure what to do, found no info on the web.
Brandmetric SA
@carlosorrego
Hi. I ma playing with the neuralstyletransfer example. I manage to make it run. But i still do not undertand de image size thing. Why 244x244 size? has to be square? Can i use hd images? bigger size? thanks in advance
MVreijling
@MVreijling
Could it be that the Anomaly Detection example using the basic autoencoder on mnist digits isn't really working properly.
When I train this configuration on 100k examples of a single digit and test on a mixed set of the original digit and another (untrained) digit it keeps giving me (also) examples of the untrained digits as “best” (low reconstruction error).
This is not what I hoped for since I am looking for a way to sort a set of mixed images into one type vs anything else.
Is this expected behaviour or should I be looking for an implementation error?
siddadel
@siddadel
G
Hi All, I wonder if anyone else in this room is facing problems working with ZooModel. I am trying to work with VGG16 and the weights do not download because blob.deeplearning4j.org is offline. Any help or reference will be appreciated. @agibsonccc
siddadel
@siddadel
@AlexDBlack - Hi - I was wondering if you could share some information on blog.deeplearning4j.org please. I have been trying to initialize VGG16 but it fails. The link with the zip fails. If I can get an alternate location -- even if it is behind a paywall -- I would really appreciate this. Additionally, is there any code reference on how to use init without the pretrained weights?
siddadel
@siddadel
@beoy - Sorry to put you on the spot on the forum, but was your zoomodel able to download "vgg16_dl4j_inference.zip". Do you know any location where it has been re-hosted or if you have a copy, could you share this with the community (assuming it is under commons/gpl copyright)
Roland Gude
@rjtg
Hi, can anybody point me to examples of how to use the various AttentionLayers in DL4J?
ashingel
@ashingel
Good evening, is there any example how to port BERT model for sequence tagging (for NER task) to deeplearning4j? Most of the examples which I have found just about or around MNIST or similar datasets. Thanks in advance.
val conf: MultiLayerConfiguration = new NeuralNetConfiguration.Builder()
    .seed(rngSeed) //include a random seed for reproducibility
    .optimizationAlgo(OptimizationAlgorithm.STOCHASTIC_GRADIENT_DESCENT) // use stochastic gradient descent as an optimization algorithm
    .iterations(1) // in nearly all cases should be 1
iterations seems to be on 0.9.1 but not on the latest beta6
there are no releases on github so I'm just trying to find the changelogs to see what to do about iterations
.iterations(int) method has been removed (was rarely used and confusing to users) k, nvm
stefanojj
@stefanojj
good morning i am new in the AI topic what would be the first place for me to start about setting up an AI
?
acon10
@acon10
Hello friends, is it possible to run a java program on Google Colab?
Sean
@seanradford

Hello,

Ideas appreciated...

Have a test network that trains happily with ParameterAveragingTrainingMaster on a 2 machine cluster, but when I switch to SharedTrainingMaster it errors with the following:

20/03/20 10:23:40 ERROR BaseTransport: Exception: {}
java.lang.RuntimeException: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
    at org.nd4j.parameterserver.distributed.v2.transport.impl.BaseTransport.internalProcessMessage(BaseTransport.java:523)
    at org.nd4j.parameterserver.distributed.v2.transport.impl.BaseTransport$2.run(BaseTransport.java:183)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.nd4j.parameterserver.distributed.v2.transport.impl.BaseTransport$1$1.run(BaseTransport.java:115)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
    at java.util.ArrayList.rangeCheck(ArrayList.java:657)
    at java.util.ArrayList.get(ArrayList.java:433)
    at org.nd4j.parameterserver.distributed.v2.transport.impl.BaseTransport.getRandomDownstreamFrom(BaseTransport.java:562)
    at org.nd4j.parameterserver.distributed.v2.ModelParameterServer$4.accept(ModelParameterServer.java:295)
    at org.nd4j.parameterserver.distributed.v2.ModelParameterServer$4.accept(ModelParameterServer.java:288)
    at org.nd4j.parameterserver.distributed.v2.transport.impl.BaseTransport.internalProcessMessage(BaseTransport.java:521)

If I drop the 2nd slave (so 1 Spark master and 1 Spark slave on 1 physical machine), it is fine. My thought is related to open ports, but there are no firewalls enabled on either box.

The only interesting lines in the log that I can see are:

20/03/20 10:41:02 WARN BaseTransport: Got request from known node [aeron:udp?endpoint=192.168.0.12:49876]. Remapping.
20/03/20 10:41:02 INFO AeronUdpTransport: Trying to disconnect failed node: [aeron:udp?endpoint=192.168.0.12:49876]
20/03/20 10:41:02 INFO AeronUdpTransport: Trying to add failed node back again: [aeron:udp?endpoint=192.168.0.12:49876]
20/03/20 10:41:02 INFO AeronUdpTransport: Adding UDP connection: [aeron:udp?endpoint=192.168.0.12:49876]

(a couple of lines before the exception)

Help...

onil-innarin
@onil-innarin
@acon10 I don't think that would be possible
acon10
@acon10
Thank you for reply @onil-innarin
Hello friends, i have a question
I trained my dataset with Resnet50 and saved it as 'model.bin'
but i have no idea for predict any jpg image,
i restored my model as
ComputationGraph graph = ModelSerializer.restoreComputationGraph("D:\zooModel\resnet50\model.bin");
bu how can i predict jpg image on this model,
can you help me to complete the code?
Kenfack Arnaud
@linuxciscoarnaud_gitlab
Hello,
I need some help with the following issue:
eclipse/deeplearning4j#8811
Thanks.
Siddhartha Gadgil
@siddhartha-gadgil
I was experimenting with representation learning in dl4j, and for now was just trying to understand how to run the activate function for a layer (obatined by getLayer). However this needs a LayerWorkspaceMgr as an argument. What is the recommended way to get one? Thanks.
Siddhartha Gadgil
@siddhartha-gadgil
I see that there is a LayerWorkspaceMgr.Builder class which I can instantiate and build. I will experiment with that. If there is a better way recommended, I would be grateful if you let me know.
acon10
@acon10
@acon10 By the way i found the solution, thank you for your helping
Nino
@weinino
Hi there
I have some performance issues during the validation:
1.) I would like to have the test loss and some additional evaluations. Therefore, I would call model.score(testDataSet) to get the loss and model.doEvaluation(testDataSet, eval1, eval2). This however runs the inference twice, right? What would be the way to go here?
2.) My test dataset is too large to fit on the gpu. Thus, I would like to evaluate the model in batches as well. Is there a form of evaluation that will aggragate the metrics over several test data batches as if it would have been run as one single large batch?
Thank you very much :-)
Aleksandar Rogovic
@lavajaw
Hello, i need examples for dl4j unsupervised models (clustering). Any starting point you will recommend?