Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Aug 26 2020 22:03
    User @agibsonccc unbanned @farizrahman4u
  • Aug 26 2020 22:02
    User @agibsonccc unbanned @RobAltena
  • Aug 26 2020 22:02
    User @agibsonccc unbanned @gsw85
  • Aug 26 2020 22:02
    User @agibsonccc unbanned @raver120
  • Aug 26 2020 22:01
    User @agibsonccc unbanned @cagneymoreau
  • Aug 20 2020 21:19
    @chrisvnicholson banned @cagneymoreau
  • Aug 20 2020 21:15
    @chrisvnicholson banned @konduitops
  • Aug 20 2020 21:13
    User @chrisvnicholson unbanned @eraly
  • Aug 20 2020 21:12
    User @chrisvnicholson unbanned @treo
  • Aug 20 2020 21:12
    User @chrisvnicholson unbanned @AlexDBlack
  • Jul 07 2020 22:38
    @chrisvnicholson banned @raver120
  • May 31 2020 20:46
    @chrisvnicholson banned @ShamsUlAzeem
  • May 31 2020 20:46
    @chrisvnicholson banned @farizrahman4u
  • May 31 2020 20:44
    @chrisvnicholson banned @gsw85_twitter
  • May 31 2020 20:44
    @chrisvnicholson banned @gsw85
  • May 31 2020 20:43
    @chrisvnicholson banned @AlexDBlack
  • May 31 2020 20:43
    @chrisvnicholson banned @agibsonccc
  • May 31 2020 20:43
    @chrisvnicholson banned @RobAltena
  • May 31 2020 20:42
    @chrisvnicholson banned @eraly
  • May 31 2020 20:42
    @chrisvnicholson banned @treo
dkarampi
@dkarampi
Hi. Not sure if I can directly ask a question here, but I'll give it a try: I'm trying to import a keras model (via an h5) that contains a custom Layer. May I ask if I need to implement these layers myself by extending the KerasLayer class? There are two exapmpes for that at github but they didn't help my much. Is there any other pointer?
Dave Cramer
@davecramer
is there a recent cnn example for images ?
Hanyang Li
@harryli:matrix.org
[m]
Hello, I am completely new to DL4J so sorry if I say anything incorrect. I'm currently stuck on the mvn clean install step of building. I have all the environment variables set up as well as everything downloaded. Attached below is a screenshot of the error message. Please let me know if you need any more information, Thanks so much.
Dave Cramer
@davecramer
Is there a Gramian encoder anywhere in DL4J
Eduardo Gonzalez
@wmeddie
That is new to me. No Google results either. What is it?
dkarampi
@dkarampi
I've noticed that there hasn't been any release since May 2020. When is the next one coming?
dkarampi
@dkarampi
It seems there are quite a few fixes not out yet. I tried to build from source but it didn't work out
Eduardo Gonzalez
@wmeddie
Real soon
@Override
public INDArray preProcess(INDArray input, int miniBatchSize) {
if(input.rank() != 4) throw new IllegalArgumentException("Invalid input: expect CNN activations with rank 4 (received input with shape " + Arrays.toString(input.shape())+")");
//Input: 4d activations (CNN)
//Output: 3d activations (RNN)
    if(input.ordering() != 'c') input = input.dup('c');

    int[] shape = input.shape();    //[timeSeriesLength*miniBatchSize, numChannels, inputHeight, inputWidth]

    //First: reshape 4d to 2d, as per CnnToFeedForwardPreProcessor
    INDArray twod = input.reshape('c',input.size(0), ArrayUtil.prod(input.shape())/input.size(0));
    //Second: reshape 2d to 3d, as per FeedForwardToRnnPreProcessor
    INDArray reshaped = twod.dup('f').reshape('f',miniBatchSize,shape[0]/miniBatchSize,product);
    return reshaped.permute(0,2,1);
}
how can i know about timeSeriesLength value?
is there any way to check that?
anonhm09
@anonhm09
@Bren077s hi can you share java code of Graph
Adam Gibson
@agibsonccc
@dkarampi you can use snapshots for now, those are published. For now, I'm doing a major pass on the tests and the CI to ensure we can release more often and consistently. We're also prepping for a major 1.0 release.
chinproisbestpro
@chinproisbestpro

Hi, I'm running into a problem when I use CUDA for my model with custom dataset.
When I use CPU to run my model, it works fine, but when I use
nd4j-cuda-10.1 with my 2060Super, my evaluation metrics are just all 0's

========================Evaluation Metrics========================
 # of classes:    2
 Accuracy:        0.0000
 Precision:       0.0000
 Recall:          0.0000
 F1 Score:        0.0000
Precision, recall & F1: reported for positive class (class 1 - "1") only

Warning: 1 class was never predicted by the model and was excluded from average precision
Classes excluded from average precision: [0]
Warning: 1 class was never predicted by the model and was excluded from average recall
Classes excluded from average recall: [1]

When I run example code with CUDA it works fine. When I use my same model but with a DataSet like MnistDataSetIterator it also works fine. So it must be something with my Dataset. Are there some restrictions or gotchas with CUDA only? Again, my model works correctly with CPU
Thanks!

chinproisbestpro
@chinproisbestpro
Figured it out with the help of Adam. It was an issue of using dataset.shuffle() with GPU.
Joao Madeira
@jmadeira
Hi! Anyone still here?
How do I implement HuberLoss function?
Eduardo Gonzalez
@wmeddie
By implementing ILoss.
Joao Madeira
@jmadeira
Thank you. Is there any example? :D
Don’t yet have an example using SameDiffLoss though. Pull Requests appreciated there.
Joao Madeira
@jmadeira
Thanks. btw @wmeddie are you guys going to implement the huber loss function in the near future?
Eduardo Gonzalez
@wmeddie
It already exists in SameDiff and ND4J… try this?
image.png
Joao Madeira
@jmadeira
I will. Thanks Eduardo
karlmcphee
@karlmcphee
I'm trying to run a machine learning model on AWS EMR and I'm getting the error java.lang.ClassNotFoundException: org.nd4j.linalg.learning.config.IUpdater even though the model works in intellij. Is there a good way for me to bugcheck this and make sure that I have the appropriate libraries?
karlmcphee
@karlmcphee
Just to add, I'm importing org.nd4j.linalg.learning.config.IUpdater already even though it isn't used directly in my model.
Adam Gibson
@agibsonccc
@karlmcphee hard to tell what your issue is. Can you please post a completely reproducible problem (dependencies, how you're running your commands, versions of software,..) on the forums? https://community.konduit.ai thanks!
karlmcphee
@karlmcphee
karlmcphee
@karlmcphee
I realized that I didn't have an nd4j backend set up. Is there a good practice to set up maven dependencies so that my project will work both on an EMR with CUDA 9.2 and my settings? Do I need CUDA version 9.2 for this?
xiptos
@xiptos
Hello, everyone! Just a quick question, to see if there is a better way. I know that it is possible to use CnnSentenceDataSetIterator for RNN and CNN training/classification using WordVectors. Is it also possible to use it with a Vectorizer (e.g. TfidfVectorizer)?
Violet981
@Violet981
image.png
Hi! I am wondering if the ComposableRecordReader could compose different numbers of records from different recordReader. Let's say we have imageRecordReader, wavFileRecordReader, CSVRecordReader, while we have 3records of images and 3 records of wavfiles and 2 records of lables. Technically, can we use composablerecordreader to compose them together, having 2 records containing images, wav filesand labels, and 1 record only image and wave files? (I know it is meaningless in the practical process, I just wonder that since I look into the source code of ComposableRecordReader and there seems no such constraint)
Rhys Compton
@basedrhys

I've converted a bunch of Keras models into the Dl4j .zip format for WekaDeeplearning4j, a Weka wrapper for Dl4j. I was wondering whether the Dl4j maintainers would be interested in using these zip files for Dl4j? It would expand the Dl4j Model Zoo substantially:

  • Keras DenseNet121
  • Keras DenseNet169
  • Keras DenseNet201
  • Keras EfficientNetB0
  • Keras EfficientNetB1
  • Keras EfficientNetB2
  • Keras EfficientNetB3
  • Keras EfficientNetB4
  • Keras EfficientNetB5
  • Keras EfficientNetB6
  • Keras EfficientNetB7
  • Keras InceptionV3
  • Keras NASNetLarge
  • Keras NASNetMobile
  • Keras ResNet101
  • Keras ResNet101V2
  • Keras ResNet152
  • Keras ResNet152V2
  • Keras ResNet50
  • Keras ResNet50V2
  • Keras VGG16
  • Keras VGG19
  • Keras Xception

You can download them from here

I ask because I don't have the capacity to code up the architectures in Dl4j for each one of them, so the model architectures would only be available in pretrained format, but maybe that's ok (up to the maintainers I suppose).

Charles
@azanux:matrix.org
[m]
is there an example somewhere to load json file , split in train and test like DataSet trainingData = testAndTrain.getTrain();
DataSet testData = testAndTrain.getTest(); i am getting this error java.lang.IllegalStateException: Cannot split DataSet with <= 1 rows (data set has 1 example) , when i try to use JacksonLineRecordReader
lw7M
@lw7M
Hey, I feel kind of dumb here so sorry in advance. For the Doc2Vec sample: how do I structure the input data? I only found info for FileLabelAwareIterator with the labels as file structure. But I do not want to use a file system. So I would use LabelAwareDocumentIterator(?). How to fill that and how to structure the data? Is there a sample/example available?
Adam Gibson
@agibsonccc
@basedrhys @azanux:matrix.org @lw7M mind moving discussion to the forums at https://community.konduit.ai? It's a discourse which allows async discussions and actual indexable search results
lw7M
@lw7M
@agibsonccc hey, sure and thanks for the reply :)
Luke Czapla
@lukeczapla
greetings all, long time no see! I have been working with drop out for regularization and was wondering if there are built-in methods to actually drop out nodes at scoring time and not just during the training process
Luke Czapla
@lukeczapla
I seem to have a poorly posed problem with way too many inputs (features) and not enough samples, so I want to try a Monte Carlo technique
MankaranSingh
@MankaranSingh
hey guys, i am fairly new to java. So I want to keep running a model for inference in a while true loop. for all preprocessing and post processing, i am using nd4j INDArray. I have written quiet a lot of functions and each one of them creates many ndarrays and get called at every iteration of loop.
how can i improve the performance and is their a better way to manage this ?
Adam Gibson
@agibsonccc
@MankaranSingh @lukeczapla please post over on the forums at https://community.konduit.ai/ with more details and we'll reply when we can. Thanks!
Luke Czapla
@lukeczapla
ok thanks! posted and hopefully someone has some ideas or could point out the class / source code.
Raviteja
@raviteja777

the tfidf vectorizer has a transform method

What is the difference between transform and vectorize methods ?

Daniel Cotter
@danielcdev
Hey all, apologies for the rudimentary question, but what DataSetIterator are you supposed to use with a single layer LSTM that's -not- using regression for calculating precision numerical sequences of data? I'm attempting to use CSVSequenceRecordReader, but according to the code it's attempting to turn all labels into one-hot representation and failing because it's not a 1 or 0. I've attempted reading the documentation, but I can't seem to skim the nugget of information I need. What am I missing here? Happy to answer any follow up questions on this, as I'm not sure what all information you would need to discern this up front.
lzd010
@lzd010
@agibsonccc Will the next release be 1.0.0-M1?
AdrienDS72
@AdrienDS72

Hello everyone. I need some help to make a pipeline between TfidfRecordReader and RecordReaderDataSetIterator. This is my code :

   int batchSize = 100;
    int seed = 123;

    File rootTrainingTxtFolder = new File("./TRAINING/TXT");
    String[] allowedFormats=new String[]{".txt"};
    FileSplit fileSplit = new FileSplit(rootTrainingTxtFolder ,allowedFormats,new Random(seed));

    Configuration config = new Configuration();
    config.setBoolean(RecordReader.APPEND_LABEL, true);
    config.setInt(TextVectorizer.MIN_WORD_FREQUENCY, 1);

    TfidfRecordReader recordReader = new TfidfRecordReader();
    recordReader.initialize(config, fileSplit);

    int nbrLabel = recordReader.getLabels().size();

    DataSetIterator trainIter = new RecordReaderDataSetIterator.Builder(recordReader, batchSize)
            .classification(1, nbrLabel)
            .build();

    trainIter.next();

And i've got an exception :
Exception in thread "main" java.lang.IllegalStateException: Cannot put array: array should have leading dimension of 1 and equal rank to output array. Attempting to put array of shape [9418] into output array of shape [100]
at org.nd4j.common.base.Preconditions.throwStateEx(Preconditions.java:641)
at org.nd4j.common.base.Preconditions.checkState(Preconditions.java:340)
at org.deeplearning4j.datasets.datavec.RecordReaderMultiDataSetIterator.putExample(RecordReaderMultiDataSetIterator.java:544)
at org.deeplearning4j.datasets.datavec.RecordReaderMultiDataSetIterator.convertWritablesHelper(RecordReaderMultiDataSetIterator.java:516)
at org.deeplearning4j.datasets.datavec.RecordReaderMultiDataSetIterator.convertWritables(RecordReaderMultiDataSetIterator.java:454)
at org.deeplearning4j.datasets.datavec.RecordReaderMultiDataSetIterator.convertFeaturesOrLabels(RecordReaderMultiDataSetIterator.java:364)
at org.deeplearning4j.datasets.datavec.RecordReaderMultiDataSetIterator.nextMultiDataSet(RecordReaderMultiDataSetIterator.java:327)
at org.deeplearning4j.datasets.datavec.RecordReaderMultiDataSetIterator.next(RecordReaderMultiDataSetIterator.java:213)
at org.deeplearning4j.datasets.datavec.RecordReaderDataSetIterator.next(RecordReaderDataSetIterator.java:378)
at org.deeplearning4j.datasets.datavec.RecordReaderDataSetIterator.next(RecordReaderDataSetIterator.java:453)
at org.deeplearning4j.datasets.datavec.RecordReaderDataSetIterator.next(RecordReaderDataSetIterator.java:85)

What is wrong ? Need to use a WritableConverter ?