These are chat archives for deeplearning4j/deeplearning4j/earlyadopters

18th
Mar 2018
Fariz Rahman
@farizrahman4u
Mar 18 18:22
@treo I am on windows and trying to build nd4j, but the uberjar is empty
java version "1.8.0_161"
gitterBot
@raver120
Mar 18 18:23
@farizrahman4u Welcome! Here's a link to Deeplearning4j's Gitter Guidelines, our documentation and other DeepLearning resources online. Please explore these and enjoy! https://github.com/deeplearning4j/deeplearning4j/blob/master/GITTER_GUIDELINES.md
Fariz Rahman
@farizrahman4u
Mar 18 18:24
Maven 3.5.2
Paul Dubs
@treo
Mar 18 18:30
@farizrahman4u can you tell me what you are doing, step by step?
Fariz Rahman
@farizrahman4u
Mar 18 18:32

clone libnd4j
ran ./buildnativeoperations.sh
set LIBND4J_HOME

clone nd4j
mvn clean install -DskipTests -D maven.javadoc.skip=true -pl '!:nd4j-cuda-8.0,!:nd4j-cuda-8.0-platform,!:nd4j-tests'

Paul Dubs
@treo
Mar 18 18:33
And that last build call works?
Fariz Rahman
@farizrahman4u
Mar 18 18:33
Yes. And it creates the jar files
Paul Dubs
@treo
Mar 18 18:34
ok, so what exactly is the problem then?
Fariz Rahman
@farizrahman4u
Mar 18 18:35
The jar files are empty. Only 3kb
Paul Dubs
@treo
Mar 18 18:35
all of them?
Fariz Rahman
@farizrahman4u
Mar 18 18:35
nd4j-uberjar-0.9.2-SNAPSHOT.jar
Paul Dubs
@treo
Mar 18 18:38
if I read that pom.xml file correctly, you have to build the uberjar seperately. change into its subdirectory, and run mvn clean package -Puberjar and you should get the uberjar in the target directory
Fariz Rahman
@farizrahman4u
Mar 18 18:40
[ERROR] Failed to execute goal on project nd4j-uberjar: Could not resolve dependencies for project org.nd4j:nd4j-uberjar
:jar:0.9.2-SNAPSHOT: The following artifacts could not be resolved: org.nd4j:flatbuffers:jar:0.9.2-SNAPSHOT, org.nd4j:pr
otbuf:jar:0.9.2-SNAPSHOT: Failure to find org.nd4j:flatbuffers:jar:0.9.2-SNAPSHOT in https://oss.sonatype.org/content/re
positories/snapshots was cached in the local repository, resolution will not be reattempted until the update interval of
 sonatype-nexus-snapshots has elapsed or updates are forced -> [Help 1]
Paul Dubs
@treo
Mar 18 18:42
@agibsonccc you've added those dependencies that are missing here, what's the idea there?
@farizrahman4u you can probably comment those out for now
also you'll have to package at least one backend into that uberjar, so try it with mvn clean package -Puberjar -Pnative
Fariz Rahman
@farizrahman4u
Mar 18 18:50
I removed flatbuffers and protobuf dependencies
And did mvn clean package -Puberjar -Pnative
Now i get
[ERROR] Failed to execute goal on project nd4j-uberjar: Could not resolve dependencies for project org.nd4j:nd4j-uberjar:jar
:0.9.2-SNAPSHOT: The following artifacts could not be resolved: org.nd4j:nd4j-native:jar:android-arm:0.9.2-SNAPSHOT, org.nd4
j:nd4j-native:jar:android-arm64:0.9.2-SNAPSHOT, org.nd4j:nd4j-native:jar:android-x86:0.9.2-SNAPSHOT, org.nd4j:nd4j-native:ja
r:android-x86_64:0.9.2-SNAPSHOT, org.nd4j:nd4j-native:jar:ios-arm64:0.9.2-SNAPSHOT, org.nd4j:nd4j-native:jar:ios-x86_64:0.9.
2-SNAPSHOT, org.nd4j:nd4j-native:jar:linux-x86_64:0.9.2-SNAPSHOT, org.nd4j:nd4j-native:jar:macosx-x86_64:0.9.2-SNAPSHOT, org
.nd4j:nd4j-native:jar:linux-ppc64le:0.9.2-SNAPSHOT: Failure to find org.nd4j:nd4j-native:jar:android-arm:0.9.2-SNAPSHOT in h
ttps://oss.sonatype.org/content/repositories/snapshots was cached in the local repository, resolution will not be reattempte
d until the update interval of sonatype-nexus-snapshots has elapsed or updates are forced -> [Help 1]
Adam Gibson
@agibsonccc
Mar 18 18:51
wait hold on..
@treo that was due to samediff
oh wait..ahh
that's the shading..we got rid of that
let me make sure there's a fix there
how did that just get picked up o_0
Paul Dubs
@treo
Mar 18 18:52
@farizrahman4u tries to build nd4j-uberjar
@farizrahman4u rerun it with -U so it looksup those things again
Adam Gibson
@agibsonccc
Mar 18 18:53
weird..is that on master?
Paul Dubs
@treo
Mar 18 18:53
@agibsonccc it is
Serhii Shepel
@sshepel
Mar 18 18:53
@agibsonccc > @farizrahman4u rerun it with -U so it looksup those things again
Adam Gibson
@agibsonccc
Mar 18 18:54
@treo wait..I never even touched that pom
look at the committers :D
Whatever..let me fix it
oh nvm
sorry just saw it
yeah my fault
Paul Dubs
@treo
Mar 18 18:55
I just wanted to post a screenshot with blame on :P
you did get rid of it on your own branch, not on master :D
Fariz Rahman
@farizrahman4u
Mar 18 18:58

@farizrahman4u rerun it with -U so it looksup those things again

Ok this is downloading a ton of stuff that i dont think i need

Paul Dubs
@treo
Mar 18 18:58
well, it looks very much like you need it, after all it complained that it couldn't find some stuff locally
or if you just want to build an uberjar with the artifacts that you've created locally, you'll have to remove the nd4j-native-platform dependency from that pom as well. As it is the reason why you did get that last error
Fariz Rahman
@farizrahman4u
Mar 18 19:03

@farizrahman4u rerun it with -U so it looksup those things again

Ok so this created a 158 MB jar file but still empty :(

image.png
Paul Dubs
@treo
Mar 18 19:04
open it as a zip file
it shouldn't be empty
also, refresh your maven in that project, intellij likes to linger on old versions when working with snapshots like that
Fariz Rahman
@farizrahman4u
Mar 18 19:10
I unzipped it
No class files
Paul Dubs
@treo
Mar 18 19:16
that's pretty weird, but you are right
I just looked into my what was built for me, and it has lots of other stuff, but no class files
Fariz Rahman
@farizrahman4u
Mar 18 19:17
But the snapshots from maven has class files
Also the snapshot of each sub project also has class files
Paul Dubs
@treo
Mar 18 19:17
is there an nd4j-uberjar snapshot?
Serhii Shepel
@sshepel
Mar 18 19:18
@treo yes
Paul Dubs
@treo
Mar 18 19:19
it is empty though
Fariz Rahman
@farizrahman4u
Mar 18 19:19
Ok so I just want to run some samediff tests.
What is the easiest way to include nd4j in another project
Serhii Shepel
@sshepel
Mar 18 19:20
@treo yeah, I have just checked to...
Paul Dubs
@treo
Mar 18 19:21
@farizrahman4u for the most part, if you want to run something in nd4j-tests, then you can simply go into it, and run the test via maven: mvn test -Dtest=yourtesthere
I use that all the time when I'm too lazy to recompile all of nd4j :D
Adam Gibson
@agibsonccc
Mar 18 19:23
Sorry had to find an outlet, I"ll commit a change here in a sec
Just building locally really fast before I merge
Paul Dubs
@treo
Mar 18 19:25
@agibsonccc while you wait for the build to finish, I've got a question: what do you think about a LossFunction that simply combines other loss functions? i.e. something like (using Kotlin syntax) LossCombine(LossMSE() to 1.0, LossBinaryXENT() to 2.0)
they are combined to a weighted average
Adam Gibson
@agibsonccc
Mar 18 19:25
I'm not against that
submit a PR
Paul Dubs
@treo
Mar 18 19:26
Sure, I'll have to convert it to java first, but that shouldn't be too much of an issue
I'm currently using it already with some custom loss functions :D
Adam Gibson
@agibsonccc
Mar 18 19:30
merged
The rest of this should be fine now
Paul Dubs
@treo
Mar 18 19:31
actually it is still kind of weirdly behaving
Adam Gibson
@agibsonccc
Mar 18 19:32
What's missing?
Paul Dubs
@treo
Mar 18 19:32
class files
Adam Gibson
@agibsonccc
Mar 18 19:32
I'd seriously just rather fix this
Let's not hand wave here :D
Paul Dubs
@treo
Mar 18 19:32
all of them
Adam Gibson
@agibsonccc
Mar 18 19:32
This is only the nd4j-uberjar?
Let me look
Paul Dubs
@treo
Mar 18 19:32
the uberjar includes a lot of stuff, but not a single class file
minimizeJar boolean 1.4 When true, dependencies will be stripped down on the class level to only the transitive hull required for the artifact. Note: Usage of this feature requires Java 1.5 or higher.
Adam Gibson
@agibsonccc
Mar 18 19:35
settle down folks
it's wrong and I see why
Paul Dubs
@treo
Mar 18 19:36
so I guess all that is needed is to set that to false :)
Adam Gibson
@agibsonccc
Mar 18 19:36
yeah that's part of it..the other problem is the way the builds are setup
it has platform dependencies in there when it shouldn't
I added a -snapshots profile that gets rid of that
I'm testing that now
Paul Dubs
@treo
Mar 18 19:36
:)
Serhii Shepel
@sshepel
Mar 18 19:37
:sparkles:
Adam Gibson
@agibsonccc
Mar 18 19:41
k folks give it a run
mvn -Pnative-snapshots -Puberjar clean install -DskipTests
That will work with the new build
Paul Dubs
@treo
Mar 18 19:42
:+1:
works fine
Adam Gibson
@agibsonccc
Mar 18 20:13
great!