dmlc/mxnet

Lightweight, Portable, Flexible Distributed/Mobile Deep Learning with Dynamic, Mutation-aware Dataflow Dep Scheduler; for Python, R, Julia, Go, and more

Bob Poekert

@bobpoekert

is there some way to manually specify the input shape?

@akuz

I've built MXNet from source yesterday, and I really want to use it from my C++ project. But I find it unreasonably complicated to find out how to link to it properly, so that my app can find all the header files and all the compiled libraries. I am using cmake. Can anyone recommend a good tutorial/example? I've been searching and trying for two days without success.

Anton Chernov

@lebeg

@akuz Now it is indeed very complicated to use the C++ module of MXNet because of multiple reasons including linking. The most simple way I see is using the c_api.h directly - it has all the needed functionality and is used by all language bindings. You can see an example of using the inference API in the (image-classification) example.

khaotik

@khaotik

Hi, all. What's the right way send/recv tensors across nodes over network? I guess I can hack some zmq sockets together, but is there a standard way to do it?

khaotik

@khaotik

For my use case, it's about separating data production / training pipeline. I'm producing synthetic data on the producer node and sending them to the training node.

Steffen Rochel

@srochel

I'm suggesting to move MXNet related discussions to https://discuss.mxnet.io/ and ASF MXNet Slack channel. I'm offering to add you to ASF MXNet Slack channel (if your email is visible), but let me know if you are not interested or send request to dev@mxnet.apache.org to be added to ASF MXNet Slack channel. You also have the opportunity to subscribe to dev@mxnet.apache.org and user@mxnet.apache.org (more details as well on http://mxnet.incubator.apache.org/community/contribute.html). Looking forward to our collaboration.

sam-h-bean

@sam-h-bean

I have some questions about the Scala API for the Symbol library

I'm trying to figure out how to do something like this https://github.com/ufoym/mxnet/blob/master/example/vae/VAE.py#L83 however it seems the scala symbol api does not allow the mixing of symbols and constants like the python library does.

It seems like if I want to use constants in my loss functions I'm going to have to have a very large argsDict when I go to train. Is there a better way>?

Steffen Rochel

@srochel

Bob Poekert

@bobpoekert

it looks like I'm hitting this condition: https://github.com/apache/incubator-mxnet/blob/master/src/operator/tensor/matrix_op-inl.h#L664

should that ever happen or does that indicate a bug in mxnet?

Steffen Rochel

@srochel

Hi Bob -

Ashutosh Dwivedi

@aashudwivedi

I am trying to run the mxnet in distributed mode using spark as implemented here : https://github.com/apache/incubator-mxnet/tree/master/scala-package/spark

but I am not able to run the examples / tests.

The commented tests in the file : https://github.com/apache/incubator-mxnet/blob/master/scala-package/spark/src/test/scala/org/apache/mxnet/spark/MXNetGeneralSuite.scala keep running into error

Exception in thread "main" java.lang.NoClassDefFoundError: scala/collection/Seq
    at java.lang.Class.getDeclaredMethods0(Native Method)
    at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
    at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
    at java.lang.Class.getMethod0(Class.java:3018)
    at java.lang.Class.getMethod(Class.java:1784)
    at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
    at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: scala.collection.Seq
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 7 more

I have seen this error mostly when there's a mismatch between the scala versions but I am not able to figure out which versions might be a mismatch in this case

all the libraries in pom are 2.11 version

Aaron Markham

@aaronmarkham

@aashudwivedi there's a mxnet-scala chat room on the MXNet Slack - you might want to head over there!

Josaped Moreno

@androuino

Hi, I just some help please. I have successfully trained a yolo3_darknet53_custom model and generated a json and params files, however when testing it with Java API, I was having this strange error:

Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: Mismatch between shape (100,1) and (1,100,4)
    at scala.Predef$.require(Predef.scala:224)
    at org.apache.mxnet.NDArray$$anonfun$9.apply(NDArray.scala:448)
    at org.apache.mxnet.NDArray$$anonfun$9.apply(NDArray.scala:445)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
    at scala.collection.Iterator$class.foreach(Iterator.scala:893)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
    at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
    at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
    at scala.collection.AbstractTraversable.map(Traversable.scala:104)
    at org.apache.mxnet.NDArray$.concatenate(NDArray.scala:445)
    at org.apache.mxnet.module.BaseModule$$anonfun$2.apply(BaseModule.scala:267)
    at org.apache.mxnet.module.BaseModule$$anonfun$2.apply(BaseModule.scala:267)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
    at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
    at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
    at scala.collection.AbstractTraversable.map(Traversable.scala:104)
    at org.apache.mxnet.module.BaseModule.predict(BaseModule.scala:267)
    at org.apache.mxnet.infer.Predictor$$anonfun$11.apply(Predictor.scala:210)
    at org.apache.mxnet.infer.Predictor$$anonfun$11.apply(Predictor.scala:210)
    at org.apache.mxnet.infer.MXNetThreadPoolHandler$$anon$4.call(MXNetHandler.scala:73)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':ObjectDetectionTutorial.main()'.
> Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with non-zero exit value 1

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 1s
2 actionable tasks: 1 executed, 1 up-to-date
Process 'command '/usr/lib/jvm/java-8-openjdk-amd64/bin/java'' finished with non-zero exit value 1
11:48:21: Task execution finished 'ObjectDetectionTutorial.main()'.

My settings are: image size is 512, class is 1, and pretrained_base=False.
This is the generated json file: https://gist.github.com/androuino/0b5e36c2fb39be2bda6cd8fc8bdadd05
and this is the param model: https://filebin.net/n89gzmb285oxre2g

Thank you so much.

Aaron Markham

@aaronmarkham

@androuino - I recommend you check out these avenues as they have more traffic and people ready to answer:
https://mxnet.incubator.apache.org/versions/master/community/contribute.html#mxnet-dev-communications

the forum is pretty active and there's also a scala channel on the mxnet's slack

Rajarajan

@generaljam

MXNet scala "Try loading mxnet-scala from native path" help me please , am doing it in IntelliJ maven

Peng Cheng

@tribbloid

does any one think it's the right moment to start scaffolding scala/clojure/JVM gluon API for training?

I can't find an answer on forum: https://discuss.mxnet.io/t/what-is-the-status-of-gluon-api-for-scala-clojure-jvm-autograd-api/5088/2
so I post it here

Deepak Battini

@deepakkumar1984

Hello,

I am working on the CSharp binding for MxNet and able to complete the Imperative, Symbolic and Gluon packages where the code is implemented based on MxNet 1.5.1 version. Can this repo be mentioned on MxNet documentation. I will work on developing the C# version of the API docs since we have similar for Scala, R binding as well.

Repo: https://github.com/SciSharp/MxNet.Sharp

Thanks for your hardwork in buiding this amazing library.

Regards,
Deepak

Pablo A. F. Mercuri

@pafmercuri

Hi, has anyone tried to run a multiple instance segmentation with mxnet on scala?

Andrew O'Reilly-Nugent

@aornugent

Heyo - I'm trying to build from source on Windows. CMake runs and returns 'Generating done, build files have been written to ../mxnet/build' but I can't see a libmxnet.dll anywhere. Any ideas what I might be doing wrong?

Is there another step after running cmake?

Andrew O'Reilly-Nugent

@aornugent

Ah, apparently:

cd mxnet
cmake -S . -B build
cd build
cmake --build .

Deepak Battini

@deepakkumar1984

Working towards aligning with MxNet v2.0: https://github.com/deepakkumar1984/MxNet.Sharp. C# Gluon Package will be updated and will be including more examples.

Mikkel Hvilshøj Funch

@MikkelFunch

Hi
I have a lot of data in parquet files right now, but would like move to use a gluon dataset(custom or otherwise), which requires random access. I cannot get that from parquet files. I generate the training files with scala spark and run the training in python. Does anybody have any ideas what would be smart? I cannot write recordio files from scala spark. I am trying with Zarr right now, but I am running into issues

There is too much data for it to be in memory, that is the reason we are looking for alternatives. And we also want to start training on gpu, which seems seamless with gluon

Where communities thrive

People

Repo info

Activity