whats the best way to deal with nvidia-docker not being available on ubuntu 18.04? (only nvidia-docker2)
tried loading from a repo with the older package but still getting
The following packages have unmet dependencies:
nvidia-docker : Depends: sysv-rc (>= 2.88dsf-24) but it is not installable or
file-rc (>= 0.8.16) but it is not installable
E: Unable to correct problems, you have held broken packages.
and can't figure out how I got around this in the past
Under: "Create a New Project" heading here:
The function to invoke multiple videos is shown to be:
deeplabcut.create_new_project(`Name of the project',`Name of the experimenter', [`Full path of video 1',`Full path of video2',`Full path of video3'], working_directory=`Full path of the working directory',copy_videos=True/False)
So lets say you already have ProjectA, ProjectB, and Project C -- but you want to merge them into Project1. If you already have some of the frames of other videos labeled, they will be in
./ProjectA/labeled-data/<Video1>_labeled. So you can copy those two directories (the ones inside
./ProjectA/labeled-data/ ) to .
/Project1/labeled-data/. Then do the same for
Assuming you have enough labeled frames covering most of the unique cases in your footage, you can skip
deeplabcut.label_frames() to go straight to
P.S., I don't think
./ProjectA/labeled-data/<Video1>_labeled is actually used by the software - i think it's generated for human convenience only when
deeplabcut.check_labels() is ran. The actual labels are stored in
whats the best way to test that my container has the ability to use GPU for training?
Inside docker, nvcc / nvisida-smi seems to be working fine, e.g.
ubuntu@4d3df5a62144:/work$ nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2017 NVIDIA Corporation
Built on Fri_Sep__1_21:08:03_CDT_2017
Cuda compilation tools, release 9.0, V9.0.176
deeplabcut.train_network(config_path,shuffle=1,trainingsetindex=0,gputouse=0,max_snapshots_to_keep=1,autotune=False,displayiters=None,saveiters= None, maxiters=None)
results in CPU max loading, and no raise in GPU memory utilization (sticks at 0.4GB / 1GB regardless if training or not). Listing processes in nvidia-smi is not supported by my graphics card, however my GPU does seem to support CUDA 9 with a compute capability of 3.0
Doesn't look like I've got TF running on GPU. Inside the container, running the following in ipython
import tensorflow as tf ...: with tf.device('/gpu:0'): ...: a = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[2, 3], name=' ...: a') ...: b = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0, 6.0], shape=[3, 2], name=' ...: b') ...: c = tf.matmul(a, b) ...: ...: with tf.Session() as sess: ...: print (sess.run(c))
InvalidArgumentError (see above for traceback): Cannot assign a device for operation 'MatMul_1': Operation was explicitly assigned to /device:GPU:0 but available devices are [ /job:localhost/replica:0/task:0/device:CPU:0 ]. Make sure the device specification refers to a valid device.
@tvajtay thanks for teaching me a new command with
between then and now I ran
conda install tensorflow-gpu=1.8.0 so it definitely shows up at the moment...
I'm not sure if it was there earlier; I was using the
https://github.com/eqs/DeepLabCut-Docker/blob/master/docker/Dockerfile (eqs) docker image. If you're familiar with it, you may know whether it had tf-gpu installed.
conda install tensorflow-gpu=1.8.0inside the container I still have the same issue when I repeat the test above (