It sounded to me like your operation isn't MPI-aware, and you really just want two separate processes (one running on each GPU). Try adding a directive for
ngpu=1 and submitting bundles of size 2 with --pretend, and look at the resulting submission script.
you’re right, it’s not MPI-aware. however, he needs to make sure that the two operations end up on separate GPUs, which is not guaranteed without either using MPI with separate ranks to place them or using CUDA_VISIBLE_DEVICES like he has
To add that to your template, you'd want to edit the
for loop to include something like export CUDA_VISIBLE_DEVICES={{ loop.index0 }} (docs: https://jinja.palletsprojects.com/en/2.10.x/templates/#for)
yes, exactly. your existing template is already doing that, but to get the benefits of using flow you would need to change the template to extend our built-in one
specifically, you’d probably want to extend this template: https://docs.signac.io/projects/flow/en/latest/supported_environments/slurm.html
the section you need to change can be found in the base script https://docs.signac.io/projects/flow/en/latest/supported_environments/base.html#base-script
if you look at the
body block, you see that it's doing the loop over operations that you have in your template
you would need to rewrite the body block, probably by copy-pasting most of the block in there and just adding an
export CUDA_VISIBLE_DEVICES as you’re doing in your current script.sh template
although the tab-completion caught my eye, looks nice
@phmalek Hi Ali, the state point is usually changed through the Python interface. There you can make the change directly: http://docs.signac.io/en/latest/jobs.html#modifying-the-state-point
If you wanted to do that from the command line, you could use something like this:
If you wanted to do that from the command line, you could use something like this:
$ signac shell -c "for job in jobs: job.sp.setdefault('b', True)"
@phmalek Also, it's worth noting that you should not edit the JSON files directly, otherwise you will break the data model that signac relies on for efficient performance. The job id is a hash of the statepoint values, and if those don't correspond, searching/opening jobs may break in unexpected ways.
actually in that case signac will typically throw a
JobsCorruptedError indicating that one of your job ids doesn’t match your state point, so hopefully it wouldn’t catch you unaware. modifying through the Python API, and specifically using signac shell as shown above, is the best approach
Another great reason to use signac within a lab, I needed to pull some data from a project that another grad student did, because they used signac it was easy to use
signac schema -x and signac find to find exactly what I needed. Had to poke into a few different flow projects but it was easy to quickly figure out if I found the right one
No one will be able to find/figure out the stuff I did before I used the signac framework
I guess that reads like a testimonial, but its 100% true!
4/5 graduate students recommend signac
Not sure if this is something that signac supports better now, but we have had issues in the past with permissions when multiple of us work on the same project at once; i.e. if I run a bunch of jobs, my collaborator won't be able to access those files. By this I mean permission in the Unix sense
That sets up "when I make a new file, what permissions should it have".
There seems to be some overlap with signac-flow.
Plus it doesn't have the signac part at all
i’m curious, since i’ve never tried it myself
Flow works on conditions which can be anything returning a book
Bool