Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • 08:57
    csadorf synchronize #253
  • 08:57

    csadorf on schema-version

    Add schema version validation. Add test and use Version class. Update changelog. and 24 more (compare)

  • 08:56
    csadorf synchronize #253
  • 08:56

    csadorf on schema-version

    Add 'filelock' and 'packaging' … (compare)

  • 03:45
    bdice synchronize #235
  • 03:45
    bdice review_requested #235
  • 03:45

    bdice on h5store-examples

    Improve the reporting of benchm… Catch deprecation warning Remove csadorf from users alway… and 12 more (compare)

  • 03:45

    bdice on small-fixes

    (compare)

  • 03:45

    bdice on master

    Small documentation fixes in pr… (compare)

  • 03:45
    bdice closed #256
  • 03:45
    bdice milestoned #257
  • 03:45
    bdice labeled #257
  • 03:45
    bdice assigned #257
  • 03:45
    bdice opened #257
  • 03:42
    bdice milestoned #256
  • 03:42
    bdice labeled #256
  • 03:42
    bdice assigned #256
  • 03:42
    bdice review_request_removed #256
  • 03:41
    bdice review_requested #256
  • 03:41
    bdice review_request_removed #256
Bradley Dice
@bdice
@berceanu (I skimmed the conversation above, so forgive me if I'm missing something.) Are you sure you need to use MPI at all?
It sounded to me like your operation isn't MPI-aware, and you really just want two separate processes (one running on each GPU). Try adding a directive for ngpu=1 and submitting bundles of size 2 with --pretend, and look at the resulting submission script.
vyasr
@vyasr
sorry i lost track of this conversation during the sprint
you’re right, it’s not MPI-aware. however, he needs to make sure that the two operations end up on separate GPUs, which is not guaranteed without either using MPI with separate ranks to place them or using CUDA_VISIBLE_DEVICES like he has
Bradley Dice
@bdice
To add that to your template, you'd want to edit the for loop to include something like export CUDA_VISIBLE_DEVICES={{ loop.index0 }} (docs: https://jinja.palletsprojects.com/en/2.10.x/templates/#for)
vyasr
@vyasr
@berceanu the splitting over ranks can be accomplished similarly to what you’re already doing in your template
yes, exactly. your existing template is already doing that, but to get the benefits of using flow you would need to change the template to extend our built-in one
specifically, you’d probably want to extend this template: https://docs.signac.io/projects/flow/en/latest/supported_environments/slurm.html
the section you need to change can be found in the base script https://docs.signac.io/projects/flow/en/latest/supported_environments/base.html#base-script
if you look at the body block, you see that it's doing the loop over operations that you have in your template
you would need to rewrite the body block, probably by copy-pasting most of the block in there and just adding an export CUDA_VISIBLE_DEVICES as you’re doing in your current script.sh template
Mike Henry
@mikemhenry
User & Dev meeting in ~20 minutes https://bluejeans.com/429240743
Andrei Berceanu
@berceanu
Hey did you guys get to have a look at https://cli.dev/
Matt Thompson
@mattwthompson
I skimmed the tutorial, without getting my hands dirty I'm not able to appreciate its utility
although the tab-completion caught my eye, looks nice
Mike Henry
@mikemhenry
I think if you were building a CLI application starting today it would be worth looking into
Ali Malek
@phmalek
Hi everyone,
how can I change the values of a state point? I mean, if there is any command line such that I can modify the content of the json file.
Carl Simon Adorf
@csadorf
@phmalek Hi Ali, the state point is usually changed through the Python interface. There you can make the change directly: http://docs.signac.io/en/latest/jobs.html#modifying-the-state-point
If you wanted to do that from the command line, you could use something like this:
$ signac shell -c "for job in jobs: job.sp.setdefault('b', True)"
Ali Malek
@phmalek
great! thanks Carl!
Bradley Dice
@bdice
@phmalek Also, it's worth noting that you should not edit the JSON files directly, otherwise you will break the data model that signac relies on for efficient performance. The job id is a hash of the statepoint values, and if those don't correspond, searching/opening jobs may break in unexpected ways.
vyasr
@vyasr
actually in that case signac will typically throw a JobsCorruptedError indicating that one of your job ids doesn’t match your state point, so hopefully it wouldn’t catch you unaware. modifying through the Python API, and specifically using signac shell as shown above, is the best approach
Mike Henry
@mikemhenry
Another great reason to use signac within a lab, I needed to pull some data from a project that another grad student did, because they used signac it was easy to use signac schema -x and signac find to find exactly what I needed. Had to poke into a few different flow projects but it was easy to quickly figure out if I found the right one
No one will be able to find/figure out the stuff I did before I used the signac framework
Bradley Dice
@bdice
@mikemhenry ^ Can I quote you on that in my AIChE talk?
Mike Henry
@mikemhenry
Yes!
I guess that reads like a testimonial, but its 100% true!
4/5 graduate students recommend signac
Matt Thompson
@mattwthompson
Not sure if this is something that signac supports better now, but we have had issues in the past with permissions when multiple of us work on the same project at once; i.e. if I run a bunch of jobs, my collaborator won't be able to access those files. By this I mean permission in the Unix sense
Mike Henry
@mikemhenry
That is a umask setting that needs to be changed
That sets up "when I make a new file, what permissions should it have".
Andrei Berceanu
@berceanu
Just wanted to pop in to say hi
Andrei Berceanu
@berceanu
By the way, were you guys aware of https://github.com/snakemake/snakemake
There seems to be some overlap with signac-flow.
vyasr
@vyasr
yes, we’ve seen snakemake before. it definitely has some overlap with signac-flow, the API is quite different since it’s basically like writing a makefile but you could definitely use signac and snakemake in concert
Andrei Berceanu
@berceanu
But it seems snakemake is a subset of signac-flow
Plus it doesn't have the signac part at all
vyasr
@vyasr
right, it’s only the workflow piece. i’ve never looked at it in-depth, in what sense is it a subset of flow? what features does flow have that snakemake does not?
i’m curious, since i’ve never tried it myself
Andrei Berceanu
@berceanu
Snakemake works on targets which are files
Flow works on conditions which can be anything returning a book
Bool
vyasr
@vyasr
got it. there’s definitely a lot of flexibility gain there
Andrei Berceanu
@berceanu
There's no way to force the re-execution of an operation right?
vyasr
@vyasr
Depends on what you’re trying to accomplish. If you just need to rerun a specific operation, you can use python project.py exec
Carl Simon Adorf
@csadorf
@berceanu For the purpose of debugging? I usually add a temporary "@Project.post.never" to force the re-execution of that specific condition. Compared to exec it has the additional benefit that it won't run if the pre-conditions are not met.
Andrei Berceanu
@berceanu
Oh that's cool, didn't know about it!
Carl Simon Adorf
@csadorf
@berceanu I just added a recipe describing just that: glotzerlab/signac-docs#61
Andrei Berceanu
@berceanu
Looks good to me;)
Carl Simon Adorf
@csadorf
@berceanu Thx for reviewing!
Andrei Berceanu
@berceanu
No prob!