Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • May 14 20:39
    bdice synchronize #464
  • May 14 20:39

    bdice on introduce-aggregation

    Fix test so it works with or wi… (compare)

  • May 14 19:11
    bdice synchronize #464
  • May 14 19:11

    bdice on introduce-aggregation

    Use repr for error contents. Try splitting lines. Previous t… (compare)

  • May 14 18:58
    bdice synchronize #464
  • May 14 18:58

    bdice on introduce-aggregation

    Move tests for invalid job/aggr… (compare)

  • May 14 18:43
    bdice synchronize #464
  • May 14 18:43

    bdice on introduce-aggregation

    Add tests for invalid with_job … Fix spacing/comments. Add test for invalid job select… and 1 more (compare)

  • May 14 17:29
    bdice synchronize #464
  • May 14 17:29

    bdice on introduce-aggregation

    Improve docstrings. (compare)

  • May 14 17:13
    bdice closed #362
  • May 14 17:03
    bdice review_requested #464
  • May 14 17:03
    bdice review_requested #464
  • May 14 17:03
    bdice review_requested #464
  • May 14 16:59
    bdice synchronize #464
  • May 14 16:59

    bdice on introduce-aggregation

    Update test. (compare)

  • May 14 16:39
    bdice synchronize #464
  • May 14 16:39

    bdice on introduce-aggregation

    Add comment to clarify what hap… (compare)

  • May 14 16:30
    bdice synchronize #464
  • May 14 16:30

    bdice on introduce-aggregation

    Fix expected number of aggregat… (compare)

Vyas Ramasubramani
@vyasr
right now the problem you're running into is that an operation is only submitted if it is eligible to run at submit time. there is not dynamic dependency resolution (i'm assuming that you have pre- and post-conditions set that define the sequence of operations you posted above)
@berceanu with groups, you put a bunch of operations into a group and then submit the entire group. a group is composed of many operations, and a group can be submitted if any of its operations are eligible at submit time. inter-group dependencies will then be resolved when your submission script runs: flow will run any initially eligible operations, then once they are done flow will check if there are any new operations that are now eligible, and if so, run those
Andrei Berceanu
@berceanu
So basically I should put all the operations in my project in the same group in order to submit just once, right?
Vyas Ramasubramani
@vyasr
yes exactly
Andrei Berceanu
@berceanu
Now I've stumbled across a separate issue.
And the problem is, it exports visible devices in order, starting with 0.
Sometimes a node has many GPUs and some are used by other people, and this will try to run on GPU 0 even if it is in use, and for example GPU 7 is free.
Carl Simon Adorf
@csadorf
@berceanu I'm not quite sure how one would solve that problem. Have you talked to your cluster administrator on how to properly identify the GPU devices that are reserved for you?
Andrei Berceanu
@berceanu
Well there are no reserved ones, I just want to make use of the ones that are available at some given time.
Carl Simon Adorf
@csadorf
Maybe I don't understand the problem, but if you submit a cluster job, some resources are reserved for you, are they not?
Andrei Berceanu
@berceanu
Well what I mean is, some resources are free.
ie. some GPUs in this case. Not necessarily reserved for me, but free for anyone to use.
Andrei Berceanu
@berceanu
Is signac_project_document.json automatically produced by signac?
Andrei Berceanu
@berceanu
ie. is there a setting that controls it?
Brandon Butler
@b-butler
yes and it stores information regarding the project that can be accessed using project.doc[key].
Andrei Berceanu
@berceanu
What's the difference between --parallel and --test?
Sorry between --pretend and --test
Carl Simon Adorf
@csadorf
The --pretend option uses the actual environment and scheduler and will also query the scheduler to figure out what to submit, whereas --test does not do that.
Andrei Berceanu
@berceanu
I see, thanks @csadorf :)
Andrei Berceanu
@berceanu
Say I have a signac project, with 10 jobs folders inside my workspace. Each job has a out.data file. I want to copy all of these to a separate folder, appending the job hash, ie out_4fc56.dat etc.
Can I do this in a script using only signac?
Carl Simon Adorf
@csadorf
@berceanu I think this would be a minimal script to do this:
from pathlib import Path
import signac
project = signac.get_project()

for job in project: 
    src = Path(job.fn('out.data'))
    dst = Path(f'outdir/out_{job.id:.5}.dat')
    dst.write_bytes(src.read_bytes())
Andrei Berceanu
@berceanu
Awesome, thank you @csadorf
Javier Barbero
@javierbg
Hello! I'm integrating signac with my experimentation workflow and I had a small question: I have stored my dataset globally for the entire project as per the documentation (project.data), but I'm not really sure about what is the proper way to access this data from a job: the job._project attribute is protected, so I assume I cannot rely on that API, and loading the entire project from the CWD or the absolute path is cumbersome and doesn't play well with the with job: context manager. Any suggestions? Could job._project be made stable as job.project?
Carl Simon Adorf
@csadorf
@javierbg Hi! In principle you can always have a reference to "project" in your module-wide namespace and just access that. The reason that we have not yet publicly exposed job._project is because we worry that there might be some ambiguity as to whether that inverse relationship is well-defined. I think it would be super helpful if you could create an issue here: https://github.com/glotzerlab/signac/issues/new/choose with that feature request.
Vyas Ramasubramani
@vyasr
yes, please make an issue for this! i think exposing job.project is probably a reasonable plan for 2.0 based on the direction of some of our other conversations about how we would like to modify signac's data model, but i think we will need to wait until a 2.0 release since those changes will be breaking to some extent
Javier Barbero
@javierbg
@csadorf Ah, of course, that would work, it didn't occur to me. I'll create the issue!
Ryan S. DeFever
@rsdefever
Is there a way to do a @flow.cmd inside a container? When I use the submit --pretend it seems that the @flow.cmd is overriding the @flow.directives(executable="") that I am using to tell it to run on a container. Any ideas?
Andrei Berceanu
@berceanu
Can an operation return a function?
Andrei Berceanu
@berceanu
@Project.operation
def foo(job):
    def bar(a):
        return a * job.sp.b
    return bar

@Project.operation
def baz(job):
    c = bar(2)  # somehow
Brandon Butler
@b-butler
@rsdefever @cmdcurrently expects that the full needed command string will be given when run including any container information.
@berceanu while you could write an operation that returned a function, you would not be able to use that function output through flow. What is your use case? likely there is an alternative approach that would work with flow nicely.
Andrei Berceanu
@berceanu
Hi @b-butler , thanks for the quick reply. My use-case is that inside baz I need to run a (Python) simulation code, which expects the function bar as one of its inputs.
dens_func is my bar function :)
baz is run_fbpic
And I would like to move the definition of dens_func outside of run_fbpic, but the problem is that the definition of dens_func contains job statepoint parameters inside.
The external code is ran here:
Andrei Berceanu
@berceanu
Does this make sense? The reason I want to move dens_func outside is because I want to be able to call it from other operations as well, ie to plot the density profile.
Right now I have to save the profile to file and re-load it inside another operation.
Brandon Butler
@b-butler
Is there a reason the signature of dens_func could not be z, r, job?
If you have a reason for not doing so, you could create a function make_dens_func that took a job and created dens_func for that job. Then you could use make_dens_func to create the closure (function that captures local variables) in your operations.
Andrei Berceanu
@berceanu
Yes, the code expects the signature of dens_func to be (z,r)
Not sure I follow.
Brandon Butler
@b-butler
Are you not allowed to modify the function? If you can modify it freely, then you could choose to add an argument job to dens_func. If for some reason you are not allowed to modify that , then you could follow the make_dens_func approach I mentioned.
def dens_func(z, r, job) you could grep to see everywhere you call the function and just modify the call to add the job. Otherwise
def make_dens_func(job):
    def dens_func(z, r):
        # code here
    return dens_func
Here make_dens_func does not need to be an operation. It is merely a helper function.