Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • May 14 20:39
    bdice synchronize #464
  • May 14 20:39

    bdice on introduce-aggregation

    Fix test so it works with or wi… (compare)

  • May 14 19:11
    bdice synchronize #464
  • May 14 19:11

    bdice on introduce-aggregation

    Use repr for error contents. Try splitting lines. Previous t… (compare)

  • May 14 18:58
    bdice synchronize #464
  • May 14 18:58

    bdice on introduce-aggregation

    Move tests for invalid job/aggr… (compare)

  • May 14 18:43
    bdice synchronize #464
  • May 14 18:43

    bdice on introduce-aggregation

    Add tests for invalid with_job … Fix spacing/comments. Add test for invalid job select… and 1 more (compare)

  • May 14 17:29
    bdice synchronize #464
  • May 14 17:29

    bdice on introduce-aggregation

    Improve docstrings. (compare)

  • May 14 17:13
    bdice closed #362
  • May 14 17:03
    bdice review_requested #464
  • May 14 17:03
    bdice review_requested #464
  • May 14 17:03
    bdice review_requested #464
  • May 14 16:59
    bdice synchronize #464
  • May 14 16:59

    bdice on introduce-aggregation

    Update test. (compare)

  • May 14 16:39
    bdice synchronize #464
  • May 14 16:39

    bdice on introduce-aggregation

    Add comment to clarify what hap… (compare)

  • May 14 16:30
    bdice synchronize #464
  • May 14 16:30

    bdice on introduce-aggregation

    Fix expected number of aggregat… (compare)

Andrei Berceanu
@berceanu
I see, thanks @csadorf :)
Andrei Berceanu
@berceanu
Say I have a signac project, with 10 jobs folders inside my workspace. Each job has a out.data file. I want to copy all of these to a separate folder, appending the job hash, ie out_4fc56.dat etc.
Can I do this in a script using only signac?
Carl Simon Adorf
@csadorf
@berceanu I think this would be a minimal script to do this:
from pathlib import Path
import signac
project = signac.get_project()

for job in project: 
    src = Path(job.fn('out.data'))
    dst = Path(f'outdir/out_{job.id:.5}.dat')
    dst.write_bytes(src.read_bytes())
Andrei Berceanu
@berceanu
Awesome, thank you @csadorf
Javier Barbero
@javierbg
Hello! I'm integrating signac with my experimentation workflow and I had a small question: I have stored my dataset globally for the entire project as per the documentation (project.data), but I'm not really sure about what is the proper way to access this data from a job: the job._project attribute is protected, so I assume I cannot rely on that API, and loading the entire project from the CWD or the absolute path is cumbersome and doesn't play well with the with job: context manager. Any suggestions? Could job._project be made stable as job.project?
Carl Simon Adorf
@csadorf
@javierbg Hi! In principle you can always have a reference to "project" in your module-wide namespace and just access that. The reason that we have not yet publicly exposed job._project is because we worry that there might be some ambiguity as to whether that inverse relationship is well-defined. I think it would be super helpful if you could create an issue here: https://github.com/glotzerlab/signac/issues/new/choose with that feature request.
Vyas Ramasubramani
@vyasr
yes, please make an issue for this! i think exposing job.project is probably a reasonable plan for 2.0 based on the direction of some of our other conversations about how we would like to modify signac's data model, but i think we will need to wait until a 2.0 release since those changes will be breaking to some extent
Javier Barbero
@javierbg
@csadorf Ah, of course, that would work, it didn't occur to me. I'll create the issue!
Ryan S. DeFever
@rsdefever
Is there a way to do a @flow.cmd inside a container? When I use the submit --pretend it seems that the @flow.cmd is overriding the @flow.directives(executable="") that I am using to tell it to run on a container. Any ideas?
Andrei Berceanu
@berceanu
Can an operation return a function?
Andrei Berceanu
@berceanu
@Project.operation
def foo(job):
    def bar(a):
        return a * job.sp.b
    return bar

@Project.operation
def baz(job):
    c = bar(2)  # somehow
Brandon Butler
@b-butler
@rsdefever @cmdcurrently expects that the full needed command string will be given when run including any container information.
@berceanu while you could write an operation that returned a function, you would not be able to use that function output through flow. What is your use case? likely there is an alternative approach that would work with flow nicely.
Andrei Berceanu
@berceanu
Hi @b-butler , thanks for the quick reply. My use-case is that inside baz I need to run a (Python) simulation code, which expects the function bar as one of its inputs.
dens_func is my bar function :)
baz is run_fbpic
And I would like to move the definition of dens_func outside of run_fbpic, but the problem is that the definition of dens_func contains job statepoint parameters inside.
The external code is ran here:
Andrei Berceanu
@berceanu
Does this make sense? The reason I want to move dens_func outside is because I want to be able to call it from other operations as well, ie to plot the density profile.
Right now I have to save the profile to file and re-load it inside another operation.
Brandon Butler
@b-butler
Is there a reason the signature of dens_func could not be z, r, job?
If you have a reason for not doing so, you could create a function make_dens_func that took a job and created dens_func for that job. Then you could use make_dens_func to create the closure (function that captures local variables) in your operations.
Andrei Berceanu
@berceanu
Yes, the code expects the signature of dens_func to be (z,r)
Not sure I follow.
Brandon Butler
@b-butler
Are you not allowed to modify the function? If you can modify it freely, then you could choose to add an argument job to dens_func. If for some reason you are not allowed to modify that , then you could follow the make_dens_func approach I mentioned.
def dens_func(z, r, job) you could grep to see everywhere you call the function and just modify the call to add the job. Otherwise
def make_dens_func(job):
    def dens_func(z, r):
        # code here
    return dens_func
Here make_dens_func does not need to be an operation. It is merely a helper function.
Andrei Berceanu
@berceanu
Oh I see.
I somehow missed that :)
Thanks!
Brandon Butler
@b-butler
No problem :smile:
Andrei Berceanu
@berceanu
how can i open a random project job?
I have the project handler, proj = signac.get_project(search=False)
Andrei Berceanu
@berceanu
If I want to use proj.open_job() I need to provide an id or statepoint.
Andrei Berceanu
@berceanu
ids = [job.id for job in proj]
and a random choice on it?
Andrei Berceanu
@berceanu
    random.seed(42) 

    proj = signac.get_project(search=False)
    ids = [job.id for job in proj]

    job = proj.open_job(id=random.choice(ids))
Andrei Berceanu
@berceanu
How can I check that a job.doc key was created after an operation?
@Project.post...
Andrei Berceanu
@berceanu
@ex
@Project.operation
@Project.pre.after(save_final_histogram)
@Projet.post(job.document.get("integrated_charge", False))
def annotate_integrated_charge(job: Job) -> None:
    Q = integrated_charge(job.fn("final_histogram.npz"), from_energy=100, to_energy=200)    
    job.doc["integrated_charge"] = float("{:.1f}".format(Q))  # pC
@Projet.post(lambda job: bool(job.doc.get("integrated_charge", False)))
Andrei Berceanu
@berceanu
?
Bradley Dice
@bdice

@berceanu Hi, sorry it's been a few days! I have Gitter set to notify me when messages are sent to this channel, but I never get the notifications... :( You can also join our Slack workspace for faster responses. Also see our calendar for developer meetings / office hours that you can join anytime: https://docs.signac.io/en/latest/community.html Our developers are quite active on Slack.

Here are responses to your questions.

Getting a random job: if you just want some job (not random, but an arbitrary job) then you can use the Python iterator syntax next(iter(project)). That will return the first job by iteration order (which isn't guaranteed to follow any convention, but I believe it's alphabetized by job id). If you truly want a random job, you can use import random; random.choice(list(project)).

Testing job document keys as post-conditions: You might want @Project.post.true("integrated_charge"). Documentation here: https://docs.signac.io/projects/flow/en/latest/api.html#flow.FlowProject.post.true

Andrei Berceanu
@berceanu
Thanks!
Theodore Fobe
@tlfobe

Hey there, I had a question about how multiple jobs are assigned to multiple processes. I have some 100 jobs I want to execute on 10 processes on my local computer, however I want them all to be run independently of each other. I guess what I mean by that is not sharing memory with each other. I’m working with a program that when it loads in parameters, they are immutably set. I’m finding that when a single process finishes a job and starts another, that process still has memory of the previous job that ran on that process. This is causing all my jobs to only have the parameters of the original 10 jobs that start. Is there a way around this using the python project.py run command?

Would submitting these jobs using a local install of SLURM get around this?

Thanks in advanced!

Hardik Ojha
@kidrahahjo

@tlfobe sorry for the delayed response! I'd like to mention that we are more active on our Slack channel, so it'd be really great if you join that! Thanks a lot.

I have a question, are you using the np flag while initiating the run process i.e. python project.py run -np 10? (This flag would allow you to parallelize to the specified number of processors. You can have a look at it's documentation here: https://docs.signac.io/projects/flow/en/latest/api.html#flow.FlowProject.run)