signac/Lobby

https://signac.io

Andrei Berceanu

@berceanu

Say I have a signac project, with 10 jobs folders inside my workspace. Each job has a out.data file. I want to copy all of these to a separate folder, appending the job hash, ie out_4fc56.dat etc.

Can I do this in a script using only signac?

Carl Simon Adorf

@csadorf

@berceanu I think this would be a minimal script to do this:

from pathlib import Path
import signac
project = signac.get_project()

for job in project: 
    src = Path(job.fn('out.data'))
    dst = Path(f'outdir/out_{job.id:.5}.dat')
    dst.write_bytes(src.read_bytes())

Andrei Berceanu

@berceanu

Awesome, thank you @csadorf

Javier Barbero

@javierbg

Hello! I'm integrating signac with my experimentation workflow and I had a small question: I have stored my dataset globally for the entire project as per the documentation (project.data), but I'm not really sure about what is the proper way to access this data from a job: the job._project attribute is protected, so I assume I cannot rely on that API, and loading the entire project from the CWD or the absolute path is cumbersome and doesn't play well with the with job: context manager. Any suggestions? Could job._project be made stable as job.project?

Carl Simon Adorf

@csadorf

@javierbg Hi! In principle you can always have a reference to "project" in your module-wide namespace and just access that. The reason that we have not yet publicly exposed job._project is because we worry that there might be some ambiguity as to whether that inverse relationship is well-defined. I think it would be super helpful if you could create an issue here: https://github.com/glotzerlab/signac/issues/new/choose with that feature request.

Vyas Ramasubramani

@vyasr

yes, please make an issue for this! i think exposing job.project is probably a reasonable plan for 2.0 based on the direction of some of our other conversations about how we would like to modify signac's data model, but i think we will need to wait until a 2.0 release since those changes will be breaking to some extent

Javier Barbero

@javierbg

@csadorf Ah, of course, that would work, it didn't occur to me. I'll create the issue!

Ryan S. DeFever

@rsdefever

Is there a way to do a @flow.cmd inside a container? When I use the submit --pretend it seems that the @flow.cmd is overriding the @flow.directives(executable="") that I am using to tell it to run on a container. Any ideas?

Andrei Berceanu

@berceanu

Can an operation return a function?

Andrei Berceanu

@berceanu

@Project.operation
def foo(job):
    def bar(a):
        return a * job.sp.b
    return bar

@Project.operation
def baz(job):
    c = bar(2)  # somehow

Brandon Butler

@b-butler

@rsdefever @cmdcurrently expects that the full needed command string will be given when run including any container information.

@berceanu while you could write an operation that returned a function, you would not be able to use that function output through flow. What is your use case? likely there is an alternative approach that would work with flow nicely.

Andrei Berceanu

@berceanu

Hi @b-butler , thanks for the quick reply. My use-case is that inside baz I need to run a (Python) simulation code, which expects the function bar as one of its inputs.

Full details here: https://github.com/berceanu/signac-driven-fbpic/blob/10caaaeadefaaf7ccd2c79372d8d21338882db19/signac/src/project.py#L173

dens_func is my bar function :)

baz is run_fbpic

And I would like to move the definition of dens_func outside of run_fbpic, but the problem is that the definition of dens_func contains job statepoint parameters inside.

The external code is ran here:

https://github.com/berceanu/signac-driven-fbpic/blob/10caaaeadefaaf7ccd2c79372d8d21338882db19/signac/src/project.py#L309

Andrei Berceanu

@berceanu

Does this make sense? The reason I want to move dens_func outside is because I want to be able to call it from other operations as well, ie to plot the density profile.

Right now I have to save the profile to file and re-load it inside another operation.

https://github.com/berceanu/signac-driven-fbpic/blob/10caaaeadefaaf7ccd2c79372d8d21338882db19/signac/src/project.py#L331

Brandon Butler

@b-butler

Is there a reason the signature of dens_func could not be z, r, job?

If you have a reason for not doing so, you could create a function make_dens_func that took a job and created dens_func for that job. Then you could use make_dens_func to create the closure (function that captures local variables) in your operations.

Andrei Berceanu

@berceanu

Yes, the code expects the signature of dens_func to be (z,r)

Not sure I follow.

Brandon Butler

@b-butler

Are you not allowed to modify the function? If you can modify it freely, then you could choose to add an argument job to dens_func. If for some reason you are not allowed to modify that , then you could follow the make_dens_func approach I mentioned.

def dens_func(z, r, job) you could grep to see everywhere you call the function and just modify the call to add the job. Otherwise

def make_dens_func(job):
    def dens_func(z, r):
        # code here
    return dens_func

Here make_dens_func does not need to be an operation. It is merely a helper function.

Andrei Berceanu

@berceanu

Oh I see.

I somehow missed that :)

Thanks!

Brandon Butler

@b-butler

No problem :smile:

Andrei Berceanu

@berceanu

how can i open a random project job?

I have the project handler, proj = signac.get_project(search=False)

Andrei Berceanu

@berceanu

If I want to use proj.open_job() I need to provide an id or statepoint.

Andrei Berceanu

@berceanu

ids = [job.id for job in proj]

and a random choice on it?

Andrei Berceanu

@berceanu

    random.seed(42) 

    proj = signac.get_project(search=False)
    ids = [job.id for job in proj]

    job = proj.open_job(id=random.choice(ids))

Andrei Berceanu

@berceanu

How can I check that a job.doc key was created after an operation?

@Project.post...

Andrei Berceanu

@berceanu

@ex
@Project.operation
@Project.pre.after(save_final_histogram)
@Projet.post(job.document.get("integrated_charge", False))
def annotate_integrated_charge(job: Job) -> None:
    Q = integrated_charge(job.fn("final_histogram.npz"), from_energy=100, to_energy=200)    
    job.doc["integrated_charge"] = float("{:.1f}".format(Q))  # pC

@Projet.post(lambda job: bool(job.doc.get("integrated_charge", False)))

Andrei Berceanu

@berceanu

Bradley Dice

@bdice

@berceanu Hi, sorry it's been a few days! I have Gitter set to notify me when messages are sent to this channel, but I never get the notifications... :( You can also join our Slack workspace for faster responses. Also see our calendar for developer meetings / office hours that you can join anytime: https://docs.signac.io/en/latest/community.html Our developers are quite active on Slack.

Here are responses to your questions.

Getting a random job: if you just want some job (not random, but an arbitrary job) then you can use the Python iterator syntax next(iter(project)). That will return the first job by iteration order (which isn't guaranteed to follow any convention, but I believe it's alphabetized by job id). If you truly want a random job, you can use import random; random.choice(list(project)).

Testing job document keys as post-conditions: You might want @Project.post.true("integrated_charge"). Documentation here: https://docs.signac.io/projects/flow/en/latest/api.html#flow.FlowProject.post.true

Andrei Berceanu

@berceanu

Thanks!

Theodore Fobe

@tlfobe

Hey there, I had a question about how multiple jobs are assigned to multiple processes. I have some 100 jobs I want to execute on 10 processes on my local computer, however I want them all to be run independently of each other. I guess what I mean by that is not sharing memory with each other. I’m working with a program that when it loads in parameters, they are immutably set. I’m finding that when a single process finishes a job and starts another, that process still has memory of the previous job that ran on that process. This is causing all my jobs to only have the parameters of the original 10 jobs that start. Is there a way around this using the python project.py run command?

Would submitting these jobs using a local install of SLURM get around this?

Thanks in advanced!

Hardik Ojha

@kidrahahjo

@tlfobe sorry for the delayed response! I'd like to mention that we are more active on our Slack channel, so it'd be really great if you join that! Thanks a lot.

I have a question, are you using the np flag while initiating the run process i.e. python project.py run -np 10? (This flag would allow you to parallelize to the specified number of processors. You can have a look at it's documentation here: https://docs.signac.io/projects/flow/en/latest/api.html#flow.FlowProject.run)

Hardik Ojha

@kidrahahjo

@tlfobe sorry for the delayed response! I'd like to mention that we are more active on our Slack channel, so it'd be really great if you join that! Thanks a lot.
I have a question, are you using the np flag while initiating the run process i.e. python project.py run -np 10? (This flag would allow you to parallelize to the specified number of processors. You can have a look at it's documentation here: https://docs.signac.io/projects/flow/en/latest/api.html#flow.FlowProject.run)

This query was already answered on Slack, I must have missed it. Apologies for this.

Where communities thrive

People

Repo info

Activity