bdice on consistent-scheduler-detection
Update changelog. (compare)
bdice on consistent-scheduler-detection
Make LSF / SLURM scheduler dete… (compare)
klywang on execution-hooks
Remove FlowProject.add_operatio… Enforce Operation Requirements … Merge branch 'master' into exec… (compare)
bdice on master
Enforce Operation Requirements … (compare)
bdice on operation-requirements
signac
?
project.data
), but I'm not really sure about what is the proper way to access this data from a job: the job._project
attribute is protected, so I assume I cannot rely on that API, and loading the entire project from the CWD or the absolute path is cumbersome and doesn't play well with the with job:
context manager. Any suggestions? Could job._project
be made stable as job.project
?
job._project
is because we worry that there might be some ambiguity as to whether that inverse relationship is well-defined. I think it would be super helpful if you could create an issue here: https://github.com/glotzerlab/signac/issues/new/choose with that feature request.
job.project
is probably a reasonable plan for 2.0 based on the direction of some of our other conversations about how we would like to modify signac's data model, but i think we will need to wait until a 2.0 release since those changes will be breaking to some extent
flow
. What is your use case? likely there is an alternative approach that would work with flow
nicely.
dens_func
is my bar
function :)
baz
is run_fbpic
dens_func
outside of run_fbpic
, but the problem is that the definition of dens_func
contains job statepoint parameters inside.
make_dens_func
that took a job
and created dens_func
for that job. Then you could use make_dens_func
to create the closure (function that captures local variables) in your operations.
def dens_func(z, r, job)
you could grep to see everywhere you call the function and just modify the call to add the job. Otherwisedef make_dens_func(job):
def dens_func(z, r):
# code here
return dens_func
make_dens_func
does not need to be an operation. It is merely a helper function.
proj = signac.get_project(search=False)
@Project.post...
@ex
@Project.operation
@Project.pre.after(save_final_histogram)
@Projet.post(job.document.get("integrated_charge", False))
def annotate_integrated_charge(job: Job) -> None:
Q = integrated_charge(job.fn("final_histogram.npz"), from_energy=100, to_energy=200)
job.doc["integrated_charge"] = float("{:.1f}".format(Q)) # pC
@Projet.post(lambda job: bool(job.doc.get("integrated_charge", False)))
@berceanu Hi, sorry it's been a few days! I have Gitter set to notify me when messages are sent to this channel, but I never get the notifications... :( You can also join our Slack workspace for faster responses. Also see our calendar for developer meetings / office hours that you can join anytime: https://docs.signac.io/en/latest/community.html Our developers are quite active on Slack.
Here are responses to your questions.
Getting a random job: if you just want some job (not random, but an arbitrary job) then you can use the Python iterator syntax next(iter(project))
. That will return the first job by iteration order (which isn't guaranteed to follow any convention, but I believe it's alphabetized by job id). If you truly want a random job, you can use import random; random.choice(list(project))
.
Testing job document keys as post-conditions: You might want @Project.post.true("integrated_charge")
. Documentation here: https://docs.signac.io/projects/flow/en/latest/api.html#flow.FlowProject.post.true
Hey there, I had a question about how multiple jobs are assigned to multiple processes. I have some 100 jobs I want to execute on 10 processes on my local computer, however I want them all to be run independently of each other. I guess what I mean by that is not sharing memory with each other. I’m working with a program that when it loads in parameters, they are immutably set. I’m finding that when a single process finishes a job and starts another, that process still has memory of the previous job that ran on that process. This is causing all my jobs to only have the parameters of the original 10 jobs that start. Is there a way around this using the python project.py run
command?
Would submitting these jobs using a local install of SLURM get around this?
Thanks in advanced!
@tlfobe sorry for the delayed response! I'd like to mention that we are more active on our Slack channel, so it'd be really great if you join that! Thanks a lot.
I have a question, are you using the np
flag while initiating the run
process i.e. python project.py run -np 10
? (This flag would allow you to parallelize to the specified number of processors. You can have a look at it's documentation here: https://docs.signac.io/projects/flow/en/latest/api.html#flow.FlowProject.run)
@tlfobe sorry for the delayed response! I'd like to mention that we are more active on our Slack channel, so it'd be really great if you join that! Thanks a lot.
I have a question, are you using the
np
flag while initiating therun
process i.e.python project.py run -np 10
? (This flag would allow you to parallelize to the specified number of processors. You can have a look at it's documentation here: https://docs.signac.io/projects/flow/en/latest/api.html#flow.FlowProject.run)
This query was already answered on Slack, I must have missed it. Apologies for this.