signac/Lobby

https://signac.io

Ali Malek

@phmalek

@vyasr Thank you! Was really helpful. What I meant from FP1 and FP2 is this inheritance structure: FP1->flow.FlowProject and FP2->flow.FlowProject.

vyasr

@vyasr

is your intention to separate your three operations (func1, func2, and func3) so that they are attached to different FlowProjects? if so, then the structure you’re suggesting is correct. you would do something like

class FP1(FlowProject):
    pass
class FP2(FlowProject):
    pass
class FP3(FlowProject):
    pass

@FP1.operation
def func1(job):
    pass
@FP2.operation
def func2(job):
    pass
@FP3.operation
def func3(job):
    pass

then you would have exactly one operation in each of proj1, proj2, and proj3 if you instantiated them as

proj1 = FP1.get_project()
proj2 = FP2.get_project()
proj3 = FP3.get_project()

Ali Malek

@phmalek

Excellent! This is exactly what I want. Thanks!

I was always wondering what was the point in flow package! Now I see that it is quite useful :-)

vyasr

@vyasr

:thumbsup:

Ali Malek

@phmalek

Hi guys,
can someone explain me a very simple use case of FlowGroup? The documentation says that it builds a subset of a workflow for a project. Does it mean that it takes a subset of operations in FlowProject? Thanks

Ali Malek

@phmalek

I would also really appreciate if someone could write a very simple example like what @vyasr wrote above for post/pre.copy_from(*other_functions).

Ali Malek

@phmalek

I have tried this:

Carl Simon Adorf

@csadorf

Hi @phmalek The idea of the flow groups is to simplify the implementation the control of execution of workflows. For example, you might have a bunch of operations that you would like to run on your workstations, and others that you want to submit to a cluster. So you could group those operations and then simply run/submit them with python project run workstation-ops and python project.py submit cluster-ops or so. Especially in the latter case this will make it easier to submit these operations together without the need to resubmit individual operations.
For example, you may have an operation that prepares your simulation runs (on a single processor) and then a "simulation" operation that requires more intense resources. Without groups it is not possible to submit these two operations with different resource requirements. With groups they can be submitted together and then executed in sequence. At least in theory, we still need to work out some details with respect to the groups.

We plan on going through the signac-flow implementation in some kind of webinar/call very soon. To help developers to understand all of these concepts and answer questions. We will likely announce that on our slack channel, have you joined it already?

Ali Malek

@phmalek

Thanks Carl, I joined just now :-)

Ali Malek

@phmalek

At the moment I am still struggling with another issue. This is my code:

class FP(FlowProject):
    pass

@FP.operation
@FP.post.true("is_done")
def func1(job):
    doSomething
    job.doc["is_done"] = True

@FP.operation
@FP.pre.after(func1)
def func2(job):
    doSomethingElse

When I write:

proj = FP()
job = FP.open_job({...}).init()

proj.get_job_status(job)

In the last line of the above code I observe that both func1 and func2 are eligible! Is it a bug or I made a mistake somewhere?

Thanks for your help

Ali Malek

@phmalek

I just resent it in slack

Bradley Dice

@bdice

This issue was resolved via Slack. For anyone interested, join the signac Slack group here: https://join.slack.com/t/signac/shared_invite/enQtNzk2MTUxNjU5ODkzLWM1NDFmMzRmMTA2MjFlN2ZiOTQ4MDBjNmIwMmM4YTgyZTQ1ODFkMGNhZTc5M2IwMmE1MWJiOTliN2Y2Y2M3ZDY

ChanForPres

@ChanForPres

tutorial references from flow import FlowProject assume this is https://github.com/flow-project/flow?

does signac use this for computation?

Carl Simon Adorf

@csadorf

@ChanForPres No, flow here is the package provided by https://github.com/glotzerlab/signac-flow
You can install with pip (pip install signac-flow) or conda (conda install signac-flow).

ChanForPres

@ChanForPres

ah i see

Carl Simon Adorf

@csadorf

@ChanForPres Let us know if you have any other questions! :)

Ryan S. DeFever

@rsdefever

Hi all. I'm running into an issue with python not being able to find an executable when its running inside of signac flow. Let's say I have a conda environment called conda-env and an executable called exec.exe. Signac and flow are installed in conda-env. The executable is located at
~/anaconda3/envs/conda-env/bin/exec.exe. My python code uses shutil.which('exec.exe') to find the executable. When submitting a job with signac flow I get an error that my python code was unable to find the executable. Any ideas what could be going on here? Also, when I use python project.py run ... rather than python project.py submit ... everything works fine.

vyasr

@vyasr

@rsdefever is it possible that the conda environment is not being loaded in the submission environment?

if you run a pretend submission, can you see what python executable is being used?

Ryan S. DeFever

@rsdefever

It is the correct python executable

vyasr

@vyasr

ok, how do you activate your conda environment? is it part of your shell configuration, or do you load it manually?

i.e. is it possible that the python executable is correct in the script, but the conda shims to e.g. your PATH are not getting set in the actual submission environment?

Mike Henry

@mikemhenry

as a sanity check can you see if print(os.environ["PATH"]) prints what you expect?

Ryan S. DeFever

@rsdefever

Right now, nothing additional to activate the conda env when I do a signac submission. Do I need to be conda-activating inside my submission template?

Mike Henry

@mikemhenry

You might, it depends on the the cluster is setup, sometimes environmental variables get unset/reset when something is submitted

vyasr

@vyasr

yes i’d assume so. conda does all sorts of hidden magic when you activate a conda environment, such as adding things to environment variables like your PATH

Ryan S. DeFever

@rsdefever

Yeah that seems to work

vyasr

@vyasr

not all of these are immediately visible unfortunately

Mike Henry

@mikemhenry

The fact it works with run but not submit makes it seem like something is going on with the shell/env that gets wiped on submission

vyasr

@vyasr

yeah exactly

Ryan S. DeFever

@rsdefever

I knew that conda envs did some wonky env stuff behind the scenes, but thought maybe signac was being fancy and I didn't need to do that

vyasr

@vyasr

unfortunately this is a case where something like #PBS -V or the SLURM equivalent may not be completely trustworthy on their own

unfortunately not haha

Mike Henry

@mikemhenry

BUT using conda in HPC workflows is pretty common, maybe we should either make a note in the docs or add some sort of @with_conda_env("my_env") decorator that could help

vyasr

@vyasr

if it was as simple as a different python executable, we can handle that (see the executable directive for signac-flow), but conda also does all sorts of other things that aren’t even just python related since it also supports other languages. for instance, it affects what compilers and linkers are used when building software

Ryan S. DeFever

@rsdefever

Ok, cool. Well thats an easy fix. Thanks for the help. I had actually restrained activity conda env from within the job submission template because I was concerned it would break something.

vyasr

@vyasr

yes, i’m definitely open to something like that. also a corresponding @with_virtual_environment(“env”) since virtual environments is also a common use case

Mike Henry

@mikemhenry

I could see it being a useful directive since you might have different conda envs for different operations and it is clunky to just handle that with a submission template

Ryan S. DeFever

@rsdefever

@mikemhenry very true. Fortunately in this case everything is in a single env

vyasr

@vyasr

that’s fair. tbh i’m pretty hesitant to use multiple conda environments in HPC environments. conda is great, but there are too many subtle ways in which it can break things. i find venv to be a more robust solution on supercomputers, but having a single conda environment which i autoload in my bash profile is something that typically works fine

(i realize this may not be a viable alternative for mbuild/foyer/etc users)

Mike Henry

@mikemhenry

I'm not a huge conda fan either, HPC or not... singularity is my go-to now for managing software

I just wish there was a no-root-needed package manager that didn't suck

vyasr

@vyasr

pip to the rescue! oh wait...

it’s a hard problem. conda does a pretty good job IMO, and it’s a great goal to be able to have all of this unified. but there are just so many ways it can go wrong as a result

Where communities thrive

People

Repo info

Activity