by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • May 30 13:28
    kidrahahjo synchronize #289
  • May 30 12:07
    kidrahahjo synchronize #289
  • May 30 09:42
    kidrahahjo edited #289
  • May 30 09:41
    kidrahahjo edited #289
  • May 29 17:59
    kidrahahjo synchronize #289
  • May 29 09:45
    kidrahahjo edited #290
  • May 29 09:45
    kidrahahjo edited #290
  • May 29 09:43
    kidrahahjo synchronize #290
  • May 28 16:43
    kidrahahjo review_requested #290
  • May 28 16:43
    kidrahahjo review_requested #290
  • May 28 16:43
    kidrahahjo review_request_removed #290
  • May 28 16:43
    kidrahahjo review_requested #290
  • May 28 16:43
    kidrahahjo review_requested #290
  • May 28 16:43
    kidrahahjo review_request_removed #290
  • May 28 16:43
    kidrahahjo opened #290
  • May 28 16:36
    kidrahahjo synchronize #289
  • May 28 15:16

    vyasr on df-includes-excludes

    (compare)

  • May 28 15:16

    vyasr on master

    Add usecols and flatten to to_d… (compare)

  • May 28 15:16
    vyasr closed #330
  • May 28 15:16
    vyasr closed #327
Ali Malek
@phmalek
@vyasr Thank you! Was really helpful. What I meant from FP1 and FP2 is this inheritance structure: FP1->flow.FlowProject and FP2->flow.FlowProject.
vyasr
@vyasr
is your intention to separate your three operations (func1, func2, and func3) so that they are attached to different FlowProjects? if so, then the structure you’re suggesting is correct. you would do something like
class FP1(FlowProject):
    pass
class FP2(FlowProject):
    pass
class FP3(FlowProject):
    pass

@FP1.operation
def func1(job):
    pass
@FP2.operation
def func2(job):
    pass
@FP3.operation
def func3(job):
    pass
then you would have exactly one operation in each of proj1, proj2, and proj3 if you instantiated them as
proj1 = FP1.get_project()
proj2 = FP2.get_project()
proj3 = FP3.get_project()
Ali Malek
@phmalek
Excellent! This is exactly what I want. Thanks!
I was always wondering what was the point in flow package! Now I see that it is quite useful :-)
vyasr
@vyasr
:thumbsup:
Ali Malek
@phmalek
Hi guys,
can someone explain me a very simple use case of FlowGroup? The documentation says that it builds a subset of a workflow for a project. Does it mean that it takes a subset of operations in FlowProject? Thanks
Ali Malek
@phmalek
I would also really appreciate if someone could write a very simple example like what @vyasr wrote above for post/pre.copy_from(*other_functions).
Ali Malek
@phmalek
I have tried this:
Carl Simon Adorf
@csadorf
Hi @phmalek The idea of the flow groups is to simplify the implementation the control of execution of workflows. For example, you might have a bunch of operations that you would like to run on your workstations, and others that you want to submit to a cluster. So you could group those operations and then simply run/submit them with python project run workstation-ops and python project.py submit cluster-ops or so. Especially in the latter case this will make it easier to submit these operations together without the need to resubmit individual operations.
For example, you may have an operation that prepares your simulation runs (on a single processor) and then a "simulation" operation that requires more intense resources. Without groups it is not possible to submit these two operations with different resource requirements. With groups they can be submitted together and then executed in sequence. At least in theory, we still need to work out some details with respect to the groups.
We plan on going through the signac-flow implementation in some kind of webinar/call very soon. To help developers to understand all of these concepts and answer questions. We will likely announce that on our slack channel, have you joined it already?
Ali Malek
@phmalek
Thanks Carl, I joined just now :-)
Ali Malek
@phmalek

At the moment I am still struggling with another issue. This is my code:

class FP(FlowProject):
    pass

@FP.operation
@FP.post.true("is_done")
def func1(job):
    doSomething
    job.doc["is_done"] = True

@FP.operation
@FP.pre.after(func1)
def func2(job):
    doSomethingElse

When I write:

proj = FP()
job = FP.open_job({...}).init()

proj.get_job_status(job)

In the last line of the above code I observe that both func1 and func2 are eligible! Is it a bug or I made a mistake somewhere?

Thanks for your help

Ali Malek
@phmalek
I just resent it in slack
Bradley Dice
@bdice
ChanForPres
@ChanForPres
tutorial references from flow import FlowProject assume this is https://github.com/flow-project/flow?
does signac use this for computation?
Carl Simon Adorf
@csadorf
@ChanForPres No, flow here is the package provided by https://github.com/glotzerlab/signac-flow
You can install with pip (pip install signac-flow) or conda (conda install signac-flow).
ChanForPres
@ChanForPres
ah i see
Carl Simon Adorf
@csadorf
@ChanForPres Let us know if you have any other questions! :)
Ryan S. DeFever
@rsdefever
Hi all. I'm running into an issue with python not being able to find an executable when its running inside of signac flow. Let's say I have a conda environment called conda-env and an executable called exec.exe. Signac and flow are installed in conda-env. The executable is located at
~/anaconda3/envs/conda-env/bin/exec.exe. My python code uses shutil.which('exec.exe') to find the executable. When submitting a job with signac flow I get an error that my python code was unable to find the executable. Any ideas what could be going on here? Also, when I use python project.py run ... rather than python project.py submit ... everything works fine.
vyasr
@vyasr
@rsdefever is it possible that the conda environment is not being loaded in the submission environment?
if you run a pretend submission, can you see what python executable is being used?
Ryan S. DeFever
@rsdefever
It is the correct python executable
vyasr
@vyasr
ok, how do you activate your conda environment? is it part of your shell configuration, or do you load it manually?
i.e. is it possible that the python executable is correct in the script, but the conda shims to e.g. your PATH are not getting set in the actual submission environment?
Mike Henry
@mikemhenry
as a sanity check can you see if print(os.environ["PATH"]) prints what you expect?
Ryan S. DeFever
@rsdefever
Right now, nothing additional to activate the conda env when I do a signac submission. Do I need to be conda-activating inside my submission template?
Mike Henry
@mikemhenry
You might, it depends on the the cluster is setup, sometimes environmental variables get unset/reset when something is submitted
vyasr
@vyasr
yes i’d assume so. conda does all sorts of hidden magic when you activate a conda environment, such as adding things to environment variables like your PATH
Ryan S. DeFever
@rsdefever
Yeah that seems to work
vyasr
@vyasr
not all of these are immediately visible unfortunately
Mike Henry
@mikemhenry
The fact it works with run but not submit makes it seem like something is going on with the shell/env that gets wiped on submission
vyasr
@vyasr
yeah exactly
Ryan S. DeFever
@rsdefever
I knew that conda envs did some wonky env stuff behind the scenes, but thought maybe signac was being fancy and I didn't need to do that
vyasr
@vyasr
unfortunately this is a case where something like #PBS -V or the SLURM equivalent may not be completely trustworthy on their own
unfortunately not haha
Mike Henry
@mikemhenry
BUT using conda in HPC workflows is pretty common, maybe we should either make a note in the docs or add some sort of @with_conda_env("my_env") decorator that could help
vyasr
@vyasr
if it was as simple as a different python executable, we can handle that (see the executable directive for signac-flow), but conda also does all sorts of other things that aren’t even just python related since it also supports other languages. for instance, it affects what compilers and linkers are used when building software
Ryan S. DeFever
@rsdefever
Ok, cool. Well thats an easy fix. Thanks for the help. I had actually restrained activity conda env from within the job submission template because I was concerned it would break something.
vyasr
@vyasr
yes, i’m definitely open to something like that. also a corresponding @with_virtual_environment(“env”) since virtual environments is also a common use case
Mike Henry
@mikemhenry
I could see it being a useful directive since you might have different conda envs for different operations and it is clunky to just handle that with a submission template
Ryan S. DeFever
@rsdefever
@mikemhenry very true. Fortunately in this case everything is in a single env
vyasr
@vyasr
that’s fair. tbh i’m pretty hesitant to use multiple conda environments in HPC environments. conda is great, but there are too many subtle ways in which it can break things. i find venv to be a more robust solution on supercomputers, but having a single conda environment which i autoload in my bash profile is something that typically works fine
(i realize this may not be a viable alternative for mbuild/foyer/etc users)
Mike Henry
@mikemhenry
I'm not a huge conda fan either, HPC or not... singularity is my go-to now for managing software
I just wish there was a no-root-needed package manager that didn't suck
vyasr
@vyasr
pip to the rescue! oh wait...
it’s a hard problem. conda does a pretty good job IMO, and it’s a great goal to be able to have all of this unified. but there are just so many ways it can go wrong as a result