Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    aconkli4
    @aconkli4
    good enough. mprof definitely gives more sensible results
    it seems (if i did the math right, which is a big if) that memory is the big problem, not time to run. I am going to try to run the clustering on a subsample on hpc if i can figure out these container things
    Julian Pistorius
    @julianpistorius
    Cool. Containers are cool. Good luck!
    aconkli4
    @aconkli4
    thanks!
    joseph-long @joseph-long waves
    Julian Pistorius
    @julianpistorius
    Hi @joseph-long ! :)
    Apparently Joseph first heard about ResBaz from @agladstein - the Arizona Research grapevine is alive & well.
    Joseph D. Long
    @joseph-long
    Reasons to do software instead of hardware: https://www.youtube.com/watch?v=sI5Ftm1-jik
    aconkli4
    @aconkli4
    does anyone know how i am screwing up using singularity to pull and run a docker image?
    If I run:
    sudo singularity build qtl-clustering.simg docker://aconklin/qtl-clustering
    sudo singularity run qtl-clustering.simg 100
    I get:
    python: can't open file './hdbscan_script_for_hpc.py': [Errno 2] No such file or directory
    Julian Pistorius
    @julianpistorius
    Hmm... Is this on your laptop I assume?
    Oh, and you get the error on the second sudo command?
    @aconkli4 - I think there might be an issue with the path in that container. I assume running this as a Docker container works?
    aconkli4
    @aconkli4
    yes this is on my laptop and i got the same error on hpc (without the sudo)
    it does work as a docker container
    Julian Pistorius
    @julianpistorius

    What happens if you run this?

    ```
    singularity run qtl-clustering.simg ls -al
    singularity run qtl-clustering.simg bash -c 'echo $PATH'

    aconkli4
    @aconkli4
    same error. it doesn't seem to be able to get past its inability to find the python script
    so I have ENTRYPOINT [ "python", "./hdbscan_script_for_hpc.py"] in my Dockerfile. should i change that to "hdbscan_script_for_hpc.py" only and rebuild?
    or ENTRYPOINT [ "python", "hdbscan_script_for_hpc.py"] rather?
    Julian Pistorius
    @julianpistorius
    Oh you know what.
    aconkli4
    @aconkli4
    what's that?
    Julian Pistorius
    @julianpistorius
    I bet it's because Singularity overwrites the home directory.
    singularity run qtl-clustering.simg pwd
    That's one of the gotchas when going from Docker to Singularity.
    aconkli4
    @aconkli4
    ok. do i need to rebuild my docker image?
    Julian Pistorius
    @julianpistorius

    Yeah. Probably. It's best to not use the home directory for storing stuff in containers. Use a path like /my-software/whatever.py or /opt/my-software/whatever.py.

    Background: The home directory inside the container will be replaced with the home directory of the user on the host (your laptop or HPC) that runs the container.

    aconkli4
    @aconkli4
    oh! that's what you meant. I'll give it a shot
    thanks!
    Julian Pistorius
    @julianpistorius
    Sure thing! Let us know what happens.
    aconkli4
    @aconkli4
    works on my laptop!
    Julian Pistorius
    @julianpistorius
    Heh... Great.
    aconkli4
    @aconkli4
    thanks for the advice.
    Julian Pistorius
    @julianpistorius
    My pleasure! Happy to help.
    Elaheh Alizadeh
    @Elaheh-Alizadeh
    Hello, I am having similar issue. When I want to go from Docker to Singularity , my script does not run!
    I am not sure if I don't bind my data properly or what !
    In Docker
    docker run -it -v $PWD/Input:/Input -v $PWD/Output:/Output elizadeh/batch:4.0 -p /Output/Batch_data2019_05_29.h5
    runs with no error.
    But with Singularity
    singularity run -B $PWD/Input:/Input -B $PWD/Output:/Output cellprofiler_3.0.0.sif -p /Output/Batch_data2019_05_29.h5
    does not run. I only get a warning : "seccomp requested but not enabled, seccomp library is missing or too old"
    Julian Pistorius
    @julianpistorius
    Hi @Elaheh-Alizadeh I think it might be the -p parameter. You don't need it. It's used in Docker only for binding ports. Try running singularity without -p.
    Elaheh Alizadeh
    @Elaheh-Alizadeh
    Thanks julian I tried it. Now the error is:
    s6-mkdir: warning: unable to mkdir /var/run/s6: Read-only file system.
    Julian Pistorius
    @julianpistorius
    Hmm... are you using Singularity on HPC or on Atmosphere?
    Elaheh Alizadeh
    @Elaheh-Alizadeh
    on HPC
    Elaheh Alizadeh
    @Elaheh-Alizadeh
    I get the same warning on Atmosphere
    Julian Pistorius
    @julianpistorius
    Actually, I think that -p might be necessary for the cellprofiler command.
    I misread where the -p was. I thought it was before the Singularity image, but it's after. Can you please paste the whole singularity command here?
    And maybe send the image to me?