Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Steven R. Brandt
    @stevenrbrandt
    I haven't tested speeds, but I believe MPI should run faster this way as the intel_mpich is being used to launch the image
    Steven R. Brandt
    @stevenrbrandt
    It would be great if someone else would try this out, just to make sure it works for someone other than me.
    Roland Haas
    @rhaas80
    last tutorial day today. With GPU nodes this time (2:05PM PDT). Though right now the GPU serve are still down (it being 6am).
    Federico G Lopez Armengol
    @fedelopezar

    Hi @stevenrbrandt , I am testing the singularity section. This is what I tried in db, and failed:

    curl -kLO https://raw.githubusercontent.com/gridaphobe/CRL/ET_2021_05/GetComponents
    chmod a+x GetComponents
    ./GetComponents https://bitbucket.org/eschnett/cactusamrex/raw/master/azure-pipelines/carpetx.th
    cd Cactus/
    ./simfactory/bin//sim setup-silent
    ./simfactory/bin/sim build sim-gpu --machine db-sing-nv --thornlist thornlists/carpetx.th

    This is the error:

    Using configuration: sim-gpu
    Reconfiguring sim-gpu
    Writing configuration to: /home/flopezar/carpetx/singularityTest/Cactus/configs/sim-gpu/OptionList
    srun: error: Unable to allocate resources: Invalid account or account/partition combination specified
    (I am trying the "short way")
    Dustin Lang
    @dstndstn
    Hi @rhaas80 the GPU pool should now auto-scale (only up to 3 right now)... because nodes with GPUs act a little different on GCP, I had to run an additional command to enable that.
    Roland Haas
    @rhaas80
    thank you. I can verify that at least one server started up (mine). Would it be hard to change the pre-selected container to be the CarpetX one (which is all that is needed today)? If easy, could it be done?
    Roland Haas
    @rhaas80
    ok. got the KHI parfile to run but needed to remove openPMD_api from the ActiveThorns line.
    in principle that can be fixed in /nfs/home/sbrandt/Cactus.zip which is where the parfile is initialy located (and each user untars the file when they follow the tutorial).
    I could get the animation to show up in the notebook as well. I'll kill my server for now, so that others can try / use the resources.
    Dustin Lang
    @dstndstn
    Sure, I just set the Friday container to be the default.
    Roland Haas
    @rhaas80
    thank you.
    I am working with Steve to fix the parfile with the extra (and failing) openPMD_api activthorn.
    Roland Haas
    @rhaas80
    fixed now.
    Steven R. Brandt
    @stevenrbrandt
    @fedelopezar what do you see when you type balance?
    Jay Kalinani
    @jaykalinani
    Hi @rhaas80 , thanks for checking this! Within the latest notebook, the users should be able to write the parfile which does not use openPMD_api as activethorn. Just wanted to confirm if you are using the latest notebook. Kindly let me know.
    1 reply
    Federico G Lopez Armengol
    @fedelopezar
    @stevenrbrandt this is what I see:
    User filesystem quotas for flopezar (uid 16052): 
         Filesystem         MB used       quota       files      fquota
         /home                 2246       10000       22752           0
         /work                28687           0       47494     4000000
    
    CPU Allocation SUs:        remaining   allocated  expiration
        hpc_et_test2:           45531.50    50000.00  2023-04-01
    Steven R. Brandt
    @stevenrbrandt
    @fedelopezar, I assume you have the latest simfactory?
    Federico G Lopez Armengol
    @fedelopezar
    I guess so. I followed the commands above.
    I can confirm that db-sing-nv config files are here, for instance.
    Steven R. Brandt
    @stevenrbrandt
    @fedelopezar so the awkwardness is that we have to run in the queue to use Singularity, and simfactory does not substitute @ALLOCATION@ in the make command. So that means I had to count on you having a default allocation. If you only have one, it's supposed to be the default, but I guess that's not working. Could you login here https://accounts.hpc.lsu.edu/balances.php, set your default to hpc_et_test2, and then wait an hour or so? Thanks.
    1 reply
    Roland Haas
    @rhaas80
    I don't know which version I used. I used a notebook KelvinHelmholtzInstability.ipynb in my $HOME which referred to a parfile ./arrangements/CarpetX/KHInitial/par/KHI.par which it assumes to exist (ie does not create). I can try wiping my $HOME to see if this pullls in a newer version.
    ok. wiping my $HOME brings in a notebook that writes its own parfile, presumably the newest version. I'll give it a try.
    1 reply
    Roland Haas
    @rhaas80
    either one should now work actually. The new one seems to work as well, at least the simulation finished and the visualization looks ok.
    1 reply
    Steven R. Brandt
    @stevenrbrandt
    @fedelopezar I updated Simfactory. If you fill in the allocation in defs.local.ini, the build should work even without a default allocation.
    Federico G Lopez Armengol
    @fedelopezar
    Ok @stevenrbrandt .
    Still testing the previous way though, we overcame the error, but found a new one:
    Using configuration: sim-gpu
    Reconfiguring sim-gpu
    Writing configuration to: /home/flopezar/carpetx/singularityTest/Cactus/configs/sim-gpu/OptionList
    srun: Job is in held state, pending scheduler release
    srun: job 35843 queued and waiting for resources
    Interactive job 35843 running: 
    srun: job 35843 has been allocated resources
    yes
    FATAL:   could not open image /work/sbrandt/images/etworkshop.simg: failed to retrieve path for /work/sbrandt/images/etworkshop.simg: lstat /ddnA/work/sbrandt/images: permission denied
    srun: error: db011: task 0: Exited with exit code 255
    Steven R. Brandt
    @stevenrbrandt
    @fedelopezar fixed the permission issue, please try again
    Federico G Lopez Armengol
    @fedelopezar
    It compiled successfully now.
    Steven R. Brandt
    @stevenrbrandt
    @fedelopezar job submission should also work
    Running jobs, too
    Roland Haas
    @rhaas80

    @dstndstn @eschnett the tutorial server worked very well on Friday! No glitches (a bit of waiting for the presenter when they foolishly let go of their container while users were starting up their's, but that is "user error" :-) ).

    Congrats! And many thanks!

    Steven R. Brandt
    @stevenrbrandt
    Yes, bravo!
    Erik Schnetter
    @eschnett
    :-)
    Dustin Lang
    @dstndstn
    Hi all, I would now like to shut down the "hub" machine, so if you have any data there, can I ask you to please copy it off now. If you want more time or are still working on anything, I can leave it running, but otherwise I will shut it off in google cloud and it will poof disappear back into the ether. @rhaas80 , @stevenrbrandt
    Steven R. Brandt
    @stevenrbrandt
    @dstndstn go ahead and shut down
    Roland Haas
    @rhaas80
    @dstndstn my data can go poof. It was ephemeral anyway. Knowing my fellow humans though... is there a way to pull a tarball of the various $HOME first?
    Dustin Lang
    @dstndstn
    :) yeah, it's only 140 GB, I guess that's worth archiving
    Dustin Lang
    @dstndstn
    Alright, I archived the /nfs/home directory, and am now shutting it down!
    Erik Schnetter
    @eschnett
    does anybody have a copy of NRPyWaveToy.ipynb?
    @dstndstn it would be in that tarball you made. where is the archive?
    (a fresh copy of the notebook would also do.)
    Dustin Lang
    @dstndstn
    The tarball is on symmetry in /gpfs/dlang/ET-tutorial-server-hub/nfs-home.tgz
    Steven R. Brandt
    @stevenrbrandt
    I submitted it.
    dassoumyadeep261997
    @dassoumyadeep261997
    Hi, I was not able to attend the Summer School as I was having my exams. Is there any place where I can find the lectures and tutorials to learn from ?
    Roland Haas
    @rhaas80
    The recordings will be posted to the Einstein Toolkit YouTube channel https://www.youtube.com/channel/UC8IObWZ7_wEbWnbIKVIQRYQ soon. I do not know about the material. I may be added to the school website (https://einsteintoolkit.github.io/et2022uidaho) and some is also available on the regular ET tutorial server's (https://etk.cct.lsu.edu/) repo https://github.com/nds-org/jupyter-et/tree/master/tutorial-server/notebooks
    dassoumyadeep261997
    @dassoumyadeep261997
    Thank you so much for the information!
    s.tootle
    @irrationalnumbers
    Recordings are up!
    Roland Haas
    @rhaas80
    @johnny: the compiled files for an ExtenalLibrary end up in:
    configs/sim/scratch/external/FOO
    for and ExternalLibrary FOO
    Roland Haas
    @rhaas80
    failing stests for CarpetX :