This is on my list of things to deal with today.
https://jupyterhub.ucar.edu/dav will put you on casper (which uses slurm)
I've just discovered that we use SLURM rather than PBS
sbatch: error: You must specify an account (--account)In [4]: print(cluster.job_script())
#!/usr/bin/env bash
#SBATCH -J dask-worker
#SBATCH -n 1
#SBATCH --cpus-per-task=16
#SBATCH --mem=60G
#SBATCH -t 00:30:00
#SBATCH -C skylake
JOB_ID=${SLURM_JOB_ID%;*}
/glade/u/home/mrocklin/miniconda/envs/dev/bin/python -m distributed.cli.dask_worker tcp://10.12.203.5:39794 --nthreads 16 --memory-limit 64.00GB --name dask-worker--${JOB_ID}-- --death-timeout 60 --interface ib0
In [5]: import dask
In [6]: dask.config.get("jobqueue.slurm")
Out[6]:
{'name': 'dask-worker',
'cores': 1,
'memory': '25 GB',
'processes': 1,
'interface': 'ib0',
'walltime': '00:30:00',
'job-extra': {'-C skylake': None},
'death-timeout': 60,
'local-directory': None,
'shebang': '#!/usr/bin/env bash',
'queue': None,
'project': None,
'extra': ['--interface', 'ib0'],
'env-extra': [],
'job-cpu': None,
'job-mem': None,
'log-directory': None}
I had to copy over my project from my PBS_ACCOUNT environment variable
Is this global?
OK, I'm all set. Thanks for your help @jhamman !
@jhamman I've to customize the installation with GCSFS/Fuse as by https://gcsfs.readthedocs.io/en/latest/fuse.html and seems that the only way ( for my understanding ) is to create a clone repository follow the instruction here https://github.com/pangeo-data/pangeo-cloud-federation. If you have any other idea I'm more than open to avoid a huge headache to me
looking forward to attend the pangeo community meeting next week http://pangeo.io/meetings/2019_summer-meeting.html ... Thank you all for organizing and in particular providing remote access for those not able to attend in person (but i will be there in person)...
hi kevin , thanks for the marge, i have a question
in compute_study.md, it is indicated that
'Duplicate each study for 2, 4, 8, and 16 workers per node (reducing chunk size proportionally)'
But I do not recall this reduction of chunk size for each increase of workers per node in utils.py,
am i missing something??
Let me look more closely...
@tinaok Also, if you are planning on attending the Pangeo community meeting remotely, you might be able to present some of your benchmarking work in a Lightning Talk. It would be a very short (and remote) presentation, but it might be possible. @jhamman would know for sure.
@tinaok Ok. I've looked more closely at the benchmarking code. (@andersy005 may have something more to say about this, but I'll take a stab myself.) The design of the current code assumes 1 chunk per worker (see
benchmarks/datasets.py), and it therefore assumes that the total dataset size will be equal to the (chunk size) * (number of nodes) * (number of workers per node).
This may not be optimal, but I think this can be amended in later versions.
The thought behind the
compute_study.md writeup was to sketch out what would be needed to generate some preliminary scaling studies for various "common" operations done with xarray and dask. The results of each study should be a plot of "Number of Nodes" vs "Operation Runtime". However, the "Operation Runtime" depends on much more than "Number of Nodes", including "Number of Workers per Node", "Number of Threads per Worker", "Total Number of Chunks", "Chunk Size", etc.
I wanted to consider 2 kinds of studies, strong and weak, since these are considered "canonical" in the HPC world. In the strong scaling studies, the "Total Data Size" should be fixed while the "Number of Nodes" is varied. In the weak scaling studies, the "Data Size per Node" should be fixed while the "Number of Nodes" is varied.
When I was coming up with the
compute_study.md document, I tried to find a way to fix all of the other parameters in a way such that each study was "fair." I chose 1 "Chunk per Worker" and 1 "Thread per Worker", and I chose to vary the "Chunk Size" and "Number of Workers per Node". And later we tried to come up with a way of varying the "Chunking Scheme" (i.e., chunk over all dimensions, chunk over only spatial dimensions, chunk over only time), too. But we need to generate data that looks at how these numbers vary with "Chunks per Worker" and "Threads per Worker", too.
@/all - the agenda and attendee list for next week’s community meeting in Seattle is now final. See details here: http://pangeo.io/meetings/2019_summer-meeting.html#
Remote participation details are also available on this page. @kmpaul - we’ll probably need to wait and see if remote lightning talks will work. We may need a proxy presenter.
thought folks on this channel might be interested in this job opening at JPL https://jpl.jobs/jobs/2019-10892-Big-Data-Software-Lead
Is anyone else experiencing the
Error displaying widget with e.g. the dask_kubernetes.KubeCluster widget (or any other widget)? It looks like this is related to ipywidgets==7.5. I have a Pangeo environment with jupyterlab=0.35, tornado=5.1.1 and dask_labextension==0.3.3, because I noticed that it was a working configuration at some point, but I'm not sure this is still the recommended configuration.
