Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Apr 15 2016 01:45
    amoise opened #49
  • Oct 09 2015 03:46
    rensa closed #48
  • Oct 09 2015 03:46
    rensa commented #48
  • Oct 09 2015 01:45
    DamienIrving commented #48
  • Oct 09 2015 00:48
    rensa opened #48
  • Sep 25 2015 02:02
    rensa commented #47
  • Sep 16 2015 05:54
    ccarouge commented #47
  • Sep 16 2015 05:50
    DamienIrving labeled #47
  • Sep 16 2015 05:50
    DamienIrving commented #47
  • Sep 16 2015 05:36
    ccarouge opened #47
  • Aug 31 2015 03:17
    cet900 commented #42
  • Aug 28 2015 05:37

    captainceramic on offscreen

    Fixed up output pattern constra… (compare)

  • Aug 21 2015 05:12

    captainceramic on offscreen

    Fixed output pattern json probl… (compare)

  • Aug 21 2015 05:00

    captainceramic on offscreen

    Fixed problems after speaking t… (compare)

  • Aug 20 2015 04:47

    captainceramic on offscreen

    Added specialised monthly tos d… Improved (slightly) the excepti… Corrected constraint names for … (compare)

  • Aug 18 2015 05:45

    captainceramic on devel

    Styling changes - mostly whites… More styling changes - mostly w… Fixed docstring typo (compare)

  • Aug 18 2015 01:37

    captainceramic on offscreen

    Added VT module for the webserv… (compare)

  • Aug 17 2015 03:45

    captainceramic on offscreen

    Added single-step Nino calculat… (compare)

  • Jul 22 2015 00:58

    captainceramic on devel

    Added the extracted files to Da… (compare)

  • Jul 12 2015 23:33

    captainceramic on master

    Added a new DatasetSummary modu… Merge branch 'master' into devel Added checks for other temporar… and 8 more (compare)

didiermonselesan
@didiermonselesan
@captainceramic Any ideas on this error when trying to launch VisTrails?
[dpm599@cwsdesktop8 ~]$ vistrails &
[1] 16642
[dpm599@cwsdesktop8 ~]$ 2015-06-05 13:43:37,597 CRITICAL:
/opt/cloudapps/vistrails/2.1.2/vistrails/core/packagemanager.py, line 130
ImportError: "userpackages" sys.path: ['/home/599/dpm599/.vistrails/userpackages/..', '/opt/cloudapps/vistrails/2.1.2', '/opt/cloudapps/vtk/5.8.0/lib/python2.7/site-packages/VTK-5.8.0-py2.7.egg', '/opt/cloudapps/vtk/5.8.0/lib/python2.7/site-packages', '/opt/cloudapps/dbus-python/0.84.0-py2.7/lib/python2.7/site-packages', '/opt/cloudapps/sip/4.15.4-py2.7/lib/python2.7/site-packages', '/opt/cloudapps/pyqt4/4.10.3-py2.7/lib/python2.7/site-packages', '/opt/cloudapps/python-vlab/ep1_2/2.7.5/lib/python27.zip', '/opt/cloudapps/python-vlab/ep1_2/2.7.5/lib/python2.7', '/opt/cloudapps/python-vlab/ep1_2/2.7.5/lib/python2.7/plat-linux2', '/opt/cloudapps/python-vlab/ep1_2/2.7.5/lib/python2.7/lib-tk', '/opt/cloudapps/python-vlab/ep1_2/2.7.5/lib/python2.7/lib-old', '/opt/cloudapps/python-vlab/ep1_2/2.7.5/lib/python2.7/lib-dynload', '/opt/cloudapps/python/2.7.5/lib/python2.7', '/opt/cloudapps/python/2.7.5/lib/python2.7/plat-linux2', '/opt/cloudapps/python/2.7.5/lib/python2.7/lib-tk', '/opt/cloudapps/python-vlab/ep1_2/2.7.5/lib/python2.7/site-packages', '/opt/cloudapps/python/2.7.5/lib/python2.7/site-packages/setuptools-0.7.2-py2.7.egg', '/opt/cloudapps/python/2.7.5/lib/python2.7/site-packages/nose-1.3.0-py2.7.egg', '/opt/cloudapps/python/2.7.5/lib/python2.7/site-packages/GDAL-1.9.2-py2.7-linux-x86_64.egg', '/opt/cloudapps/python/2.7.5/lib/python2.7/site-packages', '/vistrails/src/trunk/vistrails']
Uncaught exception on initialization: No module named userpackages
Traceback (most recent call last):
  File "/opt/cloudapps/vistrails/2.1.2/vistrails/run.py", line 122, in <module>
    v = vistrails.gui.application.start_application()
  File "/opt/cloudapps/vistrails/2.1.2/vistrails/gui/application.py", line 843, in start_application
    x = VistrailsApplication.init(optionsDict)
  File "/opt/cloudapps/vistrails/2.1.2/vistrails/gui/application.py", line 190, in init
    VistrailsApplicationInterface.init(self, optionsDict)
  File "/opt/cloudapps/vistrails/2.1.2/vistrails/core/application.py", line 404, in init
    self.vistrailsStartup.set_registry(reg)
  File "/opt/cloudapps/vistrails/2.1.2/vistrails/core/startup.py", line 597, in set_registry
    self._package_manager.init_registry(registry_filename)
  File "/opt/cloudapps/vistrails/2.1.2/vistrails/core/packagemanager.py", line 181, in init_registry
    setup_basic_package()
  File "/opt/cloudapps/vistrails/2.1.2/vistrails/core/packagemanager.py", line 180, in setup_basic_package
    self.initialize_packages(prefix_dictionary)
  File "/opt/cloudapps/vistrails/2.1.2/vistrails/core/packagemanager.py", line 566, in initialize_packages
    userpackages = self.import_user_packages_module()
  File "/opt/cloudapps/vistrails/2.1.2/vistrails/core/packagemanager.py", line 128, in import_user_packages_module
    import userpackages
ImportError: No module named userpackages
Tim Bedin
@captainceramic
Hello @didiermonselesan - I think this looks like the user packages directory in your home directory might not be present. Could you try doing a ls -l ~/.vistrails/userpackages? Mine looks like:
lrwxrwxrwx 1 teb548 r87  35 May 21 14:06 cwsl -> /home/548/teb548/Code/cwsl-mas/cwsl
-rw-r--r-- 1 teb548 r87   5 Jun  3 15:48 __init__.py
-rw-r--r-- 1 teb548 r87 155 Jun  3 15:48 __init__.pyc
This could be something to do with the order of steps if you haven't run VisTrails before linking the cwsl package in.
Damien Irving
@DamienIrving
Thanks @captainceramic . We went for the delete everything and re-install from scratch and now it's working. Go figure!
willrhobbs
@willrhobbs
I'm having a config issue that's preventing my workflow from working. When I try and execute, I get the following message from the 'Merge Time Series' module:
willrhobbs
@willrhobbs

Uncaught exception: "cwsl_ctools_path is not set in package options"

Traceback (most recent call last):
File "/opt/cloudapps/vistrails/2.1.2/vistrails/core/modules/vistrails_module.py", line 400, in update
self.compute()
File "/home/581/wrh581/.vistrails/userpackages/cwsl/vt_modules/vt_cdscan.py", line 82, in compute
this_process.execute(simulate=configuration.simulate_execution)
File "/home/581/wrh581/.vistrails/userpackages/cwsl/core/process_unit.py", line 281, in execute
raise Exception("cwsl_ctools_path is not set in package options")
Exception: cwsl_ctools_path is not set in package options but as far as I can tell my package configuration is complete:Uncaught exception: "cwsl_ctools_path is not set in package options"

As far as I can tell my config for the CWSLab package is set correctly, i.e. ${HOME}/src/cwsl-ctools. Any ideas?

Tim Bedin
@captainceramic
Hi @willrhobbs - I was just looking in the [ProcessUnit] (https://github.com/CWSL/cwsl-mas/blob/devel/cwsl/core/process_unit.py) code, and that message will be printed if either the path isn't set or path doesn't exist. I don't think that the configuration path currently expands environment variables (though clearly it should) - did you use ${HOME}? If so, could you try again using your hard-coded path? (/home/123/wxh123 etc.)
Tim Bedin
@captainceramic
Certainly open an issue on that though - we should split the two cases and print a better error message. Something like "config_path not found" or "config_path not set" would be much clearer.
Damien Irving
@DamienIrving
As a result of today's discussions I've added a Sanity Check module to the top of the development roadmap: https://github.com/CWSL/cwsl-mas/wiki/Development-roadmap
Damien Irving
@DamienIrving
Another good suggestion from the Hobart workshop was to have a regular newsletter/communication updating users of the CWSLab workflow tool / NCI Virtual Desktop on the latest happenings (e.g. what new modules and bug fixes people are working on, calls for help on particular enhancements to the tool). Perhaps this could simply be a page on the wiki or a blog linked to the CWS Lab website?
willrhobbs
@willrhobbs
Thanks @captainceramic, replacing ${HOME} with the actual path on the config paths fixed the issue. As a stopgap it might be worth updating the Configuration documentation on the wiki.
Damien Irving
@DamienIrving
@captainceramic The "for the impatient" installation instructions point to a very old version of the cwsl-mas master branch at /g/data/r87/public/apps/cwsl-mas/master/cwsl. Should we keep that /g/data/r87/public/apps/ directory in sync with the latest tagged release of the workflow tool?
Tim Bedin
@captainceramic
@DamienIrving - Absolutely. That directory should certainly be kept in sync. I'll pull down a version now.
I've had a go, but @taerwin owns that directory. We can get in touch with him or @craigheady and ask to have the 0.9.0 tags for cwsl-mas and cwsl-ctools checked out to that directory.
Damien Irving
@DamienIrving

@captainceramic I've pretty much thrown cdat out of my software stack and am using xray these days for all my netCDF I/O and routine data analysis.

With respect to our discussions around replacing cdscan in the CWSL workflow tool, the new xray.open_mfdataset() (i.e. open multi-file dataset) function might be the way to go. Read all about it at this blog post - it will blow your mind.

Tim Bedin
@captainceramic
@DamienIrving - thanks for that! Very exciting library (especially for 3D fields and daily etc.). I've been using the MFDataset multi-file datasets from the unidata netCDF4 library, but xray looks like it's quite a bit more clever. Cheers.
Damien Irving
@DamienIrving
@captainceramic Very clever. This tweet kind of says it all: https://twitter.com/shoyer/status/581634755684241408
Damien Irving
@DamienIrving
I wonder if we could setup some temporary guest accounts for people who want to try it out... https://twitter.com/nickengerer/status/624069836344176644
Tim Bedin
@captainceramic
I was thinking about this too - I'm not sure about how to deal with the data access / licencing issue though. To get ua6 access to CMIP5 you need to go through the ESG procedure, so I'm not sure how to deal with that.
Nicholas Engerer
@nickengerer
Thankfully I was able to get my own allocation via NCI startup account!
@DamienIrving @captainceramic was wondering if you could provide some more examples for vistrails workflows? I'm a bit stuck trying to plot solar radiation fields in Australia. Also, I don't seem to have "Timeslice" as an option (as in the tutorial)
Damien Irving
@DamienIrving
@nickengerer The tutorial is a little dated. In the utilities menu there's a module called Crop which you can use to crop your data along a time or spatial axis.
With respect to examples, the cwsl-workflows repository has a bunch of them. Try warm_water_volume.vt or aus_mean_tas.vt as an example
Nicholas Engerer
@nickengerer
@DamienIrving thats very helpful, i didn't realise those were all examples - will check those out and get back to you, cheers!
Damien Irving
@DamienIrving
@nickengerer Yep, any workflow from VisTrails can be saved as a .vt file so we've dumped a bunch of them in that repository
Nicholas Engerer
@nickengerer
@DamienIrving appears I don't yet have access to /short/, so can't store data, the trials of a new user - having this arranged. Will add that to my list of the 'learning curve
Damien Irving
@DamienIrving
@nickengerer By default you have 66GB storage in /local, so that should be enough to play around (be aware that /local is scratch space, so it will be deleted when you shut down the virtual machine)
Nicholas Engerer
@nickengerer
@DamienIrving hmmm don't see a /local/dl0/ (my project dl0) - is there where you meant for me to look?
Damien Irving
@DamienIrving
Does /local exist? I think you can just dump stuff in there
Nicholas Engerer
@nickengerer
@DamienIrving no it doesn't, tried creating a dummy file and moving it there - will just wait for the /short/ access - its already being processed. Thanks for your help!
Damien Irving
@DamienIrving
No problem. Since I stopped working on the CWSLab project I actually no longer have access, so unfortunately I can't check these things out
Nicholas Engerer
@nickengerer
@DamienIrving and here you are, still helping - champion!
Tim Bedin
@captainceramic
Hi @nickengerer ! The local vs short issue popped up when the NCI cloud infrastructure changed over to the new system - we still are not sure of the best place to put temporary data. I think the /short/ makes sense, as long as we can sort out the administrative issues.
Tim Bedin
@captainceramic
PS - this is the workflow that I showed you to plot up the solar radiation climatologies. You will need to change the tasmax variable constraint to rsds. Let me know if you have any environment problems - some of the workflows use module files that are found in the /g/data/r87/public/modulefiles directory that are not available by default. Also, a good tip is to try running a workflow in simulation mode first by changing the simulate option in the preferences to True (in Packages -> Climate and Weather Science Laboratory -> configure -> simulate_execution
Nicholas Engerer
@nickengerer
@captainceramic - thanks for those great tips! will let you know how I go! I think I have storage space now. Also - I've started blogging about the process: http://www.nickengerer.org/research-blog/ - i hope this can help other noobs
Nicholas Engerer
@nickengerer
what do you make of this error (at "Crop"): Traceback (most recent call last):
File "/home/603/nae603/.vistrails/userpackages/cwsl/vt_modules/vt_xmltonc.py", line 194, in compute
this_process.execute(simulate=configuration.simulate_execution)
File "/home/603/nae603/.vistrails/userpackages/cwsl/core/process_unit.py", line 327, in execute
scheduler.submit()
File "/home/603/nae603/.vistrails/userpackages/cwsl/core/scheduler.py", line 221, in submit
self.job.submit(noexec=self.noexec)
File "/home/603/nae603/.vistrails/userpackages/cwsl/core/scheduler.py", line 131, in submit
output = subprocess.check_output(args, stderr=subprocess.STDOUT)
File "/opt/cloudapps/python/2.7.5/lib/python2.7/subprocess.py", line 575, in check_output
raise CalledProcessError(retcode, cmd, output=output)
CalledProcessError: Command '['sh', '/local/dl0/nae603/tmp/tmpnqxqtB.sh']' returned non-zero exit status 2
I've gotten similar errors for other scripts - some of them were obviously tied to the disk space issues; whereas here, I've changed the output directoy to my /home/ just to try it out. @captainceramic
Tim Bedin
@captainceramic
@nickengerer - That error means that the subprocess that actually does the task (the cropping in this case) has failed. I've had a quick go and haven't been able to reproduce this one - would you be able to hover your mouse over the VisTrails module once it fails and goes red? It should give you a bit more information about the failure (if I've updated the code correctly!).
As it is a CalledProcessError it is most likely related to path permissions on the system - perhaps access to the data directory, not being able to create the path it wants to write to or perhaps not being able to find a command-line program it wants to run.
willrhobbs
@willrhobbs
I have a question about the vistrails averaging modules. Does the spatial averaging include areal weighting, and likewise does temporal averaging include time weighting? The importance of the former is self-explanatory; the latter becomes important for budget calculations. Will
Damien Irving
@DamienIrving
@willrhobbs The averaging scripts are all in the aggregation directory of the cwsl-mas repo: https://github.com/CWSL/cwsl-ctools/tree/master/aggregation
You'll see that cdo_field_agg.sh and cdo_time_agg.sh simply use cdo.
It looks like cdo does apply area weighting when calculating fldmean or fldavg: https://code.zmaw.de/projects/cdo/embedded/index.html#x1-3220002.8.5
Damien Irving
@DamienIrving
As far as I'm aware time-weighted averaging is not directly supported in CDO, so we'd have to write a script to do that in another language and throw it into the aggregation directory.
Damien Irving
@DamienIrving
Nicholas Engerer
@nickengerer
@captainceramic @DamienIrving as damien shared above, I've been able to get everything to work - thanks for your help! Now i'm digging through the code repo and trying to make sense of the code you have written. I would like to plot yearly mean values of daily radiation, which isn' currently an option in time aggregation. Probably will come back with a few questions soon. For now, off to a great start - cheers!
Nicholas Engerer
@nickengerer
What is CDO? This seems to be a very important function/module, but doesn't appear to have a declaration in the CWSL repo(s) - looks like its a CAWCR tool? Is it "climate data online"?
Tim Bedin
@captainceramic
CDO stands for Climate Data Operators - a set of command-line tools to manipulate netCDF files. They do things like temporal and spatial averaging, calculating indices, doing arithmetic between datasets etc. I've found them very useful, but you do have to watch out a little bit because there are some gotchas. For example missing values don't propagate when multiplying, and sometimes the operators can make some changes to the file metadata.
Damien Irving
@DamienIrving
@nickengerer This page of the wiki will give you a bit of an idea about how the CWSL code repos relate to one another.