depends a bit on how big your data is and how much of it you need
if it is more than ~1000MB i'd think about only pulling it when you actually need it (so neither postBuild nor start) because it either makes the image huge (slow startup times) or gives your binder a lot to do on startup (slow startup times :) ) as well as a lot of network traffic
I'm going to update the GPM precipitation dataset at
gs://pangeo-data/gpm_imerg_early, because a new version has been released and the TRMM era has been merged, so it's now available from June 2000 instead of 2014. I will also include the probabilityLiquidPrecipitation field, along with the precipitationCal field, so that we have an information about solid precipitation.
@kmuehlbauer (hi from another non-vispy project) In some of my satpy repositories I've had it run a download script in the background from the
start script. The data being downloaded though was on google's cloud storage though so I wasn't too worried about the network traffic for the ~3GB I was downloading.
according to the tracker it should be fixed..
I wonder if the docker image needs updating
@scottyhq ...
if you do the data download in the
start without putting it in the background then a similar thing will happen where binderhub will wait for jupyter lab to start up but it won't happen in time because data is taking too long to download
(from experience)
BTW Great seeing @kmuehlbauer here.. Kai, I plan to build an OpenRadarScience Pangeo
the AMS Radar workshop will be a launchpad…
@scollis - looks like you can pin jupyterlab<1.1 in your environment file as a solution? if you’re up for a bit of digging a pull request to enable running the ‘start’ script would be great! https://github.com/pangeo-data/pangeo-stacks/blob/4c90b98836c66403ab81ca837ce979ec9628a232/onbuild/r2d_overlay.py#L125
Seems like the right idea! looks like you’ll have to add a line here too https://github.com/pangeo-data/pangeo-stacks/blob/4c90b98836c66403ab81ca837ce979ec9628a232/onbuild/Dockerfile#L15. Are you able to run locally to test? @yuvipanda has mentioned putting r2d-overlay in a separate repo b/c it is a useful tool! So some additional changes could be in store in the future, but this feature would be nice to have in the meantime.
if do ONBUILD RUN /usr/local/bin/r2d_overlay.py start
it would just run that on build right
(winging it :D )
and do you just check it with Docker?
wonder if it is this
(I’m also winging it ;) I think it would be best to take a look at https://github.com/pangeo-data/pangeo-stacks/issues/38#issuecomment-494125005. Then go ahead and start your pull request w/ the current modifications saying it’ll close that issue.
@scollis - yes, the start script is run as an entrypoint. You may be interested in this PR in the main repo2docker repo where the feature was first developed: https://github.com/jupyter/repo2docker/pull/363/files
(we renamed launch —> start but the idea is there)
oh cool.. yeah I can just point repo2docker to the branch
hah.. commended.. I meant commented.. sounds like I have a mouth full of marbles
thanks for working on it
(with image definition in ./geohackweek2019/binder)
then
docker push scottyhq/geohackweek2019:2019-08-01