yes - though I just checked and I am not sure weather the ftp problem for the latter case for actually downloading data is related to the HPC Firewall
I am sorry I do not know how to check the ftproot
you're running a notebook ?
ftproot is not defined apparently
I am trying to fix this first by contacting our help desk and then if it is still problematic I will get back to here - I am sorry for interrupting your work - Have a very nice weekend
You're welcome, feel free to ask any questions
but my guess is a connection issue
also, I don't know your area of interest, but carefull when requesting too large (region([lonmin,lonmax,latmin,latmax,0,4000., '1998-01-01', '2021-03-05']))
Yeah I am guessing that too - I guess they selectively let you use github, and conda...
no it is a small region in the subpolar North-Atlantic and it should be a download of maybe max 5 min.
Can someone auth the CI run for my codespell pr? It's super simple and this is usually how I start when looking at a project that I haven't seen before. euroargodev/argopy#89
Anyone have a sense of about how big it is to fetch all of vdmzrs.ifremer.fr? I've been running rsync for about 8 hours so far and have 34GB of files so far and am still in the first subdir of aoml with about 7800 subdirs so far.
Hi @schwehr I guess you should work with a snapshot of the ftp (one from here https://www.seanoe.org/data/00311/42182/#85023 as pointed out by @quai20 ). The snapshot is compressed, it will be larger on disk. And the rsync service is more for experts who want to keep up with the full dataset and last available data.
(Note that for the rsync, the 1st sync is very very long as you noticed, 20 years of data is a lot, but then all the updates are much faster, since about 300 profiles are added daily)
Ok. I accidentally removed a subdir while prepping the data so I'll need to rsync the subdir again. rsync is an old friend from decades ago. What's one persons big data (my thesis lab data fit on a floppy disc) is another person's tiny data. And now I'm in the world of satellite images where each one is order of 15GB. I found that gzip -9 gave 10% the original size and xz -9 did drastically better than that. A compressing column store should do great
Hi folks, @tylertucker202 , anyone noticed that the argovis can't return data anymore ? See more here donatagiglio/Argovis#3