you know at nasa all the earth scientist are the bummers - oh the climate is collapsing - oh a giant storm is going to hit us. we are all jealous of the helio/planetary/astro people who are like 'look we have a cool RC helicopter on mars!'
'oh look a pretty picture of the sun'
rich --- can you put some %%time in the cells?
hee hee. BTW, I'm not really using the cluster there, because we are just reading one chunk and displaying it
If we were going to do some calculations over multiple chunks, however, we definitely would appreciate the cluster
I think this is fine as is --- just want to demostrate the fast access ---
so i thought of a name for what you all are working on --- how about zarr-meta it explains what you are doing & doesn't use the term cloud-performant which lucas knows I just love...
hey - do you all want to do a write up of this for AGU's earth and space science? I would be your editor! I was just on a call this morning where we want to publish more papers that include jupyter notebooks and actually an article like this would find into ESS porfolio really well. this would be a great short paper on the process, it's implications, and then notebooks showing different data examples. I think this would be really valuable.....ESS is open access.
I'm happy to be involved in that @cgentemann
A nice calculation would be a correlation versus time, presumably shorter wavelengths lead longer wavelengths. I think that was the original ask. By the way, FITS is special, I could subdivide the frames on the biggest axis, if that would prove something interesting.
I'm particularly interested to see what is written in the section "Describe FSSPEC without saying FSSPEC"
I am aware of ASDF, the astro-specific “modern” format (but not cloud-friendly!) that has failed to gain tracton for so long. But there are many hurdles to the references method being a viabla entry to FITS for astro analysis. In order, most important first:
the majority of FITS files are whole-file compressed, and this only works for uncompressed. We could maybe cope with bzip2, or even better zstd, but gzip is the one that’s used, and it is probably impossible
omg. you are killing me. they don't use internal compression?
this is really helpful martin, thanks. if we can demo it working, that would be a win, and we can work more with nasa --- if we can show any advancement for data access - just maybe by uncompressing the gzip files after pushing to AWS, maybe that would be a path forward? the astro community is so FITS FITS FITS.... just like I was binary binary binary 30 years ago....
So if I did that same dataset, but with
the wavelength as a coordinate (so you can select it with a slider)
mapping to helio lat/lon (or just show how you can recreate the world coordinates)
a version with sub-selection in one of the dimensions, so you do a timeseries on a section of the image better … is that enough of an argument? Should I do the same for the much more massive set of downsampled JPEG images on the public SDO server? I was hoping to get around to the dataset in intake/fsspec-reference-maker#78 , as a nice example of a dataset made by merging on multiple dimensions.
Perhaps more importantly: that dataset has FIVE dimensions, three aggregated (and two-dimensional images in each chunk)
Yeah, that is also very cool
@martindurant , did you get contacted by Ivelina Momcheva from the Space Telescope Science Institute? I spoke with her a few days ago and she is very interested in your FITS work (and in Pangeo in general).
I wasn't planning on it since I'm already presenting this at AGU, but if are able to send me to Honolulu I certainly won't complain ;)
kk working on invited slot
start working on an abstract!
deadline is tomorrow midnigth
okay - ken knows your abstract is coming in - he said you have to organize it after submission - & he will look for yours. submit to : OD12 Big Data for a Big Ocean 2022
@rsignell-usgs do you have an up-to-date NWM + intake catalog example notebook? I'm presenting fsspec-reference-maker at a geospatial workshop on campus on Monday and thought it might be a cool thing to show off