Discussion channel to work on creating metadata files that can provide zarr type access speeds to older formated data of many different types
??
but if I open a terminal and do "conda list" it's there
Does this look right?
sources:
nwm-reanalysis:
driver: intake_xarray.xzarr.ZarrSource
description: 'National Water Model Reanalysis, version 2.1'
args:
urlpath: 'reference://'
simple_templates: True
storage_options:
target_options:
anon: true
compression: 'zstd'
target_protocol: s3
fo: 's3://esip-qhub-public/noaa/nwm/nwm_reanalysis.json.zst'
remote_options:
anon: true
remote_protocol: s3
Otherwise good
If you have ideas about how to render it better, please let me know!
(or I guess I could ask Jim Bednar)
The one thing we might consider is whether to remove the "forecast" dataset from that catalog, as we don't have that updating yet to match the rolling forecast archive so it's not being kept up to date)
Here's a better nbviewer link for the NWM demo notebook: https://nbviewer.jupyter.org/gist/rsignell-usgs/02da7d9257b4b26d84d053be1af2ceeb
I think it’s fine to leave forecast. When talking about it, we can say how this is a specific line of discussion in pageo-forge, how to keep rolling datasets up to date.
Oops, I had a few typos in my strftime, here's an hopefully final notebook link: https://nbviewer.jupyter.org/gist/rsignell-usgs/89767200a0722462d37ea971b9588004
I tried recreating the combined json all in one go (using dask.bag to create individual reference dicts, passing them to mzz, combining everything into a combined json) and when I open the result with xarray I get the following errors for some (but not all) of my variables:
ds = xr.open_dataset(fs.get_mapper(""), engine='zarr')
/home/conda/store/896e738a7fff13f931bce6a4a04b3575ecd1f4cbd0e7da9d83afcc7273e57b60-pangeo/lib/python3.8/site-packages/xarray/conventions.py:512: SerializationWarning: variable 'qBtmVertRunoff' has multiple fill values {-9999000, 0}, decoding all values to NaN.
So some of the variables have been 100% converted to NaN, apprently due to conflicting fill values
I get that same message.
It actually would be nice if we fixed the metadata so that 0 was not converted to NaN. I doubt they meant for that to happen -- the providers just didn't understand the CF conventions well enough, which is not that uncommon.
Lucas, If you are still working today, can you take a look at this notebook and try to figure out why the streamflow encoding in the original NetCDF files is different than in the consolidated dataset? In cells [18] and [19] here you can see the difference: the
scale_factor has round off error, and the _FillValue is 0 instead of 999900.
I don’t know, but it’s plausible that some values are inferred by cfgrib, as opposed to real attributes in the data, and that this inference path is different with the zarr interface versus netcdf. Note that the zarr version is stored in JSON text and loaded by python as a float64. In the original float32, this is the closest representation possible to 0.01.
In [10]: np.array(0.009999999776482582, dtype="f4")
Out[10]: array(0.01, dtype=float32)
no, that’s xarray’s view
and 0.01(32) == 0.009999999776482582(64)
$ ncdump -h 202001011100.CHRTOUT_DOMAIN1.comp | grep streamflow
int streamflow(feature_id) ;
streamflow:long_name = "River Flow" ;
streamflow:units = "m3 s-1" ;
streamflow:coordinates = "latitude longitude" ;
streamflow:grid_mapping = "crs" ;
streamflow:_FillValue = -999900 ;
streamflow:missing_value = -999900 ;
streamflow:scale_factor = 0.01f ;
streamflow:add_offset = 0.f ;
streamflow:valid_range = 0, 5000000 ;
The "f" following the values indicates floating point (32 bit)
And on the subject of missing values, the provider should have just stopped at
Everything outside that range is turned into NaN, which includes -999900
valid_range: https://cfconventions.org/Data/cf-conventions/cf-conventions-1.8/cf-conventions.html#missing-dataEverything outside that range is turned into NaN, which includes -999900
I wonder if it conflicts with zarr’s internal missing value field, which is called “fill_value”. Note that http://xarray.pydata.org/en/stable/generated/xarray.open_zarr.html explicitly mentions _FillValue and missing_value, whereas for HDF they would be handled by h5py. Maybe we are seeing an xarray bug?
^ those were two unrelated guesses, if it wasn’t clear :)
I checked to see if the attributes survive round tripping with xarray and netcdf, and they do:
https://nbviewer.jupyter.org/gist/rsignell-usgs/4ea6caac48319e0f39cd2fd1ecaee027
https://nbviewer.jupyter.org/gist/rsignell-usgs/4ea6caac48319e0f39cd2fd1ecaee027
So at least that's good!