Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
Ryan May
@dopplershift
@kedoneyes Yes, that's what I mentioned in #315
Ryan May
@dopplershift
@winash12 Again, If I go to https://www1.ncdc.noaa.gov/pub/data/igra/data/data-por/ and download the data for 42675, I see a whole bunch of missing values (-9999), which get translated to nans. Siphon's not doing anything wrong in this case, that's an issue with the underlying data, nothing we can do about. If you encounter any additional data oddities in the future, my recommendation would be to first try to download the data manually and see if what Siphon is giving you matches up with what's in the raw data file.
Ray Bell
@raybellwaves
Hi thought I would ask here.
Look to open a OpenDAP url with xarray but to handle auth not and not sure if there is syntax with the backend_kwargs?
something like
url = "https://rda.ucar.edu/thredds/dodsC/files/g/ds084.1/2020/20200101/gfs.0p25.2020010100.f000.grib2"
ds = xr.open_dataset(url, backend_kwargs={"auth": ("USER@DOMAIN.COM", "PWD")})
Screen Shot 2021-02-12 at 3.29.17 PM.png
it's cool to a jupyter notebook link
but it fails at dataset = ds.remote_access()
Screen Shot 2021-02-12 at 3.30.23 PM.png
it's not great (you have to edit some dot files) but that seems to be the way with opendap?
@raybellwaves You might also be able to use the authentication as in this example: https://unidata.github.io/siphon/latest/examples/Basic_Usage.html#sphx-glr-examples-basic-usage-py
and then try to access using CDMRemote (which siphon supports using with xarray)
Ray Bell
@raybellwaves
Thanks @dopplershift i'll take a look. Just moved a Q to SO https://stackoverflow.com/questions/66178846/read-in-authorized-opendap-url-using-xarray
but i'm happy to try /understand siphon a bit more
Ray Bell
@raybellwaves
Screen Shot 2021-02-12 at 4.20.34 PM.png
The .file worked
*dot files
Ray Bell
@raybellwaves
atmosmattbrewer
@atmosmattbrewer

Is anyone having problems pulling netCDF subsets from NCEI using TDSCatalogs? I've been trying to update some old codes that used siphon and have been looking at the Unidata python training to help but it seems like I keep getting hung up on either one of theses lines : data = ncss.get_data(query)
or
ncss = cat.datasets[f'gfsanl_4_{dt:%Y%m%d}_{dt:%H}00_000.grb2'].subset()

which gives me error messages of:
HTTPError: Error accessing https://www.ncei.noaa.gov/thredds/ncss/model-gfs-g4-anl-files-old/201801/20180104/gfsanl_4_20180104_1200_000.grb2?var=Apparent_temperature_height_above_ground&var=Pressure_reduced_to_MSL_msl&var=v-component_of_wind_height_above_ground&var=u-component_of_wind_height_above_ground&time=2018-01-04T12%3A00%3A00&accept=netcdf
Server Error (500: java.lang.IllegalStateException: No records found in dataset gfsanl_4_20180104_1200_000.grb2)
and
HTTPError: Error accessing https://www.ncei.noaa.gov/thredds/ncss/model-gfs-g4-anl-files-old/201801/20180104/gfsanl_4_20180104_1200_000.grb2/dataset.xml
Server Error (500: java.lang.IllegalStateException: No records found in dataset gfsanl_4_20180104_1200_000.grb2)

I'm running this in jupyter notebooks, so once I get the first error if I don't restart I get the second error.
Also, if you click the link in the first error message it will download the netCDF subset that you were requesting.

Joleen Feltz
@joleenf

I can't get the python logger to print to either a file or screen when using TDSCatalog. Is there an alternate way to create messages for debugging in siphon? For Example:

import logging
from siphon.catalog import TDSCatalog

FORMAT = '%(asctime)s %(clientip)-15s %(user)-8s %(message)s'
logging.basicConfig(format=FORMAT)
logger = logging.getLogger(__name__)
logger.warning("Test This")

Does not work when the siphon import is being used.

Ryan May
@dopplershift
Looks like #317 Can you comment on that issue so that it comes back on top of the pile?
Joleen Feltz
@joleenf
@dopplershift. Okay.
Joleen Feltz
@joleenf
Joleen Feltz
@joleenf
@dopplershift. I think I have a solution using itertools and fnmatch to filter as needed.
Ryan May
@dopplershift
Well, glad you have a work-around. We'll definitely address that in the next release, but that's not going to happen until early 2022 at best.