Unidata/siphon

Siphon - A collection of Python utilities for retrieving data from Unidata data technologies.

agktogether

@agktogether

https://www.ncei.noaa.gov/thredds/ncss/gfs-g4-anl-files/

Sean Arms

@lesserwhirls

(I answered @agktogether via eSupport, but posting here in case others are wondering). The issue was that the NCEI server didn't have data for 12 May yet (only up to the 9th at this point). For the most recent output, https://thredds.ucar.edu/thredds/catalog/grib/NCEP/GFS/Global_0p5deg_ana/catalog.html should work (other servers are available, too).

agktogether

@agktogether

and i thanked you @lesserwhirls there but i post here for appreciating more your answer

thanks for your fast answering i got really surprised

hannah

@story645

Hi favorite people - I'm using the unidate education gateway thing this summer (thanks @lesserwhirls ) and have like 4 students who are gonna be working on goes 16/goes 17 and was wondering what's the best way to have them not download that 4 times...also does unidata have a cloud version of the gfs-mos?

Ryan May

@dopplershift

We ourselves don't have any cloud MOS. Maybe Google? It just sounds familiar, but I could be making it up. Don't worry about them hitting it 4 times from us, that's fine. If you really care, you could have them access GOES-16/17 from the noaa-goes16 and noaa-goes17 S3 buckets

Dan Adriaansen

@DanielAdriaansen

I would like to get a version of Siphon that contains the fix in #291 - I am using Anaconda. Do I need to build from src from master, or is there a RC somewhere on conda-forge that I can grab that would include these changes? Thanks!

Ryan May

@dopplershift

Boy we really need to make a release. :sob: In the meanwhile, if you have git available, you could do: python -m pip install git+https://github.com/Unidata/siphon.git

Joleen Feltz

@joleenf

I am currently trying to read Radar Level III storm track information from a THREDDS server using Siphon. One of the data variables is text, the second is a Structure. Is there anything I can do with that second variable? Are there any documentation online regarding these files?

``` from siphon.catalog import TDSCatalog
from siphon.radarserver import RadarServer
from siphon.cdmr import Dataset
from datetime import datetime, timedelta

cat = TDSCatalog("http://thredds.ucar.edu/thredds/radarServer/catalog.xml")
url = cat.catalog_refs['NEXRAD Level III Radar from IDD'].href
rs = RadarServer(url)
query_latest = rs.query()
now = datetime.utcnow()
query_latest.lonlat_box(292.9375, 235.0625, 25.0625, 52.9375).time(now).variables('NST')

query_latest_cat = rs.get_catalog(query_latest)

data_available = list(query_latest_cat.datasets.values())
if len(data_available) > 0:
print (data_available[0].access_urls['CdmRemote'])
data_from_thredds = Dataset(data_available[0].access_urls['CdmRemote'])
else:
print ("Empty query")

print (data_from_thredds)```

Ryan May

@dopplershift

Well, one option is to try to use opendap, which translates the structure. I'm not sure if that's more helpful or not:

nc = data_available[0].remote_access(service='OPENDAP')

Joleen Feltz

@joleenf

Hi Ryan, I tried both using opendap with netcdf4 and with xarray, the second variable was available/created? with netcdf4 and I am not sure xarray provided easier access to the variable. Though perhaps presents it differently. xarray creates five data variables. Two are string ndarrays of shape (1,) and three are int16 of shape(1,). I am thinking that this dataset is comprised of table-like data...but I don't really know. I have not found format information for these files, though I am still looking.

Joleen Feltz

@joleenf

@dopplershift Looking a little closer at this through the various tools, siphon, netcdf4, xarray... I think that the problem is that the data in the file does not comply with cdm standards as far as I can tell. I will just try to get the weather and climate toolkit scripting working. Hopefully I can point that to an opendap server via a command line call.

Ryan May

@dopplershift

You can open these using MetPy:

from metpy.io import Level3File
from siphon.catalog import TDSCatalog
from siphon.radarserver import RadarServer
from datetime import datetime, timedelta

cat = TDSCatalog("http://thredds.ucar.edu/thredds/radarServer/catalog.xml")
url = cat.catalog_refs['NEXRAD Level III Radar from IDD'].href
rs = RadarServer(url)
query_latest = rs.query()
now = datetime.utcnow()
query_latest.stations('GRR').time(now).variables('NST')
query_latest_cat = rs.get_catalog(query_latest)
data_available = list(query_latest_cat.datasets.values())
f = Level3File(data_available[0].remote_open())

The internal data structure isn't great and is pretty low level, but for these products, f.sym_block[0] should give you a collection of what's in the file, in order.

Joleen Feltz

@joleenf

@dopplershift Actually, f.sym_block[0] is very helpful. It at least provides a dictionary so it is easier to work with.

Dan Adriaansen

@DanielAdriaansen

Boy we really need to make a release. :sob: In the meanwhile, if you have git available, you could do: python -m pip install git+https://github.com/Unidata/siphon.git

Thank you so much for this suggestion!

Ryan May

@dopplershift

No problem!

agktogether

@agktogether

Hi sirs ,
I am always here to ask some questions.
but this time the problem is how can i access data of the future time that gfs forecast model predicts..
i know this data is there but i can not find it ,i need geoheight, temp, press , u-wind.v-wind properties in isobaric layers...
i am here https://thredds.ucar.edu/thredds/catalog/catalog.html
but i can not find future data 3 hourly ...
for example i need forecast of 3h or 6h after 00:00 in october 30 2020.last night

thanks

Ryan May

@dopplershift

Apologies on the delayed response. I think what you're looking for should be here: https://thredds.ucar.edu/thredds/catalog/grib/NCEP/GFS/Global_0p25deg/catalog.html

winash12

@winash12

i am not able to get anything out of the IGRA soundings site

i keep getting no dates matching selection

kedoneyes

@kedoneyes

When I downloaded the data of 03005 from the University of Wyoming sounding station at 12:00 on January 5, 2019, I found that the data below was wrong

Ryan May

@dopplershift

@winash12 which site and date/time?

@kedoneyes How was it wrong?

kedoneyes

@kedoneyes

@dopplershift The elevation data of the station is not consistent with the data on the network.

winash12

@winash12

from siphon.simplewebservice.igra2 import IGRAUpperAir

date = datetime(2021, 1, 22, 0)
station = 'INM00042369'

winash12

@winash12

df, header = IGRAUpperAir.request_data(date, station)

kedoneyes

@kedoneyes

Do you have a mistake in downloading the data?

winash12

@winash12

I do not get any data at all

kedoneyes

@kedoneyes

I can get data from the Wyoming station, but the elevation data is wrong, so I can't use it.

winash12

@winash12

Hello. Please read my question again. My question is specific to IGRA soundings. There is no issue with Wyoming whatsoever

kedoneyes

@kedoneyes

I'm sorry, I was just talking about my problem, not answering yours.

Ryan May

@dopplershift

@kedoneyes If I look on the Wyoming website, for 12Z on 2021-1-5 for 03005, I see:

 Station latitude: 60.13
 Station longitude: -1.18
 Station elevation: 82.0

When I run:

WyomingUpperAir.request_data(datetime(2021, 1, 5, 12), '03005')

the data also have elevation of 82. If 82 isn't correct, that's outside our control.

kedoneyes

@kedoneyes

If you can, I would like you to download the file and compare it with the data provided on the University of Wyoming website, and look at the height values of each floor, and I'm sure you can see the problem.

Ryan May

@dopplershift

I did that with WyomingUpperAir.request_data(datetime(2021, 1, 5, 12), '03005'). I see exactly what Wyoming shows. Can you share what code you're running?

Oh, wait you mean the height column?

Ryan May

@dopplershift

@kedoneyes Unidata/siphon#315

Ryan May

@dopplershift

@winash12 If you download the data file from https://www1.ncdc.noaa.gov/pub/data/igra/data/data-por/ you can see that the most recent data available for INM00042369 is from 6 January 2021--that's why you don't get any data.

winash12

@winash12

@dopplershift I tried downloading data for another station 42675

and current data is available but full of NaNs

Index(['lvltyp1', 'lvltyp2', 'etime', 'pressure', 'pflag', 'height', 'zflag',
       'temperature', 'tflag', 'relative_humidity', 'direction', 'speed',
       'date', 'u_wind', 'v_wind', 'dewpoint'],
      dtype='object')
0     925.0
1     850.0
2     700.0
3       NaN
4       NaN
5       NaN
6       NaN
7       NaN
8       NaN
9       NaN
10      NaN
11      NaN
12      NaN
13      NaN
14      NaN
Name: pressure, dtype: float64
{'etime': 'second', 'pressure': 'hPa', 'height': 'meter', 'temperature': 'degC', 'dewpoint': 'degC', 'direction': 'degrees', 'speed': 'meter / second', 'u_wind': 'meter / second', 'v_wind': 'meter / second'}
hPa
[nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan] degree_Celsius
[nan nan nan nan nan nan nan nan nan nan nan nan nan nan nan] degree_Celsius
[0.4 2.4 4.1 -0.0 0.4 0.7 3.4 3.3 3.5 1.7 6.2 8.7 4.7 5.4 9.5] meter / second
[344.05460409907715 294.6235647861636 335.2656974709475 0.0 349.2157021324374 315.0 305.2175929681927 324.9262455066517 340.15930191603167 350.059426966887 329.91031376122334 315.0 315.0 315.0 315.0] degree
Traceback (most recent call last):
  File "igra_sounds.py", line 102, in <module>
    lcl_pressure, lcl_temperature = mpcalc.lcl(p[0], T[0], Td[0])
  File "/usr/local/lib/python3.8/dist-packages/metpy/xarray.py", line 677, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/metpy/units.py", line 320, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/metpy/calc/thermo.py", line 357, in lcl
    lcl_p = so.fixed_point(_lcl_iter, pressure.m, args=(pressure.m, w, temperature),
  File "/usr/local/lib/python3.8/dist-packages/scipy-1.5.2-py3.8-linux-x86_64.egg/scipy/optimize/minpack.py", line 937, in fixed_point
    return _fixed_point_helper(func, x0, args, xtol, maxiter, use_accel)
  File "/usr/local/lib/python3.8/dist-packages/scipy-1.5.2-py3.8-linux-x86_64.egg/scipy/optimize/minpack.py", line 891, in _fixed_point_helper
    raise RuntimeError(msg)
RuntimeError: Failed to converge after 50 iterations, value is nan

kedoneyes

@kedoneyes

kedoneyes

@kedoneyes

The first picture is the downloaded data, focusing on the elevation of 256 pressure. The second image, from the University of Wyoming Web site, shows the altitude at 10210 when the air pressure is 256

kedoneyes

@kedoneyes

I think I found the problem. Df.read _ fwf uses the first 100 rows by default to deduce the width of each column of data. If the 100 th elevation is less than the 10000 meter,The program infers four characters.

Ryan May

@dopplershift

@kedoneyes Yes, that's what I mentioned in #315

Where communities thrive

People

Repo info

Activity