Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
Andrew
@IAteAnDrew1_twitter
alright, I guess I'll stick to downloading the files with fastgrib; thanks!
Bryan Guarente
@bryanguarente
@IAteAnDrew1_twitter Have you seen any documentation from the IAState website about what the 'begints' (begin timestamp) format is? More specifically, there are some dates that make little to no sense to me: 1874-10-01 06:09:24-05:50:36. I can't seem to figure out what the second time is (after the dash). It would seemingly be a UTC Offset based on the pattern of other datetimes in this file, but I have never seen a UTC Offset like that before. Thoughts?
Andrew
@IAteAnDrew1_twitter
Which station is that? If it's international it's likely bad data
Andrew
@IAteAnDrew1_twitter
actually there's quite a lot of those... chicago, nebraska, michigan, wisconsin, midwestern states are all UTC-6 (CST) which is pretty close to -5:50:36; I suggest to just round those, or do a simple replace df['begints'] = df['begints'].str.replace('-5:50:36', '-06:00:00')
James Simkins
@jsimkins2

Hello! I'm not sure if this is the correct gitter to be posting this in, and I apologize if it isn't, but I've been unable to connect to the Unidata Thredds AWS radar server all day today. Here is a code snippet that I use and the error that I've been getting:

from siphon.catalog import TDSCatalog from siphon.radarserver import RadarServer, get_radarserver_datasets from datetime import datetime, timedelta

cat = TDSCatalog('http://thredds-aws.unidata.ucar.edu/thredds/radarServer/catalog.xml') rs = RadarServer(cat.catalog_refs['S3 NEXRAD Level II'].href) query = rs.query() query.stations('KDOX').time(datetime.utcnow()) cat = rs.get_catalog(query) raw_list = list(cat.catalog_refs) ds = list(cat.datasets.values())[0]

"ConnectionError: HTTPConnectionPool(host='thredds-aws.unidata.ucar.edu', port=80): Max retries exceeded with url"

Ryan May
@dopplershift
@jsimkins2 Here is fine. Not sure what went wrong, but a quick reboot seems to have fixed the issues I saw. Is it working for you now?
James Simkins
@jsimkins2

@dopplershift works now for python 3.5!...for 2.7, I receive this error message...

"HTTPError: 403 Client Error: Forbidden for url: http://thredds-aws.unidata.ucar.edu/thredds/radarServer/catalog.xml"

Ryan May
@dopplershift
@jsimkins2 I’m not seeing any problems when I test on Python 2.7. If 2.7 is important for your use, if you point me to the code that’s failing I can dig in further.
James Simkins
@jsimkins2

Hello!

I'm receiving a 'Connection Refused Error' due to Max entries exceeded. Last time it just required a reboot. Here's the error:

requests.exceptions.ConnectionError: HTTPConnectionPool(host='thredds-aws.unidata.ucar.edu', port=80): Max retries exceeded with url: /thredds/radarServer/nexrad/level2/S3/dataset.xml (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f24ce4c3748>: Failed to establish a new connection: [Errno 111] Connection refused',))

James Simkins
@jsimkins2
Seems to be working again without the error, thanks!
Ryan May
@dopplershift
Just bounced it. Thanks for letting me know!
James Simkins
@jsimkins2

Hello!

I'm receiving an HTTPError when trying to pull Metar data via siphon from Unidata Thredds. The problem seems to be with my bounding box. When I try the bounding box from openradar , it works fine. However, when I try any other bounding box, I receive the error. Below is an example:

from netCDF4 import Dataset
from siphon.ncss import NCSS
from siphon.catalog import TDSCatalog
from datetime import datetime
metar_cat_url = 'https://thredds.ucar.edu/thredds/catalog/nws/metar/ncdecoded/catalog.xml?dataset=nws/metar/ncdecoded/Metar_Station_Data_fc.cdmr'
metar_cat = TDSCatalog(metar_cat_url)
dataset = list(metar_cat.datasets.values())[0]
ncss_url = dataset.access_urls["NetcdfSubset"]
# bounding box
bb = {'north' : 47, 'south' : 42, 'east' : -88, 'west' : -100}

ncss = NCSS(ncss_url)
ncss.variables
now = datetime.utcnow()
now_time = datetime(now.year, now.month, now.day, now.hour)

query = ncss.query()
query.lonlat_box(north=bb['north'], south=bb['south'], east=bb['east'], west=bb['west'])
query.time(now_time)
query.variables('air_temperature', 'dew_point_temperature', 'inches_ALTIM', 'wind_speed', 'wind_from_direction', 'cloud_area_fraction', 'weather')
query.accept('csv')

data = ncss.get_data(query)
And here is the error I receive:

HTTPError: Error accessing https://thredds.ucar.edu/thredds/ncss/nws/metar/ncdecoded/Metar_Station_Data_fc.cdmr?var=wind_from_direction&var=air_temperature&var=weather&var=dew_point_temperature&var=cloud_area_fraction&var=inches_ALTIM&var=wind_speed&time=2018-10-24T14%3A00%3A00&west=-100&south=42&north=47&east=-88&accept=csv: 500 Infinite loop in linked list at recno= 7976

Does anyone know a solution for this? Thanks!

John Leeman
@jrleeman
This is actually a data problem on the server. Sorry about that! We're working on a better solution, but currently that service is sometimes broken by certain incoming data.
James Simkins
@jsimkins2
No worries, thanks for the heads up!
David Hoese
@djhoese
As someone trying to read data from an ABI THREDDS server. I give a +1 to Unidata/siphon#263
is there a semi-easy way to do walk through the catalog to get all channels for a specific time?
Ryan May
@dopplershift
There’s not an easy way to do what you want. This is really where we need to go beyond static catalogs and directory-based metadata and into server-side collections and database-y things
Pete Pokrandt
@PTH1_twitter
@djhoese sweet!
David Hoese
@djhoese
@dopplershift what is your opinion of "Intake"?
Ryan May
@dopplershift
I don’t think it solves any new problems (at least in earth science) that we weren’t already solving with THREDDS.
James Simkins
@jsimkins2

Hello!

I've been unable to pull radar data via Siphon since 6PM last night. It seems there's an error with the AWS credentials.

line 161, in get_catalog
raise BadQueryError(self.get_catalog_raw(query))
siphon.http_util.BadQueryError: b'com.amazonaws.AmazonClientException: Unable to load AWS credentials from any provider in the chain

Can you send a bit of sample code that’s failing?
James Simkins
@jsimkins2
Here is an example snipet:
from siphon.radarserver import RadarServer, get_radarserver_datasets
from siphon.catalog import TDSCatalog, get_latest_access_url
from datetime import datetime, timedelta

site = 'KDOX'
cat = TDSCatalog('http://thredds-aws.unidata.ucar.edu/thredds/radarServer/catalog.xml')
rs = RadarServer(cat.catalog_refs['S3 NEXRAD Level II'].href)
rs.stations[site]
query = rs.query()
query.stations(site).time(datetime.utcnow())
rs.validate_query(query)
catalog = rs.get_catalog(query)
dataset = list(catalog.datasets.values())[0]
Ryan May
@dopplershift
I can reproduce with that. I think we switched the server over to a new instance, so I’m guessing it’s a config problem. Let me see what I can do here...
Ryan May
@dopplershift
@jsimkins2 It should be fixed now.
James Simkins
@jsimkins2
@dopplershift it works now, thanks! Were the files altered at all? Just curious if my new netcdf error is a result of a new file structure or a pyart problem
Ryan May
@dopplershift
Nothing changed about the underlying files. Could be pyart or some other underlying problem.
Warren Pettee
@WEP11
Would adding a serial interface for hardware like Arduino be in Siphon's realm? Assuming the user programmed the Arduino to return some kind of standardized output.
Ryan May
@dopplershift
I feel like that’s beyond Siphon’s scope, since it’s highly unlikely any random user of siphon would have access to the appropriate hardware. Would probably be best to put something like that in a separate package.
Warren Pettee
@WEP11
It was just a random Saturday morning idea. The more I think about it, the more difficult it would be to envision how the implementation would fit cleanly in Siphon.
Ryan May
@dopplershift
Nothing wrong with crazy ideas :wink: