Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Sep 30 16:29
    theochit commented #210
  • Sep 30 10:16
    tirkarthi synchronize #210
  • Sep 30 03:29
    tirkarthi commented #210
  • Sep 29 21:33
    theochit commented #210
  • Sep 29 21:31
    theochit commented #203
  • Sep 14 20:13
    raybellwaves edited #225
  • Sep 14 20:13
    raybellwaves opened #225
  • Sep 14 20:07
    raybellwaves opened #224
  • Sep 14 19:42

    tomkralidis on master

    WIP: add user_charset arg (#223… (compare)

  • Sep 14 19:42
    tomkralidis closed #223
  • Sep 14 19:39
    raybellwaves commented #223
  • Sep 14 19:39
    raybellwaves commented #223
  • Sep 14 17:47
    raybellwaves synchronize #223
  • Sep 14 15:53
    raybellwaves opened #223
  • Sep 14 13:43
    raybellwaves commented #222
  • Sep 14 08:26

    tomkralidis on master

    Add 'default_charset' to open_u… (compare)

  • Sep 14 08:26
    tomkralidis closed #222
  • Sep 14 05:31
    raybellwaves commented #222
  • Sep 14 04:22
    raybellwaves commented #196
  • Jul 12 19:23
    pmreyes2 commented #216
epifanio
@epifanio
a bad hack to fix the das .. is to add an exception and try to decode using utf-8 ... which now gave me a working page for the das
but this doesn't fix the pydap.client ... as the error trying to laod such dataset is:
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-1-2bb713f8a88f> in <module>
      1 from pydap.client import open_url
----> 2 dataset = open_url('http://dap.metsis.met.no/SN99938.nc')

/usr/local/lib/python3.7/dist-packages/pydap/client.py in open_url(url, application, session, output_grid, timeout, verify)
     65     """
     66     dataset = DAPHandler(url, application, session, output_grid,
---> 67                          timeout=timeout, verify=verify).dataset
     68 
     69     # attach server-side functions

/usr/local/lib/python3.7/dist-packages/pydap/handlers/dap.py in __init__(self, url, application, session, output_grid, timeout, verify)
     61                 verify=verify)
     62         raise_for_status(r)
---> 63         das = safe_charset_text(r)
     64 
     65         # build the dataset from the DDS and add attributes from the DAS

/usr/local/lib/python3.7/dist-packages/pydap/handlers/dap.py in safe_charset_text(r)
    115     else:
    116         r.charset = get_charset(r)
--> 117         return r.text
    118 
    119 

/usr/local/lib/python3.7/dist-packages/webob/response.py in _text__get(self)
    620         decoding = self.charset or self.default_body_encoding
    621         body = self.body
--> 622         return body.decode(decoding, self.unicode_errors)
    623 
    624     def _text__set(self, value):

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 94: ordinal not in range(128)
epifanio
@epifanio
if i put a print statment in /usr/local/lib/python3.7/dist-packages/webob/response.py line 622
it tels me the decoding is set to ascii
while to work in my case it should be utf-8
decodinf is define few lines above by : decoding = self.charset or self.default_body_encoding
so adding an other try/except to switch to utf-8 ... will work
but this are hack and most important .. this is happening on the client side .. where I have no control on the pydap version used by a potential user
do you have any suggestion?
epifanio
@epifanio
i added an issue on pydap/pydap#196
epifanio
@epifanio
AS the error is not in pydap itself, but in webop, I'm goona close this issue and open a new one on the webop repository
epifanio
@epifanio
but seems that what is failing is the get_charset method in pydap, which set ascii as default if the object r doesn't have a charset property
so, the part of pydap that uses webop is in the GET method implemented in net.py
epifanio
@epifanio
perhaps the safest thing to do is to set the decoding to utf-8 as default in get_charset
epifanio
@epifanio
no, modify get_charset will not work ...
epifanio
@epifanio
@tomkralidis I updated the issue adding more details and a test dataset to reporodce the problem
I tried to manually hardcode the response.charset in the pydap code, but my attempt fails .. when I try to print put the response.charset from the webop call, it still returns ascii
epifanio
@epifanio
Nobody can help with this? I am not ure where the problem is .. in pydap or in the webop library
epifanio
@epifanio
@tomkralidis do you have time to have a quick look into this? I am not sure I should open an issue on webop, as it seems like PyDap is not passing the correct decoding to webop
I tried to manually set the r.charset value to UTF-8 in src/pydap/handlers/dap.py in DAPHandler() and in get_charset() with no luck.
it seems PyDap always send to webop the decode = 'ASCII'
epifanio
@epifanio
I am attempting a build for a debian package for pydap
to do so I first have to build a debian package to replace webop
It will require a patched webop .. as for the current implementation I have the decode bug (see the issue discussed above)
epifanio
@epifanio
I added basic dockerfile to build and serve pydap as wsgi with apache https://github.com/epifanio/pydap/blob/master/Dockerfile the docker image will build two debian packages, one wor webob and one for pydap .. which I'm planning to modify to fix the unicode error
the docker image as it is, reqires only apache to start and then wills erve pydap on localhost:80 it includes the file which is giving me headhack for the unicode issue
the hack to get it working is to add a try/except stament to force use unicode in case of failure with ascii
epifanio
@epifanio
the Docker imagge now uses the fork I made with the 2 line chages. IS it worth a PR?
it is one line in pydap and one more in webob
epifanio
@epifanio
HI
I am having trouble in serving a netcdf dataset using pydap
the file can be loaded correctly using the netCDF4 python module
but pydap gives an internal server when trying to reach the file via its url
the log says:
epifanio
@epifanio

but the key y is present, reading the file directly with python-netcdf4:

https://gist.github.com/epifanio/0ee266eec05672eff43c66088750f248

I can see:
dimensions(sizes): time(1), x(25676), y(19326), gcp_index(231)
do you have any clue on what it can be?
epifanio
@epifanio
interesting to see that dimensions(sizes): time(1), x(25676), y(19326), gcp_index(231) doesn't match the dataset.variables which are:
>>> list(dataset.variables.keys())
['time', 'lat', 'lon', 'Amplitude_HH', 'Amplitude_HV', 'crsWGS84', 'dn_HH', 'sigmaNought_HH', 'gamma_HH', 'betaNought_HH', 'betaNought_HV', 'gamma_HV', 'dn_HV', 'sigmaNought_HV', 'noiseCorrectionMatrix_HH', 'noiseCorrectionMatrix_HV', 'swathList', 'GCP_incidenceAngle_HH', 'GCP_slantRangeTime_HV', 'GCP_azimuthTime_HV', 'GCP_longitude_HH', 'GCP_pixel_HH', 'GCP_elevationAngle_HH', 'GCP_latitude_HH', 'GCP_height_HH', 'GCP_incidenceAngle_HV', 'GCP_longitude_HV', 'GCP_latitude_HV', 'GCP_height_HV', 'GCP_elevationAngle_HV', 'GCP_pixel_HV', 'GCP_azimuthTime_HH', 'GCP_line_HV', 'GCP_slantRangeTime_HH', 'GCP_line_HH', 's1Level1ProductSchema_HH', 's1Level1ProductSchema_HV', 'orbitList_HH', 'swathMergeList_HH', 'coordinateConversionList_HH', 'orbitList_HV', 'swathMergeList_HV', 'coordinateConversionList_HV']
epifanio
@epifanio
I did a simple test, I copied a working netcdf file (correctly served via opendap) and I checked the dimensions and variables. I see it had the dimension time listed also in the dataset.variables
I then copied the file and used ncrename to rename the time variable: ncrename -d time,timex SN99938_dubbed.nc
by doing this the pydap service crashes in serving the directory whre the modified file is stored
epifanio
@epifanio

correct me if I am wrong, but the line:

self.dataset[grid][dim] = BaseType(dim, vars[dim][:]

Assumes that there must be (must exist) variables with the same name of the dimensions found in the dataset

is it correct?
epifanio
@epifanio
seems I am facing this exact issue: pydap/pydap#128
epifanio
@epifanio
I applied the fix proposed by @nathanlcarlson (thanks!) . it works great, sentinel1 data on a silverplate , served via pydap
epifanio
@epifanio
Hi, at work I have been asked to report about PyDap. which I chosed as main provider for handling netcdf data.
I saw some important fix, proposed in a recent PR pydap/pydap#201 didn’t receive any comment. Also the issue I had with handling unicode characters in metadata pydap/pydap#196 (mentioned here in the past) didn’t received any reply/comment.
I wonder if there is a reason, or just luck of man-power ?
epifanio
@epifanio
With just these two fix (handling unicode and dims not in vars) pydap does a great job in serving hundreds of GB for the sentinel daily products. which is cool … but it is hard to sell if those changes are not merged upstream