These are chat archives for astropy/astropy

5th
Jun 2017
matrixbot
@matrixbot
Jun 05 2017 10:28
bsipocz @ddasilva - As Cadair said above, it's not totally clear what you would like to know, so my response may not be the answer you're looking for:
astropy core uses both local tests data stored in the repository, remote data stored on our server, and temporarily generated data. I would suggest you to look at the usage of these in the tests functions, e.g. local data is used in the io.fits and wcs modules while remote ones are primarily in visualization.wcsaxes and a few in astropy.coordinates. Look for the @remote_data decorator. There are a few helpers utility in astropy.utils.data to help deal with test datasets.
matrixbot
@matrixbot
Jun 05 2017 11:20
Cadair ok so can someone tell me if this is a feature or a bug
Cadair I am reading a delerberaly broken FITS file
Cadair (I opened it in a HEX editor and broke it)
Cadair I changed the NAXIS1 key to be the wrong number
Cadair both smaller and larger
Cadair and io.fits will happily read the file anyway with out even a warning
matrixbot
@matrixbot
Jun 05 2017 12:42
astrofrog are there other HDUs in the file?
Cadair n
Cadair no
astrofrog did you make it large enough that it should read beyond the end of the file
Cadair yes
astrofrog then normally it should emit a truncation warning
Cadair I would think
Cadair I will make it silly large
astrofrog let me try here too
astrofrog tried too

astrofrog In [1]: from astropy.io import fits

In [2]: hdu = fits.open('fake.fits')
WARNING: File may have been truncated: actual file length (1045440) is smaller than the expected size (10388160) [astropy.io.fits.file]

Cadair I have seen that before
Cadair so why was it not emitting it
Cadair how did you break it?
astrofrog I just made NAXIS1 10 times bigger
astrofrog can you share the file you are using?
Cadair ok I only made it slightly wrong
astrofrog there is sometimes padding
Cadair test_2.fits
astrofrog so you have to make it quite wrong
Cadair ok that's annoying
Cadair I want it to tell me if it's any level of wrong lol
astrofrog yeah in the hex editor look at the end of the file, there will be a bunch of 0000
astrofrog does does it warn if you make it very wrong at least?
astrofrog FITS is made up of 2880 byte data blocks
matrixbot
@matrixbot
Jun 05 2017 12:47
astrofrog so they will have padding
Cadair bah
astrofrog yeah all my FITS files have a size that is a multple of 2880 bytes
astrofrog what a waste of space
Cadair what about the other way around
Cadair what if you make the header smaller than the data?
astrofrog if there is only one HDU it won't matter
astrofrog if there are multiple HDUs it probably will barf if you try and access HDUs > 0
astrofrog BUT we could potentially check if there is non-zero data in the remainder of the last 2880 byte block that is read
astrofrog as that would indicate an issue
astrofrog io.fits doesn't do that but it could
astrofrog however not sure if there is a requirement padding has to be zeros
astrofrog FITS says '
If the data array does not fill the final data block, the remain- der of the data block shall be filled by setting all bits to zero'
astrofrog so that is one thing that could be checked
astrofrog but isn't currently
astrofrog maybe could open an issue or PR for that?
Cadair sure
Cadair this is actual work so why not
matrixbot
@matrixbot
Jun 05 2017 12:52
Cadair as long as this rabbit hole does not go too deep
astrofrog I mean you are screwed if the size you are reading is a multiple of 2880 because you shouldn't continue to read the next one
astrofrog but at least in the case where product of naxis is not multiple of 2880, can check when reading last block
Cadair ok so that's the next thing I am going to try and check
Cadair what happens if I change the amount of actual data
astrofrog in what sense?
astrofrog I mean if you add fake data at end of FITS file?
astrofrog or if you truncate file without changing naxis?
Cadair both
astrofrog I think the same as what would happen when changing naxis
astrofrog well for a start if you added data that wasn't a multiple of 2880 bytes that's easy to catch
astrofrog so that's something that could be checked, I think all FITS files should be a multiple of 2880
astrofrog if you add n x 2880 bytes they will just be ignored
astrofrog but that should also raise a warning because after the 2880 byte blocks for the data it should either stop or start a new header
astrofrog so if it's just more data bocks can warn
astrofrog and if you remove n x 2880 blocks it should emit a warning as it does at the moment
astrofrog so I think you can catch most invalid cases actually, it just doesnt' do it right now
matrixbot
@matrixbot
Jun 05 2017 12:57
astrofrog The lowest hanging fruit I think is probably checking that the file size is multiple of 2880 since if it's not that's a red flag
astrofrog then the next thing to check is that when reading the last 2880 bytes of a data record, ignored bytes are all zero
astrofrog asdf is much nicer in this respect...
matrixbot
@matrixbot
Jun 05 2017 13:03
astrofrog this padding nonsense is a pain
matrixbot
@matrixbot
Jun 05 2017 13:45
Cadair why the padding?
Cadair I can't really see a reason for it
Cadair is it just so you can chunk the IO?
matrixbot
@matrixbot
Jun 05 2017 14:09
astrofrog yeah I don't know
astrofrog probably 'historical reasons'
astrofrog headers have padding too - I think that's so that you can edit the header without re-writing the file