Interesting--I'll have to see if this helps with the Python interface as well.

I believe that said data must have been written by Astropy, since I found a bug that would cause exactly that: Duplicate EXTNAME values. I guess if Astropy caused the bug we have to eat our own garbage and handle it, so thank you for the fix. Next step is to fix the original issue (I am working on a fix but it's a bit tricky...)

In any case, if your goal is to write signed bytes to a FITS table, you will have to change your approach, because it's not something supported by the FITS format, for some reason :/

Thank you for the report. This is more specifically a lower-level issue about astropy.io.fits than Table or the unified-io interface.

The FITS standard does not allow for signed bytes in table columns. It is simply not one of the supported data types for FITS tables. Only unsigned bytes are supported.

For some historical reason that I can't recall, however, the code interprets an array of signed bytes as a boolean column. I'd have to investigate more to see why that is.

I think, at the very least, we could provide a better warning about this.

@lastephey Thank you for your work on this. I think we can keep your existing changes as they are. They are probably unnecessary with the other changes, but I think they can't hurt either.

However, I think we should change the changelog message, since the main thing being changed here is now much broader: That astropy.io.fits is not using unbuffered I/O by default anymore.

I'm also tempted to deprecate that fileobj_open() call and remove any use of it. I can help with that if you want.

As for tests I don't think any special tests are needed, because if this change keeps all the existing tests passing then that covers existing known issues just fine.

@lastephey

To our knowledge, astropy is the only library where we've encountered this particular issue at NERSC, although that could be a function of how closely we have explored and profiled DESI's code.

This might be because for most applications unbuffered I/O is fairly uncommon and usually enabled for specific applications. So the overhead of very small writes doesn't show up as much.

Anyways, I don't think we need to over worry about this. I don't think we need a test for this apparently obscure (to the extent that no one even knows what causes it, and there is no known issue about it) filesystem issue. The fix I proposed to simply let the fits module used buffered I/O again should be sufficient.

@dhomeier

The role of buffering still remains a bit unclear

The role of buffering is clear to me. If you're using buffered writes the library code handling the buffering (in this case Python's io module) will not perform an actual write() system call until the buffer is filled. Zero-byte writes have no impact because they do not append anything to the buffer.

I'm still not convinced this issue is specific to zero-byte writes which seems very strange to me. From my understanding making any repeated small changes (as well a writing many small files in general) on Lustre can be a bottleneck for it.

Related: https://github.com/numpy/numpy/issues/2504

The following patch should fix this issue, and all the astropy.io.fits tests pass with it (have not run the full test suite yet). I haven't checked if there are other potential drawbacks.

IIRC a long time ago, in the Python 2 days, ndarray.tofile did not work well with the Python io module when it was first introduced, and in particular had problems with buffered I/O. I think those days are in the past but I'm not 100% sure.

diff --git a/astropy/io/fits/util.py b/astropy/io/fits/util.py
index 157b8634f..e829d0e00 100644
--- a/astropy/io/fits/util.py
+++ b/astropy/io/fits/util.py
@@ -380,14 +380,9 @@ def isfile(f):
 def fileobj_open(filename, mode):
     """
     A wrapper around the `open()` builtin.
-
-    This exists because `open()` returns an `io.BufferedReader` by default.
-    This is bad, because `io.BufferedReader` doesn't support random access,
-    which we need in some cases.  We must call open with buffering=0 to get
-    a raw random-access file reader.
     """
 
-    return open(filename, mode, buffering=0)
+    return open(filename, mode)
 
 
 def fileobj_name(f):
@@ -612,7 +607,12 @@ def _array_to_file(arr, outfile):
     `ndarray.tofile`.  Otherwise a slower Python implementation is used.
     """
 
-    if isfile(outfile) and not isinstance(outfile, io.BufferedIOBase):
+    try:
+        seekable = outfile.seekable()
+    except AttributeError:
+        seekable = False
+
+    if isfile(outfile) and seekable:
         write = lambda a, f: a.tofile(f)
     else:
         write = _array_to_file_like

I think maybe the fix from #7850 was too broad. It shouldn't prevent using ndarray.tofile for all BufferedIO objects. Just ones that are not seekable (like sys.stdin).

Changing to buffered I/O invokes a regression specifically on this issue: #2960

Changing to used buffered I/O passes almost all the test, except for a handful of strange failures related to NumPy arrays.

I think the deeper issue here, beyond just the case of zero-byte writes, is that astropy.io.fits is always using unbuffered writes due to the fact that it uses this utility function when opening files (whether for reading or writing): https://github.com/astropy/astropy/blob/189d9daf4f7fa82232df6d3fc3d1560a1e3f888a/astropy/io/fits/util.py#L380

There's some note there about buffered I/O not allowing random access, but I'm not sure if that's still true. I need to check.

If it were using buffered I/O, which is the default, it would not be making these empty write calls at all, which I confirmed with strace. I don't know what this specific issue is with Lustre, but unbuffered I/O in general might be very slow for network/distributred filesystems for small writes. I can't seem to find a specific reference on this in the case of Lustre (and would love to find one) but it makes sense that making many small write operations would create a bottleneck. (What I'd be really interested to find out is if there is an ideal buffer size, but this might require local experimenting, possibly dependent on how the filesystem is configured).

I think, rather than lots of micro-targeted fixes, there needs to be at a minimum an option when writing (and reading) FITS files to allow buffered I/O. That would solve this problem in one fell swoop. The issue about random access is relevant in some cases (updating existing files) but for writing new files I'm not sure how relevant it is.

The Lustre filesystem has a known issue that is triggered when zero-length writes are attempted.

Do you have a reference for this? I've never heard of anything like this. Why is a zero-byte write not a no-op in this case? And why would this only impact the astropy.io.fits code specifically (it sounds like this would be a problem for a lot of software)?

I'd like to study this issue a little more closely. I'm not sure it requires any "Rube Goldberg" scripts being included in the package.

It's not only that--I've been on extended travel lately and haven't had regular access to my Windows machine. Cloud-based virtualization platforms have proven too slow and unreliable in the past so I haven't tried touching it, though maybe things have gotten better in that area. I know @mkoeppe has made significant progress on building Sage for Windows on GitHub's build machines in multiple stages, in order to get around job time limits.

And my work laptop is too low on disk space at the moment to even run a Windows VM. (Part of what's taking up the space is my Sage sources which take up around 30GB ^^).

You can find information about how to build Sage on Cygwin yourself at https://doc.sagemath.org/html/en/installation/source.html#id4

In the meantime, as @williamstein wrote for the time being this is a volunteer project. If I'm not available, someone else with the resources is welcome to step in and help.

The regular reST include directive has support for :start-after: and :end-before: options allowing specific to text to search for for setting the start/end points of the inclusion.

This is extremely useful since it allows including, say, only specific sections of a markdown file, without always having to chase the exact line numbers. Happy to make a PR since I see you're looking for help on this package.

Description

When creating a compressed image HDU, adding a keyword to CompImageHDU.header causes it to be duplicated in the underlying CompImageHDU._header if it already existed in the underlying header.

Steps to Reproduce

This is most easily seen with the EXTNAME keyword since all CompImageHDU headers start with a default of EXTNAME = 'COMPRESSED_IMAGE', though the same can be reproduced by running something like:

>>> chdu = fits.CompImageHDU()
>>> chdu._header['FOO'] = 'A'
>>> chdu.header['FOO'] = 'A"
>>> chdu.header
XTENSION= 'IMAGE   '           / Image extension                                
BITPIX  =                    8 / array data type                                
NAXIS   =                    0 / number of array dimensions                     
PCOUNT  =                    0 / number of parameters                           
GCOUNT  =                    1 / number of groups                               
FOO     = 'A       '                                                            
>>> chdu._header
XTENSION= 'BINTABLE'           / binary table extension                         
BITPIX  =                    8 / array data type                                
NAXIS   =                    2 / number of array dimensions                     
NAXIS1  =                    8 / width of table in bytes                        
NAXIS2  =                    0 / number of rows in table                        
PCOUNT  =                    0 / number of group parameters                     
GCOUNT  =                    1 / number of groups                               
TFIELDS =                    1 / number of fields in each row                   
TTYPE1  = 'COMPRESSED_DATA'    / label for field 1                              
TFORM1  = '1PB     '           / data format of field: variable length array    
ZIMAGE  =                    T / extension contains compressed image            
ZTENSION= 'IMAGE   '           / Image extension                                
ZBITPIX =                    8 / data type of original image                    
ZNAXIS  =                    0 / dimension of original image                    
ZPCOUNT =                    0 / number of parameters                           
ZGCOUNT =                    1 / number of groups                               
ZCMPTYPE= 'RICE_1  '           / compression algorithm                          
ZNAME1  = 'BLOCKSIZE'          / compression block size                         
ZVAL1   =                   32 / pixels per block                               
ZNAME2  = 'BYTEPIX '           / bytes per pixel (1, 2, 4, or 8)                
ZVAL2   =                    1 / bytes per pixel (1, 2, 4, or 8)                
EXTNAME = 'COMPRESSED_IMAGE'   / name of this binary table extension            
FOO     = 'A       '                                                            
FOO     = 'A       '

In the latter, the FOO keyword gets repeated. This is probably the underlying cause of the bug that #11396 was trying to work around (there never should have been multiple EXTNAME keywords in the first place, but it appears to be an artifact of an Astropy bug, which probably existed for some time).

This is another bug caused by the disastrous CompImageHeader class. When assigning a keyword to a header, Header._update() gets called, which in turn calls Header.append() if the keyword is not already in the header. But for CompImageHeader, that means CompImageHeader.append() gets called, which also appends the keyword to the underling self._table_header even if it already existed there.

Not sure if there's a quick workaround.

I'm not sure I understand or agree with how this was handled. I don't think any standard dictates that a compressed image HDU must have an EXTNAME of COMPRESSED_IMAGE by default. It appears to be a convention carried over from CFITSIO.

More importantly, the FITS standard does not have any provision for "multiple EXTNAMEs", and the fact that there were ever multiple EXTNAMEs in a compressed image header is a bug to begin with is it not?

For reference CFITSIO (and imcopy) replace this default EXTNAME if you compress an HDU that already has an EXTNAME. It doesn't create a duplicate.

I guess, if there are FITS files in the wild created this way due to a buggy Astropy/PyFITS, then we should account for that case... :(

But I wonder if the original bug has even been fixed.

It says

The ZSIMPLE, ZEXTEND, and ZBLOCKED keywords may only be used if the original uncompressed image was contained in the primary array of the FITS file.

Which is true in this case. The idea here is, in the case of astropy.io.fits anyways, you could now recreate the original primary HDU that was compressed.

Confirmed bug. I was able to reproduce it like this (it has nothing to do with compression mode or checksums):

>>> data = np.arange(100*100, dtype=np.float64).reshape((100, 100))
>>> phdu = fits.PrimaryHDU(data=data)
>>> chdu = fits.CompImageHDU(data=phdu.data, header=phdu.header)
>>> chdu.writeto('tmp2.fits', overwrite=True)
>>> hdul = fits.open('tmp2.fits')
>>> hdul[1].header
SIMPLE  =                    T / conforms to FITS standard                      
BITPIX  =                  -64 / array data type                                
NAXIS   =                    2 / number of array dimensions                     
NAXIS1  =                  100                                                  
NAXIS2  =                  100                                                  
PCOUNT  =                    0 / number of group parameters                     
GCOUNT  =                    1 / number of groups                               
XTENSION= 'BINTABLE'           / binary table extension                         
EXTEND  =                    T

The issue it seems is that if you create a compressed image HDU from the header from a PRIMARY HDU, it will preserve the SIMPLE keyword as ZSIMPLE. But there's a bug that somehow causes it to include XTENSION in the "uncompressed" header in this case.

I need to double-check exactly what the FITS compression standard has to say about ZSIMPLE...

I see this in Python 3.6 test suites. Is this safe to ignore or should be fixed somehow? Should we move this to astropy-helpers? I am not sure where this is coming from.

home/travis/miniconda/envs/test/lib/python3.6/importlib/_bootstrap.py:205:
ImportWarning: can't resolve package from __spec__ or __package__,
falling back on __name__ and __path__

  return f(*args, **kwds)

p.s. Instead of decreasing the number of issues, I am increasing it. Sorry, @bsipocz !

This is OBE since astropy no longer supports Python 3.6, and I can't reproduce it. There is a similar/maybe related issue at https://github.com/pypa/setuptools/issues/1111 which I can see if I set -Werror::ImportWarning but the issue is already worked around upstream, and is only a problem when I explicitly convert the warning to an error.

I'm not sure if #3783 is the same - but you can't currently fit a CompoundModel:

from astropy.modeling import fitting
from astropy.modeling.polynomial import Polynomial2D
import numpy as np

x, y = np.mgrid[:100,:100]
z = x**2 - x*y + y
model = Polynomial2D(2) + Polynomial2D(2)
fitter = fitting.LinearLSQFitter()

print(fitter(model, x, y, z))

This leads to:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "{my-python-path}/site-packages/astropy/modeling/fitting.py", line 287, in __call__
    lhs = model_copy.fit_deriv(x, y, *model_copy.parameters)
TypeError: 'NoneType' object is not callable

This is a duplicate of the more recent (but more developed) issue #6038. There have also been some band-aids to this issue including documentation updates and better error handling.

I believe the test as currently written is still relevant and makes sense. There are several issues that go into unicode output (especially to terminals) beyond just how well Python supports it. Availability of fonts, what terminal is being used, what encoding / code page the user's environment is configured for, etc. Unfortunately, even in this day and age, it's a bit dicey to assume unicode support in all program output by default.

This could be seen as either a bug or a feature request... :)

I think it should be considered fixable by treating whitespace equivalently to the missing equal sign.

It would be nice to use more modern C standards in extension code. Is there any reason we can't update to use c99? The compilers on travis at least seem to be old enough that providing this flag explicitly is required.

cc @MSeifert04