profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/embray/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.

airspeed-velocity/asv 607

Airspeed Velocity: A simple Python benchmarking tool with web-based reporting

defeo/aws 9

Lecture notes for the M1 course AWS: Applications Web et Securité

embray/adass23-astropy 1

Poster paper on Astropy for ADASS XXIII

embray/astropy 1

My fork of the main Astropy repository (astropy/astropy)

embray/astropy-website 1

Source code for the Astropy website

ccp-codima/2020-01-21-colchester-workshop 0

Tools for Discrete Computational Mathematics (Colchester, 21-23 January 2020)

embray/2012-10-gmu-sandbox 0

Git learning sandbox for George Mason University

embray/aandc-fits 0

Paper on the FITS format for Astronomy and Computing journal

pull request commentoscar-system/GAP.jl

Revise how we handle GAP errors.

Interesting--I'll have to see if this helps with the Python interface as well.

fingolfin

comment created time in 2 days

push eventiguananaut/iguananaut.github.io

E. Madison Bray

commit sha 16f8687533b4865ed99d764cd7072f948855ef42

Manual update since the old build scripts are totally broken.

view details

push time in 5 days

pull request commentastropy/astropy

Support multiple EXTNAME in compressed images

I believe that said data must have been written by Astropy, since I found a bug that would cause exactly that: Duplicate EXTNAME values. I guess if Astropy caused the bug we have to eat our own garbage and handle it, so thank you for the fix. Next step is to fix the original issue (I am working on a fix but it's a bit tricky...)

timj

comment created time in 10 days

issue commentastropy/astropy

astropy.table writes int8 column in boolean in FITS output

In any case, if your goal is to write signed bytes to a FITS table, you will have to change your approach, because it's not something supported by the FITS format, for some reason :/

rongpu

comment created time in 10 days

issue commentastropy/astropy

astropy.table writes int8 column in boolean in FITS output

Thank you for the report. This is more specifically a lower-level issue about astropy.io.fits than Table or the unified-io interface.

The FITS standard does not allow for signed bytes in table columns. It is simply not one of the supported data types for FITS tables. Only unsigned bytes are supported.

For some historical reason that I can't recall, however, the code interprets an array of signed bytes as a boolean column. I'd have to investigate more to see why that is.

I think, at the very least, we could provide a better warning about this.

rongpu

comment created time in 10 days

pull request commentastropy/astropy

WIP: fixing problems caused by the zero-byte writes

@lastephey Thank you for your work on this. I think we can keep your existing changes as they are. They are probably unnecessary with the other changes, but I think they can't hurt either.

However, I think we should change the changelog message, since the main thing being changed here is now much broader: That astropy.io.fits is not using unbuffered I/O by default anymore.

I'm also tempted to deprecate that fileobj_open() call and remove any use of it. I can help with that if you want.

As for tests I don't think any special tests are needed, because if this change keeps all the existing tests passing then that covers existing known issues just fine.

lastephey

comment created time in 10 days

pull request commentastropy/astropy

WIP: fixing problems caused by the zero-byte writes

@lastephey

To our knowledge, astropy is the only library where we've encountered this particular issue at NERSC, although that could be a function of how closely we have explored and profiled DESI's code.

This might be because for most applications unbuffered I/O is fairly uncommon and usually enabled for specific applications. So the overhead of very small writes doesn't show up as much.

lastephey

comment created time in 10 days

pull request commentastropy/astropy

WIP: fixing problems caused by the zero-byte writes

Anyways, I don't think we need to over worry about this. I don't think we need a test for this apparently obscure (to the extent that no one even knows what causes it, and there is no known issue about it) filesystem issue. The fix I proposed to simply let the fits module used buffered I/O again should be sufficient.

lastephey

comment created time in 10 days

pull request commentastropy/astropy

WIP: fixing problems caused by the zero-byte writes

@dhomeier

The role of buffering still remains a bit unclear

The role of buffering is clear to me. If you're using buffered writes the library code handling the buffering (in this case Python's io module) will not perform an actual write() system call until the buffer is filled. Zero-byte writes have no impact because they do not append anything to the buffer.

I'm still not convinced this issue is specific to zero-byte writes which seems very strange to me. From my understanding making any repeated small changes (as well a writing many small files in general) on Lustre can be a bottleneck for it.

lastephey

comment created time in 10 days

pull request commentastropy/astropy

WIP: try to avoid fits io binary table zero-byte writes

Related: https://github.com/numpy/numpy/issues/2504

lastephey

comment created time in 11 days

pull request commentastropy/astropy

WIP: try to avoid fits io binary table zero-byte writes

The following patch should fix this issue, and all the astropy.io.fits tests pass with it (have not run the full test suite yet). I haven't checked if there are other potential drawbacks.

IIRC a long time ago, in the Python 2 days, ndarray.tofile did not work well with the Python io module when it was first introduced, and in particular had problems with buffered I/O. I think those days are in the past but I'm not 100% sure.

diff --git a/astropy/io/fits/util.py b/astropy/io/fits/util.py
index 157b8634f..e829d0e00 100644
--- a/astropy/io/fits/util.py
+++ b/astropy/io/fits/util.py
@@ -380,14 +380,9 @@ def isfile(f):
 def fileobj_open(filename, mode):
     """
     A wrapper around the `open()` builtin.
-
-    This exists because `open()` returns an `io.BufferedReader` by default.
-    This is bad, because `io.BufferedReader` doesn't support random access,
-    which we need in some cases.  We must call open with buffering=0 to get
-    a raw random-access file reader.
     """
 
-    return open(filename, mode, buffering=0)
+    return open(filename, mode)
 
 
 def fileobj_name(f):
@@ -612,7 +607,12 @@ def _array_to_file(arr, outfile):
     `ndarray.tofile`.  Otherwise a slower Python implementation is used.
     """
 
-    if isfile(outfile) and not isinstance(outfile, io.BufferedIOBase):
+    try:
+        seekable = outfile.seekable()
+    except AttributeError:
+        seekable = False
+
+    if isfile(outfile) and seekable:
         write = lambda a, f: a.tofile(f)
     else:
         write = _array_to_file_like
lastephey

comment created time in 11 days

pull request commentastropy/astropy

WIP: try to avoid fits io binary table zero-byte writes

I think maybe the fix from #7850 was too broad. It shouldn't prevent using ndarray.tofile for all BufferedIO objects. Just ones that are not seekable (like sys.stdin).

lastephey

comment created time in 11 days

pull request commentastropy/astropy

WIP: try to avoid fits io binary table zero-byte writes

Changing to buffered I/O invokes a regression specifically on this issue: #2960

lastephey

comment created time in 11 days

pull request commentastropy/astropy

WIP: try to avoid fits io binary table zero-byte writes

Changing to used buffered I/O passes almost all the test, except for a handful of strange failures related to NumPy arrays.

lastephey

comment created time in 11 days

pull request commentastropy/astropy

WIP: try to avoid fits io binary table zero-byte writes

I think the deeper issue here, beyond just the case of zero-byte writes, is that astropy.io.fits is always using unbuffered writes due to the fact that it uses this utility function when opening files (whether for reading or writing): https://github.com/astropy/astropy/blob/189d9daf4f7fa82232df6d3fc3d1560a1e3f888a/astropy/io/fits/util.py#L380

There's some note there about buffered I/O not allowing random access, but I'm not sure if that's still true. I need to check.

If it were using buffered I/O, which is the default, it would not be making these empty write calls at all, which I confirmed with strace. I don't know what this specific issue is with Lustre, but unbuffered I/O in general might be very slow for network/distributred filesystems for small writes. I can't seem to find a specific reference on this in the case of Lustre (and would love to find one) but it makes sense that making many small write operations would create a bottleneck. (What I'd be really interested to find out is if there is an ideal buffer size, but this might require local experimenting, possibly dependent on how the filesystem is configured).

I think, rather than lots of micro-targeted fixes, there needs to be at a minimum an option when writing (and reading) FITS files to allow buffered I/O. That would solve this problem in one fell swoop. The issue about random access is relevant in some cases (updating existing files) but for writing new files I'm not sure how relevant it is.

lastephey

comment created time in 11 days

issue commentastropy/astropy

Avoid filesystem writes when there are no bytes to write.

The Lustre filesystem has a known issue that is triggered when zero-length writes are attempted.

Do you have a reference for this? I've never heard of anything like this. Why is a zero-byte write not a no-op in this case? And why would this only impact the astropy.io.fits code specifically (it sounds like this would be a problem for a lot of software)?

weaverba137

comment created time in 11 days

pull request commentastropy/astropy

WIP: try to avoid fits io binary table zero-byte writes

I'd like to study this issue a little more closely. I'm not sure it requires any "Rube Goldberg" scripts being included in the package.

lastephey

comment created time in 11 days

issue commentsagemath/sage-windows

Sage 9.3

It's not only that--I've been on extended travel lately and haven't had regular access to my Windows machine. Cloud-based virtualization platforms have proven too slow and unreliable in the past so I haven't tried touching it, though maybe things have gotten better in that area. I know @mkoeppe has made significant progress on building Sage for Windows on GitHub's build machines in multiple stages, in order to get around job time limits.

And my work laptop is too low on disk space at the moment to even run a Windows VM. (Part of what's taking up the space is my Sage sources which take up around 30GB ^^).

You can find information about how to build Sage on Cygwin yourself at https://doc.sagemath.org/html/en/installation/source.html#id4

In the meantime, as @williamstein wrote for the time being this is a volunteer project. If I'm not available, someone else with the resources is welcome to step in and help.

alexjbest

comment created time in 16 days

issue openedCrossNox/m2r2

Adding support for start-after and end-before options

The regular reST include directive has support for :start-after: and :end-before: options allowing specific to text to search for for setting the start/end points of the inclusion.

This is extremely useful since it allows including, say, only specific sections of a markdown file, without always having to chase the exact line numbers. Happy to make a PR since I see you're looking for help on this package.

created time in 22 days

issue openedastropy/astropy

Keyword duplication in compressed image headers

Description

When creating a compressed image HDU, adding a keyword to CompImageHDU.header causes it to be duplicated in the underlying CompImageHDU._header if it already existed in the underlying header.

Steps to Reproduce

This is most easily seen with the EXTNAME keyword since all CompImageHDU headers start with a default of EXTNAME = 'COMPRESSED_IMAGE', though the same can be reproduced by running something like:

>>> chdu = fits.CompImageHDU()
>>> chdu._header['FOO'] = 'A'
>>> chdu.header['FOO'] = 'A"
>>> chdu.header
XTENSION= 'IMAGE   '           / Image extension                                
BITPIX  =                    8 / array data type                                
NAXIS   =                    0 / number of array dimensions                     
PCOUNT  =                    0 / number of parameters                           
GCOUNT  =                    1 / number of groups                               
FOO     = 'A       '                                                            
>>> chdu._header
XTENSION= 'BINTABLE'           / binary table extension                         
BITPIX  =                    8 / array data type                                
NAXIS   =                    2 / number of array dimensions                     
NAXIS1  =                    8 / width of table in bytes                        
NAXIS2  =                    0 / number of rows in table                        
PCOUNT  =                    0 / number of group parameters                     
GCOUNT  =                    1 / number of groups                               
TFIELDS =                    1 / number of fields in each row                   
TTYPE1  = 'COMPRESSED_DATA'    / label for field 1                              
TFORM1  = '1PB     '           / data format of field: variable length array    
ZIMAGE  =                    T / extension contains compressed image            
ZTENSION= 'IMAGE   '           / Image extension                                
ZBITPIX =                    8 / data type of original image                    
ZNAXIS  =                    0 / dimension of original image                    
ZPCOUNT =                    0 / number of parameters                           
ZGCOUNT =                    1 / number of groups                               
ZCMPTYPE= 'RICE_1  '           / compression algorithm                          
ZNAME1  = 'BLOCKSIZE'          / compression block size                         
ZVAL1   =                   32 / pixels per block                               
ZNAME2  = 'BYTEPIX '           / bytes per pixel (1, 2, 4, or 8)                
ZVAL2   =                    1 / bytes per pixel (1, 2, 4, or 8)                
EXTNAME = 'COMPRESSED_IMAGE'   / name of this binary table extension            
FOO     = 'A       '                                                            
FOO     = 'A       '

In the latter, the FOO keyword gets repeated. This is probably the underlying cause of the bug that #11396 was trying to work around (there never should have been multiple EXTNAME keywords in the first place, but it appears to be an artifact of an Astropy bug, which probably existed for some time).

This is another bug caused by the disastrous CompImageHeader class. When assigning a keyword to a header, Header._update() gets called, which in turn calls Header.append() if the keyword is not already in the header. But for CompImageHeader, that means CompImageHeader.append() gets called, which also appends the keyword to the underling self._table_header even if it already existed there.

Not sure if there's a quick workaround.

created time in 23 days

pull request commentastropy/astropy

Support multiple EXTNAME in compressed images

I'm not sure I understand or agree with how this was handled. I don't think any standard dictates that a compressed image HDU must have an EXTNAME of COMPRESSED_IMAGE by default. It appears to be a convention carried over from CFITSIO.

More importantly, the FITS standard does not have any provision for "multiple EXTNAMEs", and the fact that there were ever multiple EXTNAMEs in a compressed image header is a bug to begin with is it not?

For reference CFITSIO (and imcopy) replace this default EXTNAME if you compress an HDU that already has an EXTNAME. It doesn't create a duplicate.

I guess, if there are FITS files in the wild created this way due to a buggy Astropy/PyFITS, then we should account for that case... :(

But I wonder if the original bug has even been fixed.

timj

comment created time in 23 days

issue commentastropy/astropy

astropy.io.fits translates compressed HDU keywords into uncompressed HDU keywords in inconsistent ways.

It says

The ZSIMPLE, ZEXTEND, and ZBLOCKED keywords may only be used if the original uncompressed image was contained in the primary array of the FITS file.

Which is true in this case. The idea here is, in the case of astropy.io.fits anyways, you could now recreate the original primary HDU that was compressed.

weaverba137

comment created time in 23 days

issue commentastropy/astropy

astropy.io.fits translates compressed HDU keywords into uncompressed HDU keywords in inconsistent ways.

Confirmed bug. I was able to reproduce it like this (it has nothing to do with compression mode or checksums):

>>> data = np.arange(100*100, dtype=np.float64).reshape((100, 100))
>>> phdu = fits.PrimaryHDU(data=data)
>>> chdu = fits.CompImageHDU(data=phdu.data, header=phdu.header)
>>> chdu.writeto('tmp2.fits', overwrite=True)
>>> hdul = fits.open('tmp2.fits')
>>> hdul[1].header
SIMPLE  =                    T / conforms to FITS standard                      
BITPIX  =                  -64 / array data type                                
NAXIS   =                    2 / number of array dimensions                     
NAXIS1  =                  100                                                  
NAXIS2  =                  100                                                  
PCOUNT  =                    0 / number of group parameters                     
GCOUNT  =                    1 / number of groups                               
XTENSION= 'BINTABLE'           / binary table extension                         
EXTEND  =                    T

The issue it seems is that if you create a compressed image HDU from the header from a PRIMARY HDU, it will preserve the SIMPLE keyword as ZSIMPLE. But there's a bug that somehow causes it to include XTENSION in the "uncompressed" header in this case.

I need to double-check exactly what the FITS compression standard has to say about ZSIMPLE...

weaverba137

comment created time in 23 days

issue closedastropy/astropy

ImportError in Python 3.6 importlib

I see this in Python 3.6 test suites. Is this safe to ignore or should be fixed somehow? Should we move this to astropy-helpers? I am not sure where this is coming from.

home/travis/miniconda/envs/test/lib/python3.6/importlib/_bootstrap.py:205:
ImportWarning: can't resolve package from __spec__ or __package__,
falling back on __name__ and __path__

  return f(*args, **kwds)

p.s. Instead of decreasing the number of issues, I am increasing it. Sorry, @bsipocz !

closed time in 23 days

pllim

issue commentastropy/astropy

ImportError in Python 3.6 importlib

This is OBE since astropy no longer supports Python 3.6, and I can't reproduce it. There is a similar/maybe related issue at https://github.com/pypa/setuptools/issues/1111 which I can see if I set -Werror::ImportWarning but the issue is already worked around upstream, and is only a problem when I explicitly convert the warning to an error.

pllim

comment created time in 23 days

issue closedastropy/astropy

Cannot fit to compound model

I'm not sure if #3783 is the same - but you can't currently fit a CompoundModel:

from astropy.modeling import fitting
from astropy.modeling.polynomial import Polynomial2D
import numpy as np

x, y = np.mgrid[:100,:100]
z = x**2 - x*y + y
model = Polynomial2D(2) + Polynomial2D(2)
fitter = fitting.LinearLSQFitter()

print(fitter(model, x, y, z))

This leads to:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "{my-python-path}/site-packages/astropy/modeling/fitting.py", line 287, in __call__
    lhs = model_copy.fit_deriv(x, y, *model_copy.parameters)
TypeError: 'NoneType' object is not callable

closed time in 23 days

gkanarek

issue commentastropy/astropy

Cannot fit to compound model

This is a duplicate of the more recent (but more developed) issue #6038. There have also been some band-aids to this issue including documentation updates and better error handling.

gkanarek

comment created time in 23 days

issue commentastropy/astropy

Unicode guidelines update ?

I believe the test as currently written is still relevant and makes sense. There are several issues that go into unicode output (especially to terminals) beyond just how well Python supports it. Availability of fonts, what terminal is being used, what encoding / code page the user's environment is configured for, etc. Unfortunately, even in this day and age, it's a bit dicey to assume unicode support in all program output by default.

saimn

comment created time in 23 days

issue commentastropy/astropy

Is a card without equals character fixable?

This could be seen as either a bug or a feature request... :)

I think it should be considered fixable by treating whitespace equivalently to the missing equal sign.

MSeifert04

comment created time in 23 days

issue closedastropy/astropy

Compiling C extensions with --std=c99

It would be nice to use more modern C standards in extension code. Is there any reason we can't update to use c99? The compilers on travis at least seem to be old enough that providing this flag explicitly is required.

cc @MSeifert04

closed time in 23 days

drdavella