profile
viewpoint

computationalmodelling/nbval 313

A py.test plugin to validate Jupyter notebooks

jupyter/terminado 256

Terminals served by tornado websockets

jupyter/nbmanager 44

View and stop running IPython notebook servers

computationalmodelling/fidimag 37

FInite DIfference microMAGnetic code, based on Python, Cython and C

European-XFEL/h5glance 21

Explore HDF5 files in terminal & HTML views

jiffyclub/2015-07-06-scipy 14

Software Carpentry Workshop at SciPy 2015

ivanov/kanten 12

the enlightened pager. less paging. more content. read widely.

computationalmodelling/python-package-template 10

Attempt to provide a good-practice template for Python packages

batis-installer/batis 6

Install and distribute desktop applications on Linux

issue commenttakluyver/flit

Implicit namespace package support

There's been a PR (#159) hanging around for years; the idea would be that you package one module within the namespace, and flit reads the metadata from that, e.g. sphinxext.opengraph.

I'm just not really interested in namespace packages, so I find it hard to summon the enthusiasm for thinking about how to support them and hearing about bug reports when things go wrong. Quoting myself from that PR:

I ~never use namespace packages myself; my experiences with them have always been that they're an unnecessary complication, creating new ways for things to go wrong for little benefit. So I don't have much motivation to do all the testing to satisfy myself that this will work properly. And I don't have much enthusiasm for maintaining it if it doesn't work.

uranusjr

comment created time in a day

PR opened European-XFEL/EXtra-geom

Methods to retrieve quadrant positions and write XFEL HDF5 geometry files

More ways of getting geometry out. These are closely linked because the XFEL geometry files store offsets from quadrant positions. So to get quadrant positions compatible with an XFEL file, you need to either create both together, or read the file when creating quadrant positions. This will be used in geoAssembler.

I'm aiming for consistency within detector classes more than betweeen them, so quadrant positions for LPD & DSSC are returned in mm (like XFEL geometry files), but quadrant positions for AGIPD are in pixel units (like CrystFEL geometry files). Not ideal, but I figure that it's more important to easily roundtrip each geometry to/from files & quadrant positions than it is to compare geometries for different detectors.

I started writing an internal abstraction over the XFEL HDF5 geometry format, but the generalisation was adding much more code than it was saving, so I abandoned it for now.

Closes #18 Closes #20

+275 -4

0 comment

7 changed files

pr created time in 2 days

push eventEuropean-XFEL/EXtra-geom

Thomas Kluyver

commit sha dac192fd5b42595913c014417cfa1f33736b076e

Retrieve quadrant positions from AGIPD geometry

view details

push time in 2 days

pull request commenth5py/h5py

WIP: building wheels on Azure pipelines

I've got an effective template for testing manylinux wheels generated this way. So far, it's only testing the Python 3.8 wheel, but it shouldn't be hard to expand it to cover the other options.

Open questions:

  • Why does the Mac build fail? It manages to build h5py within tox to run the tests, but then when cibuildwheel tries to run, it fails with dlopen(libhdf5.dylib, 6): image not found
  • How can we bundle HDF5 into the Windows wheels?
  • Azure pipelines has various organisational mechanisms. At present, I'm using jobs with dependencies (so the wheel testing job can't start until after the wheel building job has finished). But it might be more elegant to do it another way, such as:
    • Making these separate steps in the same job
    • Defining build & test 'stages'. Stages contain jobs, so I think this means that all build jobs would have to finish successfully before test jobs started. This could be useful if we organised it similarly for all platforms - make wheels and then use them for tests.
takluyver

comment created time in 2 days

push eventtakluyver/h5py

Thomas Kluyver

commit sha 3170011965fb1830cb07aa25bcad5165a7b91bfa

Checkout repo to get tox.ini config file

view details

push time in 2 days

push eventtakluyver/h5py

Thomas Kluyver

commit sha 5cd79ad88ad3514b360549c0925f1532bf0e21a9

Skip apt step when HDF5_DIR is set

view details

push time in 2 days

push eventtakluyver/h5py

Thomas Kluyver

commit sha d511e65934d6eb4eabfa5ba55c6fe5da41987dcc

Don't install HDF5 for static checks

view details

Thomas Kluyver

commit sha 463464b2687838f0cb74ee7bc8064b64acf3bb1d

Shorter names for package manager steps

view details

Thomas Kluyver

commit sha 9359af88f66d9a7824db6eaaba75a94f17994bc2

Fix spelling of tox --installpkg option

view details

push time in 2 days

push eventtakluyver/h5py

Thomas Kluyver

commit sha be48c4740ba246bee6b6c7fdb08010704835bbfe

Fix YAML

view details

push time in 2 days

push eventtakluyver/h5py

Thomas Kluyver

commit sha c3e387ee6b1ec3c9b4f999264c679a8955099008

Try to run tox with manylinux wheel

view details

push time in 2 days

push eventtakluyver/h5py

Thomas Kluyver

commit sha 43c2bd6a7c27ef34315df78ebe5507f0739ea55f

Download wheels to working directory

view details

push time in 2 days

push eventtakluyver/h5py

Thomas Kluyver

commit sha f0255eaae4473c817e53285e3c3bce122efb33b5

On this one it's called artifactName

view details

push time in 2 days

push eventtakluyver/h5py

Thomas Kluyver

commit sha f52514f6534e8ef6b3269f68ec70ed924a4fd959

Back to build artifacts

view details

push time in 2 days

push eventtakluyver/h5py

Thomas Kluyver

commit sha 951217ce838a1652d9fe2aac2339bd13b11a7ed3

Of course there are different kinds of artifacts

view details

push time in 2 days

push eventtakluyver/h5py

Thomas Kluyver

commit sha a2234abf624e9a1497ceaa7f26d188f676abee33

Redownload manylinux wheels ready to test

view details

push time in 2 days

push eventh5py/hdf5-manylinux

Thomas Kluyver

commit sha 877c74655107bd2caa721fd4df67e6d88327a685

Use public link in README

view details

push time in 2 days

pull request commenth5py/h5py

WIP: building wheels on Azure pipelines

Manylinux + HDF5 Docker images are now automatically built on DockerHub: https://hub.docker.com/repository/docker/h5py/manylinux2010_x86_64-hdf5/general

I've created an h5py org there, but apparently free organisations are only allowed up to 3 members. We could use another service instead - e.g. quay.io, where the standard manylinux images are hosted.

takluyver

comment created time in 2 days

push eventtakluyver/h5py

Thomas Kluyver

commit sha 1311de58f002ddc69ed5a7d9ebc26d0f599fc763

Fix Docker repository name

view details

push time in 2 days

push eventtakluyver/h5py

Thomas Kluyver

commit sha 2d3eae2d2e7804fff5a69fb080e1b34f68cecb83

Use Docker image from h5py DockerHub

view details

push time in 2 days

push eventh5py/hdf5-manylinux

Thomas Kluyver

commit sha 54dd56bc64d94d65475462297c8e3ecc1fd2414f

Add README

view details

Thomas Kluyver

commit sha 76f897913d93cbe4804b946948b1df3410169b31

Don't run yum update

view details

push time in 3 days

push eventh5py/h5py

Doctr (Travis CI)

commit sha 592189d6157aeceb26667bcefa3337a55ed7bd84

Update docs after building Travis build 2331 of h5py/h5py The docs were built from the branch 'master' against the commit 3299da366ef184f05de0cfbb865a6c003a416f56. The Travis build that generated this commit is at https://travis-ci.org/h5py/h5py/jobs/728275147. The doctr command that was run is /home/travis/virtualenv/python3.8.0/bin/doctr deploy . --built-docs docs_api/_build/html

view details

push time in 3 days

delete branch h5py/h5py

delete branch : cleanup-np-tostring

delete time in 3 days

push eventh5py/h5py

Thomas Kluyver

commit sha e9e9e9a64a3da309040167102a59c5e8edcf8add

Replace a couple more arr.tostring with tobytes These must have been overlooked before.

view details

Thomas Kluyver

commit sha 24af808e9be028d0a93e5b389579b377308e9c75

Replace tostring with tobytes in doc example

view details

Thomas Kluyver

commit sha 3299da366ef184f05de0cfbb865a6c003a416f56

Merge pull request #1675 from h5py/cleanup-np-tostring Replace a couple more arr.tostring with tobytes

view details

push time in 3 days

PR merged h5py/h5py

Replace a couple more arr.tostring with tobytes

These must have been overlooked before. I noticed the deprecation warnings while running the tests.

+3 -3

4 comments

2 changed files

takluyver

pr closed time in 3 days

pull request commenth5py/h5py

Replace a couple more arr.tostring with tobytes

Merging this one straight away, as it's a trivial fix and it's passed CI.

takluyver

comment created time in 3 days

create barnchh5py/hdf5-manylinux

branch : master

created branch time in 3 days

created repositoryh5py/hdf5-manylinux

Adding HDF5 to manylinux images for building Linux packages

created time in 3 days

pull request commenth5py/h5py

WIP: building wheels on Azure pipelines

  • Linux: working and including gzip (actually zlib) compression now (files here)
    • [ ] Automate the creation of the Docker image.
    • [ ] Test the wheels produced by this machinery
  • Mac: Failing to find libhdf5 for the build.
  • Windows: Building successfully, but wheels are suspiciously small. I suspect it's not bundling HDF5 into the wheel. I don't know of any Windows equivalent of auditwheel/delocate to bundle DLLs.
takluyver

comment created time in 3 days

pull request commenth5py/h5py

Replace a couple more arr.tostring with tobytes

Thanks, done!

takluyver

comment created time in 3 days

push eventh5py/h5py

Thomas Kluyver

commit sha 24af808e9be028d0a93e5b389579b377308e9c75

Replace tostring with tobytes in doc example

view details

push time in 3 days

PR opened h5py/h5py

Replace a couple more arr.tostring with tobytes

These must have been overlooked before. I noticed the deprecation warnings while running the tests.

+2 -2

0 comment

1 changed file

pr created time in 3 days

create barnchh5py/h5py

branch : cleanup-np-tostring

created branch time in 3 days

issue closedh5py/h5py

hd5f to access to nested structures in *.mat files

General information

  • Operating System (e.g. Windows 10, MacOS 10.11, Ubuntu 16.04.2 LTS, CentOS 7): Windows 10
  • Python version (e.g. 2.7, 3.5): 3.7.3
  • Where Python was acquired (e.g. system Python on MacOS or Linux, Anaconda on Windows): Anaconda
  • h5py version (e.g. 2.6): 2.9
  • HDF5 version (e.g. 1.8.17) 1.10.4
  • The full traceback/stack trace shown (if it appears)

I getting access to the following structure of matlab file (*.mat): image

which contains the following set of structures:

image

I try to access to the fields inside each structure using hd5f python lib

image

using the following code:

mat_dict = h5py.File(file_path, 'r')
result = mat_dict['JKPRR']['results']['ChangingKnob'][0]['result']

But apparently I can't go deep from:

result = mat_dict['JKPRR']['results']['ChangingKnob'][0]

Do you know how I can go to the deeper fields?

closed time in 3 days

JControlReserch

issue commenth5py/h5py

hd5f to access to nested structures in *.mat files

Closing this as it looks (from the :+1:) like it's resolved. We can continue the discussion or reopen it if needed.

To summarise for other people looking at this: Matlab's .mat files do appear to be HDF5, but they might be oddly structured and require some fiddling to work out how to get the data you want. You may want to use a HDF viewer tool like HDFview or h5glance to inspect it.

JControlReserch

comment created time in 3 days

pull request commenth5py/h5py

Bump minimum numpy on Python 3.7 to 1.14.5

Thanks @kif

takluyver

comment created time in 3 days

delete branch takluyver/h5py

delete branch : py37-np-1.14.5

delete time in 3 days

push eventtakluyver/h5py

Thomas Kluyver

commit sha b798467056c53ee4b9d7f49b78540751caaf2afa

Build h5py wheels in Docker file with pre-built HDF5

view details

push time in 3 days

pull request commenth5py/h5py

WIP: building wheels on Azure pipelines

I have the Linux builds creating & publishing wheels as Azure pipeline artifacts. The wheels currently lack gzip support. In principle, Mac & Windows should be easier, because you don't need to build in a Docker container. In practice, there are probably some obstacles.

Building Linux wheels is currently quite slow, because it has to build HDF5 in the Docker container each time before building h5py. I think that the way to get round this is to create our own Docker image (manylinux + HDF5), publish it on Dockerhub, and then tell cibuildwheel to use that.

takluyver

comment created time in 3 days

pull request commenth5py/h5py

Bump minimum numpy on Python 3.7 to 1.14.5

This also makes Python 3.7 CI builds quicker on Linux & Windows, because the mindeps tox environment doesn't need to build numpy from source. It doesn't appear to make much difference to the Mac build, for some reason.

takluyver

comment created time in 3 days

push eventtakluyver/h5py

Thomas Kluyver

commit sha 6dd54ff16d5bccd4e7e3666427a54865d95bc43b

Try it on Windows too, just for fun

view details

push time in 3 days

push eventtakluyver/h5py

Thomas Kluyver

commit sha 84dabd5760c52697f79b87f391adf60e8da86f14

Bump numpy version for Python 3.7 in Tox config as well

view details

push time in 3 days

push eventtakluyver/h5py

Thomas Kluyver

commit sha 50c16b3d3160e79432a867c8749dfe716ed6ea20

Remove empty docker file

view details

Thomas Kluyver

commit sha 5d1208185459429968a4474dee770bd80350786b

Try building wheel on Mac OS too

view details

push time in 3 days

push eventtakluyver/h5py

Thomas Kluyver

commit sha 7f4ab1407a918271647310fb2f8b9d7f3fa572fc

Download HDF5 with curl instead of wget

view details

push time in 3 days

push eventtakluyver/h5py

Thomas Kluyver

commit sha f2dc1a0c86d200feeaffe435360ba99d9291b154

Incorporate HDF5 settings into CIBW_BEFORE_BUILD command

view details

push time in 3 days

push eventtakluyver/h5py

Thomas Kluyver

commit sha 83f366d713c3a8074ba31587447086a103ea601f

Fix Azure job name

view details

push time in 3 days

push eventtakluyver/h5py

Thomas Kluyver

commit sha e4a2e3a205ad3c4196095a588071aecb487cdf25

Build wheels in separate CI job

view details

push time in 3 days

push eventtakluyver/h5py

Thomas Kluyver

commit sha d1ff41ac2ce7a517133494ea1b2ffaaaf71bc9c7

only try to publish things when there are things to publish

view details

Thomas Kluyver

commit sha 3be5ed1a276bb6fab3b1b64dd434b6c603c5a174

Show CIBW identifiers as a separate step before building

view details

push time in 3 days

push eventtakluyver/h5py

Thomas Kluyver

commit sha f934192d925b288e677c4e34b39fd808c17b5ce3

Specify targets for cibuildwheel to build for

view details

push time in 3 days

PR opened h5py/h5py

WIP: building wheels on Azure pipelines ci

Using cibuildwheel

I can't believe it's really this simple, but... let's see what happens.

+6 -1

0 comment

1 changed file

pr created time in 3 days

create barnchtakluyver/h5py

branch : azure-wheels

created branch time in 3 days

issue openedh5py/h5py

Release 3.0

I think we're nearly ready to release h5py 3.0. What needs to be done?

  • [ ] Remaining pull requests to merge (please add any you think need to be in 3.0):
    • [ ] #1666
    • [ ] #1663
  • [ ] Generate release notes with rever & tidy up manually.
  • [ ] Tag release on Github
  • [ ] Generate wheels
    • [ ] Do we need to solve the uploading issues from https://github.com/MacPython/h5py-wheels/issues/11 ? Is this now easier if we generate wheels as part of our own CI, rather than in a separate repository?
    • [ ] Are we still relying on Christoph Gohlke for Windows wheels?
  • [ ] Upload wheels & sdist to PyPI

@h5py/team-h5py

created time in 3 days

issue openedh5py/h5py

h5py.h5ds functions missing docstrings

Several of the low-level functions in h5ds are missing docstrings: https://api.h5py.org/h5ds.html

For low-level functions, the docstring also describes the function signature (parameters and return values). If tackling this, please look at other low-level functions (h5py.h5* modules) to see the expected format.

Once this is done, we can remove the :undoc-members: option for Sphinx autodoc (which includes functions without a docstring).

created time in 3 days

push eventh5py/h5py

Doctr (Travis CI)

commit sha 9bd0c73d3761a90a3a234da19fd258187f022185

Update docs after building Travis build 2313 of h5py/h5py The docs were built from the branch 'master' against the commit 4e00f7977ae7eff4867b3eccce72b414cbae1f73. The Travis build that generated this commit is at https://travis-ci.org/h5py/h5py/jobs/727997296. The doctr command that was run is /home/travis/virtualenv/python3.8.0/bin/doctr deploy . --built-docs docs_api/_build/html

view details

push time in 3 days

delete branch takluyver/h5py

delete branch : rm-pavement

delete time in 3 days

push eventh5py/h5py

Thomas Kluyver

commit sha ba033fe0d1ee55b7f4df54c7a384ced284745196

Remove unused pavement.py file for Paver task runner

view details

Thomas Kluyver

commit sha 86356bd6292d40324b432a203951e8c88f35da95

Remove outdated windows build stuff

view details

Thomas Kluyver

commit sha 7b97901790bf2baadf80f02da7b5a3c8575dea5c

Remove obsolete build.cmd wrapper for 64-bit Python 3.4 The CI jobs actually needing this were removed in pull request gh-1296

view details

Thomas Kluyver

commit sha 17fa235f66a81f42db2961e2a16d74cc97d100ec

Drop removed files from MANIFEST.in

view details

Thomas Kluyver

commit sha 85a0b5eae9d41422fae9f5442a3f0879fa4613a2

Merge pull request #1669 from takluyver/rm-pavement Remove pavement.py & other outdated files related to building on Windows

view details

push time in 3 days

PR merged h5py/h5py

Remove pavement.py & other outdated files related to building on Windows

Closes #1664.

+2 -480

2 comments

9 changed files

takluyver

pr closed time in 3 days

issue closedh5py/h5py

Remove pavement.py files?

It looks like this file was once used to do releases, but I don't think it's been used recently. It hasn't been updated to create wheels, for instance. Even the tool it's written for - Paver - doesn't seem to have been updated for a couple of years.

There's also windows/pavement.py, which is meant to help with building on Windows. But that's only had a few cleanup PRs since 2014, so I suspect it's more of a distraction than a help by now.

Is anyone still using either of those files, or can we clean them up?

closed time in 3 days

takluyver

push eventh5py/h5py

Chen

commit sha 2195502916b7afae9654c62dd29a38e6aca9a9e8

add missing module h5ds in low-level API reference

view details

Chen

commit sha 92c736fe691cac8df0bac65989456e3705661e2f

fix h5fd.rst

view details

Thomas Kluyver

commit sha 2fe4ff0541a2aeb77feca63c565184f06dae510b

Document functions in h5py.h5ds once

view details

Thomas Kluyver

commit sha 09e845a20686a66278c16ded7186bdad7f30f7c7

Fix subheadings for H5FD constants

view details

Thomas Kluyver

commit sha 4e00f7977ae7eff4867b3eccce72b414cbae1f73

Merge pull request #1671 from dota17/new-module Add missing module h5ds in low-level API reference

view details

push time in 3 days

PR merged h5py/h5py

Add missing module h5ds in low-level API reference documentation

Fix https://github.com/h5py/h5py/issues/1667

+19 -2

2 comments

3 changed files

dota17

pr closed time in 3 days

issue closedh5py/h5py

Missing module H5DS in low-level API Reference.

In h5py low-level API Reference, there is no module H5E and module H5DS, I don't know if this is intentional.

closed time in 3 days

dota17

push eventdota17/h5py

Thomas Kluyver

commit sha 2fe4ff0541a2aeb77feca63c565184f06dae510b

Document functions in h5py.h5ds once

view details

Thomas Kluyver

commit sha 09e845a20686a66278c16ded7186bdad7f30f7c7

Fix subheadings for H5FD constants

view details

push time in 3 days

Pull request review commenth5py/h5py

Add missing module h5ds in low-level API reference

+Module H5DS+===========++.. automodule:: h5py.h5ds+    :members:

The :members: option tries to find and document functions inside the module automatically - so Sphinx is complaining because some of the functions get documented twice.

I've resolved this by deleting the list below, and adding :undoc-members: here to include the functions which don't yet have a docstring.

dota17

comment created time in 3 days

PullRequestReviewEvent

push eventh5py/h5py

Chen

commit sha 2d13136566b9f1fb0092e18fbf1477835fd16bfa

import mpi4py.MPI

view details

Thomas Kluyver

commit sha b56b76f1f9d3f57d18d0c9c4ffa7eeb8ad110e3a

Merge pull request #1670 from dota17/MPI Import mpi4py.MPI instead of mpi4py when using 'mpio' driver mode

view details

push time in 3 days

PR merged h5py/h5py

Import mpi4py.MPI instead of mpi4py when using 'mpio' driver mode

Close https://github.com/h5py/h5py/issues/1489

+1 -1

2 comments

1 changed file

dota17

pr closed time in 3 days

issue closedh5py/h5py

h5py does not import 'mpi4py.MPI'

To assist reproducing bugs, please include the following: h5py 2.9.0 HDF5 1.10.4 Python 3.6.5 | packaged by conda-forge | (default, Apr 6 2018, 13:39:56) [GCC 4.8.2 20140120 (Red Hat 4.8.2-15)] sys.platform linux sys.maxsize 9223372036854775807 numpy 1.17.3

When opening an HDF5-file using h5py.File('test.hdf5', 'w', driver='mpio'), it seems that h5py assumes that mpi4py.MPI has already been imported (as importing just mpi4py will not import it because of the way the package works). Therefore, executing the above, even with mpi4py installed, will crash, as this line attempts to access mpi4py.MPI.

I know that you can theoretically speaking assume that mpi4py.MPI is available when someone uses the 'mpio' driver, as they have to import mpi4py.MPI in order to initialize the MPI processes, but this is still a bug in my opinion.

closed time in 3 days

1313e

pull request commenth5py/h5py

Import mpi4py.MPI instead of mpi4py when using 'mpio' driver mode

Thanks, this looks good.

dota17

comment created time in 3 days

push eventh5py/h5py

Doctr (Travis CI)

commit sha a2996aa5fb150dbc73015880816b27c87d2d42cc

Update docs after building Travis build 2310 of h5py/h5py The docs were built from the branch 'master' against the commit 2ff6bce7c7ed776e1d1d3915983c1ee4e40ee9cd. The Travis build that generated this commit is at https://travis-ci.org/h5py/h5py/jobs/727973585. The doctr command that was run is /home/travis/virtualenv/python3.8.0/bin/doctr deploy . --built-docs docs_api/_build/html

view details

push time in 3 days

pull request commenth5py/h5py

Make arrays contiguous prior to H5T conversion

Thanks @cydanil !

cydanil

comment created time in 3 days

push eventh5py/h5py

Cyril Danilevski

commit sha 5f4d6568903a902de4c7c728254365b63f9f0841

Make arrays contiguous prior to H5T conversion Make use of numpy.ascontiguous on arrays fed to a dataset to ensure that slices are correctly stored

view details

Thomas Kluyver

commit sha ee5f6e7feae3507e4e24cf8d548268a0c2a600c5

Avoid extra memory copy in ndarray to vlen conversion * Avoid extra memory copy when converting non-contiguous ndarray to vlen * Reformat Cython import * Remove unused import again

view details

Cyril Danilevski

commit sha e1596bb3c6b7d5793488fb47da899c3e6ad0a0b8

Move non contiguous test to TestVlen

view details

Cyril Danilevski

commit sha dedf191d0e9c0822d48cb4036758ce831858c9de

Add news entry regarding non-contiguous arrays storage

view details

Cyril Danilevski

commit sha 5fac6965ad637504d1ef01ef9d954668a1476472

Add news entry regarding non-contiguous arrays storage

view details

Thomas Kluyver

commit sha 2ff6bce7c7ed776e1d1d3915983c1ee4e40ee9cd

Merge pull request #1657 from cydanil/fix/sliced_arrays Make arrays contiguous prior to H5T conversion

view details

push time in 3 days

PR merged h5py/h5py

Make arrays contiguous prior to H5T conversion bug vlen

This PR aims to close #1649, by forcing all arrays fed to conv_ndarray2vlen to be contiguous, by using numpy.ascontiguousarray.

I unsuccessfully tried to make use of PyBuffer_ToContiguous so (with appropriate imports from cpython.buffer):

cdef int conv_ndarray2vlen(void* ipt,
                           void* opt,
                           TypeID intype,
                           TypeID outtype) except -1:
    cdef:
        PyObject** buf_obj = <PyObject**>ipt
        vlen_t* in_vlen = <vlen_t*>opt
        void* data
        cnp.ndarray ndarray
        size_t len, nbytes
        PyObject* buf_obj0
        Py_buffer view
        Py_UCS4 contiguity = u'C'

    buf_obj0 = buf_obj[0]
    ndarray = np.ascontiguousarray(<cnp.ndarray> buf_obj0)
    len = ndarray.shape[0]
    nbytes = len * max(outtype.get_size(), intype.get_size())

    data = emalloc(nbytes)

    PyObject_GetBuffer(ndarray.data, &view, PyBUF_SIMPLE)
    PyBuffer_ToContiguous(data, &view, view.len, contiguity)
    PyBuffer_Release(&view)

    H5Tconvert(intype.id, outtype.id, len, data, NULL, H5P_DEFAULT)

    in_vlen[0].len = len
    in_vlen[0].ptr = data

    return 0

However, it does not do the job correctly:

AssertionError: [ True False False False] != [ True  True False False]
E       assert False
E        +  where False = all(array([ True, False, False, False]) == array([ True,  True, False, False])
E           +array([ True, False, False, False])
E           -array([ True,  True, False, False])
E           Full diff:
E           - array([ True,  True, False, False])
E           ?              -------
E           + array([ True, False, False, False])
E           ?              +++++++)

So I used numpy.ascontiguousarray, as shown in the code. I'm not too familiar with the trade-offs of using Python imports in Cython. Could I get an opinion?

Thanks! Cyril

+53 -1

7 comments

3 changed files

cydanil

pr closed time in 3 days

issue closedh5py/h5py

Writing sliced NumPy array stores wrong values

Environment

  • Operating System: Ubuntu 18.04.4 LTS
  • Python version: 3.8.3
  • Where Python was acquired: Anaconda on Linux
  • h5py version: 2.10.0
  • HDF5 version: 1.10.4

When writing a sliced NumPy array to a dataset, the value stored in the dataset does not match the value being written unless .copy() is called on the sliced array first.

import h5py
import numpy as np

f = h5py.File('/tmp/data.hdf5', 'w')
f.create_dataset('x', (10,), dtype=h5py.vlen_dtype('bool'))
x = np.array([True, False, True, True, False, False, False])
print(x[::2]) # [ True  True False False]
f['x'][0] = x[::2]
print(f['x'][0]) # [ True False  True  True]
f['x'][0] = x[::2].copy()
print(f['x'][0]) # [ True  True False False]

closed time in 3 days

KurtAhn

issue commenttakluyver/wheel2conda

How to run the program in linux

I think it technically worked well enough for pure-Python wheels. I didn't try to make it work for packages with compiled libraries, and that would probably be a much tougher nut to crack - I can expand on why if that's what you're thinking of.

One issue I do recall is that there was no documentation of the conda package format, at least in 2016. It was easy enough to inspect some existing packages and produce compatible ones, but no documentation implies no guarantee that the format is stable.

The main reason it didn't go anywhere was just that no-one really wanted it. At the time, conda-forge co-ordinated three different CI services to build on different platforms, and particularly the Mac builds sometimes had to queue for days. I thought that a shortcut for pure Python packages would be valuable - but special casing them in this way was a non-starter. Since then, free CI has got more capable, and conda has gained 'noarch' support, which allows many pure Python packages to be built on a single platform. So there's less pressure now to do something like this.

[Aside: I see from a quick stalk that you're one of the founders of Fastmail. I'm now a happy user, so :wave: :+1:, even if you've moved on to other things]

tonyng1707

comment created time in 3 days

push eventEuropean-XFEL/EXtra-geom

Thomas Kluyver

commit sha 22c038d81ced4c98542cb11f034d85167573ce7f

Get quadrant positions & write XFEL geometry for LPD

view details

push time in 4 days

push eventEuropean-XFEL/EXtra-geom

Thomas Kluyver

commit sha 84051013d130b20d89dd9c45509f93f1aea84f40

Fix docstring for DSSC quadrant positions

view details

push time in 4 days

push eventEuropean-XFEL/EXtra-geom

Thomas Kluyver

commit sha b7e676a15dd62bf15e49bde7802872485cb83c29

Retrieve DSSC quadrant positions from geometry object

view details

push time in 4 days

push eventEuropean-XFEL/EXtra-geom

Thomas Kluyver

commit sha 209c8e9128a34c6b35f622218d00da0ee22f0231

Require Python >= 3.6

view details

push time in 4 days

create barnchEuropean-XFEL/EXtra-geom

branch : write-xfel

created branch time in 4 days

issue commenth5py/h5py

Missing module H5DS in low-level API Reference.

Specifically, we'll need a new file h5ds.rst - look at e.g. h5pl.rst to see what it should contain. And it will need to be added to the list in index.rst.

dota17

comment created time in 4 days

Pull request review commenth5py/h5py

Make arrays contiguous prior to H5T conversion

+New features+------------++* <news item>++Deprecations+------------++* <news item>++Exposing HDF5 functions+-----------------------++* <news item>++Bug fixes+---------++* Correctly store non-contiguous arrays, such as numpy slices.

Did something go wrong with committing it? The text I can see on Github still looks the same as before.

cydanil

comment created time in 4 days

PullRequestReviewEvent

delete branch European-XFEL/EXtra-geom

delete branch : dssc-fixes

delete time in 4 days

push eventEuropean-XFEL/EXtra-geom

Thomas Kluyver

commit sha ea27e5ae95c2e5f05c204435a9877ceaca1688d5

Set x-axis inverted rather than toggling it

view details

Thomas Kluyver

commit sha 00bccf599884a7a18518155c22bfe8ba0576482a

Fix y-axis metres scale when plotting DSSC data

view details

Thomas Kluyver

commit sha 390fbe7216d517e480690922429c37f441ae000e

Revert "Set x-axis inverted rather than toggling it" This reverts commit ea27e5ae95c2e5f05c204435a9877ceaca1688d5.

view details

Thomas Kluyver

commit sha 95104c6294b52fef29e7ae8aa84d5560daa26b22

Merge pull request #23 from European-XFEL/dssc-fixes Fix y-axis scale for DSSC data plotted in metres

view details

push time in 4 days

PR merged European-XFEL/EXtra-geom

Fix y-axis scale for DSSC data plotted in metres

The axis_units='m' option when plotting puts the scale in metres rather than pixels. This was implemented as a simple scale factor, which works where pixels have equal aspect ratio. DSSC pixels don't, and we were scaling by the pixel width, exaggerating the vertical height.

This shouldn't change the images, which we were already squashing to the correct aspect ratio, only the y ticks.

+20 -12

1 comment

3 changed files

takluyver

pr closed time in 4 days

pull request commenth5py/h5py

Remove pavement.py & other outdated files related to building on Windows

This one is simple cleanup, so I'll merge it in a day if no-one objects.

takluyver

comment created time in 4 days

issue closedh5py/h5py

Error compiling Cython file

To assist reproducing bugs, please include the following:

  • Operating System (e.g. Windows 10, MacOS 10.11, Ubuntu 16.04.2 LTS, CentOS 7)

Ubuntu 18.04

  • Python version (e.g. 2.7, 3.5)

3.6.8

  • Where Python was acquired (e.g. system Python on MacOS or Linux, Anaconda on Windows)

Linux

Summary of the h5py configuration
---------------------------------

h5py    2.10.0
HDF5    1.10.4
Python  3.6.8 (default, Jan 14 2019, 11:02:34)
[GCC 8.0.1 20180414 (experimental) [trunk revision 259383]]
sys.platform    linux
sys.maxsize     9223372036854775807
numpy   1.18.0

  • The full traceback/stack trace shown (if it appears)
[ 1/19] Cythonizing /data/h5py/h5py/h5.pyx

Error compiling Cython file:
------------------------------------------------------------
...

INDEX_NAME      = H5_INDEX_NAME       # Index on names
INDEX_CRT_ORDER = H5_INDEX_CRT_ORDER  # Index on creation order

HDF5_VERSION_COMPILED_AGAINST = HDF5_VERSION
NUMPY_VERSION_COMPILED_AGAINST = NUMPY_BUILD_VERSION
                                ^
------------------------------------------------------------

h5py/h5.pyx:24:33: undeclared name not builtin: NUMPY_BUILD_VERSION

Error compiling Cython file:
------------------------------------------------------------

INDEX_NAME      = H5_INDEX_NAME       # Index on names
INDEX_CRT_ORDER = H5_INDEX_CRT_ORDER  # Index on creation order

HDF5_VERSION_COMPILED_AGAINST = HDF5_VERSION
NUMPY_VERSION_COMPILED_AGAINST = NUMPY_BUILD_VERSION
CYTHON_VERSION_COMPILED_WITH = CYTHON_BUILD_VERSION
                              ^
------------------------------------------------------------

h5py/h5.pyx:25:31: undeclared name not builtin: CYTHON_BUILD_VERSION
Traceback (most recent call last):
  File "setup.py", line 166, in <module>
    cmdclass = CMDCLASS,
  File "/usr/lib/python3/dist-packages/setuptools/__init__.py", line 129, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib/python3.6/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib/python3.6/distutils/dist.py", line 955, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "setup.py", line 75, in run
    buildobj.run()
  File "/usr/lib/python3.6/distutils/command/build.py", line 135, in run
    self.run_command(cmd_name)
  File "/usr/lib/python3.6/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.6/distutils/dist.py", line 974, in run_command
    cmd_obj.run()
  File "/data/h5py/setup_build.py", line 186, in run
    language_level=3)
  File "/usr/local/lib/python3.6/dist-packages/Cython/Build/Dependencies.py", line 1102, in cythonize
    cythonize_one(*args)
  File "/usr/local/lib/python3.6/dist-packages/Cython/Build/Dependencies.py", line 1225, in cythonize_one
    raise CompileError(None, pyx_file)
Cython.Compiler.Errors.CompileError: /data/h5py/h5py/h5.pyx

I am using the master branch code to test this, it's caused by recently merged PR https://github.com/h5py/h5py/pull/1627 I think. CC: @takluyver

closed time in 4 days

dota17

issue commenth5py/h5py

Error compiling Cython file

OK, I think we can close this now.

dota17

comment created time in 4 days

push eventtakluyver/h5py

Thomas Kluyver

commit sha 17fa235f66a81f42db2961e2a16d74cc97d100ec

Drop removed files from MANIFEST.in

view details

push time in 4 days

PullRequestReviewEvent

Pull request review commenth5py/h5py

Make arrays contiguous prior to H5T conversion

+New features+------------++* <news item>++Deprecations+------------++* <news item>++Exposing HDF5 functions+-----------------------++* <news item>++Bug fixes+---------++* Correctly store non-contiguous arrays, such as numpy slices.

Specifically, it's storing them as HDF5 variable length data that was failing before.

Otherwise, this looks good, thanks.

cydanil

comment created time in 4 days

PR opened h5py/h5py

Remove pavement.py & other outdated files related to building on Windows

Closes #1664.

+2 -478

0 comment

8 changed files

pr created time in 4 days

create barnchtakluyver/h5py

branch : rm-pavement

created branch time in 4 days

delete branch takluyver/h5py

delete branch : require-py35

delete time in 4 days

pull request commenth5py/h5py

Fix for zero length collective IO - rebased

test_collective_write and test_collective_write_empty_rank both write a scalar (0D) value into a 2D selection, i.e. broadcasting it.

We implement some special casing which in this case pre-broadcasts the 0D data to 1D. I'd suggest that the broadcasting check should happen before that - it's an implementation detail, and it's still logically broadcasting if we do it with numpy before passing data to HDF5.

aragilar

comment created time in 5 days

pull request commenth5py/h5py

Bump minimum numpy on Python 3.7 to 1.14.5

I'll merge this in a couple more days unless there are objections.

takluyver

comment created time in 5 days

issue commenth5py/h5py

h5py does not import 'mpi4py.MPI'

@dota17 this should be a one-line fix, and I'd like to have it solved for version 3.0. Would you like to do a PR for it?

1313e

comment created time in 5 days

pull request commenth5py/h5py

Make arrays contiguous prior to H5T conversion

Oh, I almost forgot - can you add a news fragment for this change?

cydanil

comment created time in 5 days

issue commentgroove-x/trio-util

AsyncDict vs. cancellation

Would it make sense, rather than adding to AsyncDictionary, to define a more restrictive 'expecting' async dictionary (ExpectationManager :stuck_out_tongue: ), which would guarantee:

  • You can only retrieve a value inside a with x.expecting(key) block
  • You can only set a key if any task is currently expecting it
smurfix

comment created time in 5 days

push eventh5py/h5py

Doctr (Travis CI)

commit sha 9410256c433711cf2bb8d1ff8663d1c2993bfd4e

Update docs after building Travis build 2302 of h5py/h5py The docs were built from the branch 'master' against the commit 0076bc4880bfa8b311338b21ddcd0e28cb27c443. The Travis build that generated this commit is at https://travis-ci.org/h5py/h5py/jobs/727627090. The doctr command that was run is /home/travis/virtualenv/python3.8.0/bin/doctr deploy . --built-docs docs_api/_build/html

view details

push time in 5 days

delete branch takluyver/h5py

delete branch : virtuallayout-no-default-dtype

delete time in 5 days

push eventh5py/h5py

Thomas Kluyver

commit sha ae1fba198b0b75bcaacb69fa7a59b37505c927b2

Require a dtype in VirtualLayout Closes gh-1474

view details

Thomas Kluyver

commit sha 13f5a80e68be4196d589f171206b3f24bd2f6517

Remove internal default dtype for virtual dataset

view details

Thomas Kluyver

commit sha 0076bc4880bfa8b311338b21ddcd0e28cb27c443

Merge pull request #1665 from takluyver/virtuallayout-no-default-dtype Require a dtype in VirtualLayout

view details

push time in 5 days

more