profile
viewpoint

kyleam/bog 52

Extensions for research notes in Org mode

kyleam/emacs.d 7

Emacs configuration

kyleam/blind 3

Temporarily mask file or directory names

kyleam/etc 2

configuration files

kyleam/mlb-rundiff 2

Analysis of MLB run differentials and jet lag

kyleam/ace-link 0

Quickly follow links using `ace-jump-mode'

kyleam/ace-window 0

Quickly switch windows using `ace-jump-mode'

kyleam/cask 0

Project management tool for Emacs

kyleam/cpinfo 0

Emacs package for recording information when copying files

issue commentdatalad/datalad

tests stall (again!) with recent git-annex

At least some of the hanging we're still experiencing should be addressed by git-annex's 8.20200617-134-gf912f8e5f (associated bug report). On a Xenial virtual machine, that commit resolves hanging I see with both the localhost and docker ssh targets, specifically in test_ria_postclonecfg, test_install_subds_from_another_remote, and test_install_subds_with_space.

I triggered a job with the most recent standalone snapshot (8.20200617-144-gb7a78cbb2). Sadly it still seems to hang, though it is getting farther along. All the recent cron jobs with recent annex versions were (consistently, I believe) hanging in core.distributed.tests.test_clone. I've triggered another job with -v to see which particular test is hanging.

yarikoptic

comment created time in 2 days

push eventdatalad/datalad

Kyle Meyer

commit sha 2de1bddfd330f2bf2963f971953df2c4caa25695

[tmp] git annex version pre-run

view details

push time in 2 days

push eventdatalad/datalad

Kyle Meyer

commit sha 1ca47a8f1ae42b6a724e1892b2406edf41bda5e3

[tmp] nose -v

view details

push time in 2 days

delete branch kyleam/niceman

delete branch : ssh-host-doc

delete time in 2 days

delete branch kyleam/niceman

delete branch : local-same-directory

delete time in 2 days

push eventdatalad/datalad

Kyle Meyer

commit sha d3d6c976c5416551cc2d957606fa81b979b544ce

[tmp] test with recent standalone hangs hopefully fixed with f912f8e5f (refix bug in a better way, 2020-07-02) IIRC these snapshots might not be working on Travis.

view details

push time in 2 days

create barnchdatalad/datalad

branch : scratch/annex-recent

created branch time in 2 days

pull request commentdatalad/datalad

ENH: addurls: Support reading records from stdin

Is it complicated to just slurp-in json lines into an array?

It's more complicated than what is there now, introduces more code when there are tools that can readily do the job, and conceptually introduces yet another way to input data.

It is much easier to do within DataLad than to install an additional tool and/or erect complicated pipeline outside.

There's a balance, and I'd argue that DataLad is already on the wrong side of that balance and has unnecessary bloat.

kyleam

comment created time in 2 days

issue openeddatalad/datalad

AppVeyor codecov submission failing regular

In the past, codecov submission errors on AppVeyor were common but didn't happen every time. The submission is now failing regularly and in a style I don't recall seeing until recently (a lot of the error message is one-character per line). Here's the latest example from gh-4669:

https://ci.appveyor.com/project/mih/datalad/builds/33876628/job/5g5p2qmsckpupkcw#L1923

created time in 2 days

pull request commentdatalad/datalad

ENH: addurls: Support reading records from stdin

The failing job on Travis is due to a timeout for the codecov submission: https://travis-ci.org/github/datalad/datalad/jobs/704417042#L2227

The failing AppVeyor runs are the now regular codecov submission failures. (I'm not sure there much to do on our end, but I'll open an issue.)

kyleam

comment created time in 2 days

pull request commentdatalad/datalad

ENH: addurls: Support reading records from stdin

I keep getting lost in jq "language" unless I use it consistently... Having an explicit option for json-lines or automagically determining the format (starts with [ -- pure json list, starts with { -- lines) would be of great assistance.

In my view this isn't a compelling reason to complicate things on DataLad's side.

kyleam

comment created time in 2 days

PR opened datalad/datalad

ENH: addurls: Support reading records from stdin

URL-FILE can be a csv file or a JSON file with an array of records. To allow the user to supply records on stdin, avoiding an intermediate file in some cases, read from stdin when URL-FILE is "-".

There are two main options for the JSON input on stdin: JSON lines or a JSON array. Go with an array, which has the advantage that it matches what addurls expects for a JSON file, which is good for both user expectations and not introducing divergent code paths. It also makes it clearer that addurls slurps all the records into memory rather than working with them as they come in. The disadvantage is the need for another intermediate reformatting step, though it's trivial to reformat JSON lines as an array with jq or similar tools.

The --input-type default value is "ext", which means to infer the input type based on the file extension. For stdin, we could rework things to try json.load() and then fall back to processing the input as a csv. But for now, assume that stdin should be treated as JSON by default unless --input-type specifies that it is a csv.

Closes #3557.


Note: Using this triggers what I think is a mistake (even if deliberate) in our "is interactive?" check: the check says "no" when feeding stdin even when output streams are still attached to a tty. I'll have to look more into the history of that and will open a separate PR if I think we can drop the stdin part from is_interactive.

+56 -5

0 comment

2 changed files

pr created time in 3 days

create barnchkyleam/datalad

branch : addurls-stdin

created branch time in 3 days

issue commentwarner/python-versioneer

Versioneer causes `fatal: bad revision 'HEAD'` under special circumstances

a tentative fix was proposed at datalad/datalad-osf#100 (comment)

Just a note: That defensively unsets all repository-local variables that Git reports, which I initially saw recommended at https://lore.kernel.org/git/20200228190218.GC1408759@coredump.intra.peff.net/. A more restricted fix would be unsetting just GIT_DIR.

mih

comment created time in 3 days

Pull request review commentmagit/magit

Rename magit-insert-submodules in doc to magit-insert-modules.

 the super-repository by adding ~magit-insert-submodules~ to the hook   has to return a string to be inserted or nil.  PROPS is an alist   that supports the keys ~:right-align~ and ~:pad-right~. -- Function: magit-insert-submodules+- Function: magit-insert-modules

As mentioned above, I think this actually corresponds to magit-insert-modules-overview. But that (and magit-insert-modules) are already included under "Status Module Sections", so I'd say this entry should just be deleted.

treese

comment created time in 3 days

Pull request review commentmagit/magit

Rename magit-insert-submodules in doc to magit-insert-modules.

 Also see [[man:git-submodule]] The command ~magit-list-submodules~ displays a list of the current repository's submodules in a separate buffer.  It's also possible to display information about submodules directly in the status buffer of-the super-repository by adding ~magit-insert-submodules~ to the hook+the super-repository by adding ~magit-insert-modules~ to the hook

AFAICS magit-insert-modules-overview is actually what came of magit-insert-submodules (83104db0), but, with the introduction of magit-insert-modules (b92454b7, a couple of commits before in the same series), it seems like that'd be the one to recommend here.

So, this looks like a good change.

treese

comment created time in 3 days

push eventkyleam/datalad

Andy Connolly

commit sha e6dfea698861e5fc03eee897868b4a2023259b8f

DOC: Fixed documentation builder to allow for overriding opts in eval_defaults. Closes #2762

view details

Andy Connolly

commit sha 97ac4a913c2baec1e3da0c6341e4d9d7d0df016a

Merge branch 'master' of git://github.com/datalad/datalad into iss2762

view details

Andy Connolly

commit sha d17a0cd0bf437ce2563f893a6fd1c7138b6b3c4f

DOC: corrects previous fix for defaults This should Close #2762 My previous fix added results_renderer to the _params_ in interface.Siblings which wasn't necessary as Kyle pointed out. I have incorporated his suggestions, and it seems to work. Thanks

view details

Andy Connolly

commit sha 387a4cad9f878dcd9c75c218a5862afe5508ed4c

Merge branch 'master' into iss2762

view details

Yaroslav Halchenko

commit sha c6d032c2959451c52e2da720a98d3cc3300d0a98

RF: push - introduce --transfer-data=(anything|nothing|auto) Changes are intensionally minimal. Subsequent commits might refine further

view details

Yaroslav Halchenko

commit sha a0a8cd18919b874252996d32e9ac6cc551cca028

RF: define force_git_push helper var at _push level I am luke warm about this change -- just wanted to compartmentalize "forces" to _push instead of spreading logic around all helper functions

view details

Yaroslav Halchenko

commit sha b20ed1379da540621e48a1515191a9ea32ae258f

[DATALAD RUNCMD] RF: --force=datatransfer -> --force=checkdatapresent (to be more to the point of the effect) Closes #4603 === Do not change lines below === { "chain": [], "cmd": "git-sedi datatransfer checkdatapresent", "exit": 0, "extra_inputs": [], "inputs": [], "outputs": [ "datalad/core/distributed/push.py", "datalad/core/distributed/tests/test_push.py" ], "pwd": "." } ^^^ Do not change lines above ^^^

view details

Yaroslav Halchenko

commit sha 65aae48ef0cee708a7e0fee89f076b152b64eef2

[DATALAD RUNCMD] RF: --force=pushall -> --force=all (since now could just enable all forces - orthogonal) === Do not change lines below === { "chain": [], "cmd": "git-sedi pushall all", "exit": 0, "extra_inputs": [], "inputs": [], "outputs": [ "datalad/core/distributed/push.py", "datalad/core/distributed/tests/test_push.py" ], "pwd": "." } ^^^ Do not change lines above ^^^

view details

Yaroslav Halchenko

commit sha 3492d600ac4e71084d0755796786523ef389c6f7

DOC: adjust --force docstring to be more to the point of the effects Any option is about changing "automatic decision making" IMHO. I think it is important to mention possible effects such as absence of --force for push is a safety measure. Default --fast is optimization and name "checkdatapresent" is self descriptive on the effect

view details

Yaroslav Halchenko

commit sha 1be419f0483dde6db939b2263bdbcf3d355dafb5

RF: --transfer-data=auto-if-wanted which dissolves datalad.push.copy-auto-if-wanted config I was going back and forth either we need some additional "datalad.push.transfer-data" config setting which could be set per repository to indicate what "datalad push" should do with data. But I remain of an idea that it would be counter-productive to establish yet another level of data transfer behavior instead of using what is already there provided by git-annex itself. To maintain "do the right thing" behavior by default I had decided also to make auto-if-wanted to be the default mode of operation -- it will transfer data if there is no explicit "wanted" setting for the remote.

view details

Yaroslav Halchenko

commit sha 8031fbbc11038f83631fbe6b5ca54715a7ac9866

ENH: log at debug that we do not even attempt data transfer is not git-annex To ease possible troubleshooting later on whenever some expected data transfer does not happen. Also tuned up a comment to say "copy" not "move", which is a valid (but not supported ATM) operation.

view details

Yaroslav Halchenko

commit sha 7dd99fd271cb30211e8a01f55ca6092c3aec3500

[DATALAD RUNCMD] RF: push - --transfer-data -> --data === Do not change lines below === { "chain": [], "cmd": "sed -i -e 's,transfer_data,data,g' -e 's,--transfer-data,--data,g' {outputs}", "exit": 0, "extra_inputs": [], "inputs": [], "outputs": [ "datalad/core/distributed/push.py", "datalad/core/distributed/tests/test_push.py" ], "pwd": "." } ^^^ Do not change lines above ^^^

view details

Yaroslav Halchenko

commit sha 56fd2e836d835cd09b1edd48c7e3fb65eb26d56a

ENH: RM note about anything/nothing, duplicate "data" Also placed list of constraints to a new line to make it fit 80 chars

view details

Andy Connolly

commit sha cf5919461296b55fb6c14498fc21d0db98dfe491

DOC: resolv gh-4480 w/ gh-4482

view details

Andy Connolly

commit sha f51a3c9a54db18d4a5d1074c78893eb9a0ed0e42

Merge remote-tracking branch 'gh-andycon/iss2762' into iss2762 # Conflicts: # datalad/interface/base.py

view details

Kyle Meyer

commit sha 0744b4dedceab7b43100768b1fbfec855c3d43ee

TST: travis: Pin grunt-contrib-qunit version to avoid failure The `npm install grunt-contrib-qunit` call has started to fail (e.g., <https://travis-ci.org/github/datalad/datalad/jobs/699391908>). This is presumably due to the v4.0.0 release today. Pin it to the previous release. Re: #4635

view details

Kyle Meyer

commit sha c6df67045cc8f8ae8a91e0e953d495cba1588ef3

Merge pull request #4638 from kyleam/pin-grunt-contrib-qunit

view details

Kyle Meyer

commit sha b917910464b61b2a774905a581479f1e42ada0d6

TST: clone: Fix clock-dependent test test_clone_unborn_head() is supposed to set up a scenario where the freshly cloned repo has a master branch with no commits and three non-master remote tracking branches, including origin/git-annex. As of 8f5cc450c (ENH: clone: Try to switch away from an unborn branch, 2020-04-01), clone() handles this situation by checking out the non-git-annex ref with the most recent committer date. With Git v2.27.0, specifically 7c5045fc18 (ref-filter: apply fallback refname sort only after all user sorts, 2020-05-03), this test fails intermittently. The failure is due to a bad approach in the test: it commits a change to the abc branch and then one to the chooseme branch, with the hope that chooseme will have the later committer date. Given the resolution of Git time stamps, though, both refs can easily have identical values for the committer date. Before Git's 7c5045fc18, this issue was masked by Git failing to fall back to alphabetical sorting of the refs. Use the set_date() helper to generate reliable time stamp differences.

view details

Kyle Meyer

commit sha 1d11ca9d0634772d36b9d9e3fd5e8d7eac923c3a

Merge branch 'maint'

view details

Kyle Meyer

commit sha 38ba64b1cf67964ef9a6204a24d71bf635fa9308

TST: push: Add test for data="auto" Adapt the test posted at gh-4541. Aside from tweaking the way to specify "use --auto", add a block that tests that the following two behave the same when a file does not match a remote's preferred content: datalad push --to=r --data=auto FILE git annex copy --to=r --auto FILE

view details

push time in 3 days

Pull request review commentdatalad/datalad

ENH+RF: tools/ci/install-annex.sh helper and switch to use conda build

 before_install:   - tools/ci/prep-travis-forssh.sh   # Install grunt-cli   - eatmydata npm install grunt-cli-  # Install optionally upstream current development so we are sure that they break nothing important for us-  - if [ ! -z "${_DL_UPSTREAM_GITANNEX:-}" ]; then sudo tools/ci/install-annex-snapshot.sh; sudo ln -s `find /usr/local/lib/git-annex.linux -maxdepth 1 -type f -perm /+x` /usr/local/bin/; else sudo eatmydata apt-get install git-annex-standalone ; fi-  # Install optionally -devel version of annex, and if goes wrong (we have most recent), exit right away-  - if [ ! -z "${_DL_DEVEL_ANNEX:-}" ]; then tools/ci/prep-travis-devel-annex.sh || { ex="$?"; if [ "$ex" -eq 99 ]; then exit 0; else exit "$ex"; fi; }; fi-  - if [ ! -z "${_DL_DEBURL_ANNEX:-}" ]; then wget -O /tmp/git-annex.deb "${_DL_DEBURL_ANNEX}" && sudo dpkg -i /tmp/git-annex.deb; fi-  # Optionally install the latest Git.  Exit code 100 indicates that bundled is same as the latest.-  - if [ ! -z "${_DL_UPSTREAM_GIT:-}" ]; then-      sudo tools/ci/install-latest-git.sh || { [ $? -eq 100 ] && exit 0; } || exit 1;-    fi--install:-  # Install standalone build of git-annex for the recent enough version+  # Install various basic depedencies   - travis_retry sudo eatmydata apt-get install zip pandoc p7zip-full   # needed for tests of patool compression fall-back solution   - travis_retry sudo eatmydata apt-get install xz-utils   - travis_retry sudo eatmydata apt-get install shunit2   # for metadata support   - travis_retry sudo eatmydata apt-get install exempi+  # Configure TMPDIR before trying install git-annex -- script might use it+  - if [[ "${TMPDIR:-}" =~ .*/sym\ link ]]; then echo "Symlinking $TMPDIR"; ln -s /tmp "$TMPDIR"; fi+  - if [[ "${TMPDIR:-}" =~ .*/d\ i\ r ]]; then echo "mkdir $TMPDIR"; mkdir -p "$TMPDIR"; fi+  - if [[ "${TMPDIR:-}" =~ .*/nfsmount ]]; then echo "mkdir $TMPDIR"; mkdir -p "$TMPDIR" "${TMPDIR}_"; echo "/tmp/nfsmount_ localhost(rw)" | sudo bash -c 'cat - > /etc/exports'; sudo apt-get install -y nfs-kernel-server; sudo exportfs -a; sudo mount -t nfs localhost:/tmp/nfsmount_ /tmp/nfsmount; fi+  # Install git-annex+  - OLD_PATH="$PATH"+  - eval source tools/ci/install-annex.sh ${_DL_ANNEX_INSTALL_SCENARIO}

Why is eval needed?

yarikoptic

comment created time in 4 days

create barnchdatalad/datalad

branch : scratch/travis-conda

created branch time in 4 days

pull request commentdatalad/datalad

ENH+RF: tools/ci/install-annex.sh helper and switch to use conda build

[ see you just posted, but since I just typed this out and it seems like you want to go a different route ]

The TMPDIR="/var/tmp/sym link" jobs still fail in the setup stage. The error happens in the argument parsing miniconda install script. We could get by that particular failure by switching to the temporary directory in a subshell, but the mininconda script will just fail later on.

I'm not sure what you have in mind for this script, but I don't think see much value in worrying about $TMPDIR. I think we should just install in either the current directory or /tmp (which should work for the context we need it).

<details> <summary>later failure</summary>

_miniconda_script=Miniconda3-latest-Linux-x86_64.sh
TMPDIR="/var/tmp/sym link"
_TMPDIR=$(mktemp -d "${TMPDIR:-/tmp}/ga-XXXXXXX")
wget -O "$_TMPDIR/${_miniconda_script}" "${ANACONDA_URL:-https://repo.anaconda.com/miniconda/}${_miniconda_script}"
(
    cd "$_TMPDIR"
    HOME="$_TMPDIR"
    bash "${_miniconda_script}" -b -p miniconda
)

[... 97 lines ...]
Preparing transaction: done
Executing transaction: done
rm: cannot remove '/var/tmp/sym': Is a directory
rm: cannot remove '/var/tmp/sym': Is a directory
Miniconda3-latest-Linux-x86_64.sh: line 449: /var/tmp/sym: Is a directory

</details>

https://travis-ci.org/github/datalad/datalad/jobs/703768393#L779

yarikoptic

comment created time in 4 days

delete branch datalad/datalad

delete branch : scratch/travis-conda

delete time in 4 days

pull request commentdatalad/datalad

ENH+RF: tools/ci/install-annex.sh helper and switch to use conda build

There are three jobs with tools/ci/install-annex.sh errors, and one job that has a single failing test (perhaps due to brittle commit counting, but I haven't taken a closer look).

https://travis-ci.org/github/datalad/datalad/builds/703721783

yarikoptic

comment created time in 4 days

push eventyarikoptic/datalad

Andy Connolly

commit sha e6dfea698861e5fc03eee897868b4a2023259b8f

DOC: Fixed documentation builder to allow for overriding opts in eval_defaults. Closes #2762

view details

Andy Connolly

commit sha 97ac4a913c2baec1e3da0c6341e4d9d7d0df016a

Merge branch 'master' of git://github.com/datalad/datalad into iss2762

view details

Andy Connolly

commit sha d17a0cd0bf437ce2563f893a6fd1c7138b6b3c4f

DOC: corrects previous fix for defaults This should Close #2762 My previous fix added results_renderer to the _params_ in interface.Siblings which wasn't necessary as Kyle pointed out. I have incorporated his suggestions, and it seems to work. Thanks

view details

Andy Connolly

commit sha 387a4cad9f878dcd9c75c218a5862afe5508ed4c

Merge branch 'master' into iss2762

view details

Yaroslav Halchenko

commit sha c6d032c2959451c52e2da720a98d3cc3300d0a98

RF: push - introduce --transfer-data=(anything|nothing|auto) Changes are intensionally minimal. Subsequent commits might refine further

view details

Yaroslav Halchenko

commit sha a0a8cd18919b874252996d32e9ac6cc551cca028

RF: define force_git_push helper var at _push level I am luke warm about this change -- just wanted to compartmentalize "forces" to _push instead of spreading logic around all helper functions

view details

Yaroslav Halchenko

commit sha b20ed1379da540621e48a1515191a9ea32ae258f

[DATALAD RUNCMD] RF: --force=datatransfer -> --force=checkdatapresent (to be more to the point of the effect) Closes #4603 === Do not change lines below === { "chain": [], "cmd": "git-sedi datatransfer checkdatapresent", "exit": 0, "extra_inputs": [], "inputs": [], "outputs": [ "datalad/core/distributed/push.py", "datalad/core/distributed/tests/test_push.py" ], "pwd": "." } ^^^ Do not change lines above ^^^

view details

Yaroslav Halchenko

commit sha 65aae48ef0cee708a7e0fee89f076b152b64eef2

[DATALAD RUNCMD] RF: --force=pushall -> --force=all (since now could just enable all forces - orthogonal) === Do not change lines below === { "chain": [], "cmd": "git-sedi pushall all", "exit": 0, "extra_inputs": [], "inputs": [], "outputs": [ "datalad/core/distributed/push.py", "datalad/core/distributed/tests/test_push.py" ], "pwd": "." } ^^^ Do not change lines above ^^^

view details

Yaroslav Halchenko

commit sha 3492d600ac4e71084d0755796786523ef389c6f7

DOC: adjust --force docstring to be more to the point of the effects Any option is about changing "automatic decision making" IMHO. I think it is important to mention possible effects such as absence of --force for push is a safety measure. Default --fast is optimization and name "checkdatapresent" is self descriptive on the effect

view details

Yaroslav Halchenko

commit sha 1be419f0483dde6db939b2263bdbcf3d355dafb5

RF: --transfer-data=auto-if-wanted which dissolves datalad.push.copy-auto-if-wanted config I was going back and forth either we need some additional "datalad.push.transfer-data" config setting which could be set per repository to indicate what "datalad push" should do with data. But I remain of an idea that it would be counter-productive to establish yet another level of data transfer behavior instead of using what is already there provided by git-annex itself. To maintain "do the right thing" behavior by default I had decided also to make auto-if-wanted to be the default mode of operation -- it will transfer data if there is no explicit "wanted" setting for the remote.

view details

Yaroslav Halchenko

commit sha 8031fbbc11038f83631fbe6b5ca54715a7ac9866

ENH: log at debug that we do not even attempt data transfer is not git-annex To ease possible troubleshooting later on whenever some expected data transfer does not happen. Also tuned up a comment to say "copy" not "move", which is a valid (but not supported ATM) operation.

view details

Yaroslav Halchenko

commit sha 7dd99fd271cb30211e8a01f55ca6092c3aec3500

[DATALAD RUNCMD] RF: push - --transfer-data -> --data === Do not change lines below === { "chain": [], "cmd": "sed -i -e 's,transfer_data,data,g' -e 's,--transfer-data,--data,g' {outputs}", "exit": 0, "extra_inputs": [], "inputs": [], "outputs": [ "datalad/core/distributed/push.py", "datalad/core/distributed/tests/test_push.py" ], "pwd": "." } ^^^ Do not change lines above ^^^

view details

Yaroslav Halchenko

commit sha 56fd2e836d835cd09b1edd48c7e3fb65eb26d56a

ENH: RM note about anything/nothing, duplicate "data" Also placed list of constraints to a new line to make it fit 80 chars

view details

Andy Connolly

commit sha cf5919461296b55fb6c14498fc21d0db98dfe491

DOC: resolv gh-4480 w/ gh-4482

view details

Andy Connolly

commit sha f51a3c9a54db18d4a5d1074c78893eb9a0ed0e42

Merge remote-tracking branch 'gh-andycon/iss2762' into iss2762 # Conflicts: # datalad/interface/base.py

view details

Kyle Meyer

commit sha b917910464b61b2a774905a581479f1e42ada0d6

TST: clone: Fix clock-dependent test test_clone_unborn_head() is supposed to set up a scenario where the freshly cloned repo has a master branch with no commits and three non-master remote tracking branches, including origin/git-annex. As of 8f5cc450c (ENH: clone: Try to switch away from an unborn branch, 2020-04-01), clone() handles this situation by checking out the non-git-annex ref with the most recent committer date. With Git v2.27.0, specifically 7c5045fc18 (ref-filter: apply fallback refname sort only after all user sorts, 2020-05-03), this test fails intermittently. The failure is due to a bad approach in the test: it commits a change to the abc branch and then one to the chooseme branch, with the hope that chooseme will have the later committer date. Given the resolution of Git time stamps, though, both refs can easily have identical values for the committer date. Before Git's 7c5045fc18, this issue was masked by Git failing to fall back to alphabetical sorting of the refs. Use the set_date() helper to generate reliable time stamp differences.

view details

Kyle Meyer

commit sha 38ba64b1cf67964ef9a6204a24d71bf635fa9308

TST: push: Add test for data="auto" Adapt the test posted at gh-4541. Aside from tweaking the way to specify "use --auto", add a block that tests that the following two behave the same when a file does not match a remote's preferred content: datalad push --to=r --data=auto FILE git annex copy --to=r --auto FILE

view details

Andy Connolly

commit sha d22ebe3731be7f41d675f863477b7d9fd339e0f6

[DOC] Add test for eval_params overrids added overriding parameters to existing test class TestUtils in datalad/interface/tests/test_utils.py passed my tests using : DATALAD_TESTS_SSH=1 python -m nose -s -v datalad/interface/tests/test_utils.py

view details

Andy Connolly

commit sha 9940af0d70c9ee59329f8e1a9fe9753a00f4162a

Merge branch 'master' of git://github.com/datalad/datalad into iss2762

view details

Yaroslav Halchenko

commit sha 98233826762a52b56c117228a1e04494ac42895e

[DATALAD RUNCMD] Minor typo fix (over[r]ide) === Do not change lines below === { "chain": [], "cmd": "sed -i -e s,overide,override,g datalad/interface/tests/test_utils.py", "exit": 0, "extra_inputs": [], "inputs": [], "outputs": [ "datalad/interface/tests/test_utils.py" ], "pwd": "." } ^^^ Do not change lines above ^^^

view details

push time in 4 days

pull request commentdatalad/datalad

ENH+RF: tools/ci/install-annex.sh helper and switch to use conda build

Thanks for working on this. The install script looks like a nice addition.

I'm about to push an update here. The only code change is to make the PATH adjustment seen by ssh commands. I've rebased to deal with conflicts in .travis.yml. I'm switching to maint as the base because (1) this is only related to our testing setup, (2) I think ideally maint and master should share a similar setup if possible, and (3) it'd be nice for maint's tests to be sped up too.

I've dropped the temporary commit that removed other testing because a test run with the changes here passed. (And, at least based on that one run, it looks like a good speed up.)

<details> <summary>range-diff</summary>

1:  c62a768a5 = 1:  fc973c2d5 NF: a script to centralize installation of git-annex in various scenarios
2:  848a9e679 = 2:  a462a4dcd RF: .travis.yml - install annex using script, switch to conda-forge build
3:  5e65dbd38 = 3:  8e0ae333c RM: install-annex-snapshot.sh - just use install-annex.sh snapshot from now on
4:  1fed2f498 < -:  --------- TEMP: remove all matrixes etc
5:  358d37618 = 4:  fd067d1e8 ENH: make it possible to install any specified version from conda-forge
6:  493f366c3 ! 5:  e3b69a0b8 BF: use old 7.20190819 git-annex from conda-forge
    @@ .travis.yml: env:
          # get ahead in PATH to not shadow travis' python
     -    - _DL_ANNEX_INSTALL_SCENARIO=conda-forge-last
     +    - _DL_ANNEX_INSTALL_SCENARIO="conda-forge-last 7.20190819"
    - 
    - 
    - before_install:
    +   matrix:
    + ##1    - DATALAD_REPO_VERSION=6
    +     - DATALAD_REPO_VERSION=5
7:  00ee6a82e = 6:  7bd1e6760 BF: travis - store adjusted PATH export within ~/.bashrc
-:  --------- > 7:  593094ffa TST: travis: Reposition PATH adjustment for git-annex

</details>

yarikoptic

comment created time in 4 days

push eventdatalad/datalad

Kyle Meyer

commit sha 499df4a05d0af9b1ca155003047857d2799c9ae9

[tmp] travis conda From pr 4640. This is quick check. An extra commit will be added on top of that PR if it works out.

view details

push time in 4 days

push eventdatalad/datalad

Andy Connolly

commit sha e6dfea698861e5fc03eee897868b4a2023259b8f

DOC: Fixed documentation builder to allow for overriding opts in eval_defaults. Closes #2762

view details

Andy Connolly

commit sha 97ac4a913c2baec1e3da0c6341e4d9d7d0df016a

Merge branch 'master' of git://github.com/datalad/datalad into iss2762

view details

Andy Connolly

commit sha d17a0cd0bf437ce2563f893a6fd1c7138b6b3c4f

DOC: corrects previous fix for defaults This should Close #2762 My previous fix added results_renderer to the _params_ in interface.Siblings which wasn't necessary as Kyle pointed out. I have incorporated his suggestions, and it seems to work. Thanks

view details

Andy Connolly

commit sha 387a4cad9f878dcd9c75c218a5862afe5508ed4c

Merge branch 'master' into iss2762

view details

Yaroslav Halchenko

commit sha 238145806c2357ddc8d8e1e80b4f6689a43b82cf

RM: datalad.interface.run - was deprecated in 0.12.0rc4-51-g8fb74ed18

view details

Yaroslav Halchenko

commit sha ec4afad564d15f925a5716be070f4be904cc1705

RM: run(..., saver=None) - was deprecated and ignored as of 0.12.0rc6~30^2~4

view details

Yaroslav Halchenko

commit sha 5c0ec369fee740b400c2793a16b1822ab245a107

ENH: join_cmdline to complement split_cmdline which uses quote_cmdlinearg

view details

Yaroslav Halchenko

commit sha 2e429cb583735e44331b2a4118c91ddfc3e0c5d6

RF: make use of join_cmdline Also includes a change to output produced by def _format_cmd_shorty(cmd): """Get short string representation from a cmd argument list""" which used to join then without quoting cmd_shorty = (' '.join(cmd) if isinstance(cmd, list) else cmd) I thought that quoted would be more correct, especially when it is not going to get shortened.

view details

Yaroslav Halchenko

commit sha f29fc4e7e5f5928d78c9f7c9cb82dce5d9382878

RF: move .dot_git assignment logic into a staticmethod

view details

Yaroslav Halchenko

commit sha 8fd4e9ea932885d0cec03dbcd7d0854ef867addd

ENH: add ok_missing flag so it could be used in cases where there might not yet be a repo Primarily done to provide unification with get_git_dir

view details

Yaroslav Halchenko

commit sha 0b60ad952afd092a39bdb14179403e4c39da9788

ENH: _get_dot_git - introduce option relative

view details

Yaroslav Halchenko

commit sha 0137ed4b8f31ffd77760d46006585a92fa1e395b

RF: use _get_dot_git in get_git_dir and adjust note to mention possible deprecation

view details

Yaroslav Halchenko

commit sha eb16ddbce6eba3f96609de9740d22ebd72feb999

DOC+ENH: fill out docstring for _get_dot_git and make kwargs mandatory kwargs

view details

Yaroslav Halchenko

commit sha 173b3560dbbe7465b1f08c5e15c27b48585e5ea9

DOC: strip obsolete coment, adjust another one

view details

Yaroslav Halchenko

commit sha c6d032c2959451c52e2da720a98d3cc3300d0a98

RF: push - introduce --transfer-data=(anything|nothing|auto) Changes are intensionally minimal. Subsequent commits might refine further

view details

Yaroslav Halchenko

commit sha a0a8cd18919b874252996d32e9ac6cc551cca028

RF: define force_git_push helper var at _push level I am luke warm about this change -- just wanted to compartmentalize "forces" to _push instead of spreading logic around all helper functions

view details

Yaroslav Halchenko

commit sha b20ed1379da540621e48a1515191a9ea32ae258f

[DATALAD RUNCMD] RF: --force=datatransfer -> --force=checkdatapresent (to be more to the point of the effect) Closes #4603 === Do not change lines below === { "chain": [], "cmd": "git-sedi datatransfer checkdatapresent", "exit": 0, "extra_inputs": [], "inputs": [], "outputs": [ "datalad/core/distributed/push.py", "datalad/core/distributed/tests/test_push.py" ], "pwd": "." } ^^^ Do not change lines above ^^^

view details

Yaroslav Halchenko

commit sha 65aae48ef0cee708a7e0fee89f076b152b64eef2

[DATALAD RUNCMD] RF: --force=pushall -> --force=all (since now could just enable all forces - orthogonal) === Do not change lines below === { "chain": [], "cmd": "git-sedi pushall all", "exit": 0, "extra_inputs": [], "inputs": [], "outputs": [ "datalad/core/distributed/push.py", "datalad/core/distributed/tests/test_push.py" ], "pwd": "." } ^^^ Do not change lines above ^^^

view details

Yaroslav Halchenko

commit sha 3492d600ac4e71084d0755796786523ef389c6f7

DOC: adjust --force docstring to be more to the point of the effects Any option is about changing "automatic decision making" IMHO. I think it is important to mention possible effects such as absence of --force for push is a safety measure. Default --fast is optimization and name "checkdatapresent" is self descriptive on the effect

view details

Yaroslav Halchenko

commit sha 1be419f0483dde6db939b2263bdbcf3d355dafb5

RF: --transfer-data=auto-if-wanted which dissolves datalad.push.copy-auto-if-wanted config I was going back and forth either we need some additional "datalad.push.transfer-data" config setting which could be set per repository to indicate what "datalad push" should do with data. But I remain of an idea that it would be counter-productive to establish yet another level of data transfer behavior instead of using what is already there provided by git-annex itself. To maintain "do the right thing" behavior by default I had decided also to make auto-if-wanted to be the default mode of operation -- it will transfer data if there is no explicit "wanted" setting for the remote.

view details

push time in 4 days

create barnchdatalad/datalad

branch : scratch/travis-conda

created branch time in 4 days

push eventdatalad/datalad

Kyle Meyer

commit sha f0a283974d3184cd6f9dd16367d610d650b7a2dd

TST: test_utils: Don't use 'is' with a literal This works but is an implementation detail of CPython. Python 3.8 emits a SyntaxWarning.

view details

Yaroslav Halchenko

commit sha 586a43a31ab3db75c8cebafa54bbc29c97a604dd

ENH(TST): use assert_equal to see why conda python 3.8 tests fail

view details

Kyle Meyer

commit sha 0b7209711e00401500b288780b01cb52f5918090

TST: travis: Add job for python 3.8 Before Python 3.8 was released, we tested against it and fixed up the issues at the time, but it wasn't added as a regular job. Add it now, demoting the 3.6 job to a cron job so we don't increase our testing time. Use Bionic rather than Xenial as the distribution so that we can use a pre-installed 3.8 instead of 3.8-dev [0]. Note that we can't use this across all runs because Bionic doesn't come with Python 3.5 pre-installed. [0] https://docs.travis-ci.com/user/reference/bionic/#python-support

view details

Kyle Meyer

commit sha d1345797b990febeea4a1eb03cd0e87baf1ed6b3

Merge pull request #4664 from yarikoptic/enh-open-files

view details

Michael Hanke

commit sha af72224e401e135beaafc8d4585d576c8357c91e

ENH: Let default result renderer tolerate non-path results Example use case: credential query (https://github.com/datalad/datalad-osf/pull/105) There is nothing on the file system that such credentials are (or have to be) associated with. Before this change, the default result handler would crash on results without a `path` key. After this change, it merely doesn't report a path at all. Spacing is adjusted be `<action>(<status>): [<message>]` in this case.

view details

Yaroslav Halchenko

commit sha d98eb91c403157c93c6153d17fafb9d82cb00afe

Merge pull request #4666 from mih/enh-nonpathresults ENH: Let default result renderer tolerate non-path results

view details

Kyle Meyer

commit sha 214bdc42aeea704794d8057b68721a22af885228

Merge pull request #4663 from kyleam/travis-py38

view details

Kyle Meyer

commit sha 9b33b39a3674c5dd7b7dba953e455d2d33e47a66

Merge branch 'maint' into master

view details

push time in 4 days

delete branch kyleam/datalad

delete branch : travis-py38

delete time in 4 days

push eventdatalad/datalad

Kyle Meyer

commit sha f0a283974d3184cd6f9dd16367d610d650b7a2dd

TST: test_utils: Don't use 'is' with a literal This works but is an implementation detail of CPython. Python 3.8 emits a SyntaxWarning.

view details

Kyle Meyer

commit sha 0b7209711e00401500b288780b01cb52f5918090

TST: travis: Add job for python 3.8 Before Python 3.8 was released, we tested against it and fixed up the issues at the time, but it wasn't added as a regular job. Add it now, demoting the 3.6 job to a cron job so we don't increase our testing time. Use Bionic rather than Xenial as the distribution so that we can use a pre-installed 3.8 instead of 3.8-dev [0]. Note that we can't use this across all runs because Bionic doesn't come with Python 3.5 pre-installed. [0] https://docs.travis-ci.com/user/reference/bionic/#python-support

view details

Kyle Meyer

commit sha 214bdc42aeea704794d8057b68721a22af885228

Merge pull request #4663 from kyleam/travis-py38

view details

push time in 4 days

PR merged datalad/datalad

travis: Add job for python 3.8, restrict cron
  • gh-4659 pointed to a failure in the conda py3.8 runs. Before Python 3.8 was released, we tested against 3.8 on Travis and fixed up the issues at the time, but we didn't add as a regular job. Add it now, demoting the 3.6 job to a cron job so we don't increase our testing time.

  • ~Restrict the cron builds to _DL_CRON=1 jobs.~

+7 -1

0 comment

2 changed files

kyleam

pr closed time in 4 days

push eventkyleam/datalad

push time in 5 days

push eventdatalad/datalad

Kyle Meyer

commit sha a3999154185602b11b5712ef117fc531e23d9745

[tmp] skip ria_postclonecfg

view details

push time in 6 days

delete branch datalad/datalad

delete branch : ssh-test-setup

delete time in 6 days

create barnchdatalad/datalad

branch : ssh-test-setup

created branch time in 6 days

Pull request review commentdatalad/datalad

travis: Add job for python 3.8, restrict cron

 matrix: #    - UNSET_S3_SECRETS=1  before_install:-  - if [ ! -z "${_DL_CRON:-}" ]; then-      if [ ! ${TRAVIS_EVENT_TYPE} = "cron" ]; then-        echo "Exiting since should only be ran on a weekly basis by cron!";-        exit 0;+  - if [ "${TRAVIS_EVENT_TYPE}" = "cron" ]; then+      if [ -z "${_DL_CRON:-}" ]; then+          echo "Skipping regular job on weekly cron";+          exit 0;+      fi+    else+      if [ ! -z "${_DL_CRON:-}" ]; then+          echo "Exiting since should only be ran on a weekly basis by cron!";

could you please pickup and finish #4640 to use conda instead of bundled annex. That should speed up our tests runs, helping to mitigate our general problem of lengthy tests run.

I doubt that will be a large enough speed up to really matter in this context. But no, right now I'm focused on figuring out the Travis ssh hang with newer versions of git-annex (and making some progress).

Sure thing if others vote (poll?) to make cron run only cron jobs, I would go with majority ;-)

No, I would prefer most PR discussions not turn into polls.

This PR isn't urgent. I'll let it sit another day, and if no one else cares to weigh in, I'll drop the tip commit and merge.

kyleam

comment created time in 6 days

Pull request review commentdatalad/datalad

travis: Add job for python 3.8, restrict cron

 matrix: #    - UNSET_S3_SECRETS=1  before_install:-  - if [ ! -z "${_DL_CRON:-}" ]; then-      if [ ! ${TRAVIS_EVENT_TYPE} = "cron" ]; then-        echo "Exiting since should only be ran on a weekly basis by cron!";-        exit 0;+  - if [ "${TRAVIS_EVENT_TYPE}" = "cron" ]; then+      if [ -z "${_DL_CRON:-}" ]; then+          echo "Skipping regular job on weekly cron";+          exit 0;+      fi+    else+      if [ ! -z "${_DL_CRON:-}" ]; then+          echo "Exiting since should only be ran on a weekly basis by cron!";

Why?

For the reason I give in 2542dc60c: running everything inconveniently blocks the test queue. On the two days of the week that cron jobs run, I regularly hit into this.

Cron should still run all the other tests, eg if we don't push for a while and have a complete pass

With this change, it is of course possible that a branch is dormant and we see a failure related to, say, an updated dependency later than we otherwise would. Or more likely, we see the failure within one of the cron jobs, requiring some inspection to realize the failure isn't cron specific.

That's a trade I would happily make. But, if you and others don't agree, the last commit of this series can safely be dropped.

kyleam

comment created time in 6 days

PR merged datalad/datalad

ENH(TST): use assert_equal to see why conda python 3.8 tests fail

This is the only test left failing after adding distro (see #4659)

+1 -1

0 comment

1 changed file

yarikoptic

pr closed time in 8 days

push eventdatalad/datalad

Yaroslav Halchenko

commit sha 586a43a31ab3db75c8cebafa54bbc29c97a604dd

ENH(TST): use assert_equal to see why conda python 3.8 tests fail

view details

Kyle Meyer

commit sha d1345797b990febeea4a1eb03cd0e87baf1ed6b3

Merge pull request #4664 from yarikoptic/enh-open-files

view details

push time in 8 days

pull request commentdatalad/datalad-osf

Rough draft of a Git remote helper to store Git repositories in OSF projects

fatal: bad revision 'HEAD'

[...] I have no idea how or why is happens, but the issues goes away with a non-editable deployment via pip install ..

It's due to versioneer's logic that runs git underneath, in particular _version.git_pieces_from_vcs.

Here's a minimal example that triggers it.

<details> <summary>demo</summary>

#!/bin/sh

set -eu

cd "$(mktemp -d ${TMPDIR:-/tmp}/git-remote-chdir-XXXXXXX)"
git init src
(
    cd src
    cat >setup.py <<'EOF'
from setuptools import setup
setup(name="foo",
      entry_points={"console_scripts": ["git-remote-foo=foo.bar:main"]})
EOF

    mkdir foo
    cat >foo/__init__.py <<'EOF'
import os
import subprocess
subprocess.run(["git", "describe", "--always"], cwd=os.path.dirname(__file__))
EOF

    cat >foo/bar.py <<'EOF'
def main():
    assert 0
EOF

    git add foo setup.py
    git commit -m'c0'
)

python3 -m venv ./env
. ./env/bin/activate
pip install -e src

export GIT_TRACE2=1
export GIT_TRACE_SETUP=1
git clone foo://b c

</details>

My understanding, based on looking into git rebase -x <tests that invoke git> failures in the past, is that the proper fix is something like this:

diff --git a/scratch.sh b/scratch.sh
index 3045667..a4faaba 100755
--- a/scratch.sh
+++ b/scratch.sh
@@ -16,6 +16,12 @@ EOF
     cat >foo/__init__.py <<'EOF'
 import os
 import subprocess
+
+out = subprocess.run(["git", "rev-parse", "--local-env-vars"],
+                     capture_output=True, encoding="utf8")
+for ev in out.stdout.splitlines():
+    os.environ.pop(ev, None)
+
 subprocess.run(["git", "describe", "--always"], cwd=os.path.dirname(__file__))
 EOF
 

Or in the context of the _version.py:

diff --git a/datalad_osf/_version.py b/datalad_osf/_version.py
index 7ae8c4b..6d65063 100644
--- a/datalad_osf/_version.py
+++ b/datalad_osf/_version.py
@@ -225,7 +225,15 @@ def git_pieces_from_vcs(tag_prefix, root, verbose, run_command=run_command):
     if sys.platform == "win32":
         GITS = ["git.cmd", "git.exe"]
 
-    out, rc = run_command(GITS, ["rev-parse", "--git-dir"], cwd=root,
+    local_envs, rc = run_command(GITS, ["rev-parse", "--local-env-vars"],
+                                 cwd=root, hide_stderr=True)
+    if rc != 0:
+        raise NotThisMethod("'git rev-parse --local-env-vars' returned error")
+    env = os.environ.copy()
+    for ev in local_envs.splitlines():
+        env.pop(ev, None)
+
+    out, rc = run_command(GITS, ["rev-parse", "--git-dir"], cwd=root, env=env,
                           hide_stderr=True)
     if rc != 0:
         if verbose:
@@ -237,12 +245,12 @@ def git_pieces_from_vcs(tag_prefix, root, verbose, run_command=run_command):
     describe_out, rc = run_command(GITS, ["describe", "--tags", "--dirty",
                                           "--always", "--long",
                                           "--match", "%s*" % tag_prefix],
-                                   cwd=root)
+                                   cwd=root, env=env)
     # --long was added in git-1.5.5
     if describe_out is None:
         raise NotThisMethod("'git describe' failed")
     describe_out = describe_out.strip()
-    full_out, rc = run_command(GITS, ["rev-parse", "HEAD"], cwd=root)
+    full_out, rc = run_command(GITS, ["rev-parse", "HEAD"], cwd=root, env=env)
     if full_out is None:
         raise NotThisMethod("'git rev-parse' failed")
     full_out = full_out.strip()
@@ -294,12 +302,12 @@ def git_pieces_from_vcs(tag_prefix, root, verbose, run_command=run_command):
         # HEX: no tags
         pieces["closest-tag"] = None
         count_out, rc = run_command(GITS, ["rev-list", "HEAD", "--count"],
-                                    cwd=root)
+                                    cwd=root, env=env)
         pieces["distance"] = int(count_out)  # total number of commits
 
     # commit date: see ISO-8601 comment in git_versions_from_keywords()
     date = run_command(GITS, ["show", "-s", "--format=%ci", "HEAD"],
-                       cwd=root)[0].strip()
+                       cwd=root, env=env)[0].strip()
     pieces["date"] = date.strip().replace(" ", "T", 1).replace(" ", "", 1)
 
     return pieces
mih

comment created time in 9 days

pull request commentkyleam/bog

Use lexical-binding

If these changes are merged, the copyright years should be updated, right?

Yes, please. If you don't have a preference, I've come to like the style I first noticed in Guix where each contributor just adds their own, separate copyright line. I think the other option is to switch the single line over to using something like "Bog contributors".

basil-conto

comment created time in 9 days

Pull request review commentkyleam/bog

Use lexical-binding

 level `bog-refile-maxlevel' are considered."     (cdr (assoc-string (completing-read "File: " note-paths)                        note-paths)))) +(defvar bog--agenda-map+  (let ((map (make-sparse-keymap)))+    (set-keymap-parent map org-agenda-mode-map)+    (define-key map "r" 'bog-agenda-redo)+    (define-key map "g" 'bog-agenda-redo)+    map)+  "Local keymap for Bog-related agendas.")+ (defmacro bog--with-search-lprops (&rest body)   "Execute BODY with Bog-related agenda values. Restore the `org-lprops' property value for `org-agenda-redo-command' after executing BODY."-  (declare (indent 0))-  `(let ((org-lprops (get 'org-agenda-redo-command 'org-lprops))-         (bog-lprops '((org-agenda-buffer-name "*Bog search*")-                       (org-agenda-files (bog-notes))-                       org-agenda-text-search-extra-files-                       org-agenda-sticky)))-     (put 'org-agenda-redo-command 'org-lprops bog-lprops)-     (put 'org-agenda-files 'org-restrict nil)-     (org-let bog-lprops ,@body)-     (use-local-map (let ((map (make-sparse-keymap)))-                      (set-keymap-parent map org-agenda-mode-map)-                      (define-key map "r" 'bog-agenda-redo)-                      (define-key map "g" 'bog-agenda-redo)-                      map))-     (put 'org-agenda-redo-command 'org-lprops org-lprops)))+  (declare (indent 0) (debug t))+  (let ((bog-lprops '((org-agenda-buffer-name "*Bog search*")+                      (org-agenda-files (bog-notes))+                      org-agenda-text-search-extra-files+                      org-agenda-sticky))+        (org-lprops (make-symbol "org-lprops")))+    `(let ((,org-lprops (get 'org-agenda-redo-command 'org-lprops)))+       (unwind-protect+           (let ,bog-lprops+             (put 'org-agenda-redo-command 'org-lprops ',bog-lprops)+             (put 'org-agenda-files 'org-restrict nil)+             ,@body+             (use-local-map bog--agenda-map))+         (put 'org-agenda-redo-command 'org-lprops ,org-lprops)))))

I thought I recalled trying to switch over to lexical binding. Checking my local repo, I have a branch from 2015 that basically says "didn't get agenda stuff working".

Looks like you managed to get it working nicely. I did a good amount of manual testing around bog-search{,-for-citekey}, as well as the interaction with the regular agenda, and didn't spot any issues.

I also tested with the version you proposed in the comments and that also works well. Whichever you prefer is fine by me.

basil-conto

comment created time in 9 days

pull request commentconda-forge/datalad-feedstock

Rebuild for python38

and I don't see a way to restart that build.

I guess you can retrigger the whole thing with one of the methods listed at https://conda-forge.org/docs/maintainer/infrastructure.html#restarting-builds.

But since we now know that it is not 3.8 per se -- I would just merge as is.

Yeah, the latest 3.8 build passed, so indeed it seems like an unrelated and existing issue with flaky tests.

regro-cf-autotick-bot

comment created time in 9 days

push eventkyleam/datalad

Kyle Meyer

commit sha 0b7209711e00401500b288780b01cb52f5918090

TST: travis: Add job for python 3.8 Before Python 3.8 was released, we tested against it and fixed up the issues at the time, but it wasn't added as a regular job. Add it now, demoting the 3.6 job to a cron job so we don't increase our testing time. Use Bionic rather than Xenial as the distribution so that we can use a pre-installed 3.8 instead of 3.8-dev [0]. Note that we can't use this across all runs because Bionic doesn't come with Python 3.5 pre-installed. [0] https://docs.travis-ci.com/user/reference/bionic/#python-support

view details

Kyle Meyer

commit sha 2542dc60ce05b543539b232e2140ed22da844101

TST: travis: Skip regular jobs when running cron jobs Our weekly cron builds for master and maint take even longer than our current tests and can hold up the test queue. For these builds, the cron jobs are the ones of interest, so restrict the builds to those. Note that this will not affect the coverage reports because those aren't submitted for cron runs.

view details

push time in 9 days

issue commentdatalad/datalad

one failed test on conda with python 3.8

Looks like linux_python3.8.____cpython passed now on conda, and py3.8 passed at gh-4663, so perhaps a (hopefully rare) fluke.

yarikoptic

comment created time in 10 days

issue commentdatalad/datalad

one failed test on conda with python 3.8

I will try to trigger it locally and also work on a PR for switching at least one Travis run to py3.8

No luck triggering it locally. Travis addition is at gh-4663.

yarikoptic

comment created time in 10 days

PR opened datalad/datalad

travis: Add job for python 3.8, restrict cron
  • gh-4659 pointed to a failure in the conda py3.8 runs. Before Python 3.8 was released, we tested against 3.8 on Travis and fixed up the issues at the time, but we didn't add as a regular job. Add it now, demoting the 3.6 job to a cron job so we don't increase our testing time.

  • Restrict the cron builds to _DL_CRON=1 jobs.

+16 -5

0 comment

2 changed files

pr created time in 10 days

create barnchkyleam/datalad

branch : travis-py38

created branch time in 10 days

issue commentdatalad/datalad

swarm of failed tests on conda with python 3.8

Looks like it's down to one failure. I will try to trigger it locally and also work on a PR for switching at least one Travis run to py3.8, perhaps moving an older version to a cron build. We've tested with py3.8 in the past (e.g., gh-3430), but only with temporary builds that were pruned before merge.

FAIL: datalad.tests.test_utils.test_get_open_files
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/conda/feedstock_root/build_artifacts/datalad_1593096801880/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.8/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/home/conda/feedstock_root/build_artifacts/datalad_1593096801880/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.8/site-packages/datalad/tests/utils.py", line 541, in newfunc
    return t(*(arg + (d,)), **kw)
  File "/home/conda/feedstock_root/build_artifacts/datalad_1593096801880/_test_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehol/lib/python3.8/site-packages/datalad/tests/test_utils.py", line 1215, in test_get_open_files
    assert not get_open_files(str(subd))
AssertionError

https://github.com/conda-forge/datalad-feedstock/pull/34/checks?check_run_id=807773405

yarikoptic

comment created time in 10 days

issue commentdatalad/datalad

swarm of failed tests on conda with python 3.8

Would you mind if I add you @kyleam and @mih as maintainers to conda package of datalad so you could push fixes etc?

Works for me. Thanks.

yarikoptic

comment created time in 10 days

issue commentdatalad/datalad

swarm of failed tests on conda with python 3.8

I haven't looked through all of those failures, but it seems like the first step would be to add distro as a dependency to the conda recipe. Our setup.py specifies distro as a core dependency for py3.8 and above.

yarikoptic

comment created time in 10 days

delete branch kyleam/datalad

delete branch : get-inherit-reckless

delete time in 10 days

push eventdatalad/datalad

Yaroslav Halchenko

commit sha 709ce89b14c1eac8efb29359af6738cfa2ccbb94

BF: skip running testremote with recent not yet fixed up git-annex See the bugreport https://git-annex.branchable.com/bugs/testremote_breeds_way_too_many_instances_of_the_externals_remote/?updated for more information.

view details

Yaroslav Halchenko

commit sha 25a7ee2a4c5f8d357a8771e17007c3d9861ce7bc

BF: annotate tests in test_utils_cached_dataset with @skip_if_no_network Since they all rely on access to a github repository ATM. See https://github.com/datalad/datalad/issues/4653 for a possible RF to avoid that

view details

Yaroslav Halchenko

commit sha 2f9a9f008e345ce475f33c278afdad51abb54aee

BF(TST): skip one case of test_ria_basics.test_binary_data if no network is available

view details

Kyle Meyer

commit sha bee63ce98779bc4090529230e5640a107ad39394

Merge pull request #4652 from yarikoptic/bf-testremote

view details

Kyle Meyer

commit sha 3074c1e31a7c1347d7a97786dd5b3eab88fd6abe

BF: get: Inherit reckless configuration when cloning subdatasets Cloning a subdataset is supposed to inherit the parent's `datalad.clone.reckless` value. This works as intended when using install() to get an existing dataset or using clone() to create a new subdataset, but the parent's configuration is ignored when the clone is triggered by get(). The "inherit reckless" logic is handled in Clone.__call__(), but that's upstream of clone_dataset(), where get() hooks into. Move the handling into clone_dataset() to avoid repeating it in get(). Note that we don't have access to the parent dataset in clone_dataset(), but we need to query its configuration. We could add the parent dataset as a parameter. However, there's already a cfg parameter, and Clone.__call__()---the only caller that specifies cfg---passes the parent's configuration. So instead just update get() to pass in cfg and query that, adjusting the documentation to note that cfg should be the parent's. Closes #4655.

view details

Kyle Meyer

commit sha be643aaf08d468d99c81c5de0045e80e48e49a10

Merge pull request #4654 from yarikoptic/bf-skip-no-network

view details

Michael Hanke

commit sha 1a2efe68e801cfa8d70d6e9977acebfda0654141

Merge pull request #4657 from kyleam/get-inherit-reckless BF: get: Inherit reckless configuration when cloning subdatasets

view details

Kyle Meyer

commit sha 67620e51d459c531cdc895a1c312f8cdb73e2387

Merge branch 'maint'

view details

push time in 10 days

issue closeddatalad/datalad

clone --reckless ephemeral does not inherit

% datalad clone --reckless ephemeral /data/project/enki/processed ds
% cd ds
% datalad get -n code/pipelines fmriprep
% git config -l |grep reckless
datalad.clone.reckless=ephemeral
% git -C fmriprep config -l |grep reckless

and the subdataset annex is also not linked. This is with 0.13

closed time in 10 days

mih

issue commentmagit/magit

magit-worktree-branch broken in combination with helm

This bisects to Helm's b9aa6513 (Make helm-read-file-name working with new HFF caching, 2020-06-21). Underneath magit-worktree-branch is calling magit-worktree-read-directory-name-function, which defaults to read-directory-name.

Taking Magit out of equation, this can be triggered with

(read-directory-name "dir: ")  ; then input non-existent directory

Given that (1) MUSTMATCH is nil in the above call and (2) this just recently started happening in the unreleased commit above that gives no hint about such a change in behavior, I'd say this is a regression on Helm's end.

Could you please open an issue over there?

simonpintarelli

comment created time in 10 days

issue commentmagit/magit

Interactive Rebase and Empty Commit Messages

Fixed. Thanks for the report and for providing a minimal example.

whxvd

comment created time in 10 days

push eventmagit/magit

Kyle Meyer

commit sha 3785becaf52dc61f195fdf73d3a8dcb897ee0dcd

magit-status-use-buffer-arguments: Fix doc-string typo

view details

Kyle Meyer

commit sha 7f486d47fe7042edba8d9a855a4c2c301a30bc6b

git-rebase-line-regexps: Fix handling of empty commit messages If a commit is made with an empty commit message, the commit regexp fails to match the git-rebase-todo line because these lines lack a space after the hash. Make the space optional, as the label and merge regexps already do. Fixes #4153.

view details

push time in 10 days

issue closedmagit/magit

Interactive Rebase and Empty Commit Messages

On current master (v2.90.1-1015-g0123e377) and running Emacs with what magit-emacs-Q-command produces.

In an empty directory, run:

git init
git commit --allow-empty -m0
touch 1; git add .; git commit -m1
touch 2; git add .; git commit -m2
touch 3; git add .; git commit --allow-empty-message -m ''
touch 4; git add .; git commit --allow-empty-message -m ''

Then, in Magit, trying to interactively rebase HEAD on the commit with message 0 leads to a git-rebase-todo buffer starting with:

pick 3e0ca4b 0 # empty
pick 073b6dc 1
pick 1882401 2
pick 3d4961b
pick ca20e39

However, the last two pick lines receive no fontification. And it is not possible to run commands like M-p or m on them.

Empty commits get a comment # empty at the end. Maybe this issue can be resolved by just adding something similar for commits with empty commit messages. E.g. # empty commit instead of just # empty. And # empty commit message and # empty commit, empty commit message in those cases.

closed time in 10 days

whxvd

issue commentdatalad/datalad

addurls: _url_basename is documented but unknown?

The error reporting here should be improved, but this is due to two URLs in that file lacking a path, leading to the path-derived _url_* fields being undefined.

https://cdn-akamai.6connex.comet:c208196
https://cdn-akamai.6connex.comet:c212253
yarikoptic

comment created time in 10 days

pull request commentdatalad/datalad

BF: get: Inherit reckless configuration when cloning subdatasets

The failing appveyor builds are due to a codecov issue (where a good chunk of the error message is helpfully reported one character per line).

kyleam

comment created time in 10 days

PR merged datalad/datalad

BF: annotate tests in test_utils_cached_dataset with @skip_if_no_network

Since they all rely on access to a github repository ATM. See https://github.com/datalad/datalad/issues/4653 for a possible RF to avoid that

+7 -2

3 comments

2 changed files

yarikoptic

pr closed time in 10 days

push eventdatalad/datalad

Yaroslav Halchenko

commit sha 25a7ee2a4c5f8d357a8771e17007c3d9861ce7bc

BF: annotate tests in test_utils_cached_dataset with @skip_if_no_network Since they all rely on access to a github repository ATM. See https://github.com/datalad/datalad/issues/4653 for a possible RF to avoid that

view details

Yaroslav Halchenko

commit sha 2f9a9f008e345ce475f33c278afdad51abb54aee

BF(TST): skip one case of test_ria_basics.test_binary_data if no network is available

view details

Kyle Meyer

commit sha be643aaf08d468d99c81c5de0045e80e48e49a10

Merge pull request #4654 from yarikoptic/bf-skip-no-network

view details

push time in 10 days

pull request commentdatalad/datalad

BF: annotate tests in test_utils_cached_dataset with @skip_if_no_network

Crippled FS failure looks unrelated, but then: How does it not happen in PR #4652 ?

And on rerun, a different failure, so I guess count these towards the spurious failures mentioned at gh-4621.

yarikoptic

comment created time in 10 days

PR opened datalad/datalad

BF: get: Inherit reckless configuration when cloning subdatasets
Cloning a subdataset is supposed to inherit the parent's
`datalad.clone.reckless` value.  This works as intended when using
install() to get an existing dataset or using clone() to create a new
subdataset, but the parent's configuration is ignored when the clone
is triggered by get().

The "inherit reckless" logic is handled in Clone.__call__(), but
that's upstream of clone_dataset(), where get() hooks into.  Move the
handling into clone_dataset() to avoid repeating it in get().

Note that we don't have access to the parent dataset in
clone_dataset(), but we need to query its configuration.  We could add
the parent dataset as a parameter.  However, there's already a cfg
parameter, and Clone.__call__()---the only caller that specifies
cfg---passes the parent's configuration.  So instead just update get()
to pass in cfg and query that, adjusting the documentation to note
that cfg should be the parent's.

Closes #4655.

+41 -9

0 comment

3 changed files

pr created time in 10 days

create barnchkyleam/datalad

branch : get-inherit-reckless

created branch time in 10 days

issue commentdatalad/datalad

clone --reckless ephemeral does not inherit

@bpoldrack Yeah, I haven't taken a closer look yet, but that all sounds right. Don't read too much into the quick and dirty patch. It came with no investigation.

mih

comment created time in 11 days

issue commentdatalad/datalad

clone --reckless ephemeral does not inherit

Confirmed on my end. I didn't spot a test that checks that datalad get on a subdataset inherits the reckless config. The closest I see is a test_clone check that cloning underneath a dataset inherits from the dataset:

https://github.com/datalad/datalad/blob/e730298044c36c8418d8c927ee6abd0a32e3a496/datalad/core/distributed/tests/test_clone.py#L335-L339

Something like this addresses the issue (at least for the demo below):

diff --git a/datalad/distribution/get.py b/datalad/distribution/get.py
index 014105768..ae1f539b6 100644
--- a/datalad/distribution/get.py
+++ b/datalad/distribution/get.py
@@ -785,6 +785,7 @@ def __call__(
                          if sdsres.get('state', None) == 'absent'
                          else sdsres['path'])
             assert 'contains' in sdsres
+            reck = reckless or ds.config.get('datalad.clone.reckless', None)
             # explore the unknown
             for target_path in sdsres.get('contains', []):
                 # essentially the same as done above for paths in the root
@@ -795,7 +796,7 @@ def __call__(
                         Path(target_path),
                         recursive,
                         recursion_limit,
-                        reckless,
+                        reck,
                         refds_path,
                         description):
                     known_ds = res['path'] in content_by_ds

I'll look into turning that into a proper patch later today.

<details> <summary>demo</summary>

cd "$(mktemp -d ${TMPDIR:-/tmp}/dl-XXXXXXX)"
datalad create a
datalad create -d a a/sub
echo d >a/sub/dat
datalad save -d a -r

datalad clone --reckless=ephemeral a b
(
    cd b
    datalad get -n sub
    ls -al sub/.git/annex
    git config --list | grep reckless
    git -C sub config --list | grep reckless
)

</details>

mih

comment created time in 11 days

delete branch datalad/datalad

delete branch : scratch/unborn-adjusted-merge

delete time in 11 days

delete branch datalad/datalad

delete branch : scratch/min-git-annex-version-testing

delete time in 11 days

push eventdatalad/datalad

Andy Connolly

commit sha e6dfea698861e5fc03eee897868b4a2023259b8f

DOC: Fixed documentation builder to allow for overriding opts in eval_defaults. Closes #2762

view details

Andy Connolly

commit sha 97ac4a913c2baec1e3da0c6341e4d9d7d0df016a

Merge branch 'master' of git://github.com/datalad/datalad into iss2762

view details

Kyle Meyer

commit sha 1b8e780e2276384515b084ee8ae6925f7f68efe3

DOC: build_doc: Document eval_params/_params_ override behavior containers-run builds ContainersRun._params_ from Run._params_ to extend run's interface with container-specific parameters. In doing so, it (presumably unintentionally) populates ContainersRun._params_ with parameters from eval_params because build_doc() modifies Run._params_ in place. Having eval_param keys in ContainersRun._params_ doesn't currently cause any issues because build_doc() does a blanket update of _params_ with eval_params, but a proposed change to build_doc() in gh-4480 shows how this can fail. Explicitly document this behavior to make it clear to callers that they can rely on it and to prevent future build_doc() tweakers from unknowingly changing it. Supersedes datalad/datalad-container#119.

view details

Andy Connolly

commit sha d17a0cd0bf437ce2563f893a6fd1c7138b6b3c4f

DOC: corrects previous fix for defaults This should Close #2762 My previous fix added results_renderer to the _params_ in interface.Siblings which wasn't necessary as Kyle pointed out. I have incorporated his suggestions, and it seems to work. Thanks

view details

Yaroslav Halchenko

commit sha b1bdf36c31ff0b02234c008068efabb9c2f1ed25

ENH: Travis - add cron job to run tests against some git-annex which we claim to support

view details

Yaroslav Halchenko

commit sha 9f9578088e4910f4057e2aa3d69584dad22ed117

Merge branch 'enh-tst-min-gitannex' into maint * enh-tst-min-gitannex: ENH: Travis - add cron job to run tests against some git-annex which we claim to support

view details

Kyle Meyer

commit sha 5e10bbb50620dd09a1204d907943727b2a9c0c56

DOC: addurls: Remove a stray ')' [ci skip]

view details

Michael Hanke

commit sha 548dd78b6435810479ec2efdf69a547318faad0d

ENH: Overall progressbar for AnnexJsonProtocol Until now it would only create individual progress bars for items that git-annex would report progress about. This change adds an additional global progress bar that counts the number of results, if a total number of expected results was provided. This fixes gh-4501 Or rather, it fixes the underlying desire to see something happen, it won't change how and when results are emitted.

view details

Kyle Meyer

commit sha 28b5c264dd80c42e1ac2690fe4e22018b728b3cd

TST: addurls: Avoid leakage between two subsequent checks test_addurls_unbound_dataset() looks at relative path handling, and one of its checks asserts that the expected files exist. But the same dataset is used across two calls to addurls(), which means the check for the second call could pass due to the previous call. Use a separate dataset for each call.

view details

Kyle Meyer

commit sha 0dd7cb8d3ec8d8e8bea2f3706b6e815166640c67

BF: addurls: Don't trip over "./" in the file name format As described in d46846716 (DOC: addurls: Clarify what filename format is relative to, 2019-08-02) and the tests, the file format is intended to be taken relative to the dataset. However, when the file name format begins with "./", we keep "./" on the relative name, while constructing the absolute file name without the subdirectory as documented. The relative file name is passed to AnnexRepo.addurl(), causing the file to be treated as relative to the current working directory and downloaded there. Achieve the documented behavior by applying the same handling to the top-level dataset as we use for subdatasets. Note that there is mention of changing this relative path handling in gh-4498. Even if we decide that this change in behavior is worthwhile, it still makes sense to fix this bug, and doing so doesn't make the future change harder.

view details

Michael Hanke

commit sha 79923d57b73461444b010537a5690716af370f09

BF: Avoid repeated reinit of global progressbar before first result completion Also move a few things into the constructor that need only run once.

view details

Kyle Meyer

commit sha 3c86d370e2632312d3face148aaa9ba4800c1dd6

TST: archive: Skip tests that depend on p7zip All of the checks of the test generator fail on a system without p7zip. Before gh-4041 (in particular, before beb5d045e), the subset of checks that existed at that time was skipped with (see gh-3176): SKIP: cmd:7z is missing. (Not) Funny enough but ATM we need p7zip installation to handle .gz files extraction 'correctly' However, with the change to the archive file name in 05e9353d7 (TST: Expand test coverage to additional relevant compression formats, 2020-01-17), the MissingExternalDependency exception that led to the skip for gzip formats is no longer triggered. And even if it were, the other file formats tested would fail. Mark these tests as known failures when p7zip isn't available.

view details

Michael Hanke

commit sha 7b61629d6e84168a346a7e2b94bcb50657468fe7

Merge pull request #4503 from mih/bf-4501 ENH: Overall progressbar for AnnexJsonProtocol

view details

Michael Hanke

commit sha 1ff7f5bf62159cd443aebe1b5a33b510c76485a3

DOC: Announce deprecation of `publish` Fixes gh-4493

view details

Michael Hanke

commit sha 7bacd57e5af4248573c095eb975ac5707a45395c

DOC: Strip outdated extension list and refer to handbook The Handbook URL anticipates a change in the handbook that will come shortly. Fixes gh-4490

view details

Michael Hanke

commit sha 65530feec4fca2d1d9060eed7cc4190ebc38ce05

BF: Fail early on push to unknown target. % datalad push --to mike [ERROR ] Unknown push target 'mike'. No targets configured in dataset. [push.py:__call__:200] (ValueError) % datalad push --to mike [ERROR ] Unknown push target 'mike'. Known targets: 'localtmp'. [push.py:__call__:200] (ValueError) Fixes gh-4506

view details

Benjamin Poldrack

commit sha 613a1a1e95f7bd8513a9d0b78ceafb90dcb52def

TST: Use binary data from github prior version of test depends on accessibility of an annex remote hosted in Jülich. Seems to have some kind of connection issue. Data for this test, however, is completely irrelevant as long as it's binary. (Closes #4371)

view details

Benjamin Poldrack

commit sha eda083f71ae1cb6976f7feffb095fdd0bc12caae

TST: Skip too slow test on crippled FS Even a local git-annex-testremote is just too slow on the crippled FS used by this Github workflow. Will time out if the test is executed.

view details

Benjamin Poldrack

commit sha 4c708e987ccb1bd3523104cfe240c93758c5e0da

Merge pull request #4518 from bpoldrack/fix-4371 Fixing issue 4371 (fragile test case)

view details

Yaroslav Halchenko

commit sha f03c67ae273b831fee4f135ccaab37776f6ed632

BF(DOC): strip away RF leftover Parameters in docstring for no_annex.py [ci skip]

view details

push time in 11 days

PR merged datalad/datalad

BF: skip running testremote with recent not yet fixed up git-annex

See the bugreport

https://git-annex.branchable.com/bugs/testremote_breeds_way_too_many_instances_of_the_externals_remote/?updated

for more information.

+7 -0

1 comment

1 changed file

yarikoptic

pr closed time in 11 days

push eventdatalad/datalad

Yaroslav Halchenko

commit sha 709ce89b14c1eac8efb29359af6738cfa2ccbb94

BF: skip running testremote with recent not yet fixed up git-annex See the bugreport https://git-annex.branchable.com/bugs/testremote_breeds_way_too_many_instances_of_the_externals_remote/?updated for more information.

view details

Kyle Meyer

commit sha bee63ce98779bc4090529230e5640a107ad39394

Merge pull request #4652 from yarikoptic/bf-testremote

view details

push time in 11 days

Pull request review commentdatalad/datalad

BF: skip running testremote with recent not yet fixed up git-annex

 def _test_gitannex(host, store, dspath):     init_opts = common_init_opts + ['url={}'.format(store_url)]     ds.repo.init_remote('store', options=init_opts) +    from datalad.support.external_versions import external_versions+    if '8.20200330' < external_versions['cmd:annex'] < '8.20200624':+        # https://git-annex.branchable.com/bugs/testremote_breeds_way_too_many_instances_of_the_externals_remote/?updated

Looks good to me. Just for the record: Joey said it happened around 9fa940569c7941ad794637017ae5296e657c7f8d. The first release to contain that commit is 8.20200501.

yarikoptic

comment created time in 11 days

push eventdatalad/datalad

Kyle Meyer

commit sha 26c0d96b8527b8ecab57316954a0f8183ed15d57

DOC: annexrepo: Add note about issue with older Git versions In tests with Git v2.9 (which is the oldest Git version that will build for me on Debian Buster), test_AnnexRepo_file_has_content() fails: File "[...]/datalad/support/tests/test_annexrepo.py", line 318, in test_AnnexRepo_file_has_content [ar.supports_unlocked_pointers]) AssertionError: [False] != [True] The failure is coming from the 'git diff' call unexpectedly reporting a staged type change as modified. Adding a 'git status' call before the 'git diff' call is enough to convince Git that there is no modification, so perhaps it's an issue related to not triggering the clean filter. This failure disappears with Git v2.13.

view details

Kyle Meyer

commit sha 3401b7a386c210b04c86cce348ea5297b7dce690

TST: gitrepo: test_duecredit: Loosen assumption about output string test_duecredit() expect the output to be entirely empty when duecredit doesn't emit anything. At least one time when that isn't the case is when running with an older Git: AssertionError: '[...]/datalad/support/gitrepo.py:978: OutdatedExternalDependencyWarning: Your git version (2.13.0) is too old, we will not safe-guard against creating a new repository under already known to git subdirectory\n OutdatedExternalDependencyWarning\n' != ''

view details

Kyle Meyer

commit sha ac24b5e0e40fdff422bdfb7c7c9e6979edd3b28d

DOC: OutdatedExternalDependency: Avoid extra period in str output MissingExternalDependency already tacks on a period to its str value, so unless the caller provides more text via msg, the output has an extra period: datalad.support.exceptions.OutdatedExternalDependency: cmd:git of version >= 2.19.1 is missing.. You have version 2.13.0 An extra period could also end up in the output when a caller specifies msg, but none of the callers at the moment end their message with a period.

view details

Kyle Meyer

commit sha f049e6d32e2ffc58bdd420e6d78aaee884c24aba

TST: config: Skip test when Git version is too old 'git config' doesn't raise the expected error until 2.13.2, specifically 25cd291963 (config: complain about --local outside of a git repo, 2017-05-12).

view details

Kyle Meyer

commit sha 8db852a32ea30d987ef1fdea6d2b8d93f24593c5

TST: update: Skip test that requires Git v2.13.2 This test depends on a change that came with v2.13.2: fdb69d33c4 (fetch-pack: always allow fetching of literal SHA1s, 2017-05-15).

view details

Kyle Meyer

commit sha ff042b88a5d792385518250ffc2f4a2460e91de9

ENH: gitrepo: Declare and check minimum Git version We do not specify a minimum supported Git version, which makes it difficult for us to decide whether we can use a newer feature and means that users have to rely on trial and error to know whether their version is recent enough. Set a minimum Git version, handling it in a way that closely follows what AnnexRepo already does for the minimum git-annex version. Choose 2.19.1 (September 2018) as the minimum because that (1) matches what conda currently has for its 32-bit Linux builds [0] and (2) is below what is currently in Debian stable (v2.20.1). Note that if we decide to lower the minimum more, the hard floor should probably be considered v2.13.0 (May 2017). As described a few commits back, AnnexRepo._check_files() doesn't work reliably until then and testing with v2.13.0 didn't reveal major problems [1], though the skips added in the two previous commits were needed. [0] https://github.com/datalad/datalad/issues/4636#issuecomment-645633006 [1] https://travis-ci.org/github/datalad/datalad/builds/701067382 Closes #4636.

view details

Kyle Meyer

commit sha 629e575628ba8b53bf073392837c9e4b49444838

MNT: tree-wide: Prune workarounds for unsupported Git versions With the previous commit, we now specify and verify a minimum Git version of 2.19.1.

view details

Kyle Meyer

commit sha f4b9e6bd63e66ce466e5a2a24a92d9637f0b2efd

TST: Use GitRepo.git_version instead of external_versions external_versions["cmd:git"] is now stored as GitRepo.git_version. Use that in a few tests.

view details

Kyle Meyer

commit sha 96cf43e3823c7d8b206f5b841bada11917918e8e

TST: travis: Add cron build to test with our minimum Git version

view details

Kyle Meyer

commit sha e730298044c36c8418d8c927ee6abd0a32e3a496

Merge pull request #4650 from kyleam/min-git-version

view details

push time in 11 days

PR merged datalad/datalad

Declare minimum supported Git version

This series sets a minimum Git version (currently v2.19.1, as suggested in gh-4636), and adds a cron build to test against it.


  • [X] Make sure run with v2.19.1 passes and then drop the tip commit.

    v2.19.1 will hopefully pass because v2.13.0 fared okay, though the tests needed a couple of skips for behavior that changed with v2.13.2. The idea with that test is to get an idea of what our functional minimal version is at this point, regardless of whether we choose a higher floor. I chose v2.13.0 because it is the earliest version that I didn't find issues with when testing locally.

    https://travis-ci.org/github/datalad/datalad/builds/701074711

  • [x] Decide where to introduce the minimal version. As I mentioned in gh-4636, I'd prefer not to introduce a hard floor in a bug fix release, as it runs the risk of making datalad not work at all for users where a subset of functionality may have been working fine. And we've gone a long time without specifying a minimum requirement.

    My current thinking is that we should target this series for 0.13.1. Instead of aborting if an unsupported version is detected, we should just tell the user via warnings.warn that v2.19.1 (or whatever version we end up deciding to go with) is the oldest version we test against. If their version is below v2.13.0, we might want to give a bit stronger warning, given there are known issues at that point.

    Then, on the master branch (so targeting 0.14), we can switch the warning over to the external_versions.check call that is currently in this PR.

+52 -35

3 comments

9 changed files

kyleam

pr closed time in 11 days

issue closeddatalad/datalad

We need to establish a minimal supported git version and puke informatively

relates: #4535

See https://github.com/CONP-PCNO/conp-portal/issues/318 where @tashrifbillah gets

(base) [tb571@pnl-z840-2 tmp]$ datalad install https://github.com/CONP-PCNO/conp-dataset.git

CommandError: command '['git', 'for-each-ref', '--format=%(refname:strip=2)', 'refs/heads', 'refs/remotes']' failed with exitcode 128
Failed to run ['git', 'for-each-ref', '--format=%(refname:strip=2)', 'refs/heads', 'refs/remotes'] under '/tmp/conp-dataset'. Exit code=128. out= err=fatal: unknown refname: format strip=2

fatal: unknown refname: format strip=2

so we use unsupported by his version --format.

Ideally we should run a sweep of our tests with that minimal version, similarly to how we do with minimalish git-annex now on cron job.

closed time in 11 days

yarikoptic

pull request commentdatalad/datalad

Declare minimum supported Git version

Looks good to me, thanks!

Thanks for taking a look.

I wouldn't bother with warnings at different versions/ levels and just have exception (as now) . Our minimal version is indeed far back enough

Yeah, perhaps it's true that our minimum is low enough, though I'm less confident that people aren't running with older git versions. The base of this PR is suitable for both master (now targeting 0.14.0) and maint (now targeting 0.13.1). So, given nobody else has chimed in since gh-4636 was opened with other suggestions for a minimum version, I'll merge this into master now. With the base where it is, we can easily merge this into maint as well if we decide to, though if we do so I still lean towards switching to a warning there.

kyleam

comment created time in 11 days

issue closeddatalad/datalad

git 2.27.0~rc2-1 seems to break GitPython (2.26.2-1) ok

Tried to build 0.12.7 for debian and it failed with many failures like

======================================================================
ERROR: datalad.distribution.tests.test_update.test_update_git_smoke
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/build/datalad-0.12.7/.pybuild/cpython3_3.8_datalad/build/datalad/tests/utils.py", line 732, in newfunc
    return t(*(arg + (filename,)), **kw)
  File "/build/datalad-0.12.7/.pybuild/cpython3_3.8_datalad/build/datalad/tests/utils.py", line 732, in newfunc
    return t(*(arg + (filename,)), **kw)
  File "/build/datalad-0.12.7/.pybuild/cpython3_3.8_datalad/build/datalad/distribution/tests/test_update.py", line 181, in test_update_git_smoke
    target.update(recursive=True, merge=True), 1,
  File "/build/datalad-0.12.7/.pybuild/cpython3_3.8_datalad/build/datalad/distribution/dataset.py", line 500, in apply_func
    return f(**kwargs)
  File "/build/datalad-0.12.7/.pybuild/cpython3_3.8_datalad/build/datalad/interface/utils.py", line 492, in eval_func
    return return_func(generator_func)(*args, **kwargs)
  File "/build/datalad-0.12.7/.pybuild/cpython3_3.8_datalad/build/datalad/interface/utils.py", line 480, in return_func
    results = list(results)
  File "/build/datalad-0.12.7/.pybuild/cpython3_3.8_datalad/build/datalad/interface/utils.py", line 399, in generator_func
    for r in _process_results(
  File "/build/datalad-0.12.7/.pybuild/cpython3_3.8_datalad/build/datalad/interface/utils.py", line 527, in _process_results
    for res in results:
  File "/build/datalad-0.12.7/.pybuild/cpython3_3.8_datalad/build/datalad/distribution/update.py", line 174, in __call__
    for fr in _update_repo(ds, sibling_, reobtain_data):
  File "/build/datalad-0.12.7/.pybuild/cpython3_3.8_datalad/build/datalad/distribution/update.py", line 240, in _update_repo
    repo.pull(remote=remote, refspec=active_branch)
  File "/build/datalad-0.12.7/.pybuild/cpython3_3.8_datalad/build/datalad/support/gitrepo.py", line 2243, in pull
    return self._call_gitpy_with_progress(
  File "/build/datalad-0.12.7/.pybuild/cpython3_3.8_datalad/build/datalad/support/gitrepo.py", line 2209, in _call_gitpy_with_progress
    ret = callable(**git_kwargs)
  File "/usr/lib/python3/dist-packages/git/remote.py", line 812, in pull
    res = self._get_fetch_info_from_stderr(proc, progress)
  File "/usr/lib/python3/dist-packages/git/remote.py", line 708, in _get_fetch_info_from_stderr
    output.extend(FetchInfo._from_line(self.repo, err_line, fetch_line)
  File "/usr/lib/python3/dist-packages/git/remote.py", line 708, in <genexpr>
    output.extend(FetchInfo._from_line(self.repo, err_line, fetch_line)
  File "/usr/lib/python3/dist-packages/git/remote.py", line 292, in _from_line
    raise ValueError("Failed to parse line: %r" % line)
ValueError: Failed to parse line: '  git config pull.rebase false  # merge (the default strategy)'

so I guess for now will build for everything else by recent debian sid

edit FWIW

  • bug known to debian: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=961702 but I do not think was forwarded upstream, neither reassigned to git - just told that GitPython should not do that (parsing)
  • I had tested current master of git -- the same trouble. bisecting now from v2.26.2

closed time in 11 days

yarikoptic

issue commentdatalad/datalad

git 2.27.0~rc2-1 seems to break GitPython (2.26.2-1) ok

I see you already fixed this upstream and that made it into a release. I'm assuming that's sufficient to close this (especially given the remaining uses of gitpython have been pruned in 0.13), but please reopen if I'm wrong.

yarikoptic

comment created time in 11 days

issue closeddatalad/datalad

push: yet another round on the interface, but with a new twist

Decided to not pollute #4581 or any other prior discussion thus filed separately

Current --force options are already non orthogonal (impossible to use no-datatransfer and datatransfer together). I think it would be great to reserve --force for tune ups to specific git or annex actions (like gitpush already and possibly no-annexfast #4603) while making a dedicated option to define the data transfer mode, and just change the default to follow @mih's desire for the new "right thing" behavior (#4551) to copy annexed data by default. Moreover, while going through such RFing - we gain a chance to align it with git-annex a bit more.

What if we add the same option as publish has -- --transfer-data with following possible values:

  • (default) "anything" -- transfer any file not available on remote
  • (previous publish behavior) "nothing" -- transfer nothing
  • "auto" -- pass --auto to annex copy to satisfy "wanted" expression for the remote if any, and otherwise transfer nothing (note: it would also respect a possible numcopies setting).

"anything" and "nothing" from above - are wanted expressions of annex. Further documentation could point to --annex-wanted of create-siblings (and may be siblings if exposed there) to hint on how it could be tuned to automate.

--fast will be used, and could be removed via --force option.

I do like an idea of configurable default push data transfer behavior discussed in #4581 .

Later we could expand the list if any additional specific desire, but specific for data transfer, not fusing it into the list of forces.

Alternative to --transfer-data MODE is to align with current get and its -n|--no-data, and also introduce -a|--auto-data.

edit1: "annex-auto" -> "auto", and added get inspired alternative

closed time in 11 days

yarikoptic

issue commentdatalad/datalad

push: yet another round on the interface, but with a new twist

I think we can consider this closed with the merge of #4620. Please reopen if I'm mistaken.

yarikoptic

comment created time in 11 days

issue closeddatalad/datalad

Q: Short(er) option for --transfer-data?

Decided to bring to discussion outside of #4620 to not pollute it.

publish was never blessed with a short version for --transfer-data option. --force had -f which made it possible to say -f datatransfer.

  • install has -g|--get-data
  • get has -n|--no-data

I wonder if we could/should decide on a short option for --transfer-data going forward. May be eventually it could even become common (there is no reason for get to not support auto* modes) and listed among common options of the cheat sheet.

  • Candidates such as -D, -d are already taken.
  • -t is also used in a wide variety of contexts without any consistency.
  • The only one relating to "transfer" left seems to be -T but I think it is more about "data" than "transfer" so I do not like it either.

Altogether - I do not see any short option :-/

--data

Orthogonal alternative compromise is to introduce --data instead of --transfer-data in #4620 , which would at least be shorter and easier to remember, e.g. datalad push --data anything --to myremote.

While at it also deprecate --transfer-data in publish (in favor of --data) if that one to survive. That would also "assist" with conscious migration from publish --transfer-data all to publish --data anything.

Later we could RF/enhance get to make it default to get --data=auto-if-wanted and use get --data=nothing instead(?) of get -n, making interfaces more consistent.

closed time in 11 days

yarikoptic

issue commentdatalad/datalad

Q: Short(er) option for --transfer-data?

gh-4620 has been merged.

yarikoptic

comment created time in 11 days

PR merged datalad/datalad

RM deprecated (promoted to .core.local.run) in 0.12 release interface.run and saver option
  • datalad-container is expected to fail, PR there https://github.com/datalad/datalad-container/pull/120
+1 -35

6 comments

2 changed files

yarikoptic

pr closed time in 11 days

push eventdatalad/datalad

Yaroslav Halchenko

commit sha 238145806c2357ddc8d8e1e80b4f6689a43b82cf

RM: datalad.interface.run - was deprecated in 0.12.0rc4-51-g8fb74ed18

view details

Yaroslav Halchenko

commit sha ec4afad564d15f925a5716be070f4be904cc1705

RM: run(..., saver=None) - was deprecated and ignored as of 0.12.0rc6~30^2~4

view details

Kyle Meyer

commit sha a655d9f99ad6d11b3a0b1ce80dd081f511d4f965

Merge pull request #4583 from yarikoptic/rm-deprecated

view details

push time in 11 days

release datalad/datalad-container

1.0.1

released time in 11 days

created tagdatalad/datalad-container

tag1.0.1

DataLad extension for containerized environments

created time in 11 days

push eventdatalad/datalad-container

Kyle Meyer

commit sha 634ac22d0fb42d6b3d75655d14ff3e3f2d61ed94

MNT: Release 1.0.1

view details

push time in 11 days

PR merged datalad/datalad

ENH: join_cmdline to complement split_cmdline which uses quote_cmdlinearg enhancement

Spotted the need while reviewing #4586

from the 2nd commit message:

Also includes a change to output produced by

     def _format_cmd_shorty(cmd):
         """Get short string representation from a cmd argument list"""

which used to join then without quoting

     cmd_shorty = (' '.join(cmd) if isinstance(cmd, list) else cmd)

I thought that quoted would be more correct, especially when it is not
going to get shortened.

I think the name join_cmdline is Ok, but could have been join_argvs or join_commands. Let me know if you feel it better be renamed. Originally I thought to make it quote_cmdlineargs but it doesn't fit semantically since it also joins them into a string

+46 -16

4 comments

7 changed files

yarikoptic

pr closed time in 11 days

push eventdatalad/datalad

Yaroslav Halchenko

commit sha 5c0ec369fee740b400c2793a16b1822ab245a107

ENH: join_cmdline to complement split_cmdline which uses quote_cmdlinearg

view details

Yaroslav Halchenko

commit sha 2e429cb583735e44331b2a4118c91ddfc3e0c5d6

RF: make use of join_cmdline Also includes a change to output produced by def _format_cmd_shorty(cmd): """Get short string representation from a cmd argument list""" which used to join then without quoting cmd_shorty = (' '.join(cmd) if isinstance(cmd, list) else cmd) I thought that quoted would be more correct, especially when it is not going to get shortened.

view details

Yaroslav Halchenko

commit sha 86d15d7f0a1eeb95749b539455b2f5b6350e11e4

BF(TST,WIN): fix a test since we always quote args on Windows

view details

Kyle Meyer

commit sha 38af32be0072dabff8d224ca16a3ab3e1cd7aed2

Merge pull request #4587 from yarikoptic/rf-quote_cmdlineargs

view details

push time in 11 days

PR merged datalad/datalad

RF: Make .get_git_dir to use .dot_git logic, prepare for possible deprecation of .get_git_dir corpse-in-basement

#2054 made me look into this code path.

Possible changes to consider

  • make ._get_dot_git public so it could be used instead of .get_git_dir, thus not requiring an instance if that seems needed; but I think it better be delayed until we know that such functionality is really desired

For now I just position it against master (thus for 0.13) but if we feel that better be delayed, I will reassign milestone and finally produce next branch ;)

+51 -37

3 comments

1 changed file

yarikoptic

pr closed time in 12 days

push eventdatalad/datalad

Yaroslav Halchenko

commit sha f29fc4e7e5f5928d78c9f7c9cb82dce5d9382878

RF: move .dot_git assignment logic into a staticmethod

view details

Yaroslav Halchenko

commit sha 8fd4e9ea932885d0cec03dbcd7d0854ef867addd

ENH: add ok_missing flag so it could be used in cases where there might not yet be a repo Primarily done to provide unification with get_git_dir

view details

Yaroslav Halchenko

commit sha 0b60ad952afd092a39bdb14179403e4c39da9788

ENH: _get_dot_git - introduce option relative

view details

Yaroslav Halchenko

commit sha 0137ed4b8f31ffd77760d46006585a92fa1e395b

RF: use _get_dot_git in get_git_dir and adjust note to mention possible deprecation

view details

Yaroslav Halchenko

commit sha eb16ddbce6eba3f96609de9740d22ebd72feb999

DOC+ENH: fill out docstring for _get_dot_git and make kwargs mandatory kwargs

view details

Yaroslav Halchenko

commit sha 173b3560dbbe7465b1f08c5e15c27b48585e5ea9

DOC: strip obsolete coment, adjust another one

view details

Yaroslav Halchenko

commit sha 9b555c5b906eee84ca69ba28ce560193ee3cf169

RF+BF: return full .dot_git path if .git is not subpath In my previous refactoring, while reading the old get_git_dir implementation that path would always be relative (since ".git" was returned etc). But it did resolve symlinks and read links from .git file. This change does not necessarily makes it match 100% old behavior, since we would still return relative it if it points e.g. somewhere under .git somehow, but as for the target need of discovering .git location I think it should be functionally equivalent (I would have even stripped maybe_relative completely).

view details

Kyle Meyer

commit sha fef8b094def3f23653d4b31738f9d310db0c4db1

Merge pull request #4597 from yarikoptic/rf-dotgit

view details

push time in 12 days

push eventdatalad/datalad

Kyle Meyer

commit sha 16962b5f7c2f36860bc642f671f430735964ccc6

MNT: Post-release dance (maint) [ci skip]

view details

Kyle Meyer

commit sha dd647fab35c6db714d50cc2bc0b00de65aa61d36

MNT: Post-release dance [ci skip]

view details

Kyle Meyer

commit sha 0f3f6279b7060d553c1f5a1d966107e85d06423b

Merge branch 'maint' [ci skip]

view details

push time in 12 days

push eventdatalad/datalad

Kyle Meyer

commit sha 16962b5f7c2f36860bc642f671f430735964ccc6

MNT: Post-release dance (maint) [ci skip]

view details

push time in 12 days

release datalad/datalad

0.13.0

released time in 12 days

push eventdatalad/datalad

Yaroslav Halchenko

commit sha 08a57a2e74851082ab9c9951ea25af83dac333bc

NF: a prototype for ext special remote to store files in git/objects of a remote [ci skip]

view details

Yaroslav Halchenko

commit sha 5865b0a59ce46b0345619ae23888a047327ccb90

ENH: use git rev-parse --git-dir to deduce .git. Rename var to gitdir Thanks @kyleam

view details

Yaroslav Halchenko

commit sha d1c35d8a6474c7112b9ba12228acd8fa0c25fb76

ENH: use ls-remote for CHECKPRESENT Thanks @kyleam!

view details

Yaroslav Halchenko

commit sha dcbdcacb0e85fe88f983231e681cdf660d849577

ENH: TODO + RF to use update-ref -d to remove ref (Thanks @kyleam)

view details

Yaroslav Halchenko

commit sha e6e8316148af567aa8abae780f19fa282a6475e3

RF: rename refs/annexed/ to refs/gitobjects to match the name of the remoe

view details

Yaroslav Halchenko

commit sha 95ff8597d40356eaf0735822a1fa92fcc0745498

BF: $1 somehow sneaked in instead of $file. Should be the same/identical (I hope)

view details

Benjamin Poldrack

commit sha 4711d28b946143f4031525a8d56f5ea32a360a7e

BF/ENH: get_size_from_key to match specification Suck in implementation from git-annex-ria-remote. Former implementation couldn't correctly deal with chunks for example. See https://git-annex.branchable.com/internals/key_format/ for specification.

view details

Adina Wagner

commit sha b2210c0653ce2beacb06c21600df521228e82d8a

DOC/ENH: only add example if code for the given API exists

view details

Adina Wagner

commit sha 5b1a14117b150bb8dc392cb39bf90b2335b762c7

DOC: add examples to diff

view details

Adina Wagner

commit sha 14115b1b1bd9534d01b038429eee1dc5f1d69bd0

DOC: remove duplicate examples from docstring of drop

view details

Adina Wagner

commit sha 93986a600a40b9b387985eb16cbde6991e96106e

DOC: add example with -n to clone

view details

Adina Wagner

commit sha 5b029bb1307afc796e7c11f1940fcbd6646f3ec5

DOC: add examples for remove

view details

Adina Wagner

commit sha d4c20a66f7c028d1e1d5ce1ae9ca78e82a5402a8

DOC: add examples for uninstall

view details

Adina Wagner

commit sha aa458a67b4c9565c3b4ecb51463687f98783408e

DOC: add examples for update

view details

Adina Wagner

commit sha e039d8473240799b31fcd9d5f27bcb1920a18170

DOC: add examples for download-url

view details

Adina Wagner

commit sha 770e163c4e75fbfa2c4ed53709ae1bcadc9123e7

DOC: move existing rerun examples into dedicated example structure, and translate to python if possible"

view details

Adina Wagner

commit sha 45f2f478b7bb39f80fe09acddd73d7b841e21b00

DOC: add examples for run-procedure

view details

Adina Wagner

commit sha afbfd3210e16f5a57c7167dfc48d14ac69124b67

DOC: add examples for unlock

view details

Michael Hanke

commit sha 5fb32b847e19b6682969cd760c0d47b2678f2afb

ENH: Return early if there is no example for a given API Enables to simply not have such an example key, instead of being forced to provide an empty one.

view details

Michael Hanke

commit sha aae7fd8da0f48d7762e5de790e63b9162a63461e

DOC: Leave formated examples untouched ... to prevent messing up more complex ones.

view details

push time in 12 days

push eventdatalad/datalad

Kyle Meyer

commit sha 01098d4b7d3ef633ddaf9181f469f55430894105

MNT: CHANGELOG.md: Prune 0.13.0rc* entries

view details

Kyle Meyer

commit sha 7a3eb802d5ff6ff73708de7f10b1c355e9fea952

MNT: CHANGELOG.md: Revise 0.13.0 entries

view details

Kyle Meyer

commit sha 9d739cb75715a1bf14bb2ee36880e99ca5164da0

MNT: Release 0.13.0

view details

push time in 12 days

created tagdatalad/datalad

tag0.13.0

Keep scientific data under control with git and git-annex

created time in 12 days

push eventdatalad/datalad

Kyle Meyer

commit sha 3d1d9de2e737e737914c2dd4e5c64c824c5f3cf2

MNT: CHANGELOG.md: Ninth batch for 0.13.0 [ci skip]

view details

push time in 12 days

delete branch kyleam/datalad

delete branch : tqdm-reset-kludge

delete time in 12 days

more