This works like a champ on CentOS 7.7 choosing between vfs and overlay2. Similar to my own original patch, I have no idea how to test overlay vs overlay2 detection.

I'm unfortunately away from a Docker-friendly computer for the weekend. I'll check it out on Monday and report back.

For whatever my opinion is worth, the functions are basically the same but only by happenstance rather than design (i.e. they both use overlay mount type versus this being a generic "try to mount it function"). Also, I think there is value in each filesystem detection routine being separate.

Ultimately, it's up to you maintainer folk and I'll happily make the code work either way.

@finswimmer are your concerns about environment variables targeted at this particular problem or generic? I can't think of any sane circumstance where it would be an issue here.

The only thing I can think of is if the shell launched by Python via shell=True sources a different set of environment variables than what were defined before Python was executed. However, the function in question shouldn't be dependent on any external variables.

PR is now at moby/moby#40131

Signed-off-by: Brian Turek [email protected]

<!-- Please make sure you've read and understood our contributing guidelines; https://github.com/moby/moby/blob/master/CONTRIBUTING.md

** Make sure all your commits include a signature generated with git commit -s **

For additional information on our contributing process, read our contributing guide https://docs.docker.com/opensource/code/

If this is a bug fix, make sure your description includes "fixes #xxxx", or "closes #xxxx"

Please provide the following information: -->

- What I did Fixed overlay support detection for rootless Docker on CentOS (and other non-Ubuntu based distros). The current implementation causes runtime errors when running rootless Docker on CentOS due lack of mounting permissions. See docker/for-linux#836 and docker-library/docker#193 for related tickets

- How I did it Current overlay detection works by merely checking for filesystem support for overlay rather than overlay support for that particular user (i.e. Ubuntu allows non-root users to use overlay, CentOS does not). This patch changes the logic to attempt a real overlay mount and determine overlay support by checking whether than mount was successful.

Note that this is literally the first time I've written Go so constructive criticism is appreciated.

- How to verify it

• Run the new dockerd binary as rootless on CentOS, the storage-driver falls back to vfs
• Run the new dockerd binary as root on CentOS, the storage-driver is overlay2
• Run the new dockerd binary as rootless on Ubuntu, the storage-driver is overlay2
• Run the new dockerd binary as root on Ubuntu, the storage-driver is overlay2

- Description for the changelog <!-- Write a short (one line) summary that describes the changes in this pull request for inclusion in the changelog: --> Improve rootless Docker overlay support detection

- A picture of a cute animal (not mandatory but encouraged)

<!-- This issue tracker is for bug reports and feature requests. For questions, and getting help on using docker:

• Docker documentation - https://docs.docker.com
• Docker Forums - https://forums.docker.com
• Docker community Slack - https://dockercommunity.slack.com/ (register here: http://dockr.ly/community)
• Post a question on StackOverflow, using the Docker tag -->

• [x] This is a bug report
• [ ] This is a feature request
• [ ] I searched existing issues before opening this one

<!-- DO NOT report security issues publicly! If you suspect you discovered a security issue, send your report privately to [email protected]. -->

Expected behavior

Rootless Docker detection of filesystem support should exclude CentOS's implementation of overlay as it requires root to work

Actual behavior

Rootless Docker detects the overlay kernel module is loaded and assumes it will work causing runtime failures

Steps to reproduce the behavior

<!-- Describe the exact steps to reproduce. If possible, provide a minimum reproduction example; take into account that others do not have access to your private images, source code, and environment.

REMOVE SENSITIVE DATA BEFORE POSTING (replace those parts with "REDACTED") --> I worked with the docker:dind-rootless image team on docker-library/docker#193 to work this out.

Steps:

• sudo modprobe overlay
• ./dockerd-rootless.sh --experimental
• docker -H unix:///run/user/1000/docker.sock run -ti alpine

docker: Error response from daemon: error creating overlay mount to /home/brian/.local/share/docker/overlay/918283926d7ce7e89ed73b6b17034793980a11b4f07534ba411bd54ee177dece-init/merged: operation not permitted.

I believe the problem lies with https://github.com/docker/docker-ce/blob/6d1e64f40569cf42e2f684690d1e2f5cff9546d1/components/engine/daemon/graphdriver/overlay2/overlay.go#L286 which doesn't check the return of the mount attempt to see if there was an error.

Output of docker version:

Client: Docker Engine - Community
 Version:           19.03.4
 API version:       1.40
 Go version:        go1.12.10
 Git commit:        9013bf583a
 Built:             Fri Oct 18 15:52:22 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.4
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.10
  Git commit:       9013bf583a
  Built:            Fri Oct 18 15:50:54 2019
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker info:

Client:
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 19.03.4
 Storage Driver: overlay
  Backing Filesystem: xfs
  Supports d_type: true
 Logging Driver: json-file
 Cgroup Driver: none
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
  rootless
 Kernel Version: 3.10.0-1062.4.1.el7.x86_64
 Operating System: CentOS Linux 7 (Core)
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.701GiB
 Name: localhost.localdomain
 ID: V2R5:7CUM:WUBK:CMPZ:5CXD:NMNA:DDKF:W4GJ:YZBF:JJWW:DUHW:Z6LK
 Docker Root Dir: /home/brian/.local/share/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
WARNING: the overlay storage-driver is deprecated, and will be removed in a future release.

Additional environment details (AWS, VirtualBox, physical, etc.) This testing has been done entirely on CentOS 7 VMs using various hypervisors

I've been working on getting docker:dind-rootless working on CentOS 7.7 and ran into an issue because of overlay detection. The docker:dind-rootless image apparently assumes the host OS supports overlay (there's even a TODO) which then causes the image to fail when launching child containers with a "operation not permitted" referencing the overlay mount command.

My workaround is passing --storage-driver=vfs on the docker run but I believe that the image should auto-detect overlay support or have documentation about the above flag.

So a manual sudo modprobe overlay on the host causes both cases to "succeed" in detecting overlay but then of course fail when it actually tries to use it. It appears that dockerd's detection of rootless overlay appears to be lacking.

It appears my test order was what caused the confusion. I always tried native rootless Docker first (it cannot load the overlay module) which correctly used vfs. I then always started a rootful Docker daemon to try dind-rootless and the rootful Docker daemon loaded the overlay module.

Thanks for the help; off to submit bug report with Docker proper!

It's definitely something unique to the image vs an underlying dockerd problem (I think).

Relevant log entries from running rootless Docker directly on host (note the use of vfs):

ERRO[2019-10-22T07:53:59.008188371+01:00] 'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support loaded.  storage-driver=overlay2
ERRO[2019-10-22T07:53:59.009369254+01:00] AUFS cannot be used in non-init user namespace  storage-driver=aufs
ERRO[2019-10-22T07:53:59.014333070+01:00] 'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support loaded.  storage-driver=overlay
INFO[2019-10-22T07:53:59.032246241+01:00] Loading containers: start.                   
WARN[2019-10-22T07:53:59.035343935+01:00] Running modprobe bridge br_netfilter failed with message: modprobe: ERROR: could not insert 'br_netfilter': Operation not permitted
insmod /lib/modules/3.10.0-1062.4.1.el7.x86_64/kernel/net/bridge/br_netfilter.ko.xz 
, error: exit status 1 
INFO[2019-10-22T07:53:59.119941796+01:00] Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address 
INFO[2019-10-22T07:53:59.155279439+01:00] Loading containers: done.                    
INFO[2019-10-22T07:53:59.165889078+01:00] Docker daemon                                 commit=9013bf583a graphdriver(s)=vfs version=19.03.4

Relevant log entries from running docker run -ti --privileged docker:dind-rootless on a rootful docker daemon (note use of overlay):

ERRO[2019-10-22T06:56:41.322548316Z] AUFS cannot be used in non-init user namespace  storage-driver=aufs
WARN[2019-10-22T06:56:41.325611279Z] [graphdriver] WARNING: the overlay storage-driver is deprecated, and will be removed in a future release 
INFO[2019-10-22T06:56:41.358526463Z] Loading containers: start.                   
WARN[2019-10-22T06:56:41.390562547Z] Running modprobe bridge br_netfilter failed with message: Device "bridge" does not exist.
bridge                151336  2 br_netfilter,ebtable_broute
stp                    12976  1 bridge
llc                    14552  2 bridge,stp
Device "br_netfilter" does not exist.
br_netfilter           22256  0 
bridge                151336  2 br_netfilter,ebtable_broute
modprobe: can't change directory to '/lib/modules': No such file or directory
, error: exit status 1 
INFO[2019-10-22T06:56:41.479545182Z] Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address 
INFO[2019-10-22T06:56:41.509871117Z] Loading containers: done.                    
INFO[2019-10-22T06:56:41.537544463Z] Docker daemon                                 commit=9013bf583a graphdriver(s)=overlay version=19.03.4

There are enough moving/experimental parts here that there might be something else going on. Any ideas? I'm happy to experiment.

I've been working on getting docker:dind-rootless working on CentOS 7.7 and ran into an issue because of overlay detection. The docker:dind-rootless image apparently assumes the host OS supports overlay (there's even a TODO) which then causes the image to fail when launching child containers with a "operation not permitted" referencing the overlay mount command.

My workaround is passing --storage-driver=vfs on the docker run but I believe that the image should auto-detect overlay support or have documentation about the above flag.

Rebased to master and merged in the changes. Should be good to go!

I consider it done and am just waiting on it to be merged.

As @oleastre mentioned, there is a significant code divergence between master and develop so the patch cannot be applied to master. My recommendation is to not let perfect be the enemy of good and just merge both #1402 and this PR.

Honestly, this master vs develop situation is a giant pain and, as highlighted by this situation, is impeding work. It's only getting worse since there was a major refactor in develop that changes many core functions.

Checked it today and it both pulls packages using certs and will publish. Confidence restored!

I changed the name and was very confident everything would continue working, however, there were major changes to develop that make me less confident.

NP, probably the easiest bug to squash :)

On launch, KeePass pops up a warning telling me to upgrade my current HIBPOfflineCheck from v1.7.0 to v1.7.1 but there is no v1.7.1 available in this repository. It appears that version.txt reflects this incorrect version.

I don't know what I thought I saw but all the type hinting is Python 2 friendly. Sorry about that!

It appears that the way configs were handled changed drastically between master and develop. It would have been entirely possible to write the values back in master as a handle was kept to each of the configs and they closely resembled files. In develop, the concept of the config was abstracted away and the individual config files are merged into the abstracted config; there is no longer a handle to the underlying files.

I confused source and upload repositories in my mind when I gave above options; sorry about that. There are actually fewer places that are possibilities:

• When someone configs an upload repository (like you have)
  • Pros: It can write the updated value back to the config
  • Cons: The update only happens at the time of the original config call
• In the Publisher class
  • Pros: The updated value gets updated per publish, no matter how many files
  • Cons:
    • I don't believe you can write the value back to the config
    • This involves making a new requests session (see previous comment about certs)
• In the Uploader class
  • Pros: The updated value gets updated per publish, no matter how many files
  • Cons: I don't believe you can write the value back to the config

I don't think there is a perfect solution here. In the Uploader seems best in terms of always working but then you don't get to perma-write the value back to the config. Maybe throw a warning in Uploader if an updated URL is detected? The idea was brought up in #1310. I could maybe make the argument that a warning and forcing the user to update their own config is clearer than auto updating a config :)

I'm a bit torn whether the below ideas are over-engineering the situation or not. However, I would definitely have a follow_redirects=True on the POST. The current code depends on, essentially, one redirect-change happening in the lifetime of the repository otherwise we'll have another silent error. Now, in terms of possible over-engineering ideas:

• Check the redirect on repository instantiation (happens per poetry run)?
• Check the redirect on every POST (i.e. do a HEAD everytime)?
• Check the redirect on the first usage of a repository?

To be clear here, I'm not a maintainer so my ideas are just ideas, not requests.

Wouldn't it make more sense to somehow hook URL usage versus config? Many people don't use the config command and this also means you would potentially catch a redirect once during the lifetime of a project.

Can you check to see if #1341 solves your problem?

Pull Request Check List

This is just a reminder about the most common mistakes. Please make sure that you tick all appropriate boxes. But please read our contribution guide at least once, it will save you unnecessary review cycles!

• [x] Added tests for changed code.
• [ ] Updated documentation for changed code.

Note: If your Pull Request introduces a new feature or changes the current behavior, it should be based on the develop branch. If it's a bug fix or only a documentation update, it should be based on the master branch.

If you have any questions to any of the points above, just submit and ask! This checklist is here to help you, not to deter you from contributing!

Per #1249 and pip #6776, newer versions of pip require require the username and password to be properly encoded/quoted. This PR simply quotes the URL when calling pip.

As referenced in #1249, we do not want to pre-quote the strings as that fouls up the metadata fetch which does not use pip.

I think this is closed with #1320?

Admittedly I'm a bit confused by the PR because the rest of the codebase does type hinting by comment (Python 2 friendly) whereas that file is Python3 only...

I probably struggled the most with naming the silly parameters compared to anything else about this PR. I started with custom-ca, hated it, but then figured we could come to consensus in the PR itself :)

Since I'm interfacing with both requests and pip under the hood, I need to use a combined client cert/key file as that's the only thing both take (pip is the picky one). That being said, I liked pip's --client-cert and went with it.

For CA:

• Twine (as you noted) uses --cacert
• pip (as you noted) uses --cert
• cURL uses --cacert
• wget uses --ca-certificate
• requests uses the parameter verify (that's terrible)
• Python's built-in ssl module uses the parameter cafile
• urllib3 uses the parameter ca_certs

Since I did --client-cert (with the hyphen in between), I'm thinking it should be consistent and be --ca-<SOMETHING> but that looks weird too. I'm happy to change it to anything that the maintainers will accept; I fully recognize --custom-ca is not great.

I'm forgetting if GitHub emails people on cross ticket mentions but I have a PR, #1325, that addresses this request. The developers said in Discord that they're ranking PRs by number of votes so please +1 the PR :)

I'm forgetting if GitHub emails people on cross ticket mentions but I have a PR, #1325, that addresses this request. The developers said in Discord that they're ranking PRs by number of votes so please +1 the PR :)

@brycedrennan I realize this is a substantial PR but I just successfully used my branch to both push and pull to a mutual TLS authenticated repository. This unfortunately is a blocker for my major use case of Poetry so please let me know I can help this along.

It appears that tensorflow 2.0.0-rc0's setup.py dynamically changes its requirements depending on the environment by executing code. My guess is that Poetry does not handle this case well for some reason as it appears ALL of the Python2-linked dependencies are being grabbed.

I've never really played with the ast module and I'm only taking guesses here...

Added and pushed

So os.symlink isn't available on Windows in Python 2 which would then maybe throw an AttributeError instead? I was thinking along the same lines as your try/except and also was unsure given the previous use of flags :)

Pinned down the problem: this is a brittle unit test that made strong assumptions on how a file would be tested whether it exists. My original code was correct but I edited it to make it fit the mock assumptions. I don't know why/how but my code was previously passing (locally) on Win10 Python 3.7 without the modification in place (weird).

P.S. - My last commit message should have been "by using exists" and not "by existing exists." It's getting late :(

<!-- Hi there! Thank you for discovering and submitting an issue.

Before you submit this; let's make sure of a few things. Please make sure the following boxes are ticked if they are correct. If not, please try and fulfill these first. -->

<!-- Checked checkbox should look like this: [x] -->

• [x] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [x] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

<!-- Once those are done, if you're able to fill in the following list with your information, it'd be very helpful to whoever handles the issue. -->

• OS version and name: Win10
• Poetry version: 1.0.0b1
• Link of a Gist with the contents of your pyproject.toml file: N/A

Issue

<!-- Now feel free to write your issue, but please be descriptive! Thanks again 🙌 ❤️ --> Adding a custom [[tool.poetry.source]] whose name includes a period causes issues when paired with http_basic settings. The add_property method on the ConfigSource object blindly splits on periods, including those in the repo name. The side effect of this is that the resulting TOML becomes very "deep."

I'm happy to do a PR but I'm curious to know how much name checking is appropriate.

I'm pretty sure Travis CI just hiccuped during my tests as those failures have absolutely nothing to do with the changed code.

Pull Request Check List

This is just a reminder about the most common mistakes. Please make sure that you tick all appropriate boxes. But please read our contribution guide at least once, it will save you unnecessary review cycles!

• [ ] Added tests for changed code.
• [ ] Updated documentation for changed code.

Note: If your Pull Request introduces a new feature or changes the current behavior, it should be based on the develop branch. If it's a bug fix or only a documentation update, it should be based on the master branch.

If you have any questions to any of the points above, just submit and ask! This checklist is here to help you, not to deter you from contributing!

Two conftest.py files are currently attempting to perform symlinking as part of a fixture which requires either admin privileges or "Developer Mode" to be enabled on Win10. This PR just wraps the symlink call in a try/except and falls back to copying if it fails.

Pull Request Check List

This is just a reminder about the most common mistakes. Please make sure that you tick all appropriate boxes. But please read our contribution guide at least once, it will save you unnecessary review cycles!

• [x] Added tests for changed code.
• [x] Updated documentation for changed code.

Note: If your Pull Request introduces a new feature or changes the current behavior, it should be based on the develop branch. If it's a bug fix or only a documentation update, it should be based on the master branch.

If you have any questions to any of the points above, just submit and ask! This checklist is here to help you, not to deter you from contributing!

This PR adds support for specifying a custom certificate authority to validate custom repositories as well as using client certificate-based authentication for custom repos. This was quite a bit bigger than I anticipated but I think I plumbed everything that is needed.

This should address #1012 and #297

Pull Request Check List

This is just a reminder about the most common mistakes. Please make sure that you tick all appropriate boxes. But please read our contribution guide at least once, it will save you unnecessary review cycles!

• [ ] Added tests for changed code.
• [ ] Updated documentation for changed code.

Note: If your Pull Request introduces a new feature or changes the current behavior, it should be based on the develop branch. If it's a bug fix or only a documentation update, it should be based on the master branch.

If you have any questions to any of the points above, just submit and ask! This checklist is here to help you, not to deter you from contributing!

#1272 introduced a bug on "develop" where poetry performs a blind read on the global config file without checking whether it exists first. This is a one-line fix that simply checks to make sure the file exists before the read. The existing follow-on code already handles creation of the file just fine.

This should close out #1179