profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/jsirois/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.
John Sirois jsirois @toolchainlabs @pantsbuild @apache @perkuno Bozeman, MT

jsirois/advisory-db 0

Security advisory database for Rust crates published through crates.io

jsirois/aurora 0

Mirror of Apache Aurora

jsirois/aurora-packaging 0

Mirror of Apache Aurora

jsirois/aurora-website 0

Mirror of Apache Aurora

jsirois/bazel-buildfarm 0

[WIP] Bazel remote caching and execution service

jsirois/binaries 0

A temporary home for pants static binaries and scripts

jsirois/binhost 0

Temporary, dumb way to host binaries at a known URL.

jsirois/bot 0

Source code for our Discord bot

jsirois/cargo-audit 0

Audit Cargo.lock files for crates with security vulnerabilities

issue commentpantsbuild/pex

Strange issue with et-xmlfile dependency

Thanks a lot @jsirois will test it as soon as the release is out

Djailla

comment created time in an hour

PR opened pantsbuild/pants

Reviewers
Remove references to PANTS_DEV env var

This environment variable was apparently previously used to control some kind of linting that pants no longer does, so we can get rid of the code that sets it.

+0 -14

0 comment

2 changed files

pr created time in 4 hours

pull request commentpantsbuild/pants

Scala 2.13 support for 1.x

And again another suites has failed.

notxcain

comment created time in 16 hours

Pull request review commentpantsbuild/pants

WIP Allow running pants with Python 3.9

 def all_entries(cls) -> List[Dict[str, str]]: class PythonVersion(Enum):     py37 = "py37"     py38 = "py38"+    py39 = "py39"      def __str__(self) -> str:         return str(self.value)      @property     def number(self) -> int:-        return {self.py37: 37, self.py38: 38}[self]  # type: ignore[index]+        return {self.py37: 37, self.py38: 38, self.py39: 39}[self]  # type: ignore[index]      @property     def decimal(self) -> float:-        return {self.py37: 3.7, self.py38: 3.8}[self]  # type: ignore[index]+        return {self.py37: 3.7, self.py38: 3.8, self.py39: 3.9}[self]  # type: ignore[index]      def default_stage(self, *, is_bootstrap: bool = False) -> Stage:         if is_bootstrap:-            return {self.py37: Stage.bootstrap, self.py38: Stage.bootstrap_cron}[self]  # type: ignore[index]-        return {self.py37: Stage.test, self.py38: Stage.test_cron}[self]  # type: ignore[index]+            return {self.py37: Stage.bootstrap, self.py38: Stage.bootstrap_cron, self.py39: Stage.bootstrap_cron}[self]  # type: ignore[index]+        return {self.py37: Stage.test, self.py38: Stage.test_cron, self.py39: Stage.test_cron}[self]  # type: ignore[index]

Exactly, but only for the sake of testing. Before merging, we want to revert it back to what you have currently

gshuflin

comment created time in 20 hours

Pull request review commentpantsbuild/pants

WIP Allow running pants with Python 3.9

 def all_entries(cls) -> List[Dict[str, str]]: class PythonVersion(Enum):     py37 = "py37"     py38 = "py38"+    py39 = "py39"      def __str__(self) -> str:         return str(self.value)      @property     def number(self) -> int:-        return {self.py37: 37, self.py38: 38}[self]  # type: ignore[index]+        return {self.py37: 37, self.py38: 38, self.py39: 39}[self]  # type: ignore[index]      @property     def decimal(self) -> float:-        return {self.py37: 3.7, self.py38: 3.8}[self]  # type: ignore[index]+        return {self.py37: 3.7, self.py38: 3.8, self.py39: 3.9}[self]  # type: ignore[index]      def default_stage(self, *, is_bootstrap: bool = False) -> Stage:         if is_bootstrap:-            return {self.py37: Stage.bootstrap, self.py38: Stage.bootstrap_cron}[self]  # type: ignore[index]-        return {self.py37: Stage.test, self.py38: Stage.test_cron}[self]  # type: ignore[index]+            return {self.py37: Stage.bootstrap, self.py38: Stage.bootstrap_cron, self.py39: Stage.bootstrap_cron}[self]  # type: ignore[index]+        return {self.py37: Stage.test, self.py38: Stage.test_cron, self.py39: Stage.test_cron}[self]  # type: ignore[index]

I'm not super-familiar with our travis configuration setup. Are you saying that this line should be: return {self.py37: Stage.test_cron, self.py38: Stage.test_cron, self.py39: Stage.test}[self] # type: ignore[index]

with the effect of running Python 3.9 by default in ordinary builds, and running Python 3.8/Python 3.7 during nightly "cron" builds?

gshuflin

comment created time in 20 hours

Pull request review commentpantsbuild/pants

python_requirements: provide global module_mapping with defaults.

+---+attrs:+  - attr++enum34:+  - enum++paho_mqtt:+  - paho++pyopenssl:+  - OpenSSL++pycrypto:+  - Crypto++pymongo:+  - bson+  - gridfs++pytest_runner:+  - ptr++python_dateutil:+  - dateutil++setuptools:+  - easy_install+  - pkg_resources++zope.event:

@Eric-Arellano thoughts?

kaos

comment created time in 20 hours

Pull request review commentpantsbuild/pants

python_requirements: provide global module_mapping with defaults.

+---+attrs:+  - attr++enum34:+  - enum++paho_mqtt:+  - paho++pyopenssl:+  - OpenSSL++pycrypto:+  - Crypto++pymongo:+  - bson+  - gridfs++pytest_runner:+  - ptr++python_dateutil:+  - dateutil++setuptools:+  - easy_install+  - pkg_resources++zope.event:

I think we'd want to map to the highest package that is unique to the distribution. E.g., zope.event just maps to zope.event and not to zope. The assumption being that if you consume zope.event you are always importing from zope.event and not from zope.

That seems like the most sane assumption (and means that we can omit zope here, assuming our "dist-name == top-level package name" heuristic works with dots in the name.)

kaos

comment created time in 20 hours

pull request commentpantsbuild/pants

python_requirements: provide global module_mapping with defaults.

@benjyw what do you think about this backwards compatibility discussion? I'm leaning towards it being acceptable.

If we decide this is categorically fine, @kaos I'll leave some more detailed comments for how I suggest factoring this / where to make the change.

Hmm, yes, I think on balance that I'd rather inconvenience the probably-tiny number of people who the collision might affect (and those people have already dealt with this and are savvy) than the large number of first-time users who shouldn't have to deal with this. So I think the backwards compat risk, as you describe it, is small enough to proceed.

kaos

comment created time in 20 hours

push eventpantsbuild/pants

gshuflin

commit sha eed7d2e77e040044a9fb98374fec221f02042995

Simplify pants initialization process and remove `pants_exe.py` (#11643) ### Problem The code implementing pants initialization is somewhat overcomplicated and more verbose than it needs to be. ### Solution This commit: * Moves the definition of some `PANTS_` environment variables relevant to initialization to a common file, to avoid duplicating environment variable string names in different parts of the codebase * Renames `PANTS_ENTRYPOINT` to `PANTS_DAEMON_ENTRYPOINT`, to make it clearer what role this entrypoint mechanism is playing in pants execution * Folds the fairly-minimal functionality of `pants_exe.py` into `pants_loader.py` or integration test files, and removes `pants_exe.py` entirely ### Result This is a refactoring commit that should have no user-visible change

view details

push time in 21 hours

PR merged pantsbuild/pants

Reviewers
Simplify pants initialization process and remove `pants_exe.py`

Problem

The code implementing pants initialization is somewhat overcomplicated and more verbose than it needs to be.

Solution

This commit:

  • Moves the definition of some PANTS_ environment variables relevant to initialization to a common file, to avoid duplicating environment variable string names in different parts of the codebase
  • Renames PANTS_ENTRYPOINT to PANTS_DAEMON_ENTRYPOINT, to make it clearer what role this entrypoint mechanism is playing in pants execution
  • Folds the fairly-minimal functionality of pants_exe.py into pants_loader.py or integration test files, and removes pants_exe.py entirely

Result

This is a refactoring commit that should have no user-visible change

+126 -115

0 comment

6 changed files

gshuflin

pr closed time in 21 hours

Pull request review commentpantsbuild/pants

Simplify pants initialization process and remove `pants_exe.py`

+# Copyright 2021 Pants project contributors (see CONTRIBUTORS.md).+# Licensed under the Apache License, Version 2.0 (see LICENSE).+++IGNORE_UNRECOGNIZED_ENCODING = "PANTS_IGNORE_UNRECOGNIZED_ENCODING"+RECURSION_LIMIT = "PANTS_RECURSION_LIMIT"+DAEMON_ENTRYPOINT = "PANTS_DAEMON_ENTRYPOINT"+PANTSC_PROFILE = "PANTSC_PROFILE"

Huh, I wonder why it's pantsc and not pantsd.

Is it supposed to represent "pants c_profile" perhaps?

gshuflin

comment created time in 21 hours

pull request commentpantsbuild/pants

Bump Zinc version to support Scala 2.13

@stuhood thank you!

I've taken a look at the CI results and I'm not sure how to interpret that. I have a gut feeling that they are false negative. There are some Yarn-related issues.

notxcain

comment created time in 21 hours

Pull request review commentpantsbuild/pants

WIP Allow running pants with Python 3.9

 backend_packages.add = [   "pants.backend.python.typecheck.mypy",   "pants.backend.python.mixed_interpreter_constraints",   "internal_plugins.releases",-  "toolchain.pants.auth",-  "toolchain.pants.buildsense",-  "toolchain.pants.common",+#  "toolchain.pants.auth",+#  "toolchain.pants.buildsense",+#  "toolchain.pants.common",

Yeah this is blocked by https://github.com/pantsbuild/pants/issues/11469, which Liam is working on. It's okay to temporarily turn these off.

What we will really want to do is find a way to turn off only in the Py39 jobs, but that can wait till everything is green.

gshuflin

comment created time in 21 hours

Pull request review commentpantsbuild/pants

WIP Allow running pants with Python 3.9

 def all_entries(cls) -> List[Dict[str, str]]: class PythonVersion(Enum):     py37 = "py37"     py38 = "py38"+    py39 = "py39"      def __str__(self) -> str:         return str(self.value)      @property     def number(self) -> int:-        return {self.py37: 37, self.py38: 38}[self]  # type: ignore[index]+        return {self.py37: 37, self.py38: 38, self.py39: 39}[self]  # type: ignore[index]      @property     def decimal(self) -> float:-        return {self.py37: 3.7, self.py38: 3.8}[self]  # type: ignore[index]+        return {self.py37: 3.7, self.py38: 3.8, self.py39: 3.9}[self]  # type: ignore[index]      def default_stage(self, *, is_bootstrap: bool = False) -> Stage:         if is_bootstrap:-            return {self.py37: Stage.bootstrap, self.py38: Stage.bootstrap_cron}[self]  # type: ignore[index]-        return {self.py37: Stage.test, self.py38: Stage.test_cron}[self]  # type: ignore[index]+            return {self.py37: Stage.bootstrap, self.py38: Stage.bootstrap_cron, self.py39: Stage.bootstrap_cron}[self]  # type: ignore[index]+        return {self.py37: Stage.test, self.py38: Stage.test_cron, self.py39: Stage.test_cron}[self]  # type: ignore[index]

When testing, you will want to invert these so that PY39 runs during daily and the others during nightly. Although, last time I checked Py39 shards are already working and the only thing left is the Build Wheels shards, which should happen no matter what during daily.

gshuflin

comment created time in 21 hours

Pull request review commentpantsbuild/pants

Simplify pants initialization process and remove `pants_exe.py`

+# Copyright 2021 Pants project contributors (see CONTRIBUTORS.md).

Better to put under pants/bin as these env vars are solely useful for the bin files.

gshuflin

comment created time in 21 hours

Pull request review commentpantsbuild/pants

Simplify pants initialization process and remove `pants_exe.py`

+# Copyright 2021 Pants project contributors (see CONTRIBUTORS.md).+# Licensed under the Apache License, Version 2.0 (see LICENSE).+++IGNORE_UNRECOGNIZED_ENCODING = "PANTS_IGNORE_UNRECOGNIZED_ENCODING"+RECURSION_LIMIT = "PANTS_RECURSION_LIMIT"+DAEMON_ENTRYPOINT = "PANTS_DAEMON_ENTRYPOINT"+PANTSC_PROFILE = "PANTSC_PROFILE"

Huh, I wonder why it's pantsc and not pantsd.

gshuflin

comment created time in 21 hours

PR opened pantsbuild/pants

Reviewers
Simplify pants initialization process and remove `pants_exe.py`

Problem

The code implementing pants initialization is somewhat overcomplicated and more verbose than it needs to be.

Solution

This commit:

  • Moves the definition of some PANTS_ environment variables relevant to initialization to a common file, to avoid duplicating environment variable string names in different parts of the codebase
  • Renames PANTS_ENTRYPOINT to PANTS_DAEMON_ENTRYPOINT, to make it clearer what role this entrypoint mechanism is playing in pants execution
  • Folds the fairly-minimal functionality of pants_exe.py into pants_loader.py or integration test files, and removes pants_exe.py entirely

Result

This is a refactoring commit that should have no user-visible change

+126 -115

0 comment

6 changed files

pr created time in 21 hours

PR opened pantsbuild/pants

WIP Allow running pants with Python 3.9

WIP; based on https://github.com/pantsbuild/pants/pull/11462

+339 -73

0 comment

9 changed files

pr created time in a day

pull request commentpantsbuild/pants

Bump Zinc version to support Scala 2.13

@stuhood I've bumped Zinc to the latest compatible version 1.3.5 and registered Scala 2.13 in scala_platform.py. What's the easiest way to publish my branch locally and use it in our project for testing?

notxcain

comment created time in a day

pull request commentpantsbuild/pants

Allow concurrent runs of pants on the same instance of pantsd

This might be related to https://github.com/pantsbuild/pants/issues/11626, but, I ran ./pants test src/python/pants/base: at the same time in two different tabs. Both correctly rendered the console test summary, but only one showed the std{out,err} of the failed exception_sink_integration_test.py. What would you expect? The EnrichedTestResult rule is uncacheable, but I'm not sure how that plays out with concurrent runs.

To reproduce, probably useful to do a slower test (or add time.sleep()) so it doesn't complete before starting the other run.

stuhood

comment created time in a day

Pull request review commentpantsbuild/pants

Allow concurrent runs of pants on the same instance of pantsd

 function run_pex() { function run_packages_script() {   (     cd "${ROOT}"-    ./pants --concurrent run "${ROOT}/build-support/bin/packages.py" -- "$@"+    ./pants --no-pantsd run "${ROOT}/build-support/bin/packages.py" -- "$@"

Why this?

stuhood

comment created time in a day

Pull request review commentpantsbuild/pex

Fix Requires metadata handling.

 def test_requires_dists_none(pygoogleearth_zip_sdist):     with example_distribution("MarkupSafe-1.0-cp27-cp27mu-linux_x86_64.whl") as (wheel_path, dist):         assert [] == list(requires_dists(wheel_path))         assert [] == list(requires_dists(dist))++    # This tests a strange case detailed here:+    #   https://github.com/pantsbuild/pex/issues/1201#issuecomment-791715585+    with downloaded_sdist("et-xmlfile==1.0.1") as sdist, warnings.catch_warnings(+        record=True+    ) as events:+        assert [] == list(requires_dists(sdist))+        assert len(events) == 1+        warning = events[0]+        assert PEXWarning == warning.category+        assert (+            dedent(+                """\+                Ignoring 1 `Requires` field in {sdist} metadata:+                1.) Requires: python (>=2.6.0)++                You may have issues using using the 'et_xmlfile' distribution as a result.
                You may have issues using the 'et_xmlfile' distribution as a result.
jsirois

comment created time in a day

Pull request review commentpantsbuild/pex

Fix Requires metadata handling.

 def requires_dists(     for requires_dist in pkg_info.get_all("Requires-Dist", ()):         yield Requirement.parse(requires_dist) -    if include_1_1_requires:-        for requires in pkg_info.get_all("Requires", ()):-            yield Requirement.parse(requires)+    legacy_requires = pkg_info.get_all("Requires", [])  # type: List[str]+    if legacy_requires:+        name_and_version = project_name_and_version(dist)+        project_name = name_and_version.project_name if name_and_version else dist+        pex_warnings.warn(+            dedent(+                """\+                Ignoring {count} `Requires` {field} in {dist} metadata:+                {requires}++                You may have issues using using the '{project_name}' distribution as a result.
                You may have issues using the '{project_name}' distribution as a result.
jsirois

comment created time in a day

Pull request review commentpantsbuild/pants

Require explicit environment usage

 def check_and_set(name: str, value: Optional[str]):         return FrozenDict(env_var_subset)  +@frozen_after_init+@dataclass(unsafe_hash=True)+class EnvironmentRequest:+    """Requests a subset of the variables set in the environment.++    Requesting only the relevant subset of the environment reduces invalidation caused by unrelated+    changes.+    """++    requested: FrozenOrderedSet[str]+    allowed: Optional[FrozenOrderedSet[str]]++    def __init__(self, requested: Sequence[str], allowed: Optional[Sequence[str]] = None):+        self.requested = FrozenOrderedSet(requested)+        self.allowed = None if allowed is None else FrozenOrderedSet(allowed)+++class Environment(FrozenDict[str, str]):+    """A subset of the variables set in the environment."""++ @rule async def complete_environment(session_values: SessionValues) -> CompleteEnvironment:     return session_values[CompleteEnvironment]  +@rule+async def environment_subset(

Possibly...? I think that a majority of users of Process right now are constructing an environment for the process by hand, and only some cases are pulling it in from the outside world. Most users of InteractiveProcess will use the CompleteEnvironment, but not all, etc.

stuhood

comment created time in a day

Pull request review commentpantsbuild/pants

Require explicit environment usage

 def check_and_set(name: str, value: Optional[str]):         return FrozenDict(env_var_subset)  +@frozen_after_init+@dataclass(unsafe_hash=True)+class EnvironmentRequest:+    """Requests a subset of the variables set in the environment.++    Requesting only the relevant subset of the environment reduces invalidation caused by unrelated+    changes.+    """++    requested: FrozenOrderedSet[str]+    allowed: Optional[FrozenOrderedSet[str]]++    def __init__(self, requested: Sequence[str], allowed: Optional[Sequence[str]] = None):+        self.requested = FrozenOrderedSet(requested)+        self.allowed = None if allowed is None else FrozenOrderedSet(allowed)+++class Environment(FrozenDict[str, str]):+    """A subset of the variables set in the environment."""++ @rule async def complete_environment(session_values: SessionValues) -> CompleteEnvironment:     return session_values[CompleteEnvironment]  +@rule+async def environment_subset(

Good point: can drop the async. But the reason to expose this as a @rule is that we want to discourage direct access to the CompleteEnvironment and expose an API that encourages requesting a filtered copy, because it should allow @rules to be cleaned with early-cutoff in more cases. When a @rule requests the entire environment, it will be invalidated for any environment change.

stuhood

comment created time in 2 days

pull request commentpantsbuild/pants

Allow concurrent runs of pants on the same instance of pantsd

Only the top commit is relevant: the rest is #11641.

stuhood

comment created time in 2 days

Pull request review commentpantsbuild/pants

Require explicit environment usage

 impl Scheduler {       command.current_dir(tempdir.path());     } -    if hermetic_env {-      command.env_clear();-    }+    command.env_clear();

Great, that's much better.

stuhood

comment created time in 2 days

Pull request review commentpantsbuild/pants

Require explicit environment usage

  import logging import re-from dataclasses import dataclass-from typing import Dict, Mapping, Optional, Sequence+from typing import Dict, Optional, Sequence  from pants.engine.internals.session import SessionValues from pants.engine.rules import collect_rules, rule from pants.util.frozendict import FrozenDict-from pants.util.meta import frozen_after_init  logger = logging.getLogger(__name__)  name_value_re = re.compile(r"([A-Za-z_]\w*)=(.*)") shorthand_re = re.compile(r"([A-Za-z_]\w*)")  -@frozen_after_init-@dataclass(unsafe_hash=True)-class PantsEnvironment:-    """PantsEnvironment is a representation of the environment variables the currently-executing-    Pants process was invoked with."""+class CompleteEnvironment(FrozenDict):

Good change to subclass.

stuhood

comment created time in 2 days

Pull request review commentpantsbuild/pants

Allow concurrent runs of pants on the same instance of pantsd

 def run_goal_rule(         console.flush()         return GoalRuleResult(exit_code, stdout.getvalue(), stderr.getvalue()) -    def set_options(self, args: Iterable[str], *, env: Mapping[str, str] | None = None) -> None:+    def set_options(+        self, args: Iterable[str], *, env: Mapping[str, str | None] | None = None+    ) -> None:         """Update the engine session with new options and/or environment variables.          The environment variables will be used to set the `PantsEnvironment`, which is the         environment variables captured by the parent Pants process. Some rules use this to be able         to read arbitrary env vars. Any options that start with `PANTS_` will also be used to set-        options.+        options. Giving an environment variable a value of `None` will cause it to be propagated from+        the test runner's environment (os.environ).          This will override any previously configured values.         """+        env = {k: (v if v is not None else os.getenv(k, "")) for k, v in env.items()} if env else {}

Extracted to #11641.

stuhood

comment created time in 2 days

pull request commentpantsbuild/pants

Require explicit environment usage

Commits are useful to review independently.

stuhood

comment created time in 2 days