profile
viewpoint

google/starlark-rust 165

Starlark (https://github.com/bazelbuild/starlark) in Rust

illicitonion/cargo-ensure-installed 4

Like cargo install but if you already have a suitable version, simply leaves it as-is.

illicitonion/cargo-serve-doc 4

A cargo plugin for serving a unified tree of crate-local and std documentation (as well as the Rust book).

illicitonion/cargo-ensure-prefix 1

Cargo subcommand to check that all target files have a fixed prefix.

blorente/pants 0

The Pants Build System

illicitonion/after-cfg 0

A tiny website with resources for Code First: Girls students to help them continue coding after the beginners web development course.

PullRequestReviewEvent

issue commentbazelbuild/starlark

Add grammar to permit type annotations

This sounds really exciting, particularly being able to re-use existing tools like MyPy/Pyre.

In addition to the syntax proposed, some interesting cases to consider:

  • Representing generics in some way; in Python3 this looks like foo: List[str] or bar: Callable[[str, str], int]. This goes slightly beyond type annotations being represented by "any arbitrary expression", as these are pretty far from existing expressions (though I guess the hack Python does of saying "You can put anything the language wouldn't understand in a string" could be a reasonable workaround, at least while we flesh out detail). MyPy supports metaclass type variables too, though I feel like that's maybe getting into compromising the simplicity of Starlark that Alan cites...
  • What level of expression should be annotatable? I'd argue that at least variables of any scope, and functions in any scope, should allow annotating. Sub-expressions... Unclear.
ndmitchell

comment created time in 2 days

pull request commentpantsbuild/pants

Clean nodes with uncacheable dependencies once per session

Aah! Sorry, I forgot to revert part of the PR when running the tests before the change!

Ok, that makes sense, thanks! Would it be possible to add a test with a dep on both an Uncacheable and a Dirty node, to show that the Dirty node still results in a re-run?

stuhood

comment created time in 2 days

pull request commentpantsbuild/pants

Clean nodes with uncacheable dependencies once per session

I think that's true the first time, but if path/in/repo changed again (say, because of a second formatter), would we still re-run IsSameAsGoldenFile?

I believe so, yes. For any number of file-invalidation events, IsSameAsGoldenFile would be marked Dirty, and as long as it had any uncacheable dependencies it would be "cleaned" by moving to the UncacheableDependencies state. Dirtying the node does not lose track of the fact that the dependencies are uncacheable.

A missing piece might be that nodes are marked Dirty via invalidate_from_roots (see), but they "complete as" UncacheableDependencies. So being marked Dirty is transient, but having uncacheable dependencies is persistent.

Would the answer change if there was a second rule between ReadFileFromSourceTree and IsSameAsGoldenFile that did some preprocessing?

I don't think so. Because both dirtying and being in the UncacheableDependencies state are transitive.

Then what situation does this stop from happening? Sorry, I'm finding it hard to see the nuance here :) Can you give an example sequence of changes, either re-using the graph in my example, or constructing a new one, where we do less work now than we did before? Ideally in a unit test which was failing before this PR? All of the tests in it were passing before this change, which makes it hard to see what functionality actually changed...

stuhood

comment created time in 2 days

pull request commentpantsbuild/pants

Clean nodes with uncacheable dependencies once per session

Thanks for your thoughtful responses! Yeah, I think you're right that the incremental dirtying is probably a lot more hassle than it's worth...

Let's say IsSameAsGoldenFile has already executed in this run, because it has pleasantly few dependencies. Then let's say path/in/repo changes in the source tree (say, because it was changed by formatter).

When this happens, path/in/repo will be "cleared" (moved to NotStarted), and its dependees will be marked Dirty: that includes IsSameAsGoldenFile.

With this PR as it stands, I believe the semantics are to say "We already ran IsSameAsGoldenFile in this build, because it happens to have a transitive dep on GetGitShaFromBranchName we won't re-run it, even though that's unrelated to the reason we should dirty it". So we're going to return a stale result for IsSameAsGoldenFile.

It won't: it will be marked Dirty due to path/in/repo having changed: see the definition of Entry::dirty(): https://github.com/pantsbuild/pants/pull/10814/files#diff-75bfebc7637079e53ddae93c399f8495R107 . When it is cleaned, it will go to UncacheableDependencies for the runid.

So afaict, it would re-run appropriately in that case.

I think that's true the first time, but if path/in/repo changed again (say, because of a second formatter), would we still re-run IsSameAsGoldenFile? Would the answer change if there was a second rule between ReadFileFromSourceTree and IsSameAsGoldenFile that did some preprocessing?

                                    SomeGoal
                                       |
                                  SomeOtherRule
                                       |
                                IsSameAsGoldenFile
                                  /            \   
                     PreprocessFile   ReadFileFromBranch("path/in/repo", output_of(dep))
                             |                                |   
ReadFileFromSourceTree("path/in/repo")     GetGitShaFromBranchName("path/of/name/of/golden/branch", uncacheable=true)
stuhood

comment created time in 4 days

PullRequestReviewEvent
PullRequestReviewEvent
PullRequestReviewEvent

pull request commentbazelbuild/rules_rust

Extract cargo from rust toolchain tar

@illicitonion Any plans to make it available in ToolchainInfo?

I don't need it for my use-case, but feel free to do so if you do! :)

illicitonion

comment created time in 4 days

Pull request review commentpantsbuild/pants

Local process cache validates that digests exist locally before hitting

 impl CommandRunner {       })       .await?; -    if let Some((execute_response, platform)) = maybe_execute_response {+    // Deserialize the cache entry if it existed.+    let result = if let Some((execute_response, platform)) = maybe_execute_response {       crate::remote::populate_fallible_execution_result(         self.file_store.clone(),         execute_response.get_result(),         vec![],         platform,         true,       )-      .map(Some)       .compat()-      .await+      .await?     } else {-      Ok(None)-    }+      return Ok(None);+    };++    // Ensure that all digests in the result are loadable, erroring if any are not.+    let _ = future03::try_join_all(vec![+      self+        .file_store+        .ensure_local_has_file(result.stdout_digest)+        .boxed(),+      self+        .file_store+        .ensure_local_has_file(result.stderr_digest)+        .boxed(),+      self+        .file_store+        .ensure_local_has_recursive_directory(result.output_directory)

Are you saying that ensure_local_has_recursive_directory should be reimplemented in terms of comparing list_missing_digests methods? That sounds reasonable. You'd still need to do a recursive walk first to find all relevant digests before running that method, but it would avoid "loading" in favor of the equivalent of a "batch exists" method.

Yeah (tbh I thought it was already implemented that way, but apparently not!) - we can still start kicking off a few blob fetches while we do the Directory walk if we find we're bottlenecked on it. Then this method would only use the first half of that, without doing the follow-up fetches.

stuhood

comment created time in 5 days

PullRequestReviewEvent

Pull request review commentpantsbuild/pants

Local process cache validates that digests exist locally before hitting

 impl CommandRunner {       })       .await?; -    if let Some((execute_response, platform)) = maybe_execute_response {+    // Deserialize the cache entry if it existed.+    let result = if let Some((execute_response, platform)) = maybe_execute_response {       crate::remote::populate_fallible_execution_result(         self.file_store.clone(),         execute_response.get_result(),         vec![],         platform,         true,       )-      .map(Some)       .compat()-      .await+      .await?     } else {-      Ok(None)-    }+      return Ok(None);+    };++    // Ensure that all digests in the result are loadable, erroring if any are not.+    let _ = future03::try_join_all(vec![+      self+        .file_store+        .ensure_local_has_file(result.stdout_digest)+        .boxed(),+      self+        .file_store+        .ensure_local_has_file(result.stderr_digest)+        .boxed(),+      self+        .file_store+        .ensure_local_has_recursive_directory(result.output_directory)

This is actually a much stronger guarantee than I think we need, and may actually make cache hits slower than builds in the case of fast-ish actions with large-ish outputs... There's a potential optimisation here to merge the output of store::remote::Store::list_missing_digests and the currently not-existing store::local::Store::list_missing_digests to test for missing-ness, rather than forcing an entire download.

stuhood

comment created time in 6 days

PullRequestReviewEvent
PullRequestReviewEvent

delete branch illicitonion/num_enum

delete branch : fix-nightly

delete time in 6 days

push eventillicitonion/num_enum

Daniel Wagner-Hall

commit sha 31cbcbddc89a164280ea75f1bd5a22a4aedf93b7

Allow nightly to have different error messages (#29)

view details

push time in 6 days

PR opened illicitonion/num_enum

Reviewers
Allow nightly to have different error messages
+98 -2

0 comment

3 changed files

pr created time in 6 days

create barnchillicitonion/num_enum

branch : fix-nightly

created branch time in 6 days

delete branch illicitonion/rules_rust

delete branch : cargo

delete time in 6 days

create barnchillicitonion/cargo-raze

branch : wip

created branch time in 6 days

PR opened bazelbuild/rules_rust

Extract cargo from rust toolchain tar

This allows for cargo to be used from scripts which may want to invoke cargo, while coupling the version used with the rest of the toolchain used for the build.

+1 -1

0 comment

1 changed file

pr created time in 6 days

create barnchillicitonion/rules_rust

branch : cargo

created branch time in 6 days

pull request commentillicitonion/num_enum

Ensure tests detect a regression of #27 + re-enabled no_std test back

Sounds good, thanks! I have a fix for the nightly thing I just need to clean up a little, will send it your way in a few hours!

danielhenrymantilla

comment created time in 6 days

push eventillicitonion/cargo-raze

Alex McArther

commit sha 2450b9a220a7a4488ebe6010a270596c215eb788

Up to 0.5.0 [via publish-new-version.sh]

view details

UebelAndre

commit sha fa1672cef9c6b2c4788136e9f23916201afba132

Multi target support (#212) Similar to #54 this pull request adds support for generating Bazel targets for multiple platforms. This is still kinda **WIP** but I wanted to share. I'll update this description as the feature evolves but for now am looking for any insight on what may be missing for the initial release of this feature. Output `BUILD` file: ```python """ @generated cargo-raze crate build file. DO NOT EDIT! Replaced on runs of cargo-raze """ # buildifier: disable=load load( "@io_bazel_rules_rust//rust:rust.bzl", "rust_binary", "rust_library", "rust_test", ) # buildifier: disable=load load("@bazel_skylib//lib:selects.bzl", "selects") package(default_visibility = [ # Public for visibility by "@raze__crate__version//" targets. # # Prefer access through "//cbindgen/raze", which limits external # visibility to explicit Cargo.toml dependencies. "//visibility:public", ]) licenses([ "notice", # MIT from expression "MIT" ]) # Generated targets # Unsupported target "atty" with type "example" omitted # buildifier: leave-alone rust_library( name = "atty", crate_type = "lib", deps = [ ] + selects.with_or({ # cfg(unix) ( "@io_bazel_rules_rust//rust/platform:aarch64-apple-ios", "@io_bazel_rules_rust//rust/platform:aarch64-linux-android", "@io_bazel_rules_rust//rust/platform:aarch64-unknown-linux-gnu", "@io_bazel_rules_rust//rust/platform:arm-unknown-linux-gnueabi", "@io_bazel_rules_rust//rust/platform:i686-apple-darwin", "@io_bazel_rules_rust//rust/platform:i686-linux-android", "@io_bazel_rules_rust//rust/platform:i686-unknown-freebsd", "@io_bazel_rules_rust//rust/platform:i686-unknown-linux-gnu", "@io_bazel_rules_rust//rust/platform:powerpc-unknown-linux-gnu", "@io_bazel_rules_rust//rust/platform:s390x-unknown-linux-gnu", "@io_bazel_rules_rust//rust/platform:x86_64-apple-darwin", "@io_bazel_rules_rust//rust/platform:x86_64-apple-ios", "@io_bazel_rules_rust//rust/platform:x86_64-linux-android", "@io_bazel_rules_rust//rust/platform:x86_64-unknown-freebsd", "@io_bazel_rules_rust//rust/platform:x86_64-unknown-linux-gnu", ): [ "@rules_rust_cbindgen__libc__0_2_76//:libc", ], "//conditions:default": [], }) + selects.with_or({ # cfg(windows) ( "@io_bazel_rules_rust//rust/platform:i686-pc-windows-gnu", "@io_bazel_rules_rust//rust/platform:x86_64-pc-windows-gnu", ): [ "@rules_rust_cbindgen__winapi__0_3_9//:winapi", ], "//conditions:default": [], }), srcs = glob(["**/*.rs"]), crate_root = "src/lib.rs", edition = "2015", rustc_flags = [ "--cap-lints=allow", ], version = "0.2.14", tags = [ "cargo-raze", "manual", ], crate_features = [ ], ) ``` Note that the `deps` section can be filtered down to a whitelist of platform triples using the new `targets` setting. ```toml targets = [ "aarch64-unknown-linux-gnu", "i686-apple-darwin", "i686-pc-windows-gnu", "i686-unknown-linux-gnu", "powerpc-unknown-linux-gnu", "x86_64-apple-darwin", "x86_64-pc-windows-gnu", "x86_64-unknown-linux-gnu", ] ``` The list above creates the following diff in the output BUILD file. ```diff diff --git a/cbindgen/raze/remote/atty-0.2.14.BUILD.bazel b/cbindgen/raze/remote/atty-0.2.14.BUILD.bazel index dfcf387..6002194 100644 --- a/cbindgen/raze/remote/atty-0.2.14.BUILD.bazel +++ b/cbindgen/raze/remote/atty-0.2.14.BUILD.bazel @@ -48,21 +48,12 @@ rust_library( }) + selects.with_or({ # cfg(unix) ( + "@io_bazel_rules_rust//rust/platform:aarch64-unknown-linux-gnu", "@io_bazel_rules_rust//rust/platform:i686-apple-darwin", "@io_bazel_rules_rust//rust/platform:i686-unknown-linux-gnu", + "@io_bazel_rules_rust//rust/platform:powerpc-unknown-linux-gnu", "@io_bazel_rules_rust//rust/platform:x86_64-apple-darwin", "@io_bazel_rules_rust//rust/platform:x86_64-unknown-linux-gnu", - "@io_bazel_rules_rust//rust/platform:aarch64-apple-ios", - "@io_bazel_rules_rust//rust/platform:aarch64-linux-android", - "@io_bazel_rules_rust//rust/platform:aarch64-unknown-linux-gnu", - "@io_bazel_rules_rust//rust/platform:powerpc-unknown-linux-gnu", - "@io_bazel_rules_rust//rust/platform:arm-unknown-linux-gnueabi", - "@io_bazel_rules_rust//rust/platform:s390x-unknown-linux-gnu", - "@io_bazel_rules_rust//rust/platform:i686-linux-android", - "@io_bazel_rules_rust//rust/platform:i686-unknown-freebsd", - "@io_bazel_rules_rust//rust/platform:x86_64-apple-ios", - "@io_bazel_rules_rust//rust/platform:x86_64-linux-android", - "@io_bazel_rules_rust//rust/platform:x86_64-unknown-freebsd", ): [ "@rules_rust_cbindgen__libc__0_2_76//:libc", ], ```

view details

Daniel Wagner-Hall

commit sha 40c78876fa4df8ff4a34ef5afb2e0b4087ff1ff9

Try to fill in sha256es from local index mirror If anything goes wrong, we simply fall back to the current behaviour of omitting them.

view details

push time in 6 days

pull request commentgoogle/cargo-raze

Try to fill in sha256es from local index mirror

I think the bigger issue is that after switching to cargo-metadata in #149, the lock file is no longer updated. Running cargo generate-lockfile before cargo raze works (and the hashes will be correctly propagated), but that forces updates you might not want. It's also an annoying manual step to remember to do.

If a Cargo.lock already exists, anything still relevant from it will still be used, right?

One alternative option would be to improve cargo-metadata to include hashes in its output and skip parsing the lock file at all, but I don't think anyone has made a conscious decision about cargo-raze not updating the lock file. It's just something we didn't catch in #149.

Eric Huss said they were open to doing so in https://internals.rust-lang.org/t/stable-way-of-getting-registry-index-data/13071 but even after implementing, that would realistically be a multi-month wait for release cycles - I think this filling in is still valuable until that could happen (and this code automatically becomes a no-op when that's done).

Anyways, I'm also hesitant on adding more crates.io-specific code as Cargo supports third-party registries. Raze currently hard codes the crates.io bucket URL in the template, but I think that's the only blocker for supporting them.

I've switched to the version of the code which is generic over all registries, rather than always assuming crates.io.

illicitonion

comment created time in 6 days

pull request commentgoogle/cargo-raze

Try to fill in sha256es from local index mirror

Do you know what the release cadence is for https://github.com/frewsxcv/rust-crates-index.git ? Maybe an issue can be created to try and push that through?

I mentioned that a release would be handy in a comment on https://github.com/frewsxcv/rust-crates-index/pull/42 - if there isn't one in the next few days I will nudge :)

illicitonion

comment created time in 6 days

issue commentillicitonion/num_enum

TryFromPrimitive: failed to resolve: could not find `result` in `core`

It feels like having your own crate named core is:

  • Probably a pretty bad idea
  • Pretty niche
  • Easy to avoid

So I'm inclined to say let's leave this unsupported and recommend anyone who runs into this in the future renames their crate.

I wouldn't strongly object to adding an optional feature, but it feels like the burden of needing to test this in the future, and generally be aware of it when making changes, probably doesn't merit the value the support would bring.

stijnfrishert

comment created time in 6 days

pull request commentillicitonion/num_enum

Ensure tests detect a regression of #27 + re-enabled no_std test back

FWIW I started bisecting the nightly breakage (we should maybe set up a nightly cron on travis to catch these in the future) - the tests were green in nightly 2020-09-04 and red in 2020-09-05

danielhenrymantilla

comment created time in 7 days

Pull request review commentillicitonion/num_enum

Ensure tests detect a regression of #27 + re-enabled no_std test back

  extern crate alloc; -use ::alloc::format;-use ::core::convert::TryFrom;+/// Un-rename `num_enum` when in a no-std crate, since it is not supported yet.+#[cfg(not(feature = "std"))]

I still don't entirely see the value, but I also don't think it does any harm, so... 🤷 shipit :)

danielhenrymantilla

comment created time in 7 days

PullRequestReviewEvent

PR opened google/cargo-raze

Try to fill in sha256es from local index mirror

If anything goes wrong, we simply fall back to the current behaviour of omitting them.

+191 -3

0 comment

3 changed files

pr created time in 7 days

create barnchillicitonion/cargo-raze

branch : sha256es

created branch time in 7 days

delete branch illicitonion/rust-crates-index

delete branch : new_bare_unwrap

delete time in 7 days

pull request commentfrewsxcv/rust-crates-index

Inline unwrap

This code has not yet been released (though I'd love a release if possible!), so this isn't a breaking change against any release :)

illicitonion

comment created time in 7 days

PR opened frewsxcv/rust-crates-index

Inline unwrap

The crates.io index is known to safely convert to a path, don't force the caller to consider errors in this case.

+3 -2

0 comment

1 changed file

pr created time in 7 days

create barnchillicitonion/rust-crates-index

branch : new_bare_unwrap

created branch time in 7 days

create barnchillicitonion/rust-crates-index

branch : public-bare-index-repo

created branch time in 7 days

fork illicitonion/rust-crates-index

Rust library for retrieving and interacting with the crates.io index

https://docs.rs/crates-index/

fork in 7 days

PR opened google/cargo-raze

Don't copy Cargo.toml to tempdir where possible

I'm not entirely clear on why we're copying out in the first place, but I added a comment with my best guess.

Avoid doing this where possible, because it breaks support for Cargo workspaces.

Alternatively, we could try to parse the Cargo.toml for its workspace members and copy those too, but that seems to be breaking more abstraction layers.

The limited cases of Workspaces working is probably also worth documenting, though I'm not sure where?

+20 -4

0 comment

1 changed file

pr created time in 7 days

create barnchillicitonion/cargo-raze

branch : tempdir

created branch time in 7 days

push eventillicitonion/cargo-raze

Daniel Wagner-Hall

commit sha 2451fbe9392b2fc81569d2ad00ed888e49a14a54

Use new cargo_build_script_run rule Instead of a custom genrule. See https://github.com/bazelbuild/rules_rust/pull/320

view details

Daniel Wagner-Hall

commit sha b588af0780a60d6d214bc3819c64e4526c5aa6f6

Glob more sources

view details

Daniel Wagner-Hall

commit sha e2cc36f292514918a8d17cbced1dbcccf805c68c

Bump rules_rust for examples

view details

Daniel Wagner-Hall

commit sha ff20b0bd3e0a9c60c631c8d7a8026cf4e4229004

Use crate_name_sanitized

view details

Damien Martin-Guillerez

commit sha f6ad8dd47d1fb962922adcf5fa12e20b33ac3f19

Merge pull request #166 from illicitonion/build-script Use new cargo_build_script_run rule

view details

David Freese

commit sha de9b1a9415367192eec21e61d0f2bc4234cf090b

Fix some spelling and grammar errors in README (#169) This was brought up in #68.

view details

Daniel Wagner-Hall

commit sha 7273ab4d044d58ae468a313493f45778db2f43d2

Bump example workspace to latest master (#168) This will be needed for #166

view details

Daniel Wagner-Hall

commit sha d0ed12d96c595de11ccb1c4ac6c5b1627a9504d3

Add workspace-wide default for gen_buildrs (#167) I want to optimistically default to generating build scripts by default.

view details

Daniel Wagner-Hall

commit sha cd573403a9c51351683044c10f5c91d5700d9d1b

proc_macro_deps is its own attribute (#164) This allows an unconditional transition to the `exec` config, rather than needing a conditional transition (which cannot transition to the `exec` config, only to the `host` config). Requires rules_rust commit c409198dcc or later.

view details

Gibson Fahnestock

commit sha 6e07debceb9800285c74d697de9b0fadd2f55060

docs: minor README clarifications on genmode (#173)

view details

Alex McArther

commit sha 69784a0633827fc72fb30f989f9284e243ea2f2f

Update to 0.3.4

view details

Wyatt Calandro

commit sha ad14d2bf2d1aef3103321fb3c9110983274b0b84

Fix: Make crate license identification more robust (#174) * Support more robust SPDX licenses * Update template and change nomenclature * Fix some bugs and add more tests * Fix expression assignments * Return LicenseData and fix nomenclature * Implement default for license

view details

Alex McArther

commit sha 24c311b8e5fed6d4f61b2277d22efea0131c4fd0

Update to 0.3.5

view details

Alexei Filippov

commit sha a2115bb8ecc21d5a16f5f7ea34193e2ae60067fb

Add build script dependencies on sys crates build scripts. (#170) System crates build scripts may produce environment variables that may be used by their direct dependencies build script. For example openssl-sys generates DEP_OPENSSL_VERSION variable that is then used by openssl crate build scripts. Co-authored-by: Alexei Filippov <alph@apple.com>

view details

boxdot

commit sha d966255154fd1eb9f5c7af66bcd3e5cf9161b676

Add --output param specifying output folder for generated files. (#175) Also fix: * remote folder created on dry run * remote/BUILD folder does not use prefix path parameter

view details

Alexei Filippov

commit sha de2b4f1dcff58f711d64dd9722f40957ce708805

Tag all cargo-raze rules with "cargo-raze" tag. (#176) The tags can be used to exclude cargo-raze generated rules from, e.g. "//...:all" targets. It is useful to prevent Bazel from building unsupported targets in multiplatform configurations. Co-authored-by: Alexei Filippov <alph@apple.com>

view details

Alex McArther

commit sha 99f9e37525efe654f75203c0eb22f6c134ef3161

Update to 0.3.6

view details

Damien Martin-Guillerez

commit sha 8ee049ee19fa686209a40ffc6787863a7ba3e808

Support additional enviroment variable for build script (#180) Since the switch to the cargo_build_script rule, the buildrs_additional_environment_variables was no longer working, this change add this support back. Fixes #179

view details

Alex McArther

commit sha 439f3935c1c084dd3895ad7a65c0ceeef2d5b2e9

Update to 0.3.7

view details

David Freese

commit sha 5bd36b5e6ab8fb4e955deeb3f15c727788f7a65e

Handle proc macros in build dependencies (#182) Proc macro crates were split off into their own field in #164 and then PR #166 changed handling of build scripts to invoke cargo_build_script, however, build dependencies can also have proc macros. This was an issue for markup5ever, which has a build dependency that transitively depends on serde_derive. Fixes #181

view details

push time in 7 days

Pull request review commentillicitonion/num_enum

Ensure tests detect a regression of #27 + re-enabled no_std test back

+#![cfg_attr(rustfmt, rustfmt::skip)] // args

Why skip the formatting? While I don't love what rustfmt does with this file, I think I still prefer consistency...

danielhenrymantilla

comment created time in 8 days

Pull request review commentillicitonion/num_enum

Ensure tests detect a regression of #27 + re-enabled no_std test back

  extern crate alloc; -use ::alloc::format;-use ::core::convert::TryFrom;+/// Un-rename `num_enum` when in a no-std crate, since it is not supported yet.+#[cfg(not(feature = "std"))]

It seems strange to make this conditional, given submodule contains use ::renamed which will presumably error if !std? I'd be tempted to just remove this cfg, if the thing is going to fail anyway...

danielhenrymantilla

comment created time in 8 days

PullRequestReviewEvent
PullRequestReviewEvent

pull request commentpantsbuild/pants

Add `runtime_binary_dependencies` field to `python_tests` target

How would you document the current incarnation?

New section to https://www.pantsbuild.org/v2.0/docs/python-test-goal titled "How to depend on built binaries"

In integration tests, you may want to build your binary targets, like python_binary targets, and use the built binary files in your test. You can then, for example, run the binary in the test, or inspect the contents of the binary. To do this, use the runtime_binary_dependencies field by adding the addresses to any binary targets you want built. Before your test is run, Pants will then call the equivalent of ./pants binary and populate the chroot (temporary directory) with your binary. For example:

I think it would be harder to document if files was included, as the docs would have to be more general and wouldn't have as much special-cased explanation about, e.g., running "the equivalent of ./pants binary.

Sorry, I think I was not super clear - what I meant was how would you document where you should put files targets which you want to access at runtime? That's where "Anything you want at runtime goes in runtime_dependencies" feels like a nice simple rule...

Eric-Arellano

comment created time in 8 days

delete branch illicitonion/pants

delete branch : childir

delete time in 8 days

push eventpantsbuild/pants

Daniel Wagner-Hall

commit sha fa716756acc42295a464cc3a704c5dd58f1b8354

fs_util cat-proto has --child-dir arg (#10228) This lets it operate on a subtree of a Directory, rather than the whole thing.

view details

push time in 8 days

PR merged pantsbuild/pants

fs_util cat-proto has --child-dir arg

This lets it operate on a subtree of a Directory, rather than the whole thing.

+22 -4

2 comments

2 changed files

illicitonion

pr closed time in 8 days

pull request commentpantsbuild/pants

Add `runtime_binary_dependencies` field to `python_tests` target

Should we require that files targets be in this list?

Rather than in the dependencies field? I don't think so. The reason this is a dedicated field is that we are doing something really special; we are not simply copying the sources into the chroot like normal. We are doing a super side-effecty thing of running ./pants binary.

I think having files be in a runtime_dependencies field rather than dependencies makes sense... While I can see the argument of "we're doing something different in how we make them". I find it really easy to document that rule:

"If you expect the file to be used at build time, add it to dependencies. If you expect the file to be used at runtime, add it to runtime_dependencies."

How would you document the current incarnation?

Eric-Arellano

comment created time in 9 days

push eventillicitonion/pants

Daniel Wagner-Hall

commit sha ad5746f510e4205072d087d70f53f07cefdc053a

fs_util cat-proto has --child-dir arg This lets it operate on a subtree of a Directory, rather than the whole thing.

view details

push time in 9 days

push eventillicitonion/pants

Eric Arellano

commit sha 238b3b73bf641e5f293fcd77955594779c642a99

Add new `typecheck` goal for MyPy (#10212) While it would be ideal to have `lint` run MyPy too, we realized that it does not work well with the `--changed-since` flag. Normally, we recommend running `./pants --changed-since=main lint`, which only runs over directly changed targets. With MyPy, you will also want to specify `--changed-include-dependees=transitive`. So, you must either choose between a) running over too few targets if you leave off dependees, or b) wasting time running Black/Flake8/Isort over unnecessary targets. We want to optimize for the `--changed-since` use case, especially with local runs when iterating, so this is an important blocker. [ci skip-rust-tests]

view details

Daniel Wagner-Hall

commit sha 77a111c36f2862ca5356e04dcefc2e07e3b39a69

local_cas supports GetCapabilities request (#10226)

view details

Daniel Wagner-Hall

commit sha 3a449abe23cb5fec87042f148159411c2083bf55

local_cas supports an instance-name (#10225)

view details

Benjy Weinberger

commit sha 0d0bd39bceb3c1f3707d8a09caa57be6de1f699a

Delete v1 python backend task code (#10223) [ci skip-rust-tests]

view details

Asher Foa

commit sha 334623358300a8b67b6b7b7d4f8f15664de00749

Add the ability to write/output the raw coverage file. (#10195) ### Problem Some code coverage processing tools consume the raw coverage file/DB generated during test. Currently, there is no way to get it when running under Pants. ### Solution Add another coverage report type for the raw coverage data file. Fixes: https://github.com/pantsbuild/pants/issues/9968

view details

gshuflin

commit sha 9651d48c1707930777a0852a502142bdec195e4a

Reduce time spend grabbing locks in workunit code (#10179) ### Problem This is an attempt to alleviate console UI glitches such as https://github.com/pantsbuild/pants/issues/9962 by reducing the amount of time that other code spends grabbing the same locks in `WorkunitStore` that the UI code needs to grab to figure out what workunits need to be rendered. ### Solution This commit splits the `WorkunitStore` struct into two structs containing different data structures under different locks, one responsible for handling the heavy hitters calculation, the other for handling streaming workunits. Additionally, the methods that actually add started and complete workunits to the store have been modified to push those workunits onto a Rust `mpsc::Sender`/`Receiver` queue, so the only locks they need to grab are the ones associated with the sender side of those queues. `heavy_hitters` and `with_latest_workunits` will pop workunits off these queues each time they are called, and then process them separately under different locks, so it is now possible for both of them to be running at the same time.

view details

Eric Arellano

commit sha 349e93904c3786f0f0a816551dafee4af921102c

Teach `pants_requirement()` to work with dependency inference (#10232) For third-party requirements, we must tell Pants what modules they export, e.g. `pantsbuild.pants` exporting `pants`. Our heuristic of using the dist's name does not work because every Pants dist starts with `pantsbuild.`. [ci skip-rust-tests]

view details

Eric Arellano

commit sha 0d47e6a8e20fd5f5341b0378851d9914d06c1729

Remove some v1 parts of `testutil/` (#10233) These imports are either no longer relevant or will fail with v1 fully removed. [ci skip-rust-tests]

view details

Eric Arellano

commit sha 58cd9d7ad72f9574ee406492829c6f61863725f5

Remove `core_tasks/` and most of `task/` (#10236) [ci skip-rust-tests]

view details

Eric Arellano

commit sha a5a04c58544a46e503b6883fb5d1358f59fa95c1

Fix `--changed-dependees` to work when v1 is disabled (#10235) While we were using a rule to calculate the dependees, it was using a v1 code path based on TargetAdaptor. This means that `--changed-dependees` caused a crash when v1 is disabled. To land this, we refactor `dependees` into two helper rules so that any rule can now `await Get(Dependees, DependeesRequest)`. [ci skip-rust-tests]

view details

Eric Arellano

commit sha 6af8bc8142ec5575947b8f87dccdf629bc4c45b7

Remove `TestBase.create_library() and `TestBase.target()` (#10237) These methods all use v1 code paths so are blocking the deletion of v1 code. Likely, we will want to create some new helper methods. Currently, all v2 code relies on `self.add_to_build_file` and then `self.request_single_product(WrappedTarget, Address)`. This can likely be streamlined. But we will want to create this from scratch, rather than trying to salvage the v1 methods. [ci skip-rust-tests]

view details

gshuflin

commit sha c4f9c6a021a905de297dd7adae904bdfffe6566e

Use debug level for remote store workunits #10238 To reduce verbosity in CI, change some of the levels of workunit items associated with the remote store from INFO to DEBUG.

view details

Benjy Weinberger

commit sha 65423af4de7ef7c2c66d42886a78737b857966a6

Delete the old BinaryTool mechanism. (#10239) It's replaced by ExternalTool. This change switches the last remaining users of BinaryTool over to ExternalTool, and deletes the former. Note that the native tool classes had extra utility methods on them that assumed that tools were downloaded locally. This is not the v2 way of doing things. These utilities were deleted, and the native tool using code will need to be rewritten to accept a digest+relpath, in the v2 idiom. [ci skip-rust-tests]

view details

Benjy Weinberger

commit sha d2e04c48c6d7b5b055d34bf7885bec76294aeb92

Get rid of --plugins2/--backend-packages2. (#10231) * Get rid of --plugins2/--backend-packages2. Instead use --plugins/--backend-packages, as there is now no v1 to get in the way.

view details

Eric Arellano

commit sha fc96d14e465857ff827b16e09412c3a25e99f05f

Prepare 2.0.0.dev1 (#10243) [ci skip-rust-tests]

view details

Eric Arellano

commit sha 4c391a5f57044912b2b372c5587e7fd8db845890

Simplify ci.py now that we have no v1 tests (#10241) We can remove the block list mechanism now.

view details

Benjy Weinberger

commit sha 85003848427566f9611a79009c083600f22d9043

Remove the concept of a scope category. (#10224) In v2 all optionables are subsystems, even global options. There is no meaning to categories any more. [ci skip-rust-tests]

view details

Eric Arellano

commit sha aa5d9f12cb17864ae0e1b2af04741c2acb179539

Delete the rest of v1 `pants.backend.python` (#10240) We would like to be able to remove `setup_py_runner.py`, `executable_pex_tool.py`, and `pex_build_util.py`, but those are used by the test `test_plugin_resolver.py`. [ci skip-rust-tests]

view details

Eric Arellano

commit sha aa1d24f06e0a9ed0b6e3e5bd0a27b8d469462589

Remove `TestBase.context()` (#10248) This will make it easier to delete `Task`. [ci skip-rust-tests]

view details

Eric Arellano

commit sha 40ba9d9ad77d8be6176226eab746c52bf8495034

Remove all remaining v1 Targets (#10246) This will make it easier to delete the core `Target` abstractions. [ci skip-rust-tests]

view details

push time in 9 days

Pull request review commentpantsbuild/pants

fs_util cat-proto has --child-dir arg

 async fn execute(top_match: &clap::ArgMatches<'_>) -> Result<(), ExitError> {           .unwrap()           .parse::<usize>()           .expect("size_bytes must be a non-negative number");-        let digest = Digest(fingerprint, size_bytes);+        let mut digest = Digest(fingerprint, size_bytes);++        if let Some(child_dir) = args.value_of("child-dir") {+          let mut accumulated_components = PathBuf::new();+          for component in Path::new(child_dir) {+            if component.is_empty() {+              continue;+            }+            accumulated_components.push(component);+            let (directory, _) = store.load_directory(digest).await?.ok_or_else(|| {+              format!(+                "Couldn't find digest {:?} for child dir {:?}",+                digest, accumulated_components+              )+            })?;+            let mut found_child = false;+            for directory in directory.get_directories() {+              if directory.get_name() == component {+                let maybe_digest: Result<Digest, String> = directory.get_digest().try_into();+                digest = maybe_digest.map_err(|err| {+                  format!(+                    "Digest of child dir {:?} couldn't be parsed: {:?}",+                    accumulated_components, err+                  )+                })?;+                found_child = true+              }+            }+            if !found_child {+              panic!("Didn't find child dir {:?}", accumulated_components);+            }+          }

Re-implemented in terms of SnapshotOps::strip_prefix - PTAL!

illicitonion

comment created time in 9 days

PullRequestReviewEvent

delete branch illicitonion/pants

delete branch : osfamily

delete time in 9 days

PR closed pantsbuild/pants

Use shared standard keys and values for platform

Note that this is a breaking change for some server configs.

+233 -49

2 comments

3 changed files

illicitonion

pr closed time in 9 days

pull request commentpantsbuild/pants

Use shared standard keys and values for platform

Closing in favour of https://github.com/pantsbuild/pants/pull/10713

illicitonion

comment created time in 9 days

pull request commentrust-lang/rfcs

Calling methods on generic parameters of const fns

Related to const bounds, has thought been given to blanket impls?

e.g. right now in core there is a blanket impl<T, U> Into<U> for T where U: From<T>

Ideally if I implement const From, the blanket Into will also be const. This presumably wouldn't be automatic, but I guess something like this could be written?

impl<T, U> const Into<U> for T where U: const From<T> {...}
impl<T, U> ?const Into<U> for T where U: ?const From<T> {...}

Perhaps there could be some way of condensing this down to one impl, and signifying somehow that the const-ness is sticky? I feel like swapping the prefix/suffix-ness of the ? would probably be confusing, but something like:

impl<T, U> const? Into<U> for T where U: const? From<T> {...}
oli-obk

comment created time in 13 days

issue commentbazelbuild/remote-apis

Clarify what a client should do on FAILED_PRECONDITION

Last I talked with @anakanemison, rewinding was incompatible with the default Skyframe graph implementation, but admittedly, that was 8+ months ago. @illicitonion, did you see actual rewinding, or did you 'just' enable the flag? I wonder if Bazel might be silently ignoring the flag.

IIRC my experiment was roughly (all with build without the bytes):

  • Set up a chain of 100 genrules each of which depends on output of the last
  • Execute a build for the first 99 actions
  • Flushed the remote CAS
  • Executed a build for the last action I observed all 99 of the required actions being re-run, and the 100th succeeding.

I’ve observed that enabling builds without the bytes causes Bazel to rebuild stuff unnecessarily. Did you verify that no rebuilding happened when you didn’t flush the remote CAS? It may have been the case that Bazel always did a rebuild, regardless of whether data was present in the CAS or not.

I'm pretty sure I did, but I'm not certain. I'll run some experiments tomorrow :)

sstriker

comment created time in 13 days

PullRequestReviewEvent

issue commentbazelbuild/remote-apis

Clarify what a client should do on FAILED_PRECONDITION

Last I talked with @anakanemison, rewinding was incompatible with the default Skyframe graph implementation, but admittedly, that was 8+ months ago. @illicitonion, did you see actual rewinding, or did you 'just' enable the flag? I wonder if Bazel might be silently ignoring the flag.

IIRC my experiment was roughly (all with build without the bytes):

  • Set up a chain of 100 genrules each of which depends on output of the last
  • Execute a build for the first 99 actions
  • Flushed the remote CAS
  • Executed a build for the last action I observed all 99 of the required actions being re-run, and the 100th succeeding.
sstriker

comment created time in 13 days

issue commentbazelbuild/remote-apis

Clarify what a client should do on FAILED_PRECONDITION

action rewinding is already mostly implemented, except for that one missing piece (famous last words?).

Can you elaborate on what's missing (maybe a link to an issue?) - I was trying it out a few weeks ago and it seemed to be working fine for all of the use-cases I tested out, but those use-cases were all fairly simple.

sstriker

comment created time in 14 days

issue commentbazelbuild/bazel

Obtaining output paths of the build artifacts

I ended up putting together this script which uses aquery to get the output path for a particular mnemonic:

#!/bin/bash -eu

# Prints the path to the (single) output file of the action with the given mnemonic, for the given target.

if [[ $# != 2 ]]; then
  echo >&2 "Usage: $0 target mnemonic"
  echo >&2 "e.g. $0 //some:target JavaSourceJar"
  exit 1
fi
target="$1"
mnemonic="$2" # e.g. GoLink or PythonZipper

out="$(mktemp)"
trap "rm -f ${out}" EXIT
bazel aquery --output=jsonproto "${target}" > "${out}"

outputs_id_query_base=".actions[] | select(.[\"mnemonic\"] == \"${mnemonic}\")| .outputIds"
output_count="$(jq "${outputs_id_query_base} | length" "${out}")"
if [[ "${output_count}" -ne 1 ]]; then
  echo >&2 "Saw ${output_count} outputs for the mnemonic and target but expected 1"
  exit 1
fi
output_id="$(jq "${outputs_id_query_base}[0]" "${out}")"
jq -r ".artifacts[] | select(.[\"id\"] == ${output_id}) | .execPath" "${out}"
vitarb

comment created time in 14 days

pull request commentpantsbuild/pants

Support binary dependencies in tests.

I think the key to model here is there's a difference between "Things I need to compile" and "Things I need to run". Even just that two-way split gets complicated for more complex integration tests ("The server I'm running" and "The client which runs on a mobile device and talks to the server", for instance). But I'd highly recommend introducing a separate attribute, rather than conflating the two :) Maybe instead of this, we could introduce a runtime_dependencies field?

There's also some interesting questions around how a binary can specify it has runtime files it needs to, and where they should be located. I guess the former has been handled by the _app targets in v1, and the latter has had some ad-hoc solutions but nothing super consistent...

benjyw

comment created time in 15 days

create barnchillicitonion/rust

branch : proc-macro

created branch time in 22 days

fork illicitonion/rust

Empowering everyone to build reliable and efficient software.

https://www.rust-lang.org

fork in 22 days

pull request commentpantsbuild/pants

remoting: remove target_platform property

As far as I remember, this was introduced because without it, we got invalid cache hits in the following situation:

  1. From a Mac, build a platform-specific pex remotely on Linux
  2. From the same Mac, build the same pex without remote execution Which led to us getting a cache hit for a Linux-specific pex when we should have gotten a Mac-specific one.

I think the platform-specifying stuff ended up pretty half-baked, and probably nothing meaningfully uses it, so this is probably fine, but if we're going down this road, we probably also want to remove all the MultiPlatformProcess stuff from both the Python and Rust side.

It's fine for us to say "we don't factor target platform into cache keys", but if so, we shouldn't pretend to have abstractions which rely on running on specific platforms (and will need to build those abstractions up again when they're actually needed).

tdyas

comment created time in 22 days

PullRequestReviewEvent

issue commentpantsbuild/pants

Remote execution uses non-standard Platform property where there is a standardised version

I just revived https://github.com/pantsbuild/pants/pull/10227 which I think does what you're describing? Let me know if you're suggesting something different (otherwise hopefully its CI will be green this time!)

illicitonion

comment created time in 23 days

push eventillicitonion/pants

Eric Arellano

commit sha 238b3b73bf641e5f293fcd77955594779c642a99

Add new `typecheck` goal for MyPy (#10212) While it would be ideal to have `lint` run MyPy too, we realized that it does not work well with the `--changed-since` flag. Normally, we recommend running `./pants --changed-since=main lint`, which only runs over directly changed targets. With MyPy, you will also want to specify `--changed-include-dependees=transitive`. So, you must either choose between a) running over too few targets if you leave off dependees, or b) wasting time running Black/Flake8/Isort over unnecessary targets. We want to optimize for the `--changed-since` use case, especially with local runs when iterating, so this is an important blocker. [ci skip-rust-tests]

view details

Daniel Wagner-Hall

commit sha 77a111c36f2862ca5356e04dcefc2e07e3b39a69

local_cas supports GetCapabilities request (#10226)

view details

Daniel Wagner-Hall

commit sha 3a449abe23cb5fec87042f148159411c2083bf55

local_cas supports an instance-name (#10225)

view details

Benjy Weinberger

commit sha 0d0bd39bceb3c1f3707d8a09caa57be6de1f699a

Delete v1 python backend task code (#10223) [ci skip-rust-tests]

view details

Asher Foa

commit sha 334623358300a8b67b6b7b7d4f8f15664de00749

Add the ability to write/output the raw coverage file. (#10195) ### Problem Some code coverage processing tools consume the raw coverage file/DB generated during test. Currently, there is no way to get it when running under Pants. ### Solution Add another coverage report type for the raw coverage data file. Fixes: https://github.com/pantsbuild/pants/issues/9968

view details

gshuflin

commit sha 9651d48c1707930777a0852a502142bdec195e4a

Reduce time spend grabbing locks in workunit code (#10179) ### Problem This is an attempt to alleviate console UI glitches such as https://github.com/pantsbuild/pants/issues/9962 by reducing the amount of time that other code spends grabbing the same locks in `WorkunitStore` that the UI code needs to grab to figure out what workunits need to be rendered. ### Solution This commit splits the `WorkunitStore` struct into two structs containing different data structures under different locks, one responsible for handling the heavy hitters calculation, the other for handling streaming workunits. Additionally, the methods that actually add started and complete workunits to the store have been modified to push those workunits onto a Rust `mpsc::Sender`/`Receiver` queue, so the only locks they need to grab are the ones associated with the sender side of those queues. `heavy_hitters` and `with_latest_workunits` will pop workunits off these queues each time they are called, and then process them separately under different locks, so it is now possible for both of them to be running at the same time.

view details

Eric Arellano

commit sha 349e93904c3786f0f0a816551dafee4af921102c

Teach `pants_requirement()` to work with dependency inference (#10232) For third-party requirements, we must tell Pants what modules they export, e.g. `pantsbuild.pants` exporting `pants`. Our heuristic of using the dist's name does not work because every Pants dist starts with `pantsbuild.`. [ci skip-rust-tests]

view details

Eric Arellano

commit sha 0d47e6a8e20fd5f5341b0378851d9914d06c1729

Remove some v1 parts of `testutil/` (#10233) These imports are either no longer relevant or will fail with v1 fully removed. [ci skip-rust-tests]

view details

Eric Arellano

commit sha 58cd9d7ad72f9574ee406492829c6f61863725f5

Remove `core_tasks/` and most of `task/` (#10236) [ci skip-rust-tests]

view details

Eric Arellano

commit sha a5a04c58544a46e503b6883fb5d1358f59fa95c1

Fix `--changed-dependees` to work when v1 is disabled (#10235) While we were using a rule to calculate the dependees, it was using a v1 code path based on TargetAdaptor. This means that `--changed-dependees` caused a crash when v1 is disabled. To land this, we refactor `dependees` into two helper rules so that any rule can now `await Get(Dependees, DependeesRequest)`. [ci skip-rust-tests]

view details

Eric Arellano

commit sha 6af8bc8142ec5575947b8f87dccdf629bc4c45b7

Remove `TestBase.create_library() and `TestBase.target()` (#10237) These methods all use v1 code paths so are blocking the deletion of v1 code. Likely, we will want to create some new helper methods. Currently, all v2 code relies on `self.add_to_build_file` and then `self.request_single_product(WrappedTarget, Address)`. This can likely be streamlined. But we will want to create this from scratch, rather than trying to salvage the v1 methods. [ci skip-rust-tests]

view details

gshuflin

commit sha c4f9c6a021a905de297dd7adae904bdfffe6566e

Use debug level for remote store workunits #10238 To reduce verbosity in CI, change some of the levels of workunit items associated with the remote store from INFO to DEBUG.

view details

Benjy Weinberger

commit sha 65423af4de7ef7c2c66d42886a78737b857966a6

Delete the old BinaryTool mechanism. (#10239) It's replaced by ExternalTool. This change switches the last remaining users of BinaryTool over to ExternalTool, and deletes the former. Note that the native tool classes had extra utility methods on them that assumed that tools were downloaded locally. This is not the v2 way of doing things. These utilities were deleted, and the native tool using code will need to be rewritten to accept a digest+relpath, in the v2 idiom. [ci skip-rust-tests]

view details

Benjy Weinberger

commit sha d2e04c48c6d7b5b055d34bf7885bec76294aeb92

Get rid of --plugins2/--backend-packages2. (#10231) * Get rid of --plugins2/--backend-packages2. Instead use --plugins/--backend-packages, as there is now no v1 to get in the way.

view details

Eric Arellano

commit sha fc96d14e465857ff827b16e09412c3a25e99f05f

Prepare 2.0.0.dev1 (#10243) [ci skip-rust-tests]

view details

Eric Arellano

commit sha 4c391a5f57044912b2b372c5587e7fd8db845890

Simplify ci.py now that we have no v1 tests (#10241) We can remove the block list mechanism now.

view details

Benjy Weinberger

commit sha 85003848427566f9611a79009c083600f22d9043

Remove the concept of a scope category. (#10224) In v2 all optionables are subsystems, even global options. There is no meaning to categories any more. [ci skip-rust-tests]

view details

Eric Arellano

commit sha aa5d9f12cb17864ae0e1b2af04741c2acb179539

Delete the rest of v1 `pants.backend.python` (#10240) We would like to be able to remove `setup_py_runner.py`, `executable_pex_tool.py`, and `pex_build_util.py`, but those are used by the test `test_plugin_resolver.py`. [ci skip-rust-tests]

view details

Eric Arellano

commit sha aa1d24f06e0a9ed0b6e3e5bd0a27b8d469462589

Remove `TestBase.context()` (#10248) This will make it easier to delete `Task`. [ci skip-rust-tests]

view details

Eric Arellano

commit sha 40ba9d9ad77d8be6176226eab746c52bf8495034

Remove all remaining v1 Targets (#10246) This will make it easier to delete the core `Target` abstractions. [ci skip-rust-tests]

view details

push time in 23 days

Pull request review commentpantsbuild/pants

upgrade to Rust v1.46.0

 trait GlobMatchingImplementation<E: Display + Send + Sync + 'static>: VFS<E> {       .into_inner()       .into_iter()       .collect::<Vec<_>>();+    #[allow(clippy::unnecessary_sort_by)]

What's the deal with these? Is there a bug for false positives we can link to?

tdyas

comment created time in a month

PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commentpantsbuild/pants

remoting: output directories in action results reference trees

 impl Store {       .to_boxed()   } +  /// Load a REv2 Tree from a remote CAS and cache the embedded Directory protos in the+  /// local store. Tree is used by the REv2 protocol as an optimization for encoding the+  /// the Directory protos that compromose the output directories from a remote+  /// execution reported by an ActionResult.+  ///+  /// Returns an Option<Digest> representing the `root` Directory within the Tree (if it+  /// in fact exists in the remote CAS).+  ///+  /// This method requires that this Store be configured with a remote CAS (and will return+  /// an error if this is not the case).+  pub async fn load_tree_from_remote(&self, tree_digest: Digest) -> Result<Option<Digest>, String> {+    let remote = if let Some(ref remote) = self.remote {+      remote+    } else {+      return Err("Cannot load Trees from a remote without a remote".to_owned());+    };++    let tree_opt = remote+      .load_bytes_with(EntryType::Directory, tree_digest, |b| {

Honestly let's just remove the param - there's no reason for it to exist (IIRC the local::ByteStore and remote::ByteStore used to implement the same trait, but they don't any more), so let's avoid needing to even ask the question :)

tdyas

comment created time in a month

PullRequestReviewEvent

Pull request review commentpantsbuild/pants

remoting: output directories in action results reference trees

 impl Store {       .to_boxed()   } +  /// Load a REv2 Tree from a remote CAS and cache the embedded Directory protos in the+  /// local store. Tree is used by the REv2 protocol as an optimization for encoding the+  /// the Directory protos that compromose the output directories from a remote+  /// execution reported by an ActionResult.+  ///+  /// Returns an Option<Digest> representing the `root` Directory within the Tree (if it+  /// in fact exists in the remote CAS).+  ///+  /// This method requires that this Store be configured with a remote CAS (and will return+  /// an error if this is not the case).+  pub async fn load_tree_from_remote(&self, tree_digest: Digest) -> Result<Option<Digest>, String> {+    let remote = if let Some(ref remote) = self.remote {+      remote+    } else {+      return Err("Cannot load Trees from a remote without a remote".to_owned());+    };++    let tree_opt = remote+      .load_bytes_with(EntryType::Directory, tree_digest, |b| {+        let mut tree = Tree::new();+        tree+          .merge_from_bytes(&b)+          .map_err(|e| format!("protobuf decode error: {:?}", e))?;+        Ok(tree)+      })+      .await?;++    let tree = match tree_opt {+      Some(t) => t,+      None => return Ok(None),+    };++    // Cache the returned `Directory` proto and the children `Directory` protos in+    // the local store.+    let root_digest = self.record_directory(tree.get_root(), true).await?;

Ideally this would use join_all (or some async/await version of the combinator) so these can all be done in parallel, as they don't depend on each other :)

tdyas

comment created time in a month

PullRequestReviewEvent

Pull request review commentpantsbuild/pants

remoting: output directories in action results reference trees

 impl Store {       .to_boxed()   } +  /// Load a REv2 Tree from a remote CAS and cache the embedded Directory protos in the+  /// local store. Tree is used by the REv2 protocol as an optimization for encoding the+  /// the Directory protos that compromose the output directories from a remote+  /// execution reported by an ActionResult.+  ///+  /// Returns an Option<Digest> representing the `root` Directory within the Tree (if it+  /// in fact exists in the remote CAS).+  ///+  /// This method requires that this Store be configured with a remote CAS (and will return+  /// an error if this is not the case).+  pub async fn load_tree_from_remote(&self, tree_digest: Digest) -> Result<Option<Digest>, String> {+    let remote = if let Some(ref remote) = self.remote {+      remote+    } else {+      return Err("Cannot load Trees from a remote without a remote".to_owned());+    };++    let tree_opt = remote+      .load_bytes_with(EntryType::Directory, tree_digest, |b| {

This should probably be EntryType::File which is nominally less constraining, though as you noted before this arg is maybe actually ignored?

tdyas

comment created time in a month

PullRequestReviewEvent

pull request commentpantsbuild/pants

remoting: output directories in action results reference trees

I've updated the PR to implement a load_tree_from_remote. However, cache_tests::cache_success continues to fail in this iteration of the PR (as it did in the prior iteration) because the original Tree is unavailable.

Specifically, the call to populate_fallible_execution_result in process_execution::cache::CommandRunner to decode the locally-cached ActionResult means that Pants will attempt to try and find the Tree referenced as an output directory and decode it.

How do I get that Tree into the CAS used by the cache tests? Or is there a more fundamental issue here where we need to store the Tree locally since this test is about cached results and does not seem to involve remote execution?

We only re-use the same serialisation as we use for remote execution for our cache entries because it was convenient to re-use the code - I'd recommend that instead of storing cache entries as exactly the same as the remote execution proto, we instead store them with the Directory digest instead of the Tree digest.

(You could be slightly sketchy and just store the Directory digest in the field called the Tree digest field, and use slightly different deserialisation code, you could use the reserved tag 2 to store the Directory digest, or you could abandon the protobuf serialisation completely and use some other binary serialisation protocol :))

tdyas

comment created time in a month

delete branch illicitonion/bazel

delete branch : toolpaths

delete time in a month

pull request commentpantsbuild/pants

remoting: output directories in action results reference trees

I believe an OutputDirectory is just a Directory, with two modifications:

It has a prefix
All of the children are included inline, rather than requiring a round-trip per child to fetch.

Nope. OutputDirectory has a prefix and digest of a Tree via the tree_digest field. It is the Tree that has a root Directory and children Directory inline.

Sorry, yeah, there's one level of indirection (it's not directly in-line), but all of what I said I think stands once you fetch the Tree?

tdyas

comment created time in a month

pull request commentpantsbuild/pants

remoting: output directories in action results reference trees

I'm a little surprised this adds a whole new type being stored in the LMDB - as far as I know, a Tree is just a different encoding for a Directory (which has some prefixes set in text rather than in the Directory, so needs wrapping a bit - I'm not really sure why it exists tbh), so I'd expect the remote execution client to convert it into a Directory, rather than store it separately...

The current structure of this PR is mainly due to my lack of familiarity with the Store code. If there are alternate ways to accomplish loading the bytes for a Tree, I am fully amenable to that and which is why I would rather put this PR up earlier rather than later in order to get feedback on direction even if the direction is not quite right yet.

I believe an OutputDirectory is just a Directory, with two modifications:

  1. It has a prefix
  2. All of the children are included inline, rather than requiring a round-trip per child to fetch.

So a Command may say "Expect an output directory at foo/bar", and then the response will have an OutputDirectory which looks like:

path: "foo/bar"
root: Directory {
  files: FileNode {
    name: "blah"
    digest: ...
  }
  directories: DirectoryNode {
    name: "baz"
    digest: Digest {
      hash: "abc123"
      ...
    }
  }
}
children: Directory (happens to have hash "abc123") {
  files: FileNode {
    ...
  }
}

This can be converted with no extra information to a directory which contains a single child directory, foo, which itself contains a single child directory, bar, which contains a file blah and a directory baz. This is just a transport encoding to avoid needing round-trips to fetch each parent one by one - instead we have all of the information.

The algorithm the client should use is roughly:

optional (we could instead just rely on the remote having these if we need them, but we have been given them, so we might as well store them): for each child in children, store it in the store as a Directory
construct a new empty Directory
for each path component in the Tree's path, create an empty subdirectory recursively
For the last path component, set it to the value of root

So for the above example, we'd:

  • Store the directory with digest abc123 as a Directory. Store root as a Directory.
  • Make a new empty Directory
  • Give it a child foo
  • Give foo a child bar whose digest is the digest of root.

This ends up being a Directory, we just happened to be able to populate our local cache with all of its children from the response.

In looking for a way to load bytes from a remote CAS, the remote attribute on Store is non-public and load_bytes_with seemed to be the way to load bytes. It was easiest initially to just expose load_bytes_with and use it. That method has an EntryType argument which I understand as deciding which LMDB to store the retrieved bytes into. Given the existence of EntryType::File and EntryType::Directory, neither seemed the right choice for the Tree proto given I understand them as storing file data and Directory protos, respectively, and not Tree protos.

As above - I think EntryType::Directory is the correct choice here, Tree is just a wire encoding which includes intermediate information, rather than a separate type.

@stuhood's comments point at putting a wrapper function in Store to handle Tree's and not expose load_bytes_with. One solution there is a load_tree method that could load from the remote CAS (since it would have access to Store.remote and just insert the embedded Directory protos into the LMDB store without storing the Tree protos locally.

Again, I don't think this is right - a Tree proto isn't something the remote CAS holds onto, or something we can load from remotely - it's a one-off serialisation of a Directory which the remote does hold onto.

However, even self.remote.load_bytes_with takes an EntryType argument although it appears to be unused.

Yeah, the CAS protocol doesn't have a concept of an EntryType - as far as it's concerned, bytes are bytes, and if they happen to be a valid protobuf that's your business to worry about. We introduced a distinction in the local copy for two reasons:

  1. So we don't need to do validity checks every time we load bytes we expect to be a Directory - we do one validity check when we write it, and after that, we assume we only ever wrote valid things into the Directory Store.
  2. So we can avoid garbage collecting them as aggressively. They're really small, handy metadata, often manipulated in-memory, and require lots of round trips to the server to fully retrieve. So we can keep them for longer than, say, a jar file, which we likely don't need locally if you're doing remote execution, and which is generally only used as the input to an execution or materialise, rather than manipulated in-memory.
tdyas

comment created time in a month

pull request commentpantsbuild/pants

remoting: output directories in action results reference trees

I'm a little surprised this adds a whole new type being stored in the LMDB - as far as I know, a Tree is just a different encoding for a Directory (which has some prefixes set in text rather than in the Directory, so needs wrapping a bit - I'm not really sure why it exists tbh), so I'd expect the remote execution client to convert it into a Directory, rather than store it separately...

tdyas

comment created time in a month

issue commentrust-lang/rfcs

Add impl TryFrom<T> for E where E is a C-like #[repr(T)] enum

This is a very common problem. People keep reinventing crates for this, and most crates settle on an interface that is identical or very close to what is proposed here. ...

  • https://lib.rs/num_enum

FWIW, as one of the maintainers of num_enum, I'd be happy to donate it or any parts of it into std if that's useful (though I can imagine it may be simpler to re-implement from scratch :))

canndrew

comment created time in a month

PullRequestReviewEvent

created tagillicitonion/named

tag0.1.0

Named arguments and default argument values for rust functions.

created time in a month

push eventillicitonion/named

Daniel Wagner-Hall

commit sha 4b916895bb7cbbc5c39298db791353d3bce8ed51

Initial commit

view details

push time in a month

create barnchillicitonion/named

branch : master

created branch time in a month

created repositoryillicitonion/named

Named arguments and default argument values for rust functions.

created time in a month

issue openeddtolnay/syn

Span for FnArg::Receiver in proc_macro seems to point to first token not whole receiver

See this repo for a repro.

I tried to point an error message at a function receiver. Parsing the code as a string, this seems to work fine. Parsing it from a TokenStream, in a proc_macro, the Span for the FnArg::Receiver appears to point to the first token of the receiver, rather than the whole thing:

error: Saw receiver with span #0 bytes(63..64)
 --> example/src/lib.rs:7:19
  |
7 |     fn by_mut_ref(&mut self) -> bool {
  |                   ^

error: Saw receiver with span #0 bytes(126..130)
  --> example/src/lib.rs:12:12
   |
12 |     fn own(self) -> bool {
   |            ^^^^

error: Saw receiver with span #0 bytes(188..191)
  --> example/src/lib.rs:17:16
   |
17 |     fn own_mut(mut self) -> bool {
   |                ^^^

error: Saw receiver with span #0 bytes(253..254)
  --> example/src/lib.rs:22:15
   |
22 |     fn by_ref(&self) -> bool {
   |               ^

If I parse the same code from a str, I get the spans I expect (or at least, :

bytes(21..30)
bytes(86..90)

created time in a month

create barnchillicitonion/repro-syn-receiver-span

branch : master

created branch time in a month

created repositoryillicitonion/repro-syn-receiver-span

created time in a month

pull request commentbazelbuild/bazel

Tool paths are relative to cc_toolchain not suite

@oquenchil @lberki Would it be possible to take a look at this? I'm running into issues because of not having it :) Thanks!

illicitonion

comment created time in a month

Pull request review commentpantsbuild/pants

Spawning against materialized binaries works.

 impl CommandRunner {       named_caches,       cleanup_local_dirs,       platform: Platform::current().unwrap(),+      spawn_lock: Arc::new(RwLock::new(true)),

I think the fear for a regular mutex is that if we end up blocking on a bunch of readers (or blocking readers on a writer), we end up clogging up the runtime thread while we wait? So for synchronisation purposes, a synchronous lock is fine, but it will cause problems for other tasks on the runtime.

jsirois

comment created time in a month

Pull request review commentpantsbuild/pants

Spawning against materialized binaries works.

 pub struct CommandRunner {   named_caches: NamedCaches,   cleanup_local_dirs: bool,   platform: Platform,+  spawn_lock: Arc<RwLock<bool>>,
  spawn_lock: Arc<RwLock<()>>,

To avoid people needing to think about what the bool may represent :)

jsirois

comment created time in a month

pull request commentCodeYourFuture/syllabus

React hooks updates

I think you've covered all of these perfectly in responding to the line-comments :) Sorry, this PR-comment was meant to say "You can ignore most of my line comments if you want, but it would be good to make sure to address the ones pertaining to these topics" - by acting on all of them, you made this comment unnecessary :D

We should probably be more explicit about explaining, and re-explaining, what rendering is, and how it relates to the functions people are writing.

Could you expand on this a bit, please? I'm not exactly clear on what aspects you mean.

I have re-worked the "Re-rendering components" section multiple times. My intention was to try isolate the concept of re-rendering (and to some extent the virtual DOM) from state. But I've found it difficult to do without getting into the implementation details of how to trigger a re-render.

In the state section itself, I really tried to emphasise the aspect of triggering a re-render via state changes. If it's not getting through, I'd appreciate a second perspective.

I think you've got this pretty much perfect now :)

I think the one additional suggestion I'd have is to take the week 1 sentence "We render the HelloWorld component into the element with the id of root." and add a phrase like " - The JSX elements that are returned from the HelloWorld function are turned into the DOM for you by React, and put inside the root element", like you emphasised in the re-rendering components section in week 2.

Students struggle with the React documentation because of how class-based it is, we should consider giving guidance text when we link to the React docs to help them read them.

I addressed one area that you highlighted (by removing it entirely). If there's other specific areas in the syllabus I can take a look at those too.

That was what I was mostly aiming for :)

Unfortunately I don't know there's much we can do about students finding materials on their own time. Do you think we should explicitly discuss (in the syllabus) that class components exist? I believe we did talk about it during the class, but I don't think it's written down anywhere.

I think it's probably worth mentioning in the syllabus. If we're viewing the syllabus as a way for different cities to pick up the same material and deliver it, anything not explicitly mentioned is liable to be left out for some classes :) Maybe it could be a "further reading" section at the end of week 1, so as to not disrupt the flow of the lesson? Gives us something easy to point to on Slack if students are confused, too.

40thieves

comment created time in a month

Pull request review commentCodeYourFuture/syllabus

React hooks updates

+---+id: container-components+title: React Further Reading - Container Components+---++In real world applications, the things we want to remember in state follow the [_business logic_](https://en.wikipedia.org/wiki/Business_logic) required by our users. So for example the number of caught Pokemon in the exercise increases when you click on the button _Catch Pokemon_. Most of the time, business logic is about figuring out when and how to change state.

Definitely agree this is a discussion for another venue - wrote up https://codeyourfuture.slack.com/archives/C012UUW69S8/p1596893069021300 :)

40thieves

comment created time in a month

more