profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/werkt/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.
George Gensure werkt Facebook Pittsburgh

bazelbuild/bazel-buildfarm 372

Bazel remote caching and execution service

bazelbuild/remote-apis 127

An API for caching and execution of actions on a remote system.

werkt/bazel-stress 2

Stress test used for remote execution testing

werkt/bazel-deps 1

Generate bazel dependencies for maven artifacts

werkt/bazel 0

Correct, reproducible, and fast builds for everyone.

werkt/bazel-buildfarm 0

[WIP] Bazel remote caching and execution service

werkt/bazel-toolchains 0

Repository that hosts Bazel toolchain configs for remote execution and related support tools.

PR opened bazelbuild/bazel-buildfarm

add "worker capabilities" for future support of storage workers

We are implementing "storage workers"- which are workers that contribute to the sharded CAS, but do not participate in execution.
To achieve this, we add a worker configuration for opting out of execution. We've discussed simplifying these changes into a configuration called "capabilities". Capabilities will be used to help us distinguish the type of workers we have.
Here is how we would determine our current worker types:

CAS Execution Type
true true Cpu
false true Gpu
true false Storage
+24 -8

0 comment

3 changed files

pr created time in a day

create barnchbazelbuild/bazel-buildfarm

branch : luxe/add_capabilities

created branch time in a day

issue commentbazelbuild/bazel-buildfarm

Unable to start server on macOS using tutorial

Edited .bazelversion to use 4.1.0 then verified version.

cobbler-laptop01:bazel-buildfarm loyaltyarm$ bazel --version
bazel 4.1.0

Ran bazel clean with --expunge.

cobbler-laptop01:bazel-buildfarm loyaltyarm$ bazel clean --expunge
INFO: Starting clean.

Re-ran the run command from quick start:

cobbler-laptop01:bazel-buildfarm loyaltyarm$ bazel run src/main/java/build/buildfarm:buildfarm-server $PWD/examples/server.config.example
Starting local Bazel server and connecting to it...
ERROR: /private/var/tmp/_bazel_loyaltyarm/073806cca4ca57a45150c2d667271a4b/external/bazel_tools/tools/jdk/BUILD:490:6: Configurable attribute "actual" doesn't match this configuration: Could not find a JDK for host execution environment, please explicitly provide one using `--host_javabase.`
INFO: Repository googleapis instantiated at:
  /Users/loyaltyarm/src/bazel-buildfarm/WORKSPACE:5:23: in <toplevel>
  /Users/loyaltyarm/src/bazel-buildfarm/deps.bzl:99:14: in buildfarm_dependencies
  /private/var/tmp/_bazel_loyaltyarm/073806cca4ca57a45150c2d667271a4b/external/bazel_tools/tools/build_defs/repo/utils.bzl:201:18: in maybe
Repository rule http_archive defined at:
  /private/var/tmp/_bazel_loyaltyarm/073806cca4ca57a45150c2d667271a4b/external/bazel_tools/tools/build_defs/repo/http.bzl:336:31: in <toplevel>
INFO: Repository 'googleapis' used the following cache hits instead of downloading the corresponding file.
 * Hash '745cb3c2e538e33a07e2e467a15228ccbecadc1337239f6740d57a74d9cdef81' for https://github.com/googleapis/googleapis/archive/6598bb829c9e9a534be674649ffd1b4671a821f9.zip
If the definition of 'googleapis' was updated, verify that the hashes were also updated.
INFO: Repository remote_java_tools_darwin instantiated at:
  /DEFAULT.WORKSPACE.SUFFIX:245:6: in <toplevel>
  /private/var/tmp/_bazel_loyaltyarm/073806cca4ca57a45150c2d667271a4b/external/bazel_tools/tools/build_defs/repo/utils.bzl:201:18: in maybe
Repository rule http_archive defined at:
  /private/var/tmp/_bazel_loyaltyarm/073806cca4ca57a45150c2d667271a4b/external/bazel_tools/tools/build_defs/repo/http.bzl:336:31: in <toplevel>
ERROR: Analysis of target '//src/main/java/build/buildfarm:buildfarm-server' failed; build aborted: /private/var/tmp/_bazel_loyaltyarm/073806cca4ca57a45150c2d667271a4b/external/bazel_tools/tools/jdk/BUILD:490:6: Configurable attribute "actual" doesn't match this configuration: Could not find a JDK for host execution environment, please explicitly provide one using `--host_javabase.`
INFO: Elapsed time: 14.933s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (36 packages loaded, 558 targets configured)
FAILED: Build did NOT complete successfully (36 packages loaded, 558 targets configured)
    currently loading: src/main/protobuf
    Fetching @local_config_xcode; fetching
    Fetching ...806cca4ca57a45150c2d667271a4b/external/googleapis; Extracting /private/var/tmp/_bazel_loyaltyarm/073806cca4ca57a45150c2d667271a4b/external/googleapis/temp2138561162632920666/6598bb829c9e9a534be674649ffd1b4671a821f9.zip
    Fetching @rules_proto; fetching
    Fetching ...4ca57a45150c2d667271a4b/external/rules_proto; Extracting /private/var/tmp/_bazel_loyaltyarm/073806cca4ca57a45150c2d667271a4b/external/rules_proto/temp15113028073105299790/40298556293ae502c66579620a7ce867d5f57311.tar.gz
loyaltyarm

comment created time in 2 days

pull request commentbazelbuild/bazel-buildfarm

Update external repository instructions

This might be better handled in defs.bzl . WDYT ? Happy to modify and test if that's a better way to do it.

gabware

comment created time in 2 days

PR opened bazelbuild/bazel-buildfarm

Update external repository instructions

Add a call to compat_repositories() when configuring bazel-buildfarm as an external repository. Without this bazel fails building the buildfarm-server with the following error:

ERROR: [...]/bzl-ws/external/io_grpc_grpc_java/compiler/BUILD.bazel:18:13: no such package '@com_google_guava_guava//jar': The repository '@com_google_guava_guava' could not be resolved and referenced by '@io_grpc_grpc_java//compiler:java_grpc_library_deps__do_not_reference'

TESTED=build and ran buildfarm-server and buildfarm-operationqueue-worker on windows host.

+4 -0

0 comment

1 changed file

pr created time in 2 days

delete branch bazelbuild/bazel-buildfarm

delete branch : luxe/metrics_for_dispatched

delete time in 4 days

push eventbazelbuild/bazel-buildfarm

Trevor Hickey

commit sha 04583ec6ccd26ca97532939873a4b03aa8252c32

[configuration] fix misuse of getProcessingTimeoutMillis (#713)

view details

push time in 4 days

PR merged bazelbuild/bazel-buildfarm

Reviewers
[configuration] fix misuse of getProcessingTimeoutMillis cla: yes

scanProcessing and scanDispatching are both using the same function getExpiresAt.

This results in a problem where the dispatching code is using the wrong configuration. processingOperations should use config.getProcessingTimeoutMillis. dispatchedOperations should use config.getDispatchingTimeoutMillis.

Also, the configuration value is being multiplied by 2. This is not necessary. If needed, it can be configured to be twice as much.

We resolve these above issues and represent processingOperatiosn and dispatchedOperations as containers.

+59 -34

0 comment

2 changed files

luxe

pr closed time in 4 days

push eventbazelbuild/bazel-buildfarm

George Gensure

commit sha d3a3fbab202bc2f2cf79a6f1d374e7b42b617e4c

Report more current worker set for BackplaneStatus (#808) The workers a scheduler reports should be relatively current, and consistent with those used during enumerations. Avoid requiring those enumeration requests by going through the cache invalidation mechanism.

view details

Trevor Hickey

commit sha 9136326425eb9ad0730e04863d07b54ea675a4ee

Merge branch 'master' into luxe/remove_dequeue_platform_constants

view details

push time in 5 days

push eventbazelbuild/bazel-buildfarm

Trevor Hickey

commit sha 2cf81c71d8034078a403332c7f110088b88a0631

enforce that java package names match file layout (#803)

view details

Trevor Hickey

commit sha 3a625fe059ead651d2e32749b21fed43bf947178

Gate code coverage at 40% (#804)

view details

Trevor Hickey

commit sha ac76c426a2daf5c20dd252566999cccaff11cc09

group execution wrappers for easier runtime packaging (#798)

view details

George Gensure

commit sha aaf40fdf04c9851ff3b15ac0382eaf4dd315ba70

Upgrade to latest netty and grpc-java (#806)

view details

George Gensure

commit sha d3a3fbab202bc2f2cf79a6f1d374e7b42b617e4c

Report more current worker set for BackplaneStatus (#808) The workers a scheduler reports should be relatively current, and consistent with those used during enumerations. Avoid requiring those enumeration requests by going through the cache invalidation mechanism.

view details

Trevor Hickey

commit sha 8b4cb8c4eaeafd528137b80b454534dbd510a400

Merge branch 'master' into luxe/skip_denylist

view details

push time in 5 days

push eventbazelbuild/bazel-buildfarm

George Gensure

commit sha aaf40fdf04c9851ff3b15ac0382eaf4dd315ba70

Upgrade to latest netty and grpc-java (#806)

view details

George Gensure

commit sha d3a3fbab202bc2f2cf79a6f1d374e7b42b617e4c

Report more current worker set for BackplaneStatus (#808) The workers a scheduler reports should be relatively current, and consistent with those used during enumerations. Avoid requiring those enumeration requests by going through the cache invalidation mechanism.

view details

Trevor Hickey

commit sha 6b67d6f476fa5d66c61e9fc4587aa85e3a44bb13

Merge branch 'master' into luxe/refactor_timeout

view details

push time in 5 days

push eventbazelbuild/bazel-buildfarm

Trevor Hickey

commit sha 1c8e248ef09cd14d09c702db2b354819a1e86c75

[static analysis burndown] remove identical catch branch (#721)

view details

Trevor Hickey

commit sha daf25e7c242fff5a19f43131d117019c5ab7b566

[metrics] Add metrics that explain currently dispatched work (#769)

view details

Trevor Hickey

commit sha c8f35408850eb47d9c1eec09aeb1cf624e2b11d1

Simplify platform property parsing. Improve visibility on resource limits. (#788)

view details

Trevor Hickey

commit sha 2cf81c71d8034078a403332c7f110088b88a0631

enforce that java package names match file layout (#803)

view details

Trevor Hickey

commit sha 3a625fe059ead651d2e32749b21fed43bf947178

Gate code coverage at 40% (#804)

view details

Trevor Hickey

commit sha ac76c426a2daf5c20dd252566999cccaff11cc09

group execution wrappers for easier runtime packaging (#798)

view details

George Gensure

commit sha aaf40fdf04c9851ff3b15ac0382eaf4dd315ba70

Upgrade to latest netty and grpc-java (#806)

view details

George Gensure

commit sha d3a3fbab202bc2f2cf79a6f1d374e7b42b617e4c

Report more current worker set for BackplaneStatus (#808) The workers a scheduler reports should be relatively current, and consistent with those used during enumerations. Avoid requiring those enumeration requests by going through the cache invalidation mechanism.

view details

Trevor Hickey

commit sha 0d5a3a39b671e671c912f37434726540195d13d5

Merge branch 'master' into luxe/fix_modifiers_static_analysis

view details

push time in 5 days

push eventbazelbuild/bazel-buildfarm

Trevor Hickey

commit sha 8bd1cd1b2e8a120deefb764affe57677a059c16f

add java_image to utilize tini + docker patch for entry_point (#725)

view details

Trevor Hickey

commit sha 20ec60a09e7d05eb72d8964cca66f88737d6ac7c

add linux-sandbox exec_property (#758)

view details

Thorbjorn Larsson

commit sha ce08bf115fe8a0c7b074f7d44599af95a6c42b9c

Fix access denied problem on windows during destroyExecDir (#753) * Fix access denied problem on windows during destroyExecDir Close the open filedescriptor after upload finished due to zero size, or it's already present in cache. This is a must on windows, otherwise destroyExecdir will fail(access denied). * Minor review updates Co-authored-by: Trevor Hickey <TrevorJamesHickey@gmail.com> Co-authored-by: George Gensure <werkt@users.noreply.github.com>

view details

Trevor Hickey

commit sha 919e4c1a11c9207d592bb853d38a7f180f0393ce

add unit tests for "canQueue" API (#765)

view details

Trevor Hickey

commit sha 96b73a9fba6479d45d7985085df08a8e84c17a52

fix documentation in resource decider (#739)

view details

Trevor Hickey

commit sha 28bf0f7ce7b72bbdd47151a0b09a3d8f8fde2c93

remove incorrect comment (#763)

view details

Trevor Hickey

commit sha 77f238e1d29b25a248d1cac068b06796228bbedf

[feature] implement 'block-network' for remote execution (#720)

view details

George Gensure

commit sha 34b47cdf7689ebfb6a423168cfd177990fbd462d

Use EntryPathStrategy for DirectoriesIndex (#771) Refactored into File abstract for duplicated functionality across index implementations.

view details

George Gensure

commit sha d28ec34a9cbcee2668f904234a76db35b59844c8

Upgrade grpc, guava, and bazel (#772) Grpc to 1.37.0. guava to 30.1.1, and bazel 4.0.0 No code changes required to pass tests, but guava had to be specifically upgraded to avoid a switchover to android from jre, which breaks a number of functional conversions to java native, and removes the MoreFiles utility methods for nio Path.

view details

Yuriy Belenitsky

commit sha fd2855a9e8173a959b0a64509240397c59920517

80degreeswest/additional prometheus metrics (#773) * Remove duplicate prometheus server controls * Add a number of remote invocations metric * Add key expiration prometheus metric * Add operation poller prometheus metric * Add operation metadata prometheus metrics

view details

Trevor Hickey

commit sha 283b079bcbf98d37b930a137e0e0bbe0994d7313

Allow core specification using "cores" instead of "min-cores" / "max-cores" (#737)

view details

Trevor Hickey

commit sha db0f9bd250d90fc8d6bfb80b9808cbcb0a61289f

add linux sandbox options for CLI usage in executor (#760)

view details

Trevor Hickey

commit sha 27578d76c7152bc4f4f314a35d690ec87742f9ff

allow using linux sandbox in the executor (#770)

view details

Trevor Hickey

commit sha 9ccb0137c74608e9f491babc6fdea4f969a57f77

remove slow metric calculations (#774)

view details

Trevor Hickey

commit sha e324364c64595174a0f79d682aa7fb50f21958cb

updates (#775)

view details

Trevor Hickey

commit sha 45b557ad656c4c090d5843d3e6acd3a6e53b4815

improve test debugging on the client (#748)

view details

Trevor Hickey

commit sha 994427c9c606c09cb3164aa6f45f065b32bff79f

fix bug when 'global limit execution' is used but cores are not limited (#776)

view details

Trevor Hickey

commit sha 1947831f07e25ab50b59bca9a190c835ac9d5a5b

prevent 0 cores from being chosen during 'global limit execution' (#779)

view details

George Gensure

commit sha bf15717f58421bdee781ae06a478698be6e1e6ef

Make directories writable before rename (#778) MacOS appears to deny renames for directories which are not themselves writable. Ensure that the toplevel directory being renamed is reset to writable ahead of a remove.

view details

George Gensure

commit sha 6b4752aa34373fe7edc93243f1bb5309126f51c8

Guard against exceptions in newOutput (#780) Ensure that the 'open' state is transitioned to cleanly. This prevents a hang when waiting for a Write that failed to open, but set its closedFuture member.

view details

push time in 5 days

push eventbazelbuild/bazel-buildfarm

Trevor Hickey

commit sha 63dcbf82b168d880e081bef040415e274ca421cc

enable AvoidReassigningCatchVariables (#711)

view details

Trevor Hickey

commit sha e108499f1c8213ba09a5ca398238eff8c2710d47

fix and gate on string-related performance (#714)

view details

Trevor Hickey

commit sha 6696356dbe67b111d1af82ff004976e0a0190f90

Replace guava fork with caffeine (#719) Caffeine has improved caching features over guava. This is to reduce the custom code in buildfarm, and make collecting the cache metrics easier.

view details

Gregg Donovan

commit sha f06030f57da416dc6078eefb2988002eb3abda18

Update jnr-fuse to 0.5.5. (0.5.1 was temporarily down on bintray). (#723) Co-authored-by: Trevor Hickey <TrevorJamesHickey@gmail.com>

view details

George Gensure

commit sha fadf51e624abab7a8a5cc2182a6fceb5039bb73b

Support exec_owner in operationqueue worker (#727)

view details

Johan Walles

commit sha 76fcaee869dc5b7af5f61994141d4a9e9cef28cf

Local dev: Document starting a worker (#728)

view details

George Gensure

commit sha 0462d2acec6b2e434f172eb32419470486be778f

Ensure uncancelled getActionResult receives error (#732) Fixes #731 CANCEL status exceptions can be observed from outside of the server call. Ensure that they are propagated if the server call is not cancelled. A race with cancellation should be mitigated by ignoring any thrown exceptions from onError.

view details

Trevor Hickey

commit sha 9b806bb2ae348c2651503240c37a588af27d5aea

[feature] add cgroup memory restrictions via min/max-mem flags (#694) * limit mem with cgroups

view details

Liam Miller-Cushon

commit sha a22aaced969aef1af50ba0dc9dc36c8c1cce21c9

Suppress errors for unused return value of proto `.build()` (#744) Fixes ``` src/test/java/build/buildfarm/examples/ExampleConfigsTest.java:51: error: [ProtoBuilderReturnValueIgnored] Unnecessary call to proto's #build() method. If you don't consume the return value of #build(), the result is discarded and the only effect is to verify that all required fields are set, which can be expressed more directly with #isInitialized(). builder.build(); ^ (see https://errorprone.info/bugpattern/ProtoBuilderReturnValueIgnored) Did you mean to remove this line or 'checkState(builder.isInitialized());'? src/test/java/build/buildfarm/examples/ExampleConfigsTest.java:51: error: [ReturnValueIgnored] Return value of this method must be used builder.build(); ^ (see https://errorprone.info/bugpattern/ReturnValueIgnored) Did you mean to remove this line? ```

view details

Ivo List

commit sha 4d6d101f259f554ec2f91a07b68ca17877cfd005

Suppress errors for unused return value of proto `.build()` (#747)

view details

tobbe76

commit sha 0ba9697fda74ae09157cbba06975732c8b808bfe

Fix problem with relative paths when running worker on windows (#749) ProcessBuilder on windows requires absolute paths for the executable or relative CWD. ProcessBuilder.directory() only affects cwd of the proccess to be created. Co-authored-by: Thorbjorn Larsson <thorbjornL76@gmail.com>

view details

George Gensure

commit sha 7f991bd50bcd5dd55270600a54489639103a8fbe

Revert "[feature] add cgroup memory restrictions via min/max-mem flags (#694)" (#752) This reverts commit 9b806bb2ae348c2651503240c37a588af27d5aea. Revert requires buildifier formatting

view details

George Gensure

commit sha c99f7befe7e644ee0a1b5316c2f8e6f34582b4cb

Correct arg pointer for execvp

view details

Trevor Hickey

commit sha 1363bf6d3bc67aaab747510db76a8ae5b7ac2af3

[ci] add integration test for memory instances (#755)

view details

George Gensure

commit sha aa839622df8824b7244900b864d91d2bc348e0ac

Specify ExOpMetadata for retry exceeded (#759) The ExecuteOperationMetadata for a failed job due to retry should be populated. This provides a clean update and publication for watchers, as well as preventing a stack trace log output for a broken metadata type.

view details

Trevor Hickey

commit sha 040300ace062a1dc8b0f145cf4bef5e2f4128deb

[feature] add cgroup memory restrictions via min/max-mem flags (#754)

view details

Trevor Hickey

commit sha 6235fb53b3b17b2427407929f14f46c199bd2620

[bug] [integration test] fix unit test failures within dockerized context (#660)

view details

Trevor Hickey

commit sha 05c557c3f74cee43fa11b79f04c7016997ef1e8f

[backplane] Make "prequeue max depth" a part of the prequeue (#699)

view details

Trevor Hickey

commit sha 2807cd9ba9e93789030f18428fcce987f2d78721

[backplane] Make "queue max depth" a part of the queue (#702)

view details

Trevor Hickey

commit sha c8a86f0b2418aff10a48ffffce9fa927f6350008

[feature] show the working directory while debugging actions (#703)

view details

push time in 5 days

push eventbazelbuild/bazel-buildfarm

George Gensure

commit sha aaf40fdf04c9851ff3b15ac0382eaf4dd315ba70

Upgrade to latest netty and grpc-java (#806)

view details

George Gensure

commit sha d3a3fbab202bc2f2cf79a6f1d374e7b42b617e4c

Report more current worker set for BackplaneStatus (#808) The workers a scheduler reports should be relatively current, and consistent with those used during enumerations. Avoid requiring those enumeration requests by going through the cache invalidation mechanism.

view details

Trevor Hickey

commit sha c0cc23c3fdbc23f050a66ffd3c2507ef81972bf8

Merge branch 'master' into luxe/removeDeadCode

view details

push time in 5 days

issue openedbazelbuild/bazel-buildfarm

Unable to start server on macOS using tutorial

Was last working 16 days ago as of #793

Now running commands from bazel from the tutorial results in:

bazel run src/main/java/build/buildfarm:buildfarm-server $PWD/examples/server.config.example
Starting local Bazel server and connecting to it...
ERROR: /private/var/tmp/_bazel_loyaltyarm/073806cca4ca57a45150c2d667271a4b/external/bazel_tools/tools/jdk/BUILD:490:6: Configurable attribute "actual" doesn't match this configuration: Could not find a JDK for host execution environment, please explicitly provide one using `--host_javabase.`
INFO: Repository googleapis instantiated at:
  /Users/loyaltyarm/src/bazel-buildfarm/WORKSPACE:5:23: in <toplevel>
  /Users/loyaltyarm/src/bazel-buildfarm/deps.bzl:99:14: in buildfarm_dependencies
  /private/var/tmp/_bazel_loyaltyarm/073806cca4ca57a45150c2d667271a4b/external/bazel_tools/tools/build_defs/repo/utils.bzl:201:18: in maybe
Repository rule http_archive defined at:
  /private/var/tmp/_bazel_loyaltyarm/073806cca4ca57a45150c2d667271a4b/external/bazel_tools/tools/build_defs/repo/http.bzl:336:31: in <toplevel>
INFO: Repository 'googleapis' used the following cache hits instead of downloading the corresponding file.
 * Hash '745cb3c2e538e33a07e2e467a15228ccbecadc1337239f6740d57a74d9cdef81' for https://github.com/googleapis/googleapis/archive/6598bb829c9e9a534be674649ffd1b4671a821f9.zip
If the definition of 'googleapis' was updated, verify that the hashes were also updated.
INFO: Repository remote_java_tools_darwin instantiated at:
  /DEFAULT.WORKSPACE.SUFFIX:245:6: in <toplevel>
  /private/var/tmp/_bazel_loyaltyarm/073806cca4ca57a45150c2d667271a4b/external/bazel_tools/tools/build_defs/repo/utils.bzl:201:18: in maybe
Repository rule http_archive defined at:
  /private/var/tmp/_bazel_loyaltyarm/073806cca4ca57a45150c2d667271a4b/external/bazel_tools/tools/build_defs/repo/http.bzl:336:31: in <toplevel>
ERROR: Analysis of target '//src/main/java/build/buildfarm:buildfarm-server' failed; build aborted: /private/var/tmp/_bazel_loyaltyarm/073806cca4ca57a45150c2d667271a4b/external/bazel_tools/tools/jdk/BUILD:490:6: Configurable attribute "actual" doesn't match this configuration: Could not find a JDK for host execution environment, please explicitly provide one using `--host_javabase.`
INFO: Elapsed time: 15.316s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (32 packages loaded, 547 targets configured)
FAILED: Build did NOT complete successfully (32 packages loaded, 547 targets configured)
    Fetching @local_config_xcode; fetching
    Fetching ...xternal/googleapis; Extracting /private/var/tmp/_bazel_loyaltyarm/073806cca4ca57a45150c2d667271a4b/external/googleapis/temp13309020438680985123/6598bb829c9e9a534be674649ffd1b4671a821f9.zip

And requires setting the --host_javabase

Running on darwin_arm64, have bumped the bazel version locally to 4.1.0 as 4.0.0 does not seem to include darwin binaries for arm any longer.

created time in 6 days

PR opened bazelbuild/bazel-buildfarm

remove WORKER_PLATFORM_CORES_PROPERTY

This constant can be removed and replaced now.

+2 -10

0 comment

1 changed file

pr created time in 6 days

Pull request review commentbazelbuild/bazel-buildfarm

[static analysis burndown] remove existing dead code

    public static final String INVALID_PLATFORM = "The `Platform` of the `Command` was invalid."; -  private static final String INVALID_FILE_NAME =

I've preserved the INVALID_FILE_NAME by enabling one of the places where it was commented out. This will prevent the code from being removed now.

luxe

comment created time in 6 days

push eventbazelbuild/bazel-buildfarm

Trevor Hickey

commit sha 0bf037b9598827dd9dd6d6df85836d6cac4ed5a6

updates

view details

push time in 6 days

push eventbazelbuild/bazel-buildfarm

Trevor Hickey

commit sha 135cc4b824fe1330cb509cff2140960fece641ce

updates

view details

push time in 6 days

push eventbazelbuild/bazel-buildfarm

Liam Miller-Cushon

commit sha a22aaced969aef1af50ba0dc9dc36c8c1cce21c9

Suppress errors for unused return value of proto `.build()` (#744) Fixes ``` src/test/java/build/buildfarm/examples/ExampleConfigsTest.java:51: error: [ProtoBuilderReturnValueIgnored] Unnecessary call to proto's #build() method. If you don't consume the return value of #build(), the result is discarded and the only effect is to verify that all required fields are set, which can be expressed more directly with #isInitialized(). builder.build(); ^ (see https://errorprone.info/bugpattern/ProtoBuilderReturnValueIgnored) Did you mean to remove this line or 'checkState(builder.isInitialized());'? src/test/java/build/buildfarm/examples/ExampleConfigsTest.java:51: error: [ReturnValueIgnored] Return value of this method must be used builder.build(); ^ (see https://errorprone.info/bugpattern/ReturnValueIgnored) Did you mean to remove this line? ```

view details

Ivo List

commit sha 4d6d101f259f554ec2f91a07b68ca17877cfd005

Suppress errors for unused return value of proto `.build()` (#747)

view details

tobbe76

commit sha 0ba9697fda74ae09157cbba06975732c8b808bfe

Fix problem with relative paths when running worker on windows (#749) ProcessBuilder on windows requires absolute paths for the executable or relative CWD. ProcessBuilder.directory() only affects cwd of the proccess to be created. Co-authored-by: Thorbjorn Larsson <thorbjornL76@gmail.com>

view details

George Gensure

commit sha 7f991bd50bcd5dd55270600a54489639103a8fbe

Revert "[feature] add cgroup memory restrictions via min/max-mem flags (#694)" (#752) This reverts commit 9b806bb2ae348c2651503240c37a588af27d5aea. Revert requires buildifier formatting

view details

George Gensure

commit sha c99f7befe7e644ee0a1b5316c2f8e6f34582b4cb

Correct arg pointer for execvp

view details

Trevor Hickey

commit sha 1363bf6d3bc67aaab747510db76a8ae5b7ac2af3

[ci] add integration test for memory instances (#755)

view details

George Gensure

commit sha aa839622df8824b7244900b864d91d2bc348e0ac

Specify ExOpMetadata for retry exceeded (#759) The ExecuteOperationMetadata for a failed job due to retry should be populated. This provides a clean update and publication for watchers, as well as preventing a stack trace log output for a broken metadata type.

view details

Trevor Hickey

commit sha 040300ace062a1dc8b0f145cf4bef5e2f4128deb

[feature] add cgroup memory restrictions via min/max-mem flags (#754)

view details

Trevor Hickey

commit sha 6235fb53b3b17b2427407929f14f46c199bd2620

[bug] [integration test] fix unit test failures within dockerized context (#660)

view details

Trevor Hickey

commit sha 05c557c3f74cee43fa11b79f04c7016997ef1e8f

[backplane] Make "prequeue max depth" a part of the prequeue (#699)

view details

Trevor Hickey

commit sha 2807cd9ba9e93789030f18428fcce987f2d78721

[backplane] Make "queue max depth" a part of the queue (#702)

view details

Trevor Hickey

commit sha c8a86f0b2418aff10a48ffffce9fa927f6350008

[feature] show the working directory while debugging actions (#703)

view details

Trevor Hickey

commit sha 8bd1cd1b2e8a120deefb764affe57677a059c16f

add java_image to utilize tini + docker patch for entry_point (#725)

view details

Trevor Hickey

commit sha 20ec60a09e7d05eb72d8964cca66f88737d6ac7c

add linux-sandbox exec_property (#758)

view details

Thorbjorn Larsson

commit sha ce08bf115fe8a0c7b074f7d44599af95a6c42b9c

Fix access denied problem on windows during destroyExecDir (#753) * Fix access denied problem on windows during destroyExecDir Close the open filedescriptor after upload finished due to zero size, or it's already present in cache. This is a must on windows, otherwise destroyExecdir will fail(access denied). * Minor review updates Co-authored-by: Trevor Hickey <TrevorJamesHickey@gmail.com> Co-authored-by: George Gensure <werkt@users.noreply.github.com>

view details

Trevor Hickey

commit sha 919e4c1a11c9207d592bb853d38a7f180f0393ce

add unit tests for "canQueue" API (#765)

view details

Trevor Hickey

commit sha 96b73a9fba6479d45d7985085df08a8e84c17a52

fix documentation in resource decider (#739)

view details

Trevor Hickey

commit sha 28bf0f7ce7b72bbdd47151a0b09a3d8f8fde2c93

remove incorrect comment (#763)

view details

Trevor Hickey

commit sha 77f238e1d29b25a248d1cac068b06796228bbedf

[feature] implement 'block-network' for remote execution (#720)

view details

George Gensure

commit sha 34b47cdf7689ebfb6a423168cfd177990fbd462d

Use EntryPathStrategy for DirectoriesIndex (#771) Refactored into File abstract for duplicated functionality across index implementations.

view details

push time in 6 days

push eventbazelbuild/bazel-buildfarm

Trevor Hickey

commit sha 040300ace062a1dc8b0f145cf4bef5e2f4128deb

[feature] add cgroup memory restrictions via min/max-mem flags (#754)

view details

Trevor Hickey

commit sha 6235fb53b3b17b2427407929f14f46c199bd2620

[bug] [integration test] fix unit test failures within dockerized context (#660)

view details

Trevor Hickey

commit sha 05c557c3f74cee43fa11b79f04c7016997ef1e8f

[backplane] Make "prequeue max depth" a part of the prequeue (#699)

view details

Trevor Hickey

commit sha 2807cd9ba9e93789030f18428fcce987f2d78721

[backplane] Make "queue max depth" a part of the queue (#702)

view details

Trevor Hickey

commit sha c8a86f0b2418aff10a48ffffce9fa927f6350008

[feature] show the working directory while debugging actions (#703)

view details

Trevor Hickey

commit sha 8bd1cd1b2e8a120deefb764affe57677a059c16f

add java_image to utilize tini + docker patch for entry_point (#725)

view details

Trevor Hickey

commit sha 20ec60a09e7d05eb72d8964cca66f88737d6ac7c

add linux-sandbox exec_property (#758)

view details

Thorbjorn Larsson

commit sha ce08bf115fe8a0c7b074f7d44599af95a6c42b9c

Fix access denied problem on windows during destroyExecDir (#753) * Fix access denied problem on windows during destroyExecDir Close the open filedescriptor after upload finished due to zero size, or it's already present in cache. This is a must on windows, otherwise destroyExecdir will fail(access denied). * Minor review updates Co-authored-by: Trevor Hickey <TrevorJamesHickey@gmail.com> Co-authored-by: George Gensure <werkt@users.noreply.github.com>

view details

Trevor Hickey

commit sha 919e4c1a11c9207d592bb853d38a7f180f0393ce

add unit tests for "canQueue" API (#765)

view details

Trevor Hickey

commit sha 96b73a9fba6479d45d7985085df08a8e84c17a52

fix documentation in resource decider (#739)

view details

Trevor Hickey

commit sha 28bf0f7ce7b72bbdd47151a0b09a3d8f8fde2c93

remove incorrect comment (#763)

view details

Trevor Hickey

commit sha 77f238e1d29b25a248d1cac068b06796228bbedf

[feature] implement 'block-network' for remote execution (#720)

view details

George Gensure

commit sha 34b47cdf7689ebfb6a423168cfd177990fbd462d

Use EntryPathStrategy for DirectoriesIndex (#771) Refactored into File abstract for duplicated functionality across index implementations.

view details

George Gensure

commit sha d28ec34a9cbcee2668f904234a76db35b59844c8

Upgrade grpc, guava, and bazel (#772) Grpc to 1.37.0. guava to 30.1.1, and bazel 4.0.0 No code changes required to pass tests, but guava had to be specifically upgraded to avoid a switchover to android from jre, which breaks a number of functional conversions to java native, and removes the MoreFiles utility methods for nio Path.

view details

Yuriy Belenitsky

commit sha fd2855a9e8173a959b0a64509240397c59920517

80degreeswest/additional prometheus metrics (#773) * Remove duplicate prometheus server controls * Add a number of remote invocations metric * Add key expiration prometheus metric * Add operation poller prometheus metric * Add operation metadata prometheus metrics

view details

Trevor Hickey

commit sha 283b079bcbf98d37b930a137e0e0bbe0994d7313

Allow core specification using "cores" instead of "min-cores" / "max-cores" (#737)

view details

Trevor Hickey

commit sha db0f9bd250d90fc8d6bfb80b9808cbcb0a61289f

add linux sandbox options for CLI usage in executor (#760)

view details

Trevor Hickey

commit sha 27578d76c7152bc4f4f314a35d690ec87742f9ff

allow using linux sandbox in the executor (#770)

view details

Trevor Hickey

commit sha 9ccb0137c74608e9f491babc6fdea4f969a57f77

remove slow metric calculations (#774)

view details

Trevor Hickey

commit sha e324364c64595174a0f79d682aa7fb50f21958cb

updates (#775)

view details

push time in 6 days

push eventbazelbuild/bazel-buildfarm

Trevor Hickey

commit sha 07b9afc8129ff7748aadc2334613e2d40a55dc5d

updates

view details

push time in 6 days

push eventbazelbuild/bazel-buildfarm

Trevor Hickey

commit sha 2cf81c71d8034078a403332c7f110088b88a0631

enforce that java package names match file layout (#803)

view details

Trevor Hickey

commit sha 3a625fe059ead651d2e32749b21fed43bf947178

Gate code coverage at 40% (#804)

view details

Trevor Hickey

commit sha ac76c426a2daf5c20dd252566999cccaff11cc09

group execution wrappers for easier runtime packaging (#798)

view details

Trevor Hickey

commit sha 17b10e8816f282791e154694ab88ce9cdba78877

Merge branch 'master' into luxe/rusage

view details

push time in 6 days

issue openedbazelbuild/bazel-buildfarm

Netty errors

While running a moderate load on the buildfarm with sharded workers and sharded servers, we are observing the following repeated messages:

INFO: [id: 0xd3d3371e, L:/10.97.20.194:8981 - R:/10.218.44.43:58226] ignoring DATA frame for stream RST_STREAM sent.
Jun 08, 2021 2:15:51 PM io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener shouldIgnoreHeadersOrDataFrame
INFO: [id: 0xd3d3371e, L:/10.97.20.194:8981 - R:/10.218.44.43:58226] ignoring DATA frame for stream RST_STREAM sent.
Jun 08, 2021 2:15:51 PM io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener shouldIgnoreHeadersOrDataFrame
INFO: [id: 0xd3d3371e, L:/10.97.20.194:8981 - R:/10.218.44.43:58226] ignoring DATA frame for stream RST_STREAM sent.
Jun 08, 2021 2:15:51 PM io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener shouldIgnoreHeadersOrDataFrame
INFO: [id: 0xd3d3371e, L:/10.97.20.194:8981 - R:/10.218.44.43:58226] ignoring DATA frame for stream RST_STREAM sent.
Jun 08, 2021 2:15:51 PM io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener shouldIgnoreHeadersOrDataFrame
INFO: [id: 0xd3d3371e, L:/10.97.20.194:8981 - R:/10.218.44.43:58226] ignoring DATA frame for stream RST_STREAM sent.
Jun 08, 2021 2:15:51 PM io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener shouldIgnoreHeadersOrDataFrame
INFO: [id: 0xd3d3371e, L:/10.97.20.194:8981 - R:/10.218.44.43:58226] ignoring DATA frame for stream RST_STREAM sent.
Jun 08, 2021 2:15:51 PM io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener shouldIgnoreHeadersOrDataFrame
INFO: [id: 0xd3d3371e, L:/10.97.20.194:8981 - R:/10.218.44.43:58226] ignoring DATA frame for stream RST_STREAM sent.
Jun 08, 2021 2:15:51 PM io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener shouldIgnoreHeadersOrDataFrame
INFO: [id: 0xd3d3371e, L:/10.97.20.194:8981 - R:/10.218.44.43:58226] ignoring DATA frame for stream RST_STREAM sent.
Jun 08, 2021 2:15:51 PM io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener shouldIgnoreHeadersOrDataFrame
INFO: [id: 0xd3d3371e, L:/10.97.20.194:8981 - R:/10.218.44.43:58226] ignoring DATA frame for stream RST_STREAM sent.
Jun 08, 2021 2:15:51 PM io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener shouldIgnoreHeadersOrDataFrame
INFO: [id: 0xd3d3371e, L:/10.97.20.194:8981 - R:/10.218.44.43:58226] ignoring DATA frame for stream RST_STREAM sent.
Jun 08, 2021 2:15:51 PM io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener shouldIgnoreHeadersOrDataFrame
INFO: [id: 0xd3d3371e, L:/10.97.20.194:8981 - R:/10.218.44.43:58226] ignoring DATA frame for stream RST_STREAM sent.
Jun 08, 2021 2:15:51 PM io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener shouldIgnoreHeadersOrDataFrame
INFO: [id: 0xd3d3371e, L:/10.97.20.194:8981 - R:/10.218.44.43:58226] ignoring DATA frame for stream RST_STREAM sent.

Not sure about the impact, but this might be fixed in Netty 4.1.44 - the buildfarm currently depends on Netty 4.1.38

created time in 6 days

delete branch bazelbuild/bazel-buildfarm

delete branch : luxe/filegroup_exec_wrappers

delete time in 7 days

push eventbazelbuild/bazel-buildfarm

Trevor Hickey

commit sha ac76c426a2daf5c20dd252566999cccaff11cc09

group execution wrappers for easier runtime packaging (#798)

view details

push time in 7 days

PR merged bazelbuild/bazel-buildfarm

Reviewers
group execution wrappers for easier runtime packaging cla: yes

Our internal deployment of buildfarm needs to reference each execution wrapper individually. This means any change to the public repo, requires similar updates to the private repo. By grouping all execution wrappers we can reference a single dependency which will make our internal deployment easier.

+11 -0

0 comment

1 changed file

luxe

pr closed time in 7 days

push eventbazelbuild/bazel-buildfarm

Trevor Hickey

commit sha 2cf81c71d8034078a403332c7f110088b88a0631

enforce that java package names match file layout (#803)

view details

Trevor Hickey

commit sha 3a625fe059ead651d2e32749b21fed43bf947178

Gate code coverage at 40% (#804)

view details

Trevor Hickey

commit sha 616e0b6f5937738eb3bf0cb9268449ee3c6c9169

Merge branch 'master' into luxe/filegroup_exec_wrappers

view details

push time in 7 days