profile
viewpoint

bazelbuild/bazel 14207

a fast, scalable, multi-language and extensible build system

google/skylark 1145

Skylark in Go: the Skylark configuration language, implemented in Go [MOVED to go.starlark.net]

google/starlark-go 865

Starlark in Go: the Starlark configuration language, implemented in Go

bazelbuild/bazel-gazelle 359

Gazelle is a Bazel build file generator for Bazel projects. It natively supports Go and protobuf, and it may be extended to support new languages and custom rule sets.

jayconrod/gypsum 51

An experimental new programming language

bazelbuild/bazel-integration-testing 23

Framework for integration tests that call Bazel

jayconrod/rules_go_simple 12

A simple set of Go rules for Bazel, used for learning and experimentation.

jayconrod/imp-interpreter 8

A minimal interpreter for the toy language, IMP, used as an example for building interpreters.

jayconrod/minibox 4

Tiny experimental program for running tiny experimental containers

jayconrod/pinky-mode 4

An Emacs minor mode which lets you navigate without holding Control. Save your pinkies!

issue closedbazelbuild/rules_go

go_repository_tools fails due to missing go executable

Running on an Alpine Docker image,

In WORKSPACE, we've got io_bazel_rules_go as follows. It may not be the root of the problem, since the line number of the error points golang/protobuf, also copied below. In any case, the actual failure is external/go_sdk/bin/go not found, as shown below.

http_archive(
    name = "io_bazel_rules_go",
    urls = [
        "https://storage.googleapis.com/bazel-mirror/github.com/bazelbuild/rules_go/releases/download/v0.20.3/rules_go-v0.20.3.tar.gz",
        "https://github.com/bazelbuild/rules_go/releases/download/v0.20.3/rules_go-v0.20.3.tar.gz",
    ],
    sha256 = "e88471aea3a3a4f19ec1310a55ba94772d087e9ce46e41ae38ecebe17935de7b",
)
go_repository(
    name = "com_github_golang_protobuf",
    tag="v1.3.0",
    importpath = "github.com/golang/protobuf"
)

Builds fail with

Step #1: INFO: Call stack for the definition of repository 'bazel_gazelle_go_repository_tools' which is a go_repository_tools (rule definition at /root/.cache/bazel/_bazel_root/f8087e59fd95af1ae29e8fcb7ff1a3dc/external/bazel_gazelle/internal/go_repository_tools.bzl:116:23):
Step #1:  - /root/.cache/bazel/_bazel_root/f8087e59fd95af1ae29e8fcb7ff1a3dc/external/bazel_gazelle/deps.bzl:72:5
Step #1:  - /src/WORKSPACE:191:1

Step #1: INFO: Call stack for the definition of repository 'bazel_gazelle_go_repository_tools' which is a go_repository_tools (rule definition at /root/.cache/bazel/_bazel_root/f8087e59fd95af1ae29e8fcb7ff1a3dc/external/bazel_gazelle/internal/go_repository_tools.bzl:116:23):
Step #1:  - /root/.cache/bazel/_bazel_root/f8087e59fd95af1ae29e8fcb7ff1a3dc/external/bazel_gazelle/deps.bzl:72:5
Step #1:  - /src/WORKSPACE:191:1
Step #1: ERROR: An error occurred during the fetch of repository 'bazel_gazelle_go_repository_tools':
Step #1:    list_repository_tools_srcs: env: can't execute '/root/.cache/bazel/_bazel_root/f8087e59fd95af1ae29e8fcb7ff1a3dc/external/go_sdk/bin/go': No such file or directory

closed time in a day

bimargulies-google

issue commentbazelbuild/rules_go

go_repository_tools fails due to missing go executable

I was able to reproduce this with a more minimal Dockerfile. It reproduces with a WORKSPACE file with the rules_go boilerplate code and a trivial go_binary.

# This image is based on alpine and includes nix. (https://github.com/NixOS/docker)
FROM nixos/nix
RUN apk add build-base go tzdata git patch
RUN apk add bazel --update-cache --repository http://dl-3.alpinelinux.org/alpine/edge/testing/ --allow-untrusted
# nix version of patch works with googleapis.
RUN nix-env --install patch-2.7.6
# minimal WORKSPACE
COPY src /src

The error is misleading. The loader is missing, not the go binary.

# ls external/go_sdk/bin/go
external/go_sdk/bin/go

# ldd external/go_sdk/bin/go
	/lib64/ld-linux-x86-64.so.2 (0x7f063db02000)
	libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x7f063db02000)
	libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7f063db02000)

# ls /lib64/ld-linux-x86-64.so.2
ls: /lib64/ld-linux-x86-64.so.2: No such file or directory

This was reported earlier in #1376. Unfortunately, I can't provide support for NixOS or other distributions where the Go toolchain downloaded from golang.org doesn't work. rules_go may still work if you install the Go toolchain outside the workspace, then register it using:

go_register_toolchains(go_version = "host")

Some more changes may be needed. I tried this briefly, but it looks like the system Go toolchain might not work either:

ERROR: /root/.cache/bazel/_bazel_root/f8087e59fd95af1ae29e8fcb7ff1a3dc/external/go_sdk/BUILD.bazel:43:1: error executing shell command: '/bin/bash -c 'external/go_sdk/bin/go' tool compile -o 'bazel-out/host/bin/external/go_sdk/builder'.a -I 'external/go_sdk' -trimpath=$PWD $@ && 'external/go_sdk/bin/go' tool link -o 'bazel-out/host/...' failed (Exit 1)
2020/02/22 14:21:14 cannot handle R_TLS_IE (sym sync/atomic.(*Value).Store) when linking internally
bimargulies-google

comment created time in a day

delete branch jayconrod/rules_go

delete branch : fix-stamp

delete time in 2 days

push eventbazelbuild/rules_go

Jay Conrod

commit sha cd8080a1c0a08cae100b794f42c28f6f604557d1

tests/legacy/examples/stamped_bin: work around go1.14 bug (#2380) Fixes #2379

view details

push time in 2 days

PR merged bazelbuild/rules_go

tests/legacy/examples/stamped_bin: work around go1.14 bug cla: yes

Fixes #2379

+9 -10

0 comment

1 changed file

jayconrod

pr closed time in 2 days

issue closedbazelbuild/rules_go

//tests/legacy/examples/stamped_bin:stamped_test fails with Go 1.14

Caused by golang/go#37369.

The test assigns a non-constant expression to BUILD_TIMESTAMP, and in Go 1.14, the linker doesn't overwrite it. Not sure if that's intended, but it's different than 1.13.8.

closed time in 2 days

jayconrod

issue commentgolang/go

cmd/link: -X does not work on variables with non-constant default values

Perhaps this is a separate feature request, but would it make sense for the linker to emit an error when a -X flag can't be applied? I don't have a strong opinion about the semantics either way; I'd expect -X is normally used on variables that are unassigned or are statically initialized. But I'd rather have a loud failure when I use -X incorrectly.

jayconrod

comment created time in 2 days

PR opened bazelbuild/rules_go

tests/legacy/examples/stamped_bin: work around go1.14 bug

Fixes #2379

+9 -10

0 comment

1 changed file

pr created time in 2 days

create barnchjayconrod/rules_go

branch : fix-stamp

created branch time in 2 days

issue openedbazelbuild/rules_go

//tests/legacy/examples/stamped_bin:stamped_test fails with Go 1.14

Caused by golang/go#37369.

The test assigns a non-constant expression to BUILD_TIMESTAMP, and in Go 1.14, the linker doesn't overwrite it. Not sure if that's intended, but it's different than 1.13.8.

created time in 2 days

issue commentbazelbuild/rules_go

Update github.com/golang/protobuf to v1.3.3

@noahdietz I'd be happy to do a quick sync any time. The rules_go protobuf integration is not as good as it could be, but I haven't had time to really understand it and make big changes. If you or anyone on your team is able to help improve that, I'd be very grateful. :)

noahdietz

comment created time in 2 days

issue closedbazelbuild/rules_go

Update github.com/golang/protobuf to v1.3.3

<!-- Please answer these questions before submitting your issue. Thanks! -->

What version of rules_go are you using?

<!-- Check io_bazel_rules_go in WORKSPACE if you're not sure -->

v0.21.3

What version of gazelle are you using?

<!-- Check bazel_gazelle in WORKSPACE if you're not sure -->

v0.20.0

What version of Bazel are you using?

<!-- Run "bazel version" to find out -->

2.1.0

Does this issue reproduce with the latest releases of all the above?

Yes

What operating system and processor architecture are you using?

Debian x86-64

What did you do?

<!-- If possible, provide a minimal recipe for reproducing the error. -->

Use cloud.google.com/go@v0.53.0 in a project.

go.mod

module github.com/noahdietz/repro

go 1.14

require (
	cloud.google.com/go v0.53.0
)

WORKSPACE

workspace(name = "com_noahdietz_repro")

load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")

http_archive(
    name = "com_google_protobuf",
    strip_prefix = "protobuf-3.11.4",
    urls = ["https://github.com/protocolbuffers/protobuf/archive/v3.11.4.zip"],
    sha256 = "9748c0d90e54ea09e5e75fb7fac16edce15d2028d4356f32211cfa3c0e956564",
)

load("@com_google_protobuf//:protobuf_deps.bzl", "protobuf_deps")

protobuf_deps()

http_archive(
    name = "io_bazel_rules_go",
    urls = [
        "https://storage.googleapis.com/bazel-mirror/github.com/bazelbuild/rules_go/releases/download/v0.21.3/rules_go-v0.21.3.tar.gz",
        "https://github.com/bazelbuild/rules_go/releases/download/v0.21.3/rules_go-v0.21.3.tar.gz",
    ],
    sha256 = "af04c969321e8f428f63ceb73463d6ea817992698974abeff0161e069cd08bd6",
)

load("@io_bazel_rules_go//go:deps.bzl", "go_rules_dependencies", "go_register_toolchains")

go_rules_dependencies()

go_register_toolchains()

http_archive(
    name = "bazel_gazelle",
    urls = [
        "https://storage.googleapis.com/bazel-mirror/github.com/bazelbuild/bazel-gazelle/releases/download/v0.20.0/bazel-gazelle-v0.20.0.tar.gz",
        "https://github.com/bazelbuild/bazel-gazelle/releases/download/v0.20.0/bazel-gazelle-v0.20.0.tar.gz",
    ],
    sha256 = "d8c45ee70ec39a57e7a05e5027c32b1576cc7f16d9dd37135b0eddde45cf1b10",
)

# gazelle:repo bazel_gazelle
load("@bazel_gazelle//:deps.bzl", "gazelle_dependencies", "go_repository")

gazelle_dependencies()

// all of the go_repository rules generated by gazelle follow...

BUILD.bazel

load("@io_bazel_rules_go//go:def.bzl", "go_binary", "go_library")
load("@bazel_gazelle//:def.bzl", "gazelle")

# gazelle:prefix github.com/noahdietz/repro
gazelle(name = "gazelle")

go_library(
    name = "go_default_library",
    srcs = ["main.go"],
    importpath = "github.com/noahdietz/repro",
    visibility = ["//visibility:private"],
    deps = ["@com_google_cloud_go//language/apiv1:go_default_library"],
)

go_binary(
    name = "repro",
    embed = [":go_default_library"],
    visibility = ["//visibility:public"],
)

main.go

package main

import (
	"context"
	"fmt"

	language "cloud.google.com/go/language/apiv1"
)

func main() {
	ctx := context.Background()
	c, err := language.NewClient(ctx)
	if err != nil {}
	_ = c
	fmt.Println("Hello, World!")
}

What did you expect to see?

$ bazel run :repro
INFO: Analyzed target //:repro (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
Target //:repro up-to-date:
  bazel-bin/linux_amd64_stripped/repro
INFO: Elapsed time: 0.145s, Critical Path: 0.00s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action
INFO: Build completed successfully, 1 total action
Hello, World!

What did you see instead?

I get a compilation error in cloud.google.com/go/language/apiv1 because the Go gRPC client stub generated (via golang/protobuf v1.3.2) on the fly expects a concrete type grpc.ClientConn, but is instead receiving an implementation of the grpc.ClientConnInterface from the code in cloud.google.com/go/language/apiv1.

bazel run :repro
INFO: Analyzed target //:repro (123 packages loaded, 7239 targets configured).
INFO: Found 1 target...
ERROR: ~/.cache/bazel/_bazel_ndietz/4ed524bbb1659f58cebccface4355903/external/com_google_cloud_go/language/apiv1/BUILD.bazel:3:1: GoCompilePkg external/com_google_cloud_go/language/apiv1
/linux_amd64_stripped/go_default_library%/cloud.google.com/go/language/apiv1.a failed (Exit 1) builder failed: error executing command bazel-out/host/bin/external/go_sdk/builder compilepkg -sdk external/go_sdk -ins
tallsuffix linux_amd64 -src external/com_google_cloud_go/language/apiv1/doc.go -src ... (remaining 27 argument(s) skipped)

Use --sandbox_debug to see verbose messages from the sandbox
compilepkg: error running subcommand: exit status 2
~/.cache/bazel/_bazel_ndietz/4ed524bbb1659f58cebccface4355903/sandbox/linux-sandbox/778/execroot/com_noahdietz_repro/external/com_google_cloud_go/language/apiv1/language_client.go:160:46
: cannot use connPool (type "google.golang.org/api/internal".ConnPool) as type *"google.golang.org/grpc".ClientConn in argument to language.NewLanguageServiceClient
Target //:repro failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 24.691s, Critical Path: 5.14s
INFO: 134 processes: 134 linux-sandbox.
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully

I attempted to use the Override instructions but got a bunch of other weird errors. So instead I cloned a local copy of rules_go and wired that up in my WORKSPACE. In the local copy, I changed the version of golang/protobuf being included to v1.3.3. Then everything worked.

We released the clients in cloud.google.com/go with a change to pass a new grpc.ClientConnInterface implementation after github.com/googleapis/go-genproto had been regenerated with golang/protobuf v1.3.3. We didn't realize rules_go was at risk here, so apologies on that.

closed time in 2 days

noahdietz

issue commentbazelbuild/rules_go

Update github.com/golang/protobuf to v1.3.3

Fixed by #2378

noahdietz

comment created time in 2 days

IssuesEvent

issue commentbazelbuild/rules_go

go_repository_tools fails due to missing go executable

Sorry, #2378 actually resolved #2376, not this issue.

bimargulies-google

comment created time in 2 days

delete branch jayconrod/rules_go

delete branch : update-dependencies

delete time in 2 days

push eventbazelbuild/rules_go

Jay Conrod

commit sha b64542950a81e86c37e6bc5335060b70c251cf5e

Update dependencies in preparation for v0.22.0 release (#2378) * Update dependencies in preparation for v0.22.0 release Fixes #2377

view details

push time in 2 days

PR merged bazelbuild/rules_go

Update dependencies in preparation for v0.22.0 release cla: yes

Fixes #2377

+11794 -5604

0 comment

12 changed files

jayconrod

pr closed time in 2 days

issue closedbazelbuild/rules_go

go_repository_tools fails due to missing go executable

Running on an Alpine Docker image,

In WORKSPACE, we've got io_bazel_rules_go as follows. It may not be the root of the problem, since the line number of the error points golang/protobuf, also copied below. In any case, the actual failure is external/go_sdk/bin/go not found, as shown below.

http_archive(
    name = "io_bazel_rules_go",
    urls = [
        "https://storage.googleapis.com/bazel-mirror/github.com/bazelbuild/rules_go/releases/download/v0.20.3/rules_go-v0.20.3.tar.gz",
        "https://github.com/bazelbuild/rules_go/releases/download/v0.20.3/rules_go-v0.20.3.tar.gz",
    ],
    sha256 = "e88471aea3a3a4f19ec1310a55ba94772d087e9ce46e41ae38ecebe17935de7b",
)
go_repository(
    name = "com_github_golang_protobuf",
    tag="v1.3.0",
    importpath = "github.com/golang/protobuf"
)

Builds fail with

Step #1: INFO: Call stack for the definition of repository 'bazel_gazelle_go_repository_tools' which is a go_repository_tools (rule definition at /root/.cache/bazel/_bazel_root/f8087e59fd95af1ae29e8fcb7ff1a3dc/external/bazel_gazelle/internal/go_repository_tools.bzl:116:23):
Step #1:  - /root/.cache/bazel/_bazel_root/f8087e59fd95af1ae29e8fcb7ff1a3dc/external/bazel_gazelle/deps.bzl:72:5
Step #1:  - /src/WORKSPACE:191:1

Step #1: INFO: Call stack for the definition of repository 'bazel_gazelle_go_repository_tools' which is a go_repository_tools (rule definition at /root/.cache/bazel/_bazel_root/f8087e59fd95af1ae29e8fcb7ff1a3dc/external/bazel_gazelle/internal/go_repository_tools.bzl:116:23):
Step #1:  - /root/.cache/bazel/_bazel_root/f8087e59fd95af1ae29e8fcb7ff1a3dc/external/bazel_gazelle/deps.bzl:72:5
Step #1:  - /src/WORKSPACE:191:1
Step #1: ERROR: An error occurred during the fetch of repository 'bazel_gazelle_go_repository_tools':
Step #1:    list_repository_tools_srcs: env: can't execute '/root/.cache/bazel/_bazel_root/f8087e59fd95af1ae29e8fcb7ff1a3dc/external/go_sdk/bin/go': No such file or directory

closed time in 2 days

bimargulies-google

issue openedgolang/go

cmd/link: -X does not work on variables with non-constant default values

What version of Go are you using (go version)?

<pre> $ go version go version go1.14rc1 darwin/amd64 </pre>

Does this issue reproduce with the latest release?

This is a regression in Go 1.14rc1. It does not reproduce in 1.13.8.

What operating system and processor architecture are you using (go env)?

<details><summary><code>go env</code> Output</summary><br><pre> $ go env GO111MODULE="" GOARCH="amd64" GOBIN="" GOCACHE="/Users/jayconrod/Library/Caches/go-build" GOENV="/Users/jayconrod/Library/Application Support/go/env" GOEXE="" GOFLAGS="" GOHOSTARCH="amd64" GOHOSTOS="darwin" GOINSECURE="" GONOPROXY="" GONOSUMDB="" GOOS="darwin" GOPATH="/Users/jayconrod/go" GOPRIVATE="" GOPROXY="https://proxy.golang.org,direct" GOROOT="/opt/go/installed" GOSUMDB="sum.golang.org" GOTMPDIR="" GOTOOLDIR="/opt/go/installed/pkg/tool/darwin_amd64" GCCGO="gccgo" AR="ar" CC="clang" CXX="clang++" CGO_ENABLED="1" GOMOD="/Users/jayconrod/Code/test/tmp2/go.mod" CGO_CFLAGS="-g -O2" CGO_CPPFLAGS="" CGO_CXXFLAGS="-g -O2" CGO_FFLAGS="-g -O2" CGO_LDFLAGS="-g -O2" PKG_CONFIG="pkg-config" GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/rq/x0692kqj6ml8cvrhcqh5bswc008xj1/T/go-build343063917=/tmp/go-build -gno-record-gcc-switches -fno-common" </pre></details>

What did you do?

Run the testscript below:

go run -ldflags='-X=example.com/test/stamp.STAMP=pass' ./main
stdout pass

-- go.mod --
module example.com/test

go 1.14
-- main/main.go --
package main

import (
	"fmt"

	"example.com/test/stamp"
)

func main() {
	fmt.Printf("STAMP = %s\n", stamp.STAMP)
}
-- stamp/stamp.go --
package stamp

var DEFAULT = "fail"
var STAMP = DEFAULT

What did you expect to see?

The program should print STAMP = pass, and the test should pass.

What did you see instead?

The program prints STAMP = fail.

This only happens when STAMP is assigned to another variable. The test passes if STAMP is assigned the value "fail" instead of DEFAULT.

Other thoughts

I've milestoned this Go1.14 and added the Soon label because this is a regression.

I don't know if this was intended to work before. If not, feel free to close.

If it was intended to work, I'm not sure whether this should block 1.14.

created time in 2 days

push eventjayconrod/rules_go

Jay Conrod

commit sha 16f78424ce9de49fffa5b7739a89ebd4fc65bfba

add shallow_since to rules_cc

view details

push time in 2 days

PR opened bazelbuild/rules_go

Update dependencies in preparation for v0.22.0 release

Fixes #2377

+11793 -5604

0 comment

12 changed files

pr created time in 2 days

create barnchjayconrod/rules_go

branch : update-dependencies

created branch time in 2 days

issue commentbazelbuild/rules_go

Update github.com/golang/protobuf to v1.3.3

Thanks for the offer, but I'd rather do it. Most of the work is updating patches, and reviewing patch changes is hard =P

noahdietz

comment created time in 2 days

issue commentbazelbuild/rules_go

Update github.com/golang/protobuf to v1.3.3

@noahdietz I'll take a look at this as soon as I can, but between performance reviews and the short week, I have less time than usual.

I suspect that I'll need to cut a new release branch for this, which means everything needs to be updated at the same time. cloud.google.com/go definitely needs to keep working.

Preparing a new release takes about a day, assuming there aren't issues with other dependencies.

noahdietz

comment created time in 2 days

issue commentbazelbuild/bazel-gazelle

update-repos should allow overriding the source of `go_repository` in WORKSPACE from the comment.

It sounds like there might be a couple issues here:

  1. Gazelle reports an error (from the merged.CheckGazelleLoaded function you linked) if, in WORKSPACE, a symbol is loaded from the bazel_gazelle repo, but bazel_gazelle isn't declared, either with a rule (e.g., http_archive) or a # gazelle:repo directive. That directive should look like this:
# gazelle:repo bazel_gazelle

I don't think this is actually documented, and it's not consistent with the # gazelle:repository directive. It should probably recognize something like the directive below instead:

# gazelle:repository http_archive name=bazel_gazelle

In any case, it doesn't seem like you should hit this error if you're loading everything out of a company federation. This should only come up if you're loading something from @bazel_gazelle.

  1. There's no way to override where go_repository is loaded from.

This part doesn't make sense to me. If there's already a load for go_repository, update-repos shouldn't introduce a new one or change the existing one. It will only add a load for go_repository if it's not already loaded.

If you're seeing this behavior, could you post a small test repo or gist that reproduces it?

amarshat

comment created time in 2 days

issue commentbazelbuild/rules_go

go_repository_tools fails due to missing go executable

Could you post a larger part of your WORKSPACE file, including everything related to rules_go and Gazelle?

I suspect that either go_register_toolchains is not called or the Go toolchain is downloaded with a name other than go_sdk. That's fine, but gazelle_dependencies needs to be called with the actual name:

gazelle_dependencies(go_sdk = "actual_go_sdk")
bimargulies-google

comment created time in 2 days

issue commentgolang/go

x/blog: builds failing on release-branch.go1.12 due to missing dependencies

Builders have support for testing with third-party packages in golang.org/x repos only in module mode.

How does the builder set up the workspace? I assumed the builder would go get the packages to be tested, but when I tried that locally with Go 1.12.17, it checked out github.com/yuin/goldmark, and I didn't run into any error with the go test command.

No packages in x/blog import x/website/markdown package yet, so it's strange that it is attempted to be built in GOPATH mode. Need to understand why that is.

I didn't think about this when I added the Markdown rendering, but content/static/gen.go is not excluded by build constraints, so golang.org/x/website/markdown is imported by the regular version of golang.org/x/website/content/static.

Perhaps instead, all the generator code should be moved from gen.go to makestatic.go. gen.go should just have a //go:generate pragma.

bcmills

comment created time in 3 days

issue commentgolang/go

cmd/go: 'Access is denied' when renaming module cache directory

Yeah, I'm not wild about the migration.

Maybe we could compare a directory to the zip file and check that it's fully extracted instead of deleting and re-extracting it. It's not clear that's actually less file I/O though.

kiwionly

comment created time in 3 days

issue commentgolang/go

cmd/go: 'Access is denied' when renaming module cache directory

@alfmos Thanks! It's really good to confirm this is the root cause.

@alexbrainman You're right, there's probably a way to do this that doesn't involve renaming.

In an earlier comment, I was thinking about extracting the zip file in place (while holding a lock, which we already do), then creating an empty file (let's call it a .extracted file) to indicate the directory is complete and the zip has been fully extracted.

If we later find that the module was only partially extracted by an earlier run (that is, the directory is present but the .extracted file is not), it's prudent to remove the directory before unzipping the module again. zip.Unzip requires this at the moment, though it could be changed if needed. I was worried that os.RemoveAll would fail if any files were open in that directory, and that seems to be the case, but robustio.RemoveAll works well enough. I didn't see any errors using that in a synthetic test.

kiwionly

comment created time in 3 days

push eventbazelbuild/rules_go

Robbert van Ginkel

commit sha c5a0b28c495edc83ec80477e1c8addf631f156a6

Use paramfile for invoking nogo (#2375) When calling the builder binary from starlark, the `args.use_param_file` option is used to support long argument lists. For nogo, the builder calls the nogo binary with a list of arguments for which the size is proportional to what the builder was called with. Currently nogo isn't called with a parameter file but instead gets it's arguments from the commandline. This change writes the nogo arguments to file and calls nogo with an argument to read the parameter file for arguments. Updates #2002

view details

push time in 3 days

PR merged bazelbuild/rules_go

Reviewers
Use paramfile for invoking nogo cla: yes

When calling the builder binary from starlark, the args.use_param_file option is used to support long argument lists. For nogo, the builder calls the nogo binary with a list of arguments for which the size is proportional to what the builder was called with. Currently nogo isn't called with a parameter file but instead gets it's arguments from the commandline.

This change writes the nogo arguments to file and calls nogo with an argument to read the parameter file for arguments.

References #2002.

+14 -3

1 comment

2 changed files

robbertvanginkel

pr closed time in 3 days

issue closedbazelbuild/rules_go

`rules_go` does not support identifying `@io_bazel_rules_go` with `@`

What version of rules_go are you using?

HEAD, i.e., b8db84a2f3fd7fac1801ddef376a549c2d83b682

What version of gazelle are you using?

n/a

What version of Bazel are you using?

HEAD (after https://github.com/bazelbuild/bazel/commit/81b57d605de823f8e6f846a06d8e50d9a00041e3)

Does this issue reproduce with the latest releases of all the above?

n/a

What operating system and processor architecture are you using?

not OS specfic

Any other potentially useful information about your toolchain?

see https://github.com/bazelbuild/continuous-integration

What did you do?

https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/1281

What did you expect to see?

passing test

What did you see instead?

  • https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/1281#ffbfcb90-d83a-4c54-a0dd-48b6f9b74680
  • https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/1281#4c4875ad-e973-4af1-ac72-3ae3cfb6318f
  • https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/1281#e81a9d4e-21e7-408b-9ec8-58483cbd6f0d
  • https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/1281#e81a9d4e-21e7-408b-9ec8-58483cbd6f0d

Futher information

rules_go assumes at various places that bazel-generated artifacts have a path starting with external. This is wrong if rules_go is the main repository. Examples include https://github.com/bazelbuild/rules_go/blob/b8db84a2f3fd7fac1801ddef376a549c2d83b682/go/tools/bazel_testing/bazel_testing.go#L297

Tracking issue is https://github.com/bazelbuild/bazel/issues/10246

Background on the change https://github.com/bazelbuild/bazel/issues/7130

closed time in 4 days

aehlig

issue commentbazelbuild/bazel-gazelle

Running gazelle inside repository_rule for custom language

It's cool that you have Gazelle working for Elixir. Do you have a repo up anywhere? If so, I'd be happy to add it to Supported languages (or feel free to send a PR editing that in).

About invoking Gazelle in a repository rule, I'd suggest following the pattern from go_repository_tools.

  • Define a repository rule elixir_repository_tools. It should either build or download a Gazelle binary. It may include other tools for finding and downloading code. It should be declared once with a predictable name from a dependency macro (users should not declare it directly).
    • As an example, go_repository_tools is declared in gazelle_dependencies. It builds cmd/gazelle and cmd/fetch_repo from source.
    • For this case, I'd lean toward downloading a binary instead of building from source. Building from source is complicated and requires a Go toolchain for the host system, which is a heavy dependency. For rules_go, a Go toolchain is a given, but that's not true for other languages.
  • elixir_repository can refer to the binary in @elixir_repository_tools (or whatever the name is). Make sure to do this through a label so that elixir_repository rules get invalidated when the binary changes.
    • For example, elixir_repository could define a hidden attr.label that defaults to @elixir_repository_tools//:bin/gazelle.
    • go_repository does a more complicated version of this. The dependency is dynamic, since it doesn't use Gazelle if the repository already has build files files.
greenboxal

comment created time in 4 days

issue commentgolang/go

cmd/go: go help get is missing the documentation of the -v flag

The -v flag is not specific to go get in either mode. It's shared with other build commands and documented in go help build. In GOPATH mode, it causes go get to print a couple extra lines, but it doesn't do anything interesting in module mode.

It should probably be removed from go help gopath-get and the summary line of go help module-get. No real need to mention it in either place.

perillo

comment created time in 4 days

issue commentgolang/go

cmd/go: 'Access is denied' when renaming module cache directory

Which op returned access-denied, remove or rename? I assume you're calling os.RemoveAll()? I'm not sure why you're testing that.

os.Rename fails. os.RemoveAll is just there to clean up the temporary directory after a successful rename.

I suggested os.File.Sync() because I use it and haven't seen such errors, tho maybe I haven't tested my app widely enough.

I don't understand why os.File.Sync would guard against ERROR_ACCESS_DENIED though. Have you seen ERROR_ACCESS_DENIED without using os.File.Sync?

We haven't been able to reproduce this problem with the go command, so it seems fairly rare, or at least dependent on other software running on the system we don't know about. I've only been able to reproduce the error with synthetic examples.

kiwionly

comment created time in 4 days

issue commentgolang/go

cmd/go: 'Access is denied' when renaming module cache directory

@networkimprov Could you explain more about why os.File.Sync would help with the ERROR_ACCESS_DENIED problem? I'm not that familiar with the pitfalls of NTFS and the Windows API, so I feel like I'm missing something. Coming from Linux / Darwin, I'd expect that to help avoid corruption in the case of power loss, but it doesn't seem like it would guard against other processes opening files.

I wrote a couple small test programs: one extracts a small module zip, then renames and removes the directory. The other repeatedly walks the directory and reads files. I was able to get an ERROR_ACCESS_DENIED pretty quickly.

I tried adding os.File.Sync in golang.org/x/mod/zip, but I still saw ERROR_ACCESS_DENIED in this case.

A quick benchmark showed that adding the os.File.Sync made a combined unzip-and-remove operation take 2.78x as long as before. So I'd rather not add it unless it's absolutely necessary.

kiwionly

comment created time in 4 days

issue commentgolang/go

proposal: cmd/go: add GOMODCACHE

I think it makes sense to move the STH out of GOPATH. We should avoid requiring a GOPATH directory in module mode. Since the STH needs to be somewhat durable, it should not be part of any cache (GOMODCACHE, GOCACHE, UserCacheDir). UserConfigDir is the only other place that makes sense.

Some open questions though:

  • If $GOPATH/pkg/mod/sumdb already exists, should we keep it there or actually move it? What if it also exists in UserConfigDir, for example, because both 1.14 and 1.15 are used on a system?
  • What if there is no UserConfigDir (HOME and other environment variables are not set), but GOPATH is set (for example, in a Docker container)? Is that an error, or should we continue using $GOPATH/pkg/mod/sumdb?
  • What if neither GOPATH nor HOME is set? I think this is currently an error right now because there's nowhere to put the module cache.

IMO, we should move $GOPATH/pkg/mod/sumdb to somewhere in UserConfigDir if that directory is defined, but we should continue to use $GOPATH/pkg/mod/sumdb if UserConfigDir is not defined.

mvdan

comment created time in 4 days

issue commentgolang/go

cmd/go: 'Access is denied' when renaming module cache directory

I spent some more timing looking at this today. Based on the MoveFileEx documentation and various threads on Windows Q&A sites and mailing list, I think the most likely explanation for this issue is that another process (probably antivirus or search indexer) is opening files in the module directory before it's completely extracted.

Here's our normal sequence of operations at a high level:

  • os.Stat the module version directory, for example, %GOPATH%\pkg\mod\golang.org\pkg\mod@v0.2.0. If it exists, we assume it's complete.
  • Download the module version's zip file, if it's not already in the cache.
  • os.OpenFile on the module version's lock file, possibly creating it.
  • LockFileEx to lock the lock file.
  • os.Stat the module version directory again. If it exists, another process got the lock and extracted the module zip first, so we assume the directory is complete.
  • os.RemoveAll on temporary module version directories with a ".tmp-" suffix. These were created by other processes which failed. We hold the lock for this version, so these directories should not be actively used. Errors are ignored here.
  • Create a temporary module version directory with a ".tmp-" suffix. Extract the module zip file there.
  • os.Rename the temporary module version directory to its final location. This calls MoveFileEx, which is where we're encountering the error.
  • Make the directory and its contents read-only.
  • UnlockFileEx on the lock file.
  • Close on the lock file.

We're getting ERROR_ACCESS_DENIED on the os.Rename call. I've verified this error is reported when os.Rename is called on a directory when another process has a file open in that directory. So while I can't confirm this is what's happening, it seems likely. Note that by default, the module cache is not in a hidden directory: %USERPROFILE%\pkg\mod will probably be read by search indexers, backup tools, and anything else that watches the user's home directory.

I thought about an alternate flow where we extract a module version to its final location, then create a special file indicating everything has been extracted. However, if we encounter an error extracting the module, we would need to call os.RemoveAll on the partial directory the next time, and that would fail if other processes have files open. So it's just a different kind of flaky.


I don't think we can fix this, but some mitigations are possible.

First, we can check if the error is ERROR_ACCESS_DENIED and add some information to the error message for this specific case.

Second, we can retry os.Rename a few times before reporting an error. It looks like cmd/go/internal/robustio.Rename can already do this.

kiwionly

comment created time in 5 days

issue commentgolang/go

does go module replace will support inherit

This is very much intentional. replace and exclude are intended for local development and fixing short-term problems with dependencies.

The ecosystem cannot scale if replacements are used everywhere, since conflicts can't be resolved.

yhyddr

comment created time in 5 days

issue commentgolang/go

x/tools/gopls: support and documentation for bazel-based projects

Tracking issue for that is bazelbuild/rules_go#512.

gonzojive

comment created time in 5 days

issue commentfatih/vim-go

ctrl+] behavior does not navigate to symbols in the same module but different source file

If you can reproduce the error go build foo: failed to cache compiled Go files, please open a new issue on https://github.com/golang/go with details. golang/go#29667 is related, but that was fixed in 1.13. In that issue, we were having some trouble with concurrent access to the build cache by multiple tools.

@stamblerre Retrying go list is a good workaround. Would love to fix the root cause though if we can isolate it.

PaulForgey

comment created time in 5 days

issue commentbazelbuild/bazel-gazelle

go_tool_library mode for go_repository BUILD file generation

bazelbuild/rules_go#2219 is the tracking issue for that work. I'd like to ship it with v0.22.0, ideally in early April. I can't promise it will be done by then though.

I just opened bazelbuild/rules_go#2374 for the nogo-specific part of this. That would probably be ready by the next release.

prestonvanloon

comment created time in 5 days

issue openedbazelbuild/rules_go

Depend on nogo with a transition and delete go_tool_library

Instead of having the toolchain depend on nogo, a common target like @io_bazel_rules_go//:go_context_data should depend on it. The dependency should transition to a configuration that doesn't depend on nogo. That would break the cyclic dependency.

With that change, we could deprecate go_tool_library, and we could drop a lot of the custom patching for analyses in org_golang_x_tools.

This would also make it possible to use an alternative nogo binary using a command-line flag.

created time in 5 days

issue commentgolang/go

cmd/go: add a way to obtain a package's build ID via list -json

cc @bcmills @matloob

mvdan

comment created time in 5 days

issue commentgolang/go

cmd/go: go list -m is confusing when go.mod is in a parent directory outside GOPATH

This is almost the same issue as #35070. The problem is that go list is failing to load the go.mod file because it doesn't contain a module directive. It appears that it's trying to add the module directive and is failing to infer the module path.

Instead, we should reject a go.mod file without a module directive with a clear message. This is based on @rsc's comment in CL 211597.

Tentatively tagging both issues for 1.15. I expect they will both be fixed by CL 211597.

cc @matloob

perillo

comment created time in 5 days

issue commentbazelbuild/bazel-gazelle

go_tool_library mode for go_repository BUILD file generation

Sorry, no updates right now. Using patches with the go_repository rule is the best way to work around this for now.

I'm working on rewriting the rules_go configuration system to use build settings and transitions instead of an aspect. Once that's done, I think it will be possible to move the nogo dependency out of the toolchain, onto a common target like @io_bazel_rules_go//:go_context_data. There could be a transition to a configuration that doesn't use nogo, breaking the cyclic dependency. That would mean go_tool_library could be deleted.

prestonvanloon

comment created time in 5 days

issue commentgolang/go

cmd/go: changes/corruption in module cache not detected

Most of what you describe is working as intended.

The integrity of module zip files is verified when they are downloaded and extracted into the cache. The go command computes a cryptographic sum of the contents of each zip file and compares that to the corresponding entry in go.sum for the main module. If there is no entry in go.sum, the go command retrieves the expected sum from sum.golang.org (unless the module is in GOPRIVATE or GONOSUMDB). So we're pretty confident about the integrity of each module, when it's ingested into the cache.

Files and directories within the cache are marked read-only to prevent accidental changes (unless the -modcacherw flag is used). go mod verify may be used to compare extracted zip contents with the zip files they came from, but it would expensive to do it on every build, since it reads a lot of files that aren't normally touched.

The reason I raise this is that I on rare occasions have seen empty cache folders. I'm not able to give further details about when/how this happens, but for me, it happens only on MacOS. In most situations, this will be visible (break the build), but not always.

Assuming "my empty folders" are empty after a `go mod download´, maybe check the folder hash after it is written ...

Could you explain more? Is the whole cache empty ($GOPATH/pkg/mod)? Or a specific module directory (e.g., $GOPATH/pkg/mod/golang.org/x/tools@v0.0.0-something-something)? Or another directory? Is it right after a module is downloaded or some time later?

The go command shouldn't automatically remove anything from the module cache. go clean -modcache is the only command that would do that.

cc @bcmills @matloob

bep

comment created time in 5 days

issue commentbazelbuild/rules_go

'go env GOROOT' does not match runtime.GOROOT

Separate of this thread, I would like a testwrapper that doesn't execute any test specific init functions that might have global side effects. I don't think thats possible with the current approach of linking the tests in, but if you have any suggestions I think that would be great. Tracking down why this init() thing seemed to work was a little confusing.

Not possible without a separate testwrapper binary I'm afraid. I'm open to that, but, as I think you mentioned earlier on #236, it would interfere with debugging.

linzhp

comment created time in 5 days

issue commentbazelbuild/rules_go

'go env GOROOT' does not match runtime.GOROOT

Just wanted to flag that the testwrapper itself running inits that are part of the test code has some surprising effects.

That can't really be avoided if the testwrapper is linked into the same binary. inits in all other packages will run first.

Relying on init order is always a little dicy though. If the test itself explicitly forks / execs itself, that will guarantee the process that actually runs the tests has GOROOT set when it starts.

Could you elaborate on the cross-compiling/remote-exec issues you forsee?

I imagine at some point it will be possible to build tests on one platform and execute them on another. It might already be possible. The Go toolchain would only work on the platform the test was built on.

linzhp

comment created time in 9 days

issue commentbazelbuild/rules_go

'go env GOROOT' does not match runtime.GOROOT

However because of the testwrapper added in #2348, the os.Setenv that changes the GOROOT from an init() runs in the testwrapper, and alters the goroot before the test binary is executed. If the tests are run with --test_env=GO_TEST_WRAP=0 they still fail.

I'm not sure I follow. #2348 didn't add anything that sets GOROOT in the environment of the child process, did it?

Could the test itself set GOROOT, then fork/exec itself? That might be a more sound approach, at least on LInux.

As it needs to be set outside of the test binary, maybe we could add something to rules_go for this? Something like a go_test(..., with_toolchain=True) or a go_test(..., tags=["needs-go-toolchain"]), for which rules_go would add a data dependency on @go_sdk//:files, sets up the required environment variables (PATH/GOROOT) using https://docs.bazel.build/versions/2.0.0/skylark/lib/testing.html#TestEnvironment?

I think that would work, but I'm not sure I want to support it. It's difficult (as we are seeing) to run the Go command inside a Bazel sandbox, and it may not work in cross-compiling / remote environments.

Maybe a custom rule using the toolchain API is the way to go?

linzhp

comment created time in 9 days

issue commentbazelbuild/rules_go

'go env GOROOT' does not match runtime.GOROOT

I think the best way to resolve this is to set the GOROOT environment variable explicitly in your TestMain.

runtime.GOROOT() will return the value of the GOROOT environment variable if it's not empty. Otherwise, it will return a value set by the linker, determined by the value of the GOROOT_FINAL environment variable at link time.

rules_go explicitly sets GOROOT_FINAL=GOROOT. This is the string you're seeing. That string replaces paths in std packages, so we have to set it to something constant for the build to be reproducible. We can't set it to the "real" GOROOT, because that may be different for every sandboxed action.

The generated test main can't really set GOROOT automatically either. Most tests run without the Go toolchain. Even if a toolchain is available in runfiles, it won't be at a predictable location, and there may be multiple toolchains. This test hardcodes external/go_sdk/bin, but the toolchain could be in any directory, maybe on the host system.

linzhp

comment created time in 10 days

push eventbazelbuild/bazel-gazelle

Jay Conrod

commit sha 57357431fc160b6877f632677d92657defd5ccbb

Update rules_go and boilerplate (#712)

view details

push time in 10 days

delete branch jayconrod/bazel-gazelle

delete branch : upgrade-rules

delete time in 10 days

delete branch jayconrod/rules_go

delete branch : announce-releases

delete time in 10 days

push eventbazelbuild/rules_go

Jay Conrod

commit sha b763c84aa4f124273bb6048136fe86a4de60a1ae

Announce releases v0.21.3, v0.20.7 [skip ci] (#2373)

view details

push time in 10 days

PR merged bazelbuild/rules_go

Reviewers
Announce releases v0.21.3, v0.20.7 [skip ci] cla: yes
+15 -10

0 comment

1 changed file

jayconrod

pr closed time in 10 days

created tagbazelbuild/rules_go

tagv0.21.3

Go rules for Bazel

created time in 10 days

created tagbazelbuild/rules_go

tagv0.20.7

Go rules for Bazel

created time in 10 days

PR opened bazelbuild/bazel-gazelle

Reviewers
Update rules_go and boilerplate
+8 -6

0 comment

2 changed files

pr created time in 10 days

create barnchjayconrod/bazel-gazelle

branch : upgrade-rules

created branch time in 10 days

PR opened bazelbuild/rules_go

Reviewers
Announce releases v0.21.3, v0.20.7 [skip ci]
+15 -10

0 comment

1 changed file

pr created time in 10 days

create barnchjayconrod/rules_go

branch : announce-releases

created branch time in 10 days

push eventbazelbuild/rules_go

Jay Conrod

commit sha f0e31a900dd9d9bd8ce9667390b1e29216150a59

Set RULES_GO_VERSION to 0.21.3

view details

Jay Conrod

commit sha 32270362bb39c806a3f4a140b6406ac24b2dbe94

Support go1.13.8, 1.12.17 (#2372) Fixes #2368

view details

push time in 10 days

push eventbazelbuild/rules_go

Jay Conrod

commit sha 74e85a2886a9531a3fb62cd1075ef2184268944f

Set RULES_GO_VERSION to 0.20.7

view details

Jay Conrod

commit sha 45e99c72f7d8300c0b7e5e78620fdb25a2d58b3b

Support go1.13.8, 1.12.17 (#2372) Fixes #2368

view details

push time in 10 days

delete branch jayconrod/rules_go

delete branch : update-go

delete time in 10 days

push eventbazelbuild/rules_go

Jay Conrod

commit sha 20e4606c9ecfcd21b40facb098848e42e8477c56

Support go1.13.8, 1.12.17 (#2372) Fixes #2368

view details

push time in 10 days

PR merged bazelbuild/rules_go

Reviewers
Support go1.13.8, 1.12.17 cla: yes

Fixes #2368

+27 -1

0 comment

1 changed file

jayconrod

pr closed time in 10 days

PR opened bazelbuild/rules_go

Reviewers
Support go1.13.8, 1.12.17

Fixes #2368

+27 -1

0 comment

1 changed file

pr created time in 10 days

create barnchjayconrod/rules_go

branch : update-go

created branch time in 10 days

issue commentbazelbuild/rules_go

android_386_cgo causes a linker error

@steeve Thanks for investigating.

I'm working on a change that switches cross-compilation from using an aspect to using Bazel's new build settings and transitions. Mode validation should be part of that, or a follow-up soon after. Currently there are a lot of modes that can be specified, but aren't valid, and mode.bzl does not do a good job of reporting failures.

ekuiter

comment created time in 10 days

issue commentbazelbuild/rules_go

android_386_cgo causes a linker error

cc @steeve who I think had this working in the past.

ekuiter

comment created time in 11 days

issue commentgolang/go

x/mod/zip: TestVCS is failing on linux-amd64-longtest and linux-386-longtest sometimes

Can't reproduce. Pretty sure it's a network or upstream flake though.

The answer here is probably to pack up the repo in testdata, then do some git config magic so a fetch in another repository goes there. Only on the builders.

dmitshur

comment created time in 11 days

push eventbazelbuild/rules_go

Zhongpeng Lin

commit sha 4655ce62b9988f1c971605f64594f8fbe9fcd969

upgrade org_golang_x_tools (#2367)

view details

push time in 11 days

PR merged bazelbuild/rules_go

Reviewers
upgrade x/tools cla: yes

Upgraded x/tools to the master as of 2020-02-11, because we need it...

+1122 -378

2 comments

9 changed files

linzhp

pr closed time in 11 days

issue commentbazelbuild/rules_go

How can I use the build flag -p on tests

-p sets the number of actions that cmd/go may run in parallel. rules_go doesn't use cmd/go internally; instead it tells Bazel to invoke the Go compiler, linker, and other tools directly. I think the Bazel equivalent might be --worker_max_instances? It will depend on how you're executing actions though (it will be different if you're using remote execution).

Why do you need this exactly? If you have some tests that aren't safe to run in parallel, adding tags =["exclusive"] to the test targets would be better. Bazel will only execute exclusive tests one at a time. If you want to see the output from one test at a time, --test_output=streamed also runs one test at a time.

cshunger

comment created time in 11 days

push eventlinzhp/rules_go

Jay Conrod

commit sha 23e9a29067abaa23e1ab46280076129d81037797

regenerate patches and add new analysis to tools_nogo

view details

Jay Conrod

commit sha d511298a5249290d40a9706171dec031a28ab85d

update popular_repos

view details

push time in 11 days

issue commentbazelbuild/rules_go

--test_output=errors output now also contains successfully passed tests, i.e. became too verbose...

This would be in v0.22.0, which should ship in early April. I aim to cut a new minor version of rules_go once a quarter, near the beginning of the quarter.

robbertvanginkel

comment created time in 11 days

issue openedbazelbuild/rules_go

Support go1.13.8, go1.12.17

created time in 11 days

pull request commentbazelbuild/rules_go

upgrade x/tools

Normally I don't accept PRs updating dependencies out of cycle. A lot of things depend on x/tools though, and it's difficult to override.

Are you able to follow Overriding dependencies? I'm not 100% sure it will work for x/tools because of the patches required, but that should be the first thing to try. Repositories overridden that way must be declared before calling go_rules_dependencies.

If not, please follow Updating dependencies to get this PR working.

If that doesn't work, I can also update x/tools in another PR today or tomorrow.

linzhp

comment created time in 11 days

issue commentgolang/go

cmd/go: unused function addModFlag in internal/modcmd/mod.go

Thanks. CL 219197 will remove this.

perillo

comment created time in 11 days

issue commentgolang/go

x/mod/zip: TestVCS is failing on linux-amd64-longtest and linux-386-longtest sometimes

Thanks for reporting. I'll take a look.

This looks like a failure running git fetch, so it's likely a network or upstream issue. If that's the case, there may not be anything to do. It's important that the test covers fetching from real repositories, but unfortunately that implies a bit of flakiness.

dmitshur

comment created time in 11 days

issue commentgolang/go

x/exp: reconsider using single module for all unrelated experimental packages of various maturity levels

I'd argue that packages in golang.org/x/exp should be promoted to their own repos or to other repos like x/tools if people depend on them being stable. cmd/apidiff may fit that description.

@bcmills is working on a change that would reduce the cost of a large module graph (both in load time and stability). Placeholder issue is #36460. So this may not be a justification for splitting modules in the future.

In general, multi-module repos complicate development in a lot of ways, and I think we should avoid that if we can.

dmitshur

comment created time in 12 days

issue openedgolang/go

cmd/go: error not reported for packages in vendor/ in module mode, without vendoring

What version of Go are you using (go version)?

<pre> $ go version go version go1.13.7 darwin/amd64 </pre>

Does this issue reproduce with the latest release?

Yes, also reproduces with 1.14rc1.

What operating system and processor architecture are you using (go env)?

<details><summary><code>go env</code> Output</summary><br><pre> $ go env GO111MODULE="" GOARCH="amd64" GOBIN="" GOCACHE="/Users/jayconrod/Library/Caches/go-build" GOENV="/Users/jayconrod/Library/Application Support/go/env" GOEXE="" GOFLAGS="" GOHOSTARCH="amd64" GOHOSTOS="darwin" GONOPROXY="" GONOSUMDB="" GOOS="darwin" GOPATH="/Users/jayconrod/go" GOPRIVATE="" GOPROXY="direct" GOROOT="/opt/go/installed" GOSUMDB="sum.golang.org" GOTMPDIR="" GOTOOLDIR="/opt/go/installed/pkg/tool/darwin_amd64" GCCGO="gccgo" AR="ar" CC="clang" CXX="clang++" CGO_ENABLED="1" GOMOD="/Users/jayconrod/Code/test/go.mod" CGO_CFLAGS="-g -O2" CGO_CPPFLAGS="" CGO_CXXFLAGS="-g -O2" CGO_FFLAGS="-g -O2" CGO_LDFLAGS="-g -O2" PKG_CONFIG="pkg-config" GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/rq/x0692kqj6ml8cvrhcqh5bswc008xj1/T/go-build248121802=/tmp/go-build -gno-record-gcc-switches -fno-common" </pre></details>

What did you do?

Created a regular package in a directory named vendor, not created with go mod vendor, not using -mod=vendor.

! go list -deps example.com/m

-- go.mod --
module example.com/m

go 1.14
-- use.go --
package use

import (
	_ "example.com/m/vendor"
	_ "example.com/m/vendor/x"
)
-- vendor/vendor.go --
package vendor
-- vendor/x/x.go --
package x

What did you expect to see?

Errors reported for packages in vendor/ when -mod=vendor is not active.

These are being treated as regular packages. Other modules won't be able to import these packages, since vendor directories are excluded from module zip files.

What did you see instead?

go list treats these like regular packages:

$ go list -deps example.com/m
example.com/m/vendor
example.com/m/vendor/x
example.com/m

cc @matloob @bcmills

created time in 12 days

issue commentbazelbuild/rules_go

Concurrent packages.Load at build time

If Package.Types is needed for a cgo package, the C code needs to be at least partly compiled, which is very slow for some packages. That may be the case for anything related to SQLite.

Do you think it a good idea to implement a simple GOPACKAGESDRIVER that calls go/parser.ParseDir to avoid calling go list?

Adding a GOPACKAGESDRIVER implementation that could be run inside a Bazel action is #393. A lot of the work may be shared with #512.

The idea is that either GoCompilePkg or a separate action would produce a package description as JSON. A driver could then scoop those files up for relevant packages, fix up paths, then return that JSON to packages.Load.

linzhp

comment created time in 13 days

issue commentbazelbuild/rules_go

--test_output=errors output now also contains successfully passed tests, i.e. became too verbose...

@achew22 Sorry no, coverage is pretty separate.

Currently, you can test with bazel coverage and select targets to instrument using --instrumentation_filter (kind of like -coverpkg). The instrumentation mostly works, and the generated test main gathers coverage data and writes it to a file.

There are some missing pieces in Bazel though. There needs to a common format that all languages convert their coverage into. There needs to be a way for languages to provide tools to do that. There needs to be a way to merge multiple sets of coverage data. And Bazel needs to be able to display that information in a useful way. Blaze has some facilities for that, but I don't think they're open source, and they wouldn't serve all the needs of open source devs in any case.

There were some presentations on coverage at BazelCon 2018. I'm not sure what's new since then. The roadmap and the design doc are pretty far out of date.

robbertvanginkel

comment created time in 13 days

push eventjayconrod/goissues

Jay Conrod

commit sha 9ce6dc19889fb9f471e09310865277c11ecc24c3

fork and modify

view details

push time in 13 days

issue commentbazelbuild/rules_go

--test_output=errors output now also contains successfully passed tests, i.e. became too verbose...

Good point about keeping the wrapper for #1918. GO_TEST_WRAP_TESTV=1 probably makes more sense (or maybe renamed to GO_WRAP_TEST_XML=1).

If you're up for keeping -test.v and filtering the output sent to stdout so it looks like -test.v was not set, I'd be happy to accept a PR for that. Less configuration is great, but less complexity is good, too.

robbertvanginkel

comment created time in 13 days

issue commentbazelbuild/rules_go

Concurrent packages.Load at build time

This depends a lot on how packages.Load is invoked and whether the underlying packages have cgo code. But it doesn't seem too surprising to me. packages.Load runs go list internally (assuming GOPACKAGESDRIVER is not set). go list is usually either I/O limited or memory limited, depending on whether files are in the kernel cache. go list has some parallelism internally, too. Adding more cores probably won't help.

linzhp

comment created time in 13 days

fork jayconrod/goissues

utility to export golang/go issues to CSV

fork in 13 days

issue commentgolang/go

cmd/go: Build fails when build with -trimpath due to lack of gcc

Related #33840, #27303, #26988.

narqo

comment created time in 13 days

PR closed golang/tools

Reviewers
update documentation semver package cla: no

Fix the documentation of Compare func in semver package

+1 -1

4 comments

1 changed file

cohental

pr closed time in 13 days

pull request commentgolang/tools

update documentation semver package

@cohental Thanks for fixing this... but golang.org/x/tools/internal/semver is a copy of golang.org/x/mod/semver. The copy was made before golang.org/x/mod was a separate module, but we don't need it anymore. I've created CL 218897 to delete it, along with ./internal/module.

It looks like the typo was already fixed in golang.org/x/mod/semver, so no need to resubmit this change over there.

cohental

comment created time in 13 days

delete branch jayconrod/rules_go

delete branch : fix-buildifier

delete time in 13 days

push eventbazelbuild/rules_go

Jay Conrod

commit sha 10a3322e2718ae364e97fd06c814fbbc495caacc

Fix //tests:buildifier_test (#2365) Post-submit test failed because #2348 was merged after #2356 was tested.

view details

push time in 13 days

PR merged bazelbuild/rules_go

Reviewers
Fix //tests:buildifier_test cla: yes

Post-submit test failed because #2348 was merged after #2356 was tested.

+11 -8

0 comment

1 changed file

jayconrod

pr closed time in 13 days

more