profile
viewpoint

aclements/latexrun 442

A 21st century LaTeX wrapper

aclements/libelfin 175

C++11 ELF/DWARF parser

aclements/go-misc 145

Miscellaneous Go hacks

aclements/perflock 108

Locking wrapper for running benchmarks on shared hosts

aclements/go-perf 104

Go packages and tools for Linux perf

aclements/commuter 80

Automated multicore scalability testing tool

aclements/mtrace 73

Memory access tracing QEMU

aclements/cpubars 48

Lightweight terminal-based multicore CPU usage monitor

aclements/biblib 45

Simple, faithful BibTeX parser and algorithms for Python 3

aclements/go-moremath 44

An assortment of more specialized math routines for Go

issue commentgolang/go

cmd/cgo: jmethodID/jfieldID is not mapped to uintptr if building with the Android NDK [1.15 backport]

I've read over all the CLs and I'm happy with them. The one outstanding concern, I believe, is whether this breaks any existing cgo-using code. The tip commits are currently working their way through global testing inside Google. If they pass that without introducing new build failures, then I'm good with the cherry-pick.

gopherbot

comment created time in 4 days

issue commentgolang/go

cmd/link: "x86_64-w64-mingw32/bin/ld.exe: Error: export ordinal too large" after upgrading to Go 1.15

@hackf5 , could you elaborate on what's not working with -buildmode=exe?

Also /cc @thanm @jeremyfaller

bep

comment created time in 7 days

issue openedgolang/go

make.bash: spurious "supported GOARCH" warning

What version of Go are you using (go version)?

<pre> $ go version go version devel +b4ea672009 Mon Sep 21 01:30:48 2020 +0000 linux/amd64 </pre>

Does this issue reproduce with the latest release?

Yes. And with Go 1.14 (but not Go 1.13)

What operating system and processor architecture are you using (go env)?

<details><summary><code>go env</code> Output</summary><br><pre> $ go env GO111MODULE="" GOARCH="amd64" GOBIN="" GOCACHE="/home/austin/.cache/go-build" GOENV="/home/austin/.config/go/env" GOEXE="" GOFLAGS="" GOHOSTARCH="amd64" GOHOSTOS="linux" GOINSECURE="" GOMODCACHE="/home/austin/r/go/pkg/mod" GONOPROXY="" GONOSUMDB="" GOOS="linux" GOPATH="/home/austin/r/go" GOPRIVATE="" GOPROXY="https://proxy.golang.org,direct" GOROOT="/home/austin/go.dev" GOSUMDB="sum.golang.org" GOTMPDIR="" GOTOOLDIR="/home/austin/go.dev/pkg/tool/linux_amd64" GCCGO="gccgo" AR="ar" CC="gcc" CXX="g++" CGO_ENABLED="1" GOMOD="/home/austin/go.dev/src/go.mod" CGO_CFLAGS="-g -O2" CGO_CPPFLAGS="" CGO_CXXFLAGS="-g -O2" CGO_FFLAGS="-g -O2" CGO_LDFLAGS="-g -O2" PKG_CONFIG="pkg-config" GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build542776898=/tmp/go-build -gno-record-gcc-switches" </pre></details>

What did you do?

While using Go 1.4 as the bootstrap compiler: GOOS=linux GOARCH=arm64 ./make.bash

What did you expect to see?

Logs from a successful cross-compilation.

What did you see instead?

Before the successful cross-compilation, the following scary-sounding warning: 2020/09/21 09:05:27 unsupported GOARCH arm64

This output comes from make.bash invoking go version on the bootstrap compiler with the GOOS and GOARCH provided by the user. Since I'm using 1.4 as a bootstrap and it didn't support GOARCH=arm64 (or many other GOARCHes), the bootstrap compiler complains.

It looks like this was introduced in CL 204757 as a simple debugging aid to print the bootstrap compiler's version in the make.bash log. Probably the simplest thing to do is just clear GOOS and GOARCH on the go version invocation.

/cc @bradfitz @ianlancetaylor

created time in 7 days

issue commentgolang/go

runtime: "failed to fetch any source profiles" in TestTimePprof

It looks like the underlying problem here is that testprog TimeProf (the binary TestTimePprof runs) timed out. The "failed to fetch any source profiles" is a follow-on failure because the binary didn't exit. There's not much information about the time-out, unfortunately. We may want to add in a GOTRACEBACK=crash just to get more details.

bcmills

comment created time in 11 days

issue commentgolang/go

cmd/link: panic: operation not permitted

@ianlancetaylor , that's a good point. It wouldn't surprise me at all of fallocate in particular simply wasn't on the allow list.

fogfish

comment created time in 13 days

issue commentgolang/go

x/playground: falsely detects a deadlock when running Cmd.StderrPipe example code

Is it actually viable to run the playground in real time? I would love to drop faketime. Based on #30439 I thought there were reasons we couldn't just drop it, including the present tool.

hundt

comment created time in 13 days

issue commentgolang/go

cmd/link: panic: operation not permitted

@fogfish, thanks. You don't happen to know what file system it's running on, do you (or what your test instance from your initial report was running on)?

fogfish

comment created time in 13 days

issue commentgolang/go

cmd/link: panic: operation not permitted

@fogfish, it would be helpful to see an strace of the failing link. Could you run go build -work -x hw.go (-work will let you rerun build commands by hand), then copy-paste the WORK=... line at the top into your shell, then finally run strace -f -e mmap,open,openat <link command> with the failing link command and paste the output?

fogfish

comment created time in 13 days

issue commentgolang/go

cmd/link: panic: operation not permitted

Wait, backing up a moment, I don't see at all what this would have to do with CONFIG_STRICT_DEVMEM. We're definitely not mmaping /dev/mem.

fogfish

comment created time in 13 days

issue commentgolang/go

runtime/debug: FreeOSMemory() not working on Android 9

It's really disappointing if Android kills applications for memory use without first trying to clean up their freed pages... If that is the case, then I would consider MADV_FREE quite broken on Android and we shouldn't use it.

@YouROK, your example code is likely working as expected, but you mention "If you allocate memory several times, then the program is killed by the system". That sounds like the real issue here. Could you post code that reproduces that?

YouROK

comment created time in 25 days

issue commentgolang/go

proposal: cmd/go: build option to set env variable defaults

Sorry, but that doesn't really answer my question. Why do you need these options? How does it behave differently and why is it important that you change that?

networkimprov

comment created time in 25 days

issue commentgolang/go

proposal: cmd/go: build option to set env variable defaults

These are called GODEBUG because they're meant for debugging. If you have to set them programmatically, it suggests we've done something wrong (in the case of madvdontneed, maybe Linux did something wrong, but that's another story). It's possible we could put some of these in the runtime/debug package as @rsc suggested, but I'd like to understand better why you need to change them first.

networkimprov

comment created time in 25 days

issue commentgolang/go

cmd/compile: R12 can be clobbered for write barrier call on PPC64

GopherBot is lazy (#25574), so I'm afraid you'll have to open the 1.14 backport issue by hand.

cherrymui

comment created time in a month

issue commentgolang/go

cmd/compile: R12 can be clobbered for write barrier call on PPC64

@cherrymui , does this also need a 1.14 backport? I'm pretty sure it does.

cherrymui

comment created time in a month

issue commentgolang/go

proposal: switch to a register-based calling convention for Go functions

That's an excellent point. A few possible solutions come to mind:

  1. It would be really sad if we had to make more per-goroutine space for this since you almost always have space on the stack. One possibility is to spill to the stack if there's room, and otherwise we keep a pool of allocated "spill space" objects. For that, we could keep just one pre-allocated per P and if the goroutine needs it, it can pull it off the P, spill into it, and then once it's entered the runtime it can allocate a new one (probably getting it from a global allocation pool) and attach it to the P. On return, it can return it to the allocation pool.

  2. A similar option would be to have just one spill space per P. Again, use the stack if you can, otherwise spill into the P, enter the runtime, grow the stack even if it's just a preemption, and then dump the spilled registers back onto the stack in the right place.

  3. We could revisit the nosplit rules. It's often (though not always) a mistake to call from a nosplit function to a non-nosplit function. I'm not sure exactly what this would look like. Maybe a non-nosplit function called from a nosplit function has to have a special prologue? Related: https://github.com/golang/go/issues/21314#issuecomment-346480985. I think the only cases I enumerated in that comment that intentionally call from a nosplit to a non-nosplit function aren't running on a user G stack and thus can't grow it anyway.

aclements

comment created time in a month

issue commentgolang/go

runtime: pcdata is -2 and 12 locals stack map entries error on nil pointer [1.15 backport]

@randall77 , how do you feel about backporting this given the relatively high complexity of the CL? Is there a simpler fix that might be safer to backport?

randall77

comment created time in a month

issue commentgolang/go

runtime: can't atomic access of first word of tiny-allocated struct on 32-bit architecture

Given that #36606 is stalled, I think we need to revisit this. It seems like we need to special-case 12 byte allocations on 32-bit to by 8-byte aligned. Are there any other solutions on the table?

NewbMiao

comment created time in a month

issue commentgolang/go

proposal: switch to a register-based calling convention for Go functions

@laboger , yes, we're considering special-casing just a few assembly functions. The hottest ones by far are memmove, memclrNoHeapPointers, and possibly syscall.Syscall. The write barrier calling convention is very specialized because its context is so unusual and performance-sensitive. We'll probably just switch the others to use the register ABI rather than anything more specialized (and teach the toolchain that those are special even though they're assembly).

aclements

comment created time in a month

issue openedgolang/go

proposal: switch to a register-based calling convention for Go functions

I propose that we switch the Go internal ABI (used between Go functions) from stack-based to register-based argument and result passing for Go 1.16.

I lay out the details of our proposal and the work required in this document (placeholder).

This was previously proposed in #18597, where @dr2chase did some excellent prototyping work that our new proposal builds on. With multiple ABI support, we’re now in a much better position to execute on this without breaking compatibility with the existing body of Go assembly code. I’ve opened this new issue to focus on discussion of our new proposal.

/cc @dr2chase @danscales @thanm @cherrymui @mknyszek @prattmic @randall77

created time in 2 months

issue openedgolang/go

cmd/link: merge dev.link into master branch for Go 1.16 early

This is a tracking issue for merging the dev.link branch into master for Go 1.16. A great deal of work has happened on dev.link to further improve the linker since Go 1.15, and we plan to merge this as soon as the 1.16 tree opens in order to avoid conflicts. At this point, we're wrapping up a few final changes and don't expect any more major changes to the linker in the near term.

While this is a large change, we propose to do the merge without a feature flag. Instead, we believe the following are sufficient risk mitigation:

  1. By merging as soon as Go 1.16 opens and not making further major changes to the linker, the linker changes will have maximal soak time. The changes are involved enough that we believe fixing any new bugs at this point would be preferable to backing them out, and maximizing the soak time also maximizes the time we have to fix bugs. This soak time will include regular testing across Google's code base from the start.

  2. Failures caused by the linker tend to be deterministic and are often not subtle, so testing is actually quite effective (in contrast with, say, runtime bugs, which tend to be non-deterministic and subtle). We theorized testing would be effective with the linker changes that went into 1.15 and, indeed, there were very few bugs and they were all relatively straightforward to debug.

  3. Since this work has been happening on a branch, we've been able to monitor the build dashboard throughout development. Hence, the linker changes have already had significant automated testing, and we've been able to keep the branch quite stable throughout.

In contrast, we believe attempting to feature flag this work would be so involved as to actually increase risk and add significant testing burden, as there's no "obviously correct" fallback (unlike how Go 1.15 could fall back to the 1.14 linker).

/cc @rsc @andybons @jeremyfaller @cherrymui @thanm

created time in 2 months

issue commentgolang/go

cmd/link: internal linker fails to build openshift/installer with "program too large, call relocation distance" and segfaults on linux/arm64

Thanks for the follow-up. Since 1.15 is just about out the door at this point, I'll move this to 1.16.

jcajka

comment created time in 2 months

issue commentgolang/go

runtime: fatal error: checkdead: runnable g [1.13 backport]

I agree with your assessment that the possibility of this bug in 1.13 seems so astronomically small that I'm not sure it's worth fixing, but I'll leave the final decision to the release team.

dmitshur

comment created time in 2 months

issue commentgolang/go

runtime: merge workarounds to false deadlocks

With your change to startm to reserve an ID while holding sched.lock, is the logic in sysmon even necessary?

prattmic

comment created time in 2 months

issue commentgolang/go

cmd/cgo: cgo parameter type mismatch between generated go wrapper and call to wrapper

Bisect says this was CL 230037, "cmd/cgo: use type aliases for #define type macros". cc @mdempsky

somersf

comment created time in 2 months

issue commentgolang/go

cmd/link: internal linker fails to build openshift/installer with "program too large, call relocation distance" and segfaults on linux/arm64

Note that this was also present in 1.14 and 1.13, so this isn't a regression. I've tentatively put this in the 1.15 milestone, but we probably won't fix this for the initial release (it may make a point release).

jcajka

comment created time in 2 months

issue commentgolang/go

cmd/link: internal linker fails to build openshift/installer with "program too large, call relocation distance" and segfaults on linux/arm64

Also cc @jeremyfaller .

Looks like we may have a trampoline insertion issue?

jcajka

comment created time in 2 months

issue commentgolang/go

runtime: GC: heap idle is not released to linux

I think there are too many things going on in this issue.

As to the original issue, @Anteoy , HeapIdle is expected to increase monotonically. It's not measuring OS memory held by Go. The documentation says:

	// Idle spans have no objects in them. These spans could be
	// (and may already have been) returned to the OS, or they can
	// be reused for heap allocations, or they can be reused as
	// stack memory.

Note the "could be returned to the OS". As @katiehockman pointed out, HeapReleased is the subset of HeapIdle that has been released to the OS. (I realize having all of these MemStats that are subsets of other MemStats is pretty annoying. We're working on a new API that will hopefully be much better structured.)

Everyone else is seeing a completely different issue. @CAFxX summarized it very well. Go uses MADV_FREE if the kernel supports it to release memory because it's dramatically more efficient than MADV_DONTNEED. A lot of modern memory managers do this. This has the rather annoying consequence that RSS doesn't go down unless the OS is actually under memory pressure (I'm not sure why Linux chose to do it this way). As @CAFxX mentioned, you can subtract LazyFree from the RSS to account for this, though top/htop/etc don't do this. Container memory limits do account for this, so if your process is OOMing, it's because it's just using too much memory, and not related to MADV_FREE.

Anteoy

comment created time in 2 months

push eventaclements/libelfin

Jonas Witschel

commit sha 946dde57e5caef8297b9339f3a7971401d540840

Add DESTDIR support For packaging it is useful to be able to install to a custom temporary installation location. The existing PREFIX variable is not enough for this purpose because it will lead to incorrect pkg-config files (e.g. setting PREFIX="$pkgdir/usr" leads to includedir=$pkgdir/usr/include while it should be includedir=/usr/include instead). DESTDIR is the standard name for this variable, defined by the GNU coding standards.

view details

push time in 2 months

PR merged aclements/libelfin

Add DESTDIR support

For packaging it is useful to be able to install to a custom temporary installation location. The existing PREFIX variable is not enough for this purpose because it will lead to incorrect pkg-config files (e.g. setting PREFIX="$pkgdir/usr" leads to includedir=$pkgdir/usr/include while it should be includedir=/usr/include instead). DESTDIR is the standard name for this variable, defined by the GNU coding standards.

Fixes: #35 Also see: Arch Linux FS#64881

+14 -14

0 comment

2 changed files

diabonas

pr closed time in 2 months

issue closedaclements/libelfin

DESTDIR support in make install

Setting custom install location independent from $PREFIX should be implemented. For example, I want make DESTDIR=/tmp/pkg PREFIX=/usr install to install libraries and headers to /tmp/pkg/usr, but write libdir and includedir entries to pkg-config files based only on $PREFIX (like $PREFIX/lib and $PREFIX/include, but not $DESTDIR/$PREFIX/whatever).

The main reason for it is packaging. When you are creating a package you often need to install the program or library to a temporary directory, which then will be archived and will go to the package. Simply setting PREFIX doesn't help because it will make pkg-config entries to look like this (incorrect): /path/to/pkgdir/usr/include instead of this (correct): /usr/include. You can see it in detail in this terminal transcript: https://gist.github.com/kodo-pp/482bef5afbece8bdb3c01bac461b1e03

closed time in 2 months

kodo-pp

pull request commentaclements/libelfin

CMake building and Initial REL(A) support

Hi @deadly-platypus . Unfortunately, with all of the code reformatting I can't really review or submit this. Please just follow the style of the existing code, which should be pretty consistent (though I didn't run a formatting tool, so I'm sure there are minor inconsistencies).

I'm curious about the CMake support you added, since this is the second PR to propose that. What's the benefit of adding CMake support? Make is certainly not the best thing in the world, but the existing makefiles work and are pretty simple. (I've never used CMake myself, so perhaps the answer is obvious to someone more familiar with it.)

deadly-platypus

comment created time in 2 months

Pull request review commentaclements/libelfin

Safer sections iteration

 elf::elf(const std::shared_ptr<loader> &l)         // Load sections         const void *sec_data = l->load(m->hdr.shoff,                                        m->hdr.shentsize * m->hdr.shnum);+        if (NULL == sec_data) {

loader.load is not allowed to return NULL (it's supposed to throw an exception if there's a problem) and the mmap loader implementation doesn't return NULL, so I'm afraid I don't understand what this change is accomplishing.

assafnativ

comment created time in 2 months

Pull request review commentaclements/libelfin

Safer sections iteration

 elf::elf(const std::shared_ptr<loader> &l)         // Load sections         const void *sec_data = l->load(m->hdr.shoff,                                        m->hdr.shentsize * m->hdr.shnum);+        if (NULL == sec_data) {

Please follow the coding conventions of the surrounding code, which uses forward comparison: sec_data == NULL.

assafnativ

comment created time in 2 months

Pull request review commentaclements/libelfin

Safer sections iteration

 elf::elf(const std::shared_ptr<loader> &l)         // Load sections         const void *sec_data = l->load(m->hdr.shoff,                                        m->hdr.shentsize * m->hdr.shnum);+        if (NULL == sec_data) {+            return;+        }         for (unsigned i = 0; i < m->hdr.shnum; i++) {                 const void *sec = ((const char*)sec_data) + i * m->hdr.shentsize;+                if (NULL == sec) {

It doesn't make sense for sec to be NULL here, since it's a result from a pointer offset from sec_data. In practice the only way this can happen is if sec_data itself is NULL, but then we shouldn't have even done the pointer arithmetic.

assafnativ

comment created time in 2 months

PR merged aclements/libelfin

fix data initialization of elf segment

Method segment::data depends on proper initialization of data field. Also this commit deletes commented code.

+1 -3

0 comment

1 changed file

abenkhadra

pr closed time in 2 months

push eventaclements/libelfin

Ammar Ben Khadra

commit sha 9d0db16d0a0b3c4f8aaa60a3e4dab295df34b6b2

fix data initialization of elf segment Method segment::data depends on proper initialization of data field. Also this commit deletes commented code.

view details

push time in 2 months

pull request commentaclements/libelfin

Fix line_table::find_address when the address is the last entry

I'm not sure how to interpret that line number table dump you pasted. It looks like it's already been rather processed, which makes it hard to tell what's going on at a low level. The "ET" on the last line suggests there is an end_sequence. Can you post a more raw dump, for example from objdump --dwarf=rawline?

boyou

comment created time in 2 months

push eventaclements/libelfin

Frank Tetzel

commit sha 74f11163334653866c08c32430bbad33c1a7877f

ignore dwarf extensions of gcc in the line table gcc 9 uses DWARF 4 by default with vendor extensions in the line table which results in a thrown exception. An application using libelfin cannot recover from an exception and still get a reasonable line table. Now, when a vendor extension is encountered, the type is set to invalid to get around the issue.

view details

push time in 2 months

PR merged aclements/libelfin

ignore dwarf extensions of gcc in the line table

gcc 9 uses DWARF 4 by default with vendor extensions in the line table which results in a thrown exception. An application using libelfin cannot recover from an exception and still get a reasonable line table. Now, when a vendor extension is encountered, the type is set to invalid to get around the issue.

Fixes https://github.com/aclements/libelfin/issues/39 for me.

+4 -0

2 comments

1 changed file

tetzank

pr closed time in 2 months

issue closedaclements/libelfin

coz: DW_FORM_sec_offset not expected for attribute (DW_AT)0x2137 / 0x2119

coz, the causal profiler, uses libelfin to parse dwarf data, and multiple users have stumbled upon the error message in the title (https://github.com/plasma-umass/coz/issues/107)

closed time in 2 months

timo

issue commentgolang/go

cmd/link: -buildmode=pie -linkshared generates incorrect code

@kiap , just the clarify, can you state the exact problem you're seeing more clearly? It's not clear from your original bug report. Is it specifically that the shared linkage executable is significantly larger than it used to be, indicating the linker included a lot of things it shouldn't have? Are you seeing problems besides the size? (When you say it "generates incorrect code", that suggests a crash or panic when you try to run it.)

kiap

comment created time in 2 months

push eventaclements/.emacs.d

Austin Clements

commit sha 868d9517c825ace61e6973a7238c50de94506b4c

add color-completion mode

view details

push time in 2 months

push eventaclements/my-dotfiles

Austin Clements

commit sha 3a1858ea5915b314602b59ced9d8d2c78d7e96cb

aptitude: change display

view details

Austin Clements

commit sha 5157834be2329db97aff23a4212fb962b3ce6a1e

ssh: updates on awk

view details

Austin Clements

commit sha c8c551f0bb6c15ec1b97d5e51e4fd7c1604c459f

archive old stuff and clean up

view details

push time in 2 months

push eventaclements/my-dotfiles

Austin Clements

commit sha d9c91fd004380413ee46fc738ad864415e461391

Xresources: tons of little changes

view details

Austin Clements

commit sha e785700feea55ab1d6a09bad562865eb47b5d25d

zgoogle: gcloud updates

view details

Austin Clements

commit sha 13420c221b9a7d876718536726538a9d29d2ed26

gdb: updates

view details

push time in 2 months

push eventaclements/my-bin

Austin Clements

commit sha 79c3ddb56dd70484499bd20ce5a0dcb0edc52029

g: more excludes

view details

push time in 2 months

push eventaclements/my-bin

Austin Clements

commit sha acf550f2ef88ef09bf50f31aa6eecec24430833f

big cleanup of old and unused commands

view details

push time in 2 months

push eventaclements/my-bin

Austin Clements

commit sha d5bf06f0c5b97353debe48b9b6d985998cdffa4b

awkscreen: awkjr -> awk

view details

push time in 2 months

issue commentgolang/go

runtime: fatal error: checkdead: runnable g

cc @prattmic

@slon, can you try reproducing with GODEBUG=asyncpreemptoff=1?

slon

comment created time in 2 months

issue commentgolang/go

proposal: all: drop GO386=387 support in Go 1.16

I've gone ahead and submitted the deprecation notice to the 1.15 release notes. However, this issue is still in proposal review. If it does get declined, we can revert the CL. I've asked for this to get reviewed in next week's proposal review meeting.

aclements

comment created time in 2 months

issue commentgolang/go

cmd/link: linking github.com/hashicorp/packer takes >2gb of ram

Thanks for closing the loop on this!

lyda

comment created time in 2 months

issue commentgolang/go

proposal: drop GO386=387 support in Go 1.16

We should wait a little longer for public comment, but I've gone ahead and prepared the pre-announcement for the release notes. (We could also submit the release notes change and roll it back if necessary.)

Nick Craig-Wood pointed out on golang-nuts that we should make sure GO386=387 fails with a useful error message if we do drop it (rather than, say, starting to ignore the GO386 environment variable entirely).

aclements

comment created time in 2 months

issue openedgolang/go

proposal: drop GO386=387 support in Go 1.16

I propose dropping 387 floating-point support and requiring SSE2 support for GOARCH=386 in the native gc compiler for Go 1.16. This would raise the minimum GOARCH=386 requirement to the Intel Pentium 4 (released in 2000) or AMD Opteron/Athlon 64 (released in 2003).

We would need to pre-announce this in the 1.15 release notes. (I've put it in the 1.15 milestone for this reason.)

There are several benefits to doing this:

  1. While 387 support isn’t a huge maintenance burden, it does take time away from performance and feature work and represents a fair amount of latent complexity.
  2. 387 support has been a regular source of bugs (#36400, #27516, #22429, #17357, #13923, #12970, #4798, just to name a few).
  3. 387 bugs often go undetected for a long time because we don’t have builders that support only 387 (so unsupported instructions can slip in unnoticed).
  4. Raising the minimum requirement to SSE2 would allow us to also assume many other useful architectural features, such as proper memory fences and 128 bit registers, which would simplify the compiler and runtime and allow for much more efficient implementations of core functions like memmove on 386.
  5. We’re exploring switching to a register-based calling convention in Go 1.16, which promises significant performance improvements, but retaining 387 support will definitely complicate this and slow our progress.

The gccgo toolchain will continue to support 387 floating-point, so this remains an option for projects that absolutely must use 387 floating-point.

I proposed this on golang-nuts to get feedback on significant uses of GO386=387, particularly for which using gccgo would not be a viable option.

created time in 2 months

issue commentgolang/go

runtime: missing deferreturn on linux/ppc64le

@runlevel5 , Go 1.14.5 was a security release, so it did not include the fix for this. This should be fixed in the next non-security release (which is very likely to be 1.14.6 and I think should be out soon).

4a6f656c

comment created time in 2 months

issue commentgolang/go

runtime: remove the mlock hack in Go 1.15

There's one deciding factor here: is the kernel patch widespread enough that the workaround is no longer necessary? I'm sure all the distros have it, but people aren't necessarily on top of upgrading their kernels. Unfortunately, I have no idea how to figure out the answer to this, and I think we should be fairly conservative about this given the nature of the kernel bug.

DanielShaulov

comment created time in 2 months

issue commentgolang/go

runtime: remove the mlock hack in Go 1.15

@nemith , can you explain why you need to patch the compiler? As of Go 1.14.1, the only effect of an mlock failure is an additional message if there's an uncaught panic, so I don't understand why that requires patching.

DanielShaulov

comment created time in 2 months

issue commentgolang/go

runtime: consider adding 24-byte size class

Neat! FYI, I already have a CL (not yet mailed) to move checkmarks out of the bitmap and was planning to send it for 1.16, precisely because they make everything with the bitmap annoying. :) I could send that now and you could rebase on top of it if that would be helpful?

dvyukov

comment created time in 2 months

issue commentgolang/go

runtime: windows system clock resolution issue

I just reviewed this issue from the top and my understanding (which hasn’t really changed since December) is that this remains blocked on implementing UMS for blocked syscall detection (#7876). Once we’ve done that, we believe the runtime itself will no longer depend on the high timer frequency, and we can drop timeBeginPeriod from the runtime (and probably expose them for applications that need them).

Unfortunately, we have very little Windows expertise on the runtime team at Google (in a somewhat ridiculous twist, our Windows laptop is in a drawer in my office, which remains inaccessible for the foreseeable future) and it’s difficult to solicit complex changes like this from open source contributors (though that would be extremely welcome!). I’m looking for solutions to this.

Ian suggested using entersyscallblock for all syscalls on Windows, though this would come at a performance penalty for all system calls. I don’t believe this has been tried.

History up to now

The call to timeBeginPeriod was added in 2011 as part of CPU profiling support. However, the runtime had several tight sleep loops and, as a result, increasing the timer frequency became important to the performance of the runtime and hence many Go programs.

In 2015, we'd eliminated nearly all of these sleep loops and it appeared the runtime no longer depended on higher timer frequency, so the frequency adjustment was removed. Unfortunately, it turned out that some benchmarks still suffered (#14790), so timeBeginPeriod was added back. @jstarks's analysis showed that this was because of the sysmon loop that detects blocked system calls. Hence, @alexbrainman suggested that the way to resolve this is to resolve #7876 by switching to user-mode scheduling to detect blocked system calls.

We made some progress on this in 2017 by reducing the timer frequency when the whole process goes idle. This improved the situation, but is far from ideal because the timer frequency is still regularly raised, causing battery drain and interfering with testing of other applications that also need to manipulate the timer frequency (e.g., Chrome).

However, the fundamental issue with detecting blocked system calls in the runtime that led to #14790 remains. Windows UMS is the right way to fix this (indeed, I wish other OSes had an equivalent to UMS; our blocked system call detection just targets the lowest common denominator right now). It seems likely that UMS support will obviate the need for a high timer frequency in the Go runtime itself.

defia

comment created time in 3 months

issue commentgolang/go

runtime: idle GC interferes with auto-scaling

Limiting idle GC is certainly a possibility. In the past we've talked about letting one P enter scheduler idle even if GC is running so that there's a P to quickly wake up and also to block in netpoll (though if you do wake it up, do you have to shut down idle GC running on another P to keep a spare P?).

It's actually been a very long time since we've measured the scalability of the garbage collector. It certainly has limits. I suspect they're high enough that they're significantly higher than a mostly-idle latency-sensitive application is likely to use on its own, but at least it would cap out the feedback cycle.

aclements

comment created time in 3 months

issue commentgolang/go

runtime: idle GC interferes with auto-scaling

I see, IIUC what you mean then #14812 (comment) (GC causing ~100ms latency spikes when running in containers with CPU limits) can also be considered as an additional argument in favor of fixing this.

@CAFxX , yep! (Thanks for that cross-reference, I've added it to the top post.)

aclements

comment created time in 3 months

issue commentgolang/go

runtime: idle GC interferes with auto-scaling

Oops.

I want to clarify that I didn't mean the two minute GC. That also has some downsides, but I think the benefits outweigh the costs.

I was talking about how the garbage collector schedules GC work during idle time on every GC cycle. This is what interferes with autoscaling.

aclements

comment created time in 3 months

issue openedgolang/go

runtime: idle GC interferes with auto-scaling

What version of Go are you using (go version)?

<pre> $ go version go version go1.14 linux/amd64 </pre>

But this has been true for years.

What did you do?

Run a mostly-idle application in a container with CPU limits under a mechanism that monitors CPU use and increases the job's CPU reservation if it appears to be too little. Specifically, this was observed with a latency-sensitive policy that looked at high percentile usage sampled over a short time period.

What did you expect to see?

Since the application is mostly idle, a small CPU reservation should be adequate and the auto-scaler should not need to grow that reservation.

What did you see instead?

Because the garbage collector attempts to use any idle cores up to GOMAXPROCS, even an otherwise mostly idle application will see periodic spikes of CPU activity. These will happen at least every 2 minutes if not more frequently. In this case, the auto-scaler's policy was sensitive enough that these spikes caused it to grow the job's CPU reservation. However, then the garbage collector uses all of the new CPU reservation. This leads to a feedback cycle where the auto-scaler continually grows the reservation.

See also #17969.

/cc @mknyszek

Thoughts

We've been thinking for a while now that idle GC may be more trouble than it's worth. The idea was to speed up the mark phase, so the write barrier is on for less time and we allocate black for less time. However, if an application is mostly idle, then it's not very sensitive to the (very small) performance impact of the write barrier and probably doesn't produce much floating garbage from allocate-black; and if an application isn't mostly idle, then it's not benefiting from idle GC anyway.

created time in 3 months

issue commentgolang/go

cmd/link: incorrect GC bitmap when global's type is in another shared object [1.14 backport]

Rationale for backport: This causes the garbage collector to incorrectly free reachable objects when using shared build mode, leading to memory corruption and crashes. There is no workaround (other than not using shared build mode).

gopherbot

comment created time in 3 months

push eventaclements/objbrowse

Austin Clements

commit sha 167e28ee777f0b1bb023cae1af06b24c1e68ae7f

missing git add

view details

push time in 3 months

push eventaclements/objbrowse

Austin Clements

commit sha beac01749f1fdd04fa109f91081361e1c2e4beb6

make Ctrl+F focus the symbol search box Since the page is populated dynamically, the browser search doesn't work.

view details

Austin Clements

commit sha 8277c4b0c13fa9ec30f0d101dd8362554e688cd2

handle relocations and display them in hex view For a seemingly small change, this basically required a complete redesign of the helper packages. This adds a generalized relocation representation. Data loaded from an object is now always represented by a byte slice *and* a sequence of relocations. Since relocations often target the dynamic symbol table, the ELF loader now reads in both symbol tables and combines them. This updates the hex view to display relocations at the positions where they apply. In the process, this fixes a bunch of other relocated things: - It introduces a global "symbol ID" taken from all combined symbol tables. This lets the generic relocation representation reference a symbol using a single ID, which can then easily be passed back in to an object file's methods to get data on that symbol. - Sequences of symbols and relocs are now both represented by sequence interfaces that allow for lazy decoding into the common representation. - Symbol size synthesis is now much more sophisticated. It handles symbols across different sections correctly, multiple symbols at the same base address, and symbols without addresses. - This fixes reading section data from SHT_NOBITS sections. - symtab.Table no longer mutates the order of the symbols slice. This is important for using symbol IDs. - symtab.Table now returns symbol IDs. The caller can turn this into a symbol.

view details

Austin Clements

commit sha 3450f1d982dc9d7dcdac327346175abe9736a32c

internal/symtab: improve handling of address-less symbols The symtab package had ad hoc handling of symbols without addresses from a while ago. The object loaders now report whether a symbol has a meaningful address or not. Dramatically simplify the address lookup logic.

view details

push time in 3 months

issue commentgolang/go

cmd/link: incorrect GC bitmap when global's type is in another shared object

@gopherbot, please backport to Go 1.14 and 1.15.

aclements

comment created time in 3 months

IssuesEvent

issue commentgolang/go

cmd/link: incorrect GC bitmap when global's type is in another shared object

Change reverted because it broke ppc64le.

aclements

comment created time in 3 months

issue commentgolang/go

cmd/link: incorrect GC bitmap when global's type is in another shared object

For the secondary issue, CL 215997 made decodetypeGcprogShlib return 0 on ARM64, where previously it would return the addend of the relocation on the gcdata field (these relocations were collected by ldshlibsyms). However, while returning 0 is definitely wrong, we no longer need any special case for this on ARM64 because this addend now gets applied in place in the shared object, meaning the contents of the gcdata field are already the correct pointer on ARM64, just like on all the other platforms.

aclements

comment created time in 3 months

issue commentgolang/go

cmd/link: incorrect GC bitmap when global's type is in another shared object

This happens when the linker is building the gcprog for the globals. It calls decodetypeGcmask (or decodetypeGcprog) for the global's type. Since the type is declared in another shared object, it has symbol type sym.SDYNIMPORT, so decodetypeGcmask attempts to read it out of the right section in the shared object. However, the shared object has already been closed, so sect.ReadAt returns a "file already closed error". decodetypeGcmask ignores this error and simply returns the freshly-allocated, zeroed bitmap, indicating that the type has no pointers.

There's a secondary issue on ARM64 on master where decodetypeGcprogShlib returns 0, which causes findShlibSection to return the NULL section (which has address 0 and length 0), causing decodetypeGcmask to look in the wrong section, in addition to trying to read from a closed file.

aclements

comment created time in 3 months

issue openedgolang/go

cmd/link: incorrect GC bitmap when global's type is in another shared object

What version of Go are you using (go version)?

<pre> $ go version go version go1.14 linux/amd64 </pre>

Also reproduced on master and as far back at Go 1.10.

Does this issue reproduce with the latest release?

Yes.

What did you do?

When linking in shared mode, if the type of a global variable is imported from a package in a different shared object, the global variable is marked as containing no pointers, which can cause the garbage collector to free reachable memory.

Here's an example from @cherrymui:

p.go

package p

type T [10]*int

main.go

package main

import "p"
import "runtime"

var x p.T
var y [10]*uint

func main() {
	for i := range x {
		x[i] = new(int)
		*x[i] = 12345
	}
	runtime.GC()
	runtime.GC()
	runtime.GC()
	for i := range y {
		y[i] = new(uint)
		*y[i] = 54321
	}
	runtime.GC()
	runtime.GC()
	runtime.GC()
	for i := range x {
		println(x[i], *x[i], y[i], *y[i])
	}
}
$ GOPATH=/tmp go install -buildmode=shared runtime sync/atomic
$ GOPATH=/tmp go install -buildmode=shared -linkshared p
$ GOPATH=/tmp go build -linkshared main
$ ./main
0xc00004a080 54321 0xc00004a080 54321
0xc00004a088 54321 0xc00004a088 54321
0xc00004a090 54321 0xc00004a090 54321
0xc00004a098 54321 0xc00004a098 54321
0xc00004a0a0 54321 0xc00004a0a0 54321
0xc00004a0a8 54321 0xc00004a0a8 54321
0xc00004a0b0 54321 0xc00004a0b0 54321
0xc00004a0b8 54321 0xc00004a0b8 54321
0xc00004a0c0 54321 0xc00004a0c0 54321
0xc00004a0c8 54321 0xc00004a0c8 54321

What did you expect to see?

In the example program, the garbage collector should not have freed the objects reachable from x, so it should have outputted:

0xc00004a080 12345 0xc00004a080 54321

What did you see instead?

The garbage collector incorrectly freed the objects reachable from x because its type was declared in a package in a different shared object.

/cc @cherrymui @jeremyfaller @thanm

created time in 3 months

more