profile
viewpoint

dvyukov/go-fuzz 3312

Randomized testing for Go

google/starlark-go 864

Starlark in Go: the Starlark configuration language, implemented in Go

dvyukov/go-fuzz-corpus 65

Corpus for github.com/dvyukov/go-fuzz examples

josharian/2015-oscon-go-perf-tutorial 19

Slides and code for Go performance tutorial given at 2015 OSCON

josharian/dont 11

Don't: template-based, decentralized static analysis for Go

josharian/compilecmp 7

automate Go compiler comparisons

josharian/countselectcases 3

Gather distribution information about select cases usage in Go (quick and dirty)

josharian/benchserve 2

benchserve turns a test binary into a benchmark server

josharian/compilebench 1

Compiler benchmarking

issue commentgolang/go

Unreachable "break" in switch statement causes erroneous compiler static analysis

Oh right, because vet needs the code to typechecking before it runs.

fsufitch

comment created time in a day

issue commentgolang/go

Unreachable "break" in switch statement causes erroneous compiler static analysis

Also, vet should catch this.

fsufitch

comment created time in a day

issue commentgolang/go

cmd/compile/internal/ssa: occasional "exit status 541111823" on windows-386-2008 builder

Alex, do you know what that exit code means? Googling yields no results.

Possibly related: https://github.com/golang/go/issues/36309. The problem there appeared to have been compiling the test; the changes that I just submitted over the last day or so fixed it in that issue by reducing memory required to build package ssa.

If it keeps happening, it’d be useful to know what’s happening when it fails—compiling the test, running the test (which one), etc.

bcmills

comment created time in a day

issue commentgolang/go

cmd/compile: consider dropping Greater* and Geq* generic ops

It doesn't seem to affect the generated code very much

If you want to know more precisely, try updating to the latest compilecmp and run

compilecmp -fn=changed -cl=220417 -platforms=arch

or for just (likely) regressions:

compilecmp -fn=bigger -cl=220417 -platforms=arch

And to investigate a particular function:

compilecmp -dumpssa=FUNCNAME -cl=220417 -platforms=arch

(or whatever particular platform you want to dig into, e.g. -platforms=js/wasm.

mundaym

comment created time in 2 days

IssuesEvent

issue commentgolang/go

cmd/compile: merging shifts into loads/stores isn't complete

I write in the CL “may fix” this issue. GitHub ignored the first word. :P

Leaving to @randall77 to decide what else needs to happen to declare this fixed.

randall77

comment created time in 2 days

issue commentgolang/go

cmd/go: confusing cache behavior for GOSSAFUNC

Possibly related: https://github.com/golang/go/issues/31290

cherrymui

comment created time in 2 days

issue commentgolang/go

runtime: TestSignalIgnoreSIGTRAP flaky on OpenBSD

Still happening. https://build.golang.org/log/aba4cbaa65bffbf86104babb1a8b58dd8c999cd8

bradfitz

comment created time in 2 days

issue commentgolang/go

cmd/go: TestExecutableGOROOT is flaky

I vaguely recall that a few years ago several of us undertook an effort to speed up all.bash. That included sprinkling a bunch of t.Parallel calls. Several of those had to get backed out, particularly in the runtime and cmd/go, because they caused flakiness. In some cases it was scheduling delays, in others assumptions that no one else shared resources we were tracking (fds), in others it was mutating global testing state like hooks.

Which doesn’t shed much light in this case; it is just to say “just adding t.Parallel” is known to be a common cause of flakiness. :)

josharian

comment created time in 2 days

issue commentgolang/go

cmd/compile: consider dropping Greater* and Geq* generic ops

I don’t have much of an opinion about parsimony. But if we don’t want to go that route, we could also decide to canonlicalize during rewrites, the way we do with the many lsh/rsh widths.

cc @dr2chase

mundaym

comment created time in 2 days

issue closedgolang/go

net/http: TestDontCacheBrokenHTTP2Conn flakes

Several instances on the dashboard. One example:

https://build.golang.org/log/1aa3681cf8816e6e95b24dddb3a9a47922fbc235

cc @bcmills @matloob although I don’t see how either of your changes could have caused this

closed time in 3 days

josharian

issue openedgolang/go

net/http: TestDontCacheBrokenHTTP2Conn flakes

Several instances on the dashboard. One example:

https://build.golang.org/log/1aa3681cf8816e6e95b24dddb3a9a47922fbc235

cc @bcmills @matloob although I don’t see how either of your changes could have caused this

created time in 3 days

issue openedgolang/go

cmd/go: TestExecutableGOROOT flake

Noticed on the build dashboard today:

https://build.golang.org/log/b1c4550d9629753e0fc2909f97f5610c385b0846

@matloob

created time in 3 days

issue openedgolang/go

cmd/go: TestCPUProfileMultithreaded flame

Noticed on the build dashboard today:

https://build.golang.org/log/6dc0c321d3320a47d39c96823b81db392018371c

cc @matloob

created time in 3 days

issue commentgolang/go

cmd/compile: add a dark mode for GOSSAFUNC html output

Awesome. Ask if you have questions. One item of note: It is an explicit goal that ssa.html is a single, standalone file—so go ahead and slap all that css and goo right into the html head. :)

agnivade

comment created time in 3 days

issue commentdvyukov/go-fuzz

Q: Debugging abysmal execution numbers

seems like the individual Fuzz calls are too quick and none of the profile files contain any information.

Bummer. I’ll see about adding proper profiling support to go-fuzz. May take a few days.

In the meantime, if Dmitry’s suggestions don’t help, you can try caveman profiling: start a large number of procs, use killall to send SIGQUIT to all of the them, and look for commonalities in the backtraces. If a single function dominates the runtime, it’ll also dominate randomly selected backtraces.

HerrSpace

comment created time in 3 days

issue commentgolang/go

proposal: Go 2: permit converting a string constant to a byte array type

@jfcg I think he was serious, yes. Please be polite. Although you may disagree, it is not an unreasonable position; the bar for language changes is very high.

jfcg

comment created time in 3 days

issue commentgolang/go

Invalid handling for infinity point on elliptic curves

cc @FiloSottile

catenacyber

comment created time in 3 days

issue commentgolang/go

x/crypto/ssh: support RSA SHA-2 host key signatures

cc @hanwen @FiloSottile

hansnielsen

comment created time in 4 days

issue commentgolang/go

cmd/compile: combine int tests when comparing against a constant string

Overlapping loads is a good idea, thanks!

josharian

comment created time in 4 days

issue commentgolang/go

cmd/compile: cant atomic access of first word of allocated struct occasionally?

@mknyszek would that make the tiny allocator much less efficient, by increasing the number and size of gaps? And if so, are there mitigations?

NewbMiao

comment created time in 4 days

issue commentdvyukov/go-fuzz

Q: Debugging abysmal execution numbers

Does the lexer emit a bunch of logging?

If you seed the corpus with a bunch of legitimate inputs, does fuzzing run faster (in terms of execs/sec)?

Does the lexer get much slower with longer inputs?

If you bump up/down the -procs arg to go-fuzz a lot, what impact does it have on execution speed? (Be sure to wait past the first warmup period of maybe 30s or so.)

Re: pprof from within fuzzing, I think so, mosty, with some work. From memory—I’m on my phone—you should do this. In your fuzz function, start/stop profiling like usual, writing output to a randomly named file in a known directory. Then take the fuzz.zip and unzip it. Inside will be two binaries, one with sonar and one without. Unfortunately the profiles will be a mixed batch, but most will be for the non-sonar binary, so you can just use that and expect some random noise/gibberish from the profiles taken using the sonar binary.

HerrSpace

comment created time in 4 days

issue commentgolang/go

cmd/compile: combine int tests when comparing against a constant string

cc @mundaym who has been thinking about SSA rules to combine branches

josharian

comment created time in 5 days

issue openedgolang/go

cmd/compile: optimize small runtime.memequal calls

In the SSA backend, we could optimize a call to runtime.memequal in which the length is small and one argument points to a readonly data symbol. We could replace it by reading the readonly data and checking the bytes of the other argument directly.

This is similar to the optimization in walk.go for comparing against a constant string, and could potentially supercede it. The optimization in walk.go doesn't always trigger. An example is using strings.HasPrefix to check for a constant prefix; the call gets inlined, but walk.go isn't able to do the optimization because its analysis isn't powerful enough.

Low priority.

created time in 5 days

issue openedgolang/go

cmd/compile: combine int tests when comparing against a constant string

To compare against "abc", we currently emit code to test whether uint16(first two bytes)==uint16("ab") and then whether (third byte)=="c".

I suspect that it'd be more efficient to load the three bytes into the first three bytes of a uint32 (two loads combined with |) and then compare that to uint32("abc\0").

I'm not sure whether we could do this with comparison-combining rewrite rules (note that the loads have side-effects in general, but we're sure in this case they won't fault) or whether it'd be better to fix this in walk.go, where this optimization is generated.

Low priority.

created time in 5 days

issue commentgolang/go

A

Looks like this may have been opened by mistake?

chandi1988

comment created time in 5 days

issue commentgolang/go

cmd/compile: cant atomic access of first word of allocated struct occasionally?

Maybe I'm missing something, but this looks like a malloc issue, not a compiler issue.

cc @mknyszek @aclements

NewbMiao

comment created time in 5 days

issue openedgolang/go

runtime: programs sometimes stall at launch

A weird thing has been happening to me recently, and I'm not sure how to debug it.

I'll run go install cmd/compile to install a modified compiler. (In this instance, the modifications were envvar-guarded logging, so it was not a broken compiler.)

Then I'll execute the compiler. To rule out any go tool problems or the like, I am running the compiler directly using an absolute path: /Users/josh/go/tip/pkg/tool/darwin_amd64/compile -S x.go. And the compiler hangs.

If I send a SIGQUIT, I don't get a backtrace:

$ /Users/josh/go/tip/pkg/tool/darwin_amd64/compile -S x.go
^\[1]    12198 quit       /Users/josh/go/tip/pkg/tool/darwin_amd64/compile -S x.go

It's as if the runtime stalled during launch.

Subsequent compiler runs usually work as normal.

I can't reproduce reliably, but it has happened enough that I think I'm probably not going crazy.

What should I do to debug next time this happens?

created time in 6 days

issue commentgolang/go

cmd/compile: "ICE: 'main': schedule does not include all values in block b19 crash" for Go1.12 and Go1.13 but fixed in Go1.14

I don't know for sure, but it seems likely to me that CL 178197 might have fixed this particular incident, but that there may yet be similar cases lurking. We should look to see whether there's an independent fix, not least because we shouldn't backport CL 178197.

I'm out of time for the moment, but here's a partially minimized reproduction:

package main

func main() {
	var n, a, b int64
	for i := int64(2); i < 10; i++ {
		for j := i; j < 10; j++ {
			if ((n % (i * j)) == 0) && (j > 1 && (n/(i*j)) == 1) {
				a, b = i, 0
				a = n / (i * j)
			}
		}
	}

	if a != b && a != n {
		println("YES")
	}
}

From my first look, it appears that the problem is with CSEing a Select0. When CSE moves a tuple generator to a new block it is supposed to move all selectors along with it. It did so, but somehow there was still a phi value using the old selector as an arg. The old selector thus doesn't get deadcoded, causing scheduling to fail.

That's as far as I've gotten, and as far as I'm likely to get in the very near term.

cc @cherrymui @randall77

amarjeetanandsingh

comment created time in 6 days

issue commentgolang/go

sync: shrink types in sync package

@balasanjay I think you should hack the compiler temporarily to inline this, in order to make progress. If it turns out the Load+CAS isn't performant enough, no harm done. And if it is a viable path forward, we can focus our attention on getting it past the execrable inliner.

E.g. I just noticed that we're charging a budget of 2 for intrinsic calls, one for the intrinsic and one for the CALLFUNC. But given that they're assumed to be comparable in cost to an arithmetic operation, we should just assign them a cost of 1. I suspect this is an oversight/accident. (Insofar as any of this can be called principled.) Fixing this gets your function inlined, barely. :)

diff --git a/src/cmd/compile/internal/gc/inl.go b/src/cmd/compile/internal/gc/inl.go
index 48c7de327d..4379be18ba 100644
--- a/src/cmd/compile/internal/gc/inl.go
+++ b/src/cmd/compile/internal/gc/inl.go
@@ -321,7 +321,7 @@ func (v *hairyVisitor) visit(n *Node) bool {
                }
 
                if isIntrinsicCall(n) {
-                       v.budget--
+                       // Treat like any other node.
                        break
                }

I'll plan to mail this as a CL during Go 1.15 and see whether it flies.

In the meantime, please forge ahead using whatever means necessary. :)

balasanjay

comment created time in 8 days

pull request commentlibgit2/libgit2

mwindow: set limit on number of open files

One possible future optimization: If the pack file is less than git_mwindow__window_size, close it immediately after mmap, since we know that we are going to mmap the entirety of the file anyway.

lhchavez

comment created time in 8 days

issue commentgolang/go

x/tools/go/packages, go/ast: NewPackage causes SIGSEGV

As a general point, I've definitely lost time debugging go/packages issues because I've failed to set NeedX when I've set NeedY and NeedY requires NeedX. I don't think I've hit this particular combination.

perillo

comment created time in 9 days

issue commentgolang/go

cmd/compile: the compiler should probably recognize runtime.Goexit

The relevant bit of the spec is https://golang.org/ref/spec#Terminating_statements. I’d personally be reluctant to add runtime.Goexit; everything else in that list is lower level.

It is important for vet to use the same set of terminating statements as the compiler. Otherwise you’ll be forced to add a terminating statement in some cases (like this) to make your code compile but vet will then complain about it.

perillo

comment created time in 9 days

issue commentgolang/go

cmd/vet: warn about more malformed JSON struct tags

Seems reasonable. Should fit nicely into the existing vet struct tags check. Should be pretty straightforward to implement for anyone wanting to try their hand on vet.

cc @mvdan @robpike

pfwgbe

comment created time in 9 days

pull request commentlibgit2/libgit2

mwindow: set limit on number of open files

Naive question: do we need the .pack files open, or could we mmap-and-close them too?

lhchavez

comment created time in 10 days

pull request commentlibgit2/libgit2

mwindow: set limit on number of open files

mmap’d files still keep the fd open

I should google before typing. Apparently this is not so. TIL! :)

lhchavez

comment created time in 10 days

pull request commentlibgit2/libgit2

mwindow: set limit on number of open files

I’m on my phone, so maybe I’m missing something, but mmap’d files still keep the fd open. (I was double-counting the .pack files by including mmap reference and regular reference, but the mmap-only .idx handles still count.)

lhchavez

comment created time in 10 days

issue commentgolang/go

crypto/cipher: optimize safeXORBytes

cc @FiloSottile for removing assembly in favor of pure Go

Filippo, is 2x-3x in this case within range of switching to pure Go?

There may also be further optimizations available (either in the pure Go or in the compiler); I haven't checked yet, and may not for a little while.

cc @CAFxX who I believe likes optimizing stuff like this

josharian

comment created time in 10 days

pull request commentlibgit2/libgit2

mwindow: set limit on number of open files

Is there a good way to set a limit on the number of .idx files open?

lhchavez

comment created time in 10 days

pull request commentlibgit2/libgit2

mwindow: set limit on number of open files

Oh, I bet lsof reports two lines for each open .pack file, one for the regular open and one for the mmap.

lhchavez

comment created time in 10 days

pull request commentlibgit2/libgit2

mwindow: set limit on number of open files

Update: as I continue to run, the number of open .idx files continues to grow, albeit slowly. Currently at 68 and growing. (File limit set to 32.)

lhchavez

comment created time in 10 days

pull request commentlibgit2/libgit2

mwindow: set limit on number of open files

I tried this out in my application. It definitely helps.

It doesn't seem to limit the number of open files to anywhere particularly close to the set value.

I set a limit of 32 and ran for a while, doing a bunch of little network operations, and found that lsof stabilized at around 64 open .pack files and around 45 open .idx files.

I changed the limit to 16, and found that it stabilized at around 32 open .pack files and 45 open .idx files.

At a limit of 8 (!), it stabillized at around 16 open .pack files and 45 open .idx files.

So it seems like maybe there's a factor-of-two bug in the .pack file handling, and something entirely different or missing in the .idx file handling.

The mere fact of stability and no errors is already a huge improvement, though; thank you!

lhchavez

comment created time in 10 days

push eventjosharian/git2go

Josh Bleecher Snyder

commit sha 760c67c973aa7efb41292f26f03e572c89a74f8e

Squashed 'libgit2/' changes from 575f4b8b56..6dacb116a9 6dacb116a9 mwindow: set limit on number of open files 03ac24b119 Merge pull request #5387 from pks-t/pks/transport-http-custom-headers 65ac33aeb1 Merge pull request #5382 from libgit2/pks/azure-coverity 46228d86f8 transports: http: fix custom headers not being applied 86c54cc8a8 azure: coverity: fix Coverity builds due to various issues ccffea6bfa azure: coverity: convert to use self-built containers b4eb0282a7 azure: coverity: fix invalid syntax for Docker image REVERT: 575f4b8b56 mwindow: set limit on number of open files REVERT: 00315f5ddd transports: http: fix custom headers not being applied git-subtree-dir: libgit2 git-subtree-split: 6dacb116a9d58269883919dcc4f9f1afbaae0b77

view details

Josh Bleecher Snyder

commit sha 9ed5cea183799f7a26d9973695e33f38438864c6

Merge commit '760c67c973aa7efb41292f26f03e572c89a74f8e' into tempmaster

view details

Josh Bleecher Snyder

commit sha 59bacf8d25e4138348c241e296fdb8188bb9562a

index: mark checkFatal as t.Helper to get better line numbers from failures

view details

Josh Bleecher Snyder

commit sha 723c36bfac69948f1e377a3de593d3fc0dcbb985

update settings to new name (TODO: fix exposed API name)

view details

push time in 10 days

create barnchjosharian/libgit2

branch : limit-mwindow

created branch time in 10 days

issue commentgolang/go

proposal: spec: derive array pointer from slice

Ping @golang/proposal-review. :)

rogpeppe

comment created time in 10 days

issue commentdvyukov/go-fuzz

make it easier to fuzz multiple functions

IMO, it should require a prefix of of Fuzz, like Test functions do.

It does. See func isFuzzFuncName in the commit you linked to.

josharian

comment created time in 10 days

issue commentdvyukov/go-fuzz

make it easier to fuzz multiple functions

Question, would it be acceptable to call all fuzz functions from a single parent?

You can, but there are downsides: Your corpus and your coverage information gets mixed together. Sometimes that is useful, and sometimes it isn't. It's not something that go-fuzz can do automatically.

josharian

comment created time in 10 days

issue commentdvyukov/go-fuzz

make it easier to fuzz multiple functions

I added the ability to compile multiple fuzz functions at once and then select between them at go-fuzz time. What isn’t implemented is the ability to have go-fuzz run all fuzz functions at once, sharing time between them.

josharian

comment created time in 10 days

issue commentgolang/go

proposal: new cryptographic assembly policy

@bwesterb please do file issues when you encounter specific intrinsics that are important for eliminating assembly. There’s uncertainty around vector intrinsics, but I think there’s openness to new scalar intrinsics.

FiloSottile

comment created time in 10 days

pull request commentlibgit2/git2go

make Delta and DiffLineType stringers

Thanks for the review.

I'd given up on upstreaming my changes, because the repo seemed unmaintained. Glad to see you do a pass through. (Want to hit the merge button on all the approved changes?) Do you anticipate doing more reviewing? That is, should I take the time to put together a bunch of PRs? It's ok if the answer is no--I'm doing just fine all by myself off in my fork.

josharian

comment created time in 11 days

Pull request review commentlibgit2/libgit2

mwindow: set limit on number of open files

 static git_mwindow *new_window(  	ctl->mapped += (size_t)len; -	while (git_mwindow__mapped_limit < ctl->mapped &&+	while ((git_mwindow__mapped_limit < ctl->mapped ||+	       git_mwindow__open_limit < ctl->open_windows) &&

Hooray! Thanks!! I've been under water fighting another bug, but I'll patch in your PR and give it a whirl soon.

josharian

comment created time in 11 days

issue commentgolang/go

cmd/trace: requires HTML imports, a deprecated feature that only works on Chrome

@egonelbre investigated rebuilding traceviewer in #11520. Hopefully it won't come to that, though.

mvdan

comment created time in 12 days

issue commentgolang/go

cmd/go: offer a consistent "global install" command

@bcmills I see. Which is why I needed to write a script instead of being able to just put the files in the right place. OK, I'll hide my comments.

mvdan

comment created time in 12 days

issue commentgolang/go

cmd/go: offer a consistent "global install" command

OK, I've edited my comment to s/repository/module/. I believe that the point stands.

mvdan

comment created time in 12 days

issue commentgolang/go

cmd/trace: requires HTML imports, a deprecated feature that only works on Chrome

Is the problem here how we get the data from the http request into a place where the trace viewer can render it?

If so, one solution is to have go trace emit a single standalone file, with the data embedded in it. (See also the various links in that issue.) I did this as part of a different effort, killed by Russ, back around Go 1.8 days: https://github.com/josharian/go/commit/a5a9f7595338afbfd5f814956b17dc60e0bc586e#diff-f3a5f1780e7f702229cc143e633fd754R12.

mvdan

comment created time in 12 days

issue commentgolang/go

cmd/go: offer a consistent "global install" command

To further complicate the discussion of directory replace directives, one case in which a directory replace directive probably could safely be applied is what the directory is located inside the same repository. This would be important for the kind of case considered in #34867, in which the agreed-upon workaround was a directory replacement.

mvdan

comment created time in 12 days

issue commentgolang/go

cmd/compile: rc1.14rc1 build internal/ssa out of memory

That’s good to hear. That won’t land until the 1.15 release, though.

n2vi

comment created time in 14 days

issue commentgolang/go

cmd/compile: rc1.14rc1 build internal/ssa out of memory

Sounds like Austin probably has a better bead on what’s going on here.

I’m curious though whether my planned changes will help sufficiently. To test,

git fetch "https://go.googlesource.com/go" refs/changes/03/213703/6 && git checkout FETCH_HEAD

and then try again.

n2vi

comment created time in 15 days

issue commentgolang/go

cmd/compile: rc1.14rc1 build internal/ssa out of memory

To rule out some things, let's try disabling parallel and concurrent compilation:

GOFLAGS="-p=1 -gcflags=cmd/compile/internal/ssa=-c=1" ./all.bash

If that still fails, let's see how much RAM the shell thinks it is using:

GOFLAGS=-toolexec="/usr/bin/time -l" ./all.bash

n2vi

comment created time in 15 days

issue commentgolang/go

cmd/compile: rc1.14rc1 build internal/ssa out of memory

Yeah, 12gb really ought to be enough. :) Does it reproduce consistently?

cc @mknyszek @aclements just in case

n2vi

comment created time in 15 days

issue commentgolang/go

cmd/compile: rc1.14rc1 build internal/ssa out of memory

There was actually plenty of available RAM at the time, according to top.

What does "plenty" mean here? I don't know how quickly top refreshes, but building package ssa eats ram (and potentially dies) very quickly.

I suspect this is probably a duplicate of https://github.com/golang/go/issues/27739. I have a series of CLs ready for 1.15 that will shrink package ssa by about 25%.

n2vi

comment created time in 15 days

Pull request review commentlibgit2/libgit2

Use error code GIT_EAUTH for authentication failures

 static int winhttp_stream_read( 	/* Enforce a reasonable cap on the number of replays */ 	if (replay_count++ >= GIT_HTTP_REPLAY_MAX) { 		git_error_set(GIT_ERROR_HTTP, "too many redirects or authentication replays");-		return -1;+		return GIT_ERROR; /* not GIT_EAUTH because the exact cause is not clear */

My instinct would be to return GIT_EAUTH if any were authentication failures, since the other redirects are likely to be leading up to or a consequence of the auth failure. That is, it seems to me that auth failure should be sticky. Or maybe our divergent instincts mean that we should leave it as is. Either way, I'd prefer to do that as a follow-up.

josharian

comment created time in 15 days

PR opened libgit2/libgit2

Use error code GIT_EAUTH for authentication failures

See the individual commit messages for more detail.

Fixes #5389

+28 -25

0 comment

9 changed files

pr created time in 15 days

create barnchjosharian/libgit2

branch : http-use-eauth

created branch time in 15 days

push eventjosharian/rulesfile

Josh Bleecher Snyder

commit sha 669e81f88862d2a8397d11924000dd553e73cbe2

allow | and ^ in auxint

view details

push time in 15 days

issue commentlibgit2/libgit2

use error code GIT_EAUTH for HTTP auth failures

Sheesh, there's a whole lot more to backwards compatibility than one thinks of at first.

Is there ever! :) I've a pretty active contributor to the Go compiler for a while. Go takes backwards compatibility extremely seriously, and yet folks who have been working on it for a decade are still surprised. Very recent example: https://github.com/golang/go/issues/28591#issuecomment-583093984.

Fortunately (or unfortunately?), I don't think libgit2 has quite the same level of commitment to backwards compatibility.

more precise error code for auth failures

I'm happy to do the legwork on this, if the http code has stabilized enough that I won't just be in the way. Or unless @ethomson would prefer to do it.

josharian

comment created time in 16 days

push eventjosharian/git2go

Josh Bleecher Snyder

commit sha b263e41d08ac4e2da603b58bb431c21f45cfb160

use non-deprecated constants for ErrorClass

view details

Josh Bleecher Snyder

commit sha 64cf9e02639ea7ab813ce82f3e59554c8ac37649

add missing ErrorClass constants

view details

push time in 16 days

issue openedlibgit2/libgit2

use error code GIT_EAUTH for HTTP auth failures

b9c5b15 is backwards incompatible. (Related: #5385.) Prior to that commit, HTTP auth failures had an error class of GIT_ERROR_NET. That commit changes them to GIT_ERROR_HTTP. I will adjust my code accordingly, but it should probably be called out in the release notes.

That said, it'd be nicer to have a more precise error code for auth failures. For lack of a good alternative, I currently check the error class and then do a string match for the word "authentication", because the error code is a generic GIT_ERROR. There is already an error code for authentication failures, GIT_EAUTH, but it appears that only the ssh network operations return it.

I'd like to suggest that HTTP use it too. Now is a particularly good time to switch, since people may have to update their error handling in light of b9c5b15 anyway.

cc @ethomson @pks-t

created time in 16 days

issue openedgolang/go

cmd/compile: ARM's MOVWnop causes regalloc to insert unnecessary copies

MOVWnop is a nop/copy designed to ensure type safety. It generates no code. However, it is not free; regalloc sometimes makes a copy of a value to provide to MOVWnop, since it doesn't know that MOVWnop doesn't modify its argument.

I noticed this while tracking down regressions due to changes in rewrite rule order application.

I don't know what the best approach to fixing it is. Suggestions?

cc @cherrymui

created time in 16 days

push eventjosharian/git2go

Josh Bleecher Snyder

commit sha ea082fbc525743746117efa31ab7d9df161c8895

expose setting on go side

view details

Josh Bleecher Snyder

commit sha 561236281f0d9d691c215b8fca8c402911523957

expose http expect continue

view details

push time in 16 days

create barnchjosharian/git2go

branch : tempmaster

created branch time in 16 days

create barnchjosharian/libgit2

branch : tempmaster

created branch time in 16 days

Pull request review commentlibgit2/libgit2

mwindow: set limit on number of open files

 static git_mwindow *new_window(  	ctl->mapped += (size_t)len; -	while (git_mwindow__mapped_limit < ctl->mapped &&+	while ((git_mwindow__mapped_limit < ctl->mapped ||+	       git_mwindow__open_limit < ctl->open_windows) &&

I see, I think.

From a quick look, that surgery might require more time than I really wanted to dedicate right now.

Do you think there is anyone who already knows this code that might be able to work on a proper fix? (If not, I may have to, since this issue is critical for me.)

josharian

comment created time in 16 days

issue commentlibgit2/libgit2

recent regression adding http headers to push/fetch

#5387 fixes it, thanks!

josharian

comment created time in 16 days

create barnchjosharian/git2go

branch : pr5387

created branch time in 16 days

create barnchjosharian/libgit2

branch : pr5387

created branch time in 16 days

push eventjosharian/rulesfile

Josh Bleecher Snyder

commit sha a7bd34309deffdfa646930e5c0574c704b3b1610

allow < and > in auxint they show up in shifts in auxint in arm64 rules

view details

push time in 17 days

issue commentlibgit2/libgit2

GC mwindows on too many open files

PR is #5386.

carlosmn

comment created time in 17 days

pull request commentlibgit2/libgit2

mwindow: set limit on number of open files

I'm not sure how to write a test for this.

Doing some manual testing (applied to master a little ways back, due to https://github.com/libgit2/libgit2/issues/5385), this helps with the situation in #2758, but not quite in the way I expected/hoped.

I still see O(#packfiles) open files using lsof with this patch applied. The difference this patch makes is that when the number of packfiles shrinks, so does the number of open files.

I don't understand enough about what's going on in the rest of the codebase that causes this combination of effects. I'd prefer that it strictly limited the number of open files. Hints and suggestions welcome!

For my purposes, this patch is at least enough to ameliorate the worst of #2758, because I can at least keep things under control by occasionally running git gc.

josharian

comment created time in 17 days

PR opened libgit2/libgit2

mwindow: set limit on number of open files

Doing many network operations can lead to lots of small packfiles. The existing mwindow limit was only the total size of mmap'd files, not on their number, so having lots of small packfiles could exhaust the allowed number of open files, particularly on macOS, where the default ulimit is very low.

Add a limit to the number of files as well as their total size, and set the default limit low enough that even macOS users should not hit it during normal use.

Fixes #2758

+29 -3

0 comment

3 changed files

pr created time in 17 days

create barnchjosharian/git2go

branch : mwindowj

created branch time in 17 days

issue commentlibgit2/libgit2

recent regression adding http headers to push/fetch

Just to make sure, I threaded GIT_OPT_ENABLE_HTTP_EXPECT_CONTINUE through to git2go and enabled it; it did not fix the problem.

josharian

comment created time in 17 days

push eventjosharian/git2go

Josh Bleecher Snyder

commit sha c2f40ad52734dd7ca6316f783a337220f98fff25

expose setting on go side

view details

Josh Bleecher Snyder

commit sha a8aef2d6aa844e808db42ba4fe238828962dfdb6

expose http expect continue

view details

push time in 17 days

push eventjosharian/git2go

Josh Bleecher Snyder

commit sha c32616611395fbb0162658ef7bf9b3040bca52ed

expose setting on go side

view details

Josh Bleecher Snyder

commit sha d66e084f9df43a7526076ae7dbe0112173570cc0

expose http expect continue

view details

push time in 17 days

push eventjosharian/libgit2

Josh Bleecher Snyder

commit sha 57594c2925d5926f393f91426be6832d23e9f769

mwindow: set limit on number of open files Doing many network operations can lead to lots of small packfiles. The existing mwindow limit was only the total size of mmap'd files, not on their number, so having lots of small packfiles could exhaust the allowed number of open files, particularly on macOS, where the default ulimit is very low. Add a limit to the number of files as well as their total size, and set the default limit low enough that even macOS users should not hit it during normal use. Fixes #2758

view details

push time in 17 days

create barnchjosharian/libgit2

branch : issue2758j

created branch time in 17 days

issue openedlibgit2/libgit2

recent regression adding http headers to push/fetch

Reproduction steps

I use a bearer token in an Authorization header to authorize with my server.

Using git2go:

opts := new(git.PushOptions)
opts.Headers = []string{"Authorization: Bearer " + bearer}
err := remote.Push(refspecs, opts)

This worked until recently. At current master (559751714b23f63ead726e9a773be8b07e3551ae), there is no Authorization header present when the request reaches the server.

I have bisected to one of these three commits:

  • 7372573b5f1113b8522e2588fac1c529ddcedb0a
  • bf55facf15e5b6897ff7305162a77d92089f6c82
  • b9c5b15a7958ab4ecb83504a6d858efd18610297

The first two of these commits don't compile: src/transports/http.c:89:2: error: unknown type name 'git_http_authtype_t'. The last one seems pretty plausible.

cc @ethomson

created time in 17 days

push eventjosharian/git2go

Josh Bleecher Snyder

commit sha 498c062fc5bb2b7a5e072dab3b1bfe6e880b472f

Squashed 'libgit2/' changes from b9c5b15a79..6c21c989a3 REVERT: b9c5b15a79 http: use the new httpclient REVERT: bf55facf15 tests: allow users to use expect/continue REVERT: 7372573b5f httpclient: support expect/continue git-subtree-dir: libgit2 git-subtree-split: 6c21c989a3b8cba20ad8cbe806937071a1923832

view details

Josh Bleecher Snyder

commit sha 78e06682eedd32832c11d2762ef8dfb4e677475e

Merge commit '498c062fc5bb2b7a5e072dab3b1bfe6e880b472f' into mwindow

view details

push time in 17 days

push eventjosharian/git2go

Josh Bleecher Snyder

commit sha 820c164d32662cef65906eae66c847baca0a7d34

Squashed 'libgit2/' changes from bf55facf15..b9c5b15a79 b9c5b15a79 http: use the new httpclient git-subtree-dir: libgit2 git-subtree-split: b9c5b15a7958ab4ecb83504a6d858efd18610297

view details

Josh Bleecher Snyder

commit sha 731b7176b0b088f61f9ef76c13109f33c7c15e51

Merge commit '820c164d32662cef65906eae66c847baca0a7d34' into mwindow

view details

push time in 17 days

push eventjosharian/git2go

Josh Bleecher Snyder

commit sha d8df6133607ecf74013c8e26b332795f1ffb173d

Squashed 'libgit2/' changes from 7372573b5f..bf55facf15 bf55facf15 tests: allow users to use expect/continue git-subtree-dir: libgit2 git-subtree-split: bf55facf15e5b6897ff7305162a77d92089f6c82

view details

Josh Bleecher Snyder

commit sha 165a2429d237e170f8a1c3e3fc477dd2c26a89c2

Merge commit 'd8df6133607ecf74013c8e26b332795f1ffb173d' into mwindow

view details

push time in 17 days

push eventjosharian/git2go

Josh Bleecher Snyder

commit sha 43ed62fe83fa2f7c7a6a7c4f10367ebf009a3027

Squashed 'libgit2/' changes from 3e9ee04f11..7372573b5f REVERT: 3e9ee04f11 trace: compare against an int value REVERT: 76fd406a75 http: send probe packets REVERT: b9c5b15a79 http: use the new httpclient REVERT: bf55facf15 tests: allow users to use expect/continue git-subtree-dir: libgit2 git-subtree-split: 7372573b5f1113b8522e2588fac1c529ddcedb0a

view details

Josh Bleecher Snyder

commit sha c247dd36bd5a1d870961b1854de1e596737975a1

Merge commit '43ed62fe83fa2f7c7a6a7c4f10367ebf009a3027' into mwindow

view details

push time in 17 days

push eventjosharian/git2go

Josh Bleecher Snyder

commit sha 08e183d31c8b4b590fcf87857ecb0c853760c2fd

Squashed 'libgit2/' changes from 82154e586c..3e9ee04f11 REVERT: 82154e586c remote functions: return an int REVERT: 3351506aa8 tree functions: return an int REVERT: 2e8c3b0b18 oid functions: return an int REVERT: 9893d376ee git_attr_cache_flush: return an int REVERT: 4460bf40c9 Merge pull request #5286 from libgit2/ethomson/gssapi REVERT: e9cef7c4b1 http: introduce GIT_ERROR_HTTP REVERT: 7fd9b3f597 ci: add NTLM tests REVERT: 29762e4089 httpclient: use defines for status codes git-subtree-dir: libgit2 git-subtree-split: 3e9ee04f11ab505e6e79b75b5627d1e075932330

view details

Josh Bleecher Snyder

commit sha 8cf513ee1684647c3a07c0e4ba0ed1f38e28f1e8

Merge commit '08e183d31c8b4b590fcf87857ecb0c853760c2fd' into mwindow

view details

push time in 17 days

push eventjosharian/git2go

Josh Bleecher Snyder

commit sha 74c5048014a57d5ac83969b3f0a311ff890f82da

Squashed 'libgit2/' changes from 6a095679c8..82154e586c 82154e586c remote functions: return an int 3351506aa8 tree functions: return an int 2e8c3b0b18 oid functions: return an int 9893d376ee git_attr_cache_flush: return an int 4460bf40c9 Merge pull request #5286 from libgit2/ethomson/gssapi e9cef7c4b1 http: introduce GIT_ERROR_HTTP 7fd9b3f597 ci: add NTLM tests 29762e4089 httpclient: use defines for status codes 3e9ee04f11 trace: compare against an int value 76fd406a75 http: send probe packets b9c5b15a79 http: use the new httpclient bf55facf15 tests: allow users to use expect/continue 7372573b5f httpclient: support expect/continue 6c21c989a3 httpclient: support CONNECT proxies 6b2088363f httpclient: handle chunked responses git-subtree-dir: libgit2 git-subtree-split: 82154e586c830a94059c3cfbed2d5ab4985cf3ce

view details

Josh Bleecher Snyder

commit sha 60502b5aeb04f3295663664ff61d2a120858b972

Merge commit '74c5048014a57d5ac83969b3f0a311ff890f82da' into mwindow

view details

push time in 17 days

push eventjosharian/git2go

Josh Bleecher Snyder

commit sha 3646e8baaf932c5a297b96249765d7cedeac488d

Revert "expose setting on go side" This reverts commit c246e9bc676de348e9d6edbaccefa18d9a5ea884.

view details

push time in 17 days

push eventjosharian/git2go

Josh Bleecher Snyder

commit sha 2ac03cff6c74a79297763be1fbe89a05a2b1b7b2

Squashed 'libgit2/' changes from 11ae656404..6a095679c8 REVERT: 11ae656404 mwindow: set limit on number of open files REVERT: 559751714b Merge pull request #5373 from pks-t/pks/fetchhead-strip-creds REVERT: 93a9044ff6 fetchhead: strip credentials from remote URL REVERT: a1bff63bf1 Merge pull request #5375 from pks-t/pks/test-ci REVERT: 7aa99dd347 azure-pipelines: properly expand negotiate passwords REVERT: aa4cd778b9 Merge pull request #5336 from libgit2/ethomson/credtype REVERT: f9b41a6600 Merge pull request #5371 from ayush-1506/julia_link REVERT: 103a76b424 Update link to Julia libgit2 REVERT: 3f54ba8b61 credential: change git_cred to git_credential REVERT: 4383ab402a Merge pull request #5365 from libgit2/ethomson/no_void REVERT: 4cae9e712c git_libgit2_version: return an int REVERT: f78f6bd597 error functions: return an int REVERT: 4b331f020e revwalk functions: return an int REVERT: 82050fa1be mempack functions: return an int REVERT: a3126a72d2 repository functions: return an int REVERT: cb43274a6b index functions: return an int REVERT: 82154e586c remote functions: return an int REVERT: 3351506aa8 tree functions: return an int REVERT: 2e8c3b0b18 oid functions: return an int REVERT: 9893d376ee git_attr_cache_flush: return an int REVERT: 4460bf40c9 Merge pull request #5286 from libgit2/ethomson/gssapi REVERT: e9cef7c4b1 http: introduce GIT_ERROR_HTTP REVERT: 7fd9b3f597 ci: add NTLM tests REVERT: 29762e4089 httpclient: use defines for status codes REVERT: 3e9ee04f11 trace: compare against an int value REVERT: 76fd406a75 http: send probe packets REVERT: b9c5b15a79 http: use the new httpclient REVERT: bf55facf15 tests: allow users to use expect/continue REVERT: 7372573b5f httpclient: support expect/continue REVERT: 6c21c989a3 httpclient: support CONNECT proxies REVERT: 6b2088363f httpclient: handle chunked responses git-subtree-dir: libgit2 git-subtree-split: 6a095679c83922d5b7e72a06882fe99de3bd4db6

view details

Josh Bleecher Snyder

commit sha e2257741211b7c2648f509b9a288f300f6d724a0

Merge commit '2ac03cff6c74a79297763be1fbe89a05a2b1b7b2' into mwindow

view details

push time in 17 days

push eventjosharian/git2go

Josh Bleecher Snyder

commit sha c246e9bc676de348e9d6edbaccefa18d9a5ea884

expose setting on go side

view details

push time in 17 days

push eventjosharian/git2go

Josh Bleecher Snyder

commit sha 52f15a632bc889c9de387a14ebcdbf7017be8aba

Squashed 'libgit2/' changes from a76348ee88..11ae656404 11ae656404 mwindow: set limit on number of open files 559751714b Merge pull request #5373 from pks-t/pks/fetchhead-strip-creds 93a9044ff6 fetchhead: strip credentials from remote URL a1bff63bf1 Merge pull request #5375 from pks-t/pks/test-ci 7aa99dd347 azure-pipelines: properly expand negotiate passwords aa4cd778b9 Merge pull request #5336 from libgit2/ethomson/credtype f9b41a6600 Merge pull request #5371 from ayush-1506/julia_link 103a76b424 Update link to Julia libgit2 3f54ba8b61 credential: change git_cred to git_credential 4383ab402a Merge pull request #5365 from libgit2/ethomson/no_void 4cae9e712c git_libgit2_version: return an int f78f6bd597 error functions: return an int 4b331f020e revwalk functions: return an int 82050fa1be mempack functions: return an int a3126a72d2 repository functions: return an int cb43274a6b index functions: return an int 82154e586c remote functions: return an int 3351506aa8 tree functions: return an int 2e8c3b0b18 oid functions: return an int 9893d376ee git_attr_cache_flush: return an int 4460bf40c9 Merge pull request #5286 from libgit2/ethomson/gssapi e9cef7c4b1 http: introduce GIT_ERROR_HTTP 7fd9b3f597 ci: add NTLM tests 29762e4089 httpclient: use defines for status codes 3e9ee04f11 trace: compare against an int value 76fd406a75 http: send probe packets b9c5b15a79 http: use the new httpclient bf55facf15 tests: allow users to use expect/continue 7372573b5f httpclient: support expect/continue 6c21c989a3 httpclient: support CONNECT proxies 6b2088363f httpclient: handle chunked responses 6a095679c8 httpclient: support authentication 0e39a8faf9 net: free the url's query component 0b8358c8d1 net: introduce path formatting function 1152f3618c httpclient: consume final chunk message dcd3b815a2 tests: support CLAR_TRACE_LEVEL 84b99a95bf httpclient: add chunk support to POST eacecebdcf httpclient: introduce a simple http implementation a591f3623c net: introduce url formatting function d68f2b1ab7 buf: add consume_bytes e995f74e88 net: introduce git_net_url_joinpath 471daeea55 net: refactor gitno redirect handling 297c61e41f net: add an isvalid function a194e17f97 winhttp: refactor request sending da9bc8ac11 ci: enable GSSAPI for Ubuntu builds 9b96adf33d docker build: add krb5 libraries and command-line tools 031e3e4c13 ci: don't do negotiate tests on windows fe4efe2e36 tests: test that clone returns 4321 7e0f5a6ac7 smart protocol: correct case in error messages 2d6a61bd82 gssapi: validate that we were requested Negotiate e761df5cf5 gssapi: dispose after completion for retry 89d1fc2ab9 gssapi: show information about disabled GSSAPI 5625892b2c gssapi: delete half-built security context so auth can continue 2174aa0ac2 gssapi: correct incorrect case in error message 3f6fe05495 gssapi: protect GSS_ERROR macro 73fe690db2 gssapi: protect against empty messages e5fb5fe5ad ci: perform SPNEGO tests 9bcf10e97e Merge pull request #5364 from libgit2/ethomson/typet 917ba7626f auth: update enum type name for consistency b59c71d8a4 iterator: update enum type name for consistency df3063eafe rebase: update enum type name for consistency 94beb3a3f4 merge: update enum type name for consistency git-subtree-dir: libgit2 git-subtree-split: 11ae65640495c633d9ce2ec8efeba97629d42355

view details

Josh Bleecher Snyder

commit sha 65e6374acf0251fbf9e9ba4ce7a86822b10346bc

update libgit2 to mwindow branch as experiment Merge commit '52f15a632bc889c9de387a14ebcdbf7017be8aba' into mwindow

view details

push time in 17 days

create barnchjosharian/libgit2

branch : issue2758

created branch time in 17 days

more