profile
viewpoint

apple/swift 50710

The Swift Programming Language

linux-on-ibm-z/go 98

A port of the Go programming language to the IBM LinuxONE. Issues are monitored, and comments/questions are welcome on our DeveloperWorks community.

mundaym/vector 10

Example layout for a package using Go assembly.

linux-on-ibm-z/LuaJIT 2

Mirror of the LuaJIT git repository

linux-on-ibm-z/scylla 1

NoSQL data store using the seastar framework, compatible with Apache Cassandra

linux-on-ibm-z/seastar 0

High performance server-side application framework

mundaym/rethinkdb 0

The open-source database for the realtime web.

issue commentgolang/go

cmd/compile: add a way to declare variables in rewrite rules

This could potentially be really valuable in a CL like CL 173659 that uses helper types heavily (rot.FromAux(r) is repeated a lot). Though that particular example would probably be helped more by Keith's strongly typed aux field change (CL 190197). Speaking of which that CL would probably also help your example too.

Another syntax idea might be to borrow the from if statements a little bit (if quo := auxTo32F(c) / auxTo32F(d); quo == quo {) to make the boolean evaluation more explicit:

(Div32F (Const32F [c]) (Const32F [d]))
     && quo := auxTo32F(c) / auxTo32F(d); quo == quo
     -> (Const32F [auxFrom32F(quo)])
josharian

comment created time in a day

issue commentgolang/go

cmd/compile: consider dropping Greater* and Geq* generic ops

Does it also make it less stable? I.e. when you add a new unrelated optimization, does it tend to perturb the overall output more? Now that I'm investigating regressions, I'm starting to value that stability a lot. :)

I hope not. I think it will be roughly as stable as the ordering of value IDs. They aren't designed to be stable in the presence of changes so any stability we get is luck-based.

I'm tempted to rebase the integer-in-range change back onto master so that the Greater/Geq elimination changes can be more easily reverted (or just not merged) if it turns out they are more trouble than they are worth.

mundaym

comment created time in a day

issue commentgolang/go

x/build: linux-arm trybot failure

Unassigning @bradfitz since I don't think he's working on x/build at all anymore.

cherrymui

comment created time in 2 days

issue commentgolang/go

x/build: linux-arm trybot failure

Ping. Still seeing this and it means we can't run the trybots on arm targets:

:: Running /bin/bash with args ["/bin/bash" "-c" "cd $WORKDIR/go/src && ./make.bash && cd .. && mv bin/*_*/* bin && rmdir bin/*_* && rm -rf pkg/linux_amd64 pkg/tool/linux_amd64 pkg/bootstrap pkg/obj"] and env ["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" "HOSTNAME=buildlet-linux-armhf-cross-rne6cbcb3" "DEBIAN_FRONTEND=noninteractive" "GOROOT_BOOTSTRAP=/workdir/go1.4" "GOOS=linux" "GOARCH=arm" "CC_FOR_TARGET=arm-linux-gnueabihf-gcc" "HOME=/root" "USER=root" "GO_STAGE0_NET_DELAY=7.1s" "GO_STAGE0_DL_DELAY=200ms" "WORKDIR=/workdir" "GOROOT_BOOTSTRAP=/go1.4" "CGO_ENABLED=1" "CC_FOR_TARGET=arm-linux-gnueabihf-gcc" "GOOS=linux" "GOARCH=arm" "GOARM=7" "TMPDIR=/workdir/tmp" "GOCACHE=/workdir/gocache" "GOROOT_BOOTSTRAP=/go1.4"] in dir /workdir

Building Go cmd/dist using /go1.4. (go1.4.3 linux/amd64)
Building Go toolchain1 using /go1.4.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for host, linux/amd64.
Building packages and commands for target, linux/arm.
---
Installed Go for linux/arm in /workdir/go
Installed commands in /workdir/go/bin
linux-arm at 986a3d002214c88890831023d3b971e4e4dbf322

HASH[build runtime/internal/sys]
HASH[build runtime/internal/sys]: "devel 986a3d002214c88890831023d3b971e4e4dbf322"
HASH[build runtime/internal/sys]: "compile\n"
HASH[build runtime/internal/sys]: "goos linux goarch arm\n"
HASH[build runtime/internal/sys]: "import \"runtime/internal/sys\"\n"
HASH[build runtime/internal/sys]: "omitdebug false standard true local false prefix \"\"\n"
HASH[build runtime/internal/sys]: "modinfo \"\"\n"
HASH[build runtime/internal/sys]: "compile QTyqihzk1wd_VuqRtm9G [] []\n"
HASH[build runtime/internal/sys]: "GOARM=7\n"
HASH /workdir/go/src/runtime/internal/sys/arch.go: bd41e7bff6f1a5f24fa3d61a6267935e048f7e0d6c078e99b5ce653af35d255b
HASH[build runtime/internal/sys]: "file arch.go vUHnv_bxpfJPo9YaYmeT\n"
HASH /workdir/go/src/runtime/internal/sys/arch_arm.go: 1a86e93c9d864cbb553843ebb5a860d1b2f933a823c7b1446d9b573930d9f0a9
HASH[build runtime/internal/sys]: "file arch_arm.go GobpPJ2GTLtVOEPrtahg\n"
HASH /workdir/go/src/runtime/internal/sys/intrinsics.go: d9db851bb68ed47db516a63a07cfa5dc3ce73ce2ab30248974ed171e856910dc
HASH[build runtime/internal/sys]: "file intrinsics.go 2duFG7aO1H21FqY6B8-l\n"
HASH /workdir/go/src/runtime/internal/sys/intrinsics_common.go: b009ab2f510e179af3cd495b904b9f9ccc0484e9e6afd163199b587bfbc87741
HASH[build runtime/internal/sys]: "file intrinsics_common.go sAmrL1EOF5rzzUlbkEuf\n"
HASH /workdir/go/src/runtime/internal/sys/stubs.go: 23b3e5c631b086fe7a2dec4bf044600e034bf6a8eeb25e0a19efc4ce6311423d
HASH[build runtime/internal/sys]: "file stubs.go I7PlxjGwhv56LexL8ERg\n"
HASH /workdir/go/src/runtime/internal/sys/sys.go: 55e021891200a7e6a5c371c8a1ab71b6c15aeb16ea6c1b192185d17df8c8b18f
HASH[build runtime/internal/sys]: "file sys.go VeAhiRIAp-alw3HIoatx\n"
HASH /workdir/go/src/runtime/internal/sys/zgoarch_arm.go: b5c50dbd03bfb5fcdedc9731c37c0d9b7953145b9b188bac9ed2fb0ac801ed22
HASH[build runtime/internal/sys]: "file zgoarch_arm.go tcUNvQO_tfze3Jcxw3wN\n"
HASH /workdir/go/src/runtime/internal/sys/zgoos_linux.go: 806c088d7491b4560a28a5af86a52b459ebbf155ea455af873baa0bf697355e4
HASH[build runtime/internal/sys]: "file zgoos_linux.go gGwIjXSRtFYKKKWvhqUr\n"
HASH /workdir/go/src/runtime/internal/sys/zversion.go: aad7c743c0f4d6d297ff88575813e669827944ddd21a04e2c332c779a38b3026
HASH[build runtime/internal/sys]: "file zversion.go qtfHQ8D01tKX_4hXWBPm\n"
HASH[build runtime/internal/sys]: 90e53875aeed774b1558c99d18d0851b239e4a3fea9aa87382faeebfc5bf6b70
runtime/internal/sys true
go tool dist: unexpected stale targets reported by go list -gcflags="" -ldflags="" for [std]:
	STALE archive/tar: stale dependency: internal/cpu
	STALE archive/zip: stale dependency: internal/cpu
	STALE bufio: stale dependency: internal/cpu
	STALE bytes: stale dependency: internal/cpu
	STALE compress/bzip2: stale dependency: internal/cpu
	STALE compress/flate: stale dependency: internal/cpu
	STALE compress/gzip: stale dependency: internal/cpu
	STALE compress/lzw: stale dependency: internal/cpu
	STALE compress/zlib: stale dependency: internal/cpu
	STALE container/heap: stale dependency: internal/cpu
	STALE container/list: build ID mismatch
	STALE container/ring: build ID mismatch
	STALE context: stale dependency: internal/cpu
	STALE crypto: stale dependency: internal/cpu
	STALE crypto/aes: stale dependency: internal/cpu
	STALE crypto/cipher: stale dependency: internal/cpu
	STALE crypto/des: stale dependency: internal/cpu
	STALE crypto/dsa: stale dependency: internal/cpu
	STALE crypto/ecdsa: stale dependency: internal/cpu
	STALE crypto/ed25519: stale dependency: internal/cpu
	STALE crypto/ed25519/internal/edwards25519: stale dependency: internal/cpu
	STALE crypto/elliptic: stale dependency: internal/cpu
	STALE crypto/hmac: stale dependency: internal/cpu
	STALE crypto/internal/randutil: stale dependency: internal/cpu
	STALE crypto/internal/subtle: build ID mismatch
	STALE crypto/md5: stale dependency: internal/cpu
	STALE crypto/rand: stale dependency: internal/cpu
	STALE crypto/rc4: stale dependency: internal/cpu
	STALE crypto/rsa: stale dependency: internal/cpu
	STALE crypto/sha1: stale dependency: internal/cpu
	STALE crypto/sha256: stale dependency: internal/cpu
	STALE crypto/sha512: stale dependency: internal/cpu
	STALE crypto/subtle: build ID mismatch
	STALE crypto/tls: stale dependency: internal/cpu
	STALE crypto/x509: stale dependency: internal/cpu
	STALE crypto/x509/pkix: stale dependency: internal/cpu
	STALE database/sql: stale dependency: internal/cpu
	STALE database/sql/driver: stale dependency: internal/cpu
	STALE debug/dwarf: stale dependency: internal/cpu
	STALE debug/elf: stale dependency: internal/cpu
	STALE debug/gosym: stale dependency: internal/cpu
	STALE debug/macho: stale dependency: internal/cpu
	STALE debug/pe: stale dependency: internal/cpu
	STALE debug/plan9obj: stale dependency: internal/cpu
	STALE encoding: build ID mismatch
	STALE encoding/ascii85: stale dependency: internal/cpu
	STALE encoding/asn1: stale dependency: internal/cpu
	STALE encoding/base32: stale dependency: internal/cpu
	STALE encoding/base64: stale dependency: internal/cpu
	STALE encoding/binary: stale dependency: internal/cpu
	STALE encoding/csv: stale dependency: internal/cpu
	STALE encoding/gob: stale dependency: internal/cpu
	STALE encoding/hex: stale dependency: internal/cpu
	STALE encoding/json: stale dependency: internal/cpu
	STALE encoding/pem: stale dependency: internal/cpu
	STALE encoding/xml: stale dependency: internal/cpu
	STALE errors: stale dependency: internal/cpu
	STALE expvar: stale dependency: internal/cpu
	STALE flag: stale dependency: internal/cpu
	STALE fmt: stale dependency: internal/cpu
	STALE go/ast: stale dependency: internal/cpu
	STALE go/build: stale dependency: internal/cpu
	STALE go/constant: stale dependency: internal/cpu
	STALE go/doc: stale dependency: internal/cpu
	STALE go/format: stale dependency: internal/cpu
	STALE go/importer: stale dependency: internal/cpu
	STALE go/internal/gccgoimporter: stale dependency: internal/cpu
	STALE go/internal/gcimporter: stale dependency: internal/cpu
	STALE go/internal/srcimporter: stale dependency: internal/cpu
	STALE go/parser: stale dependency: internal/cpu
	STALE go/printer: stale dependency: internal/cpu
	STALE go/scanner: stale dependency: internal/cpu
	STALE go/token: stale dependency: internal/cpu
	STALE go/types: stale dependency: internal/cpu
	STALE hash: stale dependency: internal/cpu
	STALE hash/adler32: stale dependency: internal/cpu
	STALE hash/crc32: stale dependency: internal/cpu
	STALE hash/crc64: stale dependency: internal/cpu
	STALE hash/fnv: stale dependency: internal/cpu
	STALE hash/maphash: build ID mismatch
	STALE html: stale dependency: internal/cpu
	STALE html/template: stale dependency: internal/cpu
	STALE image: stale dependency: internal/cpu
	STALE image/color: build ID mismatch
	STALE image/color/palette: stale dependency: image/color
	STALE image/draw: stale dependency: internal/cpu
	STALE image/gif: stale dependency: internal/cpu
	STALE image/internal/imageutil: stale dependency: internal/cpu
	STALE image/jpeg: stale dependency: internal/cpu
	STALE image/png: stale dependency: internal/cpu
	STALE index/suffixarray: stale dependency: internal/cpu
	STALE internal/bytealg: stale dependency: internal/cpu
	STALE internal/cfg: build ID mismatch
	STALE internal/cpu: build ID mismatch
	STALE internal/fmtsort: stale dependency: math/bits
	STALE internal/goroot: stale dependency: internal/cpu
	STALE internal/goversion: build ID mismatch
	STALE internal/lazyregexp: stale dependency: internal/cpu
	STALE internal/lazytemplate: stale dependency: internal/cpu
	STALE internal/nettrace: build ID mismatch
	STALE internal/obscuretestdata: stale dependency: internal/cpu
	STALE internal/oserror: stale dependency: internal/cpu
	STALE internal/poll: stale dependency: internal/cpu
	STALE internal/race: build ID mismatch
	STALE internal/reflectlite: stale dependency: internal/cpu
	STALE internal/singleflight: stale dependency: internal/cpu
	STALE internal/syscall/unix: stale dependency: internal/cpu
	STALE internal/testenv: stale dependency: internal/cpu
	STALE internal/testlog: stale dependency: sync/atomic
	STALE internal/trace: stale dependency: internal/cpu
	STALE internal/xcoff: stale dependency: internal/cpu
	STALE io: stale dependency: internal/cpu
	STALE io/ioutil: stale dependency: internal/cpu
	STALE log: stale dependency: internal/cpu
	STALE log/syslog: stale dependency: internal/cpu
	STALE math: stale dependency: math/bits
	STALE math/big: stale dependency: internal/cpu
	STALE math/bits: build ID mismatch
	STALE math/cmplx: stale dependency: math/bits
	STALE math/rand: stale dependency: math/bits
	STALE mime: stale dependency: internal/cpu
	STALE mime/multipart: stale dependency: internal/cpu
	STALE mime/quotedprintable: stale dependency: internal/cpu
	STALE net: stale dependency: internal/cpu
	STALE net/http: stale dependency: internal/cpu
	STALE net/http/cgi: stale dependency: internal/cpu
	STALE net/http/cookiejar: stale dependency: internal/cpu
	STALE net/http/fcgi: stale dependency: internal/cpu
	STALE net/http/httptest: stale dependency: internal/cpu
	STALE net/http/httptrace: stale dependency: internal/cpu
	STALE net/http/httputil: stale dependency: internal/cpu
	STALE net/http/internal: stale dependency: internal/cpu
	STALE net/http/pprof: stale dependency: internal/cpu
	STALE net/internal/socktest: stale dependency: internal/cpu
	STALE net/mail: stale dependency: internal/cpu
	STALE net/rpc: stale dependency: internal/cpu
	STALE net/rpc/jsonrpc: stale dependency: internal/cpu
	STALE net/smtp: stale dependency: internal/cpu
	STALE net/textproto: stale dependency: internal/cpu
	STALE net/url: stale dependency: internal/cpu
	STALE os: stale dependency: internal/cpu
	STALE os/exec: stale dependency: internal/cpu
	STALE os/signal: stale dependency: internal/cpu
	STALE os/signal/internal/pty: stale dependency: internal/cpu
	STALE os/user: stale dependency: internal/cpu
	STALE path: stale dependency: internal/cpu
	STALE path/filepath: stale dependency: internal/cpu
	STALE plugin: stale dependency: internal/cpu
	STALE reflect: stale dependency: math/bits
	STALE regexp: stale dependency: internal/cpu
	STALE regexp/syntax: stale dependency: internal/cpu
	STALE runtime: stale dependency: internal/cpu
	STALE runtime/cgo: build ID mismatch
	STALE runtime/debug: stale dependency: internal/cpu
	STALE runtime/internal/atomic: stale dependency: internal/cpu
	STALE runtime/internal/math: stale dependency: runtime/internal/sys
	STALE runtime/internal/sys: build ID mismatch
	STALE runtime/pprof: stale dependency: internal/cpu
	STALE runtime/pprof/internal/profile: stale dependency: internal/cpu
	STALE runtime/race: build ID mismatch
	STALE runtime/trace: stale dependency: internal/cpu
	STALE sort: stale dependency: internal/cpu
	STALE strconv: stale dependency: internal/cpu
	STALE strings: stale dependency: internal/cpu
	STALE sync: stale dependency: internal/cpu
	STALE sync/atomic: build ID mismatch
	STALE syscall: stale dependency: internal/cpu
	STALE testing: stale dependency: internal/cpu
	STALE testing/internal/testdeps: stale dependency: internal/cpu
	STALE testing/iotest: stale dependency: internal/cpu
	STALE testing/quick: stale dependency: internal/cpu
	STALE text/scanner: stale dependency: internal/cpu
	STALE text/tabwriter: stale dependency: internal/cpu
	STALE text/template: stale dependency: internal/cpu
	STALE text/template/parse: stale dependency: internal/cpu
	STALE time: stale dependency: internal/cpu
	STALE unicode: build ID mismatch
	STALE unicode/utf16: build ID mismatch
	STALE unicode/utf8: build ID mismatch
	STALE vendor/golang.org/x/crypto/chacha20: stale dependency: internal/cpu
	STALE vendor/golang.org/x/crypto/chacha20poly1305: stale dependency: internal/cpu
	STALE vendor/golang.org/x/crypto/cryptobyte: stale dependency: internal/cpu
	STALE vendor/golang.org/x/crypto/cryptobyte/asn1: build ID mismatch
	STALE vendor/golang.org/x/crypto/curve25519: stale dependency: internal/cpu
	STALE vendor/golang.org/x/crypto/hkdf: stale dependency: internal/cpu
	STALE vendor/golang.org/x/crypto/internal/subtle: build ID mismatch
	STALE vendor/golang.org/x/crypto/poly1305: stale dependency: internal/cpu
	STALE vendor/golang.org/x/net/dns/dnsmessage: stale dependency: internal/cpu
	STALE vendor/golang.org/x/net/http/httpguts: stale dependency: internal/cpu
	STALE vendor/golang.org/x/net/http/httpproxy: stale dependency: internal/cpu
	STALE vendor/golang.org/x/net/http2/hpack: stale dependency: internal/cpu
	STALE vendor/golang.org/x/net/idna: stale dependency: internal/cpu
	STALE vendor/golang.org/x/net/nettest: stale dependency: internal/cpu
	STALE vendor/golang.org/x/sys/cpu: stale dependency: internal/cpu
	STALE vendor/golang.org/x/text/secure/bidirule: stale dependency: internal/cpu
	STALE vendor/golang.org/x/text/transform: stale dependency: internal/cpu
	STALE vendor/golang.org/x/text/unicode/bidi: stale dependency: internal/cpu
	STALE vendor/golang.org/x/text/unicode/norm: stale dependency: internal/cpu

Error: tests failed: dist test failed: go_test:archive/tar: exit status 1
cherrymui

comment created time in 2 days

issue commentgolang/go

cmd/compile: consider dropping Greater* and Geq* generic ops

Canonicalization seems to be a reasonable solution. Though it does make the compiler output slightly less predictable. It does make some code slightly worse but only because sometimes it turns out it is better to keep the original duplicate comparisons than it is to CSE them and then let flagAlloc to regenerate them. Flags are clobbered frequently so we quite often end up regenerating them often when they are CSE'd.

mundaym

comment created time in 2 days

issue commentgolang/go

cmd/compile: consider dropping Greater* and Geq* generic ops

One possibility is to canonicalize the order of arguments to comparisons using something like the value IDs. I'll give that a go, see how unpleasant it is.

mundaym

comment created time in 3 days

issue commentgolang/go

cmd/compile: consider dropping Greater* and Geq* generic ops

I'm still investigating but there are some interesting things going on in the compilecmp output. The negative effects I've seen so far appear to stem from the fact that the lowered CSE pass can't merge comparison instructions unless the arguments are in the same order. This appears to happen more often after the CL since we are more likely to change the argument order when emitting a comparison instruction.

Example:

CMP R0, R1
BGT x
CMP R0, R1 // can be CSE'd
BLT y
CMP R1, R0
BLT x
CMP R0, R1 // can't be CSE'd
BLT y

I don't think this should necessarily stop us - there are some functions that benefit from the reversed order and overall the effect is probably not very noticeable - but I'll see if it can be mitigated at all.

mundaym

comment created time in 3 days

issue commentgolang/go

cmd/compile: combine int tests when comparing against a constant string

I just remembered (vaguely) that overlapping loads caused performance problems a few years ago because they interfered with load-store forwarding.

Yeah, this might be a problem. Though since Go programs don't tend to copy strings by value that much we might not see load-hit-store hazards very often. My guess would be that strings are rarely still in the store buffer when we are comparing them, but that is just a guess.

josharian

comment created time in 4 days

issue commentgolang/go

cmd/compile: consider dropping Greater* and Geq* generic ops

The CL above implements this change. I'm fairly happy with it. It doesn't seem to affect the generated code very much and the prove pass still seems to work reliably. It ends up removing about 4000 lines of code. I still need to run the benchmarks on it but if they look OK I think we should do this.

mundaym

comment created time in 6 days

issue commentgolang/go

cmd/compile: consider dropping Greater* and Geq* generic ops

Yeah, I've started playing around with it and it seems trivial to remove them (for the most part it is literally just deleting code and swapping arguments here and there). I'll put together a CL soon and we can use that for further discussion.

mundaym

comment created time in 6 days

issue openedgolang/go

cmd/compile: consider dropping Greater* and Geq* generic ops

We could, I think, fairly easily drop the following generic ops since they are trivial to implement by swapping the operands to Less* and Leq* ops (e.g. (Greater64 x y) is the same operation as (Leq64 y x)):

Greater64
Greater32
Greater16
Greater8
Greater64U
Greater32U
Greater16U
Greater8U
Geq64
Geq32
Geq16
Geq8
Geq64U
Geq32U
Geq16U
Geq8U

I think this might help make CL 165998 a fair bit simpler. Of course we could also drop the Less ops instead, but Greater is more characters :).

Before I start looking into this any further, does anyone have any initial concerns or objections that come to mind?

(Note that we can probably also do the same thing with the floating point comparisons - but those aren't necessary for the CL above).

created time in 6 days

issue commentgolang/go

cmd/compile: consider dropping Greater* and Geq* generic ops

cc @josharian @randall77 @cherrymui

mundaym

comment created time in 6 days

issue commentgolang/go

cmd/compile: combine int tests when comparing against a constant string

Tricky one. This transformation could slow down/bloat some code (generally when uint16(first two bytes)==uint16("ab") is unlikely), so while I think the backend could do it I'm not sure it should unless it has likeliness information.

Note that for longer strings we could overlap the loads given they are unaligned anyway (not sure if we do this already - I don't think we do from what I see in walk):

    x == "abcdefg" -> x[0:4] == "abcd" && x[3:7] == "defg"

I think that transformation could be applied greedily since it doesn't add any extra instructions to the 'not taken' path. It could also be applied in any situation where we want to check the last 3 characters in a string and we also know the length of the string is greater than 3 (i.e. we can access index -1 in the string and be in bounds). I suspect this crops up occasionally in string switch statement lowering. The overlapping characters could be masked out even when known to avoid double-load related issues in racy code. That said, this would all be quite hard in SSA, but maybe string comparison lowering could do it.

josharian

comment created time in 8 days

issue commentgolang/go

proposal: new cryptographic assembly policy

In general the policy seems good to me, though I do feel that if a contributor is following the current assembly policy that they would already be meeting the bar of the new one. Maybe we need to expand the advice even more, I'm not sure.

One thing that this proposal doesn't address is the fairly common practice of taking assembly from another project (e.g. OpenSSL) and porting it to Go assembly. I see this in reviews every so often. One concrete thing that I think we should encourage people to do is to document the repositories and, importantly, the exact version control revisions that their code is based on so that it is easier to check for upstream changes. Having said that, when reviewing such code do you think that it is better to encourage keeping the code close to the original so that it is easier to update with upstream changes (assuming anyone ever bothers to check for them) or to modify it more heavily to fit with the policy? Sometimes contributors don't fully understand the implementation or algorithm they are porting and so it will be very hard for them to rework it to fit with the policy. How firm should we be in that sort of situation, assuming it is otherwise well tested?

One other point I'd like to make is that I don't think we should mandate the use of code generation. It should only be used when it would increase maintainability. Short hand-written assembly files can be preferable in some situations.

FiloSottile

comment created time in 14 days

issue commentgolang/go

x/crypto/chacha20: add Cipher.SetCounter

(In the CL above the SetCounter method is renamed to Advance.)

I think if I saw s.Advance(1) in code I don't think it would be immediately obvious to me whether the method call is trying to skip 1 byte of the keystream, increment the counter by 1, or set the value of the counter to 1.

I therefore wonder if we should call this something less ambiguous. AdvanceCounterTo perhaps?

FiloSottile

comment created time in a month

issue commentgolang/go

x/crypto/chacha20: add Cipher.SetCounter

If I saw s.Advance(1) in code I don't think it would be immediately obvious to me whether the method call is trying to skip 1 byte of the keystream, increment the counter by 1, or set the value of the counter to 1.

I therefore wonder if we should call this something less ambiguous. AdvanceCounterTo perhaps?

FiloSottile

comment created time in a month

PR closed apple/swift

Update Ubuntu dependencies to include libblocksruntime-dev.

Some of the tests link against libBlocksRuntime so it appears to be a requirement. With this package installed the tests pass for me (Ubuntu 18.04 LTS on x86_64).

This StackOverflow link shows the same issue on Ubuntu 14.04: https://stackoverflow.com/questions/36479498/is-llvm-compiler-rt-installed-on-my-ubuntu-14-04

+1 -1

5 comments

1 changed file

mundaym

pr closed time in a month

PR closed apple/swift

IRGen: use APInt to represent spare bit masks for pointers

This allows us to define these masks in platform byte order so we can modify these masks without worrying about endianness.

When these marks are converted into spare bit vectors we will need to explicitly put them in little-endian byte order when targeting big-endian platforms. That will be done in a future change.

+57 -70

5 comments

7 changed files

mundaym

pr closed time in a month

issue openedgolang/go

x/crypto/sha3: add tests with assembly disabled

Both amd64 and s390x have assembly implementations of SHA-3. On amd64 the assembly is always used and the generic implementation is not tested at all. On s390x the generic implementation won't be tested if the SHA-3 CPU features are available. On both platforms we should add tests for the generic code to ensure it is properly tested.

Unfortunately due to the nature of the instructions involved they are implemented in different ways so this will take some thinking.

created time in 2 months

issue commentgolang/go

x/crypto/sha3: build failure on linux/s390x at Go master due to illegal instruction

(Note that KIMD won't give an incorrect result, it will always just SIGILL.)

toothrot

comment created time in 2 months

issue commentgolang/go

x/crypto/sha3: build failure on linux/s390x at Go master due to illegal instruction

The SIGILL was due to a bug in the way I rounded down the length of an input to a multiple of the rate suitable for the KIMD instruction. It occurs when more than 3KiB of data is written to the hash at once and the size is not a multiple of the rate. The KIMD instruction doesn't have SHA-3 support on z13 (the old buildlet machine) and the test that is failing now was added after the assembly implementation of SHA-3 was added which is why we didn't see this before.

The fix is simple but generally we should probably add better testing for generic vs. assembly implementations both for amd64 and s390x. That will take me a little while.

toothrot

comment created time in 2 months

issue commentgolang/go

x/crypto/sha3: build failure on linux/s390x at Go master due to illegal instruction

@mundaym, is this related of moving the s390x builder to z15?

Probably, I'll take a look.

toothrot

comment created time in 2 months

issue commentzos-go/go

Update Go version

@mckeederek I've recently tried rebuilding it and it worked OK for me. Can you e-mail me (you can see my e-mail here) with some more information and we can work on it there? This issue doesn't seem the right place to discuss.

bikashdube

comment created time in 2 months

issue commentzlib-ng/zlib-ng

IBM Z (s390x): DFLTCC code compilation failure

Thanks. Test failure looks the same to me.

mundaym

comment created time in 3 months

issue openedzlib-ng/zlib-ng

IBM Z (s390x): DFLTCC code compilation failure

I'm seeing a compilation failure when compiling zlib-ng configured using ./configure --with-dfltcc-deflate --with-dfltcc-inflate due to the use of an unknown variable name. The following code change appears to fix the issue:

diff --git a/arch/s390/dfltcc_deflate.c b/arch/s390/dfltcc_deflate.c
index 3710dc0..454a0bb 100644
--- a/arch/s390/dfltcc_deflate.c
+++ b/arch/s390/dfltcc_deflate.c
@@ -79,7 +79,7 @@ static inline void send_eobs(PREFIX3(streamp) strm, const struct dfltcc_param_v0
 {
     deflate_state *state = (deflate_state *)strm->state;
 
-    send_bits(state, bi_reverse(param->eobs >> (15 - param->eobl), param->eobl), param->eobl, s->bi_buf, s->bi_valid);
+    send_bits(state, bi_reverse(param->eobs >> (15 - param->eobl), param->eobl), param->eobl, state->bi_buf, state->bi_valid);
     flush_pending(strm);
     if (state->pending != 0) {
         /* The remaining data is located in pending_out[0:pending]. If someone

Can you take a look @iii-i? I'm also seeing make test fail - not sure if that is expected or not.

Thanks!

created time in 3 months

issue commentgolang/go

math: discrepancy in Atan2 special case

I'm happy to add the Atan2(±y, +Inf) = ±0 special case to the s390x implementation if that helps. I don't think we necessarily need to update the documentation for Atan2 but it would be good to add a test to enforce this special case even if we don't guarantee it externally.

bmkessler

comment created time in 4 months

issue commentgolang/go

math/cmplx: TestAtan and friends failing on several platforms

/cc @billotosyr for the s390x failures.

I'll try disabling the s390x vector implementations of math.Atan, math.Log and math.Log10 tomorrow. I suspect there could be subtle differences in there that are affecting these new tests (Atan looks like a ~1 ulp difference while the Log/Log10 tests look like zero sign differences).

mknyszek

comment created time in 4 months

issue commentgolang/go

Wrong elliptic multiplication in elliptic.go

Can you paste your C example so that people don't have to download the zip to read it? Thanks.

ndv

comment created time in 4 months

issue closedgolang/go

cmd/compile: make rulegen produce less verbose code

This is a continuation to #30810, once the rewrite has been merged. A number of ideas will be proposed below. The purpose of this is to:

  • Produce less lines of code, reducing the size of diffs and files on disk
  • Produce simpler code, making it more readable to humans
  • Hopefully reduce compile time of the generated code slightly

Here are my ideas so far:

  1. Simplifying of boolean expressions. For example, turn !(d == 32-c) into d != 32-c.

  2. Joining of contiguous break conditions, such as:

// verbose
if v_0.Op != OpAMD64SHLLconst {
        break
}
if v_0.AuxInt != 3 {
        break
}

// simpler
if v_0.Op != OpAMD64SHLLconst || v_0.AuxInt != 3 {
        break
}
  1. Removing of uninteresting lines, such as the cond line here:
// match: (ADDL x (SHLLconst [3] y))
// cond:
// result: (LEAL8 x y)
  1. Move declarations closer to their uses, reducing the number of variables in scope in some cases, such as:
for {
        carry := v.Args[2]
        x := v.Args[0] // Not actually used until the very end!
        v_1 := v.Args[1]
        if v_1.Op != OpAMD64MOVQconst {
                break
        }
        c := v_1.AuxInt
        if !(is32Bit(c)) {
                break
        }
        v.reset(OpAMD64ADCQconst)
        v.AuxInt = c
        v.AddArg(x)
        v.AddArg(carry)
        return true
}
  1. Reorder slice accesses to reduce the amount of manual bounds checks speed-ups:
// verbose
_ = v.Args[1]
x := v.Args[0]
v_1 := v.Args[1]

// simpler
v_1 := v.Args[1]
x := v.Args[0]

These are the ideas I have so far. If we think any of them would reduce readability, they can be dropped. I'm also open to more ideas and suggestions, of course. I'd send these as separate CLs so we can measure their impact on the generated code separately.

/cc @josharian @randall77 @laboger @mundaym @cherrymui

closed time in 4 months

mvdan

issue commentgolang/go

cmd/compile: make rulegen produce less verbose code

Happy to close this.

mvdan

comment created time in 4 months

issue closedgolang/go

x/build: update linux/s390x builder to use --reverse-type flag

@mundaym, as part of #21260 and deleting some legacy code in our build system, we need to update the linux/s390x builder to use the --reverse-type=host-linux-390x flag instead of --reverse=linux-s390x-ibm.

I emailed you an updated key.

Thanks!

closed time in 4 months

bradfitz

issue commentgolang/go

x/build: update linux/s390x builder to use --reverse-type flag

Thanks @bradfitz. I've updated the builder and it is now running the latest stage0 using the new key.

bradfitz

comment created time in 4 months

issue commentgolang/go

cmd/compile: consider flagging ops that should not be moved between blocks

Note that this would probably need to be a dynamic property of a value for generic ops since whether or not a certain op is in the base ISA differs between architectures. For example, PopCount8 could be guarded on one architecture and unguarded on another.

mundaym

comment created time in 4 months

issue openedgolang/go

cmd/compile: consider flagging ops that should not be moved between blocks

Some intrinsics (such as population count, FMA and rounding) are emitted by the compiler but are not present in the base ISA of the CPU the compiler is targeting. In these cases the compiler emits code to guard the execution of the instruction and fall back to a slower implementation if the required CPU feature is not present.

This technique is currently working fine but I am concerned it might not interact well with block fusing optimizations that we might add in the future (such as those mentioned in #30645) and this could lead to subtle bugs, especially since machines without some CPU features (e.g. SSE 4.1) are fairly rare these days. We already do some optimizations where we perform code movement and speculatively execute code in order to emit conditional select instructions.

I think we should consider marking these ops somehow, perhaps simply with the 'has side effects' flag. This would represent the possibility that these instructions could cause an illegal instruction exception and prevent the compiler from moving them.

The conditional select optimizations special case integer divide ops since they panic if the divisor is 0 for a similar reason: they should not be speculatively executed.

Example:

if cpu.HasPopCount {
    y = PopCount(x)
} else {
    y = GenericPopCount(x)
}

Could be transformed by the compiler into:

y = select(cpu.HasPopCount, PopCount(x), GenericPopCount(x)) 

Currently there is ~zero risk of this transformation occuring because the fall back is generally an expensive function call with side effect. However I think that is the only reason this code transformation wouldn't be applied and that seems a bit fragile.

created time in 4 months

issue commentgolang/go

crypto/ecdsa: buffers are too small, can cause panic on s390x

Thanks Filippo.

billotosyr

comment created time in 4 months

issue commentgolang/go

crypto/ecdsa: buffers are too small, can cause panic on s390x

Ok, thanks for elaborating. We are definitely reverting CL 174437 for Go 1.13. Can you make the CL?

Done, see CL 201360 for the removal in master and CL 201361 for the equivalent backport to Go 1.13.

The next version of the assembly policy (which is long overdue, and that's my fault) will explicitly require builders coverage for all assembly paths. We should not have merged that.

Understood.

We have a security release tomorrow but it's very late to add another change to it. Does this ever work on affected machines? If not, it sounds like no one will be blindsided in production, and we can put the fix in a regular minor release.

I am not 100% certain. The crypto/ecdsa tests pass on affected machines which bothers me and indicates the bug might not completely break applications. I'd like to get the revert into a Go 1.13.x release ASAP however I am not currently aware of any users that might be affected by this issue and so it may be ok to delay it until the normal minor release assuming that is fairly soon (I guess we are due the next one in the next week or two anyway?).

billotosyr

comment created time in 4 months

issue commentgolang/go

crypto/ecdsa: buffers are too small, can cause panic on s390x

If we don't get adequate test coverage & confidence of certain assembly, the fallback of course is to just revert that assembly to use the pure Go implementation. That's an option here if it comes to it.

I think that this is the best idea for Go 1.13.2. Apologies for all of this. In future we will avoid adding optimizations for unreleased hardware.

billotosyr

comment created time in 4 months

issue commentgolang/go

crypto/ecdsa: buffers are too small, can cause panic on s390x

  • What are the reproduction steps? We'll definitely need a new CL with a test.

Running the crypto/x509 tests on a z15 is enough to trigger a panic. This assembly is making use of a new instruction and so will not be executed on older machines such as the buildbot (which is a z13).

  • In what situations are the buffers overrun?

Potentially every time the KDSA instruction is executed, which will be the case when signing or verifying using P-256, P-384 or P-521.

  • What's the impact? Does the assembly ever write to memory beyond the bounds? Is the computation ever wrong but without a panic?

Yes, my understanding is that the assembly is reading and writing to memory out of bounds because the buffer is smaller than the instruction assumes it to be. If a different thread writes to this memory area then we could see the result of the KDSA call change. Overwriting the stack could modify variables in other functions in the call chain. We must therefore assume that it is possible that this could affect the result of a sign or verify call.

  • Why should we backport this? When was it introduced? How was it not noticed?

It was introduced in CL 174437 prior to the Go 1.13 release. Go 1.13 and Go 1.13.1 are affected.

The instruction was added based on a draft of the specification for the KDSA instruction which used a smaller size for the params buffer. We did not notice that this change had occurred.

  • Why did tests not catch this?

They do, but only when run on a z15. Unfortunately we were not regularly testing on z15 hardware until now. The buildbot is a z13.

In general I don't think we have a good story around testing whether assembly in Go is reading or writing memory out of bounds. Unfortunately if the buffer resides on the heap (or even on the stack) I don't think it will necessarily trigger any sort of panic or error.

  • How did you pick the new buffer sizes? Sounds like it should be a constant with a comment.

These are taken directly from the final specification for the instruction. Yes, it should probably be a named type or constant.

  • Do you think it has security implications?

Yes, I think it does. The hardware (z15) that is a pre-requisite for exposing this bug in Go is very new and has not been widely deployed yet which limits the likelihood that this is affecting a production application somewhat, but it is still a very nasty bug.

billotosyr

comment created time in 4 months

issue commentgolang/go

cmd/compile: restore walkinrange optimization (by moving to SSA)

I'd like to make block rewrite rules more powerful but I don't think I'll be able to get it done before the freeze because it is quite fiddly (happy to help if someone else wants to take a look). I'm happy to write a pass to do this specific optimization though if that would be useful. Most of the tricky stuff I worked through in the CL I posted previously.

josharian

comment created time in 4 months

issue closedgolang/go

cmd/asm: add missing s390x branch/move on condition suffixes and improve consistency

I'd like the compiler to be able to generate test-under-mask instructions (TMLL, TMLH, TMHL and TMHH) for s390x. To do this properly the SSA backend needs to be able to deal with (and be able to reverse) all 14 useful condition codes, rather than just the 9 it currently handles. We don't currently have suffixes for all these condition codes and in the process of adding them I'm wondering whether to also try and make the existing ones more consistent with their GNU assembler (GAS) equivalents.

The exact meaning of the bits set in the condition code vary from instruction to instruction. For example, bit 3 set by test-under-mask means 'selected bits are all ones'. However in general they correspond to the following (big endian, so 0 is MSB, 3 is LSB):

  1. equal
  2. less than
  3. greater than
  4. overflow (also commonly used to represent 'unordered' for floating point comparisons)

Here are my proposed changes:

<table> <tr><th>Condition Code</th><th>GAS Suffix</th><th>Current Go Suffix</th><th>Proposed Go Suffix</th></tr> <tr><td>0x0</td><td></td><td>VC</td><td></td></tr> <tr><td>0x1</td><td>o</td><td>VS</td><td>O (overflow)</td></tr> <tr><td>0x2</td><td>h</td><td>GT</td><td>GT (greater than)</td></tr> <tr><td>0x3</td><td>nle</td><td></td><td>NLE (not less than and not equal)</td></tr> <tr><td>0x4</td><td>l</td><td>LT</td><td>LT (less than)</td></tr> <tr><td>0x5</td><td>nhe</td><td>LTU</td><td>NGE (not greater than and not equal)</td></tr> <tr><td>0x6</td><td>lh</td><td></td><td>LG (less or greater than)</td></tr> <tr><td>0x7</td><td>ne</td><td>NE</td><td>NE (not equal)</td></tr> <tr><td>0x8</td><td>e</td><td>EQ</td><td>EQ (equal)</td></tr> <tr><td>0x9</td><td>nlh</td><td></td><td>NLG (not less than and not greater than)</td></tr> <tr><td>0xa</td><td>he</td><td>GE</td><td>GE (greater than or equal)</td></tr> <tr><td>0xb</td><td>nl</td><td></td><td>NL (not less than)</td></tr> <tr><td>0xc</td><td>le</td><td>LE</td><td>LE (less than or equal)</td></tr> <tr><td>0xd</td><td>nh</td><td>LEU</td><td>NG (not greater than)</td></tr> <tr><td>0xe</td><td>no</td><td></td><td>NO (no overflow)</td></tr> <tr><td>0xf</td><td></td><td></td><td></td></tr> </table>

I also like the idea of making each suffix exactly two characters long, but I suspect it is better to stick more closely to GAS. If every suffix were to be two characters, I would substitute as follows:

  • O -> OV (overflow)
  • NGE -> LO (less than or overflow)
  • NLE -> GO (greater than or overflow)
  • NLG -> EO (equal or overflow)

We'd need to decide whether to keep the current mnemonics around as aliases or not. I think we should keep BVS but the rest can probably go.

We could replace GT with H, EQ with E and so on but that feels like a step too far. These suffixes are more widely used than the overflow ones and H would conflict with our uses of H for half-word.

closed time in 5 months

mundaym

issue commentgolang/go

cmd/asm: add missing s390x branch/move on condition suffixes and improve consistency

CL 196558 adds mnemonics for these instructions that allow the condition code mask to be manually specified (e.g. rather than using the EQ suffix you can supply the equivalent condition code mask ,$8). This is just as flexible as the extended mnemonics so I don't think it is worth adding the extended mnemonics at this time.

mundaym

comment created time in 5 months

more