profile
viewpoint
遗忘

BLAKE3-team/BLAKE3 1900

official implementations of the BLAKE3 cryptographic hash function

oconnor663/bao 248

an implementation of BLAKE3 verified streaming

BLAKE3-team/BLAKE3-specs 102

The BLAKE3 paper: specifications, analysis, and design rationale

keybase/saltpack-python 80

A Python implementation of saltpack, mainly for testing. Only partial support for V2.

oconnor663/blake2_simd 78

high-performance implementations of BLAKE2b/s/bp/sp in pure Rust with dynamic SIMD

oconnor663/blake3-py 26

Python bindings for the BLAKE3 cryptographic hash function

keybase/go-framed-msgpack-rpc 6

Framed RPC for go

maxtaco/go-framed-msgpack-rpc 6

Framed RPC for go

oconnor663/blake2_c.rs 6

a safe wrapper around the BLAKE2 C implementation (deprecated in favor of blake2b_simd and blake2s_simd)

oconnor663/arch 4

Jack's install scripts for Arch Linux.

issue commentneoclide/coc-git

vim-coc gets out of sync when I make changes outside of Vim

A simpler repro for me is:

  1. Edit a file in a git repo. Don't save it.
  2. Undo the edit with :e!.

For me the ~ appears in step 1 (correctly) but remains in step 2 (incorrectly).

oconnor663

comment created time in 13 minutes

issue commentneoclide/coc-git

vim-coc gets out of sync when I make changes outside of Vim

I'm sorry, I should've done that at the beginning. This is on Manjaro Linux:

NVIM v0.4.3
Build type: Release
LuaJIT 2.0.5
Compilation: /usr/bin/cc -march=x86-64 -mtune=generic -O2 -pipe -fno-plt -O2 -DNDEBUG -DMIN_LOG_LEVEL=3 -Wall -Wextra -pedantic -Wno-unused-parameter -Wstrict-prototypes -std=gnu99 -Wshadow -Wconversion -Wmissing-prototypes -Wimplicit-fallthrough -Wvla -fstack-protector-strong -fdiagnostics-color=always -DINCLUDE_GENERATED_DECLARATIONS -D_GNU_SOURCE -DNVIM_MSGPACK_HAS_FLOAT32 -DNVIM_UNIBI_HAS_VAR_FROM -I/build/neovim/src/build/config -I/build/neovim/src/neovim-0.4.3/src -I/usr/include -I/build/neovim/src/build/src/nvim/auto -I/build/neovim/src/build/include

My vimrc is visible here: https://github.com/oconnor663/dotfiles/blob/44d325b3066fa77d395da23ed824b2ca0313aef4/vimrc And my full set of plugins is visible here: https://github.com/oconnor663/dotfiles/blob/44d325b3066fa77d395da23ed824b2ca0313aef4/peru.yaml

The plugins list is:

  • vim-airline: https://github.com/vim-airline/vim-airline/commit/768bd0f5c685f728922b98a919cb6fab30c5b4ec
  • vim-airline-themes: https://github.com/vim-airline/vim-airline-themes/commit/04fa4fc40f21d9490954213c1ee06c7fdea66a6d
  • vim-coc: https://github.com/neoclide/coc.nvim/commit/449dcad0b20080c4f58ae827146fc700287e4141
  • vim-commentary: https://github.com/tpope/vim-commentary.git/commit/f8238d70f873969fb41bf6a6b07ca63a4c0b82b1
  • vim-easymotion: https://github.com/easymotion/vim-easymotion/commit/dd7b4b526775bc8553e16bc402020573b04a948c
  • vim-fugitive: https://github.com/tpope/vim-fugitive/commit/1da7c133b109cd329060174a104e325e4d6bcc82
  • vim-rust: https://github.com/rust-lang/rust.vim/commit/0d8ce07aaa3b95e61bf319b25bb3b1a4ecc780c2
  • vim-solarized8: https://github.com/lifepillar/vim-solarized8/commit/5df7666374ead07a482605b53a0f36c27dc17e8d
oconnor663

comment created time in 32 minutes

push eventoconnor663/dotfiles

Jack O'Connor

commit sha 154abed2899357ae2f593da6ee43220cdf1695a0

check in starship.toml

view details

Jack O'Connor

commit sha 44d325b3066fa77d395da23ed824b2ca0313aef4

use blue for the prompt directory

view details

push time in 16 hours

push eventoconnor663/dotfiles

Jack O'Connor

commit sha 76cb5cb48a36942c5ed5c88c11f0f5b125f1da01

add the [<space> and ]<space> mappings from vim-unimpaired

view details

push time in 16 hours

push eventoconnor663/dotfiles

Jack O'Connor

commit sha 5a82b7f6b96a12436d37e0d43f88b0223f1327b2

a few more misspellable vim aliases

view details

push time in 16 hours

push eventoconnor663/dotfiles

Jack O'Connor

commit sha db910396e3ab63ee874a204a8ac078d3de9ee7c5

turn on powerline fonts for vim-airline

view details

Jack O'Connor

commit sha 966e17ed81241f2dc6464a8b1b4027907dceaa00

add vim-rust

view details

push time in 16 hours

issue openedneoclide/coc-git

vim-coc gets out of sync when I make changes outside of Vim

Here's an example that seems to trigger this:

  1. Edit a line in an open text file. Vim-coc puts the ~ symbol in the signcolumn.
  2. Run git reset --hard outside of Vim.
  3. Execute :e inside Vim.

Expected: The ~ should go away (eventually). Actual: The ~ remains indefinitely.

Am I wrong about the expected behavior here? Is there a way to make vim-coc poll for changes on the filesystem? What's the best way to manually "refresh" it when it gets into this state?

created time in 17 hours

PR opened starship/starship

proof of concept for showing the exit status

I'd like a way to automatically print the exit status of failed commands, in addition to the fact that they failed. Here's an initial stab at enabling something like this. (I haven't updated tests yet, so for now this change causes failures.) With this change, I add this to my config:

[character]
error_format = "[$code](red) [❯](bold red) "

Then my prompt looks like this:

image

+12 -14

0 comment

2 changed files

pr created time in 2 days

create barnchoconnor663/starship

branch : exit_code

created branch time in 2 days

fork oconnor663/starship

☄🌌️ The minimal, blazing-fast, and infinitely customizable prompt for any shell!

https://starship.rs

fork in 2 days

push eventoconnor663/dotfiles

Jack O'Connor

commit sha 0815a004f2c62abb1148dd993fa1752ed2e3090b

remove unused files and settings

view details

push time in 2 days

push eventoconnor663/dotfiles

Jack O'Connor

commit sha e1bce53d8a40154dc24f5b5483c63b3d417508b5

remove the safe paste config; `bindkey -e` has a similar effect

view details

push time in 2 days

push eventoconnor663/dotfiles

Jack O'Connor

commit sha a96d54154f2a2c96dde81b9c67a22b871af5ec45

add git fugitive

view details

Jack O'Connor

commit sha f433e9566f22b7b337536c623af4d8a189e6ab70

some bindings for coc-git

view details

push time in 2 days

push eventoconnor663/dotfiles

Jack O'Connor

commit sha 0118f817d71bd0e342e7e770ea14eb7eb23593ca

comment out cmdheight

view details

Jack O'Connor

commit sha 51e9b71f2c1e77d442479affbcc83f4d6851ac54

starship zsh

view details

Jack O'Connor

commit sha bef71362620239d19bf29c5f7ec0b1a2a4903bfa

switch alacritty.yml to UbuntuMono Nerd Font

view details

Jack O'Connor

commit sha ac60dfd64a18d8ced5dc6b54e05647eacf54813e

vim-commentary

view details

push time in 2 days

push eventoconnor663/dotfiles

Jack O'Connor

commit sha d5194b1fa4e58ee02b52f564415d9ec07bfc7c7c

avoid overriding 'gi' in coc configs

view details

Jack O'Connor

commit sha eecb28068ae9702351e028458a18d076979af8bb

add airline

view details

push time in 3 days

push eventoconnor663/dotfiles

Jack O'Connor

commit sha ec1e446089af092ae32bc10bf2d0ea1462ec8385

alias common typos

view details

push time in 3 days

push eventoconnor663/dotfiles

Jack O'Connor

commit sha d885708d08fe79e35dd193c28855163c4ce2621a

listchars

view details

push time in 3 days

push eventoconnor663/dotfiles

Jack O'Connor

commit sha 2d5ed3c6cbacac1020e161681a3e6c65968ab447

set ignorecase and smartcase

view details

push time in 3 days

push eventoconnor663/dotfiles

Jack O'Connor

commit sha d483a09797421f2c43ee1f52e512dadf9c2bf32e

clear search highlight with Ctrl-L

view details

push time in 3 days

push eventoconnor663/dotfiles

Jack O'Connor

commit sha a58c2d3ca34d2a674d2f24dfa62b0612577981de

make space the leader key

view details

Jack O'Connor

commit sha 571920884e2acf62e742ceb6e7b655b8107d6b8d

EasyMotion

view details

push time in 3 days

startedlifepillar/vim-solarized8

started time in 3 days

push eventoconnor663/dotfiles

Jack O'Connor

commit sha 24ae6da834462d7b0802904c1287d0acfb745cd5

number column and solarized colors

view details

Jack O'Connor

commit sha 001d02989cef40246fc05b6cb71493df212db5fd

add an alacritty truecolor setting for tmux

view details

push time in 3 days

startedrichardanaya/tour_of_rust

started time in 4 days

push eventoconnor663/dotfiles

Jack O'Connor

commit sha 94f82221cf621dbdb2bcb35a597319b213de10ae

dump in the entire example Vim config from CoC

view details

push time in 6 days

create barnchoconnor663/dotfiles

branch : vim-coc

created branch time in 6 days

issue commentoconnor663/duct.rs

inconsistency between Handle and ReaderHandle

Actually, it would probably be better for both to use a background thread to wait. Maybe we could avoid the implicit killing entirely.

oconnor663

comment created time in 6 days

created tagoconnor663/kangarootwelve_xkcp.rs

tag0.1.4

A Rust wrapper around the XKCP implementation of the KangarooTwelve hash function

created time in 6 days

created tagBLAKE3-team/BLAKE3

tag0.3.5

official implementations of the BLAKE3 cryptographic hash function

created time in 6 days

push eventoconnor663/kangarootwelve_xkcp.rs

Jack O'Connor

commit sha d34ed6a6a04aebece058894196d4ae40ce75bb1e

version 0.1.4 Changes since 0.1.3: - `k12sum` error output includes the filename.

view details

push time in 6 days

push eventBLAKE3-team/BLAKE3

Jack O'Connor

commit sha 7d0de7be14789924a4cfdb0e844eef3540237b2b

version 0.3.5 Changes since 0.3.4: - The `digest` dependency is now v0.9 and the `crypto-mac` dependency is now v0.8. - Intel CET is supported in the assembly implementations. - `b3sum` error output includes filepaths again.

view details

push time in 6 days

issue commentoconnor663/duct.rs

Set environment variables before running sh or sh_dangerous command

No problem! Always glad to have users :)

wickedchicken

comment created time in 8 days

issue commentbriansmith/ring

Adding deterministic SecureRandom for testing

Which ring features are you trying to make deterministic for testing purposes that you currently can't? ring::agreement? Something else?

I'm putting together some test vectors for ECDSA signing, and it would be nice to generate those deterministically.

realcr

comment created time in 8 days

issue commentkeybase/saltpack

Why base62 has been used for encoding?

The goal is to guarantee that an encoding won't be corrupted, no matter where on the internet you paste it. The number 62 is 26 + 26 + 10, that is all the lowercase letters, uppercase letters, and digits. Adding a couple more characters to reach that power of 2 is convenient from an encoding perspective, but it requires using some whitespace or punctuation. Unfortunately, for any punctuation character you might choose, there are some common sites that choose to interpret it as formatting and strip it out of text. For example, you can see what GitHub does right here to my tildes, asterisks, underscores,

equals signs,

and dashes.

marcofranssen

comment created time in 15 days

startedjamesjulich/saltpack4j

started time in 15 days

issue commentzoom/zoom-e2e-whitepaper

XSalsa20 explanation

The main reason for using XSalsa20 in the whitepaper was that it's the cipher underlying NaCl's crypto_secretbox. But as we work on the early implementation, it's looking like we'll switch this to XChaCha20. That's becoming the more widely standardized version of the cipher, and it's the recommended AEAD construction in libsodium also.

andrewaeva

comment created time in 16 days

pull request commentBLAKE3-team/BLAKE3

allow for construction of hash in const fn context

It looks like we might be at a place where the entire hash function could be implemented with const fn? Here's some similar work: https://www.reddit.com/r/rust/comments/hi11op/announcing_constsha1_a_sha1_implementation_for/

inanna-malick

comment created time in 17 days

startednushell/nushell

started time in 17 days

push eventBLAKE3-team/BLAKE3

Jack O'Connor

commit sha 2f6f56f3477321c9e4742f1c863e45f2cfcbfb5e

stop being a jerk and add the context string to test_vectors.json

view details

push time in 17 days

issue closedBLAKE3-team/BLAKE3

very high ram usage on 133GB file?

ran b3sum v0.3.4 64bit windows release, on a 143249768448 bytes file, on a laptop with 16GB ram. b3sum allocated over 13GB ram while running, and my ram usage was at 99% (im guessing it allocated as much ram as it possibly could without swapping?), is that intended behavior?

edit: also i didn't pipe it in, i gave the filename as an argument to b3sum

b3sum: b3sum v0.3.4 64bit windows release from https://github.com/BLAKE3-team/BLAKE3/releases/download/0.3.4/b3sum_windows_x64_bin.exe OS: win10 x64 version 1909 CPU: Intel Core i7-8565U, 4 cores, 4 threads (hyperthreading disabled) RAM: 16 GB total, ~13GB available (2x8GB LPDDR3-2133)

closed time in 17 days

divinity76

issue commentBLAKE3-team/BLAKE3

very high ram usage on 133GB file?

Ok cool, closing as expected behavior. Thanks for the report though, @divinity76. This will be a useful reference for people who see the same thing in the future.

divinity76

comment created time in 17 days

pull request commentBLAKE3-team/BLAKE3

C intrinsics: Use function attributes on GCC and Clang

This is admittedly confusing, but the Makefile in here is really only for internal testing, not for public consumption. I should probably put a big comment at the top or something. It's mentioned briefly in the README here. The way I see it, we don't really provide any build system, and the docs are supposed to tell you how to compile the files together (using whatever system you're already using).

k0001

comment created time in 18 days

pull request commentBLAKE3-team/BLAKE3

C intrinsics: Use function attributes on GCC and Clang

Looking at this again, I'm mildly against it. A "good citizen" caller will still need to supply file-specific compiler flags to build the intrinsics implementations correctly under MSVC. Getting rid of that step on Unix might be convenient for callers who are certain they'll never need Windows support, but it introduces some inconsistency, and it makes it more likely that callers who only test on GCC end up doing the wrong thing.

Similarly, it sounds like one of the intended benefits of this change is that callers could build the intrinsics implementations for all targets (again assuming MSVC is not a target), side-stepping the issue of selecting assembly for 64-bit vs intrinsics for 32-bit. But this is also something we probably don't want to encourage. The assembly implementations perform better (and more consistently), and we want them to be used as widely as possible.

k0001

comment created time in 18 days

issue commentBLAKE3-team/BLAKE3

Warnings during compilation with MAX_SIMD 1

This is related to #55 I think.

If adding that assert silences the compiler warning, it might be nice to add it, even if technically adds some runtime cost?

xnox

comment created time in 18 days

issue commentBLAKE3-team/BLAKE3

very high ram usage on 133GB file?

Hmm that is very suspicious. The blake3 library crate itself does very little allocation. There shouldn't be anything besides whatever memmap does, and whatever the global rayon thread pool does. The b3sum binary does some bookkeeping on top of that, like parsing command line args, but that should also be pretty trivial. I wonder if memory mapping works differently on Windows than on Unix, in some surprising way that could cause this?

Is there any way you can debug what part of the program is consuming memory? I don't know what the usual approach is for Windows tooling, but a memory profile of this would of course be super informative.

One thing I'm curious to try is hashing the same file from stdin (using <). I'll be pretty surprised if the problem persists in that mode. (Though if it doesn't persist, it would still be ambiguous whether the culprit is Rayon or Memmap or something else.)

Another thing I'd be curious to try is keeping memory mapping on, but only using one thread, like this: b3sum --num-threads=1.

divinity76

comment created time in 18 days

issue openedBLAKE3-team/BLAKE3

notes for adding ASan and UBSan testing

Reddit user /u/gzk11059 gave some very helpful pointers. Based on that thread, here's a first start on running sanitizers against the C implementation.

cd c/blake3_c_rust_bindings
CC=clang CFLAGS="-fsanitize=address,undefined -fno-sanitize-recover=all" RUSTFLAGS=-Zsanitizer=address cargo +nightly test

I can confirm that if I add some undefined behavior like this:

int signed_overflow = 2147483647;
signed_overflow++;

and then run the command above, I get a failure like this:

test test::test_compare_reference_impl ... ../blake3.c:11:18: runtime error: signed integer overflow: 2147483647 + 1 cannot be represented in type 'int'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ../blake3.c:11:18 in 

created time in 19 days

create barnchoconnor663/fsanitize_example

branch : master

created branch time in 19 days

created repositoryoconnor663/fsanitize_example

created time in 19 days

delete branch BLAKE3-team/BLAKE3

delete branch : shrink_buf

delete time in 20 days

push eventBLAKE3-team/BLAKE3

Jack O'Connor

commit sha c908847c3ff484f562226a4787bcb99778ca04c1

shrink a stack array that's twice as big as it needs to be It looks like I originally made this mistake when I was copying code from the baokeshed prototype (a274a9b0faa444dd842a0584483eae6e97dbf21e), and then it got replicated into the C implementation later.

view details

push time in 20 days

create barnchBLAKE3-team/BLAKE3

branch : shrink_buf

created branch time in 20 days

pull request commentBLAKE3-team/BLAKE3

Assembly: enable CET

I have no idea what this means. Can I leave it to you to hit the merge button whenever you think it's ready? :)

sneves

comment created time in 20 days

issue commentBLAKE3-team/BLAKE3

OID Value

I'm unfamiliar with how OIDs work in general, so I might be misunderstanding things. It looks like 1.3.6.1.4.1.1722.12.2.3 is registered as blake3 and 1.3.6.1.4.1.1722.12.2.3.8 (one extra number on the end) is registered to blake3-256. I guess I was hoping that only the former would exist? But if that clashes with how other hashes are defined, I suppose it's really not a big deal. Also am I seeing correctly that no OIDs are defined for any hash other than BLAKE2 and BLAKE3? Maybe this system is less widely used than I assumed from my initial impression?

dlegaultbbry

comment created time in 21 days

push eventBLAKE3-team/BLAKE3

Jack O'Connor

commit sha e0f193ddc9c0400262dae4118b0660ae8d70586e

put the file name in b3sum error output This was previously there, but got dropped in c5c07bb337d0af7522666d05308aaf24eef3709c.

view details

push time in 22 days

push eventoconnor663/kangarootwelve_xkcp.rs

Jack O'Connor

commit sha d3bef9a75daa2924968198bf01352c1326a06888

put the file name in k12sum error output This was previously there, but got dropped in a305b316811ed682a169738b9c1f409df33286db.

view details

push time in 22 days

issue commentBLAKE3-team/BLAKE3

OID Value

Apologies for not chiming in on this sooner. There isn't a lot of established practice on this yet, but I've argued for avoiding labels like blake3-256, in favor of just calling it blake3 as much as possible. I wrote out a longer version of my reasoning here: https://github.com/BLAKE3-team/BLAKE3-specs/issues/3. Here's a shorter version:

  • "BLAKE3-128" is not domain-separated from "BLAKE3-256". The former is just a truncated version of the latter. This is in contrast to BLAKE2, where shorter output lengths were domain-separated.
  • "BLAKE3-512" offers no additional security over "BLAKE3-256". This is in contrast to BLAKE2b-512, which does offer more security than BLAKE2b-256. (Note that BLAKE3 is derived from BLAKE2s, and there is no such thing as BLAKE2s-512.)

The vast majority of applications using BLAKE3 should stick to the default output length. I think truncated BLAKE3 should be about as common as truncated MD5. Extended outputs are useful in niche applications, like deriving very long or oddly-sized keys for uncommon algorithms, but again not something most applications should think about using. (And because different output lengths aren't domain separated, there's no real sense in which different lengths represent different algorithms.)

As far as I can tell, hash algorithm sets with multiple labeled sizes are only as common as they are because of old government standards. BLAKE3 tries as much as possible to remove unnecessary choices. We've avoided concerning the caller with details like the word size or the parallelism degree, and it would be nice not to concern them with output size either.

dlegaultbbry

comment created time in 23 days

pull request commentBLAKE3-team/BLAKE3-specs

Document OID identifier

The same OID can be used for both keyed and unkeyed hashing since in the latter case the key simply has zero length.

That doesn't sound right to me. The key is essentially the first 8 of 16 initial state words. In the unkeyed (default) mode, it's set to a constant. But its length never changes; it's always 8 words / 32 bytes / 256 bits.

xnox

comment created time in 23 days

issue commentBLAKE3-team/BLAKE3

very high ram usage on 133GB file?

Are you sure b3sum actually allocated that space? Does it all go away when b3sum exits, or does utilization remain high? What b3sum is supposed to be doing is memory mapping the file, so the kernel should be using as much memory as possible to cache file pages. On Linux, I think that shows up as a lot of memory "in use" but still "available", which can be a source of confusion. It can look like the machine is low on RAM, when in fact the kernel is able to reclaim those available pages whenever it needs to. However, I don't know how these things get reported on Windows, so I don't know the right questions to ask here.

divinity76

comment created time in 23 days

issue commentPyO3/maturin

make it harder to do the wrong thing with manylinux?

I think you need to be careful not to do something that's going to crash in a non-glibc environment. The standard library uses weak! linking for this, which I'll admit I don't understand very well.

oconnor663

comment created time in 23 days

startedKitware/CMake

started time in 23 days

startedalexcrichton/cmake-rs

started time in 24 days

pull request commentBLAKE3-team/BLAKE3

c: Implement ifunc based dispatcher

as dispatcher code is only executed once per application lifetime. Instead of on every invocation.

Might be worth clarifying, our dispatcher code today only executes once per process. Subsequent calls use the cached result: https://github.com/BLAKE3-team/BLAKE3/blob/master/c/blake3_dispatch.c#L85-L87

xnox

comment created time in 25 days

pull request commentBLAKE3-team/BLAKE3

Neon detection

I have a collection of Raspberry Pis sitting around that I can test this on. Let me know when this is ready for a lot of manual testing. Also apologies in advance for being slower to respond these days -- I'm in a busy period with a new job.

xnox

comment created time in 25 days

pull request commentBLAKE3-team/BLAKE3

c: Implement ifunc based dispatcher

@sneves I'll defer to you on this one. @xnox have you been able to measure a performance improvement from using ifuncs? I've always assumed that our dispatching overhead was in the noise, but I don't have much data on it. (Compiling with RUSTFLAGS="-C target-cpu=native" on my AVX2 laptop does show a few cycles of improvement per compression, but I don't know how many other effects it has besides optimizing out the dispatch.)

xnox

comment created time in 25 days

pull request commentBLAKE3-team/BLAKE3

Neon detection

Nice! This would close out https://github.com/BLAKE3-team/BLAKE3/issues/21. I think statically enabling NEON for AArch64 is great. Does AArch32 also always support NEON/ASIMD, and is there any way for us to detect AArch32 in stable Rust?

I'm not very familiar with ARM stuff in general, but my impression from StackOverflow is that there's no OS-independent way to detect NEON support on ARMv7. If I'm reading correctly, it looks like here we're only doing dynamic detection on Linux. Is that also going to work on Android? (The SO answer linked there suggests that Android needs a different method.) On the other hand, maybe Android is always going to be AArch64 anyway these days?

xnox

comment created time in 25 days

pull request commentBLAKE3-team/BLAKE3

implement `Hasher` trait for `Hasher`

Now that I understand better what's going on here, my biggest reservation is that std::hash::Hasher limits its output to 8 bytes. That's too small for most cryptographic purposes. (Though it works well for hash tables, which I think was the main (only?) design goal for the trait.) I think there's a real risk that users might not realize this limitation, and might think that the advertised cryptographic properties of BLAKE3 apply even when it's used through this interface, which could lead to security issues.

This trait makes it possible to hash any type implementing Hash with the blake3::Hasher, so one could for example create a blake3 hash for a struct, without converting it to bytes first:

Coincidentally, we've been thinking a lot about this exact problem at work. Uncanny timing! I think the problem of hashing structured data is really interesting and useful but also really subtle. (There must be rigorous papers on this, but I don't know of any. If anyone reading along can link to research, please do.) My tldr on this is that I think there are some commonly expected properties that the Rust standard library's approach is missing, so "do what the Rust standard library does" is unlikely to be a good solution.

Diving into that, here are some example questions:

  • Is the std::hash::Hash serialization of a structure guaranteed to be consistent across different machines? This is a requirement for any hash that might get published, for example by saving it in a file or writing it to a database. I think the standard library's implementation does not have this property. For example, integers are currently hashed according to their native endianness. Also the hash of a usize depends on the word size of the machine.
  • Is std::hash::Hash serialization guaranteed to be stable across different versions of the Rust compiler? This is also a requirement for any hash that might get published. I'm not aware of any cases where it's changed in practice, but also I don't think we have any guarantee that it won't.
  • What does it mean to hash an unordered container like a HashMap? Is it disallowed? Is it nondeterministic? Or does it imply sorting (and therefore an Ord requirement on the contents)? I think the standard library disallows this by not implementing std::hash::Hash for such containers.
  • What changes to a struct definition will change its hash? Changing its name? Changing the names of its fields? Changing the order of its fields? Can two structurally distinct objects give the same hash? Is there any way to extend a struct with new optional fields, while preserving its hash in cases where those fields are unset? (I believe Rust's answers are no, no, yes, yes, no.)

That very last question in particular, about whether new fields can be added in a backwards-compatible way, is a central design goal of Protobuf and a very useful property of many other serialization formats like JSON and MessagePack. I think that an Industrial Strength™ design for hashing structured data will need to have a take a position on this, and I think users are likely to need to know the details of that position and design around them.

Luro02

comment created time in 25 days

issue commentjedisct1/libsodium

thinking about alternatives to sodium_init

Still thinking about this: The considerations you mentioned are a strong argument for supporting explicit initialization, but I don't know if that's the same thing as requiring explicit initialization. Could it be the best of both worlds to allow sodium_init to work as it does today, but to also implicitly call sodium_init the first time we access the global PRNG, architecture-specific function pointers, or other shared state? I think the biggest benefit would be a cleaner story for depending on libosdium in library code, but making libsodium thread-safe in all cases also seems like a safety win.

I wonder how many applications in the wild today are getting this wrong. Do you think conjuring up some data or examples might be useful?

oconnor663

comment created time in a month

issue commentjedisct1/libsodium

thinking about alternatives to sodium_init

With gcc/clang/icc, your library can use a library constructor to initialize itself and its dependencies. You can leverage this to call sodium_init().

That's a good idea, I'll give it a shot. Thanks for the quick response.

oconnor663

comment created time in a month

issue openedjedisct1/libsodium

thinking about alternatives to sodium_init

As I get more experience working with libsodium, I find myself wishing that sodium_init was implicit and automatic rather than explicit. The biggest issue it causes for me is that it leaks through any abstraction created on top. For example, if I want to build a hash table library using crypto_shorthash, my library now needs to expose and document a global init function that wraps sodium_init. Another issue I've run into is that, counterintuitively, most of libsodium seems to work just fine if you don't call sodium_init. That makes me worry that lots of callers simply ignore the requirement, which I think exposes them to obscure race conditions on non-Posix platforms that they're very unlikely to uncover in testing.

Have you considered checking for initialization implicitly in every API function? (Or at least, at every point where we reach for a global data structure.) I assume the best way to do this would be with some global atomic flag, possible wrapped up in a "once" sync primitive, with the cost of checking it being a single read-acquire in the common case. I think that's cheap enough that it would disappear into the noise compared to e.g. computing a single block of ChaCha20. Is that right in your experience? Thinking about other problems: Does libsodium need to support platforms that haven't implemented C11 standard atomics? Would it be an interrupt-safety issue to take a lock implicitly? What else might I be missing?

created time in a month

push eventoconnor663/dotfiles

Jack O'Connor

commit sha 72a4f001663d4f12051d4e0e61b16e6e0bec7a64

alias :E to :e

view details

push time in a month

issue commentoconnor663/duct.rs

Support for process creation flags on Windows

Could you handle this case with the before_spawn method? It's intended as kind of a catch-all for these sorts of low-level operations.

Boscop

comment created time in a month

issue commentjedisct1/libsodium

Clarify/document in-place crypto_box_seal

Related to this, is it intended for crypto_aead_xchacha20poly1305_ietf_decrypt_detached (and other detached AEAD APIs) to be safe to use in-place? It looks in the code like they should be, but it's not documented the way it is for crypto_secretbox_easy. Actually, even crypto_secretbox_detached isn't documented to be safe to use in-place.

roshanr95

comment created time in a month

issue commentoconnor663/duct.rs

Set environment variables before running sh or sh_dangerous command

duct_sh is intended to be used with all the underlying features of duct, including env vars. The sh and sh_dangerous functions return a duct::Expression, so everything should Just Work. Could you give it a try and let me know if it works for you?

wickedchicken

comment created time in a month

issue commentoconnor663/kangarootwelve_xkcp.rs

`k12` crate

Oh it's probably this: https://github.com/RustCrypto/traits/blob/master/digest/src/dev.rs#L228-L230

The benchmarks only call update, but because this implementation defers all of its work to finalization, this is basically just benchmarking extend_from_slice :)

tarcieri

comment created time in a month

issue commentoconnor663/kangarootwelve_xkcp.rs

`k12` crate

Thanks for the heads up. I don't think I'll have bandwidth for that, but I'd be happy to hop on a call or whatever with anyone who wants to talk about the design. Incidentally, I should admit that I've never actually implemented K12 myself :) I've only wrapped the upstream implementation.

By the way, those benchmark figures are surprising. How is it possible to run the compression function in 9 ns? Am I misinterpreting what an iteration means? :)

tarcieri

comment created time in a month

pull request commentBLAKE3-team/BLAKE3

Vulkan implementation of b3sum

Ah well, it was exciting for a moment there :) It's going to be awesome when this works!

Apologies for being less active these days. I'm a month into my new job at Zoom, and I don't have as much time for side projects as I did before.

cesarb

comment created time in a month

issue openedoconnor663/bao

support larger "chunk groups" for reduced space overhead

The chunk size of BLAKE3 is fixed for all of time at 1024 bytes. However, there's nothing stopping Bao from using larger "chunk groups" on the wire and on disk. For example, Bao could work with chunk groups of 64 KiB. Internally, each group would still need to be hashed in 1024-byte chunks, so CPU performance would be the same as before. (And crucially, the BLAKE3 root hash would also be unchanged.) However, the space overhead of storing and transmitting interleaved parent nodes would be reduced. This is the difference between 6% space overhead, which is too much for some applications, and 0.1% space overhead, which presumably no one cares about.

Although all root hashes would still be standard BLAKE3, the encoding formats of Bao clients using different chunk group sizes would not be compatible. A big open question is how configurable this should be: Can we document clearly enough that it's a compatibility issue? And what should the default be?

created time in a month

push eventoconnor663/hashes

Jack O'Connor

commit sha 8ca8053f5929fff3ec3aa63d7617b3e7b8eb4ad9

remove the simd/simd_opt/simd_asm Cargo features for BLAKE2 On x86 targets, SSE4.1 and AVX2 implementations are always compiled. With the `std` feature enabled, runtime CPU feature detection is used to select between them. With `std` disabled (e.g. --no-default-features), the only way to activate SIMD is something like export RUSTFLAGS="-C target-cpu=native"

view details

push time in a month

push eventoconnor663/hashes

Jack O'Connor

commit sha 5b06106ad7240b4a14d8a9ff20c8847f8162b7aa

use blake2b_simd and blake2s_simd internally Replace the internal implementation of BLAKE2b and BLAKE2s with calls to the blake2b_simd and blake2s_simd crates. Those crates contain optimized implementations for SSE4.1 and AVX2, and they use runtime CPU feature detection to select the best implementation. Running the long-input benchmarks on an Intel i9-9880H with AVX2 support, this change is a performance improvement of about 1.5x for BLAKE2b and 1.35x for BLAKE2s. This change deletes the undocumented `with_parameter_block` method, as the raw parameter block is not exposed by blake2b_simd or blak2s_simd. Callers who need BLAKE2 tree mode parameters can use the upstream crates directly. They provide a complete set of parameter methods. This change also deletes the `finalize_last_node` method. This method was arguably attached to the wrong types, `VarBlake2b` and `VarBlake2s`, where it would panic with a non-default output length. It's not very useful without the other tree parameters, so rather than moving it to the fixed-length `Blake2b` and `Blake2s` types where it belongs, we just delete it. This also simplifies the addition of BLAKE2bp and BLAKE2sp support in the following commit, as those algorithms use the last node flag internally and cannot expose it.

view details

Jack O'Connor

commit sha 69eff42dbcade6944a51be48f69f5f622281aa6e

add support for BLAKE2bp and BLAKE2sp On an Intel i9-9880H with AVX2 support, both BLAKE2bp and BLAKE2sp are about 1.75x faster than BLAKE2b. Note that while these algorithms can be implemented with multi-threading, these implementations from blake2b_simd and blake2s_simd are single-threaded, using only SIMD parallelism. The blake2b_simd and blake2s_simd crates don't support salting or personalization for BLAKE2bp and BLAKE2sp, so the `with_params` methods are moved out into blake2b.rs and blake2s.rs.

view details

push time in a month

pull request commentRustCrypto/hashes

switch the BLAKE2 implementation to blake2b_simd/blake2s_simd

I've rebased (more like rewritten) the branch on top of current master. As before, I've added support and benchmarks for BLAKE2bp and BLAKE2sp, but I haven't yet added new test vectors for those. There's some complexity due to the fact that my *_simd crates don't support personalization or salting for BLAKE2bp or BLAKE2sp. (I've never seen any implementation that does, and I'm not aware of any test vectors covering those cases, official or otherwise. So I'm hesitant to "innovate" in that direction.)

oconnor663

comment created time in a month

push eventoconnor663/hashes

Jack O'Connor

commit sha 0a91489d5333c73ad9ee52c0ca461ce0efa7488e

add support for BLAKE2bp and BLAKE2sp On an Intel i9-9880H with AVX2 support, both BLAKE2bp and BLAKE2sp are about 1.75x faster than BLAKE2b. Note that while these algorithms can be implemented with multi-threading, these implementations from blake2b_simd and blake2s_simd are single-threaded, using only SIMD parallelism. The blake2b_simd and blake2s_simd crates don't support salting or personalization for BLAKE2bp and BLAKE2sp, so the `with_params` methods are moved out into blake2b.rs and blake2s.rs.

view details

push time in a month

push eventoconnor663/hashes

bold

commit sha 0ab3dba8023e1aa01d551c5aa1b2ed0f9ed04872

Add shabal hash

view details

bold

commit sha f4834ae5424f77d29d16595633ff0db771ee6fac

Remove unused unsafe

view details

bold

commit sha 2ce0db9759c474afbf1844325cf921020db77a03

Make it run with rust 1.21

view details

newpavlov

commit sha 7f3080be055b2feef44c408870654619eb14524f

fix block sizes for blake2 hashes

view details

Jack Lloyd

commit sha 9695573914ad96f234ecabf9ecee2dbb3a6a36ed

Fix an overflow in Streebog causing panic or incorrect output (#91)

view details

newpavlov

commit sha 18a28655d57b9e6a77c6d6b80299bfefc5f069e0

add test vector for streebog overflow bug

view details

Rodolfo Araujo

commit sha ab72529a170e8b84a4cd8bdd1e1f826419620afe

Small code improvements.

view details

Rodolfo Araujo

commit sha 0f29b6cd8d604623f9cdfe7c7f14395190b03ac8

Small code improvements.

view details

linkmauve

commit sha 9edb0a53f7cfd6e89e7a14becc08855e258d2299

Enable sha1 and sha2 AArch64 extensions from asm-hashes (#97)

view details

newpavlov

commit sha 3d11a0f248f5333c10de467716c1510c5ce45658

remove benchmarks

view details

newpavlov

commit sha da2452e9de57e9e4ac40c8dc18b81afd3d223219

bump sha1 and sha2 versions

view details

Rodolfo Araujo

commit sha 37c9ccd321ee0efa34bf276eee2403478a0e46f1

Merge branch 'master' into code_imp

view details

Rodolfo Araujo

commit sha 1bdada7ded792205a132f41a23cac36568f2579d

Fix build.

view details

Rodolfo Araujo

commit sha 5416689af01c5dd0fa9ad8af57c2370db38dd9b2

Matching formatter.

view details

Emmanuel Gil Peyrot

commit sha 3f9b92663c81b9495eb61e6847406e7922575210

Allow compile-time detection of crypto on AArch64 The existing way depends on the libc crate, only works on Linux, and does a getauxval() call each 64 bytes of input. This commit adds support for doing this detection at compile-time, if the target_feature crypto is enabled we can use the same code-path as amd64 and reuse the asm feature. A new compile_error!() has been added helping users figure out the best way to build this crate.

view details

Tony Arcieri

commit sha 87d1cb01d7c0541c82cb34be069dae6a069f491e

Merge pull request #104 from linkmauve/compile-time-aarch64-crypto Allow compile-time detection of crypto extension on AArch64

view details

Deirdre Connolly

commit sha f9b35c0d0152c9afea728903e847a3c6275a39e9

Expose sha256_utils with the utils feature flag

view details

Deirdre Connolly

commit sha 789e705e61328b8ac1c0389036f48223ed6e1710

utils feature has no dependencies

view details

Deirdre Connolly

commit sha a968a8db2bffae75fbb31596897be9088ddbb821

Coalesce sha256_util export

view details

Deirdre Connolly

commit sha c1afa054b869f26f42a1585b7855c22c206e9642

Rename flag to 'compress', export just compress* instead of whole module

view details

push time in a month

create barnchoconnor663/hashes

branch : blake2_simd_old

created branch time in a month

pull request commentBLAKE3-team/BLAKE3

Bump digest crypto-mac

My worry is that an automatic upgrade to the next version of blake3 could break some callers, if they're relying on the traits we re-export? Maybe this is an argument for moving the trait implementations out to a wrapper crate (defining a trivial wrapper type to implement traits on), where it would make more sense to do a backwards-incompatible version bump every time the trait does? Not sure.

Stupremee

comment created time in a month

pull request commentBLAKE3-team/BLAKE3

Bump digest crypto-mac

Thank you! I've followed up with 4c41a893a00a3ebe7b24529531ccf96d8593a57c. As far as releasing this, do you think it should be a backwards-incompatible version bump? What do most other crates do?

Stupremee

comment created time in a month

push eventBLAKE3-team/BLAKE3

Jack O'Connor

commit sha 4c41a893a00a3ebe7b24529531ccf96d8593a57c

a little bit of cleanup and more testing

view details

push time in a month

push eventBLAKE3-team/BLAKE3

Justus K

commit sha 7eea9b4c75e19588bbaf1a0996f228b81cb4531a

Bump digest to 0.9.0 and crypto-mac to 0.8.0

view details

Justus K

commit sha 1ecb14ce34790bb7b882449c77c36131af61add6

Replace std::io::copy with clone_from_slice

view details

push time in a month

PR merged BLAKE3-team/BLAKE3

Bump digest crypto-mac

This PR bumps digest to 0.9.0 and crypto-mac to 0.8.0

Resolves #89

+44 -27

0 comment

2 changed files

Stupremee

pr closed time in a month

issue closedBLAKE3-team/BLAKE3

Bump digest to v0.9; crypto-mac to v0.8

Hello! Thanks for BLAKE3!

Heads up, we've released new versions of digest and crypto-mac. They now use generic-array v0.14 with an MSRV of 1.41+. We've also moved to Initialize-Update-Finalize (IUF) terminology within the API.

Release notes:

  • digest v0.9: https://github.com/RustCrypto/traits/pull/186
  • crypto-mac v0.8: https://github.com/RustCrypto/traits/pull/169

closed time in a month

tarcieri

issue openedbriansmith/ring

ring is affected by a nightly Cargo regression

Ring seems to have the same issue as e.g. the remove_dir_all crate had, with the most recently nightly toolchain:

error: failed to download `ring v0.16.14`

Caused by:
  unable to get packages from source

Caused by:
  failed to parse manifest at `/home/jacko/.cargo/registry/src/github.com-1ecc6299db9ec823/ring-0.16.14/Cargo.toml`

Caused by:
  readme file with name 'doc/link-to-readme.md' was not found

This is tracked at rust-lang/cargo#8351 and already fixed by rust-lang/cargo#8353. However, to unbreak callers until the next nightly goes out, it might be nice to fix the error downstream.

created time in a month

issue commentXAMPPRocky/remove_dir_all

build error in CI, possibly due to this crate? unclear?

I just filed https://github.com/rust-lang/rust/issues/73282 upstream.

oconnor663

comment created time in a month

issue openedrust-lang/rust

nightly Cargo fails on missing README.md, breaks `remove_dir_all` crate and all transitive callers

Here's the downstream issue: https://github.com/XAMPPRocky/remove_dir_all/issues/19

This issue began very recently. It repros on 1.46.0-nightly (a37c32e2d 2020-06-11), but it did not repro on rustc 1.46.0-nightly (feb3536eb 2020-06-09). Missing components in that latest build mean that I think only people running rustup set profile minimal are seeing the break currently, but it will spread with the next complete nightly build.

In this case, it think the remove_dir_all crate will be able to publish a new patch version that includes a fix. However, this won't help binary callers that have committed a Cargo.lock file. This Cargo behavior change might be a stability hazard.

created time in a month

issue commentXAMPPRocky/remove_dir_all

build error in CI, possibly due to this crate? unclear?

I just confirmed that adding this to my Cargo.toml fixes the problem:

[patch.crates-io]
remove_dir_all = { git = 'https://github.com/XAMPPRocky/remove_dir_all' }

Based on that, could you publish current master as a v0.5.3 release? I think that would fix most callers automatically.

oconnor663

comment created time in a month

issue commentXAMPPRocky/remove_dir_all

build error in CI, possibly due to this crate? unclear?

Ok, now I see why I wasn't repro'ing this locally. I was using the default rustup profile, so rustup update was doing this:

info: syncing channel updates for 'nightly-x86_64-unknown-linux-gnu'                                                                                                                           
info: latest update on 2020-06-12, rust version 1.46.0-nightly (a37c32e2d 2020-06-11)                                                                                                          
info: skipping nightly which is missing installed component 'rustfmt-preview'                                                                                                                  

But on CI, I was doing rustup set profile minimal, so I was able to pick up that build. Presumably in a few days when a more complete nightly build comes out, everyone running the default profile will start seeing this issue on nightly too.

oconnor663

comment created time in a month

issue openedXAMPPRocky/remove_dir_all

build error in CI, possibly due to this crate? unclear?

https://github.com/oconnor663/blake2_simd/runs/764237325

In that run, I see this failure in one of my sub-crates:

/usr/share/rust/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/bin/cargo test

    Updating crates.io index
 Downloading crates ...
error: failed to download `remove_dir_all v0.5.2`

Caused by:
  unable to get packages from source

Caused by:
  failed to parse manifest at `/home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/remove_dir_all-0.5.2/Cargo.toml`

Caused by:
  readme file with name 'README.md' was not found
Command failed :(

In my CI setup, it only fails on the nightly compiler, but it seems to reliably fail with nightly on every OS target. However locally, I can't seem to reproduce the failure at all.

Any chance you recognize this error message? I just have no idea what could really be causing it.

created time in a month

push eventoconnor663/blake2_simd

Jack O'Connor

commit sha 008a96ee36bce91924363fab5a5b89dcbb14551f

add some crates.io search keywords

view details

push time in a month

pull request commentoconnor663/blake2_simd

Simple Fuzzing

Apologies, I dropped the ball on this one and didn't get around to pushing the merge button until now.

kirk-baird

comment created time in a month

delete branch oconnor663/blake2_simd

delete branch : kirk-baird-fuzz

delete time in a month

more