bossmc ( Andy Caldwell )

I've updated the notes for the CHANGELOG in the main PR description.

bossmc

comment created time in 6 days

pull request commentsoftprops/shiplift

Segfault in _Backtrace_Unwind

Cool, thanks @softprops!

bossmc

comment created time in 6 days

issue commentrust-lang/rust

The setup that's needed is:

A large LOAD segment (the one containing .text and .eh_frame)
- Specifically the .eh_frame section needs to be deep into the segment
- How deep it has to be is roughly determined by the first u32 of the .gcc_except_table section

So something like the following should trigger the fault:

#[derive(Clone, Copy)]
struct Foo {
    array: [u64; 10240],
}

impl Foo {
    const fn new() -> Self {
        Self {
            array: [0x1122_3344_5566_7788; 10240]
        }
    }
}

static BAR: [Foo; 10240] = [Foo::new(); 10240];

fn main() {
    let bt = backtrace::Backtrace::new();
    println!("Hello, world! {:?}", bt);
    println!("{:x}", BAR[0].array[0]);
}

This builds a huge .rodata section, which lives before the .eh_frame section so should lead to the crash.

Hedging words because I can't reproduce the issue on my development machine (I run out of RAM and get OOM-killered).

bossmc

comment created time in 19 days

pull request commentbossmc/pinboard

Update crossbeam-epoch requirement from 0.7.0 to 0.8.0

bors r+

dependabot-preview[bot]

comment created time in a month

push eventbossmc/crossbeam

commit sha aa7d12b42ec247cf4600132a1d8f4680c705cd69

Enable select_loop tests in lib.rs

commit sha 7ff83848d3967f688a144a7374b43c3c3d91cf47

Remove eval prefix

commit sha 70d0815975e3b77fe243401868b80a4b04a32dba

Bring back send with :ident

commit sha 20bc84f1c6b9e33846b4920ebe4e815426e16e37

More documentation

commit sha cfe4240085ad4c7f63c3b6d8d99190e82f7ec2ad

Update metadata

commit sha ebad07609e959ea69105a11bcc801459d82675e6

Update readme

commit sha 9f56a4b415f2d012f3b36d666a839093c5efb371

Rename the crate to crossbeam-channel

commit sha 2d86582c32f63430f45ab9d4b924baabbcf65368

Update readme

commit sha 33dfe5e335f2345b7df1977cb2d5cb1e16bd21e6

Use the master branch of crossbeam-epoch

commit sha f6d250436f2b4895d5d05383afe1700b4269c43e

Update benchmarks

Amanieu d'Antras

commit sha f2da30085360c5122e12878fb232239c4cc7140a

Add the ability to temporarily unpin a Guard (#45) * Minor code cleanup * Add the ability to temporarily release a Guard * Use scopeguard instead of a custom PanicGuard type * Rename unpin_for to repin_after

Amanieu d'Antras

commit sha 9383117fb4af2bf1472453c19209b1f65fb6bcb3

Add no_std support (#3)

commit sha d2a9f2b976b5d2ab58b2b533f5271a9c10a75f0a

Add categories and keywords

commit sha 28c8797216a5619e1e732b8969a4a1005c1b48fa

Bump version to 0.2.1

Jeehoon Kang

commit sha 983b33c4fa9d4631f65e5131abc0e93b81ab5bc3

Implement Guard::safepoint() (#43) * Implement Guard::safepoint() * Add a comment in safepoint() * Remove the SeqCst fence from repin() * Rename safepoint() into repin() * Fix a glitch in documentation

commit sha 004a124887d4e1dd5e8f121d1c16edc03190fc21

Add a changelog for the 0.1.0 release (#40)

Amanieu d'Antras

commit sha 75394e9dd35ecf160d8166060fa85e286f01094e

Update crossbeam-util version to 0.2 for no_std support (#48) * Update crossbeam-util version to 0.2 for no_std support * Enable crossbeam-utils/use_std for tests

Jeehoon Kang

commit sha 98e8d44a446a6c0171abfaeb3d74f05009e4cf96

Remove an warning on unused unsafe (#49)

commit sha 19eafc9512200198b49f2702b21036af3d984b16

Add a thread sanitization test (#39) * Add sanitization tests * Get rid of recursive thread spawning

commit sha 3ba4a22816f811ac08dcf5294fcf345fcc85f039

Switch to crossbeam-epoch 0.1.0

Undefined reference to `_Unwind_Resume`

push time in a month

delete branch bossmc/crossbeam

delete branch : revert-local

delete time in a month

delete branch bossmc/crossbeam

delete branch : linked-list

delete time in a month

issue commentrust-lang/rust

More analysis... looking at the LLVM-IR, same source code, new command rustc +nightly foo.rs -C panic=abort -C link-args=-nostartfiles --emit llvm-ir.

...
; core::result::Result<T,E>::unwrap
; Function Attrs: inlinehint nounwind nonlazybind
define internal void @"_ZN4core6result19Result$LT$T$C$E$GT$6unwrap17h1179f71b032e9d7eE"(i64, i64) unnamed_addr #0 personality i32 (...)* @rust_eh_personality {
start:
  ...
bb5:                                              ; preds = %start
  ...
  ; invoke core::result::unwrap_failed
  invoke void @_ZN4core6result13unwrap_failed17hfb1545ecedec37d1E([0 x i8]* noalias nonnull readonly align 1 bitcast (<{ [43 x i8] }>* @0 to [0 x i8]*), i64 43, {}* nonnull align 1 %22, [3 x i64]* noalias readonly align 8 dereferenceable(24) bitcast ({ void (i64*)*, i64, i64, i1 (i64*, %"core::fmt::Formatter"*)* }* @vtable.0 to [3 x i64]*))
          to label %unreachable unwind label %cleanup
...

So it seems that the libcore rlib was built with unwind on panic and thus the expanded template includes the landing pads (and thus implicit references to the stack unwinding infrastructure).

I also tested with xargo (to rebuild libcore) RUSTFLAGS='-C link-arg=-nostartfiles -C panic=abort' xargo run (I had to pass panic=abort in RUSTFLAGS, as [profile.dev] doesn't seem to be applied to core) and the link was successful.

I'm not sure what the right answer is here, either require compiling ones own (non-unwinding) libcore, or rustup shipping two (or more) libcores, or deferring codegen for unwinding until the final build.

jefftime

comment created time in a month

issue commentrust-lang/rust

Undefined reference to `_Unwind_Resume`

Updated test case:

#![no_std]
#![no_main]

#[panic_handler]
pub fn panic(_info: &core::panic::PanicInfo) -> ! {
    loop {}
}

#[no_mangle]
pub extern "C" fn _start() -> ! {
    let a: Result<(), usize> = Err(42);
    a.unwrap();
    loop {}
}

Compile with rustc foo.rs -C panic=abort -C link-args=-nostartfiles :-

-C panic=abort - disable unwind handling (in theory)
-C link-args=-nostartfiles - Don't include crt0/crt1/etc. as I'm not linking libc at all

error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-L" "/home/andy/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "foo.foo.7rcbfp3g-cgu.0.rcgu.o" "foo.foo.7rcbfp3g-cgu.1.rcgu.o" "foo.foo.7rcbfp3g-cgu.2.rcgu.o" "foo.foo.7rcbfp3g-cgu.3.rcgu.o" "-o" "foo" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro" "-Wl,-znow" "-nodefaultlibs" "-L" "/home/andy/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/home/andy/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-837ca740df32db0a.rlib" "/home/andy/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-db27c965e824589f.rlib" "/home/andy/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-68a4f8466685ed76.rlib" "-nostartfiles" "-Wl,-Bdynamic"
  = note: foo.foo.7rcbfp3g-cgu.3.rcgu.o: In function `core::result::Result<T,E>::unwrap':
          foo.7rcbfp3g-cgu.3:(.text._ZN4core6result19Result$LT$T$C$E$GT$6unwrap17hc6a8d8550c5348d4E+0x20): undefined reference to `_Unwind_Resume'
          foo.foo.7rcbfp3g-cgu.3.rcgu.o:(.data.DW.ref.rust_eh_personality[DW.ref.rust_eh_personality]+0x0): undefined reference to `rust_eh_personality'
          collect2: error: ld returned 1 exit status

So there's still references to _Unwind_Resume and rust_eh_personality, but where are they? Unleash a debugger!

Dump of assembler code for function _ZN4core6result19Result$LT$T$C$E$GT$6unwrap17hc6a8d8550c5348d4E:
   0x0000000000000000 <+0>:	48 83 ec 28	sub    $0x28,%rsp
   0x0000000000000004 <+4>:	48 89 3c 24	mov    %rdi,(%rsp)
   0x0000000000000008 <+8>:	48 89 74 24 08	mov    %rsi,0x8(%rsp)
   0x000000000000000d <+13>:	48 8b 04 24	mov    (%rsp),%rax
   0x0000000000000011 <+17>:	48 85 c0	test   %rax,%rax
   0x0000000000000014 <+20>:	74 12	je     0x28 <_ZN4core6result19Result$LT$T$C$E$GT$6unwrap17hc6a8d8550c5348d4E+40>
   0x0000000000000016 <+22>:	eb 00	jmp    0x18 <_ZN4core6result19Result$LT$T$C$E$GT$6unwrap17hc6a8d8550c5348d4E+24>
   0x0000000000000018 <+24>:	eb 20	jmp    0x3a <_ZN4core6result19Result$LT$T$C$E$GT$6unwrap17hc6a8d8550c5348d4E+58>
   0x000000000000001a <+26>:	48 8b 7c 24 18	mov    0x18(%rsp),%rdi
   0x000000000000001f <+31>:	e8 00 00 00 00	callq  0x24 <_ZN4core6result19Result$LT$T$C$E$GT$6unwrap17hc6a8d8550c5348d4E+36>
   0x0000000000000024 <+36>:	0f 0b	ud2    
   0x0000000000000026 <+38>:	0f 0b	ud2    
   0x0000000000000028 <+40>:	48 83 3c 24 00	cmpq   $0x0,(%rsp)
   0x000000000000002d <+45>:	74 3c	je     0x6b <_ZN4core6result19Result$LT$T$C$E$GT$6unwrap17hc6a8d8550c5348d4E+107>
   0x000000000000002f <+47>:	eb 3f	jmp    0x70 <_ZN4core6result19Result$LT$T$C$E$GT$6unwrap17hc6a8d8550c5348d4E+112>
   0x0000000000000031 <+49>:	48 83 3c 24 00	cmpq   $0x0,(%rsp)
   0x0000000000000036 <+54>:	74 31	je     0x69 <_ZN4core6result19Result$LT$T$C$E$GT$6unwrap17hc6a8d8550c5348d4E+105>
   0x0000000000000038 <+56>:	eb e0	jmp    0x1a <_ZN4core6result19Result$LT$T$C$E$GT$6unwrap17hc6a8d8550c5348d4E+26>
   0x000000000000003a <+58>:	48 8b 44 24 08	mov    0x8(%rsp),%rax
   0x000000000000003f <+63>:	48 89 44 24 10	mov    %rax,0x10(%rsp)
   0x0000000000000044 <+68>:	48 8d 3d 00 00 00 00	lea    0x0(%rip),%rdi        # 0x4b <_ZN4core6result19Result$LT$T$C$E$GT$6unwrap17hc6a8d8550c5348d4E+75>
   0x000000000000004b <+75>:	48 8d 0d 00 00 00 00	lea    0x0(%rip),%rcx        # 0x52 <_ZN4core6result19Result$LT$T$C$E$GT$6unwrap17hc6a8d8550c5348d4E+82>
   0x0000000000000052 <+82>:	48 8b 05 00 00 00 00	mov    0x0(%rip),%rax        # 0x59 <_ZN4core6result19Result$LT$T$C$E$GT$6unwrap17hc6a8d8550c5348d4E+89>
   0x0000000000000059 <+89>:	be 2b 00 00 00	mov    $0x2b,%esi
   0x000000000000005e <+94>:	48 8d 54 24 10	lea    0x10(%rsp),%rdx
   0x0000000000000063 <+99>:	ff d0	callq  *%rax
   0x0000000000000065 <+101>:	eb 0b	jmp    0x72 <_ZN4core6result19Result$LT$T$C$E$GT$6unwrap17hc6a8d8550c5348d4E+114>
   0x0000000000000067 <+103>:	eb c8	jmp    0x31 <_ZN4core6result19Result$LT$T$C$E$GT$6unwrap17hc6a8d8550c5348d4E+49>
   0x0000000000000069 <+105>:	eb af	jmp    0x1a <_ZN4core6result19Result$LT$T$C$E$GT$6unwrap17hc6a8d8550c5348d4E+26>
   0x000000000000006b <+107>:	48 83 c4 28	add    $0x28,%rsp
   0x000000000000006f <+111>:	c3	retq   
   0x0000000000000070 <+112>:	eb f9	jmp    0x6b <_ZN4core6result19Result$LT$T$C$E$GT$6unwrap17hc6a8d8550c5348d4E+107>
   0x0000000000000072 <+114>:	0f 0b	ud2    
   0x0000000000000074 <+116>:	48 89 44 24 18	mov    %rax,0x18(%rsp)
   0x0000000000000079 <+121>:	89 54 24 20	mov    %edx,0x20(%rsp)
   0x000000000000007d <+125>:	eb e8	jmp    0x67 <_ZN4core6result19Result$LT$T$C$E$GT$6unwrap17hc6a8d8550c5348d4E+103>

So, no explicit references to either function, presumably they are relocated in?

$ readelf -r foo.foo.7rcbfp3g-cgu.3.rcgu.o

Relocation section '.rela.text._ZN4core6result19Result$LT$T$C$E$GT$6unwrap17hc6a8d8550c5348d4E' at offset 0x2c8 contains 4 entries:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
000000000020  000800000004 R_X86_64_PLT32    0000000000000000 _Unwind_Resume - 4
000000000047  000500000002 R_X86_64_PC32     0000000000000000 .rodata..L__unnamed_1 - 4
00000000004e  000600000002 R_X86_64_PC32     0000000000000000 .data.rel.ro..L__unnam - 4
000000000055  000b00000009 R_X86_64_GOTPCREL 0000000000000000 _ZN4core6result13unwra - 4

Relocation section '.rela.data.rel.ro..L__unnamed_2' at offset 0x328 contains 2 entries:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
000000000000  000a00000001 R_X86_64_64       0000000000000000 _ZN4core3ptr18real_dro + 0
000000000018  000900000001 R_X86_64_64       0000000000000000 _ZN4core3fmt3num52_$LT + 0

Relocation section '.rela.data.DW.ref.rust_eh_personality' at offset 0x358 contains 1 entry:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
000000000000  000d00000001 R_X86_64_64       0000000000000000 rust_eh_personality + 0

Relocation section '.rela.eh_frame' at offset 0x370 contains 3 entries:
  Offset          Info           Type           Sym. Value    Sym. Name + Addend
000000000013  000700000002 R_X86_64_PC32     0000000000000000 DW.ref.rust_eh_persona + 0
000000000028  000300000002 R_X86_64_PC32     0000000000000000 .text._ZN4core6result1 + 0
000000000031  000400000002 R_X86_64_PC32     0000000000000000 .gcc_except_table + 0

So there's relocation instructions for both functions:

_Unwind_Resume is inserted into address 0x00000020 (becoming the destination of the callq instruction)
rust_eh_personality is inserted into a .data.DW.ref.rust_eh_personality section, and then that address is written into the .eh_frame section as the personality function to use for unwinding.

Performing code-flow analysis on the generated code shows that the relocation site for _Unwind_Resume is unreachable (since core::result::unwrap is an external symbol). This explains why the release build is fine, as the unreachable code segments (and thus the relocation instructions) are stripped by LLVM. Similarly (inspecting the LLVM-IR) the function is marked nounwind and thus no .eh_frame section is generated, thus removing the need for the rust_eh_personality relocation.

jefftime

comment created time in a month

issue commentrust-lang/rust

Implement support for LLVMs code coverage instrumentation

Ah yes, I'm being slow :) - so long as the expanded code knows it came from a macro then HIR is the correct point indeed.

gnzlbg

comment created time in a month

issue commentrust-lang/rust

Implement support for LLVMs code coverage instrumentation

I think (though I suspect @bob-wilson will know better and can feel free to correct me) that the coverage format has the capability to represent "macros" (remote code) sensibly (https://llvm.org/docs/CoverageMappingFormat.html#file-id) so maybe we'd want to apply instrumentation directly to the parse tree?

gnzlbg

comment created time in a month

issue commentrust-lang/rust

Implement support for LLVMs code coverage instrumentation

https://github.com/rust-lang/rust/pull/48346/files#diff-2030f049f8df2454e3720ea9403bc14aR446 is where we started adding the pgo-instr-gen pass.

gnzlbg

comment created time in 2 months

issue commentrust-lang/rust

Implement support for LLVMs code coverage instrumentation

@eggyal yes, profile-generate enables two passes:

pgo-instr-gen which heuristically adds http://llvm.org/docs/LangRef.html#llvm-instrprof-increment-intrinsic (and related intrinsics) to the LLVM-IR basically by adding them to the beginning of codeblocks.
instrprof which lowers the above intrinsics to "real code" I assume these are intrinsics so the implementation of profile counting can be smart (threadsafe in threaded programs, using native types where needed etc.)

Together (with the profile-rt library) these allow generation and subsequent use of profiling data during compilation. Unfortunately together these two passes are insufficient for code-coverage usage, since the LLVM is too far away from the AST. Part 4 of my write-up discusses writing the coverage mapping code that maps AST nodes to instrumentation intrinsics - this is what then enables frontend-guided coverage.

I suspect that adding the coverage map will require disabling the pgo-instr-gen pass and adding the intrinsics manually (this seems to be what clang calls profile-instr-generate?), because there's rust-specific knowledge that can/should be applied (e.g. generic expansions, semanticly equivalent AST changes etc.) and because there's no way to guess the profiling entries that will be generated by pgo-instr-gen.

gnzlbg

comment created time in 2 months

issue commentMetaswitch/floki

Better support for build caches

Just did a bit of testing, and so far all seems good. I even tested the hack we discussed on slack (symlinking from my real registry to the volume under .floki/cache) and that works excellently:

$ ln -sf ~/.cargo/registry /home/andy/.floki/cache/
$ cat floki.yaml 
image: rust-dev:latest
shell: bash
volumes:
  registry:
    shared: true
    mount: /opt/rust/registry
$ floki
# time cargo search --registry crates-io foo
...
real	0m0.449s
user	0m0.029s
sys	0m0.017s

Without the mounted registry this takes ~2 mins.

rlupton20

comment created time in 2 months

issue commentMetaswitch/floki

Better support for build caches

:+1:

rlupton20

comment created time in 2 months

issue commentMetaswitch/floki

Better support for build caches

Sounds good to me! Presumably if a floki.yaml asks to mount a shared volume that doesn't exist then it will be created but not backed by a host-path?

rlupton20

comment created time in 2 months

Pull request review commentsoftprops/shiplift

 impl Transport {             Transport::Tcp { .. } => (),             #[cfg(feature = "tls")]             Transport::EncryptedTcp { .. } => (),-            _ => panic!("connection streaming is only supported over TCP"),+            #[cfg(feature = "unix-socket")]+            Transport::Unix { .. } => panic!("connection streaming is only supported over TCP"),

This change prevents an "unreachable branch" warning on builds without unix-socket support (e.g. on windows)

bossmc

comment created time in 2 months

pull request commentsoftprops/shiplift

Ok, updated PR, new strategy - expose a vendored-ssl feature on shiplift that activates the openssl/vendored feature on openssl.

Windows:

c:\src\shiplift>cargo t --features vendored-ssl --no-default-features
...
test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

Linux:

$ cargo t
...
test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
$ ldd target/debug/deps/shiplift-5d6a9fe8f830c081
...
	libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007fccd9fa1000)
...
$ cargo t --features vendored-ssl
...
test result: ok. 10 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
$ ldd target/debug/deps/shiplift-792b6a711bfb1b6c
	linux-vdso.so.1 (0x00007f7568604000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f7567e7e000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f7567c76000)
	libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f7567a57000)
	libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f756783f000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f756744e000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f75683de000)
	libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f75670b0000)

bossmc

comment created time in 2 months

push eventMetaswitch/shiplift

commit sha 0accbf205e190d7d7046b136fa672b96cd21fe9e

Use a feature rather than guessing based on OS

push time in 2 months

pull request commentsoftprops/shiplift

@softprops - There's still some more work to do, the change I've made so far is being ignored by Cargo, so I'm putting together a smaller version of the change. Should be with you today.

bossmc

comment created time in 2 months

pull request commentsoftprops/shiplift

Sounds like adding a feature is a better fix then.

bossmc

comment created time in 2 months

push eventbossmc/dotfiles

commit sha 49b5a67cc5ed675fb9a5bcacfface77a7d402d3d

Refresh regolith config

branch : only-vendor-on-windows

push time in 2 months

PR opened softprops/shiplift

Don't vendor OpenSSL except on Windows

<!--

If there is a breaking or notable change please call that out as these will need to be added to the CHANGELOG.md file in this repository.
This repository tries to stick with the community style conventions using rustfmt with the default settings. If you have custom settings you may find that rustfmt clutter the diff of your change with unrelated changes. Please apply formatting with cargo fmt before submitting a pr. -->

What did you implement:

This relates to #170 where shiplift switched to using vendored openssl to assist windows builds. Unfortunately I believe that change went too far the other way, using the system-wide OpenSSL is much more common on *nix systems, and the vendoring gets in the way of other tooling (e.g. building an RPM from such a binary leads to problems, as the RPM will try to "depend" on the vendored shared object, which will not be resolvable from the repo).

This change enables the vendored feature only on windows builds, and reverts to the default behaviour (searching for a system-wide version using pkg-config if present) on other systems (mac, linux...)

I considered an alternative option, that of exposing a vendored-ssl feature that enables the openssl/vendored feature, which can be enabled if desired by builds, but that makes windows builds harder, which feels like a step backwards.

How did you verify your change:

(cross) Compiled for linux and windows.

What (if anything) would need to be called out in the CHANGELOG for the next release:

Clarifying the notes from #170 to be clear that they only apply to windows builds.

+5 -1

1 commit in Metaswitch/shiplift/tree/only-vendor-on-windows softprops/shiplift/tree/master

0 comment

1 changed file

pr created time in 2 months

create barnchMetaswitch/shiplift

created branch time in 2 months

delete branch Metaswitch/shiplift

delete branch : copy-from-container

delete time in 2 months

push eventMetaswitch/shiplift

Deployment Bot (from Travis CI)

commit sha 5871ff9ead315f10d4ccb25904f8687dba028cd5

Deploy softprops/shiplift to github.com/softprops/shiplift.git:gh-pages

push time in 2 months

push eventMetaswitch/shiplift

commit sha 2a40dc00e6aeb611db8095dc8646e5132fea356b

Copy from container (#150) * Add 'copy_from' function to 'Container' * Run clippy * Update deps

Vegard Sandengen

commit sha ebb293813fb1c92149a5cc802043ba41f4ddf1a5

Migrate serde dependency to use derive feature (#152) This is in line with best practices recommended by serde[1]. This will also resolve downstream crates depending on shiplift who enable the serde derive feature, due to Cargos unification of features for each crate[2]. [1]: https://github.com/serde-rs/serde/issues/1441 [2]: https://github.com/rust-lang/cargo/issues/4361#issuecomment-348538243

Marc Schreiber

commit sha 0ae953c69ce7a336794244adc29bbc206320705d

Copy a byte slice as file into a container (#151) - add function copy_file_into to container - add example

Austin Brown

commit sha ad95d93204cc84cc4648a1defac786db3ddd8582

Support multiple messages per chunk in streaming image pull (#154)

Marc Schreiber

commit sha eb98b1916c0220e44e2d0f3c869c01a2dd037f60

Apply rust format to fix CI checks (#153)

Anton Ageev

commit sha ac8789e257f5e4a477192bbbd8ecc2676bb485e5

Add a registry authentication. (#157)

Graham Wihlidal

commit sha 97ce095eaed9f893ce7e31dec6375e01d17b3b64

Added `ExposedPorts` mapping (#162) * Added `ExposedPorts` mapping, as per https://docs.docker.com/engine/api/v1.26/#operation/ContainerCreate * Correct the ExposedPorts json based on the latest Docker schema * Derived debug for all builder option types (useful for debugging) * Adjust container_options_expose test to match latest code * Applied cargo fmt

softprops

commit sha c866053e930a6d9560fc387eb63bad09551d08a7

relax strictness on patch versions on deps

softprops

commit sha a1f0a1051dc58c25f7732ce66abdb6317c95b7a2

alphbet sort on deps

softprops

commit sha 45b79d82887cbfdded28f7fc631f71377e641230

update changelog and docs

commit sha bc732d8be75c14b376bda096ceae6d01d430a8a5

Add explanative sentence for volumes option (#164)

Max Eliseev

commit sha 3df2c90e4855270a024de86d76ded25a0c2d4721

Add LogsOptionsBuilder::since() method (#169) * Add LogsOptionsBuilder.since() method * test for LogsOptionsBuilder

commit sha e97bf7175ddf063788df2fa630b4dedcd19388a0

Switch path type requirement to make more ergonomic (#168)

commit sha e5258c8e63f09ef7aafc7e489bb62131eb34a74f

Add import functionality (#165)

commit sha 00c304528485bfd7d6eb9e4605514b50de791255

Switch to vendored openssl (#170)

Sergey Tsaplin

commit sha 897f16a3a65dacd9a2d9f113659a443b5ade0cea

Fix build image response parsing error (#177)

Sergey Tsaplin

commit sha dd732ad2691c4ffb04506024f991a8cbb5b8034b

Add WorkingDir container's option support (#175)

Marc Schreiber

commit sha 12ec796a5c9cdc1803f59c85328456538c698cbe

copy_file_into creates sub-directories if required (#171)

0xflotus

commit sha ee7be3d566945e453404e6c7b799d997418d145c

fixed process (#181)

Doug Tangren

commit sha 7d9718b33764dea4a85859e94337e471de091332

rustfmt config option name changed (#182)

branch : rustfmt_config_fix

push time in 2 months

created tagMetaswitch/shiplift

tag0.5.0

🐳 🦀 rust interface for maneuvering docker containers

created time in 2 months

create barnchMetaswitch/shiplift

created branch time in 2 months

pull request commentmozilla/sccache

Use Rusoto to handle S3 communication

Re-enabled anonymous access to the cache (if no credentials are provided, try an anonymous read/write, if credentials are provided - use them).

bossmc

comment created time in 2 months

push eventMetaswitch/sccache

Nathan Froyd

commit sha b9141cb44d3c5f86148210d5223b2c407c8b3182

remove output files after preprocessing failures Fixes #517.

commit sha eb1c57f9690200e44294d3f1ac486b6606bb9f57

Move toolchain cache lock acquisition earlier. Fixes #519 Previously we would lock to prevent multiple toolchain creations at once, but the lock was acquired after checking for the presence of the toolchain in question and determining it wasn't present. In case multiple attempts were made to create a toolchain asynchronously the attempts would serialize on this lock but still redundantly re-create toolchains.

commit sha a28474b9007f024c9ec2c4807421de5703dc7182

Remove jobs from job list when replacing a server's identity.

commit sha e4372e2b9d72be04974d5528dccdc81e216b7ff8

Keep track of jobs assigned but not yet attended by a client in the scheduler. Fixes #513. We've found some situations where a server will retain jobs waiting for a client to request they be run. In practice this means that a client that's having trouble connecting to a server (due to #516 or another issue) can take that server offline by consuming its capacity, which will be unavailable indefinitely as far as the scheduler is concerned. This commit takes the step of tracking jobs that have been assigned but have not yet been started. When receiving a heartbeat from a server, the scheduler will check the unclaimed jobs assigned to that server and de-allocate any that have been waiting to start for too long. The threshold for now is 1 minute for jobs that start in the "Ready" state and 5 minutes for jobs in the "Pending" state in case toolchain packaging and transfer overrruns the smaller threshold.

Gabriel Féron

commit sha ad50350e3d11290886b2c4d1c06ae4483b9c5afa

Authenticate S3 GET requests if credentials are available

commit sha fdf1fced12d944f50ccb204367519461dc27225b

Support HTTPS and AWS v4 Auth

commit sha 103020342c20dec33b8dbaba24c02d976ec89d41

Force HTTP for now

commit sha ea138843b9f86fa1584606c8112506afdc1e2432

Update dependencies and patches

commit sha df6c9b3bcd7bcc4e039a4c24a39693f1ba902bdb

Bump minimum rustc version

commit sha 60ebcece30fb94d8d66b9526c1e649dc51235b17

Also fix up dist-server build

Full parser for license expressions (version II)

push time in 2 months

pull request commentrust-lang-nursery/license-exprs

@carols10cents - Who should I be talking to about the correct fix here? I read @withoutboats's suggestion but, as he's no longer a maintainer, I was hoping to get an opinion from one of the current maintainers before starting to re-work the change. I understand the worry about the dependency explosion, but I think most of these dependencies are already present in Cargo (and I don't believe that writing a parser by hand is a good idea, it's likely to be buggy and the lalrpop generator "just works":tm:)

As a heads-up, there seems to be an alternative crate for handling SPDX being created (including some of this very change) at https://github.com/EmbarkStudios/spdx, they've hand-rolled a Shunting-Yard-based parser which I'm sure they'd be happy to share if that's a preferred solution?

bossmc

comment created time in 2 months

push eventMetaswitch/sccache

commit sha 3a671ab34e02ee25877ae47d4a5fdc08dec335ad

Also fix up dist-server build

push time in 3 months

push eventMetaswitch/sccache

commit sha b7716296c654fbcbce7a53bbc8f18fa861e24f6f

Bump minimum rustc version

Use Rusoto to handle S3 communication

push time in 3 months

pull request commentmozilla/sccache

Well, that's promising @luser, thanks!

OTOH it seems I've bumped the minimum rustc version (due to dependencies)...

bossmc

comment created time in 3 months

Pull request review commentmozilla/sccache

Use Rusoto to handlle S3 communication

 pub struct S3Cache { impl S3Cache {     /// Create a new `S3Cache` storing data in `bucket`.     pub fn new(bucket: &str, endpoint: &str) -> Result<S3Cache> {-        let user_dirs = UserDirs::new().ok_or("Couldn't get user directories")?;-        let home = user_dirs.home_dir();--        let profile_providers = vec![-            ProfileProvider::with_configuration(home.join(".aws").join("credentials"), "default"),-            //TODO: this is hacky, this is where our mac builders store their-            // credentials. We should either match what boto does more directly-            // or make those builders put their credentials in ~/.aws/credentials-            ProfileProvider::with_configuration(home.join(".boto"), "Credentials"),-        ];

This change is the one bit that I can see being a problem. The Rusoto client no longer supports configuring multiple ProfileProviders for a single ChainProvider, I suspect it's not too hard to implement a local version, but if you're planning to move your credentials anyway, maybe this is your chance?

bossmc

comment created time in 3 months

PR opened mozilla/sccache

Use Rusoto to handlle S3 communication

This moves sccache over to using Rusoto instead of the inline S3 client. The code is so similar that I can only assume one was based on the other but Rusoto has been developed further and supports some extra features (and probably bug-fixes). In particular it supports AuthenticationV4 (as well as V2), which is very handy for interaction with S3 implementations that either don't support V2 or can only support one of V2 or V4 at a time (e.g. minio).

In the process, I ended up bumping quite a few dependencies and have updated the patch entries appropriately to match.

+1674 -1343

3 commit in Metaswitch/sccache/tree/rusoto-client-for-aws mozilla/sccache/tree/master

0 comment

16 changed files

pr created time in 3 months

push eventMetaswitch/sccache

commit sha 7fad391f6f6b30525400357e915f8b02eb0b017b

Update dependencies and patches

Use Rusoto client to allow connecting to non-AWS S3 instances

push time in 3 months

pull request commentMetaswitch/sccache

@richardwhiuk can you please give this a once over before I upstream it? Thanks!

bossmc

comment created time in 3 months

pull request commentrust-lang/rfcs

RFC for an operator to take a raw reference

Sorry for arriving late to the party (and arguably just adding bikeshedding), but I haven't seen this proposal discussed above, and I think it might make the syntax more intuitive.

Put simply, use *const <expr>/*mut <expr> to create a raw pointer. The semantics of *const are exactly the same as those proposed for &raw const (and *mut for &raw mut). This means that *const and *mut gain the same syntactic powers as & and &mut:

&T/&mut T/*const T/*mut T - Are the names of types of pointers
&t/&mut t/*const t/*mut t - Are the values of (different types of) pointers that point to t

This means that there's a clear split between &-like pointers which (maybe in future) must point to valid, aligned memory and conform to inbounds rules etc. and *-like pointers which are true raw pointers and are beholden to no-one.

&t as *const _ is still valid (as the requirements on & are stricter than those on *) but the other examples above would need the use of *const to make them defined behaviour:

#[repr(packed)]
#[derive(Default)]
struct Unaligned {
  field1: u8,
  field2: u32,
}

let s = Unaligned::default();
let sref = &s; // Fine
let field1ref = &s.field1; // Fine
let field2ref = &s.field2; // UB - unaligned reference
let field1ptr = *const s.field1;  // Fine
let field2ptr = *const s.field2;  // Fine
let bar = unsafe { &*field2ptr } // UB - the created ref is unaligned

My natural read of &raw const t is that it's creating a special kind of reference, but it isn't, it's creating a pointer which is a fundamentally different thing and I'm worried that this confusion will make this feature much harder to grok at a glance.

Disadvantage: & to mean "take the address of" is standard from C but Rust already breaks the C conventions by using &T to mean (something akin to) "pointer to T". Using * to mean "take the address of" might be unexpected, but (due to the attached mut/const) shouldn't actively steer people towards confusion.

RalfJung

comment created time in 3 months

push eventMetaswitch/sccache

commit sha 0d0397801e24469755570e38e4d451e17cf3b274

Update dependencies and patches

push time in 3 months

push eventMetaswitch/sccache

commit sha 5112041e0b1f95467e1a4da758eb2533bfb00d58

Add a field to the scheduler status result for the total number of cpus across active servers.

Felix Obenhuber

commit sha e516d90a804e41a27b9342f9c43bfa76171f4b42

Fix windows build by indicating anonymous lifetimes (#448) Along with the missing explicit lifetimes a couple or warnings regarding duplicate use directives are fixed.

Lzu Tao

commit sha 58556730f33ee46152a5f91a16db896e14e7bf4d

Remove usage of kernel32-sys crate

commit sha e978694d6b23a54c9d7ac6822e0f283b33b8978d

Update CACHE_VERSION for rust compilation following #441

commit sha 90dfacdf99349bb1ad7bca915749692a9cd4172d

Change lru-disk-cache module to expose Meter and LruCache. This module structure is preventing us from releasing with the `dist-client` feature enabled. The release process removes "path" dependencies, meaning we get the crates.io version of lru-disk-cache, which isn't exposing lru_cache{Meter,LruCache} at any level.

commit sha 0e776e07270c667c188a179aac424bd7e85d9fbb

use LruCache and Meter directly from lru_disk_cache

commit sha 73e52df48813fed9a77c8e071019dfb917fe5cce

Bump lru-disk-cache version

commit sha 48d13a96a7456822954b80d01ca0aa854826436b

start next development iteration 0.2.10-alpha.0

commit sha 9ba1b50e9bc41ff5fa884e9c94cda8693a649165

Update lockfile to reflect new version.

commit sha 42ebb0b8008feb6f94dc0a9e41213fa4c7dc5ade

Clarify docs for distributing from macOS clients.

commit sha c32b1ccb22d6519396d814bd9eb519cb57dea518

Try removing the cargo cache to make CI pass

Christopher Manchester

commit sha 93475f3458f7776c37b08201ae0d83ecac4b8fa2

Open files on worker threads rather than passing a file handle when hashing inputs. Fixes #468

Michael Tang

commit sha 2448687217463b1d77b451ac7f63e5a4614693e7

Add `CREATE_NO_WINDOW` to server startup processes on Windows

commit sha f4d130311f5b119acd1dbeee4e7d0477e303c42c

Misc. dist harness fixes

commit sha 2866bbe46da04e3e857e7449b59301b631f2705e

Remove jobs from assigned_jobs when allocation fails. In practice servers will occasionally become unreachable and trying to allocate to them will fail. Rather than leaking capacity in this case we remove the job on allocation failure. Fixes #477

commit sha 6ba6f6c15c106768b914a7697a763e2232fa253a

Add a version check for bubblewrap in the server. Fixes #316

commit sha f988cf92c8a6fc4bd799e04da61d7ce447f1c790

Notify the scheduler when a failed job completes. See #477

commit sha fe0f04557575b86d2bbe09ff62b7a7fb08e99d28

Choose servers without recent errors first when distributing jobs in the scheduler. Fixes #483

commit sha f5666903a41c8dc0409b67698ac2a9d1b033a7d3

Add do_get_status to the distributed client implementation and use it when starting the server.

commit sha 6b384d6bd7b2878af595c0d539287d9693e0dd9b

Treat missing auth for a configured scheduler as a fatal in the client. This makes it an error when a scheduler is configured but no token is present for the configured address. The error is surfaced to a potential compile attempt so that someone who triggers the server from a compile attempt will become aware of the misconfiguration. The semantics of loading the client dist config are also changed to always consult the file on disk to support a flow of attempting to compile, receiving an error instructing the user to run --dist-auth, and continuing without the need to restart the daemon or consult its logs.

push time in 3 months

PR opened Metaswitch/sccache

Use Rusoto client to allow connecting to non-AWS S3 instances

+2175 -1533

37 commit in Metaswitch/sccache/tree/rusoto-client-for-aws Metaswitch/sccache/tree/master

0 comment

35 changed files

pr created time in 3 months

push eventMetaswitch/sccache

commit sha 6cb3b26e01d7aedbac5f7f1c757d8bbd26f4bef3

Update dependencies and patches

push time in 3 months

push eventMetaswitch/sccache

commit sha 74f46bb3db434c6accc5475f8b8e7160b7791084

Update dependencies and patches

push time in 3 months

push eventMetaswitch/sccache

commit sha 5112041e0b1f95467e1a4da758eb2533bfb00d58

Add a field to the scheduler status result for the total number of cpus across active servers.

Felix Obenhuber

commit sha e516d90a804e41a27b9342f9c43bfa76171f4b42

Fix windows build by indicating anonymous lifetimes (#448) Along with the missing explicit lifetimes a couple or warnings regarding duplicate use directives are fixed.

Lzu Tao

commit sha 58556730f33ee46152a5f91a16db896e14e7bf4d

Remove usage of kernel32-sys crate

commit sha e978694d6b23a54c9d7ac6822e0f283b33b8978d

Update CACHE_VERSION for rust compilation following #441

commit sha 90dfacdf99349bb1ad7bca915749692a9cd4172d

Change lru-disk-cache module to expose Meter and LruCache. This module structure is preventing us from releasing with the `dist-client` feature enabled. The release process removes "path" dependencies, meaning we get the crates.io version of lru-disk-cache, which isn't exposing lru_cache{Meter,LruCache} at any level.

commit sha 0e776e07270c667c188a179aac424bd7e85d9fbb

use LruCache and Meter directly from lru_disk_cache

commit sha 73e52df48813fed9a77c8e071019dfb917fe5cce

Bump lru-disk-cache version

commit sha 48d13a96a7456822954b80d01ca0aa854826436b

start next development iteration 0.2.10-alpha.0

commit sha 9ba1b50e9bc41ff5fa884e9c94cda8693a649165

Update lockfile to reflect new version.

commit sha 42ebb0b8008feb6f94dc0a9e41213fa4c7dc5ade

Clarify docs for distributing from macOS clients.

commit sha c32b1ccb22d6519396d814bd9eb519cb57dea518

Try removing the cargo cache to make CI pass

Christopher Manchester

commit sha 93475f3458f7776c37b08201ae0d83ecac4b8fa2

Open files on worker threads rather than passing a file handle when hashing inputs. Fixes #468

Michael Tang

commit sha 2448687217463b1d77b451ac7f63e5a4614693e7

Add `CREATE_NO_WINDOW` to server startup processes on Windows

commit sha f4d130311f5b119acd1dbeee4e7d0477e303c42c

Misc. dist harness fixes

commit sha 2866bbe46da04e3e857e7449b59301b631f2705e

Remove jobs from assigned_jobs when allocation fails. In practice servers will occasionally become unreachable and trying to allocate to them will fail. Rather than leaking capacity in this case we remove the job on allocation failure. Fixes #477

commit sha 6ba6f6c15c106768b914a7697a763e2232fa253a

Add a version check for bubblewrap in the server. Fixes #316

commit sha f988cf92c8a6fc4bd799e04da61d7ce447f1c790

Notify the scheduler when a failed job completes. See #477

commit sha fe0f04557575b86d2bbe09ff62b7a7fb08e99d28

Choose servers without recent errors first when distributing jobs in the scheduler. Fixes #483

commit sha f5666903a41c8dc0409b67698ac2a9d1b033a7d3

Add do_get_status to the distributed client implementation and use it when starting the server.

commit sha 6b384d6bd7b2878af595c0d539287d9693e0dd9b

Treat missing auth for a configured scheduler as a fatal in the client. This makes it an error when a scheduler is configured but no token is present for the configured address. The error is surfaced to a potential compile attempt so that someone who triggers the server from a compile attempt will become aware of the misconfiguration. The semantics of loading the client dist config are also changed to always consult the file on disk to support a flow of attempting to compile, receiving an error instructing the user to run --dist-auth, and continuing without the need to restart the daemon or consult its logs.

push time in 3 months

push eventbossmc/dotfiles

commit sha 5b42c90210a70606e84ebdb9e99e1efdc9ff3a13

Add regolith config

push time in 3 months

push eventbossmc/dotfiles