profile
viewpoint

nagisa/django-bfm 40

[Not really maintained anymore] BFM is an Apache Licenced server file manager for Django made with ease of use and new web technologies in mind.

lucab/memfd-rs 8

A pure-Rust library to work with Linux memfd

nagisa/e4rat-preload-lite 5

More efficient way to preload e4rat file lists.

nagisa/Feeds 3

Feed Reader for GNOME.

nagisa/gnome-shell-theme-min 3

A GNOME-Shell theme without bells or whistles

cpcloud/tensorflow_proto 2

Rusty protobufs for Tensorflow

nagisa/llvm_build_utils.rs 2

LLVM build utils for cargo build scripts

nagisa/django-localflavor-lt 1

Country-specific Django helpers for Lithuania.

nagisa/kazlauskas.me 1

Hakyll based website of mine

Pull request review commentrust-lang/rust

Support custom allocators in `Box`

 impl<T: ?Sized> Box<T> {     /// ```     #[stable(feature = "box_leak", since = "1.26.0")]     #[inline]-    pub fn leak<'a>(b: Box<T>) -> &'a mut T+    pub fn leak<'a>(b: Self) -> &'a mut T     where-        T: 'a, // Technically not needed, but kept to be explicit.+        A: 'a,     {         unsafe { &mut *mem::ManuallyDrop::new(b).0.as_ptr() }     } +    /// Consumes and leaks the `Box`, returning a mutable reference,+    /// `&'a mut T`, and the allocator. Note that the type `T` must outlive the chosen lifetime+    /// `'a`. If the type has only static references, or none at all, then this+    /// may be chosen to be `'static`.+    ///+    /// This function is mainly useful for data that lives for the remainder of+    /// the program's life. Dropping the returned reference will cause a memory+    /// leak. If this is not acceptable, the reference should first be wrapped+    /// with the [`Box::from_raw`] function producing a `Box`. This `Box` can+    /// then be dropped which will properly destroy `T` and release the+    /// allocated memory.+    ///+    /// Note: this is an associated function, which means that you have+    /// to call it as `Box::leak(b)` instead of `b.leak()`. This+    /// is so that there is no conflict with a method on the inner type.+    #[unstable(feature = "allocator_api", issue = "32838")]+    #[inline]+    pub fn leak_with_alloc<'a>(b: Self) -> (&'a mut T, A)

This API seems sketchy to me. Without thinking too much about it I think one could do something along the lines of:

let (ref, allocref) = Box::leak_with_alloc(a_box);
drop(allocref); // potentially drops the backing storage
use(ref); // use of a dangling reference
TimDiekmann

comment created time in 14 hours

PullRequestReviewEvent

Pull request review commentrust-lang/rust

Support custom allocators in `Box`

 impl<T: ?Sized> Box<T> {     /// ```     #[stable(feature = "box_leak", since = "1.26.0")]     #[inline]-    pub fn leak<'a>(b: Box<T>) -> &'a mut T+    pub fn leak<'a>(b: Self) -> &'a mut T     where-        T: 'a, // Technically not needed, but kept to be explicit.+        A: 'a,     {         unsafe { &mut *mem::ManuallyDrop::new(b).0.as_ptr() }     } +    /// Consumes and leaks the `Box`, returning a mutable reference,+    /// `&'a mut T`, and the allocator. Note that the type `T` must outlive the chosen lifetime+    /// `'a`. If the type has only static references, or none at all, then this+    /// may be chosen to be `'static`.+    ///+    /// This function is mainly useful for data that lives for the remainder of+    /// the program's life. Dropping the returned reference will cause a memory+    /// leak. If this is not acceptable, the reference should first be wrapped+    /// with the [`Box::from_raw`] function producing a `Box`. This `Box` can+    /// then be dropped which will properly destroy `T` and release the+    /// allocated memory.+    ///+    /// Note: this is an associated function, which means that you have+    /// to call it as `Box::leak(b)` instead of `b.leak()`. This+    /// is so that there is no conflict with a method on the inner type.+    #[unstable(feature = "allocator_api", issue = "32838")]+    #[inline]+    pub fn leak_with_alloc<'a>(b: Self) -> (&'a mut T, A)+    where+        A: 'a,+    {+        let manually_drop = mem::ManuallyDrop::new(b);+        let alloc = unsafe { ptr::read(&manually_drop.1) };

You probably want MaybeUninit as you're effectively un-initializing part of the data contained within ManuallyDrop which is somewhat sketchy as the value is later used again to obtain reference.

TimDiekmann

comment created time in 14 hours

PullRequestReviewEvent

pull request commentrust-lang/rust

Add missing definitions required by the sparc-unknown-linux-gnu target

@bors r+

glaubitz

comment created time in 15 hours

pull request commentrust-lang/rust

Use `str::to_owned` when `format!` has no formatting arguments

@bors r+

lzutao

comment created time in a day

issue closedrust-fuzz/cargo-fuzz

Fuzz target exited with signal: 11

I'm running the fuzzer on part of my codebase. And it sometimes gives the following error:

#4993: cov: 16031 ft: 79453 corp: 4047 exec/s 23 oom/timeout/crash: 0/0/0 time: 401s job: 20 dft_time: 0
#5491: cov: 16031 ft: 79456 corp: 4050 exec/s 22 oom/timeout/crash: 0/0/0 time: 407s job: 21 dft_time: 0
────────────────────────────────────────────────────────────────────────────────

Error: Fuzz target exited with signal: 11

But the problem is there is no debug info, anywhere, no artifacts, no dump, no backtrace, ... Even when I run it in debug mode it does not give me any more info.

It even got a Segmentation fault but no info and just continued (output looks weird, but this is how it was printed in the terminal):

#429	NEW    cov: 10201 ft: 33953 corp: 208/179Kb lim: 2091 exec/s: 22 rss: 159Mb L: 2030/2091 MS: 2 ChangeByte-InsertRepeatedBytes-
#431	NEW    cov: 10201 ft: 33955 corp: 209/181Kb lim: 2091 exec/s: 22 rss: 159Mb L: 2091/2091 MS: 2 CMP-CrossOver- DE: "dest_entity_id"-
#432	REDUCE cov: 10201 ft: 33955 corp: 209/181Kb lim: 2091 exec/s: 22 rss: 159Mb L: 1911/2091 MS: 1 EraseBytes-
Segmentation fault (core dumped)
#104446	REDUCE cov: 6554 ft: 31504 corp: 5703/2771Kb lim: 2790 exec/s: 629 rss: 504Mb L: 341/2757 MS: 2 ShuffleBytes-EraseBytes-
INFO: exiting: 139 time: 843s
#104735	REDUCE cov: 6554 ft: 31504 corp: 5703/2771Kb lim: 2790 exec/s: 627 rss: 504Mb L: 523/2757 MS: 2 ChangeByte-EraseBytes-
#105054	NEW    cov: 6558 ft: 31508 corp: 5704/2772Kb lim: 2790 exec/s: 629 rss: 504Mb L: 584/2757 MS: 4 ChangeBinInt-ShuffleBytes-ChangeBit-PersAutoDict- DE: "new_leader_hfid"-
#106017	REDUCE cov: 6558 ft: 31508 corp: 5704/2772Kb lim: 2790 exec/s: 631 rss: 504Mb L: 397/2757 MS: 2 InsertRepeatedBytes-EraseBytes-

Is there something misconfigured or what can I do to actually find out where things go wrong? Otherwise there is not much need for a fuzzer if I can not find the problems... This could be false positives, as afl does not seem to find any crashes (yet).

closed time in a day

ralpha

issue commentrust-fuzz/cargo-fuzz

Fuzz target exited with signal: 11

effective duplicate of #235

ralpha

comment created time in a day

Pull request review commentrust-lang/rust

cg_llvm: split dwarf support

 options! {DebuggingOptions, DebuggingSetter, basic_debugging_options,         "hash algorithm of source files in debug info (`md5`, or `sha1`)"),     strip: Strip = (Strip::None, parse_strip, [UNTRACKED],         "tell the linker which information to strip (`none` (default), `debuginfo` or `symbols`)"),+    split_dwarf: SplitDwarfKind = (SplitDwarfKind::None, parse_split_dwarf_kind, [UNTRACKED],+        "enable generation of split dwarf"),+    split_dwarf_inlining: bool = (true, parse_bool, [UNTRACKED],

My reading of the LLVM code and the description of this flag itself suggests that this flag controls whether minimal debug info should also be generated within CUs.

davidtwco

comment created time in 2 days

PullRequestReviewEvent

Pull request review commentrust-lang/rust

cg_llvm: split dwarf support

 options! {DebuggingOptions, DebuggingSetter, basic_debugging_options,         "hash algorithm of source files in debug info (`md5`, or `sha1`)"),     strip: Strip = (Strip::None, parse_strip, [UNTRACKED],         "tell the linker which information to strip (`none` (default), `debuginfo` or `symbols`)"),+    split_dwarf: SplitDwarfKind = (SplitDwarfKind::None, parse_split_dwarf_kind, [UNTRACKED],+        "enable generation of split dwarf"),+    split_dwarf_inlining: bool = (true, parse_bool, [UNTRACKED],

bikeshed colour nit: verb this into inline_split_dwarf

davidtwco

comment created time in 2 days

Pull request review commentrust-lang/rust

cg_llvm: split dwarf support

 impl ModuleConfig {     } } -// HACK(eddyb) work around `#[derive]` producing wrong bounds for `Clone`.-pub struct TargetMachineFactory<B: WriteBackendMethods>(-    pub Arc<dyn Fn() -> Result<B::TargetMachine, String> + Send + Sync>,-);--impl<B: WriteBackendMethods> Clone for TargetMachineFactory<B> {-    fn clone(&self) -> Self {-        TargetMachineFactory(self.0.clone())-    }+/// Configuration passed to the function returned by the `target_machine_factory`.+pub struct TargetMachineFactoryConfig<'a> {+    /// Split DWARF is enabled in LLVM by checking that `TM.MCOptions.SplitDwarfFile` isn't empty,

Hmm… this probably should not be LLVM-specific, but I don’t yet know how the alternative backends could handle this being set (error out? transparently ignore?)

davidtwco

comment created time in 2 days

PullRequestReviewEvent
PullRequestReviewEvent

pull request commentrust-lang/rust

Use less divisions in display u128/i128

Still somewhat uncomfortable with us encoding a division algorithm manually, but I guess we can remove it later once LLVM does the right thing.

@bors r+

JulianKnodt

comment created time in 2 days

pull request commentrust-lang/rust

Pass tune-cpu to LLVM

This overall LGTM. There are remaining compilation issues remaining before this can land and the tune-cpu documentation still needs a note that it is unstable. Please also squash.

Artoria2e5

comment created time in 2 days

pull request commentrust-lang/rust

Return values up to 128 bits in registers

Can you add a codegen test for this?

r=me after that.

jonas-schievink

comment created time in 2 days

pull request commentrust-lang/rust

Fix FIXME in core::num test: Check sign of zero in min/max tests.

@bors r+ rollup

m-ou-se

comment created time in 2 days

issue closednagisa/rust_libloading

support opening existing library

Hello,

Currently you can do Library::new() which loads a library. For my use case it would be create to have Library::open that opens an already existing library.

  • I'm trying to query optionally exported functions in preloaded shared library.

Implementation can be accomplished with the RTLD_NOLOAD flag for dlopen on most unixes and GetModuleHandleEx in windows.

I can write the code myself and submit a PR if that works for you.

closed time in 3 days

yuval-k

issue commentrust-lang/rust

Use dynamic linking and download LLVM from CI for rustc across platforms

LLVM’s bugzilla is pretty much the only place for bug reports. If its something less defined than a precise bug report, talking it out on the mailing lists is the way to go.

jyn514

comment created time in 6 days

delete branch standard-ai/hedwig-rust

delete branch : nagisa/async

delete time in 6 days

delete branch standard-ai/hedwig-rust

delete branch : nagisa/bump-version

delete time in 6 days

delete branch standard-ai/hedwig-rust

delete branch : nagisa/readme

delete time in 6 days

created tagstandard-ai/hedwig-rust

tagv2.0.0

Message bus for micro-services that works on AWS and GCP

created time in 6 days

PR merged standard-ai/hedwig-rust

Update the readme
+10 -13

0 comment

1 changed file

nagisa

pr closed time in 6 days

push eventstandard-ai/hedwig-rust

Simonas Kazlauskas

commit sha 04185f055a7d64d2ed6c095565134f515f877329

Update the readme

view details

push time in 6 days

delete branch standard-ai/hedwig-rust

delete branch : nagisa/bump

delete time in 6 days

PR opened standard-ai/hedwig-rust

Reviewers
Update the readme
+10 -13

0 comment

1 changed file

pr created time in 6 days

push eventstandard-ai/hedwig-rust

Simonas Kazlauskas

commit sha 04185f055a7d64d2ed6c095565134f515f877329

Update the readme

view details

push time in 6 days

create barnchstandard-ai/hedwig-rust

branch : nagisa/readme

created branch time in 6 days

delete tag standard-ai/hedwig-rust

delete tag : v2.0.0

delete time in 6 days

created tagstandard-ai/hedwig-rust

tagv2.0.0

Message bus for micro-services that works on AWS and GCP

created time in 6 days

delete tag standard-ai/hedwig-rust

delete tag : 2.0.0

delete time in 6 days

created tagstandard-ai/hedwig-rust

tag2.0.0

Message bus for micro-services that works on AWS and GCP

created time in 6 days

push eventstandard-ai/hedwig-rust

Simonas Kazlauskas

commit sha ec12fe96153a5e8dccfbdcf72005ba00e6d4a9c1

Bump version to 2.0.0

view details

push time in 6 days

created tagstandard-ai/sendfd

tag0.3.2

Send file descriptors over unix sockets in Rust

created time in 6 days

PR merged standard-ai/sendfd

Adapt to musl

Got: cargo build --target=x86_64-unknown-linux-musl error[E0063]: missing field __pad1 in initializer of libc::cmsghdr --> /home/tim/project/virtcontainers/sendfd/src/lib.rs:92:13 | 92 | libc::cmsghdr { | ^^^^^^^^^^^^^ missing __pad1

error[E0451]: field __pad1 of struct libc::msghdr is private --> /home/tim/project/virtcontainers/sendfd/src/lib.rs:78:15 | 78 | ..mem::zeroed() | ^^^^^^^^^^^^^ field __pad1 is private

error[E0451]: field __pad2 of struct libc::msghdr is private --> /home/tim/project/virtcontainers/sendfd/src/lib.rs:78:15 | 78 | ..mem::zeroed() | ^^^^^^^^^^^^^ field __pad2 is private

This commit will resolve the problem.

Signed-off-by: Tim Zhang tim@hyper.sh

+17 -21

1 comment

1 changed file

Tim-Zhang

pr closed time in 6 days

push eventstandard-ai/sendfd

Tim Zhang

commit sha d24e55871c46137fbf695df5754ff9813ce1b868

Adapt to musl Got: cargo build --target=x86_64-unknown-linux-musl error[E0063]: missing field `__pad1` in initializer of `libc::cmsghdr` --> /home/tim/project/virtcontainers/sendfd/src/lib.rs:92:13 | 92 | libc::cmsghdr { | ^^^^^^^^^^^^^ missing `__pad1` error[E0451]: field `__pad1` of struct `libc::msghdr` is private --> /home/tim/project/virtcontainers/sendfd/src/lib.rs:78:15 | 78 | ..mem::zeroed() | ^^^^^^^^^^^^^ field `__pad1` is private error[E0451]: field `__pad2` of struct `libc::msghdr` is private --> /home/tim/project/virtcontainers/sendfd/src/lib.rs:78:15 | 78 | ..mem::zeroed() | ^^^^^^^^^^^^^ field `__pad2` is private This commit will resolve the problem. Signed-off-by: Tim Zhang <tim@hyper.sh>

view details

Simonas Kazlauskas

commit sha 7eb69eb4dd029c4639edd58e76f81af8f59f58c6

Bump to 0.3.2

view details

push time in 6 days

pull request commentrust-lang/rfcs

RFC: Promote aarch64-unknown-linux-gnu to a Tier-1 Rust target

@rfcbot concern should stack probes be implemented before this target is promoted

The compiler team briefly discussed this in their meeting, and we're ready to move into FCP! If someone on the compiler team feels like blocking the merge on addressing the stack probes question please add a concern!

The relevant discussion is here.

I am adding the concern as I believe we should strive to not regress the soundness properties across our T1 targets now that this particular hole has been patched up, regardless of however long it was open in the past before it got fixed. Or at very least we should make a conscious decision and figure out our policy wrt this question.

raw-bin

comment created time in 6 days

Pull request review commentstandard-ai/sendfd

Adapt to musl

 fn send_with_fd(socket: RawFd, bs: &[u8], fds: &[RawFd]) -> io::Result<usize> {         ptr::write(             cmsg_header,             libc::cmsghdr {+                #[cfg(target_env = "musl")]

Would it perhaps make more sense to construct cmsghdr by assigning fields to a zeroed value the same way, it is done for msghdr above?

Tim-Zhang

comment created time in 6 days

PullRequestReviewEvent

PR opened standard-ai/hedwig-rust

Bump version to 2.0.0
+1 -1

0 comment

1 changed file

pr created time in 6 days

create barnchstandard-ai/hedwig-rust

branch : nagisa/bump

created branch time in 6 days

delete branch standard-ai/hedwig-rust

delete branch : nagisa/error-rework

delete time in 6 days

PR merged standard-ai/hedwig-rust

Reviewers
Refactor the library

This commit refactors this library, almost from scratch, in order to:

  1. Achieve more conventional and easier-to-use APIs;
  2. More correct and flexible error handling;
  3. Internal segmentation of batches inside of google publisher.

The initial motivation was originally 3, but then old way of implicit error handling came up as a problem as well. I attempted to introduce some mechanism to handle errors in a more granular way without adjusting API of the library too much, but it turned out to be not very feasible.

The new API is centered around the new hedwig::Message trait which the users would implement to associate data types with various information that is necessary for the hewdig library – the topic name, the validators applicable to the data, the encoding mechanisms, etc. Furthermore, it now allows retaining all of auxiliary data alongside the message data as well, giving further flexibility in how this library is used by the users.

All in all, the trait was designed in a way that would allow to move the message-encoding specific pieces of code away from the business logic as much as possible. For now the implementation of this trait remains manual, but there is nothing that would prevent it from being deriveable in some way.


The error handling was made entirely explicit in the new API. In particular we no longer implicitly re-queue messages in the batch (nor do we reuse the batches, anymore). Instead the users receive a Stream of (result, topic, validated message). They can then choose to push the validated message back into a new batch they are constructing to retry the message… or just handle the error in whatever other way that makes sense to them.

One important improvement as part of this change is simplification of behaviour when the futures/streams are cancelled. Previously dropping the publish future would leave one in a fairly difficult to understand state where the batch might have some of the messages re-queued and some of them cancelled, etc.

Since there's no longer any state remaining in hedwig, dropping streams to cancel starts making significantly more sense as well.


Google PubSub has various limitations on the API requests that are made to its endpoints. As we are in charge of batching the requests up to implement publishing more efficiently, it is also up to us to ensure that we do not exceed any of the limits upstream has placed on us.

To do this we implement a batch segmentation algorithm in GoogleMessageSegmenter that the GooglePubSubPublisher uses internally to split up the messages.

Fixes #18

+1746 -1175

5 comments

15 changed files

nagisa

pr closed time in 6 days

push eventstandard-ai/hedwig-rust

Simonas Kazlauskas

commit sha 5c3afc9dd64750137792993ad9226bfe13f72c2a

Refactor the library This commit refactors this library, almost from scratch, in order to: 1. Achieve more conventional and easier-to-use APIs; 2. More correct and flexible error handling; 3. Internal segmentation of batches inside of google publisher. The initial motivation was originally 3, but then old way of implicit error handling came up as a problem as well. I attempted to introduce some mechanism to handle errors in a more granular way without adjusting API of the library too much, but it turned out to be not very feasible. The new API is centered around the new `hedwig::Message` trait which the users would implement to associate data types with various information that is necessary for the hewdig library – the topic name, the validators applicable to the data, the encoding mechanisms, etc. Furthermore, it now allows retaining all of auxiliary data alongside the message data as well, giving further flexibility in how this library is used by the users. All in all, the trait was designed in a way that would allow to move the message-encoding specific pieces of code away from the business logic as much as possible. For now the implementation of this trait remains manual, but there is nothing that would prevent it from being `derive`able in some way. --- The error handling was made entirely explicit in the new API. In particular we no longer implicitly re-queue messages in the batch (nor do we reuse the batches, anymore). Instead the users receive a `Stream` of `(result, topic, validated message)`. They can then choose to `push` the validated message back into a new batch they are constructing to retry the message… or just handle the error in whatever other way that makes sense to them. One important improvement as part of this change is simplification of behaviour when the futures/streams are cancelled. Previously dropping the `publish` future would leave one in a fairly difficult to understand state where the batch might have some of the messages re-queued and some of them cancelled, etc. Since there's no longer any state remaining in `hedwig`, dropping streams to cancel starts making significantly more sense as well. --- Google PubSub has various limitations on the API requests that are made to its endpoints. As we are in charge of batching the requests up to implement publishing more efficiently, it is also up to us to ensure that we do not exceed any of the limits upstream has placed on us. To do this we implement a batch segmentation algorithm in `GoogleMessageSegmenter` that the GooglePubSubPublisher uses internally to split up the messages. Fixes #18

view details

push time in 6 days

issue closedstandard-ai/hedwig-rust

Segment requests to google pubsub when publishing larger publish batches

There appears to be some sort of an undocumented request body size limit when making a :publish request.

Error message suggests

Request payload size exceeds the limit: 10485760 bytes.

which is ~10MiB.

We should ensure that when we format a request we don’t exceed this size.

closed time in 6 days

nagisa

issue commentrust-lang/rust

Fast algorithm for u128 (and i128) divided by small constant

I’ve drafted a LLVM differential to fix this https://reviews.llvm.org/D87976.

leonardo-m

comment created time in 9 days

push eventnagisa/rust

Simonas Kazlauskas

commit sha fa6f4e542baec87117367fd2f496384caff175ab

error to core triggers coherence right now :(

view details

push time in 9 days

issue commentrust-lang/project-error-handling

Prototype `core::error::Error` to prove that stabilizing `fn backtrace` will not prevent a later move of the error trait to core

I attempted to prototype the move to libcore with backtrace removed in this branch.

The mentioned downcast is indeed not at all an issue (I applied the suggestion I made in this comment), but

impl<'a> From<&str> for Box<dyn Error + Send + Sync + 'a> {

comes up as an issue. In particular it will trigger coherence's wrath with

error[E0119]: conflicting implementations of trait `core::convert::From<&str>` for type `boxed::Box<dyn core::error::Error + core::marker::Send + core::marker::Sync>`:
    --> library/alloc/src/errors.rs:66:1
     |
66   | impl<'a, E: Error + Send + Sync + 'a> From<E> for Box<dyn Error + Send + Sync + 'a> {
     | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ conflicting implementation for `boxed::Box<dyn core::error::Error + core::marker::Send + core::marker::Sync>`
     |
    ::: library/alloc/src/boxed.rs:1155:1
     |
1155 | impl<'a> From<&str> for Box<dyn Error + Send + Sync + 'a> {
     | --------------------------------------------------------- first implementation here
     |
     = note: upstream crates may add a new impl of trait `core::error::Error` for type `&str` in future versions
yaahc

comment created time in 9 days

create barnchnagisa/rust

branch : error-to-core

created branch time in 9 days

issue commentrust-lang/rust

[AVR] Failure to compile libcore on nightly

Deciphering the failing expression it looks like we're trying to assert that the alignment of an AtomicPtr is equivalent to that of *mut ().

As per the data-layout of the linked target file (p:16:8) the pointer alignment is indeed set to 1 byte and we explicitly set the alignment of the AtomicPtr to 2 here introduced in this commit many moons ago, hence the bug.

The assertion itself was added in this commit and makes sense for what the function does cc @m-ou-se.

I would check if you can adjust the data layout to specify pointers are aligned to 2 bytes (you probably cannot).

I suspect the correct fix is probably to figure out how to set repr(align(ptr)) for AtomicPtr.

abusch

comment created time in 10 days

pull request commentrust-lang/stacker

Convert callback to `dyn` to improve compile times

Released stacker 0.1.12 with this change.

Julian-Wollersberger

comment created time in 10 days

created tagrust-lang/stacker

tagstacker-0.1.12

Manual segmented stacks for Rust

created time in 10 days

PR merged rust-lang/stacker

Convert callback to `dyn` to improve compile times

To avoid monomorphizing psm::on_stack::with_on_stack 1500 times in rustc, this PR converts the callback passed to grow _grow to dyn FnMut(). The dynamic dispatch happens on the slow path.

This should improve rustc's compile time a bit, and surprisingly also improves it's runtime: perf-results measured in rust-lang/rust#76680

This was discovered with the cargo-llvm-lines tool.

+25 -9

1 comment

1 changed file

Julian-Wollersberger

pr closed time in 10 days

push eventrust-lang/stacker

Julian Wollersberger

commit sha 08dbc14c5b5594cbd8792665dfbeee9ac73f7269

To avoid monomorphizing `psm::on_stack::with_on_stack` 1500 times in rustc, convert the callback passed to grow `_grow` to `dyn FnMut()`. This should improve rustc's compile time a bit, and surprisingly also improves runtime in some cases. https://perf.rust-lang.org/compare.html?start=7402a394471a6738a40fea7d4f1891666e5a80c5&end=f242334c1e2e719cf1cba923923ad8ec62affb71

view details

Simonas Kazlauskas

commit sha 6d3e250a5628d5df9ad9acb0e3da371c0acd3de9

stacker -> 0.1.12

view details

push time in 10 days

Pull request review commentrust-lang/rust

Pass tune-cpu to LLVM

 This also supports the feature `+crt-static` and `-crt-static` to control Each target and [`target-cpu`](#target-cpu) has a default set of enabled features. +## tune-cpu

Moving this option to -Z made it unstable and not-actually-a-codegen-option. At the very least this should be annotated as unstable.

Artoria2e5

comment created time in 10 days

Pull request review commentrust-lang/rust

Pass tune-cpu to LLVM

 pub fn apply_target_cpu_attr(cx: &CodegenCx<'ll, '_>, llfn: &'ll Value) {     ); } +pub fn apply_tune_cpu_attr(cx: &CodegenCx<'ll, '_>, llfn: &'ll Value) {

This applies the attribute regardless of whether the option is specified or not. Is there a good reason why we're applying it even if the -Ztune-cpu is not specified?

Artoria2e5

comment created time in 10 days

PullRequestReviewEvent
PullRequestReviewEvent

issue commentrust-lang/compiler-team

Add StatementKind::Intrinsic to MIR

@rustbot third

I’m willing to review changes that implement this too, so do r? me on implementation PRs as well.

JulianKnodt

comment created time in 10 days

startedcpcloud/tensorflow_proto

started time in 10 days

PullRequestReviewEvent

push eventstandard-ai/hedwig-rust

Simonas Kazlauskas

commit sha 5c3afc9dd64750137792993ad9226bfe13f72c2a

Refactor the library This commit refactors this library, almost from scratch, in order to: 1. Achieve more conventional and easier-to-use APIs; 2. More correct and flexible error handling; 3. Internal segmentation of batches inside of google publisher. The initial motivation was originally 3, but then old way of implicit error handling came up as a problem as well. I attempted to introduce some mechanism to handle errors in a more granular way without adjusting API of the library too much, but it turned out to be not very feasible. The new API is centered around the new `hedwig::Message` trait which the users would implement to associate data types with various information that is necessary for the hewdig library – the topic name, the validators applicable to the data, the encoding mechanisms, etc. Furthermore, it now allows retaining all of auxiliary data alongside the message data as well, giving further flexibility in how this library is used by the users. All in all, the trait was designed in a way that would allow to move the message-encoding specific pieces of code away from the business logic as much as possible. For now the implementation of this trait remains manual, but there is nothing that would prevent it from being `derive`able in some way. --- The error handling was made entirely explicit in the new API. In particular we no longer implicitly re-queue messages in the batch (nor do we reuse the batches, anymore). Instead the users receive a `Stream` of `(result, topic, validated message)`. They can then choose to `push` the validated message back into a new batch they are constructing to retry the message… or just handle the error in whatever other way that makes sense to them. One important improvement as part of this change is simplification of behaviour when the futures/streams are cancelled. Previously dropping the `publish` future would leave one in a fairly difficult to understand state where the batch might have some of the messages re-queued and some of them cancelled, etc. Since there's no longer any state remaining in `hedwig`, dropping streams to cancel starts making significantly more sense as well. --- Google PubSub has various limitations on the API requests that are made to its endpoints. As we are in charge of batching the requests up to implement publishing more efficiently, it is also up to us to ensure that we do not exceed any of the limits upstream has placed on us. To do this we implement a batch segmentation algorithm in `GoogleMessageSegmenter` that the GooglePubSubPublisher uses internally to split up the messages. Fixes #18

view details

push time in 11 days

push eventstandard-ai/hedwig-rust

Simonas Kazlauskas

commit sha d6aa5b29368d84cf78a2d6d8e784cd8111927def

Refactor the library This commit refactors this library, almost from scratch, in order to: 1. Achieve more conventional and easier-to-use APIs; 2. More correct and flexible error handling; 3. Internal segmentation of batches inside of google publisher. The initial motivation was originally 3, but then old way of implicit error handling came up as a problem as well. I attempted to introduce some mechanism to handle errors in a more granular way without adjusting API of the library too much, but it turned out to be not very feasible. The new API is centered around the new `hedwig::Message` trait which the users would implement to associate data types with various information that is necessary for the hewdig library – the topic name, the validators applicable to the data, the encoding mechanisms, etc. Furthermore, it now allows retaining all of auxiliary data alongside the message data as well, giving further flexibility in how this library is used by the users. All in all, the trait was designed in a way that would allow to move the message-encoding specific pieces of code away from the business logic as much as possible. For now the implementation of this trait remains manual, but there is nothing that would prevent it from being `derive`able in some way. --- The error handling was made entirely explicit in the new API. In particular we no longer implicitly re-queue messages in the batch (nor do we reuse the batches, anymore). Instead the users receive a `Stream` of `(result, topic, validated message)`. They can then choose to `push` the validated message back into a new batch they are constructing to retry the message… or just handle the error in whatever other way that makes sense to them. One important improvement as part of this change is simplification of behaviour when the futures/streams are cancelled. Previously dropping the `publish` future would leave one in a fairly difficult to understand state where the batch might have some of the messages re-queued and some of them cancelled, etc. Since there's no longer any state remaining in `hedwig`, dropping streams to cancel starts making significantly more sense as well. --- Google PubSub has various limitations on the API requests that are made to its endpoints. As we are in charge of batching the requests up to implement publishing more efficiently, it is also up to us to ensure that we do not exceed any of the limits upstream has placed on us. To do this we implement a batch segmentation algorithm in `GoogleMessageSegmenter` that the GooglePubSubPublisher uses internally to split up the messages. Fixes #18

view details

push time in 11 days

push eventstandard-ai/hedwig-rust

Simonas Kazlauskas

commit sha eb1f69dc8c0db4fe5f8d274e9cf0224304024013

Adjust for review comments

view details

push time in 11 days

Pull request review commentstandard-ai/hedwig-rust

Refactor the library

-//! A Hedwig library for Rust. Hedwig is a message bus that works with arbitrary pubsub services-//! such as AWS SNS/SQS or Google Cloud Pubsub. Messages are validated using a JSON schema. The-//! publisher and consumer are de-coupled and fan-out is supported out of the box.+//! Hedwig is a message bus library that works with arbitrary pubsub services such as AWS SNS/SQS+//! or Google Cloud Pubsub. Messages are validated before they are published. The publisher and+//! consumer are de-coupled and fan-out is supported out of the box. //!-//! # Example: publish a new message+//! The Rust library currently only supports publishing. //!-//! ```no_run-//! use hedwig::{Hedwig, MajorVersion, MinorVersion, Version, Message, publishers::MockPublisher};-//! # use std::path::Path;-//! # use serde::Serialize;-//! # use strum_macros::IntoStaticStr;+//! # Examples //!-//! fn main() -> Result<(), Box<dyn std::error::Error>> {-//! let schema = r#"-//!     {-//!       "$id": "https://hedwig.standard.ai/schema",-//!       "$schema": "https://json-schema.org/draft-04/schema#",-//!       "description": "Example Schema",-//!       "schemas": {-//!           "user-created": {-//!               "1.*": {-//!                   "description": "A new user was created",-//!                   "type": "object",-//!                   "x-versions": [-//!                       "1.0"-//!                   ],-//!                   "required": [-//!                       "user_id"-//!                   ],-//!                   "properties": {-//!                       "user_id": {-//!                           "$ref": "https://hedwig.standard.ai/schema#/definitions/UserId/1.0"-//!                       }-//!                   }-//!               }-//!           }-//!       },-//!       "definitions": {-//!           "UserId": {-//!               "1.0": {-//!                   "type": "string"-//!               }-//!           }-//!       }-//!     }"#;+//! Publish a message. Payload encoded with JSON and validated using a JSON Schema. //!-//!     # #[derive(Clone, Copy, IntoStaticStr, Hash, PartialEq, Eq)]-//!     # enum MessageType {-//!     #    #[strum(serialize = "user.created")]-//!     #    UserCreated,-//!     # }-//!     #-//!     # #[derive(Serialize)]-//!     # struct UserCreatedData {-//!     #     user_id: String,-//!     # }-//!     #-//!     fn router(t: MessageType, v: MajorVersion) -> Option<&'static str> {-//!         match (t, v) {-//!             (MessageType::UserCreated, MajorVersion(1)) => Some("dev-user-created-v1"),-//!             _ => None,+//! ```+//! use uuid::Uuid;+//! use std::{path::Path, time::SystemTime};+//! use futures_util::stream::StreamExt;+//!+//! # #[cfg(not(feature = "json-schema"))]+//! # fn main() {}+//!+//! # #[cfg(feature = "json-schema")] // example uses a JSON Schema validator.+//! # fn main() -> Result<(), Box<dyn std::error::Error>> {+//! let schema = r#"{+//!     "$id": "https://hedwig.corp/schema",+//!     "$schema": "https://json-schema.org/draft-04/schema#",+//!     "description": "Example Schema",+//!     "schemas": {+//!         "user-created": {+//!             "1.*": {+//!                 "description": "A new user was created",+//!                 "type": "object",+//!                 "x-versions": [+//!                     "1.0"+//!                 ],+//!                 "required": [+//!                     "user_id"+//!                 ],+//!                 "properties": {+//!                     "user_id": {+//!                         "$ref": "https://hedwig.corp/schema#/definitions/UserId/1.0"+//!                     }+//!                 }+//!             }+//!         }+//!     },+//!     "definitions": {+//!         "UserId": {+//!             "1.0": {+//!                 "type": "string"+//!             } //!         } //!     }+//! }"#; //!-//!     // create a publisher instance-//!     let publisher = MockPublisher::default();-//!     let hedwig = Hedwig::new(-//!         schema,-//!         "myapp",-//!         publisher,-//!         router,-//!     )?;+//! #[derive(serde::Serialize)]+//! struct UserCreatedMessage {+//!     user_id: String,+//! } //!-//!     async {-//!         let published_ids = hedwig.publish(Message::new(-//!             MessageType::UserCreated,-//!             Version(MajorVersion(1), MinorVersion(0)),-//!             UserCreatedData { user_id: "U_123".into() }-//!         )).await;-//!     };+//! impl<'a> hedwig::Message for &'a UserCreatedMessage {+//!     type Error = hedwig::validators::JsonSchemaValidatorError;+//!     type Validator = hedwig::validators::JsonSchemaValidator;+//!     fn topic(&self) -> &'static str { "user.created" }+//!     fn encode(self, validator: &Self::Validator)+//!     -> Result<hedwig::ValidatedMessage, Self::Error> {+//!         validator.validate(+//!             Uuid::new_v4(),+//!             SystemTime::now(),+//!             "https://hedwig.corp/schema#/schemas/user.created/1.0",+//!             hedwig::Headers::new(),+//!             self,+//!         )+//!     }+//! } //!-//!     # Ok(())+//! let publisher = /* Some publisher */+//! # hedwig::publishers::NullPublisher;+//! let validator = hedwig::validators::JsonSchemaValidator::new(schema)?;+//! let mut batch = hedwig::PublishBatch::new();+//! batch.message(&validator, &UserCreatedMessage { user_id: String::from("U_123") });+//! let mut result_stream = batch.publish(&publisher);+//! let mut next_batch = hedwig::PublishBatch::new();+//! async {+//!     while let Some(result) = result_stream.next().await {+//!         match result {+//!             (Ok(id), _, msg) => {+//!                 println!("message {} published successfully: {:?}", msg.uuid(), id);+//!             }+//!             (Err(e), topic, msg) => {+//!                 eprintln!("failed to publish {}: {}", msg.uuid(), e);+//!                 next_batch.push(topic, msg);+//!             }+//!         }+//!     }+//! };+//! # Ok(()) //! # } //! ``` #![deny(     missing_docs,-    intra_doc_link_resolution_failure,+    broken_intra_doc_links,     clippy::all,     unsafe_code,     unreachable_pub,-    unused,+    unused )] #![cfg_attr(docsrs, feature(doc_cfg))] -use std::{-    collections::HashMap,-    fmt,-    future::Future,-    mem,-    time::{SystemTime, UNIX_EPOCH},-};+use std::{collections::BTreeMap, time::SystemTime}; -use futures::stream::StreamExt;+use futures_util::stream::{self, Stream, StreamExt}; use uuid::Uuid;-use valico::json_schema::{SchemaError, Scope, ValidationState};--#[cfg(feature = "google")]-#[cfg_attr(docsrs, doc(cfg(feature = "google")))]-mod google_publisher;-mod mock_publisher;-mod null_publisher;--/// Implementations of the Publisher trait-pub mod publishers {-    #[cfg(feature = "google")]-    #[cfg_attr(docsrs, doc(cfg(feature = "google")))]-    pub use super::google_publisher::GooglePubSubPublisher;-    pub use super::mock_publisher::MockPublisher;-    pub use super::null_publisher::NullPublisher;-} -const FORMAT_VERSION_V1: Version = Version(MajorVersion(1), MinorVersion(0));+pub mod publishers;+#[cfg(test)]+mod tests;+pub mod validators; -/// All errors that may be returned when instantiating a new Hedwig instance.+/// All errors that may be returned when operating top level APIs. #[derive(Debug, thiserror::Error)] #[non_exhaustive] pub enum Error {-    /// Unable to deserialize schema-    #[error("Unable to deserialize schema")]-    SchemaDeserialize(#[source] serde_json::Error),--    /// Schema failed to compile-    #[error("Schema failed to compile")]-    SchemaCompile(#[from] SchemaError),--    /// Unable to serialize message-    #[error("Unable to serialize message")]-    MessageSerialize(#[source] serde_json::Error),--    /// Message is not routable-    #[error("Message {0} is not routable")]-    MessageRoute(Uuid),--    /// Could not parse a schema URL-    #[error("Could not parse `{1}` as a schema URL")]-    SchemaUrlParse(#[source] url::ParseError, String),--    /// Could not resolve the schema URL-    #[error("Could not resolve `{0}` to a schema")]-    SchemaUrlResolve(url::Url),--    /// Could not validate message data-    #[error("Message data does not validate per the schema: {0}")]-    DataValidation(String),--    /// Publisher failed to publish a message-    #[error("Publisher failed to publish a message batch")]-    Publisher(#[source] Box<dyn std::error::Error + Send + Sync>),--    /// Publisher failed to publish multiple batches of messages-    #[error("Publisher failed to publish multiple batches (total of {} errors)", _1.len() + 1)]-    PublisherMultiple(-        #[source] Box<dyn std::error::Error + Send + Sync>,-        Vec<Box<dyn std::error::Error + Send + Sync>>,-    ),-}--type AnyError = Box<dyn std::error::Error + Send + Sync>;--/// The special result type for [`Publisher::publish`](trait.Publisher.html)-#[derive(Debug)]-pub enum PublisherResult<Id> {-    /// Publisher succeeded.-    ///-    /// Contains a vector of published message IDs.-    Success(Vec<Id>),-    /// Publisher failed to publish any of the messages.-    OneError(AnyError, Vec<ValidatedMessage>),-    /// Publisher failed to publish some of the messages.-    ///-    /// The error type has a per-message granularity.-    PerMessage(Vec<Result<Id, (AnyError, ValidatedMessage)>>),+    /// Unable to encode message payload+    #[error("Unable to encode message payload")]+    EncodeMessage(#[source] Box<dyn std::error::Error + Send + Sync>), } -/// Interface for message publishers+/// Message publishers.+///+/// Message publishers deliver a validated message to an endpoint, possibly a remote one. Message+/// publishers may also additionally validate a message for publisher-specific requirements (e.g.+/// size). pub trait Publisher {-    /// The list of identifiers for successfully published messages+    /// The identifier for a successfully published message.     type MessageId: 'static;-    /// The future that the `publish` method returns-    type PublishFuture: Future<Output = PublisherResult<Self::MessageId>> + Send;--    /// Publish a batch of messages-    ///-    /// # Return value-    ///-    /// Shall return [`PublisherResult::Success`](PublisherResult::Success) only if all of the-    /// messages are successfully published. Otherwise `PublisherResult::OneError` or-    /// `PublisherResult::PerMessage` shall be returned to indicate an error.-    fn publish(&self, topic: &'static str, messages: Vec<ValidatedMessage>) -> Self::PublishFuture;-}--/// Type alias for custom headers associated with a message-type Headers = HashMap<String, String>; -struct Validator {-    scope: Scope,-    schema_id: url::Url,-}--impl Validator {-    fn new(schema: &str) -> Result<Validator, Error> {-        let master_schema: serde_json::Value =-            serde_json::from_str(schema).map_err(Error::SchemaDeserialize)?;+    /// The error that this publisher returns when publishing of a message fails.+    type MessageError: std::error::Error + Send + Sync + 'static; -        let mut scope = Scope::new();-        let schema_id = scope.compile(master_schema, false)?;+    /// The stream of results that the `publish` method returns.+    type PublishStream: Stream<Item = Result<Self::MessageId, Self::MessageError>>; -        Ok(Validator { scope, schema_id })-    }--    fn validate<D, T>(-        &self,-        message: &Message<D, T>,-        schema: &str,-    ) -> Result<ValidationState, Error>+    /// Publish a batch of messages.+    ///+    /// The output stream shall return a result for each message in `messages` slice in order.+    fn publish<'a, I>(&self, topic: &'static str, messages: I) -> Self::PublishStream     where-        D: serde::Serialize,-    {-        // convert user.created/1.0 -> user.created/1.*-        let msg_schema_ptr = schema.trim_end_matches(char::is_numeric).to_owned() + "*";-        let msg_schema_url = url::Url::parse(&msg_schema_ptr)-            .map_err(|e| Error::SchemaUrlParse(e, msg_schema_ptr))?;-        let msg_schema = self-            .scope-            .resolve(&msg_schema_url)-            .ok_or_else(|| Error::SchemaUrlResolve(msg_schema_url))?;--        let msg_data = serde_json::to_value(&message.data).map_err(Error::MessageSerialize)?;--        let validation_state = msg_schema.validate(&msg_data);-        if !validation_state.is_strictly_valid() {-            return Err(Error::DataValidation(format!("{:?}", validation_state)));-        }-        Ok(validation_state)-    }+        I: Iterator<Item = &'a ValidatedMessage> + DoubleEndedIterator + ExactSizeIterator; } -/// Major part component in semver-#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq, serde::Serialize)]-pub struct MajorVersion(pub u8);+/// Types that can be encoded and published.+pub trait Message {+    /// The errors that can occur when calling the [`Message::encode`] method.+    ///+    /// Will typically match the errors returned by the [`Message::Validator`].+    type Error: std::error::Error + Send + Sync + 'static; -impl fmt::Display for MajorVersion {-    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {-        write!(f, "{}", self.0)-    }-}+    /// The validator to use for this message.+    type Validator; -/// Minor part component in semver-#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq, serde::Serialize)]-pub struct MinorVersion(pub u8);+    /// Topic into which this message shall be published.+    fn topic(&self) -> &'static str;

Yeah, they ought to be fairly fixed. The only kind of variation I can see is the -v{version} suffix in{topic}-{version}, but that can and probably should be a separate implementer ofMessage` anyway.

nagisa

comment created time in 11 days

PullRequestReviewEvent

push eventstandard-ai/hedwig-rust

Simonas Kazlauskas

commit sha 87e0750dee2b7d8668916effe4c7b4e413028767

Expand tests to exercise header validation

view details

push time in 12 days

Pull request review commentrust-lang/stacker

Convert callback to `dyn` to improve compile times

 pub fn maybe_grow<R, F: FnOnce() -> R>(red_zone: usize, stack_size: usize, callb /// The closure will still be on the same thread as the caller of `grow`. /// This will allocate a new stack with at least `stack_size` bytes. pub fn grow<R, F: FnOnce() -> R>(stack_size: usize, callback: F) -> R {+    // Measuring with cargo-llvm-lines revealed that `psm::on_stack::with_on_stack`+    // was monomorphized 1552 times in rustc and was responsible for 1.5% of+    // rustc's total llvm IR lines. That takes time for LLVM to process.+    // Converting the generic callback to a dynamic one removes all that duplication.+    let mut opt_callback = Some(callback);     let mut ret = None;     let ret_ref = &mut ret;-    _grow(stack_size, move || {-        *ret_ref = Some(callback());-    });++    // This wrapper around `callback` achieves two things:+    // * It converts the `impl FnOnce` to a `dyn FnMut`.+    //   `dyn` because we want it to not be generic, and+    //   `FnMut` because we can't pass a `dyn FnOnce` around without boxing it.+    // * It eliminates the generic return value, by writing it to the stack of this function.+    //   Otherwise the closure would have to return an unsized value, which isn't possible.+    let dyn_callback: &mut dyn FnMut() = &mut || {+        let taken_callback = opt_callback.take().unwrap();+        *ret_ref = Some(taken_callback());+    };++    // _grow is only monomorphized once with a `dyn` argument.+    _grow(stack_size, dyn_callback);

Would make sense to me.

Julian-Wollersberger

comment created time in 13 days

PullRequestReviewEvent

Pull request review commentrust-lang/stacker

Convert callback to `dyn` to improve compile times

 pub fn maybe_grow<R, F: FnOnce() -> R>(red_zone: usize, stack_size: usize, callb /// The closure will still be on the same thread as the caller of `grow`. /// This will allocate a new stack with at least `stack_size` bytes. pub fn grow<R, F: FnOnce() -> R>(stack_size: usize, callback: F) -> R {+    // Measuring with cargo-llvm-lines revealed that `psm::on_stack::with_on_stack`+    // was monomorphized 1552 times in rustc and was responsible for 1.5% of+    // rustc's total llvm IR lines. That takes time for LLVM to process.

This metric is probably very misleading, because with_on_stack is the base frame into which LLVM is able to inline the provided callback. with_on_stack is also super small itself (< dozen of instructions), so I have a hard time believing that by itself with_on_stack is a cause of the entirety of 1.5% of instructions.

With that in mind, making the callback dynamic only serves to prevent the inlining of the callback, and thus now all those closures end up having their own "category" or "line" in cargo-llvm-lines output. The improvement in perf/comptime still great, however.

Julian-Wollersberger

comment created time in 13 days

PullRequestReviewEvent

pull request commentstandard-ai/hedwig-rust

Refactor the library

While there are a couple remaining fixmes around in the code, I think its done enough to be in a state that's reviewable enough to produce an useful review of the approach/code.

nagisa

comment created time in 13 days

Pull request review commentstandard-ai/hedwig-rust

Segmentation of batches for google publisher

 pub enum Error {     #[error("Message data does not validate per the schema: {0}")]     DataValidation(String), -    /// Publisher failed to publish a message-    #[error("Publisher failed to publish a message batch")]-    Publisher(#[source] Box<dyn std::error::Error + Send + Sync>),--    /// Publisher failed to publish multiple batches of messages-    #[error("Publisher failed to publish multiple batches (total of {} errors)", _1.len() + 1)]-    PublisherMultiple(-        #[source] Box<dyn std::error::Error + Send + Sync>,-        Vec<Box<dyn std::error::Error + Send + Sync>>,-    ),-}--type AnyError = Box<dyn std::error::Error + Send + Sync>;+    /// Could not validate message data+    #[error("Could not add a message to the builder")]+    AddMessage(#[source] Box<dyn std::error::Error + Send + Sync>), -/// The special result type for [`Publisher::publish`](trait.Publisher.html)-#[derive(Debug)]-pub enum PublisherResult<Id> {-    /// Publisher succeeded.-    ///-    /// Contains a vector of published message IDs.-    Success(Vec<Id>),-    /// Publisher failed to publish any of the messages.-    OneError(AnyError, Vec<ValidatedMessage>),-    /// Publisher failed to publish some of the messages.-    ///-    /// The error type has a per-message granularity.-    PerMessage(Vec<Result<Id, (AnyError, ValidatedMessage)>>),+    /// Could not validate message data+    #[error("Could not publish the message")]+    PublishOneMessage(#[source] Box<dyn std::error::Error + Send + Sync>), }  /// Interface for message publishers pub trait Publisher {     /// The list of identifiers for successfully published messages     type MessageId: 'static;-    /// The future that the `publish` method returns-    type PublishFuture: Future<Output = PublisherResult<Self::MessageId>> + Send;+    /// The message error that this publisher returns+    type MessageError: std::error::Error + Send + Sync + 'static;+    /// The stream of results that the `publish` method returns+    type PublishStream: Stream<Item = Result<Self::MessageId, Self::MessageError>> + Unpin;      /// Publish a batch of messages     ///-    /// # Return value-    ///-    /// Shall return [`PublisherResult::Success`](PublisherResult::Success) only if all of the-    /// messages are successfully published. Otherwise `PublisherResult::OneError` or-    /// `PublisherResult::PerMessage` shall be returned to indicate an error.-    fn publish(&self, topic: &'static str, messages: Vec<ValidatedMessage>) -> Self::PublishFuture;+    /// The output stream shall return a result for each message in `messages` slice in the same+    /// order.+    fn publish<'a, I>(&self, topic: &'static str, messages: I) -> Self::PublishStream+    where+        I: Iterator<Item = &'a ValidatedMessage> + DoubleEndedIterator + ExactSizeIterator;

Stream cannot exactly support DoubleEndedIterator or ExactSizeIterator. While the only "real" publisher does not need either of those (and it would make some sense for it to accept a Stream) I don’t want to make such a change in this PR. Specifically because I don’t want to spend the time right now thinking about what the consequences of such a change might end up being.

nagisa

comment created time in 13 days

PullRequestReviewEvent

push eventstandard-ai/hedwig-rust

Simonas Kazlauskas

commit sha 485e87deaa5517fcff010e743463566b90b64645

Refactor the library This commit refactors this library, almost from scratch, in order to: 1. Achieve more conventional and easier-to-use APIs; 2. More correct and flexible error handling; 3. Internal segmentation of batches inside of google publisher. The initial motivation was originally 3, but then old way of implicit error handling came up as a problem as well. I attempted to introduce some mechanism to handle errors in a more granular way without adjusting API of the library too much, but it turned out to be not very feasible. The new API is centered around the new `hedwig::Message` trait which the users would implement to associate data types with various information that is necessary for the hewdig library – the topic name, the validators applicable to the data, the encoding mechanisms, etc. Furthermore, it now allows retaining all of auxiliary data alongside the message data as well, giving further flexibility in how this library is used by the users. All in all, the trait was designed in a way that would allow to move the message-encoding specific pieces of code away from the business logic as much as possible. For now the implementation of this trait remains manual, but there is nothing that would prevent it from being `derive`able in some way. --- The error handling was made entirely explicit in the new API. In particular we no longer implicitly re-queue messages in the batch (nor do we reuse the batches, anymore). Instead the users receive a `Stream` of `(result, topic, validated message)`. They can then choose to `push` the validated message back into a new batch they are constructing to retry the message… or just handle the error in whatever other way that makes sense to them. One important improvement as part of this change is simplification of behaviour when the futures/streams are cancelled. Previously dropping the `publish` future would leave one in a fairly difficult to understand state where the batch might have some of the messages re-queued and some of them cancelled, etc. Since there's no longer any state remaining in `hedwig`, dropping streams to cancel starts making significantly more sense as well. --- Google PubSub has various limitations on the API requests that are made to its endpoints. As we are in charge of batching the requests up to implement publishing more efficiently, it is also up to us to ensure that we do not exceed any of the limits upstream has placed on us. To do this we implement a batch segmentation algorithm in `GoogleMessageSegmenter` that the GooglePubSubPublisher uses internally to split up the messages. Fixes #18

view details

push time in 13 days

push eventstandard-ai/hedwig-rust

Simonas Kazlauskas

commit sha 9284641b737a8f5f669cfc1d4782553cd52c48b1

Refactor the library This commit refactors this library, almost from scratch, in order to: 1. Achieve more conventional and easier-to-use APIs; 2. More correct and flexible error handling; 3. Internal segmentation of batches inside of google publisher. The initial motivation was originally 3, but then old way of implicit error handling came up as a problem as well. I attempted to introduce some mechanism to handle errors in a more granular way without adjusting API of the library too much, but it turned out to be not very feasible. The new API is centered around the new `hedwig::Message` trait which the users would implement to associate data types with various information that is necessary for the hewdig library – the topic name, the validators applicable to the data, the encoding mechanisms, etc. Furthermore, it now allows retaining all of auxiliary data alongside the message data as well, giving further flexibility in how this library is used by the users. All in all, the trait was designed in a way that would allow to move the message-encoding specific pieces of code away from the business logic as much as possible. For now the implementation of this trait remains manual, but there is nothing that would prevent it from being `derive`able in some way. --- The error handling was made entirely explicit in the new API. In particular we no longer implicitly re-queue messages in the batch (nor do we reuse the batches, anymore). Instead the users receive a `Stream` of `(result, topic, validated message)`. They can then choose to `push` the validated message back into a new batch they are constructing to retry the message… or just handle the error in whatever other way that makes sense to them. One important improvement as part of this change is simplification of behaviour when the futures/streams are cancelled. Previously dropping the `publish` future would leave one in a fairly difficult to understand state where the batch might have some of the messages re-queued and some of them cancelled, etc. Since there's no longer any state remaining in `hedwig`, dropping streams to cancel starts making significantly more sense as well. --- Google PubSub has various limitations on the API requests that are made to its endpoints. As we are in charge of batching the requests up to implement publishing more efficiently, it is also up to us to ensure that we do not exceed any of the limits upstream has placed on us. To do this we implement a batch segmentation algorithm in `GoogleMessageSegmenter` that the GooglePubSubPublisher uses internally to split up the messages. Fixes #18

view details

push time in 13 days

push eventstandard-ai/hedwig-rust

Simonas Kazlauskas

commit sha 5bf1f5b96336ecd329858b8059f22379011fabc2

Refactor the library This commit refactors this library, almost from scratch, in order to: 1. Achieve more conventional and easier-to-use APIs; 2. More correct and flexible error handling; 3. Internal segmentation of batches inside of google publisher. The initial motivation was originally 3, but then old way of implicit error handling came up as a problem as well. I attempted to introduce some mechanism to handle errors in a more granular way without adjusting API of the library too much, but it turned out to be not very feasible. The new API is centered around the new `hedwig::Message` trait which the users would implement to associate data types with various information that is necessary for the hewdig library – the topic name, the validators applicable to the data, the encoding mechanisms, etc. Furthermore, it now allows retaining all of auxiliary data alongside the message data as well, giving further flexibility in how this library is used by the users. All in all, the trait was designed in a way that would allow to move the message-encoding specific pieces of code away from the business logic as much as possible. For now the implementation of this trait remains manual, but there is nothing that would prevent it from being `derive`able in some way. --- The error handling was made entirely explicit in the new API. In particular we no longer implicitly re-queue messages in the batch (nor do we reuse the batches, anymore). Instead the users receive a `Stream` of `(result, topic, validated message)`. They can then choose to `push` the validated message back into a new batch they are constructing to retry the message… or just handle the error in whatever other way that makes sense to them. One important improvement as part of this change is simplification of behaviour when the futures/streams are cancelled. Previously dropping the `publish` future would leave one in a fairly difficult to understand state where the batch might have some of the messages re-queued and some of them cancelled, etc. Since there's no longer any state remaining in `hedwig`, dropping streams to cancel starts making significantly more sense as well. --- Google PubSub has various limitations on the API requests that are made to its endpoints. As we are in charge of batching the requests up to implement publishing more efficiently, it is also up to us to ensure that we do not exceed any of the limits upstream has placed on us. To do this we implement a batch segmentation algorithm in `GoogleMessageSegmenter` that the GooglePubSubPublisher uses internally to split up the messages. Fixes #18

view details

push time in 13 days

push eventstandard-ai/hedwig-rust

Simonas Kazlauskas

commit sha b41bddae97e761704eaf48df7fcc0a30f03a3e71

Refactor the library This commit refactors this library, almost from scratch, in order to: 1. Achieve more conventional and easier-to-use APIs; 2. More correct and flexible error handling; 3. Internal segmentation of batches inside of google publisher. The initial motivation was originally 3, but then old way of implicit error handling came up as a problem as well. I attempted to introduce some mechanism to handle errors in a more granular way without adjusting API of the library too much, but it turned out to be not very feasible. The new API is centered around the new `hedwig::Message` trait which the users would implement to associate data types with various information that is necessary for the hewdig library – the topic name, the validators applicable to the data, the encoding mechanisms, etc. Furthermore, it now allows retaining all of auxiliary data alongside the message data as well, giving further flexibility in how this library is used by the users. All in all, the trait was designed in a way that would allow to move the message-encoding specific pieces of code away from the business logic as much as possible. For now the implementation of this trait remains manual, but there is nothing that would prevent it from being `derive`able in some way. --- The error handling was made entirely explicit in the new API. In particular we no longer implicitly re-queue messages in the batch (nor do we reuse the batches, anymore). Instead the users receive a `Stream` of `(result, topic, validated message)`. They can then choose to `push` the validated message back into a new batch they are constructing to retry the message… or just handle the error in whatever other way that makes sense to them. One important improvement as part of this change is simplification of behaviour when the futures/streams are cancelled. Previously dropping the `publish` future would leave one in a fairly difficult to understand state where the batch might have some of the messages re-queued and some of them cancelled, etc. Since there's no longer any state remaining in `hedwig`, dropping streams to cancel starts making significantly more sense as well. --- Google PubSub has various limitations on the API requests that are made to its endpoints. As we are in charge of batching the requests up to implement publishing more efficiently, it is also up to us to ensure that we do not exceed any of the limits upstream has placed on us. To do this we implement a batch segmentation algorithm in `GoogleMessageSegmenter` that the GooglePubSubPublisher uses internally to split up the messages. Fixes #18

view details

push time in 13 days

push eventstandard-ai/hedwig-rust

Simonas Kazlauskas

commit sha 0416b1ab63c2e337e65f86be1356a21b11cef426

Refactor the library This commit refactors this library, almost from scratch, in order to: 1. Achieve more conventional and easier-to-use APIs; 2. More correct and flexible error handling; 3. Internal segmentation of batches inside of google publisher. The initial motivation was originally 3, but then old way of implicit error handling came up as a problem as well. I attempted to introduce some mechanism to handle errors in a more granular way without adjusting API of the library too much, but it turned out to be not very feasible. The new API is centered around the new `hedwig::Message` trait which the users would implement to associate data types with various information that is necessary for the hewdig library – the topic name, the validators applicable to the data, the encoding mechanisms, etc. Furthermore, it now allows retaining all of auxiliary data alongside the message data as well, giving further flexibility in how this library is used by the users. All in all, the trait was designed in a way that would allow to move the message-encoding specific pieces of code away from the business logic as much as possible. For now the implementation of this trait remains manual, but there is nothing that would prevent it from being `derive`able in some way. --- The error handling was made entirely explicit in the new API. In particular we no longer implicitly re-queue messages in the batch (nor do we reuse the batches, anymore). Instead the users receive a `Stream` of `(result, topic, validated message)`. They can then choose to `push` the validated message back into a new batch they are constructing to retry the message… or just handle the error in whatever other way that makes sense to them. One important improvement as part of this change is simplification of behaviour when the futures/streams are cancelled. Previously dropping the `publish` future would leave one in a fairly difficult to understand state where the batch might have some of the messages re-queued and some of them cancelled, etc. Since there's no longer any state remaining in `hedwig`, dropping streams to cancel starts making significantly more sense as well. --- Google PubSub has various limitations on the API requests that are made to its endpoints. As we are in charge of batching the requests up to implement publishing more efficiently, it is also up to us to ensure that we do not exceed any of the limits upstream has placed on us. To do this we implement a batch segmentation algorithm in `GoogleMessageSegmenter` that the GooglePubSubPublisher uses internally to split up the messages. Fixes #18

view details

push time in 13 days

push eventstandard-ai/hedwig-rust

Simonas Kazlauskas

commit sha 8fb7ba842397168f279325c6f43b8424b264348b

Refactor the library This commit refactors this library, almost from scratch, in order to: 1. Achieve more conventional and easier-to-use APIs; 2. More correct and flexible error handling; 3. Internal segmentation of batches inside of google publisher. The initial motivation was originally 3, but then old way of implicit error handling came up as a problem as well. I attempted to introduce some mechanism to handle errors in a more granular way without adjusting API of the library too much, but it turned out to be not very feasible. The new API is centered around the new `hedwig::Message` trait which the users would implement to associate data types with various information that is necessary for the hewdig library – the topic name, the validators applicable to the data, the encoding mechanisms, etc. Furthermore, it now allows retaining all of auxiliary data alongside the message data as well, giving further flexibility in how this library is used by the users. All in all, the trait was designed in a way that would allow to move the message-encoding specific pieces of code away from the business logic as much as possible. For now the implementation of this trait remains manual, but there is nothing that would prevent it from being `derive`able in some way. --- The error handling was made entirely explicit in the new API. In particular we no longer implicitly re-queue messages in the batch (nor do we reuse the batches, anymore). Instead the users receive a `Stream` of `(result, topic, validated message)`. They can then choose to `push` the validated message back into a new batch they are constructing to retry the message… or just handle the error in whatever other way that makes sense to them. One important improvement as part of this change is simplification of behaviour when the futures/streams are cancelled. Previously dropping the `publish` future would leave one in a fairly difficult to understand state where the batch might have some of the messages re-queued and some of them cancelled, etc. Since there's no longer any state remaining in `hedwig`, dropping streams to cancel starts making significantly more sense as well. --- Google PubSub has various limitations on the API requests that are made to its endpoints. As we are in charge of batching the requests up to implement publishing more efficiently, it is also up to us to ensure that we do not exceed any of the limits upstream has placed on us. To do this we implement a batch segmentation algorithm in `GoogleMessageSegmenter` that the GooglePubSubPublisher uses internally to split up the messages. Fixes #18

view details

push time in 13 days

pull request commentnagisa/rust_tracy_client

Add support for optionally using externally built Tracy library

Even if the default behaviour of the crate remains the same?

Well, it still exposes a way for people to shoot themselves in a foot, by means that are not immediately apparent as to their severity.

For tracy-client-sys to support the use-case of using arbitrary version of tracy it would need to be rewritten to generate the bindings at build time from the provided headers (via bindgen), which by itself is a not a terrible proposition. At the very least it would make any mismatches explode loudly at compile time. With the way tracy-client-sys bindings are hand-written currently, were we to allow users to provide an arbitrary version of tracy binary code, the signature mismatches would compile correctly, but then clobber registers, memory etc at runtime, leading to difficult to debug crashes etc.

zanshi

comment created time in 14 days

pull request commentnagisa/rust_tracy_client

Add support for optionally using externally built Tracy library

I’m somewhat uncomfortable with this, mostly because tracy is not very good at maintaining a stable ABI and versioning the project accordingly.

zanshi

comment created time in 14 days

push eventnagisa/prost

Simonas Kazlauskas

commit sha 1466355d7ce9c627e8183994e8ecb3972d9957fd

Do not force an unnecessary Sized bound This allows users to call `Message` methods on `&(dyn Message)`.

view details

push time in 14 days

PR opened danburkert/prost

Do not force an unnecessary Sized bound

This allows users to call Message methods on &(dyn Message).

+8 -13

0 comment

2 changed files

pr created time in 14 days

create barnchnagisa/prost

branch : unsized

created branch time in 14 days

fork nagisa/prost

PROST! a Protocol Buffers implementation for the Rust Language

fork in 14 days

Pull request review commentseanmonstar/num_cpus

Add cgroups support for Linux targets

+use std::collections::HashMap;+use std::fs::File;+use std::io::{BufRead, BufReader, Read};+use std::mem;+use std::path::{Path, PathBuf};+use std::sync::atomic::{AtomicUsize, Ordering};+use std::sync::Once;++use libc;++macro_rules! debug {+    ($($args:expr),*) => ({+        if false {+        //if true {+            println!($($args),*);+        }+    });+}++macro_rules! some {+    ($e:expr) => ({+        match $e {+            Some(v) => v,+            None => {+                debug!("NONE: {:?}", stringify!($e));+                return None;+            }+        }+    })+}++pub fn get_num_cpus() -> usize {+    match cgroups_num_cpus() {+        Some(n) => n,+        None => logical_cpus(),+    }+}++fn logical_cpus() -> usize {+    let mut set: libc::cpu_set_t = unsafe { mem::zeroed() };+    if unsafe { libc::sched_getaffinity(0, mem::size_of::<libc::cpu_set_t>(), &mut set) } == 0 {+        let mut count: u32 = 0;+        for i in 0..libc::CPU_SETSIZE as usize {+            if unsafe { libc::CPU_ISSET(i, &set) } {+                count += 1+            }+        }+        count as usize+    } else {+        let cpus = unsafe { libc::sysconf(libc::_SC_NPROCESSORS_ONLN) };+        if cpus < 1 {+            1+        } else {+            cpus as usize+        }+    }+}++pub fn get_num_physical_cpus() -> usize {+    let file = match File::open("/proc/cpuinfo") {+        Ok(val) => val,+        Err(_) => return get_num_cpus(),+    };+    let reader = BufReader::new(file);+    let mut map = HashMap::new();+    let mut physid: u32 = 0;+    let mut cores: usize = 0;+    let mut chgcount = 0;+    for line in reader.lines().filter_map(|result| result.ok()) {+        let mut it = line.split(':');+        let (key, value) = match (it.next(), it.next()) {+            (Some(key), Some(value)) => (key.trim(), value.trim()),+            _ => continue,+        };+        if key == "physical id" {+            match value.parse() {+                Ok(val) => physid = val,+                Err(_) => break,+            };+            chgcount += 1;+        }+        if key == "cpu cores" {+            match value.parse() {+                Ok(val) => cores = val,+                Err(_) => break,+            };+            chgcount += 1;+        }+        if chgcount == 2 {+            map.insert(physid, cores);+            chgcount = 0;+        }+    }+    let count = map.into_iter().fold(0, |acc, (_, cores)| acc + cores);++    if count == 0 {+        get_num_cpus()+    } else {+        count+    }+}++/// Cached CPUs calculated from cgroups.+///+/// If 0, check logical cpus.+// Allow deprecation warnings, we want to work on older rustc+#[allow(warnings)]+static CGROUPS_CPUS: AtomicUsize = ::std::sync::atomic::ATOMIC_USIZE_INIT;++fn cgroups_num_cpus() -> Option<usize> {+    #[allow(warnings)]+    static ONCE: Once = ::std::sync::ONCE_INIT;++    ONCE.call_once(init_cgroups);++    let cpus = CGROUPS_CPUS.load(Ordering::Acquire);++    if cpus > 0 {+        Some(cpus)+    } else {+        None+    }+}++fn init_cgroups() {+    // Should only be called once

What happens if the cgroup limits are changed while a program is running?

seanmonstar

comment created time in 15 days

PullRequestReviewEvent

issue commentrust-lang/compiler-builtins

__rust_u128_mulo aborts when rhs is zero

Is it doing a widening multiplication, and checking if the high part is not zero?

No, none of the multiplications in this are “widening” in a sense that there are no 256-bit operations involved for a 128-bit UMULO (but there are some simpler 128-bit operations). Here's the algorithm above with types substituted for u128, maybe it'll be clearer:

%0 = %LHS.HI != 0 && %RHS.HI != 0
%1 = { i64, i1 } @umul.with.overflow.i64(i64 %LHS.HI, i64 %RHS.LO)
%2 = { i64, i1 } @umul.with.overflow.i64(i64 %RHS.HI, i64 %LHS.LO)
%3 = mul nuw i128 (%LHS.LOW as i128), (%RHS.LOW as i128)          ; note: regular wrapping multiplication
%4 = add i128 (%1.0 as i128) << 64, (%2.0 as i128) << 64
%5 = { i128, i1 } @uadd.with.overflow.i128(i128 %4, i128 %3)
%res = { %5.0, %0 || %1.1 || %2.1 || %5.1 }

Do note that this is a unsigned version, I couldn't figure out one for signed integers.

bjorn3

comment created time in 15 days

issue commentrust-lang/compiler-builtins

__rust_u128_mulo aborts when rhs is zero

In particular:

%0 = %LHS.HI != 0 && %RHS.HI != 0
%1 = { iNh, i1 } @umul.with.overflow.iNh(iNh %LHS.HI, iNh %RHS.LO)
%2 = { iNh, i1 } @umul.with.overflow.iNh(iNh %RHS.HI, iNh %LHS.LO)
%3 = mul nuw iN (%LHS.LOW as iN), (%RHS.LOW as iN)
%4 = add iN (%1.0 as iN) << Nh, (%2.0 as iN) << Nh
%5 = { iN, i1 } @uadd.with.overflow.iN( %4, %3 )
%res = { %5.0, %0 || %1.1 || %2.1 || %5.1 }
bjorn3

comment created time in 15 days

issue commentrust-lang/compiler-builtins

__rust_u128_mulo aborts when rhs is zero

I suggest you to copy the algorithm LLVM uses to lower the operation as I implemented in https://reviews.llvm.org/rG73e8a784e62f945a51363c8b5ec4eaedcf9f87e8.

bjorn3

comment created time in 15 days

issue commentrust-lang/rust

Fast algorithm for u128 (and i128) divided by small constant

The fact that LLVM cannot reduce the division to multiplications by a constant might be caused

I investigated this recently for https://github.com/rust-lang/rust/pull/76017#discussion_r479699174.

It is a bug in LLVM and it isn’t like it is impossible for it to strength-reduce, its just that doing so:

a) requires the upper-half of the multiplication result (i.e. for 128-bit multiplication it requires the upper 128-bits of a 256-bit result); and b) calculating that cannot be easily made less conservative because there are a couple of bad backends in LLVM – namely RISCV and 32-bit ARM.

I think I know an easy way to resolve it.


u64 numbers get divided by a small constant divisor using a fast algorithm

Turns out it isn’t on 32-bit targets!

leonardo-m

comment created time in 15 days

issue commentrust-lang/rust

CJK letters in rustdoc are not rendered in serif font when using Firefox

I can confirm this. Seems like a browser bug. Even if just serif is specified, the browser selects a sans-serif font (in my case Source Han Sans) even though I have a serif font available too (Source Han Serif).

osanshouo

comment created time in 15 days

Pull request review commentrust-lang/rust

unix: Extend UnixStream and UnixDatagram to send and receive file descriptors

+use crate::convert::TryFrom;+use crate::io::{self, IoSliceMut};+use crate::marker::PhantomData;+use crate::mem::{size_of, zeroed};+use crate::os::unix::io::RawFd;+use crate::path::Path;+use crate::ptr::{null_mut, read_unaligned};+use crate::slice::from_raw_parts;+use crate::sys::unix::ext::net::addr::{sockaddr_un, SocketAddr};+use crate::sys::unix::net::Socket;++pub(super) fn recv_vectored_with_ancillary_from(+    socket: &Socket,+    bufs: &mut [IoSliceMut<'_>],+    ancillary: &mut SocketAncillary<'_>,+) -> io::Result<(usize, bool, io::Result<SocketAddr>)> {+    unsafe {+        let mut msg_name: libc::sockaddr_un = zeroed();++        let mut msg = libc::msghdr {+            msg_name: &mut msg_name as *mut _ as *mut _,+            msg_namelen: size_of::<libc::sockaddr_un>() as libc::socklen_t,+            msg_iov: bufs.as_mut_ptr().cast(),+            msg_iovlen: bufs.len(),+            msg_control: ancillary.buffer.as_mut_ptr().cast(),+            msg_controllen: ancillary.buffer.len(),+            msg_flags: 0,+        };++        let count = socket.recv_msg(&mut msg)?;++        ancillary.length = msg.msg_controllen;+        ancillary.truncated = msg.msg_flags & libc::MSG_CTRUNC == libc::MSG_CTRUNC;++        let truncated = msg.msg_flags & libc::MSG_TRUNC == libc::MSG_TRUNC;+        let addr = SocketAddr::from_parts(msg_name, msg.msg_namelen);++        Ok((count, truncated, addr))+    }+}++pub(super) fn send_vectored_with_ancillary_to(+    socket: &Socket,+    path: Option<&Path>,+    bufs: &mut [IoSliceMut<'_>],+    ancillary: &mut SocketAncillary<'_>,+) -> io::Result<usize> {+    unsafe {+        let (mut msg_name, msg_namelen) =+            if let Some(path) = path { sockaddr_un(path)? } else { (zeroed(), 0) };++        let mut msg = libc::msghdr {+            msg_name: &mut msg_name as *mut _ as *mut _,+            msg_namelen,+            msg_iov: bufs.as_mut_ptr().cast(),+            msg_iovlen: bufs.len(),+            msg_control: ancillary.buffer.as_mut_ptr().cast(),+            msg_controllen: ancillary.length,+            msg_flags: 0,+        };++        ancillary.truncated = false;++        socket.send_msg(&mut msg)+    }+}++fn add_to_ancillary_data<T>(+    buffer: &mut [u8],+    length: &mut usize,+    source: &[T],+    cmsg_level: libc::c_int,+    cmsg_type: libc::c_int,+) -> bool {+    let source_len = if let Some(source_len) = source.len().checked_mul(size_of::<T>()) {+        if let Ok(source_len) = u32::try_from(source_len) {+            source_len+        } else {+            return false;+        }+    } else {+        return false;+    };++    unsafe {+        let additional_space = libc::CMSG_SPACE(source_len) as usize;++        let new_length = if let Some(new_length) = additional_space.checked_add(*length) {+            new_length+        } else {+            return false;+        };++        if new_length > buffer.len() {+            return false;+        }++        buffer[*length..new_length].fill(0);++        *length = new_length;++        let msg = libc::msghdr {+            msg_name: null_mut(),+            msg_namelen: 0,+            msg_iov: null_mut(),+            msg_iovlen: 0,+            msg_control: buffer.as_mut_ptr().cast(),+            msg_controllen: *length,+            msg_flags: 0,+        };++        let mut cmsg = libc::CMSG_FIRSTHDR(&msg);+        let mut previous_cmsg = cmsg;+        while !cmsg.is_null() {+            previous_cmsg = cmsg;+            cmsg = libc::CMSG_NXTHDR(&msg, cmsg);+        }++        if previous_cmsg.is_null() {+            return false;+        }++        (*previous_cmsg).cmsg_level = cmsg_level;+        (*previous_cmsg).cmsg_type = cmsg_type;+        (*previous_cmsg).cmsg_len = libc::CMSG_LEN(source_len) as usize;++        let data = libc::CMSG_DATA(previous_cmsg).cast();++        libc::memcpy(data, source.as_ptr().cast(), source_len as usize);

You’re already casting the pointer away to *mut u8 (which is always aligned) for memcpy. You can do exactly the same for ptr::copy{,_nonoverlapping}.

LinkTed

comment created time in 17 days

PullRequestReviewEvent

pull request commentrust-lang/rust

Syntactically permit unsafety on mods

@bors r+

dtolnay

comment created time in 18 days

issue commentrust-lang/rust

signal: 11, SIGSEGV: invalid memory reference

Have you overclocked your memory (by e.g. applying XMP) ? I've seen similarly bizarre and spurious issues come up with an unstable memory overclock.

AurevoirXavier

comment created time in 18 days

more