[Not really maintained anymore] BFM is an Apache Licenced server file manager for Django made with ease of use and new web technologies in mind.
A pure-Rust library to work with Linux memfd
More efficient way to preload e4rat file lists.
Feed Reader for GNOME.
nagisa/gnome-shell-theme-min 3
A GNOME-Shell theme without bells or whistles
Rusty protobufs for Tensorflow
LLVM build utils for cargo build scripts
nagisa/django-localflavor-lt 1
Country-specific Django helpers for Lithuania.
Hakyll based website of mine
Pull request review commentrust-lang/rust
Support custom allocators in `Box`
impl<T: ?Sized> Box<T> { /// ``` #[stable(feature = "box_leak", since = "1.26.0")] #[inline]- pub fn leak<'a>(b: Box<T>) -> &'a mut T+ pub fn leak<'a>(b: Self) -> &'a mut T where- T: 'a, // Technically not needed, but kept to be explicit.+ A: 'a, { unsafe { &mut *mem::ManuallyDrop::new(b).0.as_ptr() } } + /// Consumes and leaks the `Box`, returning a mutable reference,+ /// `&'a mut T`, and the allocator. Note that the type `T` must outlive the chosen lifetime+ /// `'a`. If the type has only static references, or none at all, then this+ /// may be chosen to be `'static`.+ ///+ /// This function is mainly useful for data that lives for the remainder of+ /// the program's life. Dropping the returned reference will cause a memory+ /// leak. If this is not acceptable, the reference should first be wrapped+ /// with the [`Box::from_raw`] function producing a `Box`. This `Box` can+ /// then be dropped which will properly destroy `T` and release the+ /// allocated memory.+ ///+ /// Note: this is an associated function, which means that you have+ /// to call it as `Box::leak(b)` instead of `b.leak()`. This+ /// is so that there is no conflict with a method on the inner type.+ #[unstable(feature = "allocator_api", issue = "32838")]+ #[inline]+ pub fn leak_with_alloc<'a>(b: Self) -> (&'a mut T, A)
This API seems sketchy to me. Without thinking too much about it I think one could do something along the lines of:
let (ref, allocref) = Box::leak_with_alloc(a_box);
drop(allocref); // potentially drops the backing storage
use(ref); // use of a dangling reference
comment created time in 14 hours
Pull request review commentrust-lang/rust
Support custom allocators in `Box`
impl<T: ?Sized> Box<T> { /// ``` #[stable(feature = "box_leak", since = "1.26.0")] #[inline]- pub fn leak<'a>(b: Box<T>) -> &'a mut T+ pub fn leak<'a>(b: Self) -> &'a mut T where- T: 'a, // Technically not needed, but kept to be explicit.+ A: 'a, { unsafe { &mut *mem::ManuallyDrop::new(b).0.as_ptr() } } + /// Consumes and leaks the `Box`, returning a mutable reference,+ /// `&'a mut T`, and the allocator. Note that the type `T` must outlive the chosen lifetime+ /// `'a`. If the type has only static references, or none at all, then this+ /// may be chosen to be `'static`.+ ///+ /// This function is mainly useful for data that lives for the remainder of+ /// the program's life. Dropping the returned reference will cause a memory+ /// leak. If this is not acceptable, the reference should first be wrapped+ /// with the [`Box::from_raw`] function producing a `Box`. This `Box` can+ /// then be dropped which will properly destroy `T` and release the+ /// allocated memory.+ ///+ /// Note: this is an associated function, which means that you have+ /// to call it as `Box::leak(b)` instead of `b.leak()`. This+ /// is so that there is no conflict with a method on the inner type.+ #[unstable(feature = "allocator_api", issue = "32838")]+ #[inline]+ pub fn leak_with_alloc<'a>(b: Self) -> (&'a mut T, A)+ where+ A: 'a,+ {+ let manually_drop = mem::ManuallyDrop::new(b);+ let alloc = unsafe { ptr::read(&manually_drop.1) };
You probably want MaybeUninit as you're effectively un-initializing part of the data contained within ManuallyDrop which is somewhat sketchy as the value is later used again to obtain reference.
comment created time in 14 hours
pull request commentrust-lang/rust
Add missing definitions required by the sparc-unknown-linux-gnu target
@bors r+
comment created time in 15 hours
pull request commentrust-lang/rust
Use `str::to_owned` when `format!` has no formatting arguments
@bors r+
comment created time in a day
issue closedrust-fuzz/cargo-fuzz
Fuzz target exited with signal: 11
I'm running the fuzzer on part of my codebase. And it sometimes gives the following error:
#4993: cov: 16031 ft: 79453 corp: 4047 exec/s 23 oom/timeout/crash: 0/0/0 time: 401s job: 20 dft_time: 0
#5491: cov: 16031 ft: 79456 corp: 4050 exec/s 22 oom/timeout/crash: 0/0/0 time: 407s job: 21 dft_time: 0
────────────────────────────────────────────────────────────────────────────────
Error: Fuzz target exited with signal: 11
But the problem is there is no debug info, anywhere, no artifacts, no dump, no backtrace, ... Even when I run it in debug mode it does not give me any more info.
It even got a Segmentation fault but no info and just continued (output looks weird, but this is how it was printed in the terminal):
#429 NEW cov: 10201 ft: 33953 corp: 208/179Kb lim: 2091 exec/s: 22 rss: 159Mb L: 2030/2091 MS: 2 ChangeByte-InsertRepeatedBytes-
#431 NEW cov: 10201 ft: 33955 corp: 209/181Kb lim: 2091 exec/s: 22 rss: 159Mb L: 2091/2091 MS: 2 CMP-CrossOver- DE: "dest_entity_id"-
#432 REDUCE cov: 10201 ft: 33955 corp: 209/181Kb lim: 2091 exec/s: 22 rss: 159Mb L: 1911/2091 MS: 1 EraseBytes-
Segmentation fault (core dumped)
#104446 REDUCE cov: 6554 ft: 31504 corp: 5703/2771Kb lim: 2790 exec/s: 629 rss: 504Mb L: 341/2757 MS: 2 ShuffleBytes-EraseBytes-
INFO: exiting: 139 time: 843s
#104735 REDUCE cov: 6554 ft: 31504 corp: 5703/2771Kb lim: 2790 exec/s: 627 rss: 504Mb L: 523/2757 MS: 2 ChangeByte-EraseBytes-
#105054 NEW cov: 6558 ft: 31508 corp: 5704/2772Kb lim: 2790 exec/s: 629 rss: 504Mb L: 584/2757 MS: 4 ChangeBinInt-ShuffleBytes-ChangeBit-PersAutoDict- DE: "new_leader_hfid"-
#106017 REDUCE cov: 6558 ft: 31508 corp: 5704/2772Kb lim: 2790 exec/s: 631 rss: 504Mb L: 397/2757 MS: 2 InsertRepeatedBytes-EraseBytes-
Is there something misconfigured or what can I do to actually find out where things go wrong? Otherwise there is not much need for a fuzzer if I can not find the problems... This could be false positives, as afl does not seem to find any crashes (yet).
closed time in a day
ralpha
issue commentrust-fuzz/cargo-fuzz
Fuzz target exited with signal: 11
effective duplicate of #235
comment created time in a day
Pull request review commentrust-lang/rust
options! {DebuggingOptions, DebuggingSetter, basic_debugging_options, "hash algorithm of source files in debug info (`md5`, or `sha1`)"), strip: Strip = (Strip::None, parse_strip, [UNTRACKED], "tell the linker which information to strip (`none` (default), `debuginfo` or `symbols`)"),+ split_dwarf: SplitDwarfKind = (SplitDwarfKind::None, parse_split_dwarf_kind, [UNTRACKED],+ "enable generation of split dwarf"),+ split_dwarf_inlining: bool = (true, parse_bool, [UNTRACKED],
My reading of the LLVM code and the description of this flag itself suggests that this flag controls whether minimal debug info should also be generated within CUs.
comment created time in 2 days
Pull request review commentrust-lang/rust
options! {DebuggingOptions, DebuggingSetter, basic_debugging_options, "hash algorithm of source files in debug info (`md5`, or `sha1`)"), strip: Strip = (Strip::None, parse_strip, [UNTRACKED], "tell the linker which information to strip (`none` (default), `debuginfo` or `symbols`)"),+ split_dwarf: SplitDwarfKind = (SplitDwarfKind::None, parse_split_dwarf_kind, [UNTRACKED],+ "enable generation of split dwarf"),+ split_dwarf_inlining: bool = (true, parse_bool, [UNTRACKED],
bikeshed colour nit: verb this into inline_split_dwarf
comment created time in 2 days
Pull request review commentrust-lang/rust
impl ModuleConfig { } } -// HACK(eddyb) work around `#[derive]` producing wrong bounds for `Clone`.-pub struct TargetMachineFactory<B: WriteBackendMethods>(- pub Arc<dyn Fn() -> Result<B::TargetMachine, String> + Send + Sync>,-);--impl<B: WriteBackendMethods> Clone for TargetMachineFactory<B> {- fn clone(&self) -> Self {- TargetMachineFactory(self.0.clone())- }+/// Configuration passed to the function returned by the `target_machine_factory`.+pub struct TargetMachineFactoryConfig<'a> {+ /// Split DWARF is enabled in LLVM by checking that `TM.MCOptions.SplitDwarfFile` isn't empty,
Hmm… this probably should not be LLVM-specific, but I don’t yet know how the alternative backends could handle this being set (error out? transparently ignore?)
comment created time in 2 days
pull request commentrust-lang/rust
Use less divisions in display u128/i128
Still somewhat uncomfortable with us encoding a division algorithm manually, but I guess we can remove it later once LLVM does the right thing.
@bors r+
comment created time in 2 days
pull request commentrust-lang/rust
This overall LGTM. There are remaining compilation issues remaining before this can land and the tune-cpu documentation still needs a note that it is unstable. Please also squash.
comment created time in 2 days
pull request commentrust-lang/rust
Return values up to 128 bits in registers
Can you add a codegen test for this?
r=me after that.
comment created time in 2 days
pull request commentrust-lang/rust
Fix FIXME in core::num test: Check sign of zero in min/max tests.
@bors r+ rollup
comment created time in 2 days
issue closednagisa/rust_libloading
support opening existing library
Hello,
Currently you can do Library::new() which loads a library.
For my use case it would be create to have Library::open that opens an already existing library.
- I'm trying to query optionally exported functions in preloaded shared library.
Implementation can be accomplished with the RTLD_NOLOAD flag for dlopen on most unixes and GetModuleHandleEx in windows.
I can write the code myself and submit a PR if that works for you.
closed time in 3 days
yuval-k
issue commentrust-lang/rust
Use dynamic linking and download LLVM from CI for rustc across platforms
LLVM’s bugzilla is pretty much the only place for bug reports. If its something less defined than a precise bug report, talking it out on the mailing lists is the way to go.
comment created time in 6 days
created tagstandard-ai/hedwig-rust
Message bus for micro-services that works on AWS and GCP
created time in 6 days
push eventstandard-ai/hedwig-rust
commit sha 04185f055a7d64d2ed6c095565134f515f877329
Update the readme
push time in 6 days
push eventstandard-ai/hedwig-rust
commit sha 04185f055a7d64d2ed6c095565134f515f877329
Update the readme
push time in 6 days
created tagstandard-ai/hedwig-rust
Message bus for micro-services that works on AWS and GCP
created time in 6 days
created tagstandard-ai/hedwig-rust
Message bus for micro-services that works on AWS and GCP
created time in 6 days
push eventstandard-ai/hedwig-rust
commit sha ec12fe96153a5e8dccfbdcf72005ba00e6d4a9c1
Bump version to 2.0.0
push time in 6 days
created tagstandard-ai/sendfd
Send file descriptors over unix sockets in Rust
created time in 6 days
PR merged standard-ai/sendfd
Got:
cargo build --target=x86_64-unknown-linux-musl
error[E0063]: missing field __pad1 in initializer of libc::cmsghdr
--> /home/tim/project/virtcontainers/sendfd/src/lib.rs:92:13
|
92 | libc::cmsghdr {
| ^^^^^^^^^^^^^ missing __pad1
error[E0451]: field __pad1 of struct libc::msghdr is private
--> /home/tim/project/virtcontainers/sendfd/src/lib.rs:78:15
|
78 | ..mem::zeroed()
| ^^^^^^^^^^^^^ field __pad1 is private
error[E0451]: field __pad2 of struct libc::msghdr is private
--> /home/tim/project/virtcontainers/sendfd/src/lib.rs:78:15
|
78 | ..mem::zeroed()
| ^^^^^^^^^^^^^ field __pad2 is private
This commit will resolve the problem.
Signed-off-by: Tim Zhang [email protected]
pr closed time in 6 days
push eventstandard-ai/sendfd
commit sha d24e55871c46137fbf695df5754ff9813ce1b868
Adapt to musl Got: cargo build --target=x86_64-unknown-linux-musl error[E0063]: missing field `__pad1` in initializer of `libc::cmsghdr` --> /home/tim/project/virtcontainers/sendfd/src/lib.rs:92:13 | 92 | libc::cmsghdr { | ^^^^^^^^^^^^^ missing `__pad1` error[E0451]: field `__pad1` of struct `libc::msghdr` is private --> /home/tim/project/virtcontainers/sendfd/src/lib.rs:78:15 | 78 | ..mem::zeroed() | ^^^^^^^^^^^^^ field `__pad1` is private error[E0451]: field `__pad2` of struct `libc::msghdr` is private --> /home/tim/project/virtcontainers/sendfd/src/lib.rs:78:15 | 78 | ..mem::zeroed() | ^^^^^^^^^^^^^ field `__pad2` is private This commit will resolve the problem. Signed-off-by: Tim Zhang <[email protected]>
commit sha 7eb69eb4dd029c4639edd58e76f81af8f59f58c6
Bump to 0.3.2
push time in 6 days
pull request commentrust-lang/rfcs
RFC: Promote aarch64-unknown-linux-gnu to a Tier-1 Rust target
@rfcbot concern should stack probes be implemented before this target is promoted
The compiler team briefly discussed this in their meeting, and we're ready to move into FCP! If someone on the compiler team feels like blocking the merge on addressing the stack probes question please add a concern!
The relevant discussion is here.
I am adding the concern as I believe we should strive to not regress the soundness properties across our T1 targets now that this particular hole has been patched up, regardless of however long it was open in the past before it got fixed. Or at very least we should make a conscious decision and figure out our policy wrt this question.
comment created time in 6 days
Pull request review commentstandard-ai/sendfd
fn send_with_fd(socket: RawFd, bs: &[u8], fds: &[RawFd]) -> io::Result<usize> { ptr::write( cmsg_header, libc::cmsghdr {+ #[cfg(target_env = "musl")]
Would it perhaps make more sense to construct cmsghdr by assigning fields to a zeroed value the same way, it is done for msghdr above?
comment created time in 6 days
PR merged standard-ai/hedwig-rust
This commit refactors this library, almost from scratch, in order to:
- Achieve more conventional and easier-to-use APIs;
- More correct and flexible error handling;
- Internal segmentation of batches inside of google publisher.
The initial motivation was originally 3, but then old way of implicit error handling came up as a problem as well. I attempted to introduce some mechanism to handle errors in a more granular way without adjusting API of the library too much, but it turned out to be not very feasible.
The new API is centered around the new hedwig::Message trait which the
users would implement to associate data types with various information
that is necessary for the hewdig library – the topic name, the
validators applicable to the data, the encoding mechanisms, etc.
Furthermore, it now allows retaining all of auxiliary data alongside the
message data as well, giving further flexibility in how this library is
used by the users.
All in all, the trait was designed in a way that would allow to move the
message-encoding specific pieces of code away from the business logic as
much as possible. For now the implementation of this trait remains
manual, but there is nothing that would prevent it from being
deriveable in some way.
The error handling was made entirely explicit in the new API. In
particular we no longer implicitly re-queue messages in the batch (nor
do we reuse the batches, anymore). Instead the users receive a Stream
of (result, topic, validated message). They can then choose to push
the validated message back into a new batch they are constructing to
retry the message… or just handle the error in whatever other way that
makes sense to them.
One important improvement as part of this change is simplification of
behaviour when the futures/streams are cancelled. Previously dropping
the publish future would leave one in a fairly difficult to understand
state where the batch might have some of the messages re-queued and some
of them cancelled, etc.
Since there's no longer any state remaining in hedwig, dropping
streams to cancel starts making significantly more sense as well.
Google PubSub has various limitations on the API requests that are made to its endpoints. As we are in charge of batching the requests up to implement publishing more efficiently, it is also up to us to ensure that we do not exceed any of the limits upstream has placed on us.
To do this we implement a batch segmentation algorithm in
GoogleMessageSegmenter that the GooglePubSubPublisher uses internally
to split up the messages.
Fixes #18
pr closed time in 6 days
push eventstandard-ai/hedwig-rust
commit sha 5c3afc9dd64750137792993ad9226bfe13f72c2a
Refactor the library This commit refactors this library, almost from scratch, in order to: 1. Achieve more conventional and easier-to-use APIs; 2. More correct and flexible error handling; 3. Internal segmentation of batches inside of google publisher. The initial motivation was originally 3, but then old way of implicit error handling came up as a problem as well. I attempted to introduce some mechanism to handle errors in a more granular way without adjusting API of the library too much, but it turned out to be not very feasible. The new API is centered around the new `hedwig::Message` trait which the users would implement to associate data types with various information that is necessary for the hewdig library – the topic name, the validators applicable to the data, the encoding mechanisms, etc. Furthermore, it now allows retaining all of auxiliary data alongside the message data as well, giving further flexibility in how this library is used by the users. All in all, the trait was designed in a way that would allow to move the message-encoding specific pieces of code away from the business logic as much as possible. For now the implementation of this trait remains manual, but there is nothing that would prevent it from being `derive`able in some way. --- The error handling was made entirely explicit in the new API. In particular we no longer implicitly re-queue messages in the batch (nor do we reuse the batches, anymore). Instead the users receive a `Stream` of `(result, topic, validated message)`. They can then choose to `push` the validated message back into a new batch they are constructing to retry the message… or just handle the error in whatever other way that makes sense to them. One important improvement as part of this change is simplification of behaviour when the futures/streams are cancelled. Previously dropping the `publish` future would leave one in a fairly difficult to understand state where the batch might have some of the messages re-queued and some of them cancelled, etc. Since there's no longer any state remaining in `hedwig`, dropping streams to cancel starts making significantly more sense as well. --- Google PubSub has various limitations on the API requests that are made to its endpoints. As we are in charge of batching the requests up to implement publishing more efficiently, it is also up to us to ensure that we do not exceed any of the limits upstream has placed on us. To do this we implement a batch segmentation algorithm in `GoogleMessageSegmenter` that the GooglePubSubPublisher uses internally to split up the messages. Fixes #18
push time in 6 days
issue closedstandard-ai/hedwig-rust
Segment requests to google pubsub when publishing larger publish batches
There appears to be some sort of an undocumented request body size limit when making a :publish request.
Error message suggests
Request payload size exceeds the limit: 10485760 bytes.
which is ~10MiB.
We should ensure that when we format a request we don’t exceed this size.
closed time in 6 days
nagisa
issue commentrust-lang/rust
Fast algorithm for u128 (and i128) divided by small constant
I’ve drafted a LLVM differential to fix this https://reviews.llvm.org/D87976.
comment created time in 9 days
push eventnagisa/rust
commit sha fa6f4e542baec87117367fd2f496384caff175ab
error to core triggers coherence right now :(
push time in 9 days
issue commentrust-lang/project-error-handling
I attempted to prototype the move to libcore with backtrace removed in this branch.
The mentioned downcast is indeed not at all an issue (I applied the suggestion I made in this comment), but
impl<'a> From<&str> for Box<dyn Error + Send + Sync + 'a> {
comes up as an issue. In particular it will trigger coherence's wrath with
error[E0119]: conflicting implementations of trait `core::convert::From<&str>` for type `boxed::Box<dyn core::error::Error + core::marker::Send + core::marker::Sync>`:
--> library/alloc/src/errors.rs:66:1
|
66 | impl<'a, E: Error + Send + Sync + 'a> From<E> for Box<dyn Error + Send + Sync + 'a> {
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ conflicting implementation for `boxed::Box<dyn core::error::Error + core::marker::Send + core::marker::Sync>`
|
::: library/alloc/src/boxed.rs:1155:1
|
1155 | impl<'a> From<&str> for Box<dyn Error + Send + Sync + 'a> {
| --------------------------------------------------------- first implementation here
|
= note: upstream crates may add a new impl of trait `core::error::Error` for type `&str` in future versions
comment created time in 9 days
issue commentrust-lang/rust
[AVR] Failure to compile libcore on nightly
Deciphering the failing expression it looks like we're trying to assert that the alignment of an AtomicPtr is equivalent to that of *mut ().
As per the data-layout of the linked target file (p:16:8) the pointer alignment is indeed set to 1 byte and we explicitly set the alignment of the AtomicPtr to 2 here introduced in this commit many moons ago, hence the bug.
The assertion itself was added in this commit and makes sense for what the function does cc @m-ou-se.
I would check if you can adjust the data layout to specify pointers are aligned to 2 bytes (you probably cannot).
I suspect the correct fix is probably to figure out how to set repr(align(ptr)) for AtomicPtr.
comment created time in 10 days
pull request commentrust-lang/stacker
Convert callback to `dyn` to improve compile times
Released stacker 0.1.12 with this change.
comment created time in 10 days
PR merged rust-lang/stacker
To avoid monomorphizing psm::on_stack::with_on_stack 1500 times in rustc, this PR converts the callback passed to grow _grow to dyn FnMut(). The dynamic dispatch happens on the slow path.
This should improve rustc's compile time a bit, and surprisingly also improves it's runtime: perf-results measured in rust-lang/rust#76680
This was discovered with the cargo-llvm-lines tool.
pr closed time in 10 days
push eventrust-lang/stacker
commit sha 08dbc14c5b5594cbd8792665dfbeee9ac73f7269
To avoid monomorphizing `psm::on_stack::with_on_stack` 1500 times in rustc, convert the callback passed to grow `_grow` to `dyn FnMut()`. This should improve rustc's compile time a bit, and surprisingly also improves runtime in some cases. https://perf.rust-lang.org/compare.html?start=7402a394471a6738a40fea7d4f1891666e5a80c5&end=f242334c1e2e719cf1cba923923ad8ec62affb71
commit sha 6d3e250a5628d5df9ad9acb0e3da371c0acd3de9
stacker -> 0.1.12
push time in 10 days
Pull request review commentrust-lang/rust
This also supports the feature `+crt-static` and `-crt-static` to control Each target and [`target-cpu`](#target-cpu) has a default set of enabled features. +## tune-cpu
Moving this option to -Z made it unstable and not-actually-a-codegen-option. At the very least this should be annotated as unstable.
comment created time in 10 days
Pull request review commentrust-lang/rust
pub fn apply_target_cpu_attr(cx: &CodegenCx<'ll, '_>, llfn: &'ll Value) { ); } +pub fn apply_tune_cpu_attr(cx: &CodegenCx<'ll, '_>, llfn: &'ll Value) {
This applies the attribute regardless of whether the option is specified or not. Is there a good reason why we're applying it even if the -Ztune-cpu is not specified?
comment created time in 10 days
issue commentrust-lang/compiler-team
Add StatementKind::Intrinsic to MIR
@rustbot third
I’m willing to review changes that implement this too, so do r? me on implementation PRs as well.
comment created time in 10 days
startedcpcloud/tensorflow_proto
started time in 10 days
push eventstandard-ai/hedwig-rust
commit sha 5c3afc9dd64750137792993ad9226bfe13f72c2a
Refactor the library This commit refactors this library, almost from scratch, in order to: 1. Achieve more conventional and easier-to-use APIs; 2. More correct and flexible error handling; 3. Internal segmentation of batches inside of google publisher. The initial motivation was originally 3, but then old way of implicit error handling came up as a problem as well. I attempted to introduce some mechanism to handle errors in a more granular way without adjusting API of the library too much, but it turned out to be not very feasible. The new API is centered around the new `hedwig::Message` trait which the users would implement to associate data types with various information that is necessary for the hewdig library – the topic name, the validators applicable to the data, the encoding mechanisms, etc. Furthermore, it now allows retaining all of auxiliary data alongside the message data as well, giving further flexibility in how this library is used by the users. All in all, the trait was designed in a way that would allow to move the message-encoding specific pieces of code away from the business logic as much as possible. For now the implementation of this trait remains manual, but there is nothing that would prevent it from being `derive`able in some way. --- The error handling was made entirely explicit in the new API. In particular we no longer implicitly re-queue messages in the batch (nor do we reuse the batches, anymore). Instead the users receive a `Stream` of `(result, topic, validated message)`. They can then choose to `push` the validated message back into a new batch they are constructing to retry the message… or just handle the error in whatever other way that makes sense to them. One important improvement as part of this change is simplification of behaviour when the futures/streams are cancelled. Previously dropping the `publish` future would leave one in a fairly difficult to understand state where the batch might have some of the messages re-queued and some of them cancelled, etc. Since there's no longer any state remaining in `hedwig`, dropping streams to cancel starts making significantly more sense as well. --- Google PubSub has various limitations on the API requests that are made to its endpoints. As we are in charge of batching the requests up to implement publishing more efficiently, it is also up to us to ensure that we do not exceed any of the limits upstream has placed on us. To do this we implement a batch segmentation algorithm in `GoogleMessageSegmenter` that the GooglePubSubPublisher uses internally to split up the messages. Fixes #18
push time in 11 days
push eventstandard-ai/hedwig-rust
commit sha d6aa5b29368d84cf78a2d6d8e784cd8111927def
Refactor the library This commit refactors this library, almost from scratch, in order to: 1. Achieve more conventional and easier-to-use APIs; 2. More correct and flexible error handling; 3. Internal segmentation of batches inside of google publisher. The initial motivation was originally 3, but then old way of implicit error handling came up as a problem as well. I attempted to introduce some mechanism to handle errors in a more granular way without adjusting API of the library too much, but it turned out to be not very feasible. The new API is centered around the new `hedwig::Message` trait which the users would implement to associate data types with various information that is necessary for the hewdig library – the topic name, the validators applicable to the data, the encoding mechanisms, etc. Furthermore, it now allows retaining all of auxiliary data alongside the message data as well, giving further flexibility in how this library is used by the users. All in all, the trait was designed in a way that would allow to move the message-encoding specific pieces of code away from the business logic as much as possible. For now the implementation of this trait remains manual, but there is nothing that would prevent it from being `derive`able in some way. --- The error handling was made entirely explicit in the new API. In particular we no longer implicitly re-queue messages in the batch (nor do we reuse the batches, anymore). Instead the users receive a `Stream` of `(result, topic, validated message)`. They can then choose to `push` the validated message back into a new batch they are constructing to retry the message… or just handle the error in whatever other way that makes sense to them. One important improvement as part of this change is simplification of behaviour when the futures/streams are cancelled. Previously dropping the `publish` future would leave one in a fairly difficult to understand state where the batch might have some of the messages re-queued and some of them cancelled, etc. Since there's no longer any state remaining in `hedwig`, dropping streams to cancel starts making significantly more sense as well. --- Google PubSub has various limitations on the API requests that are made to its endpoints. As we are in charge of batching the requests up to implement publishing more efficiently, it is also up to us to ensure that we do not exceed any of the limits upstream has placed on us. To do this we implement a batch segmentation algorithm in `GoogleMessageSegmenter` that the GooglePubSubPublisher uses internally to split up the messages. Fixes #18
push time in 11 days
push eventstandard-ai/hedwig-rust
commit sha eb1f69dc8c0db4fe5f8d274e9cf0224304024013
Adjust for review comments
push time in 11 days
Pull request review commentstandard-ai/hedwig-rust
-//! A Hedwig library for Rust. Hedwig is a message bus that works with arbitrary pubsub services-//! such as AWS SNS/SQS or Google Cloud Pubsub. Messages are validated using a JSON schema. The-//! publisher and consumer are de-coupled and fan-out is supported out of the box.+//! Hedwig is a message bus library that works with arbitrary pubsub services such as AWS SNS/SQS+//! or Google Cloud Pubsub. Messages are validated before they are published. The publisher and+//! consumer are de-coupled and fan-out is supported out of the box. //!-//! # Example: publish a new message+//! The Rust library currently only supports publishing. //!-//! ```no_run-//! use hedwig::{Hedwig, MajorVersion, MinorVersion, Version, Message, publishers::MockPublisher};-//! # use std::path::Path;-//! # use serde::Serialize;-//! # use strum_macros::IntoStaticStr;+//! # Examples //!-//! fn main() -> Result<(), Box<dyn std::error::Error>> {-//! let schema = r#"-//! {-//! "$id": "https://hedwig.standard.ai/schema",-//! "$schema": "https://json-schema.org/draft-04/schema#",-//! "description": "Example Schema",-//! "schemas": {-//! "user-created": {-//! "1.*": {-//! "description": "A new user was created",-//! "type": "object",-//! "x-versions": [-//! "1.0"-//! ],-//! "required": [-//! "user_id"-//! ],-//! "properties": {-//! "user_id": {-//! "$ref": "https://hedwig.standard.ai/schema#/definitions/UserId/1.0"-//! }-//! }-//! }-//! }-//! },-//! "definitions": {-//! "UserId": {-//! "1.0": {-//! "type": "string"-//! }-//! }-//! }-//! }"#;+//! Publish a message. Payload encoded with JSON and validated using a JSON Schema. //!-//! # #[derive(Clone, Copy, IntoStaticStr, Hash, PartialEq, Eq)]-//! # enum MessageType {-//! # #[strum(serialize = "user.created")]-//! # UserCreated,-//! # }-//! #-//! # #[derive(Serialize)]-//! # struct UserCreatedData {-//! # user_id: String,-//! # }-//! #-//! fn router(t: MessageType, v: MajorVersion) -> Option<&'static str> {-//! match (t, v) {-//! (MessageType::UserCreated, MajorVersion(1)) => Some("dev-user-created-v1"),-//! _ => None,+//! ```+//! use uuid::Uuid;+//! use std::{path::Path, time::SystemTime};+//! use futures_util::stream::StreamExt;+//!+//! # #[cfg(not(feature = "json-schema"))]+//! # fn main() {}+//!+//! # #[cfg(feature = "json-schema")] // example uses a JSON Schema validator.+//! # fn main() -> Result<(), Box<dyn std::error::Error>> {+//! let schema = r#"{+//! "$id": "https://hedwig.corp/schema",+//! "$schema": "https://json-schema.org/draft-04/schema#",+//! "description": "Example Schema",+//! "schemas": {+//! "user-created": {+//! "1.*": {+//! "description": "A new user was created",+//! "type": "object",+//! "x-versions": [+//! "1.0"+//! ],+//! "required": [+//! "user_id"+//! ],+//! "properties": {+//! "user_id": {+//! "$ref": "https://hedwig.corp/schema#/definitions/UserId/1.0"+//! }+//! }+//! }+//! }+//! },+//! "definitions": {+//! "UserId": {+//! "1.0": {+//! "type": "string"+//! } //! } //! }+//! }"#; //!-//! // create a publisher instance-//! let publisher = MockPublisher::default();-//! let hedwig = Hedwig::new(-//! schema,-//! "myapp",-//! publisher,-//! router,-//! )?;+//! #[derive(serde::Serialize)]+//! struct UserCreatedMessage {+//! user_id: String,+//! } //!-//! async {-//! let published_ids = hedwig.publish(Message::new(-//! MessageType::UserCreated,-//! Version(MajorVersion(1), MinorVersion(0)),-//! UserCreatedData { user_id: "U_123".into() }-//! )).await;-//! };+//! impl<'a> hedwig::Message for &'a UserCreatedMessage {+//! type Error = hedwig::validators::JsonSchemaValidatorError;+//! type Validator = hedwig::validators::JsonSchemaValidator;+//! fn topic(&self) -> &'static str { "user.created" }+//! fn encode(self, validator: &Self::Validator)+//! -> Result<hedwig::ValidatedMessage, Self::Error> {+//! validator.validate(+//! Uuid::new_v4(),+//! SystemTime::now(),+//! "https://hedwig.corp/schema#/schemas/user.created/1.0",+//! hedwig::Headers::new(),+//! self,+//! )+//! }+//! } //!-//! # Ok(())+//! let publisher = /* Some publisher */+//! # hedwig::publishers::NullPublisher;+//! let validator = hedwig::validators::JsonSchemaValidator::new(schema)?;+//! let mut batch = hedwig::PublishBatch::new();+//! batch.message(&validator, &UserCreatedMessage { user_id: String::from("U_123") });+//! let mut result_stream = batch.publish(&publisher);+//! let mut next_batch = hedwig::PublishBatch::new();+//! async {+//! while let Some(result) = result_stream.next().await {+//! match result {+//! (Ok(id), _, msg) => {+//! println!("message {} published successfully: {:?}", msg.uuid(), id);+//! }+//! (Err(e), topic, msg) => {+//! eprintln!("failed to publish {}: {}", msg.uuid(), e);+//! next_batch.push(topic, msg);+//! }+//! }+//! }+//! };+//! # Ok(()) //! # } //! ``` #![deny( missing_docs,- intra_doc_link_resolution_failure,+ broken_intra_doc_links, clippy::all, unsafe_code, unreachable_pub,- unused,+ unused )] #![cfg_attr(docsrs, feature(doc_cfg))] -use std::{- collections::HashMap,- fmt,- future::Future,- mem,- time::{SystemTime, UNIX_EPOCH},-};+use std::{collections::BTreeMap, time::SystemTime}; -use futures::stream::StreamExt;+use futures_util::stream::{self, Stream, StreamExt}; use uuid::Uuid;-use valico::json_schema::{SchemaError, Scope, ValidationState};--#[cfg(feature = "google")]-#[cfg_attr(docsrs, doc(cfg(feature = "google")))]-mod google_publisher;-mod mock_publisher;-mod null_publisher;--/// Implementations of the Publisher trait-pub mod publishers {- #[cfg(feature = "google")]- #[cfg_attr(docsrs, doc(cfg(feature = "google")))]- pub use super::google_publisher::GooglePubSubPublisher;- pub use super::mock_publisher::MockPublisher;- pub use super::null_publisher::NullPublisher;-} -const FORMAT_VERSION_V1: Version = Version(MajorVersion(1), MinorVersion(0));+pub mod publishers;+#[cfg(test)]+mod tests;+pub mod validators; -/// All errors that may be returned when instantiating a new Hedwig instance.+/// All errors that may be returned when operating top level APIs. #[derive(Debug, thiserror::Error)] #[non_exhaustive] pub enum Error {- /// Unable to deserialize schema- #[error("Unable to deserialize schema")]- SchemaDeserialize(#[source] serde_json::Error),-- /// Schema failed to compile- #[error("Schema failed to compile")]- SchemaCompile(#[from] SchemaError),-- /// Unable to serialize message- #[error("Unable to serialize message")]- MessageSerialize(#[source] serde_json::Error),-- /// Message is not routable- #[error("Message {0} is not routable")]- MessageRoute(Uuid),-- /// Could not parse a schema URL- #[error("Could not parse `{1}` as a schema URL")]- SchemaUrlParse(#[source] url::ParseError, String),-- /// Could not resolve the schema URL- #[error("Could not resolve `{0}` to a schema")]- SchemaUrlResolve(url::Url),-- /// Could not validate message data- #[error("Message data does not validate per the schema: {0}")]- DataValidation(String),-- /// Publisher failed to publish a message- #[error("Publisher failed to publish a message batch")]- Publisher(#[source] Box<dyn std::error::Error + Send + Sync>),-- /// Publisher failed to publish multiple batches of messages- #[error("Publisher failed to publish multiple batches (total of {} errors)", _1.len() + 1)]- PublisherMultiple(- #[source] Box<dyn std::error::Error + Send + Sync>,- Vec<Box<dyn std::error::Error + Send + Sync>>,- ),-}--type AnyError = Box<dyn std::error::Error + Send + Sync>;--/// The special result type for [`Publisher::publish`](trait.Publisher.html)-#[derive(Debug)]-pub enum PublisherResult<Id> {- /// Publisher succeeded.- ///- /// Contains a vector of published message IDs.- Success(Vec<Id>),- /// Publisher failed to publish any of the messages.- OneError(AnyError, Vec<ValidatedMessage>),- /// Publisher failed to publish some of the messages.- ///- /// The error type has a per-message granularity.- PerMessage(Vec<Result<Id, (AnyError, ValidatedMessage)>>),+ /// Unable to encode message payload+ #[error("Unable to encode message payload")]+ EncodeMessage(#[source] Box<dyn std::error::Error + Send + Sync>), } -/// Interface for message publishers+/// Message publishers.+///+/// Message publishers deliver a validated message to an endpoint, possibly a remote one. Message+/// publishers may also additionally validate a message for publisher-specific requirements (e.g.+/// size). pub trait Publisher {- /// The list of identifiers for successfully published messages+ /// The identifier for a successfully published message. type MessageId: 'static;- /// The future that the `publish` method returns- type PublishFuture: Future<Output = PublisherResult<Self::MessageId>> + Send;-- /// Publish a batch of messages- ///- /// # Return value- ///- /// Shall return [`PublisherResult::Success`](PublisherResult::Success) only if all of the- /// messages are successfully published. Otherwise `PublisherResult::OneError` or- /// `PublisherResult::PerMessage` shall be returned to indicate an error.- fn publish(&self, topic: &'static str, messages: Vec<ValidatedMessage>) -> Self::PublishFuture;-}--/// Type alias for custom headers associated with a message-type Headers = HashMap<String, String>; -struct Validator {- scope: Scope,- schema_id: url::Url,-}--impl Validator {- fn new(schema: &str) -> Result<Validator, Error> {- let master_schema: serde_json::Value =- serde_json::from_str(schema).map_err(Error::SchemaDeserialize)?;+ /// The error that this publisher returns when publishing of a message fails.+ type MessageError: std::error::Error + Send + Sync + 'static; - let mut scope = Scope::new();- let schema_id = scope.compile(master_schema, false)?;+ /// The stream of results that the `publish` method returns.+ type PublishStream: Stream<Item = Result<Self::MessageId, Self::MessageError>>; - Ok(Validator { scope, schema_id })- }-- fn validate<D, T>(- &self,- message: &Message<D, T>,- schema: &str,- ) -> Result<ValidationState, Error>+ /// Publish a batch of messages.+ ///+ /// The output stream shall return a result for each message in `messages` slice in order.+ fn publish<'a, I>(&self, topic: &'static str, messages: I) -> Self::PublishStream where- D: serde::Serialize,- {- // convert user.created/1.0 -> user.created/1.*- let msg_schema_ptr = schema.trim_end_matches(char::is_numeric).to_owned() + "*";- let msg_schema_url = url::Url::parse(&msg_schema_ptr)- .map_err(|e| Error::SchemaUrlParse(e, msg_schema_ptr))?;- let msg_schema = self- .scope- .resolve(&msg_schema_url)- .ok_or_else(|| Error::SchemaUrlResolve(msg_schema_url))?;-- let msg_data = serde_json::to_value(&message.data).map_err(Error::MessageSerialize)?;-- let validation_state = msg_schema.validate(&msg_data);- if !validation_state.is_strictly_valid() {- return Err(Error::DataValidation(format!("{:?}", validation_state)));- }- Ok(validation_state)- }+ I: Iterator<Item = &'a ValidatedMessage> + DoubleEndedIterator + ExactSizeIterator; } -/// Major part component in semver-#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq, serde::Serialize)]-pub struct MajorVersion(pub u8);+/// Types that can be encoded and published.+pub trait Message {+ /// The errors that can occur when calling the [`Message::encode`] method.+ ///+ /// Will typically match the errors returned by the [`Message::Validator`].+ type Error: std::error::Error + Send + Sync + 'static; -impl fmt::Display for MajorVersion {- fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {- write!(f, "{}", self.0)- }-}+ /// The validator to use for this message.+ type Validator; -/// Minor part component in semver-#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq, serde::Serialize)]-pub struct MinorVersion(pub u8);+ /// Topic into which this message shall be published.+ fn topic(&self) -> &'static str;
Yeah, they ought to be fairly fixed. The only kind of variation I can see is the -v{version} suffix in{topic}-{version}, but that can and probably should be a separate implementer ofMessage` anyway.
comment created time in 11 days
push eventstandard-ai/hedwig-rust
commit sha 87e0750dee2b7d8668916effe4c7b4e413028767
Expand tests to exercise header validation
push time in 12 days
Pull request review commentrust-lang/stacker
Convert callback to `dyn` to improve compile times
pub fn maybe_grow<R, F: FnOnce() -> R>(red_zone: usize, stack_size: usize, callb /// The closure will still be on the same thread as the caller of `grow`. /// This will allocate a new stack with at least `stack_size` bytes. pub fn grow<R, F: FnOnce() -> R>(stack_size: usize, callback: F) -> R {+ // Measuring with cargo-llvm-lines revealed that `psm::on_stack::with_on_stack`+ // was monomorphized 1552 times in rustc and was responsible for 1.5% of+ // rustc's total llvm IR lines. That takes time for LLVM to process.+ // Converting the generic callback to a dynamic one removes all that duplication.+ let mut opt_callback = Some(callback); let mut ret = None; let ret_ref = &mut ret;- _grow(stack_size, move || {- *ret_ref = Some(callback());- });++ // This wrapper around `callback` achieves two things:+ // * It converts the `impl FnOnce` to a `dyn FnMut`.+ // `dyn` because we want it to not be generic, and+ // `FnMut` because we can't pass a `dyn FnOnce` around without boxing it.+ // * It eliminates the generic return value, by writing it to the stack of this function.+ // Otherwise the closure would have to return an unsized value, which isn't possible.+ let dyn_callback: &mut dyn FnMut() = &mut || {+ let taken_callback = opt_callback.take().unwrap();+ *ret_ref = Some(taken_callback());+ };++ // _grow is only monomorphized once with a `dyn` argument.+ _grow(stack_size, dyn_callback);
Would make sense to me.
comment created time in 13 days
Pull request review commentrust-lang/stacker
Convert callback to `dyn` to improve compile times
pub fn maybe_grow<R, F: FnOnce() -> R>(red_zone: usize, stack_size: usize, callb /// The closure will still be on the same thread as the caller of `grow`. /// This will allocate a new stack with at least `stack_size` bytes. pub fn grow<R, F: FnOnce() -> R>(stack_size: usize, callback: F) -> R {+ // Measuring with cargo-llvm-lines revealed that `psm::on_stack::with_on_stack`+ // was monomorphized 1552 times in rustc and was responsible for 1.5% of+ // rustc's total llvm IR lines. That takes time for LLVM to process.
This metric is probably very misleading, because with_on_stack is the base frame into which LLVM is able to inline the provided callback. with_on_stack is also super small itself (< dozen of instructions), so I have a hard time believing that by itself with_on_stack is a cause of the entirety of 1.5% of instructions.
With that in mind, making the callback dynamic only serves to prevent the inlining of the callback, and thus now all those closures end up having their own "category" or "line" in cargo-llvm-lines output. The improvement in perf/comptime still great, however.
comment created time in 13 days
pull request commentstandard-ai/hedwig-rust
While there are a couple remaining fixmes around in the code, I think its done enough to be in a state that's reviewable enough to produce an useful review of the approach/code.
comment created time in 13 days
Pull request review commentstandard-ai/hedwig-rust
Segmentation of batches for google publisher
pub enum Error { #[error("Message data does not validate per the schema: {0}")] DataValidation(String), - /// Publisher failed to publish a message- #[error("Publisher failed to publish a message batch")]- Publisher(#[source] Box<dyn std::error::Error + Send + Sync>),-- /// Publisher failed to publish multiple batches of messages- #[error("Publisher failed to publish multiple batches (total of {} errors)", _1.len() + 1)]- PublisherMultiple(- #[source] Box<dyn std::error::Error + Send + Sync>,- Vec<Box<dyn std::error::Error + Send + Sync>>,- ),-}--type AnyError = Box<dyn std::error::Error + Send + Sync>;+ /// Could not validate message data+ #[error("Could not add a message to the builder")]+ AddMessage(#[source] Box<dyn std::error::Error + Send + Sync>), -/// The special result type for [`Publisher::publish`](trait.Publisher.html)-#[derive(Debug)]-pub enum PublisherResult<Id> {- /// Publisher succeeded.- ///- /// Contains a vector of published message IDs.- Success(Vec<Id>),- /// Publisher failed to publish any of the messages.- OneError(AnyError, Vec<ValidatedMessage>),- /// Publisher failed to publish some of the messages.- ///- /// The error type has a per-message granularity.- PerMessage(Vec<Result<Id, (AnyError, ValidatedMessage)>>),+ /// Could not validate message data+ #[error("Could not publish the message")]+ PublishOneMessage(#[source] Box<dyn std::error::Error + Send + Sync>), } /// Interface for message publishers pub trait Publisher { /// The list of identifiers for successfully published messages type MessageId: 'static;- /// The future that the `publish` method returns- type PublishFuture: Future<Output = PublisherResult<Self::MessageId>> + Send;+ /// The message error that this publisher returns+ type MessageError: std::error::Error + Send + Sync + 'static;+ /// The stream of results that the `publish` method returns+ type PublishStream: Stream<Item = Result<Self::MessageId, Self::MessageError>> + Unpin; /// Publish a batch of messages ///- /// # Return value- ///- /// Shall return [`PublisherResult::Success`](PublisherResult::Success) only if all of the- /// messages are successfully published. Otherwise `PublisherResult::OneError` or- /// `PublisherResult::PerMessage` shall be returned to indicate an error.- fn publish(&self, topic: &'static str, messages: Vec<ValidatedMessage>) -> Self::PublishFuture;+ /// The output stream shall return a result for each message in `messages` slice in the same+ /// order.+ fn publish<'a, I>(&self, topic: &'static str, messages: I) -> Self::PublishStream+ where+ I: Iterator<Item = &'a ValidatedMessage> + DoubleEndedIterator + ExactSizeIterator;
Stream cannot exactly support DoubleEndedIterator or ExactSizeIterator. While the only "real" publisher does not need either of those (and it would make some sense for it to accept a Stream) I don’t want to make such a change in this PR. Specifically because I don’t want to spend the time right now thinking about what the consequences of such a change might end up being.
comment created time in 13 days
push eventstandard-ai/hedwig-rust
commit sha 485e87deaa5517fcff010e743463566b90b64645
Refactor the library This commit refactors this library, almost from scratch, in order to: 1. Achieve more conventional and easier-to-use APIs; 2. More correct and flexible error handling; 3. Internal segmentation of batches inside of google publisher. The initial motivation was originally 3, but then old way of implicit error handling came up as a problem as well. I attempted to introduce some mechanism to handle errors in a more granular way without adjusting API of the library too much, but it turned out to be not very feasible. The new API is centered around the new `hedwig::Message` trait which the users would implement to associate data types with various information that is necessary for the hewdig library – the topic name, the validators applicable to the data, the encoding mechanisms, etc. Furthermore, it now allows retaining all of auxiliary data alongside the message data as well, giving further flexibility in how this library is used by the users. All in all, the trait was designed in a way that would allow to move the message-encoding specific pieces of code away from the business logic as much as possible. For now the implementation of this trait remains manual, but there is nothing that would prevent it from being `derive`able in some way. --- The error handling was made entirely explicit in the new API. In particular we no longer implicitly re-queue messages in the batch (nor do we reuse the batches, anymore). Instead the users receive a `Stream` of `(result, topic, validated message)`. They can then choose to `push` the validated message back into a new batch they are constructing to retry the message… or just handle the error in whatever other way that makes sense to them. One important improvement as part of this change is simplification of behaviour when the futures/streams are cancelled. Previously dropping the `publish` future would leave one in a fairly difficult to understand state where the batch might have some of the messages re-queued and some of them cancelled, etc. Since there's no longer any state remaining in `hedwig`, dropping streams to cancel starts making significantly more sense as well. --- Google PubSub has various limitations on the API requests that are made to its endpoints. As we are in charge of batching the requests up to implement publishing more efficiently, it is also up to us to ensure that we do not exceed any of the limits upstream has placed on us. To do this we implement a batch segmentation algorithm in `GoogleMessageSegmenter` that the GooglePubSubPublisher uses internally to split up the messages. Fixes #18
push time in 13 days
push eventstandard-ai/hedwig-rust
commit sha 9284641b737a8f5f669cfc1d4782553cd52c48b1
Refactor the library This commit refactors this library, almost from scratch, in order to: 1. Achieve more conventional and easier-to-use APIs; 2. More correct and flexible error handling; 3. Internal segmentation of batches inside of google publisher. The initial motivation was originally 3, but then old way of implicit error handling came up as a problem as well. I attempted to introduce some mechanism to handle errors in a more granular way without adjusting API of the library too much, but it turned out to be not very feasible. The new API is centered around the new `hedwig::Message` trait which the users would implement to associate data types with various information that is necessary for the hewdig library – the topic name, the validators applicable to the data, the encoding mechanisms, etc. Furthermore, it now allows retaining all of auxiliary data alongside the message data as well, giving further flexibility in how this library is used by the users. All in all, the trait was designed in a way that would allow to move the message-encoding specific pieces of code away from the business logic as much as possible. For now the implementation of this trait remains manual, but there is nothing that would prevent it from being `derive`able in some way. --- The error handling was made entirely explicit in the new API. In particular we no longer implicitly re-queue messages in the batch (nor do we reuse the batches, anymore). Instead the users receive a `Stream` of `(result, topic, validated message)`. They can then choose to `push` the validated message back into a new batch they are constructing to retry the message… or just handle the error in whatever other way that makes sense to them. One important improvement as part of this change is simplification of behaviour when the futures/streams are cancelled. Previously dropping the `publish` future would leave one in a fairly difficult to understand state where the batch might have some of the messages re-queued and some of them cancelled, etc. Since there's no longer any state remaining in `hedwig`, dropping streams to cancel starts making significantly more sense as well. --- Google PubSub has various limitations on the API requests that are made to its endpoints. As we are in charge of batching the requests up to implement publishing more efficiently, it is also up to us to ensure that we do not exceed any of the limits upstream has placed on us. To do this we implement a batch segmentation algorithm in `GoogleMessageSegmenter` that the GooglePubSubPublisher uses internally to split up the messages. Fixes #18
push time in 13 days
push eventstandard-ai/hedwig-rust
commit sha 5bf1f5b96336ecd329858b8059f22379011fabc2
Refactor the library This commit refactors this library, almost from scratch, in order to: 1. Achieve more conventional and easier-to-use APIs; 2. More correct and flexible error handling; 3. Internal segmentation of batches inside of google publisher. The initial motivation was originally 3, but then old way of implicit error handling came up as a problem as well. I attempted to introduce some mechanism to handle errors in a more granular way without adjusting API of the library too much, but it turned out to be not very feasible. The new API is centered around the new `hedwig::Message` trait which the users would implement to associate data types with various information that is necessary for the hewdig library – the topic name, the validators applicable to the data, the encoding mechanisms, etc. Furthermore, it now allows retaining all of auxiliary data alongside the message data as well, giving further flexibility in how this library is used by the users. All in all, the trait was designed in a way that would allow to move the message-encoding specific pieces of code away from the business logic as much as possible. For now the implementation of this trait remains manual, but there is nothing that would prevent it from being `derive`able in some way. --- The error handling was made entirely explicit in the new API. In particular we no longer implicitly re-queue messages in the batch (nor do we reuse the batches, anymore). Instead the users receive a `Stream` of `(result, topic, validated message)`. They can then choose to `push` the validated message back into a new batch they are constructing to retry the message… or just handle the error in whatever other way that makes sense to them. One important improvement as part of this change is simplification of behaviour when the futures/streams are cancelled. Previously dropping the `publish` future would leave one in a fairly difficult to understand state where the batch might have some of the messages re-queued and some of them cancelled, etc. Since there's no longer any state remaining in `hedwig`, dropping streams to cancel starts making significantly more sense as well. --- Google PubSub has various limitations on the API requests that are made to its endpoints. As we are in charge of batching the requests up to implement publishing more efficiently, it is also up to us to ensure that we do not exceed any of the limits upstream has placed on us. To do this we implement a batch segmentation algorithm in `GoogleMessageSegmenter` that the GooglePubSubPublisher uses internally to split up the messages. Fixes #18
push time in 13 days
push eventstandard-ai/hedwig-rust
commit sha b41bddae97e761704eaf48df7fcc0a30f03a3e71
Refactor the library This commit refactors this library, almost from scratch, in order to: 1. Achieve more conventional and easier-to-use APIs; 2. More correct and flexible error handling; 3. Internal segmentation of batches inside of google publisher. The initial motivation was originally 3, but then old way of implicit error handling came up as a problem as well. I attempted to introduce some mechanism to handle errors in a more granular way without adjusting API of the library too much, but it turned out to be not very feasible. The new API is centered around the new `hedwig::Message` trait which the users would implement to associate data types with various information that is necessary for the hewdig library – the topic name, the validators applicable to the data, the encoding mechanisms, etc. Furthermore, it now allows retaining all of auxiliary data alongside the message data as well, giving further flexibility in how this library is used by the users. All in all, the trait was designed in a way that would allow to move the message-encoding specific pieces of code away from the business logic as much as possible. For now the implementation of this trait remains manual, but there is nothing that would prevent it from being `derive`able in some way. --- The error handling was made entirely explicit in the new API. In particular we no longer implicitly re-queue messages in the batch (nor do we reuse the batches, anymore). Instead the users receive a `Stream` of `(result, topic, validated message)`. They can then choose to `push` the validated message back into a new batch they are constructing to retry the message… or just handle the error in whatever other way that makes sense to them. One important improvement as part of this change is simplification of behaviour when the futures/streams are cancelled. Previously dropping the `publish` future would leave one in a fairly difficult to understand state where the batch might have some of the messages re-queued and some of them cancelled, etc. Since there's no longer any state remaining in `hedwig`, dropping streams to cancel starts making significantly more sense as well. --- Google PubSub has various limitations on the API requests that are made to its endpoints. As we are in charge of batching the requests up to implement publishing more efficiently, it is also up to us to ensure that we do not exceed any of the limits upstream has placed on us. To do this we implement a batch segmentation algorithm in `GoogleMessageSegmenter` that the GooglePubSubPublisher uses internally to split up the messages. Fixes #18
push time in 13 days
push eventstandard-ai/hedwig-rust
commit sha 0416b1ab63c2e337e65f86be1356a21b11cef426
Refactor the library This commit refactors this library, almost from scratch, in order to: 1. Achieve more conventional and easier-to-use APIs; 2. More correct and flexible error handling; 3. Internal segmentation of batches inside of google publisher. The initial motivation was originally 3, but then old way of implicit error handling came up as a problem as well. I attempted to introduce some mechanism to handle errors in a more granular way without adjusting API of the library too much, but it turned out to be not very feasible. The new API is centered around the new `hedwig::Message` trait which the users would implement to associate data types with various information that is necessary for the hewdig library – the topic name, the validators applicable to the data, the encoding mechanisms, etc. Furthermore, it now allows retaining all of auxiliary data alongside the message data as well, giving further flexibility in how this library is used by the users. All in all, the trait was designed in a way that would allow to move the message-encoding specific pieces of code away from the business logic as much as possible. For now the implementation of this trait remains manual, but there is nothing that would prevent it from being `derive`able in some way. --- The error handling was made entirely explicit in the new API. In particular we no longer implicitly re-queue messages in the batch (nor do we reuse the batches, anymore). Instead the users receive a `Stream` of `(result, topic, validated message)`. They can then choose to `push` the validated message back into a new batch they are constructing to retry the message… or just handle the error in whatever other way that makes sense to them. One important improvement as part of this change is simplification of behaviour when the futures/streams are cancelled. Previously dropping the `publish` future would leave one in a fairly difficult to understand state where the batch might have some of the messages re-queued and some of them cancelled, etc. Since there's no longer any state remaining in `hedwig`, dropping streams to cancel starts making significantly more sense as well. --- Google PubSub has various limitations on the API requests that are made to its endpoints. As we are in charge of batching the requests up to implement publishing more efficiently, it is also up to us to ensure that we do not exceed any of the limits upstream has placed on us. To do this we implement a batch segmentation algorithm in `GoogleMessageSegmenter` that the GooglePubSubPublisher uses internally to split up the messages. Fixes #18
push time in 13 days
push eventstandard-ai/hedwig-rust
commit sha 8fb7ba842397168f279325c6f43b8424b264348b
Refactor the library This commit refactors this library, almost from scratch, in order to: 1. Achieve more conventional and easier-to-use APIs; 2. More correct and flexible error handling; 3. Internal segmentation of batches inside of google publisher. The initial motivation was originally 3, but then old way of implicit error handling came up as a problem as well. I attempted to introduce some mechanism to handle errors in a more granular way without adjusting API of the library too much, but it turned out to be not very feasible. The new API is centered around the new `hedwig::Message` trait which the users would implement to associate data types with various information that is necessary for the hewdig library – the topic name, the validators applicable to the data, the encoding mechanisms, etc. Furthermore, it now allows retaining all of auxiliary data alongside the message data as well, giving further flexibility in how this library is used by the users. All in all, the trait was designed in a way that would allow to move the message-encoding specific pieces of code away from the business logic as much as possible. For now the implementation of this trait remains manual, but there is nothing that would prevent it from being `derive`able in some way. --- The error handling was made entirely explicit in the new API. In particular we no longer implicitly re-queue messages in the batch (nor do we reuse the batches, anymore). Instead the users receive a `Stream` of `(result, topic, validated message)`. They can then choose to `push` the validated message back into a new batch they are constructing to retry the message… or just handle the error in whatever other way that makes sense to them. One important improvement as part of this change is simplification of behaviour when the futures/streams are cancelled. Previously dropping the `publish` future would leave one in a fairly difficult to understand state where the batch might have some of the messages re-queued and some of them cancelled, etc. Since there's no longer any state remaining in `hedwig`, dropping streams to cancel starts making significantly more sense as well. --- Google PubSub has various limitations on the API requests that are made to its endpoints. As we are in charge of batching the requests up to implement publishing more efficiently, it is also up to us to ensure that we do not exceed any of the limits upstream has placed on us. To do this we implement a batch segmentation algorithm in `GoogleMessageSegmenter` that the GooglePubSubPublisher uses internally to split up the messages. Fixes #18
push time in 13 days
pull request commentnagisa/rust_tracy_client
Add support for optionally using externally built Tracy library
Even if the default behaviour of the crate remains the same?
Well, it still exposes a way for people to shoot themselves in a foot, by means that are not immediately apparent as to their severity.
For tracy-client-sys to support the use-case of using arbitrary version of tracy it would need to be rewritten to generate the bindings at build time from the provided headers (via bindgen), which by itself is a not a terrible proposition. At the very least it would make any mismatches explode loudly at compile time. With the way tracy-client-sys bindings are hand-written currently, were we to allow users to provide an arbitrary version of tracy binary code, the signature mismatches would compile correctly, but then clobber registers, memory etc at runtime, leading to difficult to debug crashes etc.
comment created time in 14 days
pull request commentnagisa/rust_tracy_client
Add support for optionally using externally built Tracy library
I’m somewhat uncomfortable with this, mostly because tracy is not very good at maintaining a stable ABI and versioning the project accordingly.
comment created time in 14 days
push eventnagisa/prost
commit sha 1466355d7ce9c627e8183994e8ecb3972d9957fd
Do not force an unnecessary Sized bound This allows users to call `Message` methods on `&(dyn Message)`.
push time in 14 days
PR opened danburkert/prost
This allows users to call Message methods on &(dyn Message).
pr created time in 14 days
Pull request review commentseanmonstar/num_cpus
Add cgroups support for Linux targets
+use std::collections::HashMap;+use std::fs::File;+use std::io::{BufRead, BufReader, Read};+use std::mem;+use std::path::{Path, PathBuf};+use std::sync::atomic::{AtomicUsize, Ordering};+use std::sync::Once;++use libc;++macro_rules! debug {+ ($($args:expr),*) => ({+ if false {+ //if true {+ println!($($args),*);+ }+ });+}++macro_rules! some {+ ($e:expr) => ({+ match $e {+ Some(v) => v,+ None => {+ debug!("NONE: {:?}", stringify!($e));+ return None;+ }+ }+ })+}++pub fn get_num_cpus() -> usize {+ match cgroups_num_cpus() {+ Some(n) => n,+ None => logical_cpus(),+ }+}++fn logical_cpus() -> usize {+ let mut set: libc::cpu_set_t = unsafe { mem::zeroed() };+ if unsafe { libc::sched_getaffinity(0, mem::size_of::<libc::cpu_set_t>(), &mut set) } == 0 {+ let mut count: u32 = 0;+ for i in 0..libc::CPU_SETSIZE as usize {+ if unsafe { libc::CPU_ISSET(i, &set) } {+ count += 1+ }+ }+ count as usize+ } else {+ let cpus = unsafe { libc::sysconf(libc::_SC_NPROCESSORS_ONLN) };+ if cpus < 1 {+ 1+ } else {+ cpus as usize+ }+ }+}++pub fn get_num_physical_cpus() -> usize {+ let file = match File::open("/proc/cpuinfo") {+ Ok(val) => val,+ Err(_) => return get_num_cpus(),+ };+ let reader = BufReader::new(file);+ let mut map = HashMap::new();+ let mut physid: u32 = 0;+ let mut cores: usize = 0;+ let mut chgcount = 0;+ for line in reader.lines().filter_map(|result| result.ok()) {+ let mut it = line.split(':');+ let (key, value) = match (it.next(), it.next()) {+ (Some(key), Some(value)) => (key.trim(), value.trim()),+ _ => continue,+ };+ if key == "physical id" {+ match value.parse() {+ Ok(val) => physid = val,+ Err(_) => break,+ };+ chgcount += 1;+ }+ if key == "cpu cores" {+ match value.parse() {+ Ok(val) => cores = val,+ Err(_) => break,+ };+ chgcount += 1;+ }+ if chgcount == 2 {+ map.insert(physid, cores);+ chgcount = 0;+ }+ }+ let count = map.into_iter().fold(0, |acc, (_, cores)| acc + cores);++ if count == 0 {+ get_num_cpus()+ } else {+ count+ }+}++/// Cached CPUs calculated from cgroups.+///+/// If 0, check logical cpus.+// Allow deprecation warnings, we want to work on older rustc+#[allow(warnings)]+static CGROUPS_CPUS: AtomicUsize = ::std::sync::atomic::ATOMIC_USIZE_INIT;++fn cgroups_num_cpus() -> Option<usize> {+ #[allow(warnings)]+ static ONCE: Once = ::std::sync::ONCE_INIT;++ ONCE.call_once(init_cgroups);++ let cpus = CGROUPS_CPUS.load(Ordering::Acquire);++ if cpus > 0 {+ Some(cpus)+ } else {+ None+ }+}++fn init_cgroups() {+ // Should only be called once
What happens if the cgroup limits are changed while a program is running?
comment created time in 15 days
issue commentrust-lang/compiler-builtins
__rust_u128_mulo aborts when rhs is zero
Is it doing a widening multiplication, and checking if the high part is not zero?
No, none of the multiplications in this are “widening” in a sense that there are no 256-bit operations involved for a 128-bit UMULO (but there are some simpler 128-bit operations). Here's the algorithm above with types substituted for u128, maybe it'll be clearer:
%0 = %LHS.HI != 0 && %RHS.HI != 0
%1 = { i64, i1 } @umul.with.overflow.i64(i64 %LHS.HI, i64 %RHS.LO)
%2 = { i64, i1 } @umul.with.overflow.i64(i64 %RHS.HI, i64 %LHS.LO)
%3 = mul nuw i128 (%LHS.LOW as i128), (%RHS.LOW as i128) ; note: regular wrapping multiplication
%4 = add i128 (%1.0 as i128) << 64, (%2.0 as i128) << 64
%5 = { i128, i1 } @uadd.with.overflow.i128(i128 %4, i128 %3)
%res = { %5.0, %0 || %1.1 || %2.1 || %5.1 }
Do note that this is a unsigned version, I couldn't figure out one for signed integers.
comment created time in 15 days
issue commentrust-lang/compiler-builtins
__rust_u128_mulo aborts when rhs is zero
In particular:
%0 = %LHS.HI != 0 && %RHS.HI != 0
%1 = { iNh, i1 } @umul.with.overflow.iNh(iNh %LHS.HI, iNh %RHS.LO)
%2 = { iNh, i1 } @umul.with.overflow.iNh(iNh %RHS.HI, iNh %LHS.LO)
%3 = mul nuw iN (%LHS.LOW as iN), (%RHS.LOW as iN)
%4 = add iN (%1.0 as iN) << Nh, (%2.0 as iN) << Nh
%5 = { iN, i1 } @uadd.with.overflow.iN( %4, %3 )
%res = { %5.0, %0 || %1.1 || %2.1 || %5.1 }
comment created time in 15 days
issue commentrust-lang/compiler-builtins
__rust_u128_mulo aborts when rhs is zero
I suggest you to copy the algorithm LLVM uses to lower the operation as I implemented in https://reviews.llvm.org/rG73e8a784e62f945a51363c8b5ec4eaedcf9f87e8.
comment created time in 15 days
issue commentrust-lang/rust
Fast algorithm for u128 (and i128) divided by small constant
The fact that LLVM cannot reduce the division to multiplications by a constant might be caused
I investigated this recently for https://github.com/rust-lang/rust/pull/76017#discussion_r479699174.
It is a bug in LLVM and it isn’t like it is impossible for it to strength-reduce, its just that doing so:
a) requires the upper-half of the multiplication result (i.e. for 128-bit multiplication it requires the upper 128-bits of a 256-bit result); and b) calculating that cannot be easily made less conservative because there are a couple of bad backends in LLVM – namely RISCV and 32-bit ARM.
I think I know an easy way to resolve it.
u64 numbers get divided by a small constant divisor using a fast algorithm
Turns out it isn’t on 32-bit targets!
comment created time in 15 days
issue commentrust-lang/rust
CJK letters in rustdoc are not rendered in serif font when using Firefox
I can confirm this. Seems like a browser bug. Even if just serif is specified, the browser selects a sans-serif font (in my case Source Han Sans) even though I have a serif font available too (Source Han Serif).
comment created time in 15 days
Pull request review commentrust-lang/rust
unix: Extend UnixStream and UnixDatagram to send and receive file descriptors
+use crate::convert::TryFrom;+use crate::io::{self, IoSliceMut};+use crate::marker::PhantomData;+use crate::mem::{size_of, zeroed};+use crate::os::unix::io::RawFd;+use crate::path::Path;+use crate::ptr::{null_mut, read_unaligned};+use crate::slice::from_raw_parts;+use crate::sys::unix::ext::net::addr::{sockaddr_un, SocketAddr};+use crate::sys::unix::net::Socket;++pub(super) fn recv_vectored_with_ancillary_from(+ socket: &Socket,+ bufs: &mut [IoSliceMut<'_>],+ ancillary: &mut SocketAncillary<'_>,+) -> io::Result<(usize, bool, io::Result<SocketAddr>)> {+ unsafe {+ let mut msg_name: libc::sockaddr_un = zeroed();++ let mut msg = libc::msghdr {+ msg_name: &mut msg_name as *mut _ as *mut _,+ msg_namelen: size_of::<libc::sockaddr_un>() as libc::socklen_t,+ msg_iov: bufs.as_mut_ptr().cast(),+ msg_iovlen: bufs.len(),+ msg_control: ancillary.buffer.as_mut_ptr().cast(),+ msg_controllen: ancillary.buffer.len(),+ msg_flags: 0,+ };++ let count = socket.recv_msg(&mut msg)?;++ ancillary.length = msg.msg_controllen;+ ancillary.truncated = msg.msg_flags & libc::MSG_CTRUNC == libc::MSG_CTRUNC;++ let truncated = msg.msg_flags & libc::MSG_TRUNC == libc::MSG_TRUNC;+ let addr = SocketAddr::from_parts(msg_name, msg.msg_namelen);++ Ok((count, truncated, addr))+ }+}++pub(super) fn send_vectored_with_ancillary_to(+ socket: &Socket,+ path: Option<&Path>,+ bufs: &mut [IoSliceMut<'_>],+ ancillary: &mut SocketAncillary<'_>,+) -> io::Result<usize> {+ unsafe {+ let (mut msg_name, msg_namelen) =+ if let Some(path) = path { sockaddr_un(path)? } else { (zeroed(), 0) };++ let mut msg = libc::msghdr {+ msg_name: &mut msg_name as *mut _ as *mut _,+ msg_namelen,+ msg_iov: bufs.as_mut_ptr().cast(),+ msg_iovlen: bufs.len(),+ msg_control: ancillary.buffer.as_mut_ptr().cast(),+ msg_controllen: ancillary.length,+ msg_flags: 0,+ };++ ancillary.truncated = false;++ socket.send_msg(&mut msg)+ }+}++fn add_to_ancillary_data<T>(+ buffer: &mut [u8],+ length: &mut usize,+ source: &[T],+ cmsg_level: libc::c_int,+ cmsg_type: libc::c_int,+) -> bool {+ let source_len = if let Some(source_len) = source.len().checked_mul(size_of::<T>()) {+ if let Ok(source_len) = u32::try_from(source_len) {+ source_len+ } else {+ return false;+ }+ } else {+ return false;+ };++ unsafe {+ let additional_space = libc::CMSG_SPACE(source_len) as usize;++ let new_length = if let Some(new_length) = additional_space.checked_add(*length) {+ new_length+ } else {+ return false;+ };++ if new_length > buffer.len() {+ return false;+ }++ buffer[*length..new_length].fill(0);++ *length = new_length;++ let msg = libc::msghdr {+ msg_name: null_mut(),+ msg_namelen: 0,+ msg_iov: null_mut(),+ msg_iovlen: 0,+ msg_control: buffer.as_mut_ptr().cast(),+ msg_controllen: *length,+ msg_flags: 0,+ };++ let mut cmsg = libc::CMSG_FIRSTHDR(&msg);+ let mut previous_cmsg = cmsg;+ while !cmsg.is_null() {+ previous_cmsg = cmsg;+ cmsg = libc::CMSG_NXTHDR(&msg, cmsg);+ }++ if previous_cmsg.is_null() {+ return false;+ }++ (*previous_cmsg).cmsg_level = cmsg_level;+ (*previous_cmsg).cmsg_type = cmsg_type;+ (*previous_cmsg).cmsg_len = libc::CMSG_LEN(source_len) as usize;++ let data = libc::CMSG_DATA(previous_cmsg).cast();++ libc::memcpy(data, source.as_ptr().cast(), source_len as usize);
You’re already casting the pointer away to *mut u8 (which is always aligned) for memcpy. You can do exactly the same for ptr::copy{,_nonoverlapping}.
comment created time in 17 days
pull request commentrust-lang/rust
Syntactically permit unsafety on mods
@bors r+
comment created time in 18 days
issue commentrust-lang/rust
signal: 11, SIGSEGV: invalid memory reference
Have you overclocked your memory (by e.g. applying XMP) ? I've seen similarly bizarre and spurious issues come up with an unstable memory overclock.
comment created time in 18 days