Any update on this please?

Currently brew services isn't able to start materialized because this param was renamed.

Hi, I'm trying to send a payload without a key to Kafka and I'm obtaining this error:

cannot infer type for type parameter K

My IDE shows me this error:

type inside async fn body must be known in this context E0698 cannot infer type for type parameter K declared on the associated function send Note: the type is part of the async fn body because of this await

Without the key:

async fn create_json(data: String, producer: Data<FutureProducer>) {
    producer
        .send(
            FutureRecord::to("test").payload(&format!("{}", Json(AddResp { result: data }).result)),
            Timeout::Never,
        )
        .await;
}

But it's working with the key:

async fn create_json(data: String, producer: Data<FutureProducer>) {
    producer
        .send(
            FutureRecord::to("test")
                .payload(&format!(
                    "{}",
                    Json(AddResp {
                        result: data.clone()
                    })
                    .result
                ))
                .key(&data),
            Timeout::Never,
        )
        .await;
}

Versions:

• rdkafka: 0.26 (cmake-build)
• rustc: 1.52.1 (9bc8c42bb 2021-05-09)

Have you any idea? Thanks

Ah, it was just that thing 😕 I see that I'm really new on rust… Thank you a lot

Look at the definition of the FutureProducer:

pub struct FutureRecord<'a, K: ToBytes + ?Sized, P: ToBytes + ?Sized> {
    /// Required destination topic.
    pub topic: &'a str,
    /// Optional destination partition.
    pub partition: Option<i32>,
    /// Optional payload.
    pub payload: Option<&'a P>,
    /// Optional key.
    pub key: Option<&'a K>,
    /// Optional timestamp.
    pub timestamp: Option<i64>,
    /// Optional message headers.
    pub headers: Option<OwnedHeaders>,
}
[...]

    /// Sets the destination payload of the record.
    pub fn payload(mut self, payload: &'a P) -> FutureRecord<'a, K, P> {
        self.payload = Some(payload);
        self
    }

    /// Sets the destination key of the record.
    pub fn key(mut self, key: &'a K) -> FutureRecord<'a, K, P> {
        self.key = Some(key);
        self
    }

As you never set the key the compiler doesn't know the type of K when compiling the program. One way to solve this is by providing a type when initialising the variable.

So in your case you could solve it via:

async fn create_json(data: String, producer: Data<FutureProducer>) {
    let record: FutureRecord<String, String> = FutureRecord::to("test").payload(&format!("{}", Json(AddResp { result: data }).result));
    producer
        .send(
            record,
            Timeout::Never,
        )
        .await;
}

Now K has a type when compiling.

Part of https://github.com/MaterializeInc/materialize/issues/6957

This is example produces 1_000_000 messages in around 1-2secs depending on the machine that it is being ran on.

Closes #369

Hi, I'm trying to send a payload without a key to Kafka and I'm obtaining this error:

cannot infer type for type parameter K

My IDE shows me this error:

type inside async fn body must be known in this context E0698 cannot infer type for type parameter K declared on the associated function send Note: the type is part of the async fn body because of this await

Without the key:

async fn create_json(data: String, producer: Data<FutureProducer>) {
    producer
        .send(
            FutureRecord::to("test").payload(&format!("{}", Json(AddResp { result: data }).result)),
            Timeout::Never,
        )
        .await;
}

But it's working with the key:

async fn create_json(data: String, producer: Data<FutureProducer>) {
    producer
        .send(
            FutureRecord::to("test")
                .payload(&format!(
                    "{}",
                    Json(AddResp {
                        result: data.clone()
                    })
                    .result
                ))
                .key(&data),
            Timeout::Never,
        )
        .await;
}

Have you any idea? Thanks

Ah––I really learned a lot here. I clearly didn't understand the alignment issue, but I've since closely read the docs and come up with some test cases that fail in the old implementation. ty for your patience and hope this moves the PR forward.

If you take the output of to_raw_parts and blat them down somewhere else, e.g. in a row, this function gives you a way to determine the number of bytes you need to read out of the row to provide valid input to from_raw_parts as a function of digits, i.e. this is used on my fork of MZ locally for deserializing rows back to Datum::APD.

This doesn't need to be exported from the library, but we would need to have the same logic somewhere else; figured this was the most ergonomic place and addressed the oddity in the doc comment.

Want to apologize bc last week you pointed out the issues I tried to fix this morning (MaterializeInc/materialize#6883). I thought this approach was an optimization and hadn't realized you filed an issue bc it actually fixed something! Nonetheless, this code makes implementing the fix very straightforward.

Want to apologize bc last week you pointed out the issues I tried to fix this morning (MaterializeInc/materialize#6883). I thought this approach was an optimization and hadn't realized you filed an issue bc it actually fixed something! Nonetheless, this code makes implementing the fix very straightforward.

Hey there,

Hope you all are doing fine!

As https://github.com/fede1024/kafka-benchmark/issues/10 cannot be built anymore is there any chance that the examples could be extended with an example showing how to produce around 1,000,000 messages per second.

I tried running the simple producer from the examples but stopped it after running it for 1min on my machine. Then I updated it with buffering of futures which resulted in 4814ms for 250_000 messages but that is still far off that 1,000,000 messages per second.

Would really appreciate an example for producing a lot of messages or fixing the kafka-benchmark so it can be ran locally!

Best regards,

Dario

Code snippets with the following toml:

[dependencies]
rdkafka = { version = "0.26", features = ["cmake-build"] }
tokio = { version = "1", features = ["full"] }
futures = "0.3.15"

<details> <summary>Producer with futures buffering 4814ms for 250_000 messages</summary> <pre> use std::time::{Duration, Instant};

use rdkafka::{ClientConfig, producer::{FutureProducer, FutureRecord}, util::get_rdkafka_version};

use futures::{StreamExt, stream::{self}};

#[tokio::main] async fn main() {

let (version_n, version_s) = get_rdkafka_version();
println!("rd_kafka_version: 0x{:08x}, {}", version_n, version_s);

let producer: &FutureProducer = &ClientConfig::new()
    .set("bootstrap.servers", "localhost:9092")
    .set("message.timeout.ms", "5000")
    .create()
    .expect("Kafka Producer creation error");

let start = Instant::now();

let amount =250_000;

let futures = (0..amount)
    .map(|i| async move {
        let delivery_status = producer
            .send(
                FutureRecord::to("topic-test")
                    .payload(&format!("Message {}", i))
                    .key(&format!("Key {}", i)),
                Duration::from_secs(0),
            )
            .await;

        delivery_status
    })
    .collect::<Vec<_>>();


let mut futures = stream::iter(futures).buffered(500);

while let Some(future) = futures.next().await {
    future.unwrap();
}

println!("Took {}ms to publish {} msg", start.elapsed().as_millis(), amount);

}

Running target/debug/rust_playground rd_kafka_version: 0x000001ff, 1.6.1 Took 4814ms to publish 250000 msg </pre> </details>

<details> <summary>Producer as in examples ??(stopped after 1minute) for 250_000 messages</summary> <pre> use std::time::{Duration, Instant};

use rdkafka::{ClientConfig, producer::{FutureProducer, FutureRecord}, util::get_rdkafka_version};

use futures::{StreamExt, stream::{self}};

#[tokio::main] async fn main() {

let (version_n, version_s) = get_rdkafka_version();
println!("rd_kafka_version: 0x{:08x}, {}", version_n, version_s);

let producer: &FutureProducer = &ClientConfig::new()
    .set("bootstrap.servers", "localhost:9092")
    .set("message.timeout.ms", "5000")
    .create()
    .expect("Kafka Producer creation error");

let start = Instant::now();

let amount =250_000;

let futures = (0..amount)
    .map(|i| async move {
        let delivery_status = producer
            .send(
                FutureRecord::to("topic-test")
                    .payload(&format!("Message {}", i))
                    .key(&format!("Key {}", i)),
                Duration::from_secs(0),
            )
            .await;

        delivery_status
    })
    .collect::<Vec<_>>();

// This loop will wait until all delivery statuses have been received.
for future in futures {
    future.await.unwrap();
}

println!("Took {}ms to publish {} msg", start.elapsed().as_millis(), amount);

}

</pre> </details>

Oh derp; ty for catching