profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/NiteshKant/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.
Nitesh Kant NiteshKant Apple San francisco

NiteshKant/Aeron 1

Efficient reliable UDP unicast, UDP multicast, and IPC message transport

NiteshKant/Agrona 1

High Performance data structures and utility methods for Java

NiteshKant/akka 1

Akka Project

NiteshKant/armeria 1

Asynchronous RPC/REST library built on top of Java 8, Netty, HTTP/2, Thrift and gRPC

NiteshKant/netty-playground 1

My playground to experiment with netty.

NiteshKant/DataflowJavaSDK 0

Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines.

NiteshKant/envoy 0

Cloud-native high-performance edge/middle/service proxy

NiteshKant/eureka 0

AWS Service registry for resilient mid-tier load balancing and failover.

NiteshKant/feign 0

Feign makes writing java http clients easier

push eventNiteshKant/netty

Nitesh Kant

commit sha 29cae0445a7bf9bb9ee9b9f2785c47b74dfd2592

Introduce `ByteToMessageDecoderForBuffer` (#11654) __Motivation__ In order to migrate all codec incrementally to use `Buffer`, we need a version of `ByteToMessageDecoder` that uses `Buffer`. __Modification__ - Added the new version of `ByteToMessageDecoder` with a new name so that both old and new version can co-exist and we can incrementally migrate all codecs - Migrated `FixedLengthFrameDecoder` as it was simple and used in tests. __Result__ We have the basic building block to start migrating all codecs to the new `Buffer` API.

view details

Chris Vest

commit sha cf1ab852d1ad7339fb4a9b79dbd459d2c65279fc

Add pinnedHeap/DirectMemory methods to ByteBufAllocatorMetric (#11667) Motivation: The "used memory" is the amount of memory that a pooled allocator has currently allocated and committed for itself. Ths is useful for managing resource usage of the pool versus the available system resources. However, it is not useful for managing resources of the currently circulating buffer instances versus the pool. The pinned memory is the memory currently in use by buffers in circulation, plus memory held in the thread-local caches. Modification: Add pinned memory accounting to PoolChunk. We cannot just use the existing freeBytes because that field is only updated when pool subpages are retired, and a chunk will never retire its last subpage instance. The accounting statistics are available on the PooledByteBufAllocator only, since the metrics interfaces cannot be changed due to backwards compatibility. Result: It is now possible to get a fairly accurate (with slight over-counting due to the thread-local caches) picture of how much memory is held up in buffer instances at any given moment. Fixes #11637

view details

Chris Vest

commit sha 000f2a0934196c94601f6df9b075a25e542e0b3c

Add CompositeBuffer.decomposeBuffer method (#11683) Motivation: It may in some cases be useful to unwrap a composite buffer and work on the array of buffers directly. The decomposeBuffer method makes this possible safely, by killing the composite buffer in the process. Modification: Add a CompositeBuffer.decomposeBuffer method, which returns the array of the constituent component buffers of the composite buffer, and at the same time closes the composite buffer without closing its components. The caller effectively takes ownership of the component buffers away from the composite buffer. Result: This API makes buffer composition fully reversible.

view details

Norman Maurer

commit sha 8339a00fd694d375eccc2719ba11d22e3722683e

Ensure bom don't define extra dependencies (#11682) Motivation: The bom shouldnt depend on the parent as it may define extra dependencies that would be "pulled" in. See https://github.com/netty/netty/pull/11672#discussion_r707619462 Modifications: - Revert commit d6383bf2478267eb62b1f71807c2a3c34794d1b0. - Add tcnative version and add comments to ensure we keep the version in-sync Result: Correct bom for netty

view details

Chris Vest

commit sha 0973673fce993597e7fde5a033cd6660a0483c3d

Make CompositeBuffer an interface (#11687) Motivation: Having composite buffer as a class has caused problems in the past. Making it an interface makes it possible to stub or decorate the composite buffer implementation. Modification: Make CompositeBuffer an interface, and move the implementation to DefaultCompositeBuffer. Then hide the implementation and permit construction only through static factory methods. Result: CompositeBuffer is now an interface.

view details

NiteshKant

commit sha 819bba61bafc64d105567718c92609292865e2c6

Add more utility methods to Buffer __Motivation__ During the process of migrating codecs to use `Buffer`, I stumbled upon a few utility methods that may be useful to be added to `Buffer` __Modification__ - Added `[read, write]Split()` methods to split at an offset from the current reader/writer offset. - Added `accumulate[Reader,Writer]Offset()` methods to increment/decrement reader/writer offsets. - Added `[read, write]CharSequence()` methods which is used extensively in HTTP. __Result__ More utility methods to be used by codecs.

view details

NiteshKant

commit sha d0cbce19be54a06f4569716bacd434ed4605b3fc

Review comments

view details

push time in 10 hours

Pull request review commentnetty/netty

Add more utility methods to Buffer

+/*+ * Copyright 2021 The Netty Project+ *+ * The Netty Project licenses this file to you under the Apache License, version 2.0 (the+ * "License"); you may not use this file except in compliance with the License. You may obtain a+ * copy of the License at:+ *+ * https://www.apache.org/licenses/LICENSE-2.0+ *+ * Unless required by applicable law or agreed to in writing, software distributed under the License+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express+ * or implied. See the License for the specific language governing permissions and limitations under+ * the License.+ */+package io.netty.buffer.api;++import io.netty.util.AsciiString;+import io.netty.util.CharsetUtil;++import java.nio.charset.Charset;++import static io.netty.util.CharsetUtil.US_ASCII;++/**+ * Utilities for buffers.+ */+final class BufferUtils {+    private BufferUtils() {+        // no instances.+    }++    static CharSequence readCharSequence(Buffer source, int length, Charset charset) {+        byte[] data = new byte[length];+        source.copyInto(source.readerOffset(), data, 0, length);+        source.accumulateReaderOffset(length);+        if (US_ASCII.equals(charset)) {+            return new AsciiString(data);+        }+        return new String(data, 0, length, charset);+    }++    static void writeCharSequence(CharSequence source, Buffer destination, Charset charset) {+        if (US_ASCII.equals(charset) && source instanceof AsciiString) {+            AsciiString asciiString = (AsciiString) source;+            destination.writeBytes(asciiString.array(), asciiString.arrayOffset(), source.length());+            return;+        }+        // TODO: Copy optimized writes from ByteBufUtil

When I have the HTTP changes in place, let us discuss in the context of those which seems to be the reason why the optimizations were added to ByteBufUtil.

NiteshKant

comment created time in 10 hours

PullRequestReviewEvent

push eventNiteshKant/netty

NiteshKant

commit sha a829941637f19906ec0e301f34fc613bb0c15324

Review comments

view details

push time in 10 hours

Pull request review commentnetty/netty

Add more utility methods to Buffer

+/*+ * Copyright 2021 The Netty Project+ *+ * The Netty Project licenses this file to you under the Apache License, version 2.0 (the+ * "License"); you may not use this file except in compliance with the License. You may obtain a+ * copy of the License at:+ *+ * https://www.apache.org/licenses/LICENSE-2.0+ *+ * Unless required by applicable law or agreed to in writing, software distributed under the License+ * is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express+ * or implied. See the License for the specific language governing permissions and limitations under+ * the License.+ */+package io.netty.buffer.api.tests;++import io.netty.buffer.api.Buffer;+import io.netty.buffer.api.BufferAllocator;+import io.netty.util.AsciiString;+import org.checkerframework.checker.units.qual.A;+import org.junit.Assert;+import org.junit.jupiter.api.Assertions;+import org.junit.jupiter.params.ParameterizedTest;+import org.junit.jupiter.params.provider.MethodSource;++import java.nio.charset.StandardCharsets;++import static java.nio.charset.StandardCharsets.US_ASCII;++public class BufferCharSequenceTest extends BufferTestSupport {++    @ParameterizedTest+    @MethodSource("allocators")+    void readCharSequence(Fixture fixture) {+        try (BufferAllocator allocator = fixture.createAllocator()) {+            try (Buffer buf = allocator.allocate(32)) {+                String data = "Hello World";+                buf.writeBytes(data.getBytes());+                assertEquals(data.length(), buf.writerOffset());+                assertEquals(0, buf.readerOffset());++                final CharSequence charSequence = buf.readCharSequence(data.length(), US_ASCII);+                Assertions.assertEquals(data, charSequence.toString());+                assertEquals(data.length(), buf.writerOffset());+                assertEquals(data.length(), buf.readerOffset());+            }+        }+    }++    @ParameterizedTest+    @MethodSource("allocators")+    void writeCharSequence(Fixture fixture) {+        try (BufferAllocator allocator = fixture.createAllocator()) {+            try (Buffer buf = allocator.allocate(32)) {+                AsciiString data = new AsciiString("Hello world".getBytes(US_ASCII));+                buf.writeCharSequence(data, US_ASCII);+                assertEquals(data.length(), buf.writerOffset());+                assertEquals(0, buf.readerOffset());++                final byte[] read = readByteArray(buf);+                Assertions.assertEquals(data.toString(), new String(read, US_ASCII));+                assertEquals(data.length(), buf.writerOffset());+                assertEquals(data.length(), buf.readerOffset());+            }+        }+    }++    @ParameterizedTest+    @MethodSource("allocators")+    void readAndWriteCharSequence(Fixture fixture) {+        try (BufferAllocator allocator = fixture.createAllocator()) {+            try (Buffer buf = allocator.allocate(32)) {+                AsciiString data = new AsciiString("Hello world".getBytes(US_ASCII));+                buf.writeCharSequence(data, US_ASCII);+                assertEquals(data.length(), buf.writerOffset());+                assertEquals(0, buf.readerOffset());++                final CharSequence read = buf.readCharSequence(data.length(), US_ASCII);+                Assertions.assertEquals(data, read);+                assertEquals(data.length(), buf.writerOffset());+                assertEquals(data.length(), buf.readerOffset());+            }+        }+    }

In the interest of time and the fact that we are depending on the contract of CharSequence, lemme punt that for now

NiteshKant

comment created time in 10 hours

PullRequestReviewEvent

issue commentnetty/netty

`SingleThreadEventExecutor.shutdownGracefully` starts thread to shut it down?

creation of hundreds of threads on shutdown, might not be desirable either.

I understand what you are saying but this statement isn't really true. Threads are not created "on shutdown", but the code ensures that the associated thread per eventloop is created before shutdown, optimistically assuming that if an eventloop is created the intent is to use it. The downside is that if this optimistic assumption is violated, as in your case, then it seems unnecessary to create the thread.

The reason thread is created and run before graceful shutdown is that we want to ensure if there is a race between task submission and graceful shutdown, we try to drain the queue. The complexity associated with not doing that IMHO outweighs the non ideal behavior for this edge case you are describing.

kristian

comment created time in 10 hours

PullRequestReviewEvent
PullRequestReviewEvent

issue commentnetty/netty

`SingleThreadEventExecutor.shutdownGracefully` starts thread to shut it down?

@kristian is there a use case where this is causing an issue for you or you are just trying to reason about the code?

kristian

comment created time in a day

issue commentnetty/netty

Flaky test on Windows: io.netty.channel.ThreadPerChannelEventLoopGroupTest

@hyperxpro sure thing, please send a PR

hyperxpro

comment created time in a day

push eventNiteshKant/netty

Nitesh Kant

commit sha 29cae0445a7bf9bb9ee9b9f2785c47b74dfd2592

Introduce `ByteToMessageDecoderForBuffer` (#11654) __Motivation__ In order to migrate all codec incrementally to use `Buffer`, we need a version of `ByteToMessageDecoder` that uses `Buffer`. __Modification__ - Added the new version of `ByteToMessageDecoder` with a new name so that both old and new version can co-exist and we can incrementally migrate all codecs - Migrated `FixedLengthFrameDecoder` as it was simple and used in tests. __Result__ We have the basic building block to start migrating all codecs to the new `Buffer` API.

view details

push time in 2 days

push eventnetty/netty

Nitesh Kant

commit sha 29cae0445a7bf9bb9ee9b9f2785c47b74dfd2592

Introduce `ByteToMessageDecoderForBuffer` (#11654) __Motivation__ In order to migrate all codec incrementally to use `Buffer`, we need a version of `ByteToMessageDecoder` that uses `Buffer`. __Modification__ - Added the new version of `ByteToMessageDecoder` with a new name so that both old and new version can co-exist and we can incrementally migrate all codecs - Migrated `FixedLengthFrameDecoder` as it was simple and used in tests. __Result__ We have the basic building block to start migrating all codecs to the new `Buffer` API.

view details

push time in 2 days

delete branch NiteshKant/netty

delete branch : b2md

delete time in 2 days

PR merged netty/netty

Reviewers
Introduce `ByteToMessageDecoderForBuffer` netty5

Motivation

In order to migrate all codec incrementally to use Buffer, we need a version of ByteToMessageDecoder that uses Buffer.

Modification

  • Added the new version of ByteToMessageDecoder with a new name so that both old and new version can co-exist and we can incrementally migrate all codecs
  • Migrated FixedLengthFrameDecoder as it was simple and used in tests.

Result

We have the basic building block to start migrating all codecs to the new Buffer API.

+1367 -13

2 comments

5 changed files

NiteshKant

pr closed time in 2 days

pull request commentnetty/netty

Introduce `ByteToMessageDecoderForBuffer`

@chrisvest I will pull this in

NiteshKant

comment created time in 2 days

PullRequestReviewEvent

PR opened netty/netty

Add more utility methods to Buffer

Motivation

During the process of migrating codecs to use Buffer, I stumbled upon a few utility methods that may be useful to be added to Buffer

Modification

  • Added [read, write]Split() methods to split at an offset from the current reader/writer offset.
  • Added accumulate[Reader,Writer]Offset() methods to increment/decrement reader/writer offsets.
  • Added [read, write]CharSequence() methods which is used extensively in HTTP.

Result

More utility methods to be used by codecs.

+471 -5

0 comment

6 changed files

pr created time in 2 days

create barnchNiteshKant/netty

branch : buf-new-methods

created branch time in 2 days

push eventNiteshKant/netty

Norman Maurer

commit sha a39fea736d5dc1b25a570bb0f89a1fd6cc7b43f3

Fix release problems caused by the testsuite-shading module (#11640) Motivation: Last time when we tried to do a release it failed due the fact that no javadoc jar was generated. Beside this how we did configure the shading module did introduce a lot of duplication Modifications: - Generate an empty javadoc jar - Share config for shading plugin Result: No more problems during release process

view details

Francesco Nigro

commit sha 23601902abb35d681be569bd06f02e13f1c31a24

O(1) buffer next capacity computation (#11641) Motivation: Enlarging buffers approaching 4 MiB size requires n iterations Modification: Use a single instruction to compute the next buffer capacity Result: Faster/Simpler calculateNewCapacity

view details

Chris Vest

commit sha 0cb4cc4e498afe8a44aede3f5bec10ec477a4536

Make Promise not extend Future (#11634) Motivation: We wish to separate these two into clearer write/read interfaces. In particular, we don't want to be able to add listeners to promises, because it makes it easy to add them out of order. We can't prevent it entirely, because any promise can be freely converted to a future where listeners can be added. We can, however, discourage this in the API. Modification: The Promise interface no longer extends the Future interface. Numerous changes to make the project compile and its tests run. Result: Clearer separation of concerns in the code.

view details

Norman Maurer

commit sha cba3b4dd57824039bae82ec064fdbf276c26d7cf

Change visibility of DefaultPromise to push users to use EventExecutor methods (#11646) Motivation: While we use DefaultPromise as our implementation of Promise and Future users should not really use it directly. Users should always use the EventExecutor / EventLoop to create a Promise / Future. Modifications: - Change static Promise methods to be package-private - Add default implementations for Promise and Future creation to EventExecutor - Change public constructor to protected - Remove usage of DefaultPromise in our tests Result: Less likely users will depend on the actual Promise implementation

view details

Idel Pivnitskiy

commit sha 6a2b9a76fe3f79cee679c5e44479974cdf856c59

Always log Http2UnknownStreamError at `DEBUG` level (#11643) Motivation: Exception logged by `onHttp2UnknownStreamError` is propagated to the upper layers anyway. Receiver of the exception is responsible for correct handling. Inside netty, it's enough to log at `DEBUG` level. Modifications: - `Http2FrameCodec#onHttp2UnknownStreamError` always logs at `DEBUG` level; Result: Less noise in logs when `Http2UnknownStreamError` is properly handled by upper layers. Co-authored-by: Norman Maurer <norman_maurer@apple.com> Co-authored-by: Chris Vest <mr.chrisvest@gmail.com>

view details

skyguard1

commit sha 6339b240d4e797adc275d0482d6e1159c8d245bc

Remove useless imports and redundant type cast (#11639) Motivation: There are some redundant imports and unnecessary type cast Modification: Remove useless imports and unnecessary type cast Result: The code is cleaner than original Signed-off-by: xingrufei <xingrufei@sogou-inc.com>

view details

Violeta Georgieva

commit sha ced712d4f7d9d7ab0d1a84ee0e65e8d6a590f30f

Ensure HttpData#addContent/setContent releases the buffer before throwing IOException (#11621) Motivation: When the ByteBuf size exceeds the max limit/defined size, IOException is thrown. HttpData#addContent/setContent should release the buffer in such cases otherwise memory leak will happen. Modification: - Release the provided ByteBuf before throwing IOException - Add unit tests Result: Fixes #11618

view details

Idel Pivnitskiy

commit sha 3a1a3de4e9f985033bb063873187894a443a6f0c

Include number of maximum active streams in exception message (#11644) Motivation: When users receive "Maximum active streams violated for this endpoint" exception, it's useful to know what is the current max streams limit on HTTP/2 connection. Modifications: - Include current number of maximum active streams in exception message; Result: Easier debugging of HTTP/2 connections.

view details

Norman Maurer

commit sha 06bc52f59eb943a7097a6917933bfb77d4d998ac

Remove UnaryPromiseNotifier and just use Future.cascadeTo(...) (#11650)

view details

Norman Maurer

commit sha e97cb12b2455ab0b1b0ff09f25c370d18ed57a1b

Remove deprecated EventLoopGroups (#11648) Motivation: The way how an EventLoopGroup is created for a specific transport has changed. We should remove all the old deprecated implementations and change all our code to use the new way how to init groups Modifications: - Remove LocalEventLoopGroup, NioEventLoopGroup, EpollEventLoopGroup and KQueueEventLoopGroup - Adjust code to use the new way how to setup EventLoopGroups Result: Remove deprecate classes and usages

view details

Nitesh Kant

commit sha 683ff4230e2a68b6f50c944d33a981a6d2ec50e8

Add `Channel#bufferAllocator()` (#11651) __Motivation__ As we start to migrate codecs to use the new `Buffer` API, we need a way for them to get a handle of `BufferAllocator`. __Modification__ Added `bufferAllocator()` method to `ChannelConfig`, `Channel` and `ChannelHandlerContext` __Result__ Codecs can allocate `Buffer` instances

view details

Norman Maurer

commit sha ee54ea725ad0b3afa7c7bea35cc1bf77a63234de

Add default methods to EventExecutor / EventExecutorGroup / EventLoop / EventLoopGroup (#11649) Motivation: We can remove some classes and duplication if we add default methods Modifications: - Add default methods to EventExecutor / EventExecutorGroup / EventLoop / EventLoopGroup - Remove code duplication - Remove AbstractEventExecutorGroup as it is not needed anymore Result: Cleanup and removal of code-duplication. Also makes it easier for people to implement their custom executors / groups

view details

kushalagrawal

commit sha 3a23094b814f75a37505e5926be890ef6c91fa43

Updated "CipherSuitesConverter" to make it public. (#11656) Motivation: While using netty there is sometimes need to handle the cipher suites and signature algorithm in more strict environment. CipherSuiteConverter is quite helpful in converting Cipher-Suites to and from java to openSSL. Making it public would be helpful for the project which are using netty. Modification: Updated "CipherSuitesConverter" to make it public. updated "CipherSuitesConverter.toOpenssl" to public. updated "CipherSuitesConverter.toJava" to public. Result: Fixes #11655

view details

Chris Vest

commit sha 3cbb41a4781375b1a714f2d4b1b6e91be00db4b4

Make ByteCursor implementations static final inner classes (#11662) Motivation: People might be tempted to use mocking tools like Mockito.spy() on the ByteCursors. By returning instances where the concrete classes are final, we will be forcing integrators to use stub-like wrappers instead. Such stubs are more well-behaved since they are implemented in terms of the real instance. This prevents the mocked objects from (easily) producing behaviour that violates the API specification. Modification: All ByteCursor implementations have changed from using anonymous inner classes, to using static-final named inner classes. Result: The concrete ByteCursor classes can no longer be extended via byte code generation, such as from mocking tools.

view details

Chris Vest

commit sha 59275fba525ceaf7a493c8115e20d69ee97272b1

Netty Future no longer extends JDK Future (#11647) Motivation: It is important to avoid blocking method calls in an event loop thread, since that can stall the system. Netty's Future interface was extending the JDK Future interface, which included a number of blocking methods of questionable use in Netty. We wish to reduce the number of blocking methods on the Future API in order to discourage their use a little. Further more, the Netty Future specification of the behaviour of the cancel() and isDone() methods are inconsistent with those of the JDK Future. If Netty's Future stop extending the JDK Future interface, it will also no longer be bound by its specification. Modification: Make Netty's Future no longer extend the JDK Future interface. Change the EvenExecutorGroup interface to no longer extend ScheduledExecutorService. The EventExecutorGroup still extends Executor, because Executor does not dictate any return type of the `execute()` method — this is also useful in the DefaultFutureCompletionStage implementation. The Netty ScheduledFuture interface has been removed since it provided no additional features that were actually used. Numerous changes to use sites that previously relied on the JDK types. Remove the `Future.cancel()` method that took a boolean argument — this argument was always ignored in our implementations, which was another spec deviation. Various `invoke*` and `shutdown*` methods have been removed from the EvenExecutorGroup API since it no longer extends ScheduledExecutorService — these were either not used anywhere, or deprecated with better alternatives available. Updates to cancellation javadocs. Result: Cleaner code, leaner API.

view details

Ran

commit sha 3152ec76db0a4e0818f3e2bb7f1941c49c5a09dd

Throw exceptions when rule violating UDS paths been passed in. (#11663) Motivation: Currently, Netty is silently truncating all over the limit UDS paths and ignoring the `sun_path`'s null-termination role, which hurts compatibility with other UDS clients and servers. Modifications: Adding a validation in the JNI code, if the UDS path is not satisfying the system limit or Linux spec throw a NativeIoException. Result: All UDS paths Netty can successfully bind are connectable by other programs.

view details

Chris Vest

commit sha 1eb9a9764eca54c7c31e16729b5a91e69aa6e4e7

Buffer should not expose nativeAddress() directly (#11665) Motivation: Accessing the native address, if a buffer has any, violates the no-aliasing rule for Buffers. Also, it inherently assumes that a buffer has a single, native memory allocation. The native address, or addresses, are already available via the Readable- and WritableComponents. Accessing these via forEachReadable and forEachWritable will also ensure that composite buffers will be handled correctly. Modification: Remove the nativeAddress() method from the Buffer API. Update the ByteBufAdaptor and a few tests to cope with this change. Result: Less error prone code, and make unsafe APIs a bit more hidden.

view details

Norman Maurer

commit sha dcf1e12556c73cea9cff1219feebf88ee25f6a85

Add support for mac m1 (#11666) Motivation: As more and more people switch to a mac m1 we should support it Modifications: - Add profiles for cross-compile for mac m1 - Adjust script to finish release Result: Mac m1 is supported

view details

Norman Maurer

commit sha ebd3f8b4fb503ff3a767d6d7f6b6115693fe767d

Respect jdk.tls.namedGroups when using native SSL implementation (#11660) Motivation: When using the JDK implementation for SSL its possible to adjust the used named groups. We should allow to do this as well and also select some default groups that will reduce the number of roundtrips. Modifications: - Upgrade netty-tcnative so we can set the curves - Respect jdk.tls.namedGroups - Use default groups of "P-256", "P-384", "X25519" so its compatible with what the JDK versions < 13 support as well. Result: Be able to set the used groups Co-authored-by: Nitesh Kant <nitesh_kant@apple.com>

view details

Norman Maurer

commit sha f2cc94c7d4aeb77c1d4ada151dfab00608ab283a

Merge pull request from GHSA-grg4-wf29-r9vv Motivation: We should do the Bzip2 decoding in a streaming fashion and so ensure we propagate the buffer as soon as possible through the pipeline. This allows the users to release these buffers as fast as possible. Modification: - Change the Bzip2Decoder to do the decompression of data in a streaming fashion. - Add some safety check to ensure the block length never execeeds the maximum (as defined in the spec) Result: No more risk of an OOME by decompress some large data via bzip2. Thanks to Ori Hollander of JFrog Security for reporting the issue. (we got acquired during the process and now Vdoo is part of JFrog company)

view details

push time in 2 days

push eventNiteshKant/netty

NiteshKant

commit sha 48acfdf338f89658418fcd57a58f7a3b04225c95

Review comments

view details

push time in 2 days

Pull request review commentnetty/netty

Introduce `ByteToMessageDecoderForBuffer`

 protected void decodeLast(ChannelHandlerContext ctx, Buffer in) throws Exception         }     } -    private static Buffer expandCumulation(BufferAllocator alloc, Buffer oldCumulation, Buffer in) {-        Buffer newCumulation =-                alloc.allocate(safeFindNextPositivePowerOfTwo(oldCumulation.readableBytes() + in.readableBytes()));-        Buffer toRelease = newCumulation;+    private static Buffer expandCumulationAndWrite(BufferAllocator alloc, Buffer oldCumulation, Buffer in) {+        final int newSize = safeFindNextPositivePowerOfTwo(oldCumulation.readableBytes() + in.readableBytes());+        Buffer newCumulation = oldCumulation.readOnly() ? alloc.allocate(newSize) :+                oldCumulation.ensureWritable(newSize, in.readableBytes(), true);

Let me just use the default ensureWritable for now.

NiteshKant

comment created time in 2 days

PullRequestReviewEvent
PullRequestReviewEvent

PR merged netty/netty-incubator-codec-http3

Reviewers
Log cause of failed attempt to open control stream

Motivation

I encountered the error Unable to open control stream due to the limit on the number of streams on the client side. I want to clarify the reason for the inability to open a control stream in logs.

Modification

Add cause of failed attempt to log message.

Result

Error looks like:

io.netty.incubator.codec.http3.Http3Exception: Unable to open control stream, cause: io.netty.incubator.codec.quic.QuicException: QUICHE_ERR_STREAM_LIMIT
	at io.netty.incubator.codec.http3.Http3ConnectionHandler.lambda$createControlStreamIfNeeded$0(Http3ConnectionHandler.java:91)
...
io.netty.incubator.codec.http3.Http3ConnectionHandler.createControlStreamIfNeeded(Http3ConnectionHandler.java:88)
	at io.netty.incubator.codec.http3.Http3ConnectionHandler.channelActive(Http3ConnectionHandler.java:152)
+1 -1

1 comment

1 changed file

abezhovets

pr closed time in 3 days

push eventnetty/netty-incubator-codec-http3

abezhovets

commit sha db173a2f0cfbdfe539f88ee71c27512d16b2d31a

Log cause of failed attempt to open control stream (#167) **Motivation** I encountered the error `Unable to open control stream` due to the limit on the number of streams on the client side. I want to clarify the reason for the inability to open a control stream in logs. **Modification** Add cause of failed attempt to log message. **Result** Error looks like: ``` io.netty.incubator.codec.http3.Http3Exception: Unable to open control stream, cause: io.netty.incubator.codec.quic.QuicException: QUICHE_ERR_STREAM_LIMIT at io.netty.incubator.codec.http3.Http3ConnectionHandler.lambda$createControlStreamIfNeeded$0(Http3ConnectionHandler.java:91) ... io.netty.incubator.codec.http3.Http3ConnectionHandler.createControlStreamIfNeeded(Http3ConnectionHandler.java:88) at io.netty.incubator.codec.http3.Http3ConnectionHandler.channelActive(Http3ConnectionHandler.java:152) ```

view details

push time in 3 days

pull request commentnetty/netty-incubator-codec-http3

Log cause of failed attempt to open control stream

Thanks a lot @abezhovets

abezhovets

comment created time in 3 days

PR merged netty/netty-incubator-codec-http3

Separate the processing of the QuicException and Http3Exception error…

Motivation

Change the Http3RequestStreamInboundHandler such that by default it does not propagate the QuicException and Http3Exception to the pipeline further and reduce noisy logs.

Modification

Separate the processing of these errors into separate methods and simply log by default.

Result

Fixes #164

+37 -0

1 comment

1 changed file

abezhovets

pr closed time in 3 days

push eventnetty/netty-incubator-codec-http3

abezhovets

commit sha 96270a66e110c84d7a0d01d964a7b530274ed407

Separate the processing of the QuicException and Http3Exception error… (#166) Motivation Change the Http3RequestStreamInboundHandler such that by default it does not propagate the QuicException and Http3Exception to the pipeline further and reduce noisy logs. Modification Separate the processing of these errors into separate methods and simply log by default. Result Fixes #164

view details

push time in 3 days

issue closednetty/netty-incubator-codec-http3

`QuicException: STREAM_RESET` in `exceptionCaught()`

If the client sends RESET_STREAM, then the STREAM_RESET will reach exceptionCaught() as a QuicException. Is this an error or valid behavior? Is it correct that STREAM_RESET reaches the custom logic handler?

Code to reproduce: <details><summary>ResetStreamTest</summary> <p>

package io.netty.incubator.codec.http3;

import io.netty.bootstrap.Bootstrap;
import io.netty.channel.*;
import io.netty.channel.socket.nio.NioDatagramChannel;
import io.netty.handler.logging.ByteBufFormat;
import io.netty.handler.logging.LogLevel;
import io.netty.handler.logging.LoggingHandler;
import io.netty.handler.ssl.util.InsecureTrustManagerFactory;
import io.netty.incubator.codec.quic.QuicChannel;
import io.netty.incubator.codec.quic.QuicSslContext;
import io.netty.incubator.codec.quic.QuicSslContextBuilder;
import io.netty.incubator.codec.quic.QuicStreamChannel;
import io.netty.util.NetUtil;
import lombok.extern.log4j.Log4j2;
import org.junit.Test;

import java.net.InetSocketAddress;
import java.util.concurrent.TimeUnit;

@Log4j2
public class ResetStreamTest extends BaseTest {

    @Test
    public void testUpload() throws Exception {

        QuicSslContext context = QuicSslContextBuilder.forClient()
                .trustManager(InsecureTrustManagerFactory.INSTANCE)
                .applicationProtocols(Http3.supportedApplicationProtocols()).build();
        ChannelHandler codec = Http3.newQuicClientCodecBuilder()
                .sslContext(context)
                .initialMaxStreamsBidirectional(100)
                .maxIdleTimeout(5000, TimeUnit.MILLISECONDS)
                .initialMaxData(10000000)
                .initialMaxStreamDataBidirectionalLocal(1000000)
                .build();

        Channel channel = new Bootstrap()
                .group(GROUP)
                .channel(NioDatagramChannel.class)
                .option(ChannelOption.SO_REUSEADDR, true)
                .handler(codec)
                .bind(new InetSocketAddress(0))
                .sync().channel();

        QuicChannel quicChannel = QuicChannel.newBootstrap(channel)
                .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 10_000)
                .remoteAddress(new InetSocketAddress(NetUtil.LOCALHOST4, PORT))
                .handler(new ChannelInitializer<>() {
                    @Override
                    protected void initChannel(Channel ch) {
                        ch.pipeline().addLast(new Http3ClientConnectionHandler());
                    }
                })
                .connect()
                .sync().get();

        QuicStreamChannel streamChannel = Http3.newRequestStream(quicChannel, new ChannelInitializer<>() {
            @Override
            protected void initChannel(Channel ch) {
                ch.pipeline()
                        .addLast(new LoggingHandler(LogLevel.DEBUG, ByteBufFormat.SIMPLE))
                        .addLast(new Http3RequestStreamInboundHandler() {

                            @Override
                            protected void channelRead(ChannelHandlerContext ctx, Http3HeadersFrame frame, boolean isLast) throws Exception {
                                log.info("channelRead {}, {}", frame, isLast);
                            }

                            @Override
                            protected void channelRead(ChannelHandlerContext ctx, Http3DataFrame frame, boolean isLast) throws Exception {
                                log.info("channelRead {}, {}", frame, isLast);
                            }
                        });
            }
        }).sync().getNow();

        streamChannel.shutdownOutput(Http3ErrorCode.H3_INTERNAL_ERROR.code);

        TimeUnit.SECONDS.sleep(1);
    }
}

</p> </details>

<details><summary>BaseTest</summary> <p>

package io.netty.incubator.codec.http3;

import io.netty.bootstrap.Bootstrap;
import io.netty.channel.Channel;
import io.netty.channel.ChannelHandler;
import io.netty.channel.ChannelHandlerContext;
import io.netty.channel.nio.NioEventLoopGroup;
import io.netty.channel.socket.nio.NioDatagramChannel;
import io.netty.handler.ssl.util.SelfSignedCertificate;
import io.netty.incubator.codec.quic.InsecureQuicTokenHandler;
import io.netty.incubator.codec.quic.QuicSslContext;
import io.netty.incubator.codec.quic.QuicSslContextBuilder;
import org.junit.After;
import org.junit.Before;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.net.InetSocketAddress;
import java.util.concurrent.TimeUnit;

public abstract class BaseTest {

    protected static final int PORT = 9999;
    protected static final NioEventLoopGroup GROUP = new NioEventLoopGroup(1);

    protected final Logger log = LoggerFactory.getLogger(this.getClass());

    @Before
    public void setUp() throws Exception {
        SelfSignedCertificate cert = new SelfSignedCertificate();
        QuicSslContext sslContext = QuicSslContextBuilder.forServer(cert.key(), null, cert.cert())
                .applicationProtocols(Http3.supportedApplicationProtocols()).build();

        ChannelHandler codec = Http3.newQuicServerCodecBuilder()
                .sslContext(sslContext)
                .maxIdleTimeout(5000, TimeUnit.MILLISECONDS)
                .initialMaxData(10000000)
                .initialMaxStreamDataBidirectionalLocal(1000000)
                .initialMaxStreamDataBidirectionalRemote(1000000)
                .initialMaxStreamsBidirectional(100)
                .tokenHandler(InsecureQuicTokenHandler.INSTANCE)
                .handler(new Http3ServerConnectionHandler(new Http3RequestStreamInboundHandler() {

                    @Override
                    protected void channelRead(ChannelHandlerContext ctx, Http3HeadersFrame frame, boolean isLast) throws Exception {
                        log.info("channelRead {}, {}", frame, isLast);
                    }

                    @Override
                    protected void channelRead(ChannelHandlerContext ctx, Http3DataFrame frame, boolean isLast) throws Exception {
                        log.info("channelRead {}, {}", frame, isLast);
                    }
                }))
                .build();

        Bootstrap bs = new Bootstrap();
        Channel serverChannel = bs.group(GROUP)
                .channel(NioDatagramChannel.class)
                .handler(codec)
                .bind(new InetSocketAddress(PORT)).sync().channel();
    }

    @After
    public void cleanUp() {
        GROUP.shutdownGracefully();
    }
}

</p> </details>

<details><summary>Stacktrace</summary> <p>

[nioEventLoopGroup-2-1] WARN channel.DefaultChannelPipeline: An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
io.netty.incubator.codec.quic.QuicException: STREAM_RESET
	at io.netty.incubator.codec.quic.Quiche.newException(Quiche.java:645) ~[netty-incubator-codec-quic-0.0.17.Final-osx-x86_64.jar:0.0.17.Final]
	at io.netty.incubator.codec.quic.Quiche.throwIfError(Quiche.java:669) ~[netty-incubator-codec-quic-0.0.17.Final-osx-x86_64.jar:0.0.17.Final]
	at io.netty.incubator.codec.quic.QuicheQuicChannel.streamRecv(QuicheQuicChannel.java:765) ~[netty-incubator-codec-quic-0.0.17.Final-osx-x86_64.jar:0.0.17.Final]
	at io.netty.incubator.codec.quic.QuicheQuicStreamChannel$QuicStreamChannelUnsafe.recv(QuicheQuicStreamChannel.java:860) ~[netty-incubator-codec-quic-0.0.17.Final-osx-x86_64.jar:0.0.17.Final]
	at io.netty.incubator.codec.quic.QuicheQuicStreamChannel$QuicStreamChannelUnsafe.beginRead(QuicheQuicStreamChannel.java:602) ~[netty-incubator-codec-quic-0.0.17.Final-osx-x86_64.jar:0.0.17.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.read(DefaultChannelPipeline.java:1362) ~[netty-all-4.1.67.Final.jar:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeRead(AbstractChannelHandlerContext.java:686) ~[netty-all-4.1.67.Final.jar:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.read(AbstractChannelHandlerContext.java:671) ~[netty-all-4.1.67.Final.jar:4.1.67.Final]
	at io.netty.incubator.codec.http3.Http3FrameCodec.read(Http3FrameCodec.java:625) ~[netty-incubator-codec-http3-0.0.8.Final.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeRead(AbstractChannelHandlerContext.java:686) ~[netty-all-4.1.67.Final.jar:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.read(AbstractChannelHandlerContext.java:671) ~[netty-all-4.1.67.Final.jar:4.1.67.Final]
	at io.netty.incubator.codec.http3.Http3FrameTypeDuplexValidationHandler.read(Http3FrameTypeDuplexValidationHandler.java:85) ~[netty-incubator-codec-http3-0.0.8.Final.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeRead(AbstractChannelHandlerContext.java:686) ~[netty-all-4.1.67.Final.jar:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.read(AbstractChannelHandlerContext.java:671) ~[netty-all-4.1.67.Final.jar:4.1.67.Final]
	at io.netty.channel.DefaultChannelPipeline.read(DefaultChannelPipeline.java:1004) ~[netty-all-4.1.67.Final.jar:4.1.67.Final]
	at io.netty.channel.DefaultChannelPipeline.read(DefaultChannelPipeline.java:46) ~[netty-all-4.1.67.Final.jar:4.1.67.Final]
	at io.netty.incubator.codec.quic.QuicheQuicStreamChannel.read(QuicheQuicStreamChannel.java:287) ~[netty-incubator-codec-quic-0.0.17.Final-osx-x86_64.jar:0.0.17.Final]
	at io.netty.incubator.codec.quic.QuicheQuicStreamChannel.read(QuicheQuicStreamChannel.java:51) ~[netty-incubator-codec-quic-0.0.17.Final-osx-x86_64.jar:0.0.17.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.readIfIsAutoRead(DefaultChannelPipeline.java:1422) ~[netty-all-4.1.67.Final.jar:4.1.67.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelActive(DefaultChannelPipeline.java:1400) ~[netty-all-4.1.67.Final.jar:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:230) ~[netty-all-4.1.67.Final.jar:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:216) ~[netty-all-4.1.67.Final.jar:4.1.67.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelActive(DefaultChannelPipeline.java:895) ~[netty-all-4.1.67.Final.jar:4.1.67.Final]
	at io.netty.incubator.codec.quic.QuicheQuicStreamChannel$QuicStreamChannelUnsafe.register(QuicheQuicStreamChannel.java:474) ~[netty-incubator-codec-quic-0.0.17.Final-osx-x86_64.jar:0.0.17.Final]
	at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:87) ~[netty-all-4.1.67.Final.jar:4.1.67.Final]
	at io.netty.channel.SingleThreadEventLoop.register(SingleThreadEventLoop.java:81) ~[netty-all-4.1.67.Final.jar:4.1.67.Final]
	at io.netty.incubator.codec.quic.QuicheQuicChannel$1.onUnhandledInboundMessage(QuicheQuicChannel.java:360) ~[netty-incubator-codec-quic-0.0.17.Final-osx-x86_64.jar:0.0.17.Final]
	at io.netty.channel.DefaultChannelPipeline$TailContext.channelRead(DefaultChannelPipeline.java:1296) ~[netty-all-4.1.67.Final.jar:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-all-4.1.67.Final.jar:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-all-4.1.67.Final.jar:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-all-4.1.67.Final.jar:4.1.67.Final]
	at io.netty.incubator.codec.http3.Http3ConnectionHandler.channelRead(Http3ConnectionHandler.java:172) ~[netty-incubator-codec-http3-0.0.8.Final.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-all-4.1.67.Final.jar:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-all-4.1.67.Final.jar:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[netty-all-4.1.67.Final.jar:4.1.67.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-all-4.1.67.Final.jar:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[netty-all-4.1.67.Final.jar:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[netty-all-4.1.67.Final.jar:4.1.67.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-all-4.1.67.Final.jar:4.1.67.Final]
	at io.netty.incubator.codec.quic.QuicheQuicChannel$QuicChannelUnsafe.recvStream(QuicheQuicChannel.java:1298) ~[netty-incubator-codec-quic-0.0.17.Final-osx-x86_64.jar:0.0.17.Final]
	at io.netty.incubator.codec.quic.QuicheQuicChannel$QuicChannelUnsafe.connectionRecv(QuicheQuicChannel.java:1237) ~[netty-incubator-codec-quic-0.0.17.Final-osx-x86_64.jar:0.0.17.Final]
	at io.netty.incubator.codec.quic.QuicheQuicChannel.recv(QuicheQuicChannel.java:777) ~[netty-incubator-codec-quic-0.0.17.Final-osx-x86_64.jar:0.0.17.Final]
	at io.netty.incubator.codec.quic.QuicheQuicCodec.lambda$handlerAdded$0(QuicheQuicCodec.java:83) ~[netty-incubator-codec-quic-0.0.17.Final-osx-x86_64.jar:0.0.17.Final]
	at io.netty.incubator.codec.quic.QuicHeaderParser.parse(QuicHeaderParser.java:123) [netty-incubator-codec-quic-0.0.17.Final-osx-x86_64.jar:0.0.17.Final]
	at io.netty.incubator.codec.quic.QuicheQuicCodec.handleQuicPacket(QuicheQuicCodec.java:140) [netty-incubator-codec-quic-0.0.17.Final-osx-x86_64.jar:0.0.17.Final]
	at io.netty.incubator.codec.quic.QuicheQuicCodec.channelRead(QuicheQuicCodec.java:131) [netty-incubator-codec-quic-0.0.17.Final-osx-x86_64.jar:0.0.17.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.67.Final.jar:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.67.Final.jar:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-all-4.1.67.Final.jar:4.1.67.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-all-4.1.67.Final.jar:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-all-4.1.67.Final.jar:4.1.67.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-all-4.1.67.Final.jar:4.1.67.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-all-4.1.67.Final.jar:4.1.67.Final]
	at io.netty.channel.nio.AbstractNioMessageChannel$NioMessageUnsafe.read(AbstractNioMessageChannel.java:97) [netty-all-4.1.67.Final.jar:4.1.67.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:719) [netty-all-4.1.67.Final.jar:4.1.67.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:655) [netty-all-4.1.67.Final.jar:4.1.67.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:581) [netty-all-4.1.67.Final.jar:4.1.67.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-all-4.1.67.Final.jar:4.1.67.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-all-4.1.67.Final.jar:4.1.67.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.67.Final.jar:4.1.67.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.67.Final.jar:4.1.67.Final]
	at java.lang.Thread.run(Thread.java:832) [?:?]

</p> </details>

closed time in 3 days

abezhovets