Skip to content

Fix the read timeout implementation in NettyStream #635

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from

Conversation

stIncMale
Copy link
Member

Both the new approach and the original one achieve the guarantee that
there are no concurrent read timeouts scheduled for a channel.
This is an essential property needed for a timeout implementation,
let us call it "at-most-one".

The original approach of achieving the at-most-one property:

  • Schedule timeouts only by an event loop thread.
  • If another thread needs to schedule a timeout, it submits a new scheduleTimeout task
    to the channel's event pool (asynchronous timeout scheduling).
    This task schedules a new timeout if none is scheduled.

The original approach allowed executions in which a scheduleTimeout task runs after
completion of the read operation that submitted the task,
which resulted in unexpected timeout exceptions.

The new approach achieves the at-most-one property by using a lock.
As a result, timeouts can be scheduled by any thread and there is no
asynchronous timeout scheduling. This means we cannot miss removing a timeout
because it was submitted for scheduling, but has not been scheduled yet.

Other notable changes:

  1. ReadTimeoutHandler is now actually implemented as a handler. It reacts to Netty events,
    learns about the context and when to remove the timeout automatically
    because the channel is closed, or because the channel is no longer registered with an event loop
    (we do not use channels this way, but it is possible,
    as described here
    https://netty.io/wiki/new-and-noteworthy-in-4.0.html#deregistration-and-re-registration-of-a-channel-fromto-an-io-thread).
  2. Netty channel handlers (all except for ReadTimeoutHandler) do not try to schedule timeouts anymore.
    This was unnecessary even with the original approach.
  3. Fields NettyStream.pendingReader, pendingException are always written/read
    inside synchronized blocks that use the same NettyStream object,
    so marking them volatile is unnecessary and potentially misleading.

JAVA-3920

Both the new approach and the original one achieve the guarantee that
there are no concurrent read timeouts scheduled for a channel.
This is an essential property needed for a timeout implementation,
let us call it "at-most-one".

The original approach of achieving the at-most-one property:
- Schedule timeouts only by an event loop thread.
- If another thread needs to schedule a timeout, it submits a new scheduleTimeout task
to the channel's event pool (asynchronous timeout scheduling).
This task schedules a new timeout if none is scheduled.

The original approach allowed executions in which a scheduleTimeout task runs after
completion of the read operation that submitted the task,
which resulted in unexpected timeout exceptions.

The new approach achieves the at-most-one property by using a lock.
As a result, timeouts can be scheduled by any thread and there is no
asynchronous timeout scheduling. This means we cannot miss removing a timeout
because it was submitted for scheduling, but has not been scheduled yet.

Other notable changes:
1) ReadTimeoutHandler is now actually implemented as a handler. It reacts to Netty events,
learns about the context and when to remove the timeout automatically
because the channel is closed, or because the channel is no longer registered with an event loop
(we do not use channels this way, but it is possible,
as described here
https://netty.io/wiki/new-and-noteworthy-in-4.0.html#deregistration-and-re-registration-of-a-channel-fromto-an-io-thread).
2) Netty channel handlers (all except for ReadTimeoutHandler) do not try to schedule timeouts anymore.
This was unnecessary even with the original approach.
3) Fields NettyStream.pendingReader, pendingException are always written/read
inside synchronized blocks that use the same NettyStream object,
so marking them volatile is unnecessary and potentially misleading.

JAVA-3920
@stIncMale
Copy link
Member Author

stIncMale commented Jan 16, 2021

After discussing the MR internally, we decided not to go with the approach that preserves ReadTimeoutHandler because even if it respects the Netty channel/handler lifecycle (which adds a fair bit of code, even though the code is simple), we still have to call some of its methods explicitly from the NettyStream class in order to have timeouts. This makes the handler not pure.

The fix is done in #636.

@stIncMale stIncMale closed this Jan 16, 2021
@stIncMale stIncMale deleted the JAVA-3920 branch January 29, 2021 12:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant