Skip to content

Fix the Unsubscribe timeout problem #313

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Sep 29, 2023
Merged

Fix the Unsubscribe timeout problem #313

merged 9 commits into from
Sep 29, 2023

Conversation

Gsantomaggio
Copy link
Member

@Gsantomaggio Gsantomaggio commented Sep 23, 2023

  • The Unsubscribe function sync (like the others). The consumers' list is updated in the right way.
    The connection is correctly closed since there are no pending consumers.

  • Fix the handle delivery if the consumer is removed, but there are still chunks on the wire.
    Add debug logs for this situation.

  • Add Degug Asserts to validate the buffer size, which can help understand some parse chunk errors.
    That can be temporary at some point; we can remove them.
    Ref Consumer timeout and pending connection  #310

  • Add the cancellation token to the connection class
    it helps when the reader is blocked and the consumer is closed.

  • Given this use case so short-life consumers, the initial credits should be low to avoid keeping the read handler busy.

  • Minor change is replace Memory to ReadOnlyMemory it is a bit faster and safe

  • Uniform the log debug and error messages with consumer info it adds all the consumer info that can be helpful in case of debugging and errors

The wait Unsubscribe now is sync so it is removed from the consumers list correctly.
The connection can be closed correctly since there are not pendig consumers.

Fix the handle deliver in case of the consumer is removed but there are still chunks on the wire.
Add Degug Asserts to validate the buffer size, it can help to understand some parse chunk error.
That can be temporany at some point we can remove them.
Ref #310

Signed-off-by: Gabriele Santomaggio <[email protected]>
Signed-off-by: Gabriele Santomaggio <[email protected]>
@Gsantomaggio
Copy link
Member Author

@jonnepmyra do you have a chance to test?
It adds more detailed logs to understand the chunk parse error

@codecov
Copy link

codecov bot commented Sep 23, 2023

Codecov Report

Attention: 40 lines in your changes are missing coverage. Please review.

Comparison is base (a2e06bf) 92.79% compared to head (355e635) 92.49%.

❗ Current head 355e635 differs from pull request most recent head 2125dcd. Consider uploading reports for the commit 2125dcd to get more accurate results

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #313      +/-   ##
==========================================
- Coverage   92.79%   92.49%   -0.31%     
==========================================
  Files         112      112              
  Lines        9804     9884      +80     
  Branches      785      825      +40     
==========================================
+ Hits         9098     9142      +44     
- Misses        536      564      +28     
- Partials      170      178       +8     
Files Coverage Δ
RabbitMQ.Stream.Client/Connection.cs 91.39% <81.25%> (+2.73%) ⬆️
RabbitMQ.Stream.Client/Deliver.cs 82.71% <0.00%> (-3.19%) ⬇️
RabbitMQ.Stream.Client/Client.cs 88.16% <56.00%> (-2.35%) ⬇️
RabbitMQ.Stream.Client/RawConsumer.cs 84.33% <72.61%> (-2.10%) ⬇️

... and 2 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@jonnepmyra
Copy link
Contributor

@jonnepmyra do you have a chance to test? It adds more detailed logs to understand the chunk parse error

Thanks! I'm on it!

Zerpet
Zerpet previously approved these changes Sep 25, 2023
replace Memory to ReadOnlyMemory on the chunk data

Signed-off-by: Gabriele Santomaggio <[email protected]>
task cancellation on the connection class

Signed-off-by: Gabriele Santomaggio <[email protected]>
in the rawconsumer. It gives all the information about the consumer. It is helpful in case of debugging

Signed-off-by: Gabriele Santomaggio <[email protected]>
@Gsantomaggio Gsantomaggio marked this pull request as ready for review September 28, 2023 13:08
@jonnepmyra
Copy link
Contributor

I've been thoroughly testing this pull request over the past few days, and it appears to be a clear improvement.

Before this fix, we encountered the following errors during high-throughput scenarios with short-lived consumers. However, after applying this fix, these issues do not seem to occur anymore:

Error while processing chunk: XYZ

System.NullReferenceException: Object reference not set to an instance of an object.
   at RabbitMQ.Stream.Client.RawConsumer.<>c__DisplayClass14_0.<<ParseChunk>g__DispatchMessage|1>d.MoveNext() in C:\git\rabbitmq-streamclient\rabbitmq-stream-dotnet-client\RabbitMQ.Stream.Client\RawConsumer.cs:line 201

An error occurred while calling Dispose

System.ArgumentNullException: Value cannot be null. (Parameter 'array')
   at System.Buffers.TlsOverPerCoreLockedStacksArrayPool`1.Return(T[] array, Boolean clearArray)
   at System.IO.Pipelines.BufferSegment.ResetMemory()
   at System.IO.Pipelines.BufferSegment.Reset()
   at System.IO.Pipelines.StreamPipeReader.CompleteAndGetNeedsDispose()
   at System.IO.Pipelines.StreamPipeReader.Complete(Exception exception)
   at RabbitMQ.Stream.Client.Connection.Dispose() in C:\git\rabbitmq-streamclient\rabbitmq-stream-dotnet-client\RabbitMQ.Stream.Client\Connection.cs:line 223
   at RabbitMQ.Stream.Client.Client.Close(String reason) in C:\git\rabbitmq-streamclient\rabbitmq-stream-dotnet-client\RabbitMQ.Stream.Client\Client.cs:line 583

While we are still experiencing some Close() timeouts with short-lived stream consumers, it's worth noting that this may be related to a non-default InitialCredits value (specifically, 100 instead of the default 2) mentioned by @Gsantomaggio. At present, we can consider this as a non-issue.

Great work and thanks for helping out!

@Gsantomaggio Gsantomaggio merged commit 0711e13 into main Sep 29, 2023
@Gsantomaggio Gsantomaggio deleted the timeout_error branch September 29, 2023 09:01
@Gsantomaggio Gsantomaggio modified the milestones: 1.7.1, 1.7.2 Sep 29, 2023
@Gsantomaggio
Copy link
Member Author

Thank you @jonnepmyra for the contribution

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants