Skip to content

Max segment size #110

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Apr 28, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -196,6 +196,18 @@ await system.CreateStream(new StreamSpec(stream)

Set a policy is highly recommended.

RabbitMQ does not store the whole stream in a single file, but splits it in segment files.
This is also used for truncate the stream: when a stream reaches his maximum size, an entire segment file is deleted. For this reason `MaxLengthBytes` (the max dimension of the entire stream) is usually significantly higher than `MaxSegmentSizeBytes` (the max dimension of a single segment file).
RabbitMQ enforces the retention policy when the current segment has reached its maximum size and is closed in favor of a new one.

```csharp
await system.CreateStream(new StreamSpec(stream)
{
MaxLengthBytes = 20_000,
MaxSegmentSizeBytes = 1000
});
```

### Producer

A Producer instance is created from the `System`.
Expand Down
4 changes: 4 additions & 0 deletions RabbitMQ.Stream.Client/StreamSpec.cs
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,10 @@ public LeaderLocator LeaderLocator
{
set => Args["queue-leader-locator"] = $"{value.ToString()}";
}
public int MaxSegmentSizeBytes
{
set => Args["stream-max-segment-size-bytes"] = $"{value}";
}

public IDictionary<string, string> Args => args;
}
Expand Down
1 change: 1 addition & 0 deletions Tests/ApiApproval.Approve.verified.txt
Original file line number Diff line number Diff line change
Expand Up @@ -861,6 +861,7 @@ namespace RabbitMQ.Stream.Client
public RabbitMQ.Stream.Client.LeaderLocator LeaderLocator { set; }
public System.TimeSpan MaxAge { set; }
public int MaxLengthBytes { set; }
public int MaxSegmentSizeBytes { set; }
public string Name { get; set; }
public virtual RabbitMQ.Stream.Client.StreamSpec <Clone>$() { }
public void Deconstruct(out string Name) { }
Expand Down
10 changes: 9 additions & 1 deletion Tests/SystemTests.cs
Original file line number Diff line number Diff line change
Expand Up @@ -80,9 +80,17 @@ public async void Create_Delete_Stream()
var stream = Guid.NewGuid().ToString();
var config = new StreamSystemConfig();
var system = await StreamSystem.Create(config);
var spec = new StreamSpec(stream) { MaxAge = TimeSpan.FromHours(8), LeaderLocator = LeaderLocator.Random };
var spec = new StreamSpec(stream)
{
MaxAge = TimeSpan.FromHours(8),
LeaderLocator = LeaderLocator.Random,
MaxLengthBytes = 20_000,
MaxSegmentSizeBytes = 1000
};
Assert.Equal("28800s", spec.Args["max-age"]);
Assert.Equal("random", spec.Args["queue-leader-locator"]);
Assert.Equal("1000", spec.Args["stream-max-segment-size-bytes"]);
Assert.Equal("20000", spec.Args["max-length-bytes"]);
await system.CreateStream(spec);
await system.DeleteStream(stream);
await system.Close();
Expand Down