chore: add high availability etcd docker compose #1477
Conversation
WalkthroughThe existing single-node etcd service in the Docker Compose configuration has been replaced with a three-node etcd cluster. Each node is configured with specific environment variables for cluster membership and communication, and unique host port mappings are assigned to avoid conflicts. Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant etcd0
participant etcd1
participant etcd2
User->>etcd0: Connect via client port 2379
User->>etcd1: Connect via client port 23791
User->>etcd2: Connect via client port 23792
etcd0-->>etcd1: Peer communication (2380/23801)
etcd0-->>etcd2: Peer communication (2380/23802)
etcd1-->>etcd0: Peer communication (23801/2380)
etcd1-->>etcd2: Peer communication (23801/23802)
etcd2-->>etcd0: Peer communication (23802/2380)
etcd2-->>etcd1: Peer communication (23802/23801)
Poem
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
Documentation and Community
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (1)
lib/runtime/docker-compose.yml (1)
57-71: Apply same client advertise URL fix to etcd2The recommendation for
etcd1also applies here—replace0.0.0.0withetcd2in theETCD_ADVERTISE_CLIENT_URLSsetting.
🧹 Nitpick comments (1)
lib/runtime/docker-compose.yml (1)
41-56: Use service hostname in ETCD_ADVERTISE_CLIENT_URLS for etcd1Advertising
0.0.0.0is not routable by other containers. Change to the service name so peers and clients can resolve it internally:- - ETCD_ADVERTISE_CLIENT_URLS=http://0.0.0.0:2379 + - ETCD_ADVERTISE_CLIENT_URLS=http://etcd1:2379You may also want to mirror the restart policy and version-pinning changes from
etcd0here.
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
lib/runtime/docker-compose.yml(1 hunks)
⏰ Context from checks skipped due to timeout of 90000ms (4)
- GitHub Check: pre-merge-rust (lib/bindings/python)
- GitHub Check: pre-merge-rust (lib/runtime/examples)
- GitHub Check: pre-merge-rust (.)
- GitHub Check: Build and Test - vllm
There was a problem hiding this comment.
Take any of the 3 ETCD replicas down (we need 2 for consensus). Logs looks like
What happens if 2 go down? Does 1 standalone no longer work with ETCD_*_CLUSTER settings?
There was a problem hiding this comment.
If user starts all 3 with docker-compose up, but then sets only export ETCD_ENDPOINTS=localhost:2379, does that still work as well?
There was a problem hiding this comment.
Yes - this works out of the box. That's why the first endpoint is the default 2379
We now spin up 3 ETCD replicas with the first one having the default
2379port so a user does not have to change anything. The other two are also up allowing developers to test what happens when 1 goes down.If a user wants to test - they can now do
export ETCD_ENDPOINTS=localhost:2379,localhost:23791,localhost:23792 dynamo serve graphs.agg:Frontend -f configs/agg.yamlTake any of the 3 ETCD replicas down (we need 2 for consensus). Logs looks like
But lease won't be canceled taking everything down
Summary by CodeRabbit