Conversation
b03473a to
1ceee04
Compare
1ceee04 to
b1dd5a6
Compare
This comment has been minimized.
This comment has been minimized.
b1dd5a6 to
da212d3
Compare
da212d3 to
dbf0974
Compare
This comment has been minimized.
This comment has been minimized.
Smoke Tests on k8sVerify the failure: bulk load -> k8s deploy reproduces the stale address bugRan a standalone Zero locally with --my=localhost:5080, bulk loaded the 21M movie dataset (3 shards), then injected the Zero WAL and Alpha p directories into a fresh Docker Desktop k8s cluster (1 Zero, 3 Alphas, sharded config, dgraph/dgraph:latest v25.3.3). When the k8s Zero started, WAL replay overwrote its live --my address with localhost:5080 — confirmed via /state. All three Alphas got stuck in a connection-refused loop trying to dial localhost:5080 inside their containers and never became Ready. Verified fix works: bulk load -> k8s deploy with dgraph/dgraph:local (built on shiva/fqdn)Rebuilt dgraph/dgraph:local from the PR branch (shiva/fqdn), redeployed the k8s cluster, and injected the same stale Zero WAL (localhost:5080) and bulk-loaded Alpha data from the bug repro. On startup, Zero's reconciliation loop detected the mismatch within seconds of leader election and corrected it via Raft: All three Alphas came up Ready, connected to Zero at the corrected address, and served queries against the 21M movie dataset across all 3 shard groups. This is the same stale WAL that caused a total cluster failure on main — the fix resolves it without manual intervention. |
Description
When a Zero node is bootstrapped, its --my address is permanently recorded in the Raft WAL via a ConfChangeAddNode entry. On every subsequent restart, WAL replay restores that original address into MembershipState, overwriting the current --my flag. This causes Alphas to connect to a stale address (e.g., localhost:5080 from an initial bulk load) even when Zero is restarted with a production FQDN.
This PR introduces a leader-driven reconciliation loop that detects mismatches between the live --my address and what is stored in MembershipState, and proposes a ConfChangeUpdateNode through Raft to durably correct the address across all nodes. An in-memory override in the Connect RPC ensures Alphas receive the correct address immediately, even before the Raft entry is committed.
fixes #9676