Handling “split-brain” in a VDB cluster
Split-brain occurs when a network partition (or node failures) divides the cluster into two or more groups that cannot communicate. If each group believes it is the primary and accepts writes, the same logical data can diverge in both partitions, leading to conflicting updates and data loss when the partition heals.
Summary
- Split-brain: a network partition (or node failures) divides the cluster so each group cannot communicate; if each believes it is primary and accepts writes, data can diverge and conflict when the partition heals.
- Prevention: consensus and quorum—only a majority (or primary elected via Raft/Paxos) can accept writes. A partition without a majority must not accept writes; at most one partition makes progress. Trade-off: minority side may be read-only or unavailable.
- After recovery: reconcile (discard or merge conflicting writes). Metadata often managed by consensus; vector data may use primary-replica so only primary accepts writes. See disaster recovery. Pipeline: partition occurs, minority refuses writes, on heal reconcile. Practical tip: use quorum writes and a single primary for writes; monitor partition events.
Prevention with consensus
Prevention relies on consensus and quorum: only a majority (or a designated primary elected via Raft/Paxos) can accept writes. If a partition does not have a majority, it must not accept writes even if it cannot reach the other side. That way at most one partition can make progress, avoiding split-brain. Trade-off: during a partition, the minority side may be read-only or unavailable for writes to preserve consistency.
Recovery and reconciliation
After the network recovers, nodes reconcile: the partition that was not the primary may need to discard its writes or merge via a conflict-resolution policy (e.g. last-writer-wins, or application-defined). For vector DBs, metadata (schema, shard assignment) is often managed by consensus; vector data itself may be replicated with a primary-replica model so that only the primary accepts writes, and replicas in the other partition become read-only until they rejoin. Monitoring and alerting on partition events and leader identity help operators detect and resolve split-brain quickly. Pipeline: partition occurs, minority partition refuses writes (no quorum), on network heal nodes reconcile (discard or merge). Practical tip: use quorum writes and a single primary for writes; monitor partition events and leader identity.
Frequently Asked Questions
What is split-brain?
When a network partition divides the cluster and each group cannot communicate. If each group believes it is the primary and accepts writes, the same logical data can diverge, leading to conflicting updates and data loss when the partition heals. See disaster recovery.
How do I prevent split-brain?
Use consensus and quorum: only a majority (or a primary elected via Raft/Paxos) can accept writes. A partition without a majority must not accept writes. At most one partition can make progress. Trade-off: the minority side may be read-only or unavailable.
What happens after the partition heals?
Nodes reconcile: the partition that was not the primary may need to discard its writes or merge via conflict-resolution (e.g. last-writer-wins). Metadata is often managed by consensus; vector data may use primary-replica so only the primary accepted writes and replicas in the other partition were read-only until rejoin.
How do VDBs use consensus for this?
Metadata (schema, shard assignment) is often managed by Raft/Paxos; vector data may be replicated with a primary-replica model so only the primary accepts writes. Monitoring and alerting on partition events and leader identity help detect and resolve split-brain. See leader-based vs. leaderless.