I once watched a cache cluster fall over because we added one node. Not ten. One. Every key moved at once, every cache missed at once, and the database behind it went down in minutes. That night taught me the difference between a hash that moves everything and a hash that moves almost nothing. This lesson follows one key onto a ring, and shows why the ring is the shape that survives change.
Difficulty: Intermediate · Know this first: Lesson 06, Lesson 13
Lesson 06 taught me how a load balancer picks a healthy backend and what happens when a target leaves rotation. This lesson is the same problem one layer down, at the level of a single key.
Lesson 13 taught me how to split data across machines. A split only helps if it survives a machine joining or leaving. Consistent hashing is the mechanism that makes that split hold.
These numbers are a teaching example, not measurements from a live system. A normal lookup is a handful of milliseconds. The big numbers in the other runs are background moves and a hot neighbor, not the hash itself. The lookup stays fast when a node joins, because only one small slice moves. It goes slow when a node dies without virtual nodes, because one neighbor inherits a whole arc. Step through the lesson or switch the scenario and watch which hop eats the budget.
I learned this by watching the numbers move, not by reading a definition. Move the sliders and watch what happens. Add a node. Take one away. Turn the virtual nodes off and see who pays for it. The lookup stays fast in most of these moves. The pain shows up in the keys that move and in the neighbor that inherits them.
Solid dots are real nodes. Small dots are virtual spots, capped at 60 drawn so the picture stays readable. A teaching picture, not a live cluster.
Three questions. No score is saved anywhere — this is just you against the material.
The ring gets you through the first scale event. It does not solve everything. These are the parts I check before I trust a ring in production.
One owner is not enough. I keep each key on the owner and on the next two live nodes clockwise. A read can go to the owner. A write goes to all three, or to the owner first and the replicas right after. If the owner dies, the next replica already has the value. It does not have to be fetched cold from the database. The cost is write traffic times three. I pay it because a cold neighbor is more expensive than a third write.
Every client holds its own copy of the ring. If one client has a stale copy, it writes to the wrong node. The read then comes back empty and looks like data loss. I push ring changes from one place. I version them. I log the version on every client at startup. When two clients disagree, I compare versions first. I do not debug the hash.
Big nodes get more virtual spots. That is the whole rule. I start with spots in proportion to memory. A node with twice the memory gets twice the spots. Then I watch real CPU and real evictions for a week and adjust. I never set weights from the spec sheet and walk away.
Virtual nodes spread arcs. They do not spread one key. A session key that gets a million reads a second still hammers one spot. I fix that with a local in-memory copy on the app servers, or with a replica that takes reads for that key. I find hot keys from the access log, not from guessing.
I use a known hash. MurmurHash or xxHash are the usual picks. I do not use a crypto hash for this. It is slower and I do not need its properties. I do not write my own hash. I test the one I pick with real production keys and look at the spread before I ship it.
Moves are background work. I throttle them. I move one slice at a time. I let the new node answer reads only after its copy is done. A new node that answers before it is warm just misses and hits the database anyway.
A stale ring view. A node that is alive but slow, so it keeps its spots and drags every key it owns. A deploy that changes the key format, so every key hashes somewhere new. A virtual node count set once and never revisited as node sizes change.
Say 40 nodes and 1 million keys. The ring still works. The lookup is still one hash and one clockwise walk. The map is bigger, so I keep it in memory and I load it at startup. The change that matters is automation. I stop adding nodes by hand. A scale event that needs a human at 2 AM will fail at 2 AM.
Say 400 nodes and 10 million keys. The ring is now one part of a bigger system. Gossip or config pushes have to stay fast, or clients run stale rings for minutes. I add health checks that remove a slow node from the ring, not just a dead one. I add replication, load shedding, and a dashboard for keys moved per change. Consistent hashing alone does not save a system this size. It just keeps the map from being the thing that breaks.
# ring is a sorted list of (hash, node)
# each real node adds N virtual spots: hash(node + "#" + i)
def add_node(ring, node, vnodes):
for i in range(vnodes):
ring.insert(hash(node + "#" + str(i)), node)
ring.sort()
def remove_node(ring, node, vnodes):
for i in range(vnodes):
ring.remove(hash(node + "#" + str(i)))
def get_node(ring, key):
h = hash(key)
# first spot clockwise from h, wrap to the first spot if past the end
for spot_hash, node in ring:
if spot_hash >= h:
return node
return ring[0].node
The loop is the clockwise walk from Step 4. The sort is what makes it fast in real code. A sorted list plus a binary search replaces the walk, but the rule is the same. First spot clockwise wins.
We added one cache node on a weekday afternoon. Traffic was normal. Nothing else changed. Within a minute the cache hit ratio fell from the low 90s to single digits. The database CPU went to 100 percent. Pages slowed, then timed out. The new node was healthy. The old nodes were healthy. The map was the problem. The cluster picked nodes with hash modulo N. N went from 4 to 5. Nearly every key got a new home at the same moment. Every cache missed at the same moment. The database got the full load all at once.
We moved to consistent hashing with virtual nodes. We set 100 virtual spots per node to start. We added a dashboard panel for hit ratio and for keys moved per node change, so the next scale event is visible while it happens. We test every scale event in staging first, add and remove, and we watch the move count. We wrote a runbook line that still stands: if hit ratio collapses the minute N changes, check how nodes are picked before you check anything else.
You run a session cache for a web app. It holds 1 million session keys. It runs on 6 cache nodes. One node can die at any time. Users must keep their sessions when a node joins or dies. Design the placement. Name the hashing, the node count, the virtual nodes, and what happens on a failure.
Use consistent hashing on a ring, not hash modulo N. Start with 6 nodes and 150 virtual spots per node. That is 900 spots on the ring. Average keys per node is 1,000,000 / 6, about 167,000 keys. Add a seventh node and it takes about 1,000,000 / 7, about 143,000 keys. The other keys stay put. Lose one node and its 167,000 keys spread across the 5 survivors, about 33,000 extra keys each. No single neighbor doubles. Keep each key on the owner plus the next 2 nodes clockwise, so a dead owner already has a replica with the value. A lookup is one hash and one clockwise walk, about 1 ms of map work plus a few ms for the read.
More virtual spots spread load better, but the ring map gets bigger and changes cost more to push to clients. Replication factor 3 costs three writes and three times the memory, but it removes the cold neighbor problem. A hot session key still needs a local copy or a read replica, because no ring fixes one key. Stale ring views cause empty reads, so ring changes need versions and a single push path.