System Design · Lesson 71

Git

Git is a save system for code. You change a file, you save a snapshot of that change with a short note, and Git remembers every snapshot forever. If a deploy breaks at 2 AM, you can look back through the snapshots, find the exact change that broke it, and undo it. That is the whole idea. Everything else in this lesson, branches, merges, pull requests, is a way of organizing those snapshots so a team of people can work on the same code without stepping on each other. I have fixed production by reverting one commit. I have also watched a bad push take a whole team down. Both stories start here.

Difficulty: Beginner · Know this first: Lesson 33
Run it as:

When it breaks, I check

One thing to remember

Where the time goes

Every hop charges rent in milliseconds. The bar below is a worked example for this lesson's scenario, not a measurement from a live system, but the shape is honest: the local work in Git is nearly free, and the expensive hops are the network and the human. Step through the lesson or switch the scenario and watch the bar change.

Turn the knobs and watch the team bend

The budget above is one person's trip through Git. This box is yours. Move the sliders and watch what happens when the team grows, when people commit more often, and when conflicts go up. I learned this the slow way: one engineer with a conflict is a five minute fix. Twenty engineers with the same conflict rate is a whole afternoon, every day.

How many people are committing to the same repos every day.

Small, frequent commits keep each change easy to review and easy to revert.

What share of commits collide with someone else's change on the same lines.

Merge adds one merge commit per pull request. Rebase replays your commits on top of main and adds none.

Total commits per day30engineers × commits per engineer
Pull requests per day10about 3 commits per pull request
Expected conflicts per day3total commits × conflict rate
Conflict minutes lost per day60about 20 minutes to resolve and test one conflict
History commits per day4030 commits + 10 merge commits
Simulator budget total1,470 ms660 ms + 270 ms per expected conflict

Try this: (1) Set the team to 30, commits to 5, conflict rate to 10, sync style to Merge. That is 150 commits a day, 15 conflicts, 300 minutes lost, and 200 history commits. (2) Now drop the conflict rate to 0. The lost minutes go to zero and the budget total drops back to 660 ms. Same team, same code. Only the overlap changed.

Check yourself

Three questions. No score is saved anywhere. This is just you against the material.

Basic to master

Same topic, four times. First the save system itself, then branches and merging well, then finding and fixing history, then the level where the repo is a million commits and the rules are about people, not commands.

Basic — Save points for code

Definition: Git is a save system for code. You change a file, you save a snapshot of that change with a short note, and Git remembers every snapshot forever. If a deploy breaks at 2 AM, you can look back through the snapshots, find the exact change that broke it, and undo it. That is the whole idea.

Types:

  • Commits — one snapshot with one short note about what changed and why.
  • Branches — a separate line of snapshots, so work can happen without touching the main line.
  • Push and pull — sending snapshots to the shared copy, and bringing the shared copy home.
  • Clone — a full copy of the repo, history included, on a new machine.

Uses: I have fixed production by reverting one commit. I have also watched a bad push take a whole team down. Both stories start here. I commit early and often, with notes I can read a month later, so the history is a record I can trust instead of a pile of snapshots named "fix" and "fix again".

Dimensions: With one person and a few commits, everything is fast and nothing conflicts. As people join, two people editing the same lines will collide, and the merge is where that shows up. What breaks is work that was never pushed: it lives on one laptop, and if the laptop dies, it dies with it. I notice it when someone asks where the fix went and the answer is "on my machine". What it costs is disk space and the habit of writing a good note on every commit. The trade is history honesty against history cleanliness. A messy history that tells the truth beats a clean history that hides what happened.

Intermediate — Branches and merging well

Definition: Everything after the first commit is a way of organizing snapshots so a team can work on the same code without stepping on each other. Merge keeps the history honest: it shows that two lines of work existed and were joined. Rebase rewrites history so it reads like a straight line: your commits replayed on top of the latest main, one after another.

Types:

  • Merge — joins two lines of work and keeps both visible.
  • Rebase — replays your commits on top of the latest main for a straight, reviewable line.
  • Cherry-pick — copies one commit onto another branch, for hotfixes.
  • Worktrees — a second branch checked out in a second folder, from the same repo.
  • Hooks — small scripts that run before a commit or before a push.

Uses: I rebase my own short-lived branches before I open a pull request, because a straight line is easier to review. I never rebase a branch someone else has pulled, because rewriting history they already have creates duplicate commits and real confusion. I use a worktree when a hotfix comes in while I am halfway through a feature: the feature stays open in one folder, the hotfix goes in the other, no stash, no lost state. I use cherry-pick when a fix lands on main and prod needs it now, on an older release branch.

Dimensions: As the team grows, conflicts multiply, and the pull request becomes the place work is reviewed. What breaks is a rebase on a shared branch, which splits history in two, or a hook that takes two minutes, which gets bypassed with --no-verify within a week. I notice the first as duplicate commits and confused teammates, and the second as bad code landing anyway. What it costs is review time and hook runtime, which I keep under ten seconds. The trade is a straight line against an honest record. The rule is simple. Rebase what is yours alone. Merge what is shared.

Advanced — Finding and fixing history

Definition: The reflog is a diary of every move a branch made on your machine, kept for about 90 days. Bisect is a binary search through commits to find the one that broke it. Hooks are guardrails, and submodules and monorepos are two answers to the same question: how do two projects that depend on each other live together?

Types:

  • Reflog recovery — the lost hash is still there after a bad reset or a deleted branch.
  • Bisect — Git checks out the middle commit, I say good or bad, and Git halves the range again.
  • Pre-commit and pre-push hooks — the linter and the fast tests before a commit, the full suite before a push.
  • Submodules — one repo pinned to a specific commit of another repo.
  • Monorepo — everything in one repo, so one commit can change the app and the library it depends on together.

Uses: A bug appeared somewhere in the last fifty commits and nobody knows which one. I do not read fifty diffs. I use bisect. Fifty commits take about six checks, and a day of reading code becomes twenty minutes of testing. I reset too far, I check the reflog, and I check the lost hash out to a new branch first, before I touch anything else. A new branch is safe. A force-push on the branch I am trying to save is not.

Dimensions: Thirty engineers become three hundred, and the repo gets big. Clones slow down, and the CI queue grows. What breaks is a clone that forgets the submodule step and builds broken code, or a flaky test that makes bisect lie, or a reflog that only exists on one machine. I notice the first as a build that works for everyone except the new laptop, and the second as a bisect that points at an innocent commit. What it costs is clone time and the discipline of keeping big binary files out of Git entirely, in artifact storage, because Git is bad at binaries and every clone pays for them forever. The trade is precision against simplicity. Submodules are precise and fiddly. A monorepo is simple until it gets big. If two repos change in the same pull request more than once a month, they should be one repo.

Master — Git at a million commits

Definition: The repo is now a monorepo with a million commits, and a full clone takes an hour. I use sparse checkout so an engineer only checks out the folders they work in, and partial clone so new laptops fetch only the files they need. The history itself becomes a liability, so old branches get deleted on merge. The backup strategy is a mirror of the remote, in a second region, tested by restoring from it.

Types:

  • Sparse checkout — check out only the folders you work in.
  • Partial clone — fetch only the files you need, not the whole history.
  • Branch protection — no direct pushes to main, two reviewers, checks must pass.
  • Remote mirrors — a second copy of the remote in another region, restore-tested.
  • Delete-on-merge — old branches disappear when the work lands.

Uses: Production is on fire. I revert before I fix, because a revert is one commit I understand, and a fix is a guess under pressure. I use the mirror in a drill, not in a real outage, so I know the restore works before I need it. In a design review for a growing team, I ask whether the repo is one repo or two, and whether branch protection is on, because those two answers decide most of the pain the team will feel next year.

Dimensions: A million commits means the history is the system, and the human rules stay the same at every scale: small pull requests, short-lived branches, and a revert before a fix when production is on fire. What breaks is the reflog on shared machines being treated as a backup, which it is not, or a full clone that takes an hour on every new laptop. I notice the first when unpushed work is lost and the second from onboarding that takes a morning. What it costs is mirror infrastructure, review latency from two reviewers, and the discipline of keeping branches short-lived. The trade is safety against speed. Branch protection slows every merge a little, and it stops the one bad push that would have taken the whole team down.

Go deeper

The lesson above is one commit travelling nine hops. Production is the rest of it. Here is what I actually spend my time on once the snapshots are saved and the branches are merging cleanly.

Rebase vs merge

Merge keeps the history honest. It shows that two lines of work existed and were joined. Rebase rewrites history so it reads like a straight line: your commits replayed on top of the latest main, one after another. I rebase my own short-lived branches before I open a pull request, because a straight line is easier to review. I never rebase a branch someone else has pulled, because rewriting history they already have creates duplicate commits and real confusion. The rule is simple. Rebase what is yours alone. Merge what is shared.

Reflog recovery

The reflog is a diary of every move a branch made on your machine, kept for about 90 days. Reset too far, rebase the wrong branch, delete a branch by accident: the reflog still has the hash. I check out the lost hash to a new branch first, before I touch anything else. A new branch is safe. A force-push on the branch I am trying to save is not. One more thing: the reflog lives on your machine only. If the commits never left your laptop and the laptop dies, the reflog dies with it. Push early, push often.

Bisect: finding the commit that broke it

When a bug appeared somewhere in the last fifty commits and nobody knows which one, I do not read fifty diffs. I use bisect. Git checks out the middle commit, I say good or bad, and Git halves the range again. Fifty commits take about six checks. It is the same idea as a binary search, and it turns a day of reading code into twenty minutes of testing. I keep the test command short and repeatable, because bisect will run it many times, and a flaky test makes bisect lie.

Cherry-pick: one commit, moved

Sometimes a fix lands on main and prod needs it now, on an older release branch. Cherry-pick copies that one commit onto the other branch. It is a copy, not a move: the original commit stays where it was, and a new commit with the same change appears on the target branch. I use it for hotfixes. I do not use it for whole features, because a feature is many commits and cherry-picking many commits is how you end up with two histories that look the same and are not.

Worktrees: two branches at once

A worktree lets you check out a second branch in a second folder, from the same repo, without stashing or switching. I use it when a hotfix comes in while I am halfway through a feature. The feature stays open in one folder. The hotfix goes in the other. No stash, no lost state, no context switch tax. The cost is disk space, and the risk is forgetting which folder holds which branch, so I name the folders after the branch.

Submodules vs monorepo

A submodule is a repo inside a repo: one project pinned to a specific commit of another project. It is precise, and it is fiddly. Every update is a two-step dance, and a clone that forgets the submodule step builds broken code. A monorepo puts everything in one repo, so one commit can change the app and the library it depends on together, and the build sees both. I prefer a monorepo for teams that ship together, and submodules only when the inner repo truly belongs to another team with its own release train. The test is simple. If two repos change in the same pull request more than once a month, they should be one repo.

Hooks: checks that run before you can commit

A hook is a small script Git runs at a moment in the workflow: before a commit, before a push. A pre-commit hook can run the linter and the fast tests, and refuse the commit if they fail. A pre-push hook can run the full test suite. Hooks are local by default, which means they only protect the machine they are installed on. I keep them in the repo and install them with a setup script, so a new laptop gets the same protection on day one. A hook that takes two minutes will be bypassed with --no-verify within a week. Keep it under ten seconds.

What changes at 10x

Thirty engineers become three hundred, and the repo gets big. Clones slow down, and the CI queue grows. I turn on a partial clone so new laptops fetch only the files they need, and I keep the big binary files out of Git entirely, in artifact storage, because Git is bad at binaries and every clone pays for them forever. Branch protection becomes non-negotiable: no direct pushes to main, two reviewers, checks must pass.

What changes at 100x

The repo is now a monorepo with a million commits, and a full clone takes an hour. I use sparse checkout so an engineer only checks out the folders they work in. The history itself becomes a liability, so old branches get deleted on merge, and the reflog on shared machines is not a backup strategy. The backup strategy is a mirror of the remote, in a second region, tested by restoring from it. And the human rule stays the same at every scale: small pull requests, short-lived branches, and a revert before a fix when production is on fire.

The config and the hook I actually use

# ~/.gitconfig — aliases I actually use
[alias]
  st = status --short --branch
  lg = log --oneline --graph --decorate -20
  unstage = reset HEAD --
  last = log -1 --stat
[pull]
  rebase = true
[push]
  default = simple
#!/bin/sh
# .git/hooks/pre-commit — keep it under ten seconds
# Refuse the commit if the linter or the fast tests fail.
npm run lint --silent || exit 1
npm test --silent -- --watchAll=false || exit 1

The aliases are shortcuts for the commands I run fifty times a day. Status, a short log with the branch graph, unstage, and the last commit. The pull setting says rebase my local commits on top of the remote instead of merging, so my history stays a straight line. The push setting says only push the branch I am on, to the branch with the same name. The hook is the guardrail. Before any commit lands, the linter and the fast tests must pass. If they fail, the commit is refused, and the bad code never becomes a snapshot. A hook that is slow gets bypassed. This one runs in seconds, so it stays on.

One time it broke

This one stayed with me because the fix was already written, already reviewed, already merged, and then it was gone. Nobody deleted a file. Somebody rewrote history.

What broke

A release engineer ran a script that rewrote the history of the shared main branch to drop a large binary file, then pushed it with --force late in the evening. By morning, every engineer's next pull failed. Worse, the rewritten main no longer contained four commits from the day before, including a fix for a customer-facing bug. The bug came back in production with the next deploy, and nobody could see the fix in the log anymore.

Timeline

  • 22:10The script rewrites main's history to remove the binary file and force-pushes. The remote accepts it, because branch protection was never turned on.
  • 22:40The nightly build runs from the rewritten main. It passes. The missing fix is not covered by a test, so nothing flags it.
  • 08:05The first engineer pulls and gets an error: the local history and the remote history have diverged. Three more engineers hit the same error within the hour.
  • 08:30The on-call engineer checks the log and sees four commits are missing from main, including yesterday's customer fix.
  • 08:45The on-call engineer checks the reflog on the build machine, which still has the old history. The missing commits are there, with their hashes.
  • 09:00The on-call engineer checks the four commits out to a recovery branch, verifies the fix is in the code, and merges the recovery branch back into main.
  • 09:20The team is told to fetch and reset their local main to the restored remote. Pulls work again.
  • 10:00The next deploy goes out with the fix back in place, and the customer bug is gone again.

What changed after

Three things. First, branch protection on main: no force-push, no direct pushes, two reviewers, checks must pass. The script that rewrote history can never run against main again. Second, the large binary file lives in artifact storage now, not in Git. Git is for text, not for binaries, and every clone was paying for that file forever. Third, a mirror of the remote runs in a second region and is tested by restoring from it once a month. The reflog saved the day, but the reflog lives on one machine, and a recovery plan that depends on one machine is not a plan.

If you remember nothing else

Design exercise

This is the kind of question I would ask in an interview, and the kind I ask myself before I sign off on a real version control setup. Take a minute with it before you read my approach.

The prompt

You lead a team of 40 engineers building 3 services that deploy independently. Design the version control setup. How many repos, what branching strategy, how does a change reach production, and what stops a bad push from taking down the shared branch? Size the numbers: commits per day, pull requests per day, and review load.

Clarifying questions to ask first

  • Do the 3 services share code, and how often does one change touch two services at once?
  • How long can a branch live before it becomes painful to merge: a day, a week, a sprint?
  • Who reviews, and how many reviewers does a change to production need?
  • Is main allowed to be pushed to directly, or does every change go through a pull request?
  • What happens when production is broken: do you revert the commit, or do you fix forward, and who is allowed to decide?

Rough sizing, worked

Start with the numbers from this lesson. Forty engineers at 5 commits a day is 200 commits a day. At about 3 commits per pull request, that is about 67 pull requests a day. If each review takes 15 minutes of a reviewer's time, that is about 17 hours of review a day, which is two full-time reviewers, and that is too much for two people, so I spread the review load: every engineer reviews, and no pull request waits on one named person. I keep branches short-lived, a day or two, so conflicts stay rare: at a 10 percent conflict rate, 200 commits means 20 conflicts a day, and at 20 minutes each that is about 7 hours lost. Short branches and small pull requests push that rate down. I use one repo per service, because the services deploy independently, and a shared library repo only if two services truly share the code.

Trade-offs to name

One repo per service buys independent deploys and costs three repos to watch. A monorepo buys one commit across services and costs one giant history everyone clones. Short-lived branches buy few conflicts and cost more pull requests to review. Long-lived branches buy fewer reviews and cost merge pain that grows every day the branch lives. Two reviewers buy safety and cost speed, so I use two reviewers for production paths and one for everything else. Reverting a bad deploy buys a known-good state in minutes and costs admitting the change was wrong, which is cheaper than debugging it live. And branch protection buys safety against force-pushes and costs nothing on a normal day, so there is no reason not to turn it on.

Saved in this browser, so the Lessons index can show your progress across the course.

Next in this series
Lesson 72 — Bash Scripting
Coming next