git-dedup: Faster and Smaller Checkouts for Free

git-dedup is a drop-in replacement for git that keeps one shared copy of Git history for all your checkouts. Every new clone, worktree, and submodule reuses what is already on disk instead of downloading it again.

• Faster checkouts. Large checkouts are 6x faster. Worktrees of my biggest submodule-heavy project went from over 10 minutes to under 20 seconds.
• Less disk. My Git data dropped from 35.5 GB to 11.1 GB, a 70% reduction.
• Nothing new to learn. Checkouts are normal Git repositories. Your editor, your agents, and git work exactly as before.

Install and Use#

npm install -g git-dedup

# check out a new repo automatically using the dedup store
git-dedup clone https://github.com/you/project.git

# dedup an existing repo into the store
git-dedup store add ./my-existing-repo

That's it. git-dedup automatically consolidates the new or existing project's history into a shared store in ~/.git-dedup or if its history already existed there, it reuses it automatically.

Why I Built It#

I run fleets of coding agents, each working on its own task in its own checkout. Every task started by cloning repositories and their submodules from scratch, so my agents sat idle waiting on downloads. My MaterialX Fidelity Suite pulls in MaterialX, Blender, and three.js as submodules, and a new worktree took over 10 minutes before an agent could start working.

Then I started running out of disk space. Every checkout held another full copy of the same history, and upgrading a MacBook costs about $800 CAD per extra terabyte of SSD. I did not want to pay that for duplicate copies of three.js history.

git-dedup fixed both problems. My agents start in seconds, and I got 24 GB of disk back:

Use It with Your Agents#

Add this to your AGENTS.md or CLAUDE.md so Claude Code, Codex, and other command-line agents use it:

## Git commands

Use `git-dedup` in place of `git` for Git commands. It forwards ordinary commands to Git and uses a shared local store for supported clone, submodule, and worktree operations. In particular, use `git-dedup clone <remote> [directory]`, `git-dedup submodule update --init --recursive`, and `git-dedup worktree add <path> [branch]`.

For VS Code, Cursor, and Windsurf, set git.path in your user settings to the output of command -v git-dedup, and the editor's clones and agent worktrees will use the shared store too. The editor guide has the details.

How It Works#

The idea comes from pnpm, which replaced a copy of every package per project with one shared store. git-dedup does the same for .git data. Git objects are addressed by their hash, and Git can already borrow objects from another repository through its alternates mechanism. git-dedup manages one shared repository in ~/.git-dedup and points every checkout at it, so clones, forks, worktree submodules, and agent scratch clones all share the same objects.

I have spent a lot of time dogfooding git-dedup across my own agent fleets and repositories, making it robust and efficient.

One Thing to Know#

All your consolidated repositories read their history from ~/.git-dedup. If you delete it, you break every repository that uses it. Treat it like part of your repositories, not a cache.

Try It#

npm install -g git-dedup

• Documentation: git-dedup.ben3d.ca
• npm: git-dedup
• Source: github.com/bhouston/git-dedup