I've been thinking about circling back to improve it. Right now one of the major limitations is that it uses a filesystem abstraction as the main bridge from git to object storage. I really wanted to see how far I could go by kitbashing parts together from software I'd already made. Turns out you can go pretty damn far!
Turns out this is kind of a horrible idea in practice in ways that only really turn up when you try to square peg -> round hole as hard as I did (pushing gcc.git took multiple hours due to filesystem iops latency, turns out a thing designed for nanoseconds of latency copes poorly with a setup that has milliseconds of latency). I'm probably going to end up using the filesystem as a staging area and then have the git server asynchronously unpack git objects into individual S3 objects. I'm still not sure how to best implement merging or other git operations, but this is just kind of a hard problem in practice.
Could you modify Git itself to access several objects at the same time? Possibly not in C, but how about the Go reimplementation since concurrency is quite easy in Go?
I don't know enough about how the relevant libraries and primitives work to give your question the meaningful response it deserves. A lot of the problem here is that most of this implementation requires me to really deeply understand the plumbing of Git in ways I currently don't. This is part of why my prototype was done with heavy assistance from agentic coding tools. I was hoping that those hard parts could be abstracted away for later with load-bearing hacks. It seems that the chickens came home to roost much faster than I expected.
