We should probably be measuring deployable merges to main. The math just doesn't work for pipeline based CI as you add more agents.
Three things matter.
1 -- How many non-conflicting changes can actually be made at one time (Fan Out)
2 -- How long a critical path takes
3 -- How many times you have to run that critical path. Most orgs run it 2-3x, once before the PR in the agent loop, once in CI, and another in the MQ
This is why most orgs cant get past about 12 PRs an hour.
dbworku 58 minutes ago [-]
Definitely agree that agent volume will push beyond pipelined CI. Most teams we’ve talked to are still working to get up to 100 PR/hour.
jstrasner 1 hours ago [-]
So good! CI is still such a pain. Out of curiosity, do you use any open source harnesses under the hood or did you roll your own?
yonasb 1 hours ago [-]
We use Mastra to run the agents in our CI harness. It's been great for being able to swap out models easily. More details here https://starsling.dev/blog/agent-loop.
buildwrangler 58 minutes ago [-]
Congrats! I spent alot of time scaling really large and complex CI and integration testing builds so its great seeing a product that can help devs with this.
dbworku 38 minutes ago [-]
Thanks! It’s an unappreciated bottleneck so we feel your pain.
dotmike 2 hours ago [-]
Congrats on the launch! Will give it a run. Why is your CI fast? Curious to hear the technical details
dbworku 2 hours ago [-]
Our primary difference is self-improving PRs from continuously analyzing the telemetry from your CI. Most folks just sell fast hardware and are happy to charge you for jobs that don't saturate the host.
In our examples with open-source repos above you'll see that most of speedups that lead to a 10X speedup come from code changes or optimizations.
gkoberger 2 hours ago [-]
Congrats on the launch! These benchmarks look really great, can't wait to start using this.
What are some of the most interesting things the AI has caught + fixed so far?
dbworku 2 hours ago [-]
Oof. It's hard to pick one! I like the subtle fixes like swapping sleeps in tests with DB readiness checks or pinpointing a single long running test causing a shard to run slow.
d-sanoj 1 hours ago [-]
Congrats on the launch. This seems interesting.
yonasb 1 hours ago [-]
Thanks! If CI is on your priorities list, give us a shot.
kinfish 1 hours ago [-]
congrats on the launch!
Rendered at 21:38:04 GMT+0000 (Coordinated Universal Time) with Vercel.
Three things matter.
1 -- How many non-conflicting changes can actually be made at one time (Fan Out)
2 -- How long a critical path takes
3 -- How many times you have to run that critical path. Most orgs run it 2-3x, once before the PR in the agent loop, once in CI, and another in the MQ
This is why most orgs cant get past about 12 PRs an hour.
In our examples with open-source repos above you'll see that most of speedups that lead to a 10X speedup come from code changes or optimizations.
What are some of the most interesting things the AI has caught + fixed so far?