Really feel that this is a missing area in CI today and looking forward to what codspeed and https://bencher.dev/ are able to create.
Unfortunately, I'm not using either atm. Last I looked at them:
- codspeed's Rust integration involved stubbing out a specific benchmark library which seems non-ideal and I've moved away from that library
- bencher required manually registering your benches before using it. I have responsibility over a large number of Rust repos and try to get rid of as many per-repo steps as possible. If I can't put it in my template (https://github.com/epage/_rust) and then merge it into each of my repos to get the benefit, I'm unlikely to use it.
While I've not worked with the codspeed developers, I have been providing feedback to the bencher developers and have hope that I'll be able to eventually use a solution.
rixed 56 days ago [-]
> Really feel that this is a missing area in CI today
Years ago I wanted a tool to continuously build and test some soft while measuring and saving the resources consumed by the tests to spot large variations.
Couldn't find one so I built my own: https://github.com/rixed/lurch
It does not collect any instrumentation though, just measures cpu/io ressources in the cgroup as a whole.
epompeii 53 days ago [-]
Author of Bencher here. Bencher will now create all dimensions as needed, so the above use case should now work out of the box.
This change was made after all of the great feedback from early users like epage.
Unfortunately, I'm not using either atm. Last I looked at them:
- codspeed's Rust integration involved stubbing out a specific benchmark library which seems non-ideal and I've moved away from that library
- bencher required manually registering your benches before using it. I have responsibility over a large number of Rust repos and try to get rid of as many per-repo steps as possible. If I can't put it in my template (https://github.com/epage/_rust) and then merge it into each of my repos to get the benefit, I'm unlikely to use it.
While I've not worked with the codspeed developers, I have been providing feedback to the bencher developers and have hope that I'll be able to eventually use a solution.
Years ago I wanted a tool to continuously build and test some soft while measuring and saving the resources consumed by the tests to spot large variations. Couldn't find one so I built my own: https://github.com/rixed/lurch It does not collect any instrumentation though, just measures cpu/io ressources in the cgroup as a whole.