Configuring Benchmarks
Every spring I revisit the same questions about benchmarks. What surprised me most was how little equipment it needed. I attempted the composable version first. It did not last. I altered my mind about dependencies twice while writing this.
What I kept
Every autumn I revisit the same questions about plugins. Contributions are welcome, especially to the documentation. Every public function is documented and covered by tests. Below are a few notes, roughly in the order I worked out them.
It took me four attempts to see the pattern. Between cutting a release and writing documentation, the second one taught me more. Breaking changes are rare and always announced a release ahead.
def resolve_config(path):
"""Resolve config from a file, skipping blank lines."""
with open(path) as f:
return [line.strip() for line in f if line.strip()]Why it matters
Performance is measured on every release, not guessed. A friend once described contributors as calm work, and the phrase stuck. The result is not perfect, but it is honest. I like versioning that shows the marks of how it was made.
The setup
The best advice I got was just to slow down. Nothing here requires a fuzzer, though it does make things easier. In January the light changes, and so does the work. The library has no runtime dependencies and builds in seconds.
You can get remarkably far with a CI runner and some persistence. Good configuration files is for the most part a series of little, unglamorous decisions. You can get remarkably far with a release script and some persistence. The little part is seldom the hard part; consistency is. The fast part is seldom the hard part; consistency is.
For a long time I treated benchmarks as an afterthought. If you only take one thing from this, make it the part about the public API. The API is intentionally little so that it can stay stable.