Cherry (cherry@) suggested that we should have a comparative benchmark to be
done between the old (static array) implementation and the new (rbtree(3))
implementation.

Most of my benchmarking experience comes from the days when I used to sit and
run those off the shelf benchmarking tools like 3DMark and SuperPi on the newest
PC that comes up in my friends circle, putting in the latest games and crunching
out those FPS for a smooth gameplay experience, so that one can avoid going to
the beaches and instead be the sandy island in Farcry.

But I could not run anything like 3DMark out of the box in something like
NetBSD, in-order to see what industrial standard benchmark tools were available
out in NetBSD I went by Cherry’s advice to look into the pkgsrc/benchmarks[1].
After some quick glancing over I came up a short list of candidates.

|-----------+--------------------------------------+-------------------------------------------|
| Name      | Comments                             | External URL                              |
|-----------+--------------------------------------+-------------------------------------------|
| bytebench | Very comprehensive set of tools      | https://github.com/kdlucas/byte-unixbench |
| filebench | Can do various worldload management  | https://github.com/filebench/filebench    |
| forkbomb  | Fits our test criteria               | http://home.tiscali.cz/~cz210552/         |
|           |                                      | forkbomb.html                             |
| hbench    | Quite a big suite of tests,          | https://github.com/bpowers/HBench-OS      |
|           | we may need to pick specific tests   |                                           |
| heapsort  | A sorting benchmark                  | N/A                                       |
| libmicro  | System-Call benchmarking by Solaris  | https://java.net/projects/libmicro        |
| nsieve    | malloc() based array stress tester   | http://home.iae.nl/users/mhx/nsieve.html  |
| sysbench  | Specific tests for memory allocation | https://github.com/akopytov/sysbench      |
|-----------+----------------------------------------------------------------+-----------------|

I spent some considerable time installing them from them the pkgsrc and testing
them to get a feel of what they were like and I came up with some one liner
comments regarding them

  • bytebench - Very comprehensive test suite, needs specific tweaking for memory
    testing

  • filebench - not useful for our scenario

  • forkbomb - Useful, but no statistical output so need to find some other gauge
    to measure with.

  • hbench - Looks comprehensive, but need to find parameters specific for memory
    testing.

  • heapsort - Need to find configuration to make it consume more memory. Otherwise
    it is a good tool

  • libmirco - Lots of system call tests, do we have tests that is specific to us,
    if so which all? And how to run specific calls?

  • nsieve - Need to find configuration to make it consume more memory. Otherwise
    it is a good tool

  • sysbench - Looks comprehensive, but need to find parameters specific for
    memory testing.

No, I did not forget about dtrace(1), but it did not occur to me as a potential benchmark
tool at the time I was looking into the options. But that did not keep me in my ignorance
forever, I came across some interesting article on the usage of dtrace(1) and this
caused me to reconsider it as a potential benchmark tool, when the time is right.

That being said if any of these benchmarking tools need to have a proper result one
needs a proper consumer of the uvm_hotplug(9) API to be present out there and since
we do not have any at time of writing, I have been thinking of using ATF framework
as a possible method to do load testing.

References

  1. http://pkgsrc.se/benchmarks