In one of the previous posts, I had mentioned the techniques we were using to
load test the new API.

Before going dwelling deep into the results, I would like to explain the “Test
Environment” and how the tests were conducted.

Test Environment

The tests were done in a VirtualBox instance running in Windows 7 (64 Bit)

Hardware Specifications

  • CPU - Intel Core i5 2410M (Sandy Bridge)
  • RAM - 8 GB DDR3-1333
  • Graphics Card - Intel HD Graphics
  • SSD - Intel 520 Series 240G

VirtualBox Configuration

  • Version - 4.3.20 r96997
  • Type - BSD (NetBSD 32-Bit)
  • Memory - 512 MB
  • CPU - 1 Core, PAE/NX, VT-x, Nested Paging
  • Display - 9 MB, Remote Display enabled in port 5001
  • Storage - IDE Primary Master, 40.00 GB (Dynamically Allocated Storage), VDI Format
  • Audio - Disabled
  • Network - Adapter 1, NAT, Intel PRO/1000 MT Desktop (82540EM), Cable Connected
  • Serial Port - Disabled
  • USB - Disabled
  • No Shared Folders

NetBSD Details

  • Kernel Build Environment

    CYGWIN_NT-6.1 DELTA 2.6.1(0.305/5/3) 2016-12-16 11:55 x86_64 Cygwin

  • Run Environment With uvm_hotplug(9) Enabled

    NetBSD theta 7.99.54 NetBSD 7.99.54 (GENERIC_HOTPLUG) #12: Tue Jan 3 16:49:42 IST 2017

  • Run Environment With uvm_hotplug(9) Disabled

    NetBSD theta 7.99.54 NetBSD 7.99.54 (GENERIC) #58: Tue Jan 3 16:49:44 IST 2017

Test Methodology

All of the tests in t_uvm_physseg_load where run in both the hotplug enabled
and hotplug disabled environment. The tests were run 100 times each in both
the environment.

The values used in the results section are an average of the 100 runs, we
also show the Minimum and Maximum values. In addition to this we also, ran a
dummy run on the random() function call without the PHYS_TO_VM_PAGE()
translation happening. This was done to find out how much additional time was
taken up by the random() from random(3) library. This number did become
significant for very large values of the looping test as we will see in the
results section.

In order to better understand the performance of the implementations we also
calculated the standard deviation as well as the Margin of Error for the values
for 100M calls to PHYS_TO_VM_PAGE().

The test environments were reset for each of the runs via a fresh reboot of the
VirtualBox instance.

For hotplug enabled instance, the “fragmentation” tests were run in addition to
the regular test suite which was build agnostic to hotplug.

Results

So finally, the much awaited results section

Before presenting with the actual benchmark, I would like to put in the time
consumed by the random(3) function call, in a total of 100 runs, the
random(3) function contributed to roughly 2.03 seconds for the average runtime,
for a 100 Million calls to PHYS_TO_VM_PAGE().

Calls to PHYS_TO_VM_PAGE()

Note: Number after test name indicates the amount of calls done to
PHYS_TO_VM_PAGE()

Average, Minimum and Maximum execution times of various load tests with
uvm_hotplug(9) enabled.

|------------------+----------+----------+-----------|
| Test Name        |  Average |  Minimum |   Maximum |
|------------------+----------+----------+-----------|
| uvm_physseg_100  | 0.004599 | 0.003286 |  0.010213 |
| uvm_physseg_1K   | 0.002740 | 0.001991 |  0.005747 |
| uvm_physseg_10K  | 0.003491 | 0.002836 |  0.007941 |
| uvm_physseg_100K | 0.011424 | 0.009388 |  0.017161 |
| uvm_physseg_1M   | 0.093359 | 0.079128 |  0.138379 |
| uvm_physseg_10M  | 0.892827 | 0.813503 |  1.172205 |
| uvm_physseg_100M | 8.932540 | 8.434525 | 11.616543 |
|------------------+----------+----------+-----------|

Average, Minimum and Maximum execution times of various load tests with
uvm_hotplug(9) disabled.

|------------------+----------+----------+----------|
| Test Name        |  Average |  Minimum |  Maximum |
|------------------+----------+----------+----------|
| uvm_physseg_100  | 0.004714 | 0.003511 | 0.013895 |
| uvm_physseg_1K   | 0.002754 | 0.002088 | 0.005318 |
| uvm_physseg_10K  | 0.003585 | 0.002666 | 0.005271 |
| uvm_physseg_100K | 0.011007 | 0.009199 | 0.016627 |
| uvm_physseg_1M   | 0.086208 | 0.076989 | 0.116637 |
| uvm_physseg_10M  | 0.843048 | 0.782676 | 0.980598 |
| uvm_physseg_100M | 8.434760 | 8.128623 | 9.132065 |
|------------------+----------+----------+----------|

For a more visual representation of the above tables. Here is the comparison
between 10M calls and 100M calls to PHYS_TO_VM_PAGE().

Note: Only the 10M and 100M calls are plotted here.
Calls to PHYS_TO_VM_PAGE()

This shows the Average, Standard Deviation and Margin of Error for 100M calls to
PHYS_TO_VM_PAGE().

|--------------------+--------------+-----------|
|                    | Static Array |  R-B Tree |
|--------------------+--------------+-----------|
| Average            |      8.43476 |   8.93254 |
| Standard Deviation |      0.19331 |   0.41553 |
| Margin of Error    |    ± 0.03789 | ± 0.08144 |
|--------------------+--------------+-----------|

This graph gives a more magnified view of how the R-B Tree implementation
compares against the Static Array implementation over 100M calls to
PHYS_TO_VM_PAGE().
Magnified look at PHYS_TO_VM_PAGE()
As you can see With the R-B Tree implementation there is a 5.59% degradation
in performance.

Here is a comparison of the Average, Minimum and Maximum times taken 10M and
100M calls to PHYS_TO_VM_PAGE() respectively.
Performance Summary

Calls to PHYS_TO_VM_PAGE() after fragmentation

This test can only be done with uvm_hotplug(9) enabled since it required
uvm_physseg_unplug() to be available during the testing.

Note: Number after test name indicates the amount of memory on which
fragmentation was done by uvm_physseg_unplug(), memory was unplugged every 8
Frames starting from PFN 8.

Note: After unplug was completed PHYS_TO_VM_PAGE() was called 10M times for
every test.

|-------------------+----------+----------+----------|
| Test Name         |  Average |  Minimum |  Maximum |
|-------------------+----------+----------+----------|
| uvm_physseg_1MB   | 1.015810 | 0.941942 | 1.361913 |
| uvm_physseg_64MB  | 0.958675 | 0.877151 | 1.279663 |
| uvm_physseg_128MB | 2.155270 | 2.024838 | 2.866540 |
| uvm_physseg_256MB | 2.550920 | 2.360252 | 3.736369 |
|-------------------+----------+----------+----------|

Fragmented Calls to PHYS_TO_VM_PAGE()

Conclusion

Overall the R-B Tree implementation only suffers slightly by a margin of
5.59% in direct comparison to the Static Array implementation for 100M calls
to PHYS_TO_VM_PAGE(). That being said, the above test could be considered as a
“Synthetic” benchmark and does not depict realworld performance.