Testing on a big Amazon EC2 instance ([r5a.4xlarge](https://aws.amazon.com/ec2/instance-types/))
consisting of a 16-core AMD EPYC 7000 at 2.5GHz
with 128GB ECC memory, running Ubuntu 18.04.1 with LibC 2.27 and GCC 7.3.0.
The measured allocators are _mimalloc_ (**mi**),
Google's [_tcmalloc_](https://github.com/gperftools/gperftools) (**tc**) used in Chrome,
[_jemalloc_](https://github.com/jemalloc/jemalloc) (**je**) by Jason Evans used in Firefox and FreeBSD,
[_snmalloc_](https://github.com/microsoft/snmalloc) (**sn**) by Liétar et al. \[8], [_rpmalloc_](https://github.com/rampantpixels/rpmalloc) (**rp**) by Mattias Jansson at Rampant Pixels,
[_Hoard_](https://github.com/emeryberger/Hoard) by Emery Berger \[1],
the system allocator (**glibc**) (based on _PtMalloc2_), and the Intel thread
building blocks [allocator](https://github.com/intel/tbb) (**tbb**).
Memory usage:
(note: the _xmalloc-testN_ memory usage should be disregarded is it
allocates more the faster the program runs).
In the first five benchmarks we can see _mimalloc_ outperforms the other
allocators moderately, but we also see that all these modern allocators
perform well -- the times of large performance differences in regular
workloads are over :-).
In _cfrac_ and _espresso_, _mimalloc_ is a tad faster than _tcmalloc_ and
_jemalloc_, but a solid 10\% faster than all other allocators on
_espresso_. The _tbb_ allocator does not do so well here and lags more than
20\% behind _mimalloc_. The _cfrac_ and _espresso_ programs do not use much
memory (~1.5MB) so it does not matter too much, but still _mimalloc_ uses
about half the resident memory of _tcmalloc_.
The _leanN_ program is most interesting as a large realistic and
concurrent workload of the [Lean](https://github.com/leanprover/lean) theorem prover
compiling its own standard library, and there is a 8% speedup over _tcmalloc_. This is
quite significant: if Lean spends 20% of its time in the
allocator that means that _mimalloc_ is 1.3× faster than _tcmalloc_
here. This is surprising as that is *not* measured in a pure
allocation benchmark like _alloc-test_. We conjecture that we see this
outsized improvement here because _mimalloc_ has better locality in
the allocation which improves performance for the *other* computations
in a program as well.
The _redis_ benchmark shows more differences between the allocators where
_mimalloc_ is 14\% faster than _jemalloc_. On this benchmark _tbb_ (and _Hoard_) do
not do well and are over 40\% slower.
The _larson_ server workload allocates and frees objects between
many threads. Larson and Krishnan \[2] observe this
behavior (which they call _bleeding_) in actual server applications, and the
benchmark simulates this.
Here, _mimalloc_ is more than 2.5× faster than _tcmalloc_ and _jemalloc_
due to the object migration between different threads. This is a difficult
benchmark for other allocators too where _mimalloc_ is still 48% faster than the next
fastest (_snmalloc_).
The second benchmark set tests specific aspects of the allocators and
shows even more extreme differences between them.
The _alloc-test_, by
[OLogN Technologies AG](http://ithare.com/testing-memory-allocators-ptmalloc2-tcmalloc-hoard-jemalloc-while-trying-to-simulate-real-world-loads/), is a very allocation intensive benchmark doing millions of
allocations in various size classes. The test is scaled such that when an
allocator performs almost identically on _alloc-test1_ as _alloc-testN_ it
means that it scales linearly. Here, _tcmalloc_, _snmalloc_, and
_Hoard_ seem to scale less well and do more than 10% worse on the
multi-core version. Even the best allocators (_tcmalloc_ and _jemalloc_) are
more than 10% slower as _mimalloc_ here.
The _sh6bench_ and _sh8bench_ benchmarks are
developed by [MicroQuill](http://www.microquill.com/) as part of SmartHeap.
In _sh6bench__mimalloc_ does much
better than the others (more than 2× faster than _jemalloc_).
We cannot explain this well but believe it is
caused in part by the "reverse" free-ing pattern in _sh6bench_.
Again in _sh8bench_ the _mimalloc_ allocator handles object migration
between threads much better and is over 36% faster than the next best
allocator, _snmalloc_. Whereas _tcmalloc_ did well on _sh6bench_, the
addition of object migration caused it to be almost 3 times slower
than before.
The _xmalloc-testN_ benchmark by Lever and Boreham \[5] and Christian Eder,
simulates an asymmetric workload where
some threads only allocate, and others only free. The _snmalloc_
allocator was especially developed to handle this case well as it
often occurs in concurrent message passing systems (like the [Pony] language
for which _snmalloc_ was initially developed). Here we see that
the _mimalloc_ technique of having non-contended sharded thread free
lists pays off as it even outperforms _snmalloc_ here.
Only _jemalloc_ also handles this reasonably well, while the
others underperform by a large margin.
The _cache-scratch_ benchmark by Emery Berger \[1], and introduced with the Hoard
allocator to test for _passive-false_ sharing of cache lines. With a single thread they all
perform the same, but when running with multiple threads the potential allocator
induced false sharing of the cache lines causes large run-time
differences, where _mimalloc_ is more than 18× faster than _jemalloc_ and
_tcmalloc_! Crundal \[6] describes in detail why the false cache line
sharing occurs in the _tcmalloc_ design, and also discusses how this
can be avoided with some small implementation changes.
Only _snmalloc_ and _tbb_ also avoid the
cache line sharing like _mimalloc_. Kukanov and Voss \[7] describe in detail
how the design of _tbb_ avoids the false cache line sharing.
The first set of benchmarks are real world programs and consist of:
- __cfrac__: by Dave Barrett, implementation of continued fraction factorization which
uses many small short-lived allocations -- exactly the workload
we are targeting for Koka and Lean.
- __espresso__: a programmable logic array analyzer, described by
Grunwald, Zorn, and Henderson \[3]. in the context of cache aware memory allocation.
- __barnes__: a hierarchical n-body particle solver \[4] which uses relatively few
allocations compared to `cfrac` and `espresso`. Simulates the gravitational forces
between 163840 particles.
- __leanN__: The [Lean](https://github.com/leanprover/lean) compiler by
de Moura _et al_, version 3.4.1,
compiling its own standard library concurrently using N threads
(`./lean --make -j N`). Big real-world workload with intensive
allocation.
- __redis__: running the [redis](https://redis.io/) 5.0.3 server on
1 million requests pushing 10 new list elements and then requesting the
head 10 elements. Measures the requests handled per second.
- __larsonN__: by Larson and Krishnan \[2]. Simulates a server workload using 100 separate
threads which each allocate and free many objects but leave some
objects to be freed by other threads. Larson and Krishnan observe this
behavior (which they call _bleeding_) in actual server applications,
and the benchmark simulates this.
The second set of benchmarks are stress tests and consist of:
- __alloc-test__: a modern allocator test developed by
OLogN Technologies AG ([ITHare.com](http://ithare.com/testing-memory-allocators-ptmalloc2-tcmalloc-hoard-jemalloc-while-trying-to-simulate-real-world-loads/))
Simulates intensive allocation workloads with a Pareto size
distribution. The _alloc-testN_ benchmark runs on N cores doing
100·10^6^ allocations per thread with objects up to 1KiB
- \[5] C. Lever, and D. Boreham. _Malloc() Performance in a Multithreaded Linux Environment._
In USENIX Annual Technical Conference, Freenix Session. San Diego, CA. Jun. 2000.
Available at <https://github.com/kuszmaul/SuperMalloc/tree/master/tests>
- \[6] Timothy Crundal. _Reducing Active-False Sharing in TCMalloc._
2016.<http://courses.cecs.anu.edu.au/courses/CSPROJECTS/16S1/Reports/Timothy*Crundal*Report.pdf>. CS16S1 project at the Australian National University.
- \[7] Alexey Kukanov, and Michael J Voss.
_The Foundations for Scalable Multi-Core Software in Intel Threading Building Blocks._
Intel Technology Journal 11 (4). 2007
- \[8] Paul Liétar, Theodore Butler, Sylvan Clebsch, Sophia Drossopoulou, Juliana Franco, Matthew J Parkinson,
Alex Shamis, Christoph M Wintersteiger, and David Chisnall.
_Snmalloc: A Message Passing Allocator._
In Proceedings of the 2019 ACM SIGPLAN International Symposium on Memory Management, 122–135. ACM. 2019.
