622 Commits

Author SHA1 Message Date
daan
394b796ea0 fix over-eager page reset in segment reclamation 2020-01-25 13:43:56 -08:00
daan
4faf412f53 move 'memory.c' to 'region.c' 2020-01-25 13:28:49 -08:00
daan
7785139201 fix warning on gcc on attribute ignore in templates 2020-01-25 13:28:24 -08:00
daan
62b8fb26b1 fix freeing of segments on forced reclaim 2020-01-25 13:27:47 -08:00
daan
cb433d272a Merge branch 'dev' into dev-exp-abandon 2020-01-25 13:03:06 -08:00
daan
f4630d43a7 allow reset on large pages; check commit status before reset 2020-01-25 12:49:14 -08:00
daan
5e32d00aab add visit count to abandoned to limit list length 2020-01-25 12:26:08 -08:00
daan
3ec691cfb4 Merge branch 'dev' of https://github.com/microsoft/mimalloc into dev 2020-01-25 11:20:17 -08:00
David Carlier
2b667bd3ae enable arc4random abi under apple 2020-01-25 14:47:09 +00:00
daan
072ceb2efe Merge branch 'dev' into dev-exp-abandon 2020-01-24 20:20:55 -08:00
daan
ecece57284 fix bug in committed check in arena allocation 2020-01-24 20:20:43 -08:00
daan
47300eeda3 avoid memset 2020-01-24 20:17:33 -08:00
daan
b31bc52618 add cache alignment directives for contended variables 2020-01-24 19:02:13 -08:00
daan
e070eba112 fix tagged null encoding, search segment cache before reclaim 2020-01-24 16:31:18 -08:00
daan
9a400ecce9 Merge branch 'dev' into dev-exp-abandon 2020-01-24 15:48:34 -08:00
daan
4ae51096ec add warning on region exhaustion 2020-01-24 15:45:03 -08:00
daan
28c14d99c3 clean up comments 2020-01-24 11:03:12 -08:00
daan
e68293741e fix assertion, add check for page committed before doing reset 2020-01-23 21:44:32 -08:00
daan
58fdcbb0cd fix bug in collect where has_page was not set on free pages 2020-01-23 21:37:14 -08:00
daan
4a2a0c2d50 initial abandon based on fine-grained reclamation 2020-01-23 19:50:35 -08:00
daan
12701b1aac do not reclaim segments on collect 2020-01-23 15:48:51 -08:00
daan
76e727f7d1 fix assertion on page destroy 2020-01-23 15:39:38 -08:00
daan
0316859e06 improve codegen for mi_free 2020-01-23 10:03:03 -08:00
daan
66818bf632 use atomic yield on delayed-freeing; clarify code 2020-01-23 09:57:55 -08:00
daan
3bbbe6c686 enable atomic yield when delayed_freeing is encountered 2020-01-22 20:12:42 -08:00
daan
0193a15f7e nicer message on huge OS page reservation 2020-01-22 19:46:43 -08:00
daan
e3b16fe4ef reduce type casts in atomic operations 2020-01-22 19:44:54 -08:00
daan
3957b2fd28 add mi_new_realloc(n) to support C++ style reallocation that raises std::bad_alloc on out-of-memory 2020-01-20 15:41:56 -08:00
daan
b77be05e40 only collect retired at fresh page allocation 2020-01-20 12:14:34 -08:00
daan
e8d7c80c74 fix build warnings on linux 2020-01-19 17:33:36 -08:00
daan
41e717c2e0 fix assertion in mi_block_zero_init (issue #194) 2020-01-18 20:30:12 -08:00
daan
dc58388968 Add ability to register custom error function called on various error conditions; including ENOMEM 2020-01-17 19:59:55 -08:00
daan
6dd636d82d improve STL allocator using mi_new_n and removing unused parameter names; follow up from pr #193 and #188 2020-01-17 15:41:52 -08:00
daan
6b7356a10a make mimalloc compile with the highest warning level on msvc 2020-01-16 18:12:01 -08:00
daan
c6037ac8f2 fix assertion index 2020-01-16 16:56:02 -08:00
daan
3f8b8b7a55 initialize thread_delayed_free field atomically 2020-01-16 16:45:34 -08:00
daan
da709bcf11 test for gcc version for aligned new/delete overrides 2020-01-16 16:30:44 -08:00
daan
7a98a461a3 fix type of bitmap field to avoid C++ error on older gcc 2020-01-16 16:21:59 -08:00
daan
7a9502973d rearrange STL allocator code: remove pragma, ifdef for C++11 2020-01-16 15:57:11 -08:00
daan
4e91eab8fc specialize mi_mallocn for count=1 2020-01-16 14:12:37 -08:00
daan
8d8f355ed0 add option to reset eagerly when a segment is abandoned 2020-01-16 11:25:02 -08:00
daan
b8072aaacb fix debug build 2020-01-16 03:54:51 -08:00
daan
9629d73188 fix options 2020-01-15 18:07:29 -08:00
daan
7a1e86fc20 merge from dev 2020-01-15 18:03:37 -08:00
daan
ad32eb1dfb eager collect on page reclamation 2020-01-15 17:57:19 -08:00
daan
0099707af9 use delayed free for all pages; reduce size of the page structure for improved address calculation 2020-01-15 17:19:01 -08:00
daan
c9b5ac80b3 update page reset queue to just do delayed page resets 2020-01-15 12:00:44 -08:00
daan
783e3377f7 add output argument to stat printing 2020-01-15 10:53:54 -08:00
daan
0956a05bf6 add argument pointer to the register deferred free callback 2020-01-15 10:21:32 -08:00
daan
f92a2a7264 add argument pointer to the register output routine 2020-01-15 10:18:32 -08:00
daan
941c55ee42 wip: first implementation of page free list on segments for effecient delayed page reset 2020-01-14 21:47:18 -08:00
daan
65f4f5144b fix out-of-bounds error in huge OS page bitmap 2020-01-13 17:06:25 -08:00
daan
5f61a9e896 add mprotect error when the mmap limit might be reached in secure mode (see issue #77) 2020-01-09 17:52:28 -08:00
daan
8f75444e7a fix windows debug build at MI_DEBUG=2 2020-01-08 23:21:32 -08:00
daan
12ef2816ed fix bug exposed by commit 59fa286 where reclaimed pages could be stuck to NEVER_DELAYED 2020-01-08 19:00:03 -08:00
daan
683d8998d4 fix potential A-B-A problem with segment abandonment; noticed by Manual Poeter and Sam Gross 2020-01-08 17:45:38 -08:00
daan
50b3f6d7ae fix assertion 2020-01-08 16:33:45 -08:00
daan
d4ab0ff08c fix timeout on huge page reservation if set to 0 2020-01-07 14:15:37 -08:00
daan
4223caac0f on Linux dynamically detect if getrandom is supported and fall back to /dev/urandom if needed 2020-01-06 22:08:21 -08:00
daan
d8d69c2c94 disable MAP_NORESERVE on huge pages 2020-01-05 22:07:16 -08:00
daan
45582d1fb5 revert a2a9230 (remove empty page removal on search): this is not generally valid when concurrent frees do not always add to thread_delayed_free. 2020-01-05 13:58:49 -08:00
daan
59fa286294 fix bug where continue would wrongly exit the do-while loop for delayed freeing 2020-01-05 13:52:07 -08:00
daan
a2a9230ad6 remove empty page removal on page search (no longer needed with retired collection and delayed freeing) 2020-01-03 22:52:52 -08:00
daan
d596f08569 fix thread_free read in assertion 2020-01-03 22:06:27 -08:00
daan
2b108c8748 increase retire expiration to 4 2020-01-03 21:39:18 -08:00
daan
eeb623e6af increase retire limit, collect retired pages 2020-01-03 17:06:41 -08:00
daan
9629a0190f fix eager commit on large pages (issue #182) 2020-01-02 17:25:00 -08:00
daan
1b5a08cd25 remove unused parameter in check double free 2020-01-02 17:24:32 -08:00
daan
384808e340 Merge branch 'dev-random' into dev 2019-12-28 16:19:45 -08:00
daan
fc3e537bd4 improve double free detection with faster same page check 2019-12-28 15:28:13 -08:00
daan
77134e1ad0 update free list encoding to stronger formula with addition last 2019-12-28 15:17:49 -08:00
daan
e3391d9a53 stronger encoding of free lists using two keys per page 2019-12-28 00:57:42 -08:00
daan
ce02986d56 variable renaming 2019-12-27 22:30:23 -08:00
daan
6f0b35d513 merge from master 2019-12-24 10:44:49 -08:00
daan
e05a1edc03 fix large OS page size on Linux (issue #184, due to fix for #179) 2019-12-24 10:32:44 -08:00
daan
ba87a39d9f updated random cookie generation using OS primitives and chacha20 2019-12-22 19:37:49 -08:00
daan
f9b942d80d fix compilation of region descriptor on 32-bit 2019-12-22 17:08:46 -08:00
Daan Leijen
e31e609414 add preload check in option initialization (issues #179) 2019-12-01 01:03:39 -08:00
daan
36d168a2d9 add preload check to options initialization 2019-12-01 00:03:35 -08:00
daan
a407f35c64 add arena.c into the static override object 2019-12-01 00:01:14 -08:00
daan
1d998af854 clean up options; make secure work with eager_page_commit 2019-11-25 10:47:17 -08:00
daan
1643273357 fix unix bug in decommit size 2019-11-25 10:11:29 -08:00
daan
c6df7a199c experimental eager page commit option 2019-11-24 22:02:40 -08:00
daan
4452431b6c reenable segment cache and fix initial segment commit 2019-11-24 15:25:19 -08:00
daan
727d33b96f more precise memory reset 2019-11-24 14:40:47 -08:00
daan
4a0d35afd0 improve secure guard page allocation to work with non-eager commit 2019-11-23 11:59:19 -08:00
daan
6568059cc6 merge from dev 2019-11-21 16:28:28 -08:00
daan
74dbfc30be improved security by encoding NULL values; double free mitigation on by default; more precise free list corruption detection 2019-11-21 15:21:23 -08:00
daan
1674d551ff add verbose message with secure build level 2019-11-20 20:46:13 -08:00
Daan Leijen
049dbf41ba fix commit bits for huge page allocations 2019-11-20 15:44:07 -08:00
Daan Leijen
211f1aa519 remove reset delay slots; add reset tracking per page and segment 2019-11-20 14:55:12 -08:00
Daan Leijen
30e2c54adb remove delayed reset option (for now) 2019-11-20 14:13:02 -08:00
Daan Leijen
56b9fac4bf merge from dev 2019-11-20 13:31:04 -08:00
Daan Leijen
4d4a2885f5 use atomic read/write on the page->heap field where concurrent interaction is possible 2019-11-20 13:19:17 -08:00
daan
17ade3fca5 merge from dev 2019-11-13 17:52:20 -08:00
daan
a4ed63d127 Adresses pr #165 and issue #164 by @colesbury:
On Mac OS, the thread-local _mi_default_heap may get reset before
_mi_thread_done is called, leaking the default heap on non-main threads.

Now the current default heap is also stored in mi_pthread_key (or mi_fls_key
on Windows). The _mi_thread_done function is called with this value.
2019-11-13 17:22:03 -08:00
Daan Leijen
d01ed42bcb replace max_numa_nodes by use_numa_nodes (to help with wrong detection of numa nodes on WSL for example) 2019-11-13 13:35:50 -08:00
Daan Leijen
bdb8274819 change max_numa_node to max_numa_nodes option 2019-11-12 12:04:43 -08:00
daan
d4f54dcf30 remove numaif dependency on linux 2019-11-12 10:37:15 -08:00
daan
867d78f877 reserve huge OS pages earlier on at process_init 2019-11-12 10:19:52 -08:00
daan
af746ca4c1 inline bitmap_mask 2019-11-12 10:17:39 -08:00
daan
ef179a6377 avoid allocation at numa node detection on linux 2019-11-12 10:16:59 -08:00
daan
165ee45845 initialize delay slots for the main thread 2019-11-11 17:31:48 -08:00
daan
a0958b2da6 enable more reset delay slots 2019-11-11 17:06:16 -08:00
daan
5e6754f3f7 track commit status per block in a region 2019-11-11 15:45:31 -08:00
daan
db3f1c4bfa add commit info to arenas 2019-11-11 15:26:11 -08:00
daan
741a80256d merge from dev-exp 2019-11-11 14:51:16 -08:00
daan
2bb058bd25 remove cache_reset parameter 2019-11-11 14:44:32 -08:00
daan
534e1e39ef allow allocation in committed regions even if not requested 2019-11-11 14:42:29 -08:00
daan
93a6463383 only allow commit delay for small and medium objects 2019-11-11 14:16:45 -08:00
daan
83a066fd2d remove reset_decommits option 2019-11-11 09:46:02 -08:00
daan
21bbb1be87 fix warnings 2019-11-10 12:36:55 -08:00
daan
181bef382c merge dev-exp 2019-11-10 11:39:53 -08:00
Daan Leijen
9f08ddd0d0 refactor regions; add commit tracking on a segment basis 2019-11-09 19:30:53 -08:00
Daan Leijen
7b72a4cd50 fix region suitable bug 2019-11-08 11:55:43 -08:00
daan
6ccfb5a93e merge from dev 2019-11-07 11:00:50 -08:00
daan
9b65388807 fix space leak in secure mode where a non-null free list would be discarded 2019-11-07 10:59:19 -08:00
daan
27f1a8b3d2 fix avg display; set secure default to 0` 2019-11-07 10:35:30 -08:00
daan
36090dee5c Merge branch 'dev' into dev-exp 2019-11-07 10:34:06 -08:00
daan
31d11f64d5 fix secure free list extension where a non-empty initial free list was discarded 2019-11-07 10:33:45 -08:00
daan
378716c467 refactor and improve atomic bitmap usage 2019-11-07 10:26:52 -08:00
daan
b09282bc0d change arena allocator to atomic bitmap as well 2019-11-06 22:49:01 -08:00
daan
00e19cad9a refactor region code, split out atomic bitmap 2019-11-06 21:37:23 -08:00
daan
2887266063 optimize get numa node for single node systems 2019-11-06 14:17:36 -08:00
daan
829fd872f4 initial delay slots 2019-11-04 11:48:41 -08:00
daan
3d0a1e249f remove all floating point types and arithmetic 2019-11-04 09:40:10 -08:00
daan
c38af8f7c1 merge d1d65fbc: make max error messages configurable 2019-11-04 08:49:59 -08:00
daan
aece753dce fix output during preloading enabling stderr only after the crt has loaded 2019-11-04 08:46:17 -08:00
daan
8afd06b248 use int64 for time (instead of double) 2019-11-04 08:44:40 -08:00
daan
9d6a5acb22 fix unix build warnings 2019-11-03 13:34:54 -08:00
daan
d1d65fbca4 make max error messages configurable 2019-11-03 13:25:41 -08:00
daan
520a8dafee divide huge pages more even 2019-11-03 13:25:28 -08:00
daan
f36ec5d9d8 reserve huge pages incrementally 2019-11-03 13:16:07 -08:00
daan
e320488791 add numa nodes to stats 2019-11-03 12:18:32 -08:00
daan
723fbba259 fix output during preloading enabling stderr only after the crt has loaded 2019-11-03 12:18:20 -08:00
daan
62cd0237fc fix aligned huge page allocation on windows 2019-11-02 17:49:34 -07:00
daan
ee323aabac fix vs2017 build 2019-11-02 15:56:21 -07:00
daan
fd9d8c85ae change numa support on linux to use getcpu 2019-11-02 11:55:03 -07:00
daan
70748ee1ee fix missing semi colon 2019-11-02 10:39:09 -07:00
daan
a69016c33e improve and document numa support 2019-11-02 10:30:16 -07:00
daan
2c12d7f223 optimized numa calls; better Linux support 2019-11-01 22:01:52 -07:00
daan
57dd69265a normalize numa node 2019-11-01 20:30:01 -07:00
daan
2d10c78587 fix linux compilation 2019-11-01 20:19:00 -07:00
daan
3fadf4abaf initial numa awareness for regions 2019-11-01 20:01:08 -07:00
daan
a6499be074 initial numa support for arenas 2019-11-01 19:53:07 -07:00
daan
aaf01620f4 improve allocation of the huge OS page arena 2019-10-31 19:39:49 -07:00
daan
d36d04b4a6 add arena for huge page management 2019-10-31 15:35:10 -07:00
daan
8725a88fba fix assertion in debug secure mode 2019-10-30 15:22:40 -07:00
daan
4a4d74927c protect against double-free in multi-threaded free list 2019-10-30 14:53:21 -07:00
daan
6cf16b1201 fix reset error on windows when disabling eager commit option 2019-10-30 14:32:28 -07:00