Mathieu Desnoyers [Mon, 18 Mar 2024 15:04:56 +0000 (11:04 -0400)] 
mempool: do not overwrite same per-cpu values
If the per-cpu items already have the correct values, do not write to
the pages. This eliminates useless COW in populate-all pools over the
zero page.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I4e47db82ccf15c3cca30a8df48d367cbc05f3887
Mathieu Desnoyers [Mon, 18 Mar 2024 11:28:08 +0000 (07:28 -0400)]
mempool cow test: use default stride
Fix ppc64 which has a 64kB page size.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I2debd1c50f1ecc0486a557b69af3fda70bae2dcd
Mathieu Desnoyers [Sun, 17 Mar 2024 20:31:11 +0000 (16:31 -0400)]
mempool: Test COW vs malloc_init race
Test that the entire malloc init value is visible in CPU mappings. If
the COW page copy race vs init happens while init is in the middle of
storing to the newly allocated area, iteration on all CPUs comparing the
visible content to the init value is responsible for detecting and
mitigating uninitialized or partially initialized init value from the
point of view of a CPU. Validate that this scheme has the intended
effect wrt a concurrent COW caused by storing to a nearby per-cpu area
on the same page.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Iebde10eb61e80cc0d8eb0ebc89925f4a2ad9de9c
Mathieu Desnoyers [Sun, 17 Mar 2024 02:43:48 +0000 (22:43 -0400)]
mempool tests: Adapt to free list new location
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: If582864dcc7aceea99f866ad3ca0f6391d6ca5c5
Mathieu Desnoyers [Sun, 17 Mar 2024 02:42:48 +0000 (22:42 -0400)]
Move robust pool free list to own memory area
Consumes more memory in the robust pool, but removes blind spots for
validation of poison due to aliasing with free list.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ia8187cf3049aad8e54daaced4005fbe2e841ae7e
Mathieu Desnoyers [Sat, 16 Mar 2024 21:16:57 +0000 (17:16 -0400)]
mempool: Fix off_t cast warning on 32-bit
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I93978e7f0f583ae4a9bf62d3f707258c385bb0a8
Mathieu Desnoyers [Sat, 16 Mar 2024 20:23:45 +0000 (16:23 -0400)]
mempool test: test populate none policy
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I2eb34565ba84ae9479eb235743fbba07fe277165
Mathieu Desnoyers [Thu, 14 Mar 2024 01:50:56 +0000 (21:50 -0400)]
mempool: Implement populate none policy
Implement lazy-populate policy (populate none) relying on kernel COW of
init values pages.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ie4467f1806208271a5462605445abc2f0c1c1d38
Mathieu Desnoyers [Fri, 15 Mar 2024 19:37:56 +0000 (15:37 -0400)]
mempool: Fix leak on pool destroy
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I3534084f1055b0db73ac9ec5005a6c88d8056d78
Mathieu Desnoyers [Thu, 14 Mar 2024 00:22:08 +0000 (20:22 -0400)]
mempool: Introduce mmap_addr and mmap_len range fields
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ia91571ba9f3535402f456074973875503e688690
Mathieu Desnoyers [Sun, 10 Mar 2024 19:14:08 +0000 (15:14 -0400)]
mempool_test: test rseq_mempool_percpu_malloc_init
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ia266d69a4007340ffe1d98950619ce90582c2e40
Mathieu Desnoyers [Sun, 10 Mar 2024 19:06:16 +0000 (15:06 -0400)]
mempool: implement malloc_init
Add a *_malloc_init to the API to allow allocating memory initialized to
a given initial value.
This replaces application patterns where a loop on all possible CPUs is
needed after allocation to initialize memory to a given nonzero value,
and will eventually allow doing the initialization lazily (the first
time the cpu touches memory within the pool).
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I3acb1505d7b4f46ceb6a8128d2ebda346b8e359c
Mathieu Desnoyers [Sun, 10 Mar 2024 03:03:49 +0000 (22:03 -0500)]
mempool_test: test poison corruption detection
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ia284caa599f2c9c7c142efe6508779d36d828e17
Mathieu Desnoyers [Sat, 9 Mar 2024 22:02:40 +0000 (17:02 -0500)]
mempool: Use default poison value when robust is set
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I84a75bd14f6798261fd630aaf076cc3981bad0a1
Mathieu Desnoyers [Sat, 9 Mar 2024 21:55:24 +0000 (16:55 -0500)]
mempool: check poison on destroy
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I40fe7d4179e42a392f3528cc9697552fe7c4ad09
Mathieu Desnoyers [Sat, 9 Mar 2024 21:44:17 +0000 (16:44 -0500)]
mempool: Detect poison corruption on alloc
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I1d4181f089aa08d1c14466751661bba8a3727cf5
Mathieu Desnoyers [Sat, 9 Mar 2024 21:43:40 +0000 (16:43 -0500)]
mempool test: add one extra object after free
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ie4b5f47ac4e2108cecb079c63767259ee0493a48
Mathieu Desnoyers [Sat, 9 Mar 2024 21:14:54 +0000 (16:14 -0500)]
mempool: introduce poison attribute
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I056c0d8c4f0d5b1d2c401592a08b22ff6f239d11
Mathieu Desnoyers [Sat, 9 Mar 2024 20:40:22 +0000 (15:40 -0500)]
mempool test: Only use a single range for large stride
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I62baaa751d46abf0341e4e1c3dffecd1b2402986
Mathieu Desnoyers [Sat, 9 Mar 2024 20:38:54 +0000 (15:38 -0500)]
mempool test: test max_nr_ranges > 1
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I7d3b48690c5e61789f0bc5f50db27898499c335a
Mathieu Desnoyers [Sat, 9 Mar 2024 20:26:12 +0000 (15:26 -0500)]
mempool: Introduce rseq_mempool_attr_set_max_nr_ranges
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I5417673d16ce7cfa5e8f2b9c120978f8633664bc
Mathieu Desnoyers [Sat, 9 Mar 2024 19:59:34 +0000 (14:59 -0500)]
mempool: Introduce support for ranges linked list
Each pool can allocate additional ranges when the initial range is full.
There is currently a limit of 1 range (to be made configurable in the
future).
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ibc59f140a8dc1ac9152b0a4ae82d32db76a2c42e
Mathieu Desnoyers [Sat, 9 Mar 2024 20:09:00 +0000 (15:09 -0500)]
mempool: remove unused FIRST_POOL
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I545c7f2ff7f4b760c92bf0689b5f4edbe1894329
Mathieu Desnoyers [Sat, 9 Mar 2024 18:39:31 +0000 (13:39 -0500)]
mempool: cpu=-1 for init_func argument (global pool)
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I956006f59186252d3dcc767fa21e5928f68d5cb8
Mathieu Desnoyers [Sat, 9 Mar 2024 16:40:05 +0000 (11:40 -0500)]
mempool: init_func can return an error
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I212fd5a6ec6ae7580a28068de4ee88d7955da9d0
Mathieu Desnoyers [Sat, 9 Mar 2024 15:46:15 +0000 (10:46 -0500)]
mempool: aligned_mmap_anonymous: support non-power-of-2 nr_cpus
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I7231ee9ac6e822b9d43dae596763d84b0abfcbe1
Mathieu Desnoyers [Sat, 9 Mar 2024 14:40:27 +0000 (09:40 -0500)]
rseq: Introduce rseq_get_max_nr_cpus
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ie7d24470994b3a6ec805bfec7111ff3492b46997
Mathieu Desnoyers [Sat, 9 Mar 2024 14:31:29 +0000 (09:31 -0500)]
param_test: Use mempool max_nr_cpus rather than CPU_SETSIZE
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I6401e5584ee97ea8958898e377e1ec52476814c6
Mathieu Desnoyers [Sat, 9 Mar 2024 14:14:39 +0000 (09:14 -0500)]
mempool: Introduce rseq_mempool_get_max_nr_cpus
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ibd5a7abf40e56d8e57c7b51a5dd9495c51bd9624
Mathieu Desnoyers [Sat, 9 Mar 2024 13:59:57 +0000 (08:59 -0500)]
mempool: Introduce rseq_mempool_range_init_numa helper
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Icc0ac582fbd21a71f01212d23b86a9ecd5679465
Mathieu Desnoyers [Sat, 9 Mar 2024 13:35:44 +0000 (08:35 -0500)]
mempool: Introduce init_func attribute
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I1e36bae465b0f459b4b6a4f8182cf524e0e6ade5
Mathieu Desnoyers [Sat, 9 Mar 2024 13:24:49 +0000 (08:24 -0500)]
mempool: reenable rseq_mempool_range_init_numa
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I8c7d3ccdf0279cca44e8ad33f8904e34b38f8a79
Mathieu Desnoyers [Sat, 9 Mar 2024 03:48:44 +0000 (22:48 -0500)]
smp.c: fix includes and macro namespace
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I5ef3c45240120f7f415c3bb38a280e3a7ceb410a
Mathieu Desnoyers [Sat, 9 Mar 2024 03:16:21 +0000 (22:16 -0500)]
Mempool: default as global
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I1a2c44188404e1703a9996a83495086d2b0e3ca3
Mathieu Desnoyers [Sat, 9 Mar 2024 02:47:22 +0000 (21:47 -0500)]
mempool: Receive mempool type (percpu/global) as attribute
Also introduce a smp.c/h helper to get the number of possible cpus,
copied from libside (MIT).
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I4dfc8fa56ae29e5586eb029377abad62c93de6a9
Mathieu Desnoyers [Sat, 9 Mar 2024 02:16:33 +0000 (21:16 -0500)]
mempool test: iterate from page size
Fix test on ppc64, which has a 64kB page size.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I6e309855c190b056d886aa3c187a9dc1929d7391
Mathieu Desnoyers [Sat, 9 Mar 2024 01:55:08 +0000 (20:55 -0500)]
mempool: update comments in header file
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I35a017716d5f49e2eb827cd7ed4f56855d0f8e1c
Mathieu Desnoyers [Sat, 9 Mar 2024 01:33:25 +0000 (20:33 -0500)]
mempool: reduce default RSEQ_PERCPU_STRIDE to 64kB on 64-bit
aarch64 has issues with mmap() of 16MB * 1000 cpus (ENOMEM).
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: If22aab4f9afacaf9bdf0318c8876486cdda2ec25
Mathieu Desnoyers [Sat, 9 Mar 2024 00:24:13 +0000 (19:24 -0500)]
membarrier rseq fence test: adapt to RSEQ_PERCPU_STRIDE
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ifaa59012cf19695eabe9977cae73a281e34afaa7
Mathieu Desnoyers [Sat, 9 Mar 2024 00:20:14 +0000 (19:20 -0500)]
Add missing rseq_after_asm_goto in rseq_load_add_load_load_add_store__ptr
Change-Id: If2a966e39302d5163203769855ed0d5f5e6012a7
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Sat, 9 Mar 2024 00:18:39 +0000 (19:18 -0500)]
Revert "x86-64: Implement/use load_cbne_load_add_load_add_store"
Now that the percpu mempool allocator has a stride known in advance, it
is possible to use the offset approach in the critical section to index
by cpu number.
This reverts commit
d87b0cd222a1c11751be7f8b655fc0dc45b7dc12.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I8b394650f1dbe409eb9ed3db88afcc43cd230133
Mathieu Desnoyers [Fri, 8 Mar 2024 22:22:07 +0000 (17:22 -0500)]
mempool: namespacing, add global alloc/free
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I6b3a6f9948f8400ae691738d7caa15164e634b74
Mathieu Desnoyers [Fri, 8 Mar 2024 21:32:22 +0000 (16:32 -0500)]
mempool: Introduce optional stride parameter
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ic7119bb33f86cf72c73c2d50dbe55e4769ce0b22
Mathieu Desnoyers [Fri, 8 Mar 2024 21:13:09 +0000 (16:13 -0500)]
mempool: mass renaming of API
- rseq_percpu_pool -> rseq_mempool
- rseq_pool_attr -> rseq_mempool_attr
- rseq_percpu_pool_set -> rseq_mempool_set
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ic34a4163854e91838b2f0727d26469d55c69a0e0
Olivier Dion [Fri, 8 Mar 2024 15:59:57 +0000 (10:59 -0500)] 
tests/mempool_test: Add robust testing
Change-Id: I0e7c9dd7ad03070f0524edb4252cb11988418108
Signed-off-by: Olivier Dion <odion@efficios.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Fri, 8 Mar 2024 20:53:27 +0000 (15:53 -0500)]
mempool: check for NULL pool on destroy
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I1a095ac04c4545095ed53699402a30d2d1ace8a7
Mathieu Desnoyers [Fri, 8 Mar 2024 20:51:24 +0000 (15:51 -0500)]
mempool: update header comments
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I7648cd668ddb4a18dd6d04d021198833bfc078b5
Mathieu Desnoyers [Fri, 8 Mar 2024 20:47:25 +0000 (15:47 -0500)]
mempool: Remove unused defines
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I35f3ce87da25c47097bb76ab9c3aeadee46f5e32
Mathieu Desnoyers [Fri, 8 Mar 2024 20:44:37 +0000 (15:44 -0500)]
mempool: Remove pool array and lock
Those are not needed anymore with the new mask-based scheme.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Id0d401d21b4ac3b8b7c7e50fecf6bd92513bd162
Mathieu Desnoyers [Fri, 8 Mar 2024 20:34:01 +0000 (15:34 -0500)]
mempool: Use masks to get pool range from pointer and stride for percpu offset
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I45bb460be66bc4843ead316d7b870958cd1217c2
Mathieu Desnoyers [Fri, 8 Mar 2024 19:52:31 +0000 (14:52 -0500)]
Memory map ranges aligned on the stride
This will allow finding the range header with a mask applied on the
pointer in free().
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I25821ebb4a7333fe53658932f58c6d9cad1bd64a
Mathieu Desnoyers [Fri, 8 Mar 2024 19:07:11 +0000 (14:07 -0500)]
percpu alloc: Introduce stride parameter
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I6469725da2610560872d2fe678128455d7ff22f1
Olivier Dion [Fri, 8 Mar 2024 17:29:36 +0000 (12:29 -0500)]
rseq-mempool: Call set_alloc_slot() on all paths
set_alloc_slot() was not called when getting free slot from free-list.
Change-Id: I21d633ced6592039fe39e0ae121128958048f80e
Signed-off-by: Olivier Dion <odion@efficios.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Fri, 8 Mar 2024 16:37:05 +0000 (11:37 -0500)]
mempool: implement one range
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I189ee936cd11229b5770ed95c8ac32b67fede20d
Mathieu Desnoyers [Fri, 8 Mar 2024 14:51:41 +0000 (09:51 -0500)]
Update mempool test
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I57fe799bcdb4df1d6d6fb6e605e17cf581f13197
Mathieu Desnoyers [Fri, 8 Mar 2024 14:48:30 +0000 (09:48 -0500)]
Add mempool test to make check
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ia998f1157083df566e8e8e9c331983bab256880f
Mathieu Desnoyers [Fri, 8 Mar 2024 14:25:16 +0000 (09:25 -0500)]
mempool test: safe iteration for free
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Id5fe8f73dff0c914646a76af45738a2e80a8e2b2
Mathieu Desnoyers [Fri, 8 Mar 2024 13:51:33 +0000 (08:51 -0500)]
Introduce mempool test
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Fri, 8 Mar 2024 12:04:19 +0000 (07:04 -0500)]
Rename percpu-alloc files to mempool
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I0a1ba042649cf3b2224a90a05e095733e9a597fc
Mathieu Desnoyers [Fri, 8 Mar 2024 12:06:31 +0000 (07:06 -0500)]
Update gitignore
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I55d5e3571e7115cec5d09fac52ce51fa3601c12d
Mathieu Desnoyers [Thu, 7 Mar 2024 22:01:53 +0000 (17:01 -0500)]
percpu alloc: Remove RSEQ_POOL_ROBUST attribute
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: If90ce5767d0d2516674efda08fb0d694b4f8d1a2
Olivier Dion [Thu, 7 Mar 2024 21:14:36 +0000 (16:14 -0500)]
percpu-alloc: Add move_pages batching
Change-Id: I5e9f40d73f45f2abeded1d1f62056a478fd8455a
Signed-off-by: Olivier Dion <odion@efficios.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Thu, 7 Mar 2024 21:12:37 +0000 (16:12 -0500)]
percpu pool: Move robust flag to atttribute
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ie83fff22b71dc9be39d90e4be97d094b90542715
Mathieu Desnoyers [Thu, 7 Mar 2024 20:57:40 +0000 (15:57 -0500)]
percpu pool: mmap set attr: return error value
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Id2debf769299ee1d6624097b38e79f916b8c0f2b
Mathieu Desnoyers [Thu, 7 Mar 2024 20:53:35 +0000 (15:53 -0500)]
percpu pool: Introduce generic attributes
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I7641218c7f69b73fc998eddf2839312fcf5726e9
Mathieu Desnoyers [Thu, 7 Mar 2024 19:03:07 +0000 (14:03 -0500)]
percpu alloc: add pool name
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I886c840f394cb1de0b5ab71b7417b99675ea1468
Mathieu Desnoyers [Thu, 7 Mar 2024 18:47:07 +0000 (13:47 -0500)]
Fix typos in fprintf
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ic82493c649a2389391808f079d9ed58c81e61018
Olivier Dion [Thu, 7 Mar 2024 18:16:03 +0000 (13:16 -0500)]
percpu-alloc: Add free-list check on pool destroy
Change-Id: I68bab4673aabde92dc0d8321de2d5561c5e7a11c
Signed-off-by: Olivier Dion <odion@efficios.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Thu, 7 Mar 2024 17:39:34 +0000 (12:39 -0500)]
rseq percpu: print caller ip on double-free/alloc corruption and leak
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I5b1bc7a2a5711e6c9de4bc82df92a49df5654f8b
Mathieu Desnoyers [Thu, 7 Mar 2024 17:33:24 +0000 (12:33 -0500)]
Fix: add missing \n in fprintf
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ibbb61e3b7f66f8d9b7ed125e4123581b2c1b93d9
Mathieu Desnoyers [Thu, 7 Mar 2024 17:32:36 +0000 (12:32 -0500)]
Implement rseq_hweight_ulong with __builtin_popcountl
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I483d53c7c5402e034b8d468e0ed2ab048184f069
Mathieu Desnoyers [Thu, 7 Mar 2024 17:29:50 +0000 (12:29 -0500)]
Fix destroy_alloc_bitmap size
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ic53da70a6fbcbbdd87c885d01d80a5ba65befb5a
Mathieu Desnoyers [Thu, 7 Mar 2024 17:24:43 +0000 (12:24 -0500)]
percpu alloc: Improve robust pools
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I4966965cca5143640347a4b21fef3e2c353c8366
Olivier Dion [Thu, 7 Mar 2024 15:39:29 +0000 (10:39 -0500)]
percu-allocator: Implement robust pool validation
When set, the RSEQ_POOL_ROBUST flag will add a bitmap to the pool which
is used to track double-free and allocation leaks in the pool.
Signed-off-by: Olivier Dion <odion@efficios.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I531782ee57d33c9704b9cca44af1a9464f60cb90
Mathieu Desnoyers [Tue, 5 Mar 2024 16:13:55 +0000 (11:13 -0500)]
percpu: remove rseq_percpu_pool_ptr_offset
The rseq_percpu_pool_ptr_offset API is now unused, and it leaks
implementation details about per-cpu pools which may prevent
implementing auto-extensible pools efficiently in the future.
Remove it.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ib62c4af34c04f9c4e5227dab2f487356478b32fa
Mathieu Desnoyers [Tue, 5 Mar 2024 15:06:51 +0000 (10:06 -0500)]
x86-64: Implement/use load_cbne_load_add_load_add_store
Change the current load_add_load_load_add_store for a
load_cbne_load_add_load_add_store, which allows validating that the
percpu pointer did not change since it was loaded from C.
This allows precomputing the address of the percpu memory area in C and
provide it as a second pointer argument.
The comparison approach is prefered to the offset-from-pointer approach
because it does not leak implementation details of the percpu allocator.
Add missing rseq_after_asm_goto() in the static inline function.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I3fe9d57d13f7507d5af95ef37391ad36fe2221fe
Mathieu Desnoyers [Tue, 5 Mar 2024 15:10:53 +0000 (10:10 -0500)]
Remove riscv has_load_add_load_load_add_store implementation
The implementation is currently buggy: RSEQ_ASM_OP_R_DEREF_ADDV()
should increment the address _target_ as a final store, not just the
address within the temporary register.
Remove it for now.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I0f2a854cec4ab5e58ea6cbb74d40ca29b256d27e
Mathieu Desnoyers [Tue, 5 Mar 2024 15:31:19 +0000 (10:31 -0500)]
param test: membarrier: validate total number of increments
While reviewing the riscv implementation for this test, I noticed that
the final store is only incrementing the content of its register,
without ever storing it to memory.
This passes testing because the increment is effectively a no-op, and
the test never validates that any increment happen in the first place.
Introduce a validation of the increment total to eliminate those
false-positives.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ie9ee1c7902509da75dd85babd07bc180ad0b9ae9
Mathieu Desnoyers [Tue, 5 Mar 2024 13:42:54 +0000 (08:42 -0500)]
percpu allocator: Add flags argument for future extensions
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I4c507a99351c450ef2c6396af5391f019f8e3d91
Mathieu Desnoyers [Mon, 4 Mar 2024 21:26:50 +0000 (16:26 -0500)]
percpu alloc: introduce rseq mmap attributes
Introduce rseq mmap attribute argument to pool creation, allowing
users to override the memory allocator used to allocate/free the pool
memory.
Move NUMA page move to a separate rseq_percpu_pool_init_numa() API.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ie08c7250fce5f567c7a4aee469e010c5dcf39bc6
Mathieu Desnoyers [Mon, 4 Mar 2024 18:47:34 +0000 (13:47 -0500)]
Cleanup: Remove whiteline
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I537b82707f1f9a95005da672b98196ab44875e2a
Mathieu Desnoyers [Mon, 4 Mar 2024 16:50:11 +0000 (11:50 -0500)]
Rename RSEQ_ARCH_HAS_OFFSET_DEREF_ADDV to rseq_arch_has_load_add_load_load_add_store
Fix param test to use rseq_arch_has_load_add_load_load_add_store
rather than rseq_arch_has_load_cbne_load_add_store.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I373c91dc9178dd69ed159d568090e8c732cf4485
Mathieu Desnoyers [Mon, 4 Mar 2024 16:43:18 +0000 (11:43 -0500)]
Revert "Introduce rseq_load_cbne_load_add_store pseudocode"
This reverts commit
3eff770448484493fa15ea90db643f86c124db6c.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Mon, 4 Mar 2024 16:43:08 +0000 (11:43 -0500)]
Revert "x86: Introduce rseq_load_cbne_load_add_store"
This reverts commit
18f21b27f01b9147cd95c538ece07f95c37150a2.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ieffcd50965fe6dcf2c7d067f5fddb4e9f62e82f3
Mathieu Desnoyers [Mon, 4 Mar 2024 16:42:17 +0000 (11:42 -0500)]
Revert "rseq.h: Introduce rseq_load_cbne_load_add_store__ptr"
This is not needed anymore.
This reverts commit
040147907ff31cdc279568ffcb74197650430e4b.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Mathieu Desnoyers [Mon, 4 Mar 2024 16:36:03 +0000 (11:36 -0500)]
param tests: membarrier: pre-decode pointer offset
Loading "cpulist->head" outside of the rseq critical section is
unintended: loading cpulist->head loads memory from the per-cpu list
outside of the rseq critical section, which happens to work because the
memory is not actually freed in this specific test case, but would break
if memory would be reclaimed after the rseq fence.
So go back to the originally used rseq_load_add_load_load_add_store__ptr(),
and use the new rseq_percpu_pool_ptr_offset() to pre-calculate the
offset.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I5e621c724e143e38a6dfabd26bbf682a6fcc5182
Mathieu Desnoyers [Mon, 4 Mar 2024 16:34:41 +0000 (11:34 -0500)]
percpu alloc: introduce rseq_percpu_pool_ptr_offset
Introduce rseq_percpu_pool_ptr_offset to pre-decode the offset from
all __rseq_percpu pointers for a given pool. This is useful to prepare
offsets that would need to be calculated within rseq critical sections.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I42d2669ed5ed81d74aae551b721c9cb04b25fe45
Mathieu Desnoyers [Mon, 4 Mar 2024 15:08:14 +0000 (10:08 -0500)]
Document percpu alloc
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I8fa4b68e8a68c57da8ab726d0f68ee12aad2f15e
Mathieu Desnoyers [Mon, 4 Mar 2024 14:47:32 +0000 (09:47 -0500)]
rseq percpu: Remove __rseq_percpu annotation from returned pointer
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Iea04ce173cc529612b1024393794a304b0e7e219
Mathieu Desnoyers [Sun, 3 Mar 2024 17:08:15 +0000 (12:08 -0500)]
x86-32: ip-relative: return to post-call address
This simplifies the assembler returns to the expected (speculated)
address.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I68c59f72ac32011d37cc9399414ff328f7b8fdd0
Mathieu Desnoyers [Sat, 2 Mar 2024 21:57:58 +0000 (16:57 -0500)]
param test: x86-32: use ip-relative addressing for RSEQ_INJECT_ASM
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Id8349df846f5b0491b348b603f0423b726c55b8c
Mathieu Desnoyers [Sat, 2 Mar 2024 21:57:32 +0000 (16:57 -0500)]
x86-32: Save ip reference for ip-relative addressing
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ie95e1bf409f0ca0c271f529b694839905d4d0d91
Mathieu Desnoyers [Sat, 2 Mar 2024 17:57:28 +0000 (12:57 -0500)]
x86-32: Use ip-relative addressing to store rseq_cs
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I294a4844bf8022be480059a028b9a17a5a2dcc4b
Mathieu Desnoyers [Sat, 2 Mar 2024 16:54:28 +0000 (11:54 -0500)]
x86-32: work-around register pressure limitations
Use explicit register clobbers and load input values from "m" input
operands to reduce register pressure.
Note that old gcc does not support output operands for asm goto, so we
cannot simply re-use an input register as output.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I745d27f416ef303b1049e8c12b122e5b4a271f63
Mathieu Desnoyers [Sat, 2 Mar 2024 16:19:51 +0000 (11:19 -0500)]
x86-32: Document linker warning
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I35e1db5eda78ea4b6349039ed34d739b33b81d6d
Mathieu Desnoyers [Sat, 2 Mar 2024 16:08:25 +0000 (11:08 -0500)]
arm: work-around register pressure limitations
The recent move to percpu allocator breaks the 32-bit ARM build due to
register pressure. Fix this by using explicit register clobbers and
loading input values from "m" input operands.
Note that old gcc does not support output operands for asm goto, so we
cannot simply re-use an input register as output.
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I1d546e427c7f6b62035433815806c8ec275a55f8
Mathieu Desnoyers [Sat, 2 Mar 2024 01:13:52 +0000 (20:13 -0500)]
param tests: percpu memcpu buffer: use percpu alloc
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I86c975aa8578469f685dcdb4951f4af50f06b3fe
Mathieu Desnoyers [Sat, 2 Mar 2024 00:58:22 +0000 (19:58 -0500)]
param tests: percpu buffer: use percpu alloc
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I49f29134679931eed4e06140a72cc56592cc12b8
Mathieu Desnoyers [Fri, 1 Mar 2024 19:55:24 +0000 (14:55 -0500)]
param tests: percpu list: use percpu alloc
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: Ie1415e33187902e76dd3d492cd8ed2fe9b980565
Mathieu Desnoyers [Sat, 2 Mar 2024 00:44:59 +0000 (19:44 -0500)]
rseq.h: Introduce rseq_load_cbne_load_add_store__ptr
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@efficios.com>
Change-Id: I5011d4115d5dc0064b0d3fc359c037dcda02508c
This page took 0.040096 seconds and 4 git commands to generate.
projects / librseq.git / log
