# Gigablock series — plan
**This document describes two series.** The **prequel** (§13's Phase P) is pageblock evacuation: it
stands alone upstream, needs nothing from the rest, and has its own measurable claim. The **gigablock
series** is everything else and builds on it. Wherever both appear below, which one owns a piece is
stated.
2026-08-06. Canonical and self-consistent as of this date. Superseded decisions, and the scenario
that killed each, are in `gigablock-plan-history.md` and `gigablock-plan-v1-archaeology.md`. Neither
is safe to cite from: their section numbers overlap this file's and mean different things.
Base for the series is `scratch/riel/gigablock` at `0f23d56f17fd` (linus-upstream, 7.3-rc1 merge
window), which has `848acc8ffe1b` as an ancestor. Code facts below were derived against
`e1914add2799`. Re-verified on the new base: §12a's seven `zone->free_area` references in
`page_alloc.c`, still exactly seven in the same seven functions. **Everything else still needs
re-verifying** — `mm/page_alloc.c` differs by 422 lines from `e1914add2799` and `mm/vmscan.c` by 464. Measurements are from
`~/debug/gigablock-phase0/` on devvm34971, node 0 Normal zone.
## 1. Problem, and the measured baseline
### 1.0 Why 1 GB pages, and why they cannot be allocated on demand
**The kernel allocates two kinds of memory, and only one of them can be moved.** Most of a machine's
memory is allocated on behalf of userspace — page cache, anonymous memory — and the kernel knows every
mapping of it, so it can copy such a page elsewhere and repoint the page tables. That is migration, and
it is what compaction and `alloc_contig_range()` are built on. The kernel also allocates for itself:
slab objects, page tables, per-cpu areas, DMA buffers. Those generally cannot move, because some other
part of the kernel holds a pointer to the object and there is no way to find and update every such
pointer. Everything below follows from that split.
**A TLB entry covers one page whatever its size, so page size is TLB reach.** One entry maps 4 KB, or
2 MB, or 1 GB. The TLB is a small fixed hardware resource, so a workload whose working set exceeds
`entries x page_size` pays a page walk on the misses, and the walk itself is shorter for a huge page: a
1 GB mapping terminates at the PUD, skipping the PMD and PTE levels a 4 KB mapping has to load. Under
virtualization both effects are multiplied, because a guest TLB miss walks two page tables rather than
one.
Page table memory follows the same ratio. A 1 GB region mapped at 4 KB needs 262144 PTEs; mapped at PUD
level it needs one entry. For a database or a VM with a large, sparsely touched heap that is the
difference between page tables measured in gigabytes and page tables that fit in cache.
**What a 1 GB page costs to produce is 262144 physically contiguous, PUD-aligned pages, and one page of
the second kind anywhere in that range is fatal.** Movable content is migrated out of the way, which is
work but always possible. A kernel allocation is not: it stays where it is for as long as it lives, and
one slab object or page table in a 1 GB range disqualifies the range for that whole time.
**There is no free list for 1 GB, and that is a deliberate tradeoff rather than an oversight.** The
buddy allocator keeps one free list per order and searches upward from the requested order —
`__rmqueue_smallest()` loops `current_order` from `order` to `NR_PAGE_ORDERS`. Every extra order
lengthens that loop for every allocation that has to climb it, and enlarges the per-zone `free_area[]`
array. What those extra orders would hold is almost nothing: blocks much above a handful of orders
rarely arise by chance on a running system, and even order 5 is uncommon. Paying an iteration on every
allocation to check lists that are nearly always empty is a bad trade, so `MAX_PAGE_ORDER` is 10 and the
largest buddy block is 4 MB.
So gigantic pages are not popped off a list; they are **assembled** by `alloc_contig_pages()`, which
isolates a range and migrates whatever movable content is in it. That is why the requirement is not
"free" but **free or evacuable** — and why one unmovable page defeats it, since evacuation has nothing
to offer content that cannot move.
**The existing anti-fragmentation machinery does not reach this scale.** Migratetype grouping keeps
unmovable allocations in unmovable pageblocks, and a pageblock is 2 MB. That is what makes 2 MB THP
work. Nothing in it arranges for **512 consecutive pageblocks** to stay free of kernel content, which is
what a 1 GB range is. Worse, when a kernel allocation cannot find a pageblock of its own type,
`try_to_claim_block()` steals a movable one and leaves behind whatever it could not move, so kernel
content spreads rather than concentrating.
**So 1 GB pages work today only if they are reserved at boot**, through `hugepagesz=1G hugepages=N` or
CMA, which takes the memory out of general use whether or not anything needs it. Allocating them on
demand — growing a VM, starting a database, backing a guest after the host has been up for a week — is
what does not work, and the measurements below say why.
### 1.1 The measured baseline
Confine kernel allocations so that PUD-aligned 1 GB ranges stay free or evacuable, and 1 GB
hugepages can be allocated reliably.
Node 0 Normal, 822 GB present, 766 PUD ranges, 391808 pageblocks. **The three do not reconcile and
want re-deriving**: 391808 pageblocks is 765 GB and 766 PUD ranges is 766 GB, so the span the gigablock
formula divides is about 766 GB, which is what yields 12 GB gigablocks. Where 822 GB comes from is not
established here; the derived numbers below use the span.
free 25.9%
movable (LRU/anon) 61.8%
kernel (upper bound) 12.3% 101 GB, 24.7M pages
PUD ranges holding non-free non-LRU content 766 / 766 (100%)
PUD ranges with kernel pages in MOVABLE pbs 763 / 766 (99.6%)
ideal packing of kernel content 94 PUDs (12.3%)
ranges lost to placement alone 672
Pollution is universal but thin: per range min 0.95%, p10 1.69%, median 10.73%, p90 19.79%. Only 3
ranges are under 1% polluted, so it is not a few ranges absorbing the damage.
Where the kernel pages sit, by pageblock migratetype (pages):
pageblock free lru/anon slab reserved flagless
UNMOVABLE 5340945 289171 2309989 1946 4318301
MOVABLE 41342579 124333200 4779242 3281450 1954376
RECLAIMABLE 3921532 355517 8216906 320 59965
Four numbers drive the design:
- **39 GB of kernel pages sit in MOVABLE pageblocks**, in 99.6% of ranges. Stopping that leak is the
series' job; pageblock-type accounting alone understates the problem.
- **36 GB of free memory is stranded in kernel-typed pageblocks.** Space to absorb kernel
allocations already exists and is not being used, which is why fresh pageblocks keep being claimed.
- **12.8 GB of the leak is the vmemmap `struct page` array**, allocated from memblock before
pageblock types are set and permanently pinned.
- **A third of the kernel pages are in RECLAIMABLE pageblocks**, shrinkable rather than
unmigratable, so the two need separate accounting.
The 12.3% is an upper bound: it counts reclaimable slab, `movable_ops` pages (zsmalloc/zswap pool,
~529k pages here, migratable), and vmemmap. Refining it needs a `page_has_movable_ops()` check plus
slab-cache attribution in the drgn pass.
### 1.2 The current mechanism does not contain fragmentation, measured two ways
The machinery this series replaces is migratetype fallback plus `should_try_claim_block()`, so the
case for the series rests on showing it does not work.
**Observational.** All 766 PUD ranges hold non-free non-LRU content while the kernel content would
fit in 94. 672 ranges lost to placement alone.
**Causal, in a VM, in 36 seconds.** `~/debug/gigablock-phase0/vm-fallback-experiment/`, unmodified
kernel, ftrace hist triggers only. Normal zone pageblock census before and after a concurrent
kernel-plus-movable churn workload:
Unmovable 36 -> 642 (+606)
Reclaimable 4 -> 22 (+18)
Movable 31192 -> 30568 (-624)
Kernel content held at the end: 171 MB unreclaimable slab, 0.5 MB page tables, 41 MB reclaimable
slab. Densely packed, 172 MB of unmovable content is 86 pageblocks. **It occupied 642 — 7.5x more
territory than content.** Every traced fallback was a kernel allocation converting a movable
pageblock, and 306 of 317 were order 0:
alloc_mt=UNMOVABLE <- MOVABLE, change_ownership=1 : 309
alloc_mt=RECLAIMABLE <- MOVABLE, change_ownership=1 : 7
alloc_mt=RECLAIMABLE <- UNMOVABLE : 2
alloc_mt=MOVABLE <- anything : 0
`should_try_claim_block()` returns true unconditionally for UNMOVABLE and RECLAIMABLE at any order,
so one order-0 kernel allocation that misses its own free list converts a whole 2 MB pageblock. The
sentence for the cover letter: **the current policy spends territory in 2 MB units to satisfy 4 KB
requests, and nothing bounds how much it spends.**
**And half the conversions cannot currently be seen.** The census moved 624 blocks against 318
traced conversions. The residual comes from paths that emit no tracepoint — `change_pageblock_range()`
on the `__free_one_page()` merge path, and the highatomic reserve and unreserve paths.
`set_pageblock_migratetype()` and `set_pfnblock_flags_mask()` are both inlined, so they cannot be
kprobed either. Closing that gap is Phase -1.
## 2. Design in one page
- A **gigablock** is `round_up(zone_span / 64, PUD_SIZE)`, at most 64 per zone, 12 GB on this host.
Boundaries are PUD-aligned and gigablocks tile the zone, so every pfn in a gigablock zone is inside
exactly one and its free pages live on that gigablock's lists. A gigablock is a free-list container
and search unit, nothing more.
- **Gigablocks exist in `ZONE_NORMAL` only, when it spans at least 8 of them, for memory present at
boot.** On this host that is one zone of five, so `zone->gb == NULL` is the common case and the path
through it stays exactly mainline's, in cost as well as behaviour. `ZONE_DMA` and `ZONE_DMA32` are
device-constrained memory that 32-bit devices depend on, and a 1 GB page from there is worth nothing.
`ZONE_MOVABLE` needs no confinement because every page in it is migratable already. `ZONE_DEVICE` and
hot-added memory are out of scope for v1. Below 8 gigablocks a tainted set and a clean set have no
room to coexist. Details in §10.3.
- **What is tracked per gigablock is how much kernel content it holds**, as a count of pageblocks.
A gigablock is *tainted* by its first kernel pageblock, and graded by how full it is.
- **What is tracked per pageblock is content class**, in three bits: `PB_has_unmovable`,
`PB_has_reclaimable`, `PB_has_movable`. All three clear means no allocated content.
- **Whether a PUD range can host a 1 GB page is derived, never stored**: the range must lie wholly
in this zone, contain no CMA pageblock, and have no pageblock with either kernel bit set.
- **Every choice is a bitmap query.** Kernel allocations search the tainted set, movable searches
the clean set first, and the one positional decision — which clean gigablock to spoil — takes the
one with the least left to lose. No sweep directions and no cursors, because the bitmap states
directly what direction was only a proxy for.
- **Headroom watermarks** per tainted gigablock decide whether a sleepable kernel allocation must
evacuate first and whether background evacuation is kicked. They do **not** gate movable: movable
competes only for content-free pageblocks, and §6.2's barrier decides its access to those.
- **Tainted describes a gigablock, content bits describe a pageblock.** A gigablock is tainted when
`nr_kernel_pageblocks > 0`; a pageblock holds content classes. The two are never the same statement.
- **Evacuation creates space rather than selecting it**: it migrates movable content out, and never
reclaims. This is the load-bearing mechanism; the bitmaps are the cheap index telling it where to work.
Success metric: the number of PUD ranges passing the derived candidate test, against the 94 that
§1 says the kernel content actually needs.
**What the prequel does, and does not, contain.** The pageblock content bits and the pageblock-scale
evacuation helper are useful without any of the above and ship first (§13 Phase P). Their claim is
narrower and entirely local: a pageblock stolen for kernel use should become usable by that use, rather
than staying half-full of movable pages that push the next kernel allocation into stealing another one.
Nothing in the prequel knows what a gigablock is.
## 3. Data structures
Shapes, sizes, who writes and who reads each field are tabulated once in §3.0; every later mention of a
name resolves against that table.
struct zone {
...
struct gigablock_data *gb; /* NULL when this zone has none */
};
struct gigablock_data {
u8 nr_gigablocks; /* <= 64 */
unsigned long gb_size_pages; /* multiple of PUD_PAGES */
unsigned long start_pfn; /* PUD-aligned */
/* one bit per gigablock, so exactly one word each */
unsigned long has_free[MIGRATE_TYPES][NR_PAGE_ORDERS];
unsigned long gb_tainted; /* nr_kernel_pageblocks > 0 */
unsigned long gb_fullness[2]; /* §3.4, two buckets */
unsigned long gb_above[2][__NR_GB_WATERMARKS]; /* per class, §6.1 */
unsigned long gb_evac_failed; /* TAINTED_FULL (§6.2) */
unsigned long gb_evac_retry_at; /* jiffies, §6.2 retry */
struct gigablock gb[]; /* nr_gigablocks entries */
};
struct gigablock {
struct free_area free_area[NR_PAGE_ORDERS];
u32 nr_free; /* free pages */
u16 nr_kernel_pageblocks; /* pageblocks the kernel cannot move */
u32 nr_free_kernel; /* free pages on kernel lists, §6.1 */
};
Per pageblock, three bits in the existing flat `pageblock_flags` array. One accessor performs the
list operation, both counters and the `has_free` bit flip, so they cannot diverge.
**The four levels, because the search unit and the protected thing are not the same size.** A gigablock
is 12x coarser than the PUD range a hugepage needs, which is why a gigablock's fullness says nothing
about how many of its ranges survive (§4):
zone ~766 PUD ranges, 64 gigablocks
+-- gigablock 12 GB the search unit: one bit in every mask of §3
+-- PUD range 1 GB what a 1 GB hugepage needs: 12 per gigablock
+-- pageblock 2 MB 512 per PUD range, 6144 per gigablock
+-- page 4 KB
Total per zone: `has_free` is 6 x 11 words = 528 bytes, plus `gb_tainted`,
`gb_evac_failed`, `gb_evac_retry_at`, two fullness words and eight watermark words of which seven are
used, so 632 bytes of index against `zone->free_area[]`'s 1144 bytes — still under half the size of the structure it
indexes.
### 3.0 Shape, size and who touches what
The structures above are declared once and then referred to by name for the rest of this document:
`has_free` appears in six later sections, `gb_tainted` in seven. A reader meeting a name in §5.1 cannot
tell from the surrounding text whether it is one bit per gigablock or one per pageblock, and the two are
6144x apart. This table is the one place that answers that, and every later mention resolves against it.
Reference host throughout is §1's: a `ZONE_NORMAL` spanning ~766 GB, 766 PUD ranges and 391,808
pageblocks, tiled by 64 gigablocks of 12 GB. Sizes below use the tiling, so 6144 pageblocks per
gigablock and 64 x 12 GB = 768 GB of tiling over a slightly smaller span; the last gigablock is the one
that overhangs (§10.2a).
**The index — one word per mask, one bit per gigablock.** In `gigablock_data`, reached through
`zone->gb`. Every mask is exactly one `unsigned long`, because §3.2 caps gigablocks at 64.
| name | size | written by | read by | path |
|---|---|---|---|---|
| `has_free[mt][order]` | 528 B | the free-list accessor, with the list op and both counters | §4.1, §5.1, §5.2, §5.4 | hot |
| `gb_tainted` | 8 B | `gigablock_account_pageblock()` on the 0 crossing (§3.4) | §4, §5.1, §5.2 | hot |
| `gb_fullness[2]` | 16 B | `gb_get_fullness()` on the one boundary crossing (§3.4) | §5.1, §5.2 ranking | hot |
The fullness numerator differs by population and is easy to swap: `nr_kernel_pageblocks/total` for a
tainted gigablock, grading concentration; `(total - nr_free)/total` for a clean one, grading how little
is left to protect (§3.4).
| `gb_above[2][4]` | 64 B, 7 used | `gigablock_watermark_update()` (§6.1, §6.2) | §4.1, indexed by request class | hot |
| `gb_evac_failed` | 8 B | set by evacuation on failure; cleared by the free accessor and the 10 s retry (§6.2) | §4.1, evacuation trigger | hot read, hot clear, slow set |
| `gb_evac_retry_at` | 8 B | the evacuation trigger (§6.2) | same, lazily | slow |
| total | 632 B | | | |
**What the allocation fast path actually reads:**
elig = gb_tainted & gb_above[class][LOW] & ~gb_evac_failed; /* 3 words */
cand = elig & has_free[mt][order]; /* 1 word */
gb = rank_by_fullness(cand); /* 2 words */
Six `unsigned long` loads, no per-pageblock access and no counter reads. Only one word of `has_free`'s
528 B is touched per query, the `[mt][order]` cell, so **the five single-word masks should be laid out
adjacent and ahead of `has_free`**: the query then costs one cache line plus one line of `has_free`,
where the other order costs two lines for the same six words. The free side adds one conditional clear
of `gb_evac_failed`, almost always already clear.
**Per gigablock**, same allocation as the index (§10.1a):
| name | size | written by | read by | path |
|---|---|---|---|---|
| `free_area[NR_PAGE_ORDERS]` | 1144 B | the free-list accessor | `pfn_free_area()`, `for_each_free_area()` | hot |
| `nr_free` | 4 B | the accessor | §3.4 for clean gigablocks, §7 | hot |
| `nr_kernel_pageblocks` | 2 B | `gigablock_account_pageblock()` (§3.3) | `gb_tainted`, §3.4, §5.0, §5.1, evacuation ranking (§8) | hot write, slow read for ranking |
| `nr_free_kernel` | 4 B | the accessor, beside `nr_free` | `headroom_pages` (§6.1) | hot |
| per gigablock | 1154 B | | | |
| x64 | 72 KB | | | |
**Per pageblock is a different array, and this is the distinction the table exists for.** The three
content bits are not in `gigablock_data`; they are three spare bits in the pre-existing flat
`pageblock_flags`, one entry per pageblock, 384 KB for this zone. `__NR_PAGEBLOCK_BITS` is 5 today and
`NR_PAGEBLOCK_BITS` is `roundup_pow_of_two(5)` = 8, so the design uses all three spare bits (§3.2).
Per-pageblock reads happen in the background contraction pass that re-derives stale-set bits (§3.3),
and in the deferred claim cursor of §4a if it is ever built. The allocation-path search of §4.1 touches
none of it.
**Not structures, though §6 and §7 read like they are:** `headroom_pages` and `headroom_blocks` are
quantities computed per gigablock (§6.1). `gb_above[]` is the min over both, folded into a mask at
update time, which is why it has four levels and not seven.
**Assumptions these sizes rest on.** A change to one invalidates the numbers above rather than quietly
staling them:
- **8 bits per pageblock**, giving 384 KB of `pageblock_flags` and 6 KB of content bits per gigablock.
One `unsigned long` per pageblock instead would make those 3 MB and 48 KB and remove the
three-spare-bit ceiling; it would not touch §4.1.
- **At most 64 gigablocks per zone**, which is what makes every mask one word and what §10.1's sizing
exists to hold.
- **6144 pageblocks per gigablock.** §4a's deferred cursor is a `u16`, so gigablocks above 128 GB would
need a wider one.
- **`struct free_area` is 104 B**, six `list_head` plus `nr_free`. The 1144 B and 72 KB figures move if
`MIGRATE_TYPES` changes.
**Which patch introduces what** (§13), so an implementer knows what must exist before a name is usable:
| patch | introduces |
|---|---|
| 1 | `gigablock_data`, `zone->gb`, sizing, `nr_gigablocks`, `gb_size_pages`, `start_pfn` |
| 2 | `pfn_to_gigablock()`, `zone_has_gigablocks()`, accessors |
| 3 | the three `PB_has_*` bits, maintenance, init sweep, `PageOffline` and hwpoison hooks |
| 4 | `nr_free`, `nr_free_kernel`, `nr_kernel_pageblocks`, the taint threshold |
| 5 | `has_free[mt][order]` — written, not read |
| 9-10 | `free_area[]` moves into `struct gigablock` |
| 17 | `gb_above[]`, `gb_fullness[2]`, `gb_evac_failed` and `gb_evac_retry_at` become load-bearing |
### 3.1 Why a pointer rather than fields in `struct zone`
**The layout argument is the strongest and it is not about memory.** `struct zone` has
`CACHELINE_PADDING(_pad1_)` before `free_area[]` and `_pad2_` after `lock` and
`trylock_free_pages`, deliberately isolating the write-intensive allocator fields. Inlining 544 bytes
there moves `lock` and `trylock_free_pages` onto different cache lines than today, on every zone of
every node including those that will never have a gigablock. That is an unrelated performance change
smuggled into the series. A pointer perturbs nothing.
**NULL is a better "absent" than a count.** `zone_has_gigablocks()` becomes a NULL test, the boot
ordering problem gets a natural "not built yet", and every scope exclusion in §10 collapses to leaving
the pointer NULL. The `!CONFIG` stub is a NULL too.
**One allocation, contiguous.** Bitmaps and gigablock array come from a single allocation, so the index
and the data it indexes share pages and there is one lifetime to reason about. At boot that allocation
is `memblock_alloc_node()`; see §10.1a.
Costs, to measure rather than dismiss: one extra dependent load on the allocator fast path, and two
cache lines instead of one on first touch. Mitigate by placing `zone->gb` inside the existing `_pad1_`
region next to `free_area` and `lock`, so the pointer arrives on a line the allocator has already
touched. One load against the ~18 cache lines mainline's ascending `free_area` scan already touches.
### 3.2 Maintain a bitmap only if it is read on a hot path and cannot be derived
Every maintained bit is a coherence bug waiting to happen. Each has to earn its place against two
questions: is it read often enough that deriving it would cost something, and can it be derived from
state already maintained.
**What survives.** `has_free[mt][order]` is read on the allocation fast path and cannot be derived
without walking the free lists. `gb_tainted` and the `gb_fullness` masks are read by every
fallback query and although all are functions of `nr_kernel_pageblocks`, deriving a 64-bit *mask* would
mean reading 64 counters. `gb_tainted` flips once per gigablock lifetime and a gigablock moves between
fullness masks only on a bucket crossing, so both are cheap to keep.
**What was dropped, and why, so nobody adds it back.**
- **A per-PUD "clean" bitmap.** Nothing on any hot path consults it; the 1 GB target search and the
headline metric are its only consumers, both slow paths, and both can run the derived test of §10.2
at 512 bytes of reads per candidate range.
- **A `gb_eligible` bitmap.** §10.2's three-condition test covers every case, and two of the three
conditions are structural constants the test can check directly.
- **A fourth pageblock bit for "no content".** `__NR_PAGEBLOCK_BITS` is 5 today and
`NR_PAGEBLOCK_BITS` is `roundup_pow_of_two(5)` = 8, so there are exactly **three spare bits**. A
fourth would round to 16 and **double `pageblock_flags`** — 392 KB to 784 KB per zone on this host.
It is also unnecessary: "no allocated content" is precisely the state where all three content bits
are clear.
- **A derived `has_free_any[order]`** to save the ORs in §4's queries. A bit that is the OR over
migratetypes can only be cleared once every migratetype's list at that order is empty, which needs
a count or a rescan per clear. Two ORs at query time are cheaper.
- **A `has_content_free_pageblock` bitmap.** A content-free pageblock is a fully free one, and a fully
free pageblock coalesces to `pageblock_order` — §3.3 already sets `PB_has_unmovable` on reserved,
offline and poisoned pages, so the cases that cannot coalesce are not content-free either. It is
therefore the OR of `has_free[mt][order]` over every migratetype and every order at or above
`pageblock_order`, which is 12 words on this configuration:
static bool gb_content_free_pageblock(gbd, order_rows)
{
unsigned long m = 0;
for (o = pageblock_order; o < NR_PAGE_ORDERS; o++)
for (mt = 0; mt < MIGRATE_TYPES; mt++)
m |= gbd->has_free[mt][o];
return m;
}
Keeping it as a bitmap would trade those reads for a write on every pageblock content transition, on
a line every CPU freeing into the zone touches. Reads scale, contended writes do not, and the rows
being read are written anyway for the search. The minimal set is also easier to justify upstream.
- **Group population counts and a zone-level summary word**, and any third level above them. Those
exist to make a *wide* level-0 bitmap cheap to scan. With at most 64 gigablocks the level-0 bitmap
is one `unsigned long`, so a search is one load and one `find_next_bit()`, and every level above it
is machinery with nothing to do.
### 3.3 The three content bits
Introduced by the prequel (§13 P2), where their consumer is the evacuation trigger. The gigablock series
adds readers: `nr_kernel_pageblocks`, the taint threshold, and the derived candidate test of §10.2.
has_unmovable has_reclaimable has_movable meaning
0 0 0 no allocated content
0 0 1 movable only: evacuable
0 1 0 reclaimable only: shrinkable
0 1 1 reclaimable + movable: both mechanisms
apply, can still reach empty
1 0 0 unmovable only: permanent
1 0 1 unmovable + movable: blocks a hugepage,
evacuation cannot fully empty it
1 1 0 mixed kernel: shrinking cannot fully
empty it
1 1 1 all three: neither mechanism empties it
Eight states in three bits, which is every distinction §5, §6 and §8 need, at zero additional memory.
**The dividing line is `has_unmovable`.** The four states with it clear can still reach "no allocated
content" — shrinking empties the reclaimable part, evacuation migrates the movable part — so those
pageblocks are recoverable supply for `headroom_blocks`. The four with it set cannot, whatever runs.
That is the distinction §5.1 step 5's asymmetry turns on: putting content into a pageblock that already
has `has_unmovable` set costs nothing, putting unmovable content into one that does not costs its
recoverability.
**Reserved, offline and poisoned pages set `PB_has_unmovable`.** The bit means "this pageblock holds
memory the kernel cannot move", not "an unmovable allocation happened here", and the largest source of
unmovable content on a real machine never goes through the page allocator: `PageReserved` from
memblock covers the vmemmap, kernel image, initrd, crashkernel, firmware regions, and the hole pages
`init_unavailable_range()` marks.
Without tainting them, a gigablock holding nothing but vmemmap reports zero kernel pageblocks, is
classed clean, and is then *protected* as a 1 GB candidate it can never satisfy. Set them at init in
one sweep over `for_each_reserved_mem_range()` — a handful of ranges, not a walk over pages — ordered
after the gigablock array exists. `PageOffline` and hwpoison need the same treatment at their own
hooks; `memory_failure()` is the hook for the latter, and a single poisoned page kills a range
permanently.
**Clearing is conservative and corrected on a slow path.** All three bits clear together when a
pageblock has no content left. That is exact for the common case but leaves one hole: a pageblock whose
kernel content was freed while movable content remains keeps `PB_has_unmovable` set, so
`nr_kernel_pageblocks` never decrements and the gigablock never returns to clean — precisely the
"kernel had a lot of memory and then freed it" case. Per-content-class counts would fix it exactly and
cost a 784 KB per-zone array. Instead: **the bits are a hot-path hint that may be stale-set and never
stale-clear, and the background contraction pass re-derives them** for the gigablock it examines.
Contraction happens, just not instantly.
### 3.4 Taint on first contact, graded by fullness
gb_tainted = nr_kernel_pageblocks > 0
GB_MOSTLY_DEDICATED at or above ~25% of that class
GB_MOSTLY_EMPTY below it
For a tainted gigablock the numerator is `nr_kernel_pageblocks`, since concentrating kernel content is
what the bucket is steering. For a clean one it is `total - nr_free`, which measures how little is left
to protect. This is the `gb_lists[category][fullness]` structure from the v28 superblock implementation,
and both halves of it are load-bearing.
**Taint on first contact, because a threshold smears.** Under a 25% rule a kernel allocation spilling
into a clean gigablock leaves it clean, at 1 pageblock out of 6144. Nothing then attracts the *next*
kernel allocation there, so it opens a new front, and the one after that a third. Kernel content spreads
across many gigablocks, every one below the threshold, none tainted, all unable to host a 1 GB page. The
threshold meant to bound the damage spreads it instead. Tainting on the first kernel pageblock makes the
gigablock a magnet: the second allocation follows the first.
**Fullness, because taint alone latches.** A bare `> 0` bit is a one-way write-off of 12 GB triggered by
one stray page. A boot-time burst or a percpu allocation touching every gigablock would taint the whole
zone and leave no clean set, so the design would do nothing. The fullness bucket restores the gradation
the bit throws away: a gigablock tainted by one allocation is `TAINTED, MOSTLY_EMPTY` and sorts last
among tainted gigablocks, so it is still the one the kernel reaches for last, still worth protecting, and
still the prime evacuation candidate — without needing a second classification to say so. The vmemmap
case makes the point concretely: the gigablock holding the vmemmap's last 0.84 GB is 7% kernel and holds
eleven good PUD ranges, and it lands in `TAINTED, MOSTLY_EMPTY`, which is exactly where it belongs.
Write-off therefore stops being a predicate. "Beyond saving" is exhaustion, not a bucket:
no content-free pageblock and an empty `has_free` row, which is §9.3's own split of density as
a cheap predictor and exhaustion as the hard gate. Everything short of that shades into "recoverable",
so the decisions that used to test `gb_tainted` — protect it, admit movable, target evacuation — test
the bucket instead.
**Two buckets, because every consumer wants an extreme rather than a rank.** Steps 1a, 1b, 7, 8 and the
spill all ask for the fullest; evacuation asks for the emptiest; nothing asks to tell 50-74% from 25-49%.
Five buckets, which is what v28 used, cost five word ANDs per ranking and four boundary crossings per
gigablock as it fills, each crossing a clear and a set under `zone->lock`. Two cost two ANDs and one
crossing, and the magnet argument above survives intact at a 25% boundary.
**Evacuation ranks on the counter, not on a mask.** Inside `MOSTLY_EMPTY` the cost of evacuating spans
three orders of magnitude, one kernel pageblock against 1474 at 24%. But `gigablock_evacuate()` is the
sleepable path choosing one gigablock, so reading 64 `u16` counters is 128 bytes. §3.2's "deriving a
64-bit mask would mean reading 64 counters" is a fast-path argument and does not apply here; ranking on
`nr_kernel_pageblocks` directly is finer than five buckets ever were, with less state.
**The two boundaries are independent constants.** The numerator is `nr_kernel_pageblocks/total` for
tainted, grading concentration, and `(total - nr_free)/total` for clean, grading how little is left to
protect. Different questions, so the values are not tied to each other. ~25% each is a starting point to
measure, not a constant to defend.
**Hysteresis is deferrable.** A gigablock hovering at a boundary clears and sets two mask words per
alloc-free pair. Adding hysteresis needs no new state, since the previous bucket is already recorded by
which mask holds the bit, so it is a comparison change in one function and can land whenever the
counters show it is worth having.
**Open: the bitmap index caps gigablocks at 64, which is what forces 12 GB granularity here.** §3.2's
one-word-per-mask design needs at most 64 gigablocks per zone, so §10.1 sizes them at
`round_up(zone_span / 64, PUD_SIZE)` — 12 GB, twelve PUD ranges, on a 768 GB machine. v28 had no such
cap because it used lists rather than masks, and its gigablock was one PUD range. The bitmap buys O(1)
mask queries and costs granularity on exactly the machines the series targets; per-fullness lists buy
granularity and cost list operations under `zone->lock`. Measure before assuming the bitmap wins.
## 4. Search mechanism
**One load and one `find_next_bit()` per query.** With at most 64 gigablocks every per-gigablock
bitmap is a single `unsigned long`, so there is no hierarchy, no population count, no summary level and
no cursor. `[mt][order]` is the right layout because both scans that matter iterate orders for a fixed
migratetype: 11 contiguous words, 88 bytes, 2 cache lines covering every order.
**Each population searches its own set.** Kernel allocations search `gb_tainted & ...`; movable
searches `~gb_tainted & ...` first and considers tainted gigablocks only under the §6 watermark rules.
That is what direction used to approximate — "prefer the gigablocks that are already lost" — so the
bitmap replaces it, and no cursor or sweep order is needed. Contiguity of the clean set is worth
nothing here either, because a 1 GB hugepage needs one PUD range and ranges are independent.
**The queries are where the policy lives.** `has_free[mt][order]` answers only "which gigablock has a
free block of this order and type", which is step 1 below. Every fallback step needs a different
candidate set, and handing those steps to mainline's `__rmqueue_claim()` would be gigablock-blind and
would claim in a clean gigablock — the one thing the design exists to prevent. So each step names its
set:
fb = fallbacks[start_mt] /* {RECLAIMABLE, MOVABLE} for UNMOVABLE */
step 1 elig & has_free[start_mt][order], then rank by fullness (§4.1)
elig = gb_tainted & gb_above[...] & ~gb_evac_failed
step 1' same, with ~gb_tainted then fullest clean
step 2 gb_tainted & gb_content_free_pageblock()
step 3 gb_tainted & (has_free[fb[0]][o] | has_free[fb[1]][o])
step 4 as step 3, largest o that hits
step 5 gb_tainted & (has_free[fb[0]][order] | has_free[fb[1]][order])
step 7 ~gb_tainted & gb_content_free_pageblock() last resort
spill ~gb_tainted, fullest bucket first which clean one to spoil
Two or three word operations per step, all on paths that are already the slow ones. A fallback query
touches fewer cache lines than mainline's existing ascending `free_area` scan does on its fast path
today, so the index is a reduction rather than a cost to apologise for.
**Which clean gigablock to spoil is a quality question, not a positional one.** Take the fullest
bucket — least left to protect, so the fewest usable candidate ranges to lose. A gigablock with several
structurally doomed ranges is a better target than a pristine one.
**Order-of-orders is unchanged from upstream:** ascending for the matching migratetype, descending on
the claim path where the largest block is the existing proxy for "the pageblock with the most free
pages". Both are orthogonal to the gigablock dimension.
**Which pageblock inside a gigablock: today it is whatever the list head offers, and that is not good
enough.** A claim takes `get_page_from_free_area(&gb->free_area[pageblock_order], mt)`, so the pageblock
is whichever was freed most recently — arbitrary with respect to position.
The gigablock is 12x coarser than the thing being protected. A 12 GB gigablock is 6144 pageblocks and 12
PUD ranges, and 1536 kernel pageblocks — the bucket boundary, exactly 3 ranges' worth — say nothing about
where those pageblocks sit. Packed into 3 ranges that gigablock still offers **9 candidate ranges**;
smeared evenly across all 12 it offers **none**:
1536 kernel pageblocks, 12 PUD ranges in the gigablock
packed [KKK][KKK][KKK][ ][ ][ ][ ][ ][ ][ ][ ][ ]
3 ranges spoiled 9 candidates survive
smeared [K ][K ][K ][K ][K ][K ][K ][K ][K ][K ][K ][K ]
every range spoiled 0 survive
Leaving the choice to LIFO reproduces §1.2's spend-territory-in-2 MB-units pathology one level down.
**V1 accepts that, and the arithmetic says it can afford to.** 64 gigablocks tile 766 PUD ranges against
a target of 94, so 8 clean gigablocks meet the target on their own and up to **56 of 64 may be tainted**
before within-gigablock position matters at all. At the 8 tainted gigablocks this host expects, the clean
set alone yields 672 ranges, 7.1x the requirement, and dense packing inside the tainted 8 adds at most 72
marginal ranges to a surplus of 578. It cannot change the outcome.
Within-gigablock packing only becomes load-bearing once the tainted set passes roughly 56 gigablocks, and
at that point concentration has already failed badly enough that packing the wreckage is treating a
symptom. **So v1 ships without it**, and the bet is stated rather than hidden: v1 assumes the tainted set
stays small, which is exactly what taint-on-first-contact (§3.4) and fullness steering exist to deliver,
and exactly what v1 is already measuring.
**The trigger for building it** is the success metric of §1 falling toward 94 with clean gigablocks alone,
or equivalently the tainted count passing ~48 of 64, leaving margin. Both are in the debugfs dump of
Phase 1 patch 8.
### 4a. Deferred: the per-gigablock claim cursor
Kept because if the trigger above fires, re-deriving this costs more than reading it.
**One `u16` per gigablock** recording the pageblock index of the last claim; a claim takes the first
content-free pageblock at or after it, wrapping. Apply it to the steps that pick a content-free pageblock,
steps 2 and 7 of §5.1, where every candidate is equivalent and position is the only thing to choose on.
Leave step 4 taking the largest fallback block, where the existing largest-first heuristic is doing real
work and position is secondary.
This is a cursor, which §3.2 rejected for choosing *between* gigablocks. The difference is that a bitmap
answers the inter-gigablock question directly and there is no bitmap for position within a gigablock, so
a cursor is the right tool for the second question and the wrong one for the first.
**It is a linear scan, and the amortised claim holds in only one direction.** While a gigablock fills,
the cursor moves forward and wraps once, so across n claims the scan visits each pageblock about once:
O(1) per claim in aggregate, and the next pageblock is usually the one. In a nearly-full fragmented
gigablock every claim walks a long run of allocated pageblocks and wraps, and there is no amortisation
left. Nearly-full is not an exotic case, since §5.1 prefers tainted gigablocks precisely because they are
already spoiled.
What makes it survivable is the shape of the scan rather than its length: a contiguous byte array, one
byte per pageblock, 6144 bytes and 96 sequential prefetchable cache lines at worst, against the same
length in *dependent* cache misses for a free-list walk. "All three content bits clear" is a mask test,
so one 8-byte load screens 8 pageblocks and the worst case is 768 loads.
**Bound the walk anyway.** The cursor is a packing heuristic, not a correctness requirement, so nothing
breaks if it occasionally picks a worse pageblock. Scan at most N pageblocks from the cursor and fall
back to the free-list head, which bounds the work under `zone->lock` and makes the degradation traceable:
count the fallbacks, and a high rate says N is too small before the density number goes bad.
Not taken: a per-gigablock content-free bitmap, 6144 bits = 768 B each and 48 KB per zone, where
`find_next_bit()` makes the scan 96 words. It passes §3.2's test — read on a hot path, and deriving it is
the scan being bounded — but 48 KB and a second coherence point against the content bits is a steep price
for a heuristic a bounded walk already handles.
### 4.1 Eligibility is one mask; fullness only ranks the survivors
Written literally, step 1 is a nested loop — fullness buckets by eleven orders, a word AND per cell
before the first candidate on a miss. That is the wrong shape, because the two dimensions are not the same kind
of thing. Eligibility is correctness: a gigablock below `LOW` may not take movable, and `TAINTED_FULL`
has nothing to give. Fullness is preference: among gigablocks that may all serve the request, one is a
better victim. So test eligibility once, and rank only what survives.
/* one word per level, cumulative: above[HIGH] subset of above[LOW] ... */
unsigned long gb_above[__NR_GB_WATERMARKS];
class = (order >= 4) ? GB_LARGE : GB_SMALL; /* §6.1 */
elig = gb_tainted & gb_above[class][LOW] & ~gb_evac_failed;
cand = elig & has_free[mt][order];
if (cand)
gb = rank_by_fullness(cand);
**Fold each class's capability into its mask at update time.** §6.1 grades order 0-3 on
`headroom_pages + headroom_blocks * (1 << pageblock_order)` and order 4 and above on `headroom_blocks`, so
a query that computed either would be doing at every allocation what a bit flip does once per crossing.
The class is a property of the request, known without a load, so indexing by it costs nothing: the query
is three loads either way.
**Cumulative rather than one word per band.** The hot path asks "may this gigablock serve me", which is
an "at least" question and one load. The background evacuation worker asks "which gigablocks are in the
`EVACUATE` band exactly", which is `gb_above[EVAC] & ~gb_above[LOW]` — two loads and an andn, on a path
that already sleeps. Exclusive per-band words would invert that trade and cost two words per transition
instead of one.
**Every hard exclusion belongs in the same word, not in a branch.** `TAINTED_FULL` is the one the search
would otherwise rediscover per query, since "evacuation failed here and nothing has been freed since" is
a property of the gigablock, not of the request. Anything the request decides — `ALLOC_CMA`,
`ALLOC_HIGHATOMIC`, `ALLOC_NOFRAG_TAINTED_OK` — stays a per-request mask combined at query time,
because a per-request bit maintained per gigablock is a coherence bug with extra steps.
**What not to add.** A precombined mask per search step: there are nine steps across two migratetypes
and the combinations multiply, while each is one or two ANDs of words already in L1 from the same
`kvmalloc` (§3.1). This is §3.2's rule applied to the watermarks — maintain a bit only if it is read on
a hot path and cannot be derived cheaply from state already maintained.
**Maintenance is the existing accessor's job.** The watermark words flip only when a gigablock crosses a
level, under `zone->lock`, in the same helper that already updates `nr_free`, `nr_kernel_pageblocks`,
the `has_free` bit and the fullness bucket. A second writer is how these get out of step, and
`CONFIG_DEBUG_VM` should assert `gb_above[]` against a recomputation the way it already does for
`has_free`.
## 5. Allocation policy — the complete search order
Every requesting migratetype, every class of gigablock, every kind of pageblock. Oversights here caused
bugs in earlier implementations, so nothing may be left to "and then it falls through to the existing
code".
### 5.0 The classes being searched
**Gigablocks.** Two classes, plus one whole-zone case:
tainted nr_kernel_pageblocks > 0, graded MOSTLY_EMPTY / MOSTLY_DEDICATED
clean no kernel pageblocks, graded the same way by total use
no gigablocks zone->gb is NULL, every page on zone->free_area, behaviour is
mainline's (§5.9, §10.3)
**PUD ranges that can never host a 1 GB page live inside gigablocks like any other range**, on
gigablock free lists, and are entirely ordinary to the allocator: ranges with a memory hole, spanning
two zones, containing CMA, dominated by the vmemmap, or currently isolated. They are a property of a
range rather than a third class of gigablock, they matter only to the 1 GB target search, and §10.2
derives them rather than listing them.
**Candidate pageblocks, and what taking one costs a kernel allocation.** This is the cost model §5.1's
order is derived from. Two independent properties decide it: migratetype, which says whether the
pageblock may be converted at all, and content, which says what conversion leaves stranded.
pageblock migratetype content bits a kernel allocation may cost
the requested type any allocate from its free list none
the other kernel type any claim, or steal one block near zero: kernel
content either way
MOVABLE all three clear claim best: 512 pages of
headroom, nothing
stranded
MOVABLE movable only, all reclaim, which empties it, no copy, no
clean page cache then claim destination, no debt
MOVABLE movable only, not claim: converts, strands evacuation debt
all reclaimable its movable content
MOVABLE movable only, not steal one block, no creates a mixed
all reclaimable conversion pageblock
HIGHATOMIC any nothing ALLOC_HIGHATOMIC only
CMA any nothing movable requests only
isolate bit set any nothing not allocatable
**A kernel allocation may use only the first six rows**; HIGHATOMIC, CMA and isolated pageblocks are
exclusions rather than options. `migratetype_is_mergeable()` is the kernel's own name for the line — it is
`mt < MIGRATE_PCPTYPES`, so UNMOVABLE, MOVABLE and RECLAIMABLE may be converted and HIGHATOMIC, CMA
and ISOLATE may not.
The "all clean page cache" row cannot be identified from the bits: `PB_has_movable` does not
distinguish clean file from dirty file from anon. The step is therefore speculative in one specific
sense — the helper already exists, since the prequel shipped it (§13 P4), and the cost of running it is
what cannot be predicted, so run it on a movable-content-only pageblock and let it discover the cost. **Do not add
a bit for this**, or the array doubles (§3.2).
### 5.1 UNMOVABLE and RECLAIMABLE requests
**The numbered steps are a preference order, not a call chain.** Steps 6 and above cannot run under
`zone->lock`; the fast path returns no page and `__alloc_pages_slowpath()` re-enters (§9.3).
`fb = fallbacks[start_mt]`, which is `{RECLAIMABLE, MOVABLE}` for UNMOVABLE and `{UNMOVABLE, MOVABLE}`
for RECLAIMABLE.
0 the structurally doomed but usable ranges (§10.2a), if any
at most two per zone, fixed at init: the partial PUD ranges at the zone
edges. They can never host a 1 GB page and they hold real free memory,
so kernel content there costs nothing and spares a good range.
1a matching type in a tainted gigablock, fullest bucket first, order ascending
gb_tainted & gb_fullness[f] & has_free[start_mt][o]. Fullest first
concentrates kernel content where it already is (§3.4); split a larger
block here rather than take a smaller one from a clean gigablock.
1b matching type in a clean gigablock, fullest bucket first, order ascending
~gb_tainted & gb_fullness[f] & has_free[start_mt][o]. Spoils a gigablock
nothing has touched, so it comes after every tainted option, evacuation
included, and takes the one with least left to protect.
2 tainted gigablock, content-free pageblock, claim
gb_tainted & gb_content_free_pageblock() (§3.2)
3 tainted gigablock, movable-content-only pageblock: migrate out, then claim
speculative, see §5.0
4 tainted gigablock, largest fallback block, claim
gb_tainted & (has_free[fb[0]][o] | has_free[fb[1]][o]), o descending
5 tainted gigablock, steal a single fallback block, no conversion
only while headroom_pages > MIN, and at the requested order only (§6.2).
Prefer a pageblock with PB_has_unmovable already set: see below
6 sleepable: shrink or evacuate inside the tainted set (§9.2), retry from 2
7 clean gigablock, content-free pageblock, claim, fullest first
~gb_tainted & gb_content_free_pageblock(). The tainted set grows; trace this.
8 clean gigablock, largest fallback block, claim, fullest first
worst case: spoils a gigablock and strands movable content in it
9 ALLOC_NOFRAG_TAINTED_OK retry, or ALLOC_OOM / ALLOC_RESERVES:
skip all steering and take what mainline would take
Steps 5 and 6 are ordered by the watermark, not fixed. Step 9 is invariant 1 made concrete. §9.3 gives
the rule for how hard to try steps 3 through 6 before reaching step 7.
**Fullness ordering dissolves the step-1c question.** An earlier draft split the clean class into
"already holds kernel content" and "pristine" and could not place the pristine step: measured in
gigablocks lost it belonged below steps 7 and 8, measured in pages converted it belonged above them.
With a graded ladder there is no binary to place. Every step that touches the clean set walks the same
fullness order, so the choice inside each step is "least left to protect first" and the choice between
steps stays what it always was, cost per allocation.
**Gigablock class outranks order within the matching migratetype.** For an order-0 unmovable request
with free order-0 pages only in clean gigablocks and a free order-6 block in a tainted one, take the
order-6 block and split it. `expand()` puts one block of each intermediate order back on that
gigablock's lists, so the residue serves the next several order-0 requests from the same already-lost
gigablock, and the split costs nothing that matters. Taking the order-0 page from the clean gigablock
costs up to a whole 1 GB range. So the search runs class on the outer loop and order on the inner,
the same shape §5.2 uses for movable with the classes the other way round.
**An earlier version of this section claimed step 1 "cannot spoil anything". That was wrong**, and it
mattered because step 1 is the common path. The claim came from `__rmqueue_smallest()` allocating
within the *requested* migratetype, so no pageblock changes type. But a clean gigablock can hold an
UNMOVABLE-typed pageblock with no content — one converted earlier and since emptied — and allocating
there adds kernel content, sets `PB_has_unmovable`, raises `nr_kernel_pageblocks` and taints the
gigablock (§3.4). Pageblock migratetype and gigablock taint are different things.
**Stealing between the two kernel types is not symmetric, so step 5 prefers by direction.** Both
directions leave `nr_kernel_pageblocks` unchanged, which is why stealing beats claiming a fresh
pageblock at the gigablock level — and with the claim cursor deferred (§4a) it also spoils no new PUD
range, where a claim may. But the two directions differ in what they destroy:
- **A RECLAIMABLE request stealing from an UNMOVABLE pageblock costs nothing.** That pageblock has
`has_unmovable` set and was never going to return to content-free, so adding shrinkable content to it
changes nothing.
- **An UNMOVABLE request stealing from a RECLAIMABLE pageblock costs that pageblock's
recoverability.** It moves from `010` to `110` in §3.3's table: every other object in it can still be
shrunk and it will never be content-free again. §1 measures 8.2M slab pages in RECLAIMABLE
pageblocks, a third of all kernel pages, and that is the supply §9.2 is trying to turn back into
`headroom_blocks`.
So step 5 prefers a victim that already has `PB_has_unmovable` set, which is one bit test on a
pageblock it is already looking at. The damaging direction stays available — it is still better than
spoiling a clean gigablock at step 8 — but it is the last of the tainted-set options rather than an
equal one.
Step 4 is the same choice with conversion, and there the answer is simply no: claiming a RECLAIMABLE
pageblock for UNMOVABLE use loses the whole pageblock's recoverability rather than one block's worth,
so step 4 should skip a `010` victim while step 5 may take one.
**Step 1a covers the case that used to leak.** A gigablock holding one kernel pageblock out of 6144 is
tainted (§3.4), so 1a finds it and the next kernel allocation lands there rather than opening a new
front. Its `MOSTLY_EMPTY` bucket sorts it last among tainted gigablocks, so a fuller one absorbs the
allocation first if there is one — the magnet without the write-off.
**1c is where a gigablock is lost, so it is the only step that needs a reason to run at all.** Nothing
below it is cheaper: every tainted option, including synchronous evacuation, is preferable to spoiling a
range nothing has touched. §9.3 governs how hard to try those before reaching it.
**The remaining imprecision is inside a bucket, and it grows with machine size.** A gigablock is
`round_up(zone_span / 64, PUD_SIZE)`, 12 GB on this host — twelve PUD ranges. A fullness bucket says how
much of the group is in use, not which of the twelve ranges holds it, so the search concentrates
allocations into the right 12 GB and then picks blindly inside it. On a 32 GB machine a gigablock is
exactly one PUD range and the blindness does not exist; on a 768 GB machine every allocation inside a
bucket is a twelve-way coin flip. v28 did not have this problem because its gigablock was one PUD range;
§3.4 records why the bitmap index reintroduced it.
**Nothing else was guarding it either.** `should_try_claim_block()` gates only the fallback paths, so
§5.8's "clean, no permission -> no" never sees an allocation that comes off the matching-type free
list. For step 1 the search order is the whole protection, which is why 1a and 1b are separate steps
rather than one ascending scan.
**Interaction with high-order headroom.** Splitting the order-6 block consumes that gigablock's
largest free run, which is the supply `headroom_blocks` exists to protect (§6.1). No extra rule is
needed: if the split would take `headroom_blocks` below `EVAC`, that is the signal to evacuate
or to try the next tainted gigablock, not to reach into a clean one. Ascending order within 1a already
takes the smallest sufficient block, so the largest run is consumed only when nothing smaller exists.
### 5.2 MOVABLE requests
0 ZONE_MOVABLE, if populated. Not a step in this zone's search at all:
build_zonerefs_node() walks zone_type from MAX_NR_ZONES downward, so a
GFP_HIGHUSER_MOVABLE allocation tries ZONE_MOVABLE before ZONE_NORMAL.
Movable pressure reaches the gigablock zone only once ZONE_MOVABLE is
exhausted, and ZONE_MOVABLE never has gigablocks.
1 CMA, when ALLOC_CMA and NR_FREE_CMA_PAGES > NR_FREE_PAGES / 2
unchanged from mainline; CMA ranges never sit in a gigablock.
2 matching type in the clean set, order ascending
~gb_tainted & has_free[MOVABLE][o]
3 CMA fallback (__rmqueue_cma_fallback), mainline's RMQUEUE_CMA step
4 matching type in a tainted gigablock, emptiest bucket first, order
ascending
gb_tainted & gb_fullness[f] & has_free[MOVABLE][o]. Free pages in an
already-spoiled gigablock cost nothing; emptiest first leaves the ones
closest to recovery alone.
5 barrier: stop. Return no page and let __alloc_pages_slowpath() reclaim.
Kernel headroom is behind this, not the tainted set.
6 ALLOC_NOFRAG_TAINTED_OK retry: barrier released. Only here may movable take
kernel headroom, in this order:
6a steal a single block from a kernel pageblock at the requested order
only, no splitting. Consumes the stranded free space of §1. Prefer
the gigablocks with the most kernel content, so the nearly-recoverable
ones are left alone.
6b claim a kernel pageblock. Converts kernel territory back to movable,
which shrinks the kernel footprint, but costs a whole pageblock of
headroom -- hence after 6a.
Claim and steal are ordered opposite to §5.1 for a one-line reason: a kernel allocation wants the
pageblock converted, so claim first; a movable allocation does not want to consume a whole pageblock of
kernel headroom, so steal first.
**The barrier sits between taking free memory and taking kernel headroom, not between clean and
tainted.** Step 4 is movable using pages that are simply free in a gigablock already spoiled, which
costs the design nothing and is memory the machine has. Steps 6a and 6b convert or consume space the
kernel is counting on, which is why they need the zone to have already failed: `ALLOC_NOFRAG_TAINTED_OK`
records exactly that, the clean set could not serve the request and reclaim did not help.
**When the kernel frees a lot and leaves half-empty tainted gigablocks**, movable consumes them at step
4, without needing the barrier released at all, and that is right: the pages are genuinely free, those gigablocks are already
spoiled, and refusing free memory in order to reclaim instead is the pathology §7 exists to prevent.
Exclusion is graded by headroom, not absolute — nailing the door shut at the taint bit would strand that
memory. The refinement is 6a's preference: consume the *most* spoiled gigablocks first, because a nearly-empty
tainted gigablock is the prime candidate for returning to clean, and filling it converts free recovery
into recovery that needs migration work.
**What a populated ZONE_MOVABLE means for this series.** It is the static version of what gigablocks do
dynamically: `movablecore`/`kernelcore` carve a range kernel allocations may never enter, sized once at
boot and never resized. With a large ZONE_MOVABLE the competition §1 measured is reduced and the series
buys less; with none — both nodes here have one with zero pages, the fleet configuration — it buys the
most. State that in the cover letter: gigablocks give the containment ZONE_MOVABLE gives, without having
to size it correctly at boot or stranding the reservation when the workload changes.
### 5.3 HIGHATOMIC
alloc __rmqueue_smallest(zone, order, MIGRATE_HIGHATOMIC), only with
ALLOC_HIGHATOMIC. Never a fallback source, never falls back itself.
reserve reserve_highatomic_pageblock() takes from an already-tainted gigablock:
the reserve is long-lived and unmigratable, so it must not spoil a
clean one.
drain unreserve_highatomic_pageblock() returns the pageblock to its previous
type. nr_kernel_pageblocks must be updated on both transitions or
invariant 7 drifts.
watermark its free pages are already excluded by __zone_watermark_unusable_free()
and must also be excluded from headroom, for the same reason.
### 5.4 CMA
Movable requests only, `MIGRATE_CMA` lists only, never a claim or steal source or target because
`migratetype_is_mergeable()` excludes it and `fallbacks[]` never names it. A PUD range containing CMA
fails the §10.2 candidate test, so CMA never appears in a 1 GB target. It does live inside gigablocks
like anything else, so `has_free[MIGRATE_CMA][*]` is maintained but never searched by §5.1.
### 5.5 ISOLATE
An isolated range is unallocatable whatever the policy thinks of it, so steering is unaffected. Free
lists and counters still track it.
**Isolation has two causes with opposite meanings.** `start_isolate_page_range()` is called by
`offline_pages()`, where the range is going away, and by `alloc_contig_range()`, where isolation is the
*success path* of building a 1 GB page. A fully isolated PUD range is more likely the goal in progress
than a disqualification. The persistent state cannot tell them apart — both set the same bit, and only
the caller knows via `MEMORY_OFFLINE`. No consequence for allocation, one for **reporting**: counting a
range mid-`alloc_contig_range()` as lost undercounts the outcome the series exists to produce. v27's
`contig_allocated`/`gigablock_contig_mark()` is the record of the success case and is where the metric
should read from.
### 5.6 Which bitmap rows are searched
UNMOVABLE, MOVABLE, RECLAIMABLE searched by §5.1 and §5.2
HIGHATOMIC searched only for ALLOC_HIGHATOMIC
CMA maintained, never searched by kernel requests
ISOLATE maintained for list consistency, never searched
Stated explicitly because a search consulting the wrong row is exactly the class of oversight that bit
earlier implementations.
**The unsearched rows are still maintained, and that is the cheap option.** The single accessor of §3 is
uniform over migratetype: it flips `has_free[mt][order]` for whatever migratetype it was called with. So
maintaining the CMA and ISOLATE rows costs one bit flip like any other, while skipping them would need a
conditional on the hot path. The only cost is 176 bytes of the 528-byte array.
### 5.7 `page_group_by_mobility_disabled`
When mobility grouping is off, `should_try_claim_block()` returns true unconditionally and migratetype
separation collapses. It is only set on systems too small to host a gigablock, so the two cannot
coexist — but assert it rather than rely on it, because the coupling is invisible:
`VM_WARN_ON(page_group_by_mobility_disabled && zone_has_gigablocks(zone))`.
### 5.8 What `should_try_claim_block()` becomes
Today it decides on order alone: true at `order >= pageblock_order`, true at
`order >= pageblock_order / 2`, true for UNMOVABLE or RECLAIMABLE at any order, true when mobility
grouping is off, false otherwise. So kernel allocations always claim and movable claims only at order 4
and above.
**It gets rebased rather than deleted.** For movable the order thresholds are right and stay unchanged. For
kernel requests the order terms are already vestigial — every path returns true — so a gigablock-class
term replaces them. One new input is needed: whether spoiling a clean gigablock is permitted yet, which
is retry state rather than a property of the candidate. Signature becomes
`should_try_claim_block(order, start_mt, gb, alloc_flags)`.
UNMOVABLE / RECLAIMABLE request:
tainted yes, any order already lost, and claiming yields
512 pages of headroom. This is
where claiming should be most
aggressive, the opposite of a
threshold.
tainted, headroom < MIN yes still better than spoiling a clean
gigablock
clean, no permission no the new behaviour and the point of
the series
clean, TAINTED_OK or ALLOC_OOM yes invariant 1
zone has no gigablocks yes today's behaviour, invariant 2
MOVABLE request:
any class, order >= pageblock_order yes pageblock was entirely free
tainted, TAINTED_OK, order >= yes converts kernel territory back to
pageblock_order/2 movable; §5.2 tries steal first
tainted, no TAINTED_OK no the barrier (§6.2)
clean, order < pageblock_order / 2 no unchanged from today
zone has no gigablocks unchanged from today
Two things fall out. **A movable request needs no gigablock-class term at all**, because movable content
never spoils anything — clean means free of *kernel* content, so a THP landing in a clean gigablock
leaves it clean. The class dimension exists for kernel requests only. And **for kernel requests the
order dimension disappears**: the answer is yes in every class except an unpermitted clean gigablock, at
every order.
### 5.9 Zones without gigablocks
The common case, not an edge case. On this host node 0 has DMA at 4095 pages, DMA32 at 4.3 GB, Normal at
821.6 GB, Movable and Device empty, so **one zone of five gets gigablocks** and every allocation walking
the zonelist crosses both kinds. The `zone->gb == NULL` path is hot and must be free.
pfn_free_area() returns &zone->free_area[order]; every list operation is
mainline's, unchanged
for_each_free_area() yields exactly one element, so the Phase 2 macro degenerates
to the loop it replaced
PB_has_* maintenance skipped entirely. The NULL test is already being made, so
this costs nothing, and it is what makes "small systems stay
bit-identical" true of cost as well as behaviour.
should_try_claim_block() today's answers exactly
watermarks NR_FREE_KERNEL_TAINTED reads zero, so the arithmetic is
unchanged
headroom, evacuation none. No watermark states, no kcompactd wakeups.
alloc_contig_range() works exactly as today: not blocked, just not helped
CONFIG_DEBUG_VM checks no-ops for this zone, not failures
debugfs reports the zone as having no gigablocks rather than omitting
it, so selftests skip cleanly instead of failing to parse
**Mixed nodes need nothing special** — each zone answers for itself. Worth stating only because it means
the NULL path is on the hot path for nearly every allocation, so it must be a predicted branch off an
already-loaded cache line, not a function call.
## 6. Watermarks, states and the barrier
### 6.1 Headroom is two numbers
A page count says nothing about order capability: a gigablock can hold ten thousand free pages in
kernel pageblocks, all at order 0, and fail every high-order kernel request.
headroom_pages free pages in kernel-typed pageblocks, excluding highatomic
reserves. Serves the must-succeed kernel allocations, which
are all order 3 or below.
headroom_blocks content-free pageblocks in the tainted set. Serves order 4
and above, and is what evacuation must produce for an
order >= pageblock_order request -- one at a time, through
the prequel's pageblock helper (§8).
**Both numbers are kernel headroom, and neither gates movable.** Movable competes for content-free
pageblocks and for nothing else, so §6.2's barrier decides its access to them; there is no movable
watermark.
**Where they come from.** `headroom_blocks` is exactly derivable from counters already kept, so nothing
maintains it:
for (o = pageblock_order; o < NR_PAGE_ORDERS; o++)
blocks += gb->free_area[o].nr_free << (o - pageblock_order);
`headroom_pages` is not derivable — `struct free_area` keeps one `nr_free` across all migratetypes — so
`struct gigablock` carries `u32 nr_free_kernel`, free pages on the UNMOVABLE and RECLAIMABLE lists,
written by the accessor that already writes `nr_free`. Same line, same lock, no new contention point.
**Each caller is graded on the capability it can actually reach**, not on the worse of the two numbers:
kernel order 0-3 headroom_pages + headroom_blocks * (1 << pageblock_order)
HIGH, LOW, EVAC, MIN
kernel order >= 4 headroom_blocks
HIGH, LOW, EVAC
An order-3-or-below kernel request can claim a content-free pageblock (§5.1 step 2), so content-free
pageblocks are capacity for it too. Grading it on the worse of the two numbers would send a sleepable
`GFP_KERNEL` allocation to migrate pages and shrink slab while thousands of pageblocks it could have
claimed sat free — the common shape, since kernel pageblocks filling is what tainted means.
The two graded quantities cost 8 mask words against 4, 32 bytes with one word unused, and **nothing on
the fast path**: the
caller knows its own order and indexes the mask for its class, so the query is the same three loads
(§4.1).
`headroom_blocks` needs no `MIN` because its callers are order 4 and above and already have
`MIGRATE_HIGHATOMIC` as their reserve, whose free pages `headroom_pages` excludes for exactly that
reason. At `headroom_blocks` zero a sleepable high-order kernel allocation spills to a clean gigablock
(§5.1 step 8) and spoils it. That is the floor, stated rather than left as an absence.
Derive the thresholds from one ratio per quantity rather than exposing seven knobs.
**`headroom_blocks` is the high-order reserve, built from what already exists.** A content-free
pageblock inside a tainted gigablock already is one, because claiming it yields an order-9 block, and
§5.1 step 2 already prefers exactly those, so maintaining the watermark is the whole reservation. Compare `MIGRATE_HIGHATOMIC`, which is the same idea for atomic allocations with
its free pages excluded from the zone watermark — the difference is that this reserve is *usable* by
ordinary kernel allocations rather than held idle, so an idle system does not pay for it. Start
`headroom_blocks` `EVAC` at 2 to 4 pageblocks per active gigablock, enough for a vmalloc order-10
attempt, and measure.
### 6.2 Exclusion, the evacuation trigger, and the barrier
**Movable is not gated on kernel headroom; the barrier gates its access to kernel headroom.** The line
that matters is §5.2's, between taking free memory and taking space the kernel is counting on. Movable
taking free pages from movable-typed pageblocks of an already-tainted gigablock (§5.2 step 4) consumes
no content-free pageblock and no kernel-typed free page, so it costs the design nothing and is memory
the machine has. Refusing it leaves that memory unused and sends the allocation to spoil a clean
gigablock instead, which is strictly worse.
**What protects the tainted set from movable is ordering, not a watermark.** Step 4 takes the emptiest
bucket first, so movable lands in the gigablocks least worth saving and leaves the nearly-recoverable
ones alone. That is a per-gigablock preference rather than a cliff that treats every tainted gigablock
alike.
**The barrier sits between §5.2 step 4 and step 6**, and it is `ALLOC_NOFRAG_TAINTED_OK`, not a
watermark: only after the clean set has failed and reclaim has not helped may movable steal from a
kernel pageblock (6a) or claim one (6b). Movable content is migratable by definition and §10.3
guarantees it most of the zone, so this is the reversible kind of damage (§7).
The cost admitted here is that evacuation later has more pages to move. It is bounded by the same
emptiest-first ordering, and moving movable content is what evacuation is for.
**Fullness orders, headroom admits.** The two now overlap enough to be worth separating explicitly, and
it is §3.4's lesson applied again — one number cannot answer two questions. The fullness bucket decides
*which* gigablock the search prefers, and it is a pageblock count because concentration is about
position. The headroom watermark decides *whether* movable may enter the one it picked, and it is a free
count because admission is about capacity. A gigablock can be `MOSTLY_EMPTY` and still closed to movable
if its few kernel pageblocks have no free pages left, and `MOSTLY_DEDICATED` and still open if they do.
§10.3 makes the premise safe: gigablocks exist only in a `ZONE_NORMAL` spanning at least 8 of them, so
the movable workload always has most of the zone to work in.
**Hysteresis comes back with readmission, and it is the price of grading the exclusion.** Exclude below
`LOW`, readmit only above `HIGH`. Without the gap, evacuating a pageblock invites movable straight back
into it and the gigablock oscillates, paying migration cost forever, so `HIGH - LOW` must exceed what
movable allocation can consume between two watermark checks — on the order of the PCP batch size times
the CPU count. An earlier draft deleted this pair by making exclusion absolute at the taint bit. That
was an over-correction: it bought a simpler watermark set by throwing away every half-empty tainted
gigablock as a home for movable, which is memory the machine has and §5.2 was right to want.
The bands, all read off the two kernel-headroom numbers, `HIGH > LOW > EVAC > MIN`. Two mechanisms fire
in them and they act on different content: **evacuation migrates movable content out** (§8, it never
reclaims), while **slab reclaim shrinks kernel content in place** (§9.2). A tainted gigablock needs both,
since evacuation raises headroom but only the kernel side can lower `nr_kernel_pageblocks`.
capability vs band what fires
----------------- ---------------- --------------------------------------
(no kernel pbs) CLEAN nothing; kernel entering taints it
>= HIGH TAINTED_OPEN kernel proceeds
>= LOW TAINTED_TIGHT that class prefers another gigablock
>= EVAC TAINTED_EVACUATE async: migrate movable, shrink slab
>= MIN TAINTED_CRITICAL sleepable kernel does both synchronously
below MIN .. non-sleepable kernel only
(any) TAINTED_FULL overrides the ladder; retry in 10 s
Movable appears nowhere in that column: it is gated by §5.2's barrier, not by these bands. In full:
CLEAN no kernel pageblocks. Movable and kernel both allowed.
TAINTED_OPEN both numbers >= HIGH. Kernel proceeds, movable allowed.
TAINTED_TIGHT a class below its LOW. Kernel requests of that class prefer
elsewhere; movable is unaffected.
TAINTED_EVACUATE a class below its EVAC. Async evacuation and slab reclaim run a
bounded pass and stop; the next check re-evaluates.
TAINTED_CRITICAL order 0-3 capability below MIN. Sleepable kernel allocations
evacuate or shrink synchronously; only non-sleepable ones use
the rest.
TAINTED_FULL an evacuation attempt isolated pages and failed to migrate them,
and nothing has been freed here since. Kernel allocation looks
elsewhere. An attempt that found nothing movable to isolate is a
different outcome and does not set this; see below.
**A bounded pass replaces evacuation hysteresis.** A background worker triggered at one level needs
either a higher stop level or a work limit, or it stops the moment it makes progress and immediately
re-triggers. `kcompactd` already takes the second option: do a pass, stop, re-check next cycle. That
costs one watermark instead of two and needs no gap sized against an allocation rate.
**`TAINTED_FULL` overrides the band rather than sitting below it.** Once evacuation has failed there is
nothing for either evacuating band to do, so the gigablock skips to spilling regardless of its counters.
Otherwise `TAINTED_EVACUATE` retries a known-failed migration every cycle.
**Failure is optimistic: two outcomes, and only one of them latches.** Most reasons a migration fails are
temporary — a GUP or DMA pin dropped a moment later, writeback completing, `migrate_pages()` returning
`-ENOMEM` under momentary pressure — and none of them frees a page in this gigablock, so the clear
condition above never fires for them. Latching on the first failure would write off 12 GB for the rest of
the boot on a transient, and because the bit gates allocation as well as evacuation, the gigablock would
be neither evacuated nor used.
So the attempt reports which of two things happened. **Nothing movable to isolate** means evacuation can
never help here and only a free can change that; it does not set `TAINTED_FULL`. **Isolated pages,
migration failed** is the transient case, and only it does. The attempt already walked the gigablock, so
distinguishing them costs nothing.
**Retry after 10 seconds**, which is a starting constant rather than a tuned one. It covers writeback
and transient `-ENOMEM`; a long-lived DMA pin outlives any period. The quantity worth watching is not the
period but how often `TAINTED_FULL` is entered at all: near zero and the period does not matter, high and
the period is the least of the problems. Count entries in the debugfs dump.
**The retry is lazy.** `gb_evac_retry_at` holds one jiffies stamp per zone, checked by the
evacuation trigger, which already sleeps:
if (gb->gb_evac_failed && time_after(jiffies, gb->gb_evac_retry_at)) {
gb->gb_evac_failed = 0;
gb->gb_evac_retry_at = jiffies + GB_EVAC_RETRY; /* 10 * HZ */
}
No timer and no wakeup on an idle system, and the retry happens when something is asking for memory,
which is the only time the answer is worth having. Retry only while the gigablock is still blocking:
`gb_evac_failed` set, no content-free pageblock and an empty `has_free` row. If something is
free there is nothing to evacuate for.
**The whole word clears at once, deliberately.** An attempt begins with `drain_all_pages(zone)`, an IPI
to every CPU with a non-empty list, so one drain and one bounded pass should serve every retry in the
cycle. Per-gigablock clocks would multiply the drains by the number of stuck gigablocks: on a 176-CPU
host with eight stuck, roughly one IPI-to-all-CPUs per second, forever, to rediscover that they are still
pinned.
**The bottom band is safe only because spilling exists.** Closing a gigablock to sleepable kernel
allocations would be an OOM if this were a zone-level reserve. It is not: everyone else moves to another
gigablock, and invariant 1 guarantees a spill eventually succeeds. The reserve is for callers with
nowhere else to go, on the `__GFP_HIGH` model.
**Headroom is only defined for tainted gigablocks**, since a clean one has no kernel-typed pageblocks to
count free pages in. A clean gigablock is never a barrier.
**The exclusion is a barrier, not a filter.** If movable is excluded from a gigablock and the search
merely skips it, movable lands in the next tainted gigablock instead — deeper into kernel territory, so
the exclusion would be causing the damage it exists to prevent. Movable stops at exclusion and reclaims
instead, which needs no new mechanism: the fast path returns no page and `__alloc_pages_slowpath()` does
what it always does. §5.2 step 5 is that stop, and step 6 is the one condition that releases it.
**Movable steals a block at exactly the requested order, leaving higher orders intact.**
`__rmqueue_steal()` walks
`current_order` ascending and `page_del_and_expand()` splits, so a movable order-0 steal would split an
order-10 block inside a kernel pageblock and destroy that gigablock's high-order capability for a request
that could have come from a movable pageblock elsewhere. A movable allocation has a claim on the pages of
kernel headroom, not on its contiguity. It still gets the stranded space of §1, which is stranded
precisely because it is fragmented. This also keeps the two headroom numbers nearly independent: movable
consumption depletes `headroom_pages` without touching `headroom_blocks`, while evacuation raises both.
**Background evacuation** wakes `kcompactd` with a new reason rather than adding a kthread, and only for
the gigablock kernel allocation is currently using.
### 6.3 Higher-order kernel allocations, checked against the actual GFP flags
**vmalloc asks opportunistically and handles failure itself, so the design owes it nothing.**
`vm_area_alloc_pages()` starts at
`large_order = ilog2(nr_remaining)` capped at `MAX_PAGE_ORDER`, so a large vmalloc asks for order 10
first. But it asks with
large_gfp = vmalloc_gfp_adjust(gfp, large_order) & ~__GFP_DIRECT_RECLAIM
`vmalloc_gfp_adjust()` adds `__GFP_NOWARN` and clears `__GFP_NOFAIL` for large orders, and the caller
strips `__GFP_DIRECT_RECLAIM`. Every high-order attempt is non-sleeping, silent and opportunistic: it
cannot reclaim, cannot compact, and fails immediately if the free list has nothing, after which the loop
walks down to the order-0 bulk allocator. On success it calls `split_page()` immediately, so it never
needed the block to stay high-order.
Failing those attempts is normal and already handled. The design owes vmalloc nothing beyond not making
order 0 fail. The cost of the fallback is still real — PTE instead of PMD mappings for the lifetime of
the mapping, for BPF JIT text, module text, `VMAP_STACK` stacks and large hash tables — which is what
`headroom_blocks` exists to reduce, not to guarantee.
**The high-order kernel allocations that must succeed are all order 3 or below**: kernel stacks at
order 2 without `VMAP_STACK`, `dma_alloc_coherent()`, `kmalloc` above 4 KB, ring buffers. They carry
`__GFP_DIRECT_RECLAIM` and sit at or under `PAGE_ALLOC_COSTLY_ORDER`, so the existing retry, reclaim and
compaction path serves them. A `GFP_KERNEL` order 4 to 9 allocation that must not fail is already
considered a bug upstream.
**Evacuation for an order >= `pageblock_order` kernel request is forced by the bits.** Emptying a mixed
pageblock cannot produce a contiguous run, because its kernel content stays. Only a pageblock whose
content is exclusively movable can be emptied into a content-free pageblock, so those are the only
useful targets. `MAX_PAGE_ORDER` is 10, so an order-10 request needs **two adjacent** such pageblocks.
## 7. Keeping the watermark honest when free memory is in the wrong place
If migration fails for want of a destination and the fallback is reclaim, reclaim must make progress even
though the zone is above its watermark. The hazard is real, but the mechanism is that **the watermark
lies**: `__zone_watermark_ok()` counts free pages inside tainted gigablocks that a movable allocation may
not touch, so the allocator concludes all is well, kswapd is not woken, direct reclaim is skipped, and
nothing progresses.
`__zone_watermark_unusable_free()` already exists for this and already has two instances of the pattern —
it subtracts `nr_free_highatomic` when the caller has no rights to reserves, and `NR_FREE_CMA_PAGES` when
the allocation cannot use CMA. This is a third:
if (!(alloc_flags & ALLOC_NOFRAG_TAINTED_OK) && movable request)
unusable_free += zone_page_state(z, NR_FREE_KERNEL_TAINTED);
With an honest watermark every existing mechanism engages correctly and needs no special case: kswapd
wakes, direct reclaim runs, compaction is considered, both retry paths see a real shortage, and the OOM
path is reached only when there is genuinely nothing to do. It also makes the pressure valve automatic —
when reclaim and migration both fail to help, the retry relaxes the barrier and lets movable spill into
the tainted set, which is the reversible kind of damage. Reuse `ALLOC_NOFRAG_TAINTED_OK` for both that
relaxation and the `ALLOC_NOFRAGMENT` retry: one flag, two uses, no new retry state machine.
Cost: one zone counter maintained where the per-gigablock free counts already are. It mirrors
`NR_FREE_CMA_PAGES`, which is also the argument that lands it upstream.
## 8. Evacuation
**The helper already exists.** `__alloc_contig_migrate_range()` isolates the range and calls
`migrate_pages()`, with a retry limit, `lru_cache_disable()` around the loop and
`putback_movable_pages()` on failure. The Phase 3 helper is a refactor of that, not new code, which
removes most of the risk from what looked like the largest new piece of machinery.
**Movable content is migrated, never reclaimed.** `__alloc_contig_migrate_range()` also calls
`reclaim_clean_pages_from_list()` on what it isolated, and the Phase 3 helper drops that step:
clean file pages migrate a copy, but no page is destroyed
dirty file pages migrate copying 4 KB beats a writeback
anon migrate
pinned give up on this range
Two reasons, and the second is why nothing is lost.
**Migration relocates a page; reclaim destroys it.** Dropping a clean file page belonging to a cgroup
below its `memory.min` or `memory.low` is a protection violation, and background headroom maintenance
would make that routine rather than rare. Migration moves the page and leaves the charge where it is, so
no protection is involved and no policy question arises.
**Reclaim still happens when it is needed, through the allocator rather than through us.** Migration
allocates a destination page for every page it moves, and those allocations take the ordinary path: if
memory is short they wake kswapd, enter direct reclaim, and obey every existing watermark and cgroup
rule. Reclaiming inline would be doing badly, with none of that policy, what the destination allocations
already do correctly. §7's honest watermark is what makes them see the real shortage.
The cost is a copy per clean file page, and a destination allocation that can fail where dropping the
page could not. A failed evacuation is the `TAINTED_FULL` case of §6.2, which retries.
**The kernel side of the same pass does shrink, for the same reasons.** This restriction is about
movable content only. A tainted gigablock needs both halves: migrating movable pages raises headroom but
cannot lower `nr_kernel_pageblocks`, because taint is defined by kernel pageblocks, so evacuation alone
can never return a gigablock to `CLEAN`. Only the kernel side can, which is why the two run together
(§9.2).
The same argument that makes migration right for movable content makes shrinking right for kernel
content: a shrinker hands objects back to the allocator rather than destroying a user's page, and the
`memory.min` question does not arise. What it recovers is described in §9.2 — mostly pages and
slab-internal space rather than whole gigablocks.
Three rules keep the kernel half from becoming the expensive one:
- **Bounded like the evacuation half.** A fixed request per trigger, then stop and re-check, which is
§6.2's bounded-pass discipline rather than a second watermark.
- **Node-local.** `shrink_slab()` is not gigablock-aware, so pressure applied while evacuating one
gigablock frees objects whose pages are anywhere. Pass the nid. The value is not "help this
gigablock", it is reducing the rate at which new pageblocks are tainted zone-wide, and framed as
targeted help it will look ineffective in tracing and get removed.
- **Behind patch 18's tunable**, defaulting conservative. §8's throttle argument applies harder here
than to migration: page reclaim's cost is a refault, slab reclaim's is rebuilding metadata from disk,
and shredding the dentry cache to preserve 1 GB availability is the wrong trade for most workloads.
**The two halves report separately.** §6.2's `TAINTED_FULL` is a statement about the migration half
only. A pass that finds nothing movable to isolate but shrinks a useful amount of slab has not failed to
evacuate — it had nothing to evacuate — so it must not arm the retry clock, and a pass whose migration
failed must arm it even if the shrink went well.
**Evacuation has two scales, and the plan needs both.** `__alloc_contig_migrate_range()` works on a
range and is what a 1 GB target or a whole-gigablock pass wants. One pageblock at a time is what
`headroom_blocks` is made of, and what §5.1 step 3 reaches for when a movable-content-only pageblock
stands between a kernel allocation and a claimable block.
**The pageblock-scale helper is the prequel's** (§13 Phase P), so the gigablock series inherits it and
adds only a second caller. An isolate-and-migrate loop over one pageblock, looping because `isolate_migratepages_range()` stops at `COMPACT_CLUSTER_MAX`, queued through
a fixed 64-entry preallocated work-item pool. Best effort by construction — a request is dropped when
the pool is empty and a block whose pages will not move is left alone — which bounds *concurrency*
rather than only work per pass, so the worst case holds however many gigablocks want service at once.
It rechecks the pageblock's migratetype before starting, since anything queued asynchronously can be
stale by the time it runs. `queue_work()` is not safe under `zone->lock`, so the queue hop is irq_work
first.
**Two triggers, one helper, one per series.** The reactive trigger is the prequel's: a movable pageblock
has just been claimed for kernel use, so migrate the movable pages left behind before they keep later
kernel allocations out of it. The proactive trigger is the gigablock series': a tainted gigablock is
short of `headroom_blocks`, so produce one (§6.2's `TAINTED_EVACUATE`). Background use takes
`MIGRATE_ASYNC`; the synchronous `TAINTED_CRITICAL` path wants a stronger mode.
Keeping the helper's signature free of gigablock types is therefore a requirement rather than good
manners: it takes a pfn and a migrate mode, so the prequel can land without anticipating the second
caller.
**Destinations must be clean, not merely not-the-source.** Putting evacuated pages in another tainted
gigablock moves the problem: that gigablock's headroom shrinks and it becomes the next target, with
ping-pong the worst case. Excluding only the source is not enough, because the nearly-recoverable
gigablocks are exactly the ones with the most headroom and therefore the ones a movable allocation is
admitted to. So: **an evacuation destination may only be a clean gigablock, or a zone with no
gigablocks. If none is available, the evacuation does not happen.** The justification is definitional
rather than tuned — the purpose of evacuation is to reduce movable content in the tainted set, so putting
evacuated pages there is counterproductive at any watermark. A gigablock-less zone is a fine destination
because it cannot host a hugepage anyway, and `alloc_migration_target()` is already node-scoped rather
than zone-scoped, so DMA32 is reachable without new plumbing.
**An evacuation destination is allocated with the barrier enforced.** A user allocation may relax it
with `ALLOC_NOFRAG_TAINTED_OK` as a last resort to avoid OOM; a destination never may, because that
would let evacuation spill into the set it is trying to empty. Same flag, opposite default, decided by
caller.
**The tool switches under pressure, which is what makes the restriction safe.** Once clean space runs
out migration is unavailable, and that is exactly when memory is tight — at which point migration's own
destination allocations drive reclaim on the ordinary path, which is where that judgement belongs.
**LRU order works against cleaning up mixed gigablocks.** Pages that spilled into kernel territory during
a spike are the most recently allocated, hence youngest, hence the last thing LRU-ordered reclaim frees.
kswapd will not clean them; only directed evacuation will. Undirected reclaim still helps
`headroom_pages`, since any freed page in a kernel pageblock counts. Keep the two metrics separate for
this reason, and do not defer directed evacuation because kswapd is busy.
**Evacuation has to drain the per-cpu lists first, or it fails on pages that are merely cached.**
`nr_kernel_pageblocks` counts content, and a pageblock whose only remaining content sits on some CPU's
`pcp->lists[]` still counts, so a gigablock can look occupied when nothing owns its pages. Isolation and
migration cannot reclaim those: the pages are neither on a free list nor on the LRU. `drain_all_pages(zone)`
before an evacuation attempt is what turns them back into free pages. It costs an IPI to every CPU with a
non-empty list, which is why this belongs on the sleepable path only, and it is a second reason
`TAINTED_FULL` must be defined by outcome — an attempt that fails because of caching would otherwise be
recorded as an unemptiable gigablock. The two outcomes §6.2 distinguishes are reported from here:
nothing movable found to isolate, against pages isolated and migration failed.
**Background evacuation throttles on the zone watermarks** and stops where the zone itself would need to
reclaim. Unthrottled it can drive a system into swap to preserve 1 GB availability, which is the wrong
trade for almost every workload. Synchronous evacuation for a specific allocation may proceed past that
point, because that allocation was going to stall anyway.
## 9. Invariants, and the tradeoffs behind the first one
1. **Every steering rule has a fallthrough to the path mainline would have taken, and it is reachable
without any evacuation, migration or reclaim having succeeded.** Spilling into a clean gigablock is
that fallthrough: bounded attempts, then take what mainline would have taken. So policy is never the
reason an allocation fails. Losing a 1 GB range is recoverable; an OOM is not.
2. A pfn with no gigablock behaves exactly as mainline.
3. Kernel content is confined to the tainted set: a clean gigablock holds no kernel pageblocks at all,
and the derived candidate test of §10.2 is the authority on which ranges are usable.
4. `zone->free_area[order].nr_free` equals the sum over gigablocks, and `NR_FREE_PAGES` stays
consistent. `CONFIG_DEBUG_VM` checks both.
5. A `has_free` bit is set if and only if the corresponding list is non-empty. `CONFIG_DEBUG_VM` checks
it.
6. Movable allocations enter a tainted gigablock while its kernel headroom is above `LOW`, or below it
only under `ALLOC_NOFRAG_TAINTED_OK`.
7. A gigablock returns to `CLEAN` when its last kernel pageblock is freed. Without
this the tainted set only ever grows, one transient spike costs a gigablock permanently, and the
mechanism decays with uptime until it is useless.
### 9.1 Invariant 1 constrains the last resort, and everything before it is policy
Read as a goal — "never fail" — it would imply always spilling, which lets the tainted set grow
monotonically until every gigablock is in it and the series has cost with no benefit. What it actually
constrains is the **last resort**. Everything ahead of that is policy, and policy may be arbitrarily
aggressive precisely because it cannot cause a failure.
### 9.2 The missing mechanism: reclaim on the kernel side
When the tainted set wants to grow, the options are evacuate, spill, or fail. The third option is to
**shrink kernel memory**, and it is not a small omission: Phase 0 measured `SReclaimable` at 50 GB of
66 GB total slab on this host, so most kernel memory is shrinkable by machinery that already exists
(`shrink_slab()` and the dentry and inode shrinkers). The kernel shrinks slab under *memory* pressure and
not under *fragmentation* pressure; adding the second trigger is the missing piece.
**What it recovers, from prior experience: pages, not gigablocks.** Shrinking mostly returns individual
pages, and a large part of what it frees is space *inside* slab pages that is never returned to the page
allocator at all. Whole gigablocks coming back clean has not been observed. So scope §9.2 to what it
actually does — feed `headroom_pages` (§6.1) so the tainted set can absorb more kernel allocation — and
do not count it toward contraction back to `CLEAN` (patch 19), which needs the pages of a whole gigablock
to leave, not scattered pages anywhere.
**The slab-internal free space is not a loss, it is the magnet working.** Space freed inside a slab page
stays with the slab allocator and is handed to the next kernel allocation, which lands in a pageblock
that is already tainted rather than opening a new front. That is taint-on-first-contact (§3.4) getting
help from an allocator that already prefers reuse, and it is a reason to expect the tainted set to stay
small — the assumption §4 now rests on.
### 9.3 Density as a cheap predictor, exhaustion as the hard gate
A fixed rule like "never spill while any tainted gigablock is under half used" has a failure mode:
density can be low because *movable* content is occupying the tainted set, in which case the blocker is
not density and the fix is evacuation. So density does not gate spilling. It is an excellent cheap
predictor, though, and that matters because the alternative gate is expensive: proving evacuation and
shrinking cannot help requires trying them, on an allocation path.
state sleepable cannot sleep
density below ~30% evacuate, bounded shrink, spill
then retry
any tainted gigablock not FULL evacuation has somewhere spill
to work: try it
all FULL and a shrink pass tried spill spill
everything above failed spill unconditionally (invariant 1)
**Only a sleepable allocation can choose evacuation over spilling**, and that asymmetry is not a
softening of the rule, it is the whole rule: `migrate_pages()` takes page locks, allocates destinations
and may wait on writeback. A `GFP_NOWAIT` or `GFP_ATOMIC` caller cannot execute a single row of the
left column, so for it every row reads "spill". Refusing to spill there would turn a lost gigablock
into an atomic allocation failure, which is the worse outcome by a wide margin. §6.2's
`TAINTED_CRITICAL` states the same split for the headroom watermark.
**Evacuation is a slowpath retry, not a step in the search.** The §5.1 order runs in `__rmqueue()`
under `zone->lock`, where nothing may sleep, so "evacuate, then retry from 2" cannot happen inline.
The fast path returns no page and `__alloc_pages_slowpath()` does the migration and re-enters — the
same shape §6.2 uses for the movable barrier. Reading §5.1's numbered list as a single call chain is
the easiest way to mis-implement this section.
**A successful evacuation needs a bound too.** `TAINTED_FULL` covers repeated *failure*, being defined
as "last attempt failed and nothing freed here since". Nothing yet limits repeated success: a stream of
sleepable kernel allocations can each trigger a migration pass, paying migration bandwidth to keep one
gigablock's headroom just above `MIN`. §6.2's bounded pass is the intended defence — a triggered
evacuation does a fixed amount of work and stops rather than tracking a watermark — so measure whether
that bound is sufficient before adding a rate limit.
### 9.4 The tradeoffs, spelled out
- **Latency against containment.** Every mechanism tried before spilling adds latency to a kernel
allocation. Evacuation is milliseconds; a dentry shrink on a large cache is worse. Strict containment
means slow kernel allocations under fragmentation pressure. This is the fundamental cost of the
approach and belongs in the cover letter.
- **Kernel reclaim can be a bad trade.** Shrinking dentry and inode caches to recover a gigablock throws
away work to gain a hugepage, and whether that is worth it is workload-dependent. Bounded and tunable,
defaulting conservative, with an off switch that returns mainline behaviour.
- **Contraction matters more than any spill rule.** If invariant 7 works the tainted set breathes and
unbounded growth does not happen. If it does not, no gate saves the design — it converts growth into
allocation stalls. Recovery is the primary defence; the spill ordering is second line.
- **The gate can never be a refusal**, or we have built an OOM source.
- **It is measurement-dependent.** If reclaimable slab is smeared one object per pageblock, shrinking
frees pages everywhere and whole gigablocks nowhere, and §9.2 is worthless. Measure first.
## 10. Scope and geometry
### 10.1 Gigablocks tile the zone
gb_start_pfn = round_down(zone->zone_start_pfn, PUD_PAGES)
gb_size_pages = round_up(zone_span / 64, PUD_PAGES)
nr_gigablocks = DIV_ROUND_UP(zone_end_pfn - gb_start_pfn, gb_size_pages)
Every gigablock exists and owns free lists, including incomplete ones at the zone edges, because their
pages need somewhere to live. Some gigablocks contain PUD ranges that can never be hugepage candidates.
Those are two different facts and must not be conflated.
**Gigablock boundaries are PUD-aligned**, which the formulas guarantee. Without that a PUD range could
straddle two gigablocks and "which gigablock owns this candidate range" would be ambiguous.
### 10.1a Boot ordering, taken from the existing implementation
Not a new problem. The v27/v28 superblock series solved it and v1 copies the shape:
**Build the arrays in `free_area_init_core()`, after `setup_usemap()` and before
`init_currently_empty_zone()`.** That is before the zone's free lists exist, so it is before any
`__free_one_page()` can run, and no maintenance site needs a "not built yet" guard.
**Allocate with `memblock_alloc_node()`, not `kvmalloc()`.** The page allocator does not exist yet,
which is the whole reason the ordering works. `kvmalloc` and the RCU teardown of §12a belong to
`resize_zone_gigablocks()`, and v1 has no resize: memory hot-add is out of scope (§10.1) and Phase 1
item 1 says boot-present memory only.
**A failed allocation leaves `zone->gb` NULL**, which is already the "this zone has none" state of §3.1,
so warn and continue. The zone behaves exactly like one that legitimately has no gigablocks, and every
call site no-ops through the same path rather than a second one.
Deferred struct page init frees large ranges where the maintenance is pure overhead. The arrays exist by
then, so this is a cost question rather than a correctness one, and §13's Phase 1 patch 3 is where the
init sweep decides what to do about it.
### 10.2 The candidate test for a PUD range
1. the whole range lies within this zone
excludes the partial ranges at the zone edges and ranges straddling a zone
boundary. Structural and cheap.
2. no pageblock in the range is MIGRATE_CMA
structural, and not expressible in the content bits: CMA content is movable,
but a CMA pageblock never changes migratetype, so the range can never be
handed to hugetlb.
3. no pageblock in the range has PB_has_unmovable or PB_has_reclaimable
the content test. Memory holes land here rather than in a case of their own,
because init_unavailable_range() marks hole pages PageReserved and §3.3 makes
PageReserved taint.
Derived on demand by the 1 GB allocator and by the debugfs metric: 512 bytes of reads per candidate
range, and 48 KB for the whole zone.
### 10.2a The doomed-but-usable ranges at the zone edges
Neither `zone_start_pfn` nor `zone_end_pfn` is PUD-aligned in general, so the first gigablock's lowest
range extends below the zone and the last gigablock's highest range extends above it. Both fail condition
1, both can never host a 1 GB page, and both hold real usable memory — up to `PUD_PAGES - 1` pages each,
just under 2 GB per zone. A range straddling a zone boundary fails the test in *both* zones, so each zone
may fill its half with kernel content at no cost to either.
PUD grid |....1G....|....1G....|....1G....|....1G....|....1G....|
zone [====================================]
^^^ ^^^^
doomed: range straddles the zone start / end, fails
condition 1 in both zones, yet holds usable memory
**So they are a two-entry list, fixed at init, and §5.1 consumes them first.** Kernel content there costs
nothing and spares a good range.
**The effect scales badly downward.** On this host Normal starts at pfn 1048576, which is PUD-aligned, so
only the end range is doomed: 1 of 766. On a 32 GB machine `round_up(32 GB / 64, PUD_SIZE)` is 1 GB, so
gigablocks *are* PUD ranges, `ZONE_NORMAL` from 4 to 32 GB gives 28 of them, and both ends give one each:
2 of 28, against a fixed cost of one more range for the vmemmap. 28 ranges minus one vmemmap, minus two
edges, minus three or four for kernel content leaves roughly 22.
### 10.3 What gets no gigablocks
Whole zones only. `ZONE_NORMAL` is the only zone type that gets them, and only when it spans at least 8
gigablocks:
- `ZONE_DMA` and `ZONE_DMA32` are device-constrained rather than general memory. A naive
`round_up(span / 64, PUD_SIZE)` would give DMA32 four 1 GB gigablocks on this host, and steering inside
a 4.3 GB zone that 32-bit devices depend on risks starving them to chase a hugepage nobody wants from
there. Excluding by zone type is simpler than tuning a size threshold to exclude them.
- Below about 8 gigablocks there is no room for a tainted set and a clean set to coexist, so the
machinery adds only cost.
`ZONE_MOVABLE` gets none because every page there is migratable by construction — that is why hot-remove
uses it — so any 1 GB range is freeable on demand through `alloc_contig_range()` and steering protects
nothing. `ZONE_DEVICE` gets none. Hot-added memory gets none in v1.
Every one of those is the same `zone->gb == NULL` fallback, so all are free to implement, and together
they delete the gigablock array resize entirely. v27 needs three different reader-safety contracts around
`resize_zone_gigablocks()` — `rcu_read_lock()` plus `synchronize_rcu()` for non-sleeping readers,
`get_online_mems()` for `page_reporting_cycle()` which drops `zone->lock` to call into a driver, and
nothing at all for readers holding `zone->lock` throughout — purely because the old array is `kvfree()`d.
No resize, no contract to get wrong, and `page_reporting` stops needing to know what a gigablock is.
### 10.4 Known bounds, stated rather than fought
- Order-0 allocations mostly never reach the buddy allocator; PCP absorbs them, and PCP frees and
reallocations do not consult gigablock policy. This bounds achievable purity at low order.
- A single long-lived unmovable page costs one 1 GB range until it is freed. That is the point of
confinement.
- Long-term GUP pins make a movable page unmigratable with nothing to distinguish it at allocation time
(§14.1).
- The vmemmap is 12.8 GB per node of permanently pinned memory, and on this host it sits at the top of the
zone because `memblock.bottom_up` is false by default and early allocations therefore cluster at the top
of available memory. `movable_node` flips that.
## 11. Salvage from v27
Reuse close to verbatim:
- the `PB_has_*` bits and their maintenance sites
- `pfn_gb_free_area()` — this is Phase 2's `pfn_free_area()`, already written and proven
behaviour-identical for add, del and move
- the `zone->free_area[].nr_free` exact shadow
- `contig_allocated` / `gigablock_contig_mark()` for the CMA and contig-range cases, and as the metric's
record of a successful 1 GB allocation
- the `fast_isolate_freepages()` cross-gigablock freelist fix-up, as the worked example in Phase 2's
changelog
- **`evacuate_pageblock()` and its async plumbing**, which is the prequel's P3 and P4 rather than part
of the gigablock series. 224 lines already
written and debugged: the isolate-and-migrate loop over one pageblock, the fixed 64-entry work-item
pool, the staleness recheck, and the irq_work hop that makes `queue_work()` reachable from under
`zone->lock`. Drop its `reclaim_clean_pages_from_list()` call per §8
Delete: `gb_lists[category][fullness]`, the PASS_1/2/2B/2C/2D category search machinery, `gb_hint` and
`gb_warm_hints`, `GB_AGGRESSIVE_THRESHOLD`, the per-gigablock `free_area` fullness ranking, and the
resize path.
The series applies to `848acc8ffe1b`, not to the v27 branch. Say so in the cover letter with this list, so
"small diff" is not read against v27's 4666 lines.
## 12. Open questions
Items 1, 2, 5, 7 and 8 are settled and kept so they are not re-raised. Live going into Phase 1: **3**
and **9** need v1's data, **6** needs a benchmark that does not exist on either side, and **4** is
deferred to a follow-up series.
1. **Deferred out of v1, not open: within-gigablock packing** (§4a). Build it if the tainted count
passes ~48 of 64 or clean-only candidate ranges approach 94. The quantities to measure then are
candidate ranges per tainted gigablock — 9 of 12 is the ceiling, 0 is what LIFO gives — and the
bounded-walk fallback rate.
2. **Answered: kernel-side shrinking recovers pages, not gigablocks.** Prior experience is that
shrinking mostly returns individual pages, and much of what it frees is space *inside* slab pages
that is never returned to the allocator at all. Whole gigablocks coming back clean has not been
observed. §9.2 is therefore a headroom mechanism, not a contraction one; see there.
3. **The PUD fast path's lock hold**, unmeasured: removing every free block of a content-free 1 GB range
from the lists under `zone->lock`. `pageblock_order` is 9 and `MAX_PAGE_ORDER` is 10, so a fully
coalesced free 1 GB range is 256 order-10 blocks, not 512 order-9, and the per-pageblock bit updates are
512.
4. **Compaction scope, split in two.** The correctness half is not optional and is not deferred:
`isolate_freepages()` scans pfns and survives, but `__isolate_free_page()` must route through the
gigablock's `free_area`, which Phase 2 does. The policy half is deferred to a follow-up series:
compaction migrating pages into a clean gigablock does **not** taint it, since taint is
`nr_kernel_pageblocks > 0` and movable content is evacuable, so making destination selection
gigablock-aware is an optimisation rather than a correctness fix, and which direction it should
prefer is exactly what v1's data will say.
5. **Answered: evacuation migrates and never reclaims** (§8), so no page is destroyed and no cgroup
protection is involved. Where reclaim is genuinely needed it is driven by migration's own destination
allocations, on the ordinary allocator path, which respects `memory.min` and `memory.low` like any
other allocation.
6. **No hugepage-success benchmark exists** on either side.
7. **Answered: solved in the v27/v28 implementation, and copied from it.** See §10.1a.
8. **Answered: `gb_above` is `[2][4]`.** Two request classes (§6.1), four levels HIGH, LOW, EVAC, MIN.
The order 4 and above class leaves `MIN` unused, since it has no floor below spilling; 64 bytes with
one word wasted is cheaper than two arrays of different length.
9. **The two fullness boundary values, which need v1's data.** The thresholds are two independent
constants, not one shared knob: for tainted gigablocks the numerator is `nr_kernel_pageblocks/total`
and grades concentration for the magnet of §3.4, for clean ones it is `(total - nr_free)/total` and
grades how little is left to protect when choosing what to spoil. Different questions, so no reason
the values should match. ~25% each is the starting point; the distribution of both ratios across the
tainted and clean sets is what says whether either is right.
Two things need no change and should be said up front, because both are obvious reviewer objections:
`__zone_watermark_ok()` reads the exact `nr_free` shadow so watermark checking is untouched, and
`fill_contig_page_info()` with its users `fragmentation_index()`, `extfrag_for_order()` and
`unusable_free_index()` read only `nr_free`.
## 12a. Where the code lands
v28 is the warning: it added **5034 lines to `mm/page_alloc.c`** and created no new file, which is why
its steering, accounting and search policy could not be reviewed or tested apart from the buddy
allocator. The split below exists to keep that from happening again, and the target is that
`mm/gigablock.c` answers *which gigablock* while `page_alloc.c` keeps doing *buddy mechanics* and gains
only call sites.
**mm/gigablock.c (new, the bulk of the series)**
gigablock_alloc_search() §5.1, §5.2 step order for both populations
gigablock_eligible_mask() §4.1 mask composition
gigablock_rank_fullness() §4.1 ranking among survivors
gb_get_fullness() §3.4 bucket, and the mask move on a crossing
gigablock_account_pageblock() §3.3 content bits, nr_kernel_pageblocks, taint
gigablock_watermark_update() §6.1, §6.2 band transitions and gb_above[]
gigablock_evacuate() §8, the sleepable path and TAINTED_FULL
resize_zone_gigablocks() §10.1 sizing, kvmalloc, RCU teardown -- not in v1, see §10.1a
gigablock_debugfs() §15 introspection
**mm/internal.h** — `struct gigablock_data`, `struct gigablock`, `enum gb_fullness`,
`enum gb_watermark`, `zone_has_gigablocks()`, and the `!CONFIG_GIGABLOCK` stubs that make every call
site above compile to nothing. Nothing here is exported outside mm/.
**include/linux/mmzone.h** — `struct zone` gains one pointer, `zone->gb`, placed inside the existing
`_pad1_` region (§3.1). That is the only change to a public header.
**mm/page_alloc.c — the seven lines that decide which list, plus call sites**
`zone->free_area` appears exactly seven times in `page_alloc.c` on the current base, and three of those
are inside the free-list accessors:
__add_to_free_list area = &zone->free_area[order]
move_to_free_list area = &zone->free_area[order]
__del_page_from_free_list zone->free_area[order].nr_free--
__rmqueue_smallest area = &zone->free_area[current_order]
__rmqueue_claim area = &zone->free_area[current_order]
__rmqueue_steal area = &zone->free_area[current_order]
unreserve_highatomic_pageblock area = &zone->free_area[order]
Replacing each with `gb_free_area(zone, page, order)` is the whole per-gigablock-free-list change, and
because the accessors are already the chokepoint, their 24 call sites need no edit at all. The remaining
`page_alloc.c` work is call sites rather than logic: `__rmqueue()` consults
`gigablock_alloc_search()` before falling back to today's behaviour, `should_try_claim_block()` gains a
gigablock term, `__free_one_page()` and the highatomic reserve paths call the accounting helper, and
`__zone_watermark_ok()` learns the headroom subtraction of §7.
**Files that gain a call, not a policy** — `mm/mm_init.c` (build the arrays after the zone is sized),
`mm/compaction.c` (target selection asks gigablock.c which gigablock, §8), `mm/page_reporting.c`
(§10.1's `zone->lock` drop), `mm/memory_hotplug.c` (resize), `mm/vmstat.c`, `mm/show_mem.c`,
`mm/debug.c` (counters and dumps).
**What the split buys, and the one place it may not hold.** Buddy mechanics stay testable without
gigablocks, the search policy becomes unit-testable through one entry point, and `!CONFIG_GIGABLOCK`
compiles the whole thing out through internal.h stubs. The risk is `gb_free_area()` on the fast path: a
function call where there used to be an array index, in a file that is not `page_alloc.c`, is where LTO
and inlining stop being an abstract concern. If it does not inline, that helper is the one thing that
moves back into `internal.h` as a static inline — and it is small enough to. Measure before assuming
either way.
## 13. Patch series
### Phase P — the prequel: pageblock evacuation
Ships as its own series, before and independent of everything below. It needs no gigablock, no per-zone
structure and no policy, and it is justified entirely by what happens today when `try_to_claim_block()`
steals a movable pageblock and cannot move everything out: the leftovers keep the stealing type from
filling the block, so the next allocation of that type steals another one.
P1. Extract `count_alike_pages()` from `try_to_claim_block()`, and the whole-block path with it
(§16.3a). Prep for P5, which changes the same function.
P2. The three `PB_has_*` content bits and their maintenance at the migratetype-change, whole-pageblock
free and allocate sites, plus the init sweep over `for_each_reserved_mem_range()` and the
`PageOffline` and hwpoison hooks (§3.3). Three spare bits, no array growth.
P3. Deferred-work infrastructure: the fixed 64-entry preallocated work-item pool and the
irq_work-to-workqueue hop, because `queue_work()` is not safe under `zone->lock`.
P4. `evacuate_pageblock()`: isolate and migrate over one pageblock, looping because
`isolate_migratepages_range()` stops at `COMPACT_CLUSTER_MAX`, with a staleness recheck of the
migratetype before starting. Migrate only, never reclaim (§8).
P5. Queue an evacuation when a movable pageblock is claimed for kernel use, skipping blocks whose
`PB_has_movable` is clear. This is what makes P2's bits load-bearing in the series that adds them.
P6. Counters: steals, evacuations attempted, succeeded, and dropped for want of a pool entry. Without
these the series cannot be argued for or against.
**How it sits with `defrag_mode`.** Johannes Weiner's `defrag_mode` sets `ALLOC_NOFRAGMENT` on every
allocation, reclaims at `max(order, pageblock_order)`, and drops `ALLOC_NOFRAGMENT` only after reclaim
and compaction have both failed. Its strategy is to **prevent** the mixing it can; the prequel
**recovers** the mixing it cannot prevent, and the two meet exactly at that last-resort retry, which is
the moment a steal is permitted. They compose rather than compete:
- kswapd cannot do this job even at `pageblock_order`, because the leftovers in a freshly stolen block
are the youngest pages and LRU-ordered reclaim reaches them last. Only a directed pass targets them.
- Under `defrag_mode` the evacuation's own destination allocations inherit `ALLOC_NOFRAGMENT`, so they
cannot fall back and pollute a second pageblock. The cost is that they fail more often, which
best-effort already tolerates. Measure with `defrag_mode` both on and off.
- The work is sequential, not duplicated: compaction failed to produce a whole block, one was stolen,
and the prequel makes the stolen one usable.
The objection to pre-empt is "make `defrag_mode` stricter instead". It cannot be: an unmovable
allocation cannot be refused forever, so the forced path is unavoidable, and today its leftovers are
pure loss.
### Phase -1 — tracing for the pageblock conversion paths that emit nothing
Half of all pageblock conversions are invisible (§1.2), which blocks validating a series whose premise is
controlling which pageblocks change type. Prerequisite instrumentation, independently useful upstream, and
it has a hard pass/fail: rerun the VM experiment and the 624-versus-318 gap should close.
`set_pageblock_migratetype()` and `set_pfnblock_flags_mask()` are inlined, so a kprobe is not an option
and the tracepoint has to go in the source. Check whether every conversion funnels through
`set_pageblock_migratetype()` or whether `change_pageblock_range()` and the highatomic paths need their
own.
### Phase 0 — measurement
Done, in `~/debug/gigablock-phase0/`. Outstanding refinement: a `page_has_movable_ops()` check plus
slab-cache attribution in the drgn pass, to turn the 12.3% upper bound into a real number, and open
question 2. Also worth having generally: for the caches that pollute, track how many *pageblocks* a slab
cache touches rather than how many bytes it holds. A 147 MB cache spread over 36k pages can spoil more
pageblocks than a 2 GB densely packed one.
### Where the cleanup patches sit
Not in a block of their own. Each cleanup lands immediately before the patch that changes the same
function, lettered `Na` so the numbering of the functional patches stays put. A reviewer reading patch
`Na` sees a function split along theme boundaries with no behaviour change, and patch `N` then changes one
named helper instead of reaching into a 166-line body. Reviewing the pair together is also the only way to
check that the split actually helped, which a cleanup block at the front of the series makes impossible.
Each is still justifiable without mentioning the series (§16.1) — adjacency is about review order, not
about weakening that test. Detail in §16.3.
### Phase 1 — inert infrastructure, nothing reads it
1. `struct gigablock_data` and the per-zone pointer: `round_up(zone_span / 64, PUD_PAGES)`, at most 64,
PUD-aligned, boot-present memory only, no resize, `ZONE_NORMAL` only and only at 8 gigablocks or more.
2. `pfn_to_gigablock()`, `zone_has_gigablocks()`, accessors. NULL and no-op when absent.
3a. Extract the `done_merging` tail of `__free_one_page()` into its own helper (§16.3a). The merge loop
stays as it is; `buddy_pfn` escapes it.
3. *(In the prequel as P2, listed here because the gigablock series depends on it.)* The three
`PB_has_*` bits plus maintenance at the migratetype-change and whole-pageblock free and allocate
sites, plus the init sweep over `for_each_reserved_mem_range()` and the `PageOffline` and hwpoison
hooks.
4. Per-gigablock `nr_free` and `nr_kernel_pageblocks`, and the `gb_tainted` threshold.
5. `has_free[mt][order]`, maintained from `list_empty()` and the
pageblock bits. Written, not yet read.
6. The two-entry doomed-but-usable range list, computed at init.
7. Free-list iterator helper for the four external readers — `page_reporting`, hibernate
`mark_free_pages()`, `show_mem`, `/proc/pagetypeinfo` — still iterating `zone->free_area`. Pure
refactor, reviewable by those maintainers without knowing what a gigablock is.
8. debugfs state dump. Lands here rather than in Phase 4 because Phases 2 and 3 cannot be developed or
debugged without seeing the tainted set, the candidate count and the per-gigablock state. Also carries
the counters the open questions of §12 are waiting on: the distribution of both fullness ratios, the
`TAINTED_FULL` entry count, and candidate ranges from the clean set alone.
### Phase 2 — move the lists, in two steps
A page has one `buddy_list` node, so the lists cannot exist in both places at once and the move cannot be
split by caller. It can be split by indirection.
9a. Rename `free_area_empty()` to `free_list_empty()`. It reads as a question about the whole
`free_area` but tests one migratetype's list, and patch 9 makes the distinction load-bearing.
9b. Split `fast_isolate_freepages()`, 166 lines, along the six themes of §16.3: `fast_isolate_setup()`,
`fast_isolate_scan_order()` for the b+c+d group that cannot be separated, `fast_isolate_fallback()`.
9c. Extract `compact_suitable_page_free()` from `__compact_finished()`, the one theme of four this series
replaces. Both functions touch the free lists directly — three references and two respectively — so
they are modified by patch 9 as well as by patch 16, and the cleanup belongs before the first of the
two.
9. Convert every free-list search and update site to `for_each_free_area(zone, order, mt, area)` and
`pfn_free_area(zone, pfn, order)`, whose first implementations yield exactly `&zone->free_area[order]`.
Mechanical, bit-identical, no gigablock knowledge. The changelog names the sites that need
`pfn_free_area()` to re-resolve a specific found page's owner rather than merely iterate — chiefly
`fast_isolate_freepages()`, whose `move_freelist_head()` reorder must act on the list the candidate
actually lives on, not the last one iterated.
10. Gigablocks own `free_area[]`: `pfn_free_area()` returns the gigablock's, `for_each_free_area()`
iterates through `has_free`, and one accessor does the list operation, both counters and the bit flip.
Small diff, all of the risk, trivially bisectable. This is where behaviour changes, and the change is
"allocate from whichever gigablock the bitmap finds". No taint preference exists yet.
11. `CONFIG_DEBUG_VM` cross-checks for invariants 4 and 5.
### Phase 3 — policy and evacuation
12a. Split `__zone_watermark_ok()` into `watermark_min_for_flags()` and
`zone_has_suitable_free_page()` (§16.3a).
12. `NR_FREE_KERNEL_TAINTED`, the `__zone_watermark_unusable_free()` subtraction, and
`ALLOC_NOFRAG_TAINTED_OK`. First, so the watermark is honest from the first patch that can create a
shortage.
13a. Lift the CMA-balance preamble out of `__rmqueue()` as `rmqueue_cma_balanced()` (§16.3a). The
`switch (*mode)` escalation ladder stays whole.
13. Kernel candidate order (§5.1), including the doomed-range step and the content-free-pageblock
preference.
14a. *(In the prequel as P1.)* Extract `count_alike_pages()` from `try_to_claim_block()`, and the
whole-block path with it (§16.3a).
14. Movable candidate order (§5.2), including steal-at-requested-order-only, and the
`should_try_claim_block()` rebase of §5.8.
15a. Give `__alloc_contig_migrate_range()` a range and a target policy instead of a
`compact_control` it builds itself. No behaviour change; it has one caller today only because it is
not callable.
15. Build the range-scale directed evacuation helper on it, for whole gigablocks and 1 GB targets. The
pageblock-scale helper already exists by now — it is the prequel's P4 — so this patch adds the
coarse scale only, and patch 17 adds the proactive trigger that asks for a pageblock when a tainted
gigablock is short of `headroom_blocks`.
16. `compact_suitable_page_free()` becomes a one-line substitution. Teaching compaction *destination
selection* about gigablocks is deferred to a follow-up series (§12 item 4): it is an optimisation,
not a correctness fix, since migrating movable pages into a clean gigablock does not taint it.
17. Headroom watermarks and states: synchronous evacuation for sleepable allocations, `kcompactd` wakeup
for the background path, throttled on the zone watermarks. Includes `TAINTED_FULL`'s two outcomes
and its 10 s lazy retry (§6.2). The retry ships with the bit rather than waiting for patch 19, so
the bit is never load-bearing without an exit.
18. Kernel-side shrinking (§9.2) and the spill ordering of §9.3, behind a tunable defaulting
conservative.
19. Contraction back to `CLEAN`, including the background re-derive of §3.3.
20a. Extract the conversion and accounting from `unreserve_highatomic_pageblock()`, 71 lines: three
nested loops with an early return in the middle, victim selection tangled with conversion.
20. Highatomic reserves from an already-tainted gigablock.
### Phase 4 — exploit
21. PUD-order fast path: satisfy a 1 GB request from a content-free PUD range with no migration. Measure
the lock hold.
22. Tracepoints and the debugfs metric: candidate ranges, tainted-set size, mixed-gigablock count, and
taint transitions. Three numbers, not one — candidates is what the series delivers, tainted-set size
is what it costs, and the mixed count is outstanding evacuation debt.
23. Documentation.
24. Tests (§15), as their own member of the series.
Phases 1 and 2 are the arguable part; a reviewer who accepts patch 10 has accepted the design. Phase 3 is
where the behaviour lives and Phase 4 is optional, so both can land incrementally or be dropped without
leaving the tree worse than before patch 1.
## 14. Future work
### 14.1 Long-term GUP pins, and the lifetime principle behind them
A `FOLL_LONGTERM` pin makes a movable page unmigratable for the pin's lifetime, so a pinned page in an
otherwise clean gigablock quietly makes that PUD range non-evacuable, with nothing at allocation time to
distinguish it. RDMA, vfio, io_uring fixed buffers and dma-buf are the volume cases.
**Segregate pinned content, and the reason is lifetime rather than movability.** A pin ends when the
workload restarts and its memory is freed, so whole gigablocks become available again. A random kernel
allocation mixed into the same gigablock may live until reboot, and mixing the two destroys the
recoverability of the shorter-lived one.
That generalises: **the axis worth segregating on is expected lifetime**, and movability is only a proxy
for it that breaks exactly here. In bitmap terms this is one set per lifetime class rather than bands in an
address ordering, which is also why the two-boundary problem an earlier positional version of this had
disappears.
**The mechanism to extend already exists.** `folio_is_longterm_pinnable()` already returns false for
`MIGRATE_CMA` and `MIGRATE_ISOLATE`, and `check_and_migrate_movable_folios()` migrates such pages before
allowing the pin, because unmigratability is unacceptable there. Clean gigablocks are the same kind of
region.
**Pin granularity decides the cost, and the two interesting sizes behave oppositely.** A 1 GB hugetlb page
pinned long-term costs nothing to place: it already owns its PUD range exclusively, so the pin adds no
constraint the allocation did not already impose. Nothing to migrate, no set to join, just an accounting
mark, and it frees wholesale on exit. That makes RDMA workloads that consume 1 GB pages the **best** case
for this series rather than the hard case, which is worth saying because the reviewer instinct is the
opposite.
2 MB pins are the hard case: big enough to matter, small enough to scatter. One pinned THP makes one
pageblock unmigratable and a PUD range is 512 pageblocks, so a single pinned THP costs a whole candidate.
100 GB pinned as 1 GB pages consumes exactly 100 ranges; the same bytes pinned as THPs is 51200
unmigratable islands that can spoil many times that number. So the pinned set must be filled at PUD
granularity: complete a partially filled pinned range before starting a new one.
**Measure first.** `/proc/vmstat` has `nr_foll_pin_acquired` and `nr_foll_pin_released`, equal at
614962909 on this host, so nothing is pinned at the moment and these workloads may not need any of this.
### 14.2 Deferred elsewhere
- **Per-section gigablock lifetime**, if 1 GB pages on hot-added memory ever matter. A PUD range is 8
sections, so hanging the struct off the memory section ties lifetime to section online state with still
no array to reallocate.
- **Per-PUD-range taint bits**, if the §3.4 threshold proves too coarse — specifically if measurement shows
tainted gigablocks losing clean ranges faster than the design needs them.
- **Evacuation cost-awareness**: a per-gigablock skip bit and last-attempt record, dropped from v1 because
correctness comes first. Add if futile attempts turn out to be frequent.
- **`pbsteal`**, making movable steal before it claims, is a dead end unless mTHP adoption makes orders 4
through 8 a large share of movable allocation. Two runs found mechanism A firing zero times, and the
reason is structural: `should_try_claim_block()` refuses movable claims below `pageblock_order / 2`, and a
claim at or above `pageblock_order` takes an already-free pageblock and leaves nothing behind. Branch
`scratch/riel/pbsteal` is kept as the record.
## 15. Tests
Guidance: tests ship as their own member of the series (5.Posting.rst via `~/kernel-style/patch-series.md`
§3); every patch builds and runs on its own; KUnit tests for mm live in `mm/tests/` with a `mm/Kconfig`
entry following `lazy_mmu_mode_kunit.c`; behavioural tests live in `tools/testing/selftests/mm/` using
`kselftest.h`, wired into the `Makefile`, `run_vmtests.sh` and the `config` fragment.
### 15.1 The decision that makes allocator tests non-flaky
Allocator tests earn their reputation by trying to *induce* a state through a workload and then asserting
on it. Fragmentation is stochastic, so those tests get marked flaky and then disabled. Do it the other way
round: expose each operation under test as a debugfs trigger, drive it directly, and assert on the reported
state. "Evacuate gigablock N and tell me what happened" is deterministic; "allocate until a gigablock
happens to need evacuating" is not.
Assert directional and threshold properties with generous margins, never exact numbers, and always print
the measured values so a failure is diagnosable rather than just red.
### 15.2 Every invariant gets a test
I1 no allocation fails due to policy
debugfs fault injection: force every evacuation attempt to fail, then run the
existing mm selftests and a memory hog. Nothing may OOM or fail that would
have succeeded on a base kernel. Highest-value test in the set, because I1 is
the invariant whose violation is unrecoverable.
I2 a pfn with no gigablock behaves as mainline
boot a VM too small to host a gigablock and run the full mm selftest set;
results must match the base kernel exactly. Plus a KUnit case that
pfn_to_gigablock() returns NULL for excluded zones.
I3 kernel content stays in the tainted set
slab-heavy workload, then read debugfs: candidate ranges outside the tainted
set must be within a stated tolerance of the pre-workload count.
I4 counter consistency
I5 bitmap-versus-list consistency
CONFIG_DEBUG_VM checks are the assertions; the test's job is to exercise the
allocator hard while they are on.
I6 movable never enters a gigablock below LOW without ALLOC_NOFRAG_TAINTED_OK
a stat counter for barrier stops plus a tracepoint. Under movable pressure
with a saturated tainted set, barrier stops must be non-zero and movable
allocations below LOW without the flag must be zero.
I7 the tainted set contracts
build a large dentry tree to grow slab, record the tainted-set size, drop
caches, assert it returns to within a tolerance of its starting value.
Deterministic, and it catches the failure mode where the mechanism decays with
uptime.
### 15.3 KUnit: the parts that are pure logic
`mm/tests/gigablock_kunit.c`, `CONFIG_MM_GIGABLOCK_KUNIT_TEST`. Runs anywhere, in CI, with no memory
requirements:
- sizing across zone spans from 1 GB to 64 TB, including the degenerate single-gigablock case, zones too
small to host one, and the PUD-alignment of `gb_start_pfn` and `gb_size_pages`
- the candidate test: partial ranges at both edges, ranges spanning two zones, ranges containing CMA,
ranges dominated by `PageReserved`
- the `has_free` search in both index directions, including empty, full and single-bit bitmaps
- the taint threshold and the watermark state derivation, including every boundary and the hysteresis gap
- the content-bit clearing rule: a pageblock that loses all content clears all three; one that keeps a
single kernel page does not
### 15.4 Selftests
Each must `ksft_test_result_skip()` rather than fail when its preconditions are absent — not enough
memory, no debugfs, no hugetlb, no gigablocks on this zone. `run_vmtests.sh` gains a size gate.
- `gigablock_taint.c` — I3 and I7, workload plus debugfs read
- `gigablock_pressure.c` — I1, I6 and the barrier, under fault injection
- `gigablock_1g.c` — allocate 1 GB hugetlb pages before and after a fragmenting workload and assert the
count does not fall. Model on `compaction_test.c`, which measures, fragments, measures and compares
against a fraction of RAM rather than an absolute count.
- `gigablock_thp.c` — THP availability must not regress, reported per eighth of the zone, since an
improvement confined to the already-clean part of the zone would hide a regression elsewhere.
A test module covers what userspace cannot produce: `GFP_ATOMIC` bursts from non-sleepable context, and
`GFP_KERNEL` order-0 storms faster than evacuation can keep up. Same Kconfig as the KUnit tests, driven by
debugfs, in the pattern of `lib/test_*`.
### 15.5 Tests that must exist beyond the invariants
- **Both gigablock size extremes.** A 32 GB machine gets 1 GB gigablocks and a 768 GB machine gets 12 GB;
the edge effect differs by a factor of 27 and both ends want a test.
- **Fork storm and netdev surge.** Many `GFP_KERNEL` order-0 allocations arriving faster than evacuation
can free space, to size the watermarks and confirm the latency bound.
- **Movable pressure with a full zone.** Free memory present only inside the tainted set, to confirm
invariant 1 and §7's watermark accounting rather than an OOM.
- **Memory offline while gigablocks exist**, and a hot-add run confirming the added memory gets none.
### 15.5a Proving the fast path did not regress
The plan predicts three times that the cost is negligible — §3.1's "a pointer perturbs nothing", §4's
"the index is a reduction rather than a cost to apologise for", §4.1's masks on the allocation path — and
never says how any of it gets checked. On an allocator series that is the first thing a maintainer asks
for, and each of those is a claim to falsify rather than a conclusion to state.
**Three configurations, because they separate three different costs.**
CONFIG_GIGABLOCK=n the code does not exist: the baseline
=y, zone_has_gigablocks() false a zone under 8 gigablocks (§10.3), so every
hook is compiled in and every one returns
early: measures the cost of the branches alone
=y, gigablocks active measures the policy
The middle configuration is the one that has to come out free, and it is the one that will run on the
majority of machines. If it does not, the problem is a hook placement rather than the policy, which is a
different and much cheaper fix — so measure it before the third.
**What to run.** Existing tools first: `will-it-scale`'s `page_fault1` and `page_fault3` for concurrent
allocation, `hackbench` and `kernbench` for mixed kernel allocation under load. Then a targeted
microbenchmark, which does not exist yet and has to be written: a debugfs trigger doing a tight
`alloc_pages()`/`__free_pages()` loop at orders 0, 3 and 9 for each migratetype, reported in cycles per
operation, so a change inside `__rmqueue()` shows up without a workload's noise on top of it.
**Verify the inlining rather than assuming it.** §12a's one real risk is `gb_free_area()` failing to
inline across the `page_alloc.c`/`gigablock.c` boundary. That is checkable directly and cheaply:
`objdump -d` the three free-list accessors and look for a call. Do it in the patch that introduces the
helper, not after the series is written, because the fix is to move the helper into `internal.h` and that
changes where later patches put things.
**State the result either way in the cover letter.** A series that reorganises the allocator's hottest
structure and reports no numbers invites the reviewer to assume the worst, and a small honest cost with a
named cause is far easier to accept than silence.
### 15.6 Will upstream accept knobs that exist for tests
Yes, and nothing needs inventing — four established patterns cover everything above.
- **A debugfs trigger whose consumer is a selftest.** `<debugfs>/split_huge_pages` in `mm/huge_memory.c`
is write-only, `CONFIG_DEBUG_FS`-gated, and its documented consumer is
`tools/testing/selftests/mm/split_huge_page_test.c`. `/proc/sys/vm/compact_memory` is the same idea in
the most analogous subsystem.
- **Fault injection.** Do not invent one: `mm/fail_page_alloc.c` with `CONFIG_FAIL_PAGE_ALLOC` under
`FAULT_INJECTION`, `setup_fault_attr()`, a boot parameter and debugfs attributes.
- **A test-only device behind its own CONFIG.** `CONFIG_GUP_TEST` provides `/sys/kernel/debug/gup_test`
and its Kconfig help text points straight at the selftest.
- **Fragmentation observability in debugfs.** `mm/vmstat.c` already exports the extfrag and unusable
indices under `CONFIG_DEBUG_FS` via `DEFINE_SEQ_ATTRIBUTE(extfrag)`. A gigablock state dump sits beside
them.
The distinction that decides acceptance is whether a knob is *only* scaffolding. Candidate-range count,
tainted-set size, per-gigablock headroom and the doomed-range list are genuine diagnostics — the first
thing anyone debugging a failed 1 GB allocation wants — so they belong with the extfrag files on that
basis. The evacuation trigger and the failure injection are scaffolding, so they go behind default-off
CONFIGs in the `GUP_TEST` and `FAIL_PAGE_ALLOC` pattern.
### 15.7 What does not ship
The Phase 0 A/B measurement harness needs a 64 GB VM, a fixed-work workload and three runs per
configuration. That is a measurement, not a pass or fail: it stays in `~/debug/gigablock-phase0/` and its
numbers go in the cover letter. Boot-matrix cases belong in CI rather than `run_vmtests.sh`.
## 16. Tech debt in the code the series touches
### 16.1 The test a prep patch has to pass
A preparatory cleanup must be justifiable **without mentioning this series**. Maintainers reject "refactor
so my work fits" and accept "this function is 166 lines and does three things". If a cleanup can only be
motivated by what comes after it, it belongs inside the patch that needs it.
**Placement is adjacent, not up front.** Each of these lands as patch `Na` immediately before the
functional patch `N` that changes the same function (§13). That keeps the pair reviewable together, which
is the only way to see whether the split earned its place, and it means a cleanup dropped in review takes
one functional patch with it rather than stranding the whole series behind a refactor block.
The related rule is that a series owns the functions it grows (`~/kernel-style/patch-series.md` §1), and
`scripts/series-function-growth.py` reports the offenders. Its `function_lengths()` handles multi-line
declarations correctly; do not write a second parser, because a naive regex silently misses exactly the
functions this is for.
### 16.2 Lengths of the functions the series touches
Measured on `848acc8ffe1b`:
166 mm/compaction.c fast_isolate_freepages
116 mm/compaction.c isolate_freepages_block
113 mm/compaction.c isolate_freepages
109 mm/compaction.c __compact_finished
109 mm/page_reporting.c page_reporting_cycle
87 mm/page_alloc.c __free_one_page
77 mm/page_alloc.c __zone_watermark_ok
71 mm/page_alloc.c unreserve_highatomic_pageblock
65 mm/page_alloc.c try_to_claim_block
63 mm/page_alloc.c __rmqueue
55 mm/page_alloc.c __alloc_contig_migrate_range
49 mm/page_alloc.c __rmqueue_claim
45 mm/page_alloc.c prep_move_freepages_block
43 kernel/power/snapshot.c mark_free_pages
### 16.3 The prep patches, split by theme
**1. `fast_isolate_freepages()`, 166 lines.** Six themes, and the interesting part is which must *not* be
separated:
a decide where and how hard to search the cc->order <= 0 bail, the scan_start
choice, distance/low_pfn/min_pfn, the
search_order clamp
b find the best candidate in one order the reverse list walk, tracking highest,
accepting pfn >= low_pfn, remembering the
best >= min_pfn, honouring limit
c reorder the list for next time move_freelist_head(freelist, freepage)
d take the page __isolate_free_page, counters, cc->freepages[]
e what to do when nothing was found the if (!page) block
f record progress compact_cached_free_pfn, total_free_scanned,
fast_isolate_around
`fast_isolate_setup()` for a, `fast_isolate_scan_order()` for **b+c+d together**, `fast_isolate_fallback()`
for e, leaving f in a ~30-line main function. b, c and d stay together because c's correctness depends on
`freepage` referring to the list b actually walked, and d must run under the same `zone->lock` acquisition.
Separating c from b is precisely the mechanical split that would make review harder, which is why this
function is the worked example for CS-13.
**2. `__compact_finished()`, 109 lines.** Four unrelated decisions: scanners-met, proactive compaction,
`defrag_mode`, and the direct-compactor "is a suitable page free" loop. Extract the last as
`compact_suitable_page_free()`, because it is the only theme the series replaces, making the later change a
one-line substitution instead of surgery inside a four-way branch.
**3. `__alloc_contig_migrate_range()`, 55 lines.** One theme in the wrong home. Split the isolate, reclaim
and migrate loop from the `compact_control` setup so it takes a range and a target policy. It has one
caller only because it is not callable.
**4. `unreserve_highatomic_pageblock()`, 71 lines.** Victim selection across the zonelist, then conversion
and accounting. Extract the conversion; three nested loops with an early return in the middle.
**5. Rename `free_area_empty()` to `free_list_empty()`.** It reads as a question about the whole
`free_area` but tests one migratetype's list, and the series makes the distinction load-bearing.
Their series positions are 9b, 9c, 15a, 20a and 9a respectively — each one patch ahead of its consumer.
**A function modified twice gets one cleanup, before the first.** `fast_isolate_freepages()` and
`__compact_finished()` both touch the free lists, so patch 9 rewrites them and patch 16 changes their
behaviour. Splitting them at 9b and 9c rather than 16a and 16b keeps patch 9's mechanical diff readable
and leaves patch 16 the one-line substitution it was supposed to be.
### 16.3a The `page_alloc.c` functions, one at a time
Read on `848acc8ffe1b`, counting braces to braces. Two of the five come out under the cap, two come out
over it on purpose, and one should not be touched.
**`__zone_watermark_ok()`, 77 lines, 25 of them comment. Split, and it is the clearest win of the five.**
Three themes with no shared mutable state:
a lower the bar for reserve callers ALLOC_RESERVES, ALLOC_MIN_RESERVE,
ALLOC_NON_BLOCK, ALLOC_OOM: pure
arithmetic on a local `min`
b the order-0 answer two lines
c is any suitable page free the descending order loop over
free_area[] and free_area_empty()
`watermark_min_for_flags(mark, alloc_flags)` returns a long and touches nothing else.
`zone_has_suitable_free_page(z, order, alloc_flags)` is the loop. That leaves about twelve lines in which
the order of the two checks is visible at a glance, and it puts each of the series' two changes inside its
own helper: §7's headroom subtraction belongs to a, and c is what becomes gigablock-aware. Lands as 12a.
**`try_to_claim_block()`, 65 lines, 23 of them comment. Split one theme out, and it is the one carrying
the subtlety.** Computing `alike_pages` is twenty lines of reasoning about what a non-movable page in a
movable pageblock can be assumed to be, and it is a pure function of four scalars:
`count_alike_pages(start_type, block_type, free_pages, movable_pages)`. Extracting it puts the count next
to the threshold test that consumes it, instead of a page apart. The `current_order >= pageblock_order`
whole-block path is a second candidate — seven lines, self-contained, early return — and taking both
leaves roughly forty lines. §5.8's gigablock term goes into the threshold decision, which is exactly what
becomes readable. Lands as 14a.
**`__free_one_page()`, 87 lines, 16 of them comment. Split the tail only, and accept staying over the
cap.** The obvious extraction is the merge loop, and it is the wrong one: `buddy_pfn` is assigned inside
the loop and read *after* it by `buddy_merge_likely()`, and `page`, `pfn` and `order` all mutate across
iterations. A helper would need four out-parameters or a state struct to say what the loop says now, and
`compaction_capture()`'s early return carries a compensating `account_freepages()` that a careless
extraction drops. Take the `done_merging` tail instead — set the order, choose head or tail, add to the
list, notify reporting — which is self-contained, about fifteen lines, and is where the series' free-path
accounting hook goes. That leaves roughly sixty-five lines, still over the cap, and the changelog should
say the merge loop was left deliberately rather than let a reviewer assume it was missed. Lands as 3a.
**`__rmqueue()`, 63 lines, 18 of them comment. Extract the preamble; leave the switch.** The `switch
(*mode)` is four cases chained by `fallthrough`, and the escalation order — matching list, CMA, claim,
steal — is the entire content of the function. Hiding a case behind a helper would hide the fallthrough,
which is the one thing the reader is there for. The CMA-balance preamble ahead of it is a different
decision that happens to be adjacent: `rmqueue_cma_balanced(zone, alloc_flags)` as a predicate leaves
about forty-eight lines and makes the escalation ladder the first thing in the body, which is where the
gigablock search inserts. Lands as 13a.
**`__rmqueue_claim()`, 49 lines, 14 of them comment. Do not split it.** One descending loop implementing
one policy, largest fallback block first, with a three-way result and an early break; the body is under
twenty lines of code. Any extraction here is a helper called once taking six parameters, which trades a
readable loop for an unreadable signature. The series changes which `free_area` the loop reads, not what
the loop does, so there is nothing for a prep patch to prepare. Leave it at 49 and say so.
**Two of five stay over 40 after cleanup**, on the grounds above rather than by omission. `kernel-style.md`
§3 treats the cap as a prompt to examine rather than a limit to satisfy, and a declined split should be a
recorded decision.
**Every prep patch, and where it lands.**
3a __free_one_page() tail only, stays over cap §16.3a
9a free_area_empty() rename
9b fast_isolate_freepages() six themes §16.3
9c __compact_finished() one theme of four §16.3
12a __zone_watermark_ok() two helpers, biggest win §16.3a
13a __rmqueue() preamble only, switch stays §16.3a
14a try_to_claim_block() count_alike_pages() §16.3a
15a __alloc_contig_migrate_range() make it callable §16.3
20a unreserve_highatomic_pageblock() conversion out §16.3
__rmqueue_claim() declined, 49 lines §16.3a
**Left for a separate cleanup series.** `isolate_freepages()`, `isolate_freepages_block()` and
`page_reporting_cycle()` are all over the cap, but the series either does not modify them or touches a line
or two. Splitting them is somebody's cleanup series, not a prerequisite for this one, and bundling them
would make the series look like a drive-by refactor of `mm/compaction.c`.
### 16.4 Debt to name but not fix
**`struct free_area` accounting is asymmetric.** `free_list[]` is per migratetype but `nr_free` is a single
total across all of them, so "is there a free block of this order and type" is a `list_empty()` while "how
many blocks at this order" is a count that cannot answer it. That asymmetry is precisely why the design
needs a bitmap rather than reusing counters. Making it coherent means either per-migratetype counters on
the hot path or dropping `nr_free`, both behaviour-affecting refactors of the allocator's hottest
structure. State it in the cover letter as the reason the bitmap exists.
**Comments around the claim path may be stale.** The fallback code was reworked between `e1914add2799` and
`848acc8ffe1b` (`-2` became `FALLBACK_NOCLAIM`) and `should_try_claim_block()`'s comment block predates
that. Read it on the new base before quoting it.
