What a plan costs#

Prior state is rebuilt by reading the live system every time you plan, because nothing stored is trusted. That reading is two costs, and they grow along different axes. Which of the two dominates depends on the size of your estate, and the answer flips.

Both belong to hooks, and the hooks differ in when they are needed. The read pass is unconditional and costs what stock’s refresh costs. The sweep is the adoption hook, which answers a question an operator needs during a migration or an audit and not on an ordinary plan of an estate that is already adopted. That it should not run on every plan was the stale-state ruling (#604), and 09d180f921 implemented it. This page is the measurement that ruling rested on, and it is still the measurement of what the sweep costs when a run does take it.

Read this page as the sweep’s cost. Since 09d180f921 a plan of an estate that has its own evidence to narrow by - types declared in configuration, or holding a key in the record store - no longer enumerates the whole admission table, and the 79-instance fixture measured throughout this page went from 710 API calls to 157, against stock’s 150. Every full-sweep figure below still describes a run where the narrowing has nothing to narrow by - an adoption, an audit, a rebuild from markers - because every gate fails toward doing the work. The exact gates are below. It no longer describes an ordinary plan of an adopted estate. The scales above 79 instances have not been re-measured in calls since. What a steady-state plan costs and what is still outstanding is on what you pay, and when.

The two terms#

The sweep asks what this estate owns, and it is O(types). One estate-filtered tagging call covers the types whose ARNs the hand-curated join table can resolve; every other admitted type is routed to the native leg and gets its own list attempt. The work is set by the size of the admission table, and it does not grow with your estate. Counted at 5d55f4aa9f, and reproducible in under a second with no cloud and no emulator:

go test ./internal/live/discovery/ -run TestSweepUniversePartitionIsMostlyNative
sweep universe=1027 tagging_leg=35 native_leg=992

Not all 992 of those reach the network. Only 502 of them can issue a ListResources at all; the rest report a sweep gap without a call, because the type is either not listable or not taggable. Measured at 5dbe452a1e for #586, where 435 of the 502 fired against the 79-instance fixture below in its unmigrated state.

The read pass asks what each owned resource currently looks like, and it is O(resources). One or more provider Reads per instance the plan materializes, how many depending entirely on which resource types you have. That is the same work a stock refresh does, and it measured equal to stock’s per resource at every scale below, to the call.

When the native leg is narrowed, and when it is not#

The narrowing is the whole difference between the figures on this page and what a steady-state plan costs, so the conditions under which it happens are operator-facing rather than an implementation note. They are in internal/live/discovery/nativesweep.go, and every one of them fails toward doing the full work.

It narrows the native per-type leg and nothing else. The tagging leg’s single estate-filtered GetResources, the record store’s own orphan walk, and the parent-read and fold-child legs all run exactly as before. The one question being declined is the account inventory, which asks what is in my account that this estate does not know about.

It narrows only where there is positive evidence to narrow by, and that evidence is the estate’s own record store. All four of these take the full universe:

• the run asked for the account inventory (-adoption-only, or TOFU_LIVE_COLLECT_UNCLAIMED=1);
• no record store opened for the pass;
• a record store opened and would not list;
• its listing came back empty.

The last two are the ones worth planning around. A fresh estate, and an estate whose store has not been written yet, still pay the whole admission table. That is by design rather than an oversight, because an estate with no record of itself has only its markers to say what it owns, and it is also the rebuild-from-markers path. Note what the gate is not: declaring a record_store block, since an estate that names none gets an implied local one anyway.

Given a non-empty store, the kept set is deliberately generous. It holds every type the configuration declares an instance of, every type the declared set routed through discovery or through the record rung, and every type the store holds a key for. A false positive there costs one list call; a false negative costs a removal nobody proposes.

A narrowed plan gives up exactly one shape of removal, and it is worth stating in full. Take a live object carrying this estate’s marker, of a type that:

• the configuration does not declare;
• the record store has no entry for; and
• the ARN join table cannot place from an ARN.

Its destroy is not proposed. Every other removal is unaffected, which TestNarrowedNativeSweepStillProposesRemovals and the day2_remove gauntlet stages check by value rather than by argument.

A narrowed plan says so. The “Foreign resources” section prints the count it skipped and the command that asks anyway, rather than letting silence read as “there is nothing out there”. 987 below is the fixed sample TestForeign_narrowedSweepSaysSo (internal/command/views/live_plan_nativesweep_test.go) renders the message with, not a measured type count - it names the shape of the sentence, not a current value, so it is not the same quantity as the sweep universe (1027) above or the admission-table size elsewhere on this page:

This run did not ask which live resources carry no ownership marker at all, so
987 admitted types this estate has no record of ever having used were not
listed. Every resource this estate owns was still swept for. Run "choudoufu
plan -adoption-only" for the account-wide question.

There is one case where narrowing is deliberately not attempted at all. TOFU_LIVE_CLOUDCONTROL=off selects the other sweep leg, which has no cheap estate-wide oracle standing behind it. No GetResources call covers the types the narrowing would skip, so skipping them there would remove coverage with nothing underneath. That run pays the full universe whatever the record store holds.

The measured split, on a migrated estate#

This is the day-2 shape and the one to start from: an estate already adopted, every declared instance carrying its markers, which is where an operator actually plans. Generated terralith at three scales, applied with stock terraform and then migrated with choudoufu live-import -approve before anything was counted (commit cfd0dc58d4, floci pin sha256:c55d74e1, reported in the slicing measurement (#584, corrected by #634):

The three rows are not the same vintage, and only the first has been re-measured. As published, the 79-instance row read 521 / 558 / 706 / 21.0%. Re-run at 5ff7f43f5b its legs read tagging 1, native 512, configuration scan 26, boundary 9, post-sweep 0 - sweep 548, total 696. The read pass did not move. That nine-call drift in the native leg is unrelated to the #628 provider-block defect that corrupted CLI-plan counts elsewhere on this page: the in-process bench configures its provider from a literal three-flag body that never carried skip_requesting_account_id. The 301- and 745-instance rows have not been re-run and stand at their published values.

The two legs do not add up to the sweep on their own. The rest of it is the configuration scan, 26, 58 and 124 calls at the three scales, plus a boundary and post-sweep pass of about ten calls. Those four account for the sweep exactly at 79 instances and to within one call at the two larger scales, where a type whose scan records nothing fires no progress event and its calls fold into the next attribution interval.

Both terms are linear; fitted to the three rows as published, sweep = 545.9 + 0.15315N and read pass = 1.8378N + 2.8 cross at 322 instances, just past this fixture’s scale 4. Take that as the shape rather than a current number - the line was fit before the 79-instance row moved by ten calls, and re-fitting across one re-measured row and two published ones would describe no run that ever happened. Below the crossing a plan is mostly the fixed sweep; above it, cost tracks your estate - the shape has not changed. This is a crossover between choudoufu’s own two terms on a full-sweep run, not between choudoufu and stock - there is no such crossing, as what you pay, and when sets out.

The read pass is the number stock pays to read the same resources#

Measured on both sides of the same estate, the per-resource work is the same and the totals differ by a constant:

A constant two calls separates the two. Stock’s provider block resolves its own account with one GetCallerIdentity and one GetUser; the read pass has no equivalent, since nothing in it needs the account identity. The read pass fits 1.8378N + 2.8, and stock’s own two-point fit is 1.84N + 5 - the same line, two more calls of constant. 745 was not re-run on either side; 1374 is what stock’s shared slope implies rather than anything anyone counted.

So the shared term is the resource reads: the read pass is the AWS provider’s own Read implementations, which stock invokes on the same resources when it refreshes, and nothing in this fork adds to them or can subtract from them. live/plan-budget.json says the same of its own figures: the shape “is a property of the AWS provider’s own Read”.

Above stock, everything choudoufu spends is the sweep - but that is a claim about API calls on a run that sweeps in full, and it holds only there. It does not describe a steady-state plan, and it does not survive the move to seconds. At 745 resources on real AWS, counting the requests the AWS provider itself logs, stock issues 1392 and choudoufu 1399, seven apart, while the wall clock reads 22–39 s against 123–124 s. Seven requests do not cost ninety seconds. That count excludes choudoufu’s own Cloud Control and Tagging clients, which log no line per request, so it is a floor rather than a total; what is spending the ninety seconds is unaccounted for, and this page will not guess.

The sweep is the term that is genuinely ours. Stock has no equivalent, because a state file already answers the question the sweep asks.

Both tools refresh at the same size on a default plan. The honest difference shows up under -refresh=false, in what each side may skip. Stock skips everything and trusts its state file outright. Choudoufu skips only what the run can vouch for (an instance the sweep verified by marker, or one the record store attests while the run’s own listing proves it exists); everything else still reads. The state cache supplies attributes for what is vouched, the plan launcher never plans those wire reads, and reads = "full" turns the whole pass off. The unchanged-is-free claim measures it; default plans are untouched, since the read is drift detection.

The native leg does not move#

native_sweep_calls measures 512 in every configuration the slicing work covered: whole estates at all three scales above, both slices of a two-way split, and each of eight slices of an eight-way split. It does not grow with the estate, and it does not shrink when a configuration declares fewer types. (It read 521 in all thirteen when that work was published, and 512 on the re-measure at 5ff7f43f5b; the split table above accounts for the nine calls. Flat is the property that matters, and it is still flat. What did change is who pays it, which the section above and 09d180f921 cover.)

The second half of that runs the wrong way round from most people’s intuition, so here is the mechanism. sweepTypes builds its universe by removing the types the configuration declares from the admission table, so a slice declaring five types has a sweep universe of 1022 to 1026 against the whole estate’s 1021. A small slice pays slightly more than the whole estate does.

The consequence for an already-sliced estate is where the sweep actually hurts: because it does not shrink per slice, its cost multiplies with slice count even though a steady-state plan’s does not - the same 512-calls-per- slice figure 5ff7f43f5b measured above, times the number of slices, 4096 summed at eight, on any run that sweeps in full. What you pay has the steady-state ratio table (1.05x/1.07x/1.21x at k=1/2/8) and the choice this leaves an adopter with.

On real AWS the sweep was nearly the whole plan#

Call counts say what the two terms are. Seconds say which one an operator notices, and on this estate those were not the same answer.

live/live-cert/terralith-scale.sh times both binaries on the same estate inside one certification run. From #578’s real-AWS run at scale 1, 79 resources in us-east-2, provider warm on both sides, TF_LOG unset on both, three runs each, every plan reporting zero changes so that each pair is the same operation:

Stock finishes the read pass, the term both sides share, in three seconds. The sweep is the remaining 200 seconds. Spread over the 558 sweep calls counted at that scale it is about 0.36s each, which is one network round trip apiece, and at the time of that run the sweep made them one after another.

Three bounds on that paragraph. The seconds are real AWS and the call counts are the emulator, so 0.36s per call is an estimate built from two measurements, not a measured quantity - live/FLOCI.md sets out when two wall clocks may be combined. The table predates the sweep going concurrent (next section) and predates the narrowing, so a steady-state plan of this estate no longer looks like the second row at all: the same pair now reads 3, 4, 3s against 17, 18, 17s. The 200s column is what a full sweep cost sequentially on a real account; an ordinary plan costs far less.

The sweep now overlaps its own waiting#

The admission table fixes how many calls there are. Nothing requires them to be made one after another, and since #605 they are not: Discover prefetches the sweep’s per-type listings through a bounded worker pool, DefaultSweepParallelism = 10 (internal/live/discovery/sweepconcurrency.go), the same bound stock plans an estate at. It covers the sweep’s per-type listing and nothing else - the config-driven scan, the tagging leg’s single GetResources, and the parent and record-orphan reads are untouched.

The call count does not move, which is the point. Measured at 177a2579c1 against the pinned emulator at four settings and both scales, 558 calls at 79 instances and 591 at 301, identical at parallelism 1, 2, 10 and 20, with the scan-row order and the diagnostic sequence identical too:

Those are milliseconds over loopback, so they measure the overlap and not the saving. A repeat of each parallelism-1 row landed 18% lower (357.4ms and 355.7ms), so read the ratios as approximate.

This prefetch pool is one of four things that bound a plan’s seconds in v0.5.0, alongside the read pass learning the same (#626), the narrowing that takes the native leg off a steady-state plan entirely (#627), and the record store’s round trips falling to one per plan (#636; what you pay, and when carries the figure and its own staleness note). Overlapping a leg and not running it are different mechanisms, and the narrowing does most of the work at 79 resources.

What that adds up to in seconds is on what you pay, and when; it carries the wall-clock figures and states what each one rests on. This page is the mechanism; that page is the number.

Turning a phase down#

Both terms overlap their own waiting, and each has its own bound. The two are separate settings because they are separate phases. Neither of them is stock’s -parallelism - that flag bounds the graph walk and nothing on this page.

Each bounds the calls a phase has in flight. Each also has a second bound behind that one, on the answers it has fetched and the consuming loop has not used yet, and neither of those has a variable of its own: turning a phase down is turning down what the account is asked for, which is the width, and the buffer follows it.

For the read pass that is a hundred per in-flight slot, so a thousand at the default width. Until #683 one number was both, and an answer that had landed went on holding the width until the loop reached that instance in build order - so a single read in a provider backoff, 26 seconds of it on a 745-resource plan, stopped the pass from starting anything else at all.

The sweep had the same shape and the same defect, one phase over (#839): its listings were released by the scan loop in universe order rather than when the call returned, so one throttled list call held the sweep’s whole width behind it. Its buffer is ten per slot rather than a hundred, because an unconsumed listing here is every live object of its type and the scan drops those objects once it has filed its row - read-ahead the run would not otherwise pay for at all, where the read pass’s answers duplicate objects prior state ends up holding anyway. Ten per slot is worth about thirty-six seconds of sweeping at the rate the timing table above measures, which is what the straggler it covers costs.

Peak memory is still a multiple of the two bounds and never of the estate or of the admission table, which is what the single number was protecting.

What the split was worth, measured#

#867 re-took #683’s trace on the same estate after both fixes landed - us-east-2, provider 6.59.0, choudoufu built from d455a2fed4, harness and instrument at 3889d2476c, 2026-09-06. Three steady-state choudoufu plan runs, and three stock terraform plan runs of the same estate in the same session, so that the account’s own throttling is roughly the same on both sides of the comparison. An idle gap is a stretch of at least 0.8 seconds during which no AWS request is in flight at all; live/live-cert/wallclock-gaps.py is the instrument, and it is in the tree rather than on a branch this time.

The fork’s extra three hundred are not the read pass, which still makes stock’s calls call for call. They are the sweep’s two client-side-filtered listings, aws_iam_policy and aws_iam_role, which enumerate the whole ACCOUNT rather than this estate: GetPolicyVersion 102 to 201, ListRolePolicies 113 to 226, GetRolePolicy 203 to 308 between the stock column and the fork’s. That column is therefore not a property of this estate alone - the test account also held objects earlier runs had left behind - and it is why the fork’s span here is longer than #683’s on the same estate.

The fork’s idle share is not the number to read on its own. An account does not throttle the same way twice: stock’s own share moved from 20% in #683’s session to somewhere between 0% and 42% in this one. What compares is the fork’s share against stock’s in the same session. #683’s captures, put through this same instrument (3889d2476c), read 49% and 56% idle against stock’s 20% - about two and a half times stock. Here the fork is 6% to 20% against stock’s 0% to 42%, which is below stock, and the worst single stall a choudoufu plan took, 4.68s, is shorter than the worst stock took on the same estate minutes earlier, 8.04s.

Every stall on both sides, nineteen of them, ends in a retrying request line, so what is left of the idle at d455a2fed4 is the provider’s own backoff schedule rather than anything either binary decides. On the fork’s side every read-pass stall falls in the last quarter of its run: while there are reads left to launch, a stalled one holds an in-flight slot and no buffer slot, so the launcher keeps going, and the residue is the tail, where fewer instances remain than the width and a slow one has nothing left to overlap with. #683’s stalls were spread across the whole run, because back then any one of them stopped everything.

The sweep showed no straggler, and this estate cannot produce one. Two throttled list calls across the three runs, costing 1.23s and 1.51s, measured at d455a2fed4. Thirty-two of this estate’s swept types are answered by the single estate-filtered GetResources described above, which takes no per-type slot at all; only 3 types - aws_ecs_service, aws_iam_policy and aws_iam_role - take the per-type list path the sweep’s bounds cover, on the first post-migration plan and on a steady-state one alike. Three outstanding calls against a width of ten means at most three listings are ever fetched and unconsumed, so the sweep’s buffer is never reached and a factor of one would have produced the identical run. Ten per slot therefore still rests on the derivation above rather than on a measurement; testing it needs an estate whose types mostly lack a server-side tag filter, which is also the only shape in which #839’s defect could have cost anything.

Set either to 1 for the sequential loop, one call at a time in the order the phase would have made them. A value below 1 is refused rather than read as “no limit” - the read bound’s refusal lands before the run reads anything at all, because it is resolved before the configuration is even loaded.

Neither changes what a plan costs in calls. The sweep’s counts were measured identical at 1, 2, 10 and 20 in the timing table just above; the read pass makes one import and one read per instance whatever its width, which is a property of the loop rather than something anyone had to measure. What the settings change is how much of the waiting overlaps, which is why the reason to touch them is a real account answering Rate exceeded rather than a wish for a cheaper plan.

Both defaults are 10 because stock plans an estate at -parallelism 10. That argument is the stronger of the two for the read pass, which makes call for call the same requests a stock refresh of the same estate makes - the stock-versus-choudoufu table earlier on this page - so ten asks an account for exactly what it already answers for OpenTofu. Read-side throttling cannot be measured from an emulator, since floci does not throttle, and it has now been measured on a real account instead: at d455a2fed4 a steady-state plan of the 745 instances above was throttled 43 to 46 times per run at this width, every one of them retried and answered, with the section just above for what the waiting cost. That is the account tolerating ten concurrent reads, not refusing them.

live-mv honours the read bound and has no sweep to bound: a rename lists one resource type rather than the estate. live-import’s own -parallelism flag is a third thing again, the width of its stamp pass, which neither variable moves.

Turning the account inventory off, or back on#

TOFU_LIVE_COLLECT_UNCLAIMED is not a width. It is the on/off for the account inventory, the question the narrowing declines, and it is the only one of the three settings here that changes what a plan costs in calls rather than how much of the waiting overlaps.

Anything else errors and quotes the value it could not read. The variable exists beside the flag rather than instead of it because live-plan’s own -estate form and plain apply have no -adoption-only to reach for.

Turning it on is the expensive direction and it is the one to reach for deliberately: on the 79-instance fixture it is the difference between 157 and 710 API calls.

The unmigrated estate, for contrast#

The same fixture and the same pin, measured before migration with no marker on any object (commit f4611196e5, the fast-projection ruling (#579)):

sweep = 548.3 + 0.1486N, read pass = 1.1486N - 4.7, crossing at 553 instances.

This is the one table on the page the narrowing does not move. An estate with no marker on any object has no record store keys either, and an empty store is one of the four gates that takes the full universe. The adoption case still pays the whole admission table, by design.

It is the plan you run on the way in rather than the ones you run afterwards, and it is a lower bound on the read pass: nothing was bound, so much of the estate never materialized. Migration leaves the sweep alone and raises the read pass by 61% to 72%. Per instance the read pass goes from 1.15 calls to 1.84, and the crossover moves from 553 instances to 322. If you are budgeting from the unmigrated table, you are budgeting for a state your estate passes through once.

Bounds on all of the above#

• The tagging leg was available. With TOFU_LIVE_CLOUDCONTROL=off the sweep falls back to per-type listing across the whole universe, so every figure here is the cheapest production shape rather than the worst one.
• The call counts are emulator-measured. The tagging leg is ceil(tagged_resources / page) and floci’s page is 100, which is why it reads 1, 2 and 4 rather than 1 everywhere. cloudcontrol.Client.GetResources sets no ResourcesPerPage, so the real page size is the Resource Groups Tagging API’s own default and no emulator-backed run can report it.
• One fixture, one composition - the 512-call native leg is a property of the admission table and the ARN join table rather than of the estate, but that is an argument; only this estate was measured, and it declares thirteen types.
• AWS only - nothing here says anything about another provider.
• Every call-count table on this page measures a full-sweep run. None of those tables has been re-measured under the narrowing; what has is the 79-instance fixture’s headline, 157 against 710, and the real-AWS pair at 745 resources. Where a figure here disagrees with a plan you actually ran, the narrowing is the first thing to suspect.

Do not carry one resource type’s slope to another#

This is the mistake most worth avoiding, and it has already been made once in an issue.

live/plan-budget.json (re-measured at 3690d38143, checked every run by TestPlanCallBudgetAgainstFloci) ratchets an aws_s3_bucket estate at 22 calls per instance, fitting calls_total = 22*N + 8 exactly at N=20, 200 and 1000 (448, 4408, 22008). That number is not a property of choudoufu. aws_s3_bucket is an unusually chatty Read: a dozen subresource GETs for ACL, CORS, encryption, lifecycle, logging, object lock, policy, replication, request payment, versioning, website and acceleration, plus the parent-read children beside them.

The generated estate in the tables above measures 1.84 calls per instance migrated, and 1.15 unmigrated. Same tool, same code, twelve and nineteen times below the S3 figure, because the composition is different. An estate of IAM roles, inline policies and DNS records reads cheaply; an estate of S3 buckets does not.

If you want a number for your estate, measure your estate. Extrapolating from somebody else’s resource type will be wrong by whatever the ratio between the two providers’ Read implementations happens to be.

The + 8 in that fit is worth one line of its own, because an earlier version of this page described the fixed term wrongly and the correction is the more useful fact. These are not account-level probes. Six of the eight are ListBuckets: five issued by the parent-read sweep, one by the provider’s own account and region resolution. The remaining two are GetCallerIdentity and GetUser. They are 1.8% of the total at N=20 and 0.04% at N=1000. A fixed term looks expensive on a small estate and disappears on a large one, which is the opposite of how the sweep behaves and a good reason to fit a line rather than divide once.

Emulator wall clock is not on this page#

Every cost figure here is a call count, with one deliberate exception. Seconds measured against the pinned emulator grade the machine the test ran on, which is why live/plan-budget.json records a wall clock and never gates on it. The one timing table above is real AWS, where the seconds are network latency rather than a property of whatever laptop ran the suite. live/FLOCI.md covers the distinction and the three other questions an emulator-backed measurement cannot answer.