How Flux is layered¶

Nothing reaches this cluster except by being merged to master. There is no kubectl apply step, and no way to install something by hand that survives the next reconcile. What that costs is a structure that has to encode ordering, blast radius and failure isolation in configuration, because there is no operator present to sequence anything.

That structure is three layers deep, and the shape is not arbitrary.

The three layers¶

k8s/bootstrap/        applied once by hand: the GitRepository and the umbrellas
  ├── crds            the kinds every layer above it needs
  ├── namespaces      every namespace, and each app's Kustomization beside it
  └── platform-<domain>  the cluster's machinery, one Kustomization per domain,
                        each in the namespace it reconciles

The workloads are not a fourth umbrella. Each app's Kustomization is created by namespaces, in the app's own namespace — see below for what that costs and what it buys.

The current membership of each layer, with every interval, prune and wait setting, is in Flux Kustomizations.

k8s/bootstrap/ is the seed: a GitRepository pointing at this repo, and the umbrella Kustomizations. It is the only thing ever applied manually, and it exists solely so that everything after it can be applied by Flux — which also means a change to a file in it does nothing until someone runs kubectl apply -k k8s/bootstrap.

One of its umbrellas, namespaces, creates the namespace every app is deployed into, from k8s/namespaces/. They sit here rather than with the app for two reasons: a Namespace is cluster-scoped, so an app reconciling under a namespace-scoped ServiceAccount could not apply its own; and a namespace's group.rbac.thaum.xyz/<group> labels are an access grant, which must not live in a directory its own tenant can change.

The platform-<domain> namespaces sit there too, for a third reason: each is shared by several components, so keeping it in any one of their directories makes that component's prune the whole domain's blast radius.

An app's directory there holds two files, or three: the Namespace, the sync.yaml that is its Kustomization, and a ClusterRoleBinding where the app applies something cluster-scoped. The other two prerequisites are not files. The flux.rbac.thaum.xyz/role label on the Namespace makes Kyverno generate the flux-reconciler identity the Kustomization applies as and the GitRepository it reads, both from generate-flux-reconciler: a Kustomization can create neither, and both are the same for every app, so a file per namespace would only be a place for them to drift. Three properties of the directory are kept on purpose. There is no kustomization.yaml, so Flux applies every manifest it finds and a namespace cannot be silently skipped for want of a resources: entry. Every namespaced object states its own namespace, because nothing transforms it. And every Namespace carries kustomize.toolkit.fluxcd.io/prune: disabled, because pruning is inventory-based and a namespace dropped from the inventory would be deleted with everything in it. What each label on a Namespace means is in annotations and labels.

platform-cluster is the one whose directory there is also built by hand. It holds the umbrellas, so it cannot be created by namespaces — that is one of them — and k8s/bootstrap/ applies the same directory before any Kustomization exists. Every Namespace is in k8s/namespaces/, including that one; there is no longer a flux-system, and the controllers run in platform-cluster like any other component.

Each platform-<domain> is a directory rather than a file, because the Namespace is not the only thing that has to be there first. A Kustomization reconciling in that namespace reads a GitRepository in it — --no-cross-namespace-refs allows nothing else — and applies as the flux-reconciler ServiceAccount there, which --default-service-account resolves in the Kustomization's own namespace. Neither can be created by the Kustomization that needs them, so both are prerequisites of the domain rather than members of it.

That ServiceAccount and that GitRepository are files, where an app's are generated by generate-flux-reconciler from a label on its Namespace. The difference is the bootstrap path: the Kustomization that installs Kyverno impersonates platform-security's flux-reconciler and reads its GitRepository, and the one that installs the CNI reads platform-network's. An object produced by the policy engine cannot be one Flux needs in order to install that engine — on a rebuild nothing would ever reconcile. So the platform namespaces carry no flux.rbac.thaum.xyz/role label, and the files are not something to tidy away.

platform is what a workload assumes is already there — CNI, storage drivers, ingress controllers, cert-manager, admission control, the observability collectors. Breaking something here breaks things that do not mention it.

The apps are the workloads. An app can fail entirely without taking anything else with it, which is the property the split exists to preserve — and now that each reconciles under its own namespace's identity, one cannot reach another's objects even deliberately.

Why the Kustomizations live apart from the manifests¶

An app appears in two places, and the split is by who owns each. k8s/namespaces/<app>/ is platform-owned: the Namespace, the sync.yaml that is that app's Kustomization, and a ClusterRoleBinding where the app needs one. k8s/apps/<app>/ is what sync.yaml reconciles, and belongs to whoever owns the app.

That line is the review boundary rather than a convention, because it is also the privilege boundary. A Kustomization reconciles as the flux-reconciler in its own namespace, so putting the object in the app's namespace is what confines it there — and putting the object itself somewhere the app's owner cannot edit is what stops them widening it.

It also separates how a component is reconciled — its interval, its dependencies, whether it prunes — from what the component is. Those change for different reasons.

There is no apps umbrella. Each app's Kustomization is created by namespaces, alongside the namespace it runs in. That costs an ordering edge: apps used to dependsOn platform, and --no-cross-namespace-refs=true forbids a Kustomization in an app namespace naming one in platform-cluster. On a cold start an app therefore fails until the platform component it needs exists, and retries. That is the price of the confinement the same flag buys.

The platform is arranged the other way round, and the difference is deliberate. A platform component has no Kustomization of its own; it is a directory named in its domain's k8s/platform/<domain>/kustomization.yaml, and one Kustomization reconciles the whole domain. The unit an app needs is isolation — one app failing must not reach another. The unit the platform needs is a namespace: that is what the reconciler identity, the group RoleBindings and a prune's blast radius are all keyed to, and one Kustomization per namespace is what makes those the same question rather than three that can drift apart.

Each domain Kustomization is in the namespace it reconciles, reading the GitRepository there and applying as that namespace's flux-reconciler. The umbrellas reconcile from platform-cluster, which makes that namespace the seed the rest of the cluster is built from — and where the controllers themselves run, installed by the flux component like any other release.

Ordering, where it genuinely matters¶

Almost nothing declares dependsOn — the reference page lists what does. Everything else is order-independent by construction; the exceptions are all cases where an object cannot be accepted by the API server until something else exists:

Three of those are between HelmReleases rather than Kustomizations because the domains cannot depend on each other: --no-cross-namespace-refs lets a Flux object name a dependsOn target only in its own namespace, and each domain is in a different one.

The same rule is why only platform-cluster waits for crds. It is the one domain sharing a namespace with that layer, so it is the one that can — the edge is an accident of where the object lives rather than a statement that its kinds matter more. Read it as an asymmetry to leave alone, not a pattern to spread; it also buys less than it looks, because the NodeFeatureRules in that same domain need CRDs from a chart the domain installs itself.

For the other four, ordering against the kinds they need is not declared anywhere, because it cannot be. On a cold start an object whose kind has not been established fails its dry-run, and so does the pass it is part of: Flux applies a Kustomization in stages and abandons a stage on its first error rather than collecting them per object, so the whole domain waits for the retry, which runs on retryInterval and defaults to interval.

The kyverno policies were the sharpest case of that, and are the reason kyverno's CRDs are not installed by the kyverno chart. csi-nfs, piraeus-datastore and cnpg-system ship policies.kyverno.io objects, and so does platform-security. k8s/crds/kyverno establishes those kinds through kyverno-policy-crds, a chart in thaum-xyz/helm-charts whose only content is upstream's separate kyverno-api chart as a dependency -- the same chart the kyverno chart pulls in, so the CRDs are the ones the controllers expect.

The wrapper is there because kyverno-api cannot be a release by itself. Helm stores a release's own templates and its rendered manifest in one Secret; for that chart both are the same eleven CRDs, and gzipped together they exceed what a Secret may hold. A dependency's templates are not stored, so behind the wrapper the CRDs are kept once, as they are inside the kyverno release.

Four of the five domains still cannot declare that they wait for it, so this buys convergence rather than ordering: what a cold start retries against is a CRD-only release with no dependencies of its own, instead of the whole kyverno stack with its controllers and webhooks.

The kyverno chart gates that subchart and its own on a single crds.install, with no way to disable half, so the kyverno release drops those eleven CRDs from its manifest with a $patch: delete post-renderer instead. Everything outside policies.kyverno.io it still installs and upgrades itself: those kinds are used by no other domain, so nothing needs them earlier, and leaving them with the chart keeps them in lockstep with the controllers that serve them without a vendored copy to go stale.

crds is declared in k8s/bootstrap/ and applies k8s/crds/, rather than being a component of one domain, because more than one domain uses what it ships and no domain can be ordered after another. Its manifests sit outside k8s/platform/ for a second reason: everything under k8s/platform/<domain>/ belongs to that domain's Kustomization, and a directory there owned by another layer is a rule with an exception.

crds is the only Kustomization with wait: true: a dependency that is merely applied is not yet usable, and a CRD has to be established before an object of that kind will be accepted.

Pruning, and the exceptions¶

prune: true is the default here — nearly every Kustomization has it, which is what makes deleting a directory delete the objects, and what makes the tutorial's cleanup step work. The ones that opt out are listed on the reference page, and they divide into two kinds.

The layers (crds, namespaces) do not prune because a transient failure to render one of them would otherwise be read as "these components are gone" and cascade into deleting everything in the layer. For namespaces the stake is higher still: deleting a Namespace takes everything inside it, so removing one is deliberately two acts — drop the file, then delete the object.

Everything else prunes, including all five platform domains — and including Flux's own manifests, which platform-cluster reconciles like any other component. What makes that survivable is not a prune: false but the two mechanisms below, which is the same trade one level down: the guard sits on the object that carries the risk rather than on a switch covering everything near it. What used to be a component-wide prune: false on the handful whose loss is unrecoverable — cluster networking, the drivers behind every volume, the ingress path to everything — is now kustomize.toolkit.fluxcd.io/prune: disabled on the individual objects that carry the risk. Those are the HelmReleases, because pruning one uninstalls the release behind it, and each says what its own loss would cost.

That is narrower in both directions. A component-wide switch also protected the HelmRepository and values ConfigMap sitting beside the release — the orphans that had to be cleared out by hand after the traefik and topolvm namespace moves — and it protected them anonymously, leaving the next reader to infer from what the component is why it was exempt.

Those values ConfigMaps keep stable names, which is what lets a valuesFrom entry reference one directly. What makes a change to one reach its release is a label rather than a new name. See why Helm values live in a file.

What reconciliation actually costs¶

Every namespace has its own GitRepository, each polling the same repository every 60 seconds, so a merge lands within a minute. That is the cost of --no-cross-namespace-refs: a Kustomization may read only a source beside it. There is no webhook either, for the same reason — a Receiver may only name resources in its own namespace, so one of them could refresh one namespace and no other.

The trap is that reconciling in the wrong order reports success while doing nothing. flux reconcile kustomization <name> acts on whatever revision the source currently holds — which, minutes after a merge, may still be the one before it. The source has to be refreshed first, and it is the source in that Kustomization's own namespace; the CLI defaults to a flux-system that no longer exists, so every command takes -n:

flux -n <app> reconcile source git ankhmorpork               # an app, in its own namespace
flux -n <app> reconcile kustomization <app>

flux -n platform-<domain> reconcile source git ankhmorpork   # a platform domain
flux -n platform-<domain> reconcile kustomization platform-<domain>

flux -n platform-cluster reconcile source git ankhmorpork    # a layer: crds or namespaces
flux -n platform-cluster reconcile kustomization crds

--with-source does both in one command, and is the shorter way to say it:

flux -n <namespace> reconcile kustomization <name> --with-source

There is no third step for a component whose values come from a configMapGenerator. There used to be: the ConfigMap's contents changed while its name did not, so the HelmRelease spec never moved and helm-controller had no event. Those ConfigMaps now carry reconcile.fluxcd.io/watch: Enabled and helm-controller reconciles on the write; see why Helm values live in a file.

What still reconciles green without your change in it¶

Two cases survive, and in neither will the status show anything wrong.

An app's Kustomization is applied by namespaces, not by itself. A change to sync.yaml — its interval, its path, its dependsOn — is picked up when that layer next runs, so reconciling the app reports success against the spec it already had. Reconcile namespaces first.

A change to k8s/bootstrap/ is not reconciled at all. Nothing watches that directory; it takes effect when someone runs kubectl apply -k k8s/bootstrap, and until then every umbrella keeps reconciling happily on the spec it was created with.

The same shape catches the objects Kyverno generates — a namespace's flux-reconciler, its RoleBinding, its GitRepository. They are in no Kustomization's inventory, so an edit to one is reverted rather than applied, and generate-flux-reconciler recreates it from the Namespace's label. Edit the label, or the policy; see annotations and labels.