Kubernetes Cost Allocation by Team: Showback, Chargeback and Shared Clusters

To allocate Kubernetes costs by team, label every namespace with its owner, turn your bill into a price per CPU-hour and per GiB-hour, charge each namespace for the larger of what it requested and what it used, and split idle and shared costs by a rule agreed before the first report. Then show the numbers for a few months before anyone is charged for them.

The cloud bill will not do this for you. It lists machines, disks and load balancers, while your teams share those machines pod by pod. This post is about who pays for what in a shared cluster. Making the total smaller is a different job, covered in our guide to Kubernetes cost optimization with SRE practices. In practice the two go together, because a team that can see its own number is the team that lowers it.

Why Doesn't the Cloud Bill Show Cost per Team?

Cloud providers bill per resource: this node, that disk, that load balancer, for so many hours. Kubernetes schedules per pod. On a shared node pool, one node runs pods from four teams this hour and from two other teams the next, and the node itself belongs to nobody.

• Cloud tags work for resources a team creates directly. A node in a shared pool cannot carry one team's tag.
• Pods move. Autoscaling adds and removes nodes all day, so "which node is whose" has no stable answer.
• The data you need lives inside the cluster: which namespace asked for how much CPU and memory, and how much it really used, hour by hour.

So allocation is a calculation you run, not a report you download. Everything below is built from upstream pieces: namespaces and labels, kube-state-metrics, the kubelet's container metrics and Prometheus.

Should You Allocate by Namespace or by Label?

Use the namespace as the unit and put the ownership on the namespace as labels. Pod labels are set by whoever writes each deployment and drift over time. Namespace labels are set once, usually by the platform team, and every pod in the namespace inherits the owner.

• Give each team its own namespaces, one per environment: checkout-prod, checkout-staging.
• Put at least two labels on every namespace: team and cost-center. Add env if staging and production are paid from different budgets.
• A namespace shared by several teams is the exception. Split it, or allocate it by a pod label and accept the extra upkeep.

Labels that are optional end up missing. Kubernetes can refuse new namespaces without them, using a built-in ValidatingAdmissionPolicy and no extra controller:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: require-cost-labels
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE"]
        resources: ["namespaces"]
  validations:
    - expression: >-
        has(object.metadata.labels) &&
        'team' in object.metadata.labels &&
        'cost-center' in object.metadata.labels
      message: "New namespaces need the labels team and cost-center."
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: require-cost-labels
spec:
  policyName: require-cost-labels
  validationActions: [Warn, Audit]

The binding starts in Warn and Audit mode: whoever creates a namespace without the labels gets a warning, and the API server audit log records it if audit logging is on. Once the warnings stop, change validationActions to [Deny]. The policy only checks CREATE, so namespaces that already exist keep working while you label them.

One trap catches almost everyone. kube-state-metrics v2 does not export namespace or pod labels unless you list them with --metric-labels-allowlist, for example --metric-labels-allowlist=namespaces=[team,cost-center]. Without the flag, kube_namespace_labels returns nothing and every query that joins on it comes back empty. With it, the labels appear as label_team and label_cost_center, with dashes turned into underscores.

How Do You Turn CPU and Memory into Money?

You need two prices: one per CPU core per hour and one per GiB of memory per hour. No metric gives you these. You work them out from the bill and keep them as inputs.

• Take the monthly cost of the nodes in a pool, after discounts and commitments.
• Decide how much of a node's price is CPU and how much is memory. A common way is to compare a general-purpose and a memory-optimised instance with the same number of cores: the difference is roughly the price of the extra memory.
• Divide each part by the core-hours and GiB-hours the pool provided that month.

Then charge each container for the larger of its request and its usage:

• The request is capacity the scheduler sets aside. Nobody else can use it, so it costs the cluster whether the pod uses it or not.
• Usage above the request is capacity borrowed from the slack on the node. It is not free either.
• A pod with no requests at all looks free in a requests-only model. A LimitRange with default requests in every namespace closes that gap.

In Prometheus this fits in a handful of recording rules. The two prices are constants you replace with your own:

groups:
  - name: cost-allocation
    interval: 5m
    rules:
      # Example prices. Replace them with the numbers from your own bill.
      - record: cost:cpu_core_hour:eur
        expr: vector(0.031)
      - record: cost:memory_gib_hour:eur
        expr: vector(0.0042)

      # Per namespace: the larger of request and usage, for running and pending pods only.
      - record: namespace:cost_cpu_cores:billed
        expr: |
          sum by (namespace) (
            max by (namespace, pod, container) (
                kube_pod_container_resource_requests{resource="cpu", unit="core"}
                  * on (namespace, pod) group_left ()
                    max by (namespace, pod) (kube_pod_status_phase{phase=~"Pending|Running"})
              or
                rate(container_cpu_usage_seconds_total{container!="", container!="POD"}[5m])
            )
          )
      - record: namespace:cost_memory_gib:billed
        expr: |
          sum by (namespace) (
            max by (namespace, pod, container) (
                kube_pod_container_resource_requests{resource="memory", unit="byte"}
                  * on (namespace, pod) group_left ()
                    max by (namespace, pod) (kube_pod_status_phase{phase=~"Pending|Running"})
              or
                container_memory_working_set_bytes{container!="", container!="POD"}
            )
          ) / 2^30

      # Euros per hour, per namespace and per team.
      - record: namespace:cost_eur_per_hour:allocated
        expr: |
          namespace:cost_cpu_cores:billed * scalar(cost:cpu_core_hour:eur)
          + namespace:cost_memory_gib:billed * scalar(cost:memory_gib_hour:eur)
      - record: team:cost_eur_per_hour:allocated
        expr: |
          sum by (label_team) (
            namespace:cost_eur_per_hour:allocated
            * on (namespace) group_left (label_team) kube_namespace_labels
          )

      # Capacity nobody was billed for.
      - record: cluster:cost_cpu_cores:idle
        expr: |
          sum(kube_node_status_allocatable{resource="cpu", unit="core"})
          - sum(namespace:cost_cpu_cores:billed)

A few details in there matter:

• Completed Job pods keep reporting their requests in kube-state-metrics until they are deleted. The kube_pod_status_phase filter counts only pending and running pods.
• container!="" and container!="POD" drop the pod-level totals and the pause container, which would otherwise be counted twice.
• The rules give euros per hour. A monthly figure is the average over the month times the hours in it: avg_over_time(team:cost_eur_per_hour:allocated[30d]) * 24 * 30.

If you would rather not maintain these queries, OpenCost, a CNCF project, implements the same request-or-usage model and adds cloud list prices. The allocation decisions in the rest of this post apply to it just the same.

What Do You Do with Idle Capacity and Shared Services?

After allocation, two piles of cost remain, and how you treat them decides whether teams trust the report.

Idle capacity is node capacity that no namespace requested or used: the gap between what the nodes provide and what was billed. The last rule above measures it for CPU. There are two honest ways to handle it:

• Spread it over the teams in proportion to what each was allocated. Every euro ends up with a team, but idle is hidden inside everyone's number.
• Keep it as its own line, owned by the platform team. Idle then becomes a visible number with an owner whose job is to shrink it, through bin-packing, smaller nodes and autoscaling.

Shared services are what every team uses: the control plane fee, kube-system, ingress controllers, monitoring and logging, and the load balancers in front of shared ingress. Split them in proportion to allocated cost, or by a usage signal where one exists, such as log volume per namespace for the logging stack.

Agree on the method before the first report goes out. Changing it after teams have seen their numbers turns every review into an argument about the method rather than the spend.

A Worked Example: Three Teams on One Cluster

The numbers are round and made up for the example. A cluster costs €12,000 a month:

• €9,000 is nodes.
• €3,000 is shared: the control plane, monitoring and the load balancers in front of ingress.

From the rules above, the month's node cost is allocated like this:

• Checkout: €3,600
• Search: €2,700
• Data: €1,350
• Idle, allocated to nobody: €1,350

Option A spreads idle and shared costs in proportion to allocation (47%, 35% and 18%):

• Checkout pays €3,600 + €2,047 = €5,647
• Search pays €2,700 + €1,535 = €4,235
• Data pays €1,350 + €768 = €2,118

Option B spreads only the shared €3,000 and leaves idle with the platform team:

• Checkout pays €3,600 + €1,412 = €5,012
• Search pays €2,700 + €1,059 = €3,759
• Data pays €1,350 + €529 = €1,879
• The platform team shows €1,350 of idle capacity as its own line

Both add up to €12,000. Under option A, Data pays €238 for idle capacity it did not cause. Under option B, the €1,350 has an owner. Start with option B in the first reports, because it shows where the waste is. Switch to option A later if finance needs every euro on a product budget.

The same report shows something else. If Search requests 20 cores and uses 6 on a typical day, Search pays for 20. That is the conversation that leads to right-sizing requests, and it starts much more easily from a number the team owns. If you want a second pair of eyes on the method before your first report goes out, we run a free Kubernetes cost assessment.

Showback or Chargeback: Which Should Come First?

• Showback: every team gets a report of its costs, and no money moves.
• Chargeback: those costs are charged to the team's budget or cost center.

Start with showback and run it for a few monthly cycles. The first reports always find unlabelled namespaces, workloads in the wrong place and disagreements about shared costs. Fixing those while nothing is at stake is cheaper than fixing them in a budget dispute.

Move to chargeback when:

• every namespace has its labels and the policy is set to deny;
• the unallocated line is small and stays small month after month;
• the method for idle and shared costs is written down and finance has agreed to it;
• teams have stopped questioning the numbers and started asking how to lower them.

A useful monthly report per team holds the total cost and the change from last month, the most expensive namespaces, requests compared with usage, and the share of idle and shared costs it carries.

How Do You Allocate Costs in a Multi-Tenant Cluster?

The same model works when the tenants are customers or business units rather than teams. Each tenant gets its own namespaces with a tenant label, and a few controls keep the numbers honest:

• A ResourceQuota on each tenant's namespaces caps the total it can request, and with it what it can cost.
• A LimitRange gives default requests to containers that declare none, so no tenant runs for free.
• Usage above requests is charged through the request-or-usage rule, so a noisy neighbour pays for the capacity it borrows.
• A tenant with its own node pool, kept apart with taints and tolerations, pays for the whole pool, idle included. It reserved those nodes and nobody else can use them.

How Do You Compare Costs Across Several Clusters?

• Use the same label keys in every cluster. A team label in one cluster and an owner label in another cannot be added up.
• Give each Prometheus an external label with the cluster name and send the recorded series to one central store, so a team's cost can be summed across clusters.
• Keep prices per cluster. Regions, instance types and commitments differ, and one global CPU price would move cost from expensive clusters to cheap ones.
• Allocate each cluster's overhead, such as its control plane, system namespaces and per-node agents, inside that cluster before adding clusters together.
• A cluster dedicated to one team needs no allocation: all of it goes to that team. Report it in the same format anyway, so comparisons stay simple.

What About Spend Outside the Cluster That Nobody Labelled?

Pods are only part of the bill. Several costs are created by Kubernetes but billed outside it:

• Persistent volumes. A disk is billed for as long as it exists, even with no pod using it. Each PersistentVolume points to its claim through claimRef, and so to a namespace. kube_persistentvolumeclaim_resource_requests_storage_bytes gives the requested size per namespace. Many CSI drivers can also write the claim's namespace onto the cloud disk as a tag; check whether yours does.
• Load balancers. Every Service of type LoadBalancer creates one in the cloud, and kube_service_spec_type{type="LoadBalancer"} counts them per namespace.
• Snapshots, network egress and the managed databases a team uses. Tag them in your infrastructure code with the same team and cost-center keys you use on namespaces, so the cloud report and the cluster report join on the same names. Default tags in your Terraform modules make this the rule rather than the exception.

Whatever is still untagged goes into an unallocated line. Publish it with everything else. A visible unallocated line, with someone responsible for shrinking it, gets smaller. A hidden one grows.

Where to Go from Here

Allocation does not cut spend by itself. It shows where to look. The namespaces with the largest gap between requests and usage are where right-sizing pays first, and batch workloads that tolerate interruptions are candidates for spot node pools. The recording rules belong next to the rest of your Prometheus setup, and the labels and the policy fit naturally into a platform engineering golden path.

Frequently Asked Questions

Should teams pay for what they request or for what they use? For the larger of the two. Requests reserve capacity whether it is used or not, and usage above requests borrows capacity from others. Charging only usage rewards over-requesting. Charging only requests makes pods without requests free.

Do we need a commercial tool for this? No. kube-state-metrics, the kubelet's container metrics and Prometheus recording rules are enough for one cluster or several. OpenCost implements the same model if you prefer not to maintain the queries.

How often should the report run? Keep a daily dashboard for the teams, so they see the effect of a change within a day, and a monthly report for showback or chargeback, aligned with the cloud invoice.

What about GPUs? Price them per device-hour and allocate them on requests. A GPU requested by a pod is unavailable to everyone else, used or not, so the request is the fair measure.

Want your first allocation report checked before it reaches finance? We help teams set up the labels, the recording rules and the split for idle and shared costs, starting with a free Kubernetes cost assessment.