This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Architecture & Concepts

Key design decisions, personas, templating, and metadata reference for Capsule

1: Admission Policies
2: Architecture
3: Known Metadata
4: Templating

1 - Admission Policies

Recommended Admission Policies to enforce best practices in multi-tenant environments.

As Capsule we try to provide a secure multi-tenant environment out of the box, there are however some additional Admission Policies you should consider to enforce best practices in your cluster. Since Capsule only covers the core multi-tenancy features, such as Namespaces, Resource Quotas, Network Policies, and Container Registries, Classes, you should consider using an additional Admission Controller to enforce best practices on workloads and other resources.

Custom

Create custom Policies and reuse data provided via Tenant Status to enforce your own rules.

Owner Validation

Class Validation

Let’s say we have the following namespaced ObjectBucketClaim resource:

apiVersion: objectbucket.io/v1alpha1
kind: ObjectBucketClaim
metadata:
  name: admission-class
  namespace: solar-production
  finalizers:
    - objectbucket.io/finalizer
  labels:
    bucket-provisioner: openshift-storage.ceph.rook.io-bucket
spec:
  additionalConfig:
    maxSize: 2G
  bucketName: test-some-uid
  generateBucketName: test
  objectBucketName: obc-test-test
  storageClassName: ocs-storagecluster-ceph-rgw

However since we are allowing Tenant Users to create these ObjectBucketClaims we might want to consider validating the storageClassName field to ensure that only allowed StorageClasses are used.

---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-tenant-class
spec:
  validationFailureAction: Enforce
  rules:
  - name: restrict-storage-class
    context:
      - name: classes
        apiCall:
          urlPath: "/apis/capsule.clastix.io/v1beta2/tenants"
          jmesPath: "items[?contains(status.namespaces, '{{ request.namespace }}')].status.classes | [0]"

      - name: storageClass
        variable:
          jmesPath: "request.object.spec.storageClassName || 'NONE'"
    match:
      resources:
        kinds:
        - ObjectBucketClaim
        namespaceSelector:
          matchExpressions:
          - key: capsule.clastix.io/tenant
            operator: Exists
    validate:
      message: "storageclass {{ storageClass }} is not allowed in tenant ({{classes.storage}})"
      deny:
        conditions:
          - key:   "{{classes.storage}}"
            operator: AnyNotIn
            value:  "{{ storageClass }}"

Workloads

Policies to harden workloads running in a multi-tenant environment.

Disallow Scheduling on Control Planes

If a Pods are not scoped to specific nodes, they could be scheduled on control plane nodes. You should disallow this by enforcing that Pods do not use tolerations for control plane nodes.

---
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  name: solar
spec:
  owners:
  - name: alice
    kind: User
  nodeSelector:
    node-role.kubernetes.io/worker: ''

---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: disallow-controlplane-scheduling
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        resources: ["pods"]
        operations: ["CREATE","UPDATE"]
        scope: "Namespaced"
  validations:
    - expression: >
        // deny if any toleration targets control-plane taints
        !has(object.spec.tolerations) ||
        !object.spec.tolerations.exists(t,
          t.key in ['node-role.kubernetes.io/master','node-role.kubernetes.io/control-plane']
        )        
      message: "Pods may not use tolerations which schedule on control-plane nodes."
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: disallow-controlplane-scheduling
spec:
  policyName: disallow-controlplane-scheduling
  validationActions: ["Deny"]
  matchResources:
    namespaceSelector:
      matchExpressions:
        - key: capsule.clastix.io/tenant
          operator: Exists

---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: disallow-controlplane-scheduling
spec:
  validationFailureAction: Enforce
  rules:
  - name: restrict-controlplane-scheduling-master
    match:
      resources:
        kinds:
        - Pod
        namespaceSelector:
          matchExpressions:
          - key: capsule.clastix.io/tenant
            operator: Exists
    validate:
      message: Pods may not use tolerations which schedule on control plane nodes.
      pattern:
        spec:
          =(tolerations):
            - key: "!node-role.kubernetes.io/master"

  - name: restrict-controlplane-scheduling-control-plane
    match:
      resources:
        kinds:
        - Pod
        namespaceSelector:
          matchExpressions:
          - key: capsule.clastix.io/tenant
            operator: Exists
    validate:
      message: Pods may not use tolerations which schedule on control plane nodes.
      pattern:
        spec:
          =(tolerations):
            - key: "!node-role.kubernetes.io/control-plane"

Pod Disruption Budgets

Pod Disruption Budgets (PDBs) are a way to limit the number of concurrent disruptions to your Pods. In multi-tenant environments, it is recommended to enforce the usage of PDBs to ensure that tenants do not accidentally or maliciously block cluster operations.

MaxUnavailable

A PodDisruptionBudget which sets its maxUnavailable value to zero prevents all voluntary evictions including Node drains which may impact maintenance tasks. This policy enforces that if a PodDisruptionBudget specifies the maxUnavailable field it must be greater than zero.

---
# Source: https://kyverno.io/policies/other/pdb-maxunavailable/pdb-maxunavailable/
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: pdb-maxunavailable
  annotations:
    policies.kyverno.io/title: PodDisruptionBudget maxUnavailable Non-Zero
    policies.kyverno.io/category: Other
    kyverno.io/kyverno-version: 1.9.0
    kyverno.io/kubernetes-version: "1.24"
    policies.kyverno.io/subject: PodDisruptionBudget
    policies.kyverno.io/description: >-
      A PodDisruptionBudget which sets its maxUnavailable value to zero prevents
      all voluntary evictions including Node drains which may impact maintenance tasks.
      This policy enforces that if a PodDisruptionBudget specifies the maxUnavailable field
      it must be greater than zero.      
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: pdb-maxunavailable
      match:
        any:
          - resources:
              kinds:
                - PodDisruptionBudget
              namespaceSelector:
                matchExpressions:
                - key: capsule.clastix.io/tenant
                  operator: Exists
      validate:
        message: "The value of maxUnavailable must be greater than zero."
        pattern:
          spec:
            =(maxUnavailable): ">0"

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: pdb-maxunavailable
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
    - apiGroups: ["policy"]
      apiVersions: ["v1"]
      operations: ["CREATE", "UPDATE"]
      resources: ["poddisruptionbudgets"]
    namespaceSelector:
      matchExpressions:
      - key: capsule.clastix.io/tenant
        operator: Exists
  validations:
  - expression: |
      !has(object.spec.maxUnavailable) ||
      string(object.spec.maxUnavailable).contains('%') ||
      object.spec.maxUnavailable > 0      
    message: "The value of maxUnavailable must be greater than zero or a percentage."
    reason: Invalid
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: pdb-maxunavailable-binding
spec:
  policyName: pdb-maxunavailable
  validationActions: ["Deny"]

MinAvailable

When a Pod controller which can run multiple replicas is subject to an active PodDisruptionBudget, if the replicas field has a value equal to the minAvailable value of the PodDisruptionBudget it may prevent voluntary disruptions including Node drains which may impact routine maintenance tasks and disrupt operations. This policy checks incoming Deployments and StatefulSets which have a matching PodDisruptionBudget to ensure these two values do not match.

---
# Source: https://kyverno.io/policies/other/pdb-minavailable/pdb-minavailable/
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: pdb-minavailable-check
  annotations:
    policies.kyverno.io/title: Check PodDisruptionBudget minAvailable
    policies.kyverno.io/category: Other
    kyverno.io/kyverno-version: 1.9.0
    kyverno.io/kubernetes-version: "1.24"
    policies.kyverno.io/subject: PodDisruptionBudget, Deployment, StatefulSet
    policies.kyverno.io/description: >-
      When a Pod controller which can run multiple replicas is subject to an active PodDisruptionBudget,
      if the replicas field has a value equal to the minAvailable value of the PodDisruptionBudget
      it may prevent voluntary disruptions including Node drains which may impact routine maintenance
      tasks and disrupt operations. This policy checks incoming Deployments and StatefulSets which have
      a matching PodDisruptionBudget to ensure these two values do not match.      
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: pdb-minavailable
      match:
        any:
          - resources:
              kinds:
                - Deployment
                - StatefulSet
              namespaceSelector:
                matchExpressions:
                - key: capsule.clastix.io/tenant
                  operator: Exists
      preconditions:
        all:
        - key: "{{`{{ request.operation | 'BACKGROUND' }}`}}"
          operator: AnyIn
          value:
          - CREATE
          - UPDATE
        - key: "{{`{{ request.object.spec.replicas | '1' }}`}}"
          operator: GreaterThan
          value: 0
      context:
        - name: minavailable
          apiCall:
            urlPath: "/apis/policy/v1/namespaces/{{`{{ request.namespace }}`}}/poddisruptionbudgets"
            jmesPath: "items[?label_match(spec.selector.matchLabels, `{{`{{ request.object.spec.template.metadata.labels }}`}}`)] | [0].spec.minAvailable | default(`0`)"
      validate:
        message: >-
          The matching PodDisruptionBudget for this resource has its minAvailable value equal to the replica count
          which is not permitted.          
        deny:
          conditions:
            any:
              - key: "{{`{{ request.object.spec.replicas }}`}}"
                operator: Equals
                value: "{{`{{ minavailable }}`}}"

Deployment Replicas higher than PDB

PodDisruptionBudget resources are useful to ensuring minimum availability is maintained at all times.Introducing a PDB where there are already matching Pod controllers may pose a problem if the author is unaware of the existing replica count. This policy ensures that the minAvailable value is not greater or equal to the replica count of any matching existing Deployment. If other Pod controllers should also be included in this check, additional rules may be added to the policy which match those controllers.

---
# Source: https://kyverno.io/policies/other/deployment-replicas-higher-than-pdb/deployment-replicas-higher-than-pdb/
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: deployment-replicas-higher-than-pdb
  annotations:
    policies.kyverno.io/title: Ensure Deployment Replicas Higher Than PodDisruptionBudget
    policies.kyverno.io/category: Other
    policies.kyverno.io/subject: PodDisruptionBudget, Deployment
    kyverno.io/kyverno-version: 1.11.4
    kyverno.io/kubernetes-version: "1.27"
    policies.kyverno.io/description: >-
      PodDisruptionBudget resources are useful to ensuring minimum availability is maintained at all times.
      Introducing a PDB where there are already matching Pod controllers may pose a problem if the author
      is unaware of the existing replica count. This policy ensures that the minAvailable value is not
      greater or equal to the replica count of any matching existing Deployment. If other Pod controllers
      should also be included in this check, additional rules may be added to the policy which match those
      controllers.      
spec:
  validationFailureAction: Enforce
  background: true
  rules:
  - name: deployment-replicas-greater-minAvailable
    match:
      any:
      - resources:
          kinds:
          - PodDisruptionBudget
          operations:
          - CREATE
          - UPDATE
          namespaceSelector:
            matchExpressions:
            - key: capsule.clastix.io/tenant
              operator: Exists
    context:
    - name: deploymentreplicas
      apiCall:
        jmesPath: items[?label_match(`{{`{{ request.object.spec.selector.matchLabels }}`}}`, spec.template.metadata.labels)] || `[]`
        urlPath: /apis/apps/v1/namespaces/{{`{{request.namespace}}`}}/deployments
    preconditions:
      all:
      - key: '{{`{{ length(deploymentreplicas) }}`}}'
        operator: GreaterThan
        value: 0
      - key: '{{`{{ request.object.spec.minAvailable || "" }}`}}'
        operator: NotEquals
        value: ''
    validate:
      message: >-
        PodDisruption budget minAvailable ({{`{{ request.object.spec.minAvailable }}`}}) cannot be
        greater than or equal to the replica count of any matching existing Deployment.
        There are {{`{{ length(deploymentreplicas) }}`}} Deployments which match this labelSelector
        having {{`{{ deploymentreplicas[*].spec.replicas }}`}} replicas.        
      foreach:
        - list: deploymentreplicas
          deny:
            conditions:
              all:
              - key: "{{`{{ request.object.spec.minAvailable }}`}}"
                operator: GreaterThanOrEquals
                value: "{{`{{ element.spec.replicas }}`}}"

CNPG Cluster

---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: pdb-cnpg-cluster-validation
  annotations:
    policies.kyverno.io/title: Check PodDisruptionBudget minAvailable for cnpgCluster
    policies.kyverno.io/category: Other
    kyverno.io/kyverno-version: 1.9.0
    kyverno.io/kubernetes-version: "1.24"
    policies.kyverno.io/subject: PodDisruptionBudget, Cluster
    policies.kyverno.io/description: >-
      When a Pod controller which can run multiple replicas is subject to an active PodDisruptionBudget,
      if the replicas field has a value equal to the minAvailable value of the PodDisruptionBudget
      it may prevent voluntary disruptions including Node drains which may impact routine maintenance
      tasks and disrupt operations. This policy checks incoming CNPG Clusters and their .spec.enablePDB setting.      
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: pdb-cnpg-cluster-validation
      match:
        any:
          - resources:
              kinds:
                - postgresql.cnpg.io/v1/Cluster
              namespaceSelector:
                matchExpressions:
                - key: capsule.clastix.io/tenant
                  operator: Exists
      preconditions:
        any:
        - key: "{{request.operation || 'BACKGROUND'}}"
          operator: AnyIn
          value:
          - CREATE
          - UPDATE
      validate:
        message: >-
          Set `.spec.enablePDB` to `false` for CNPG Clusters when the number of instances is lower than 2.          
        deny:
          conditions:
            all:
              - key: "{{request.object.spec.enablePDB }}"
                operator: Equals
                value: true
              - key: "{{request.object.spec.instances }}"
                operator: LessThan
                value: 2

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: pdb-cnpg-cluster-validation
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
    - apiGroups: ["postgresql.cnpg.io"]
      apiVersions: ["v1"]
      operations: ["CREATE", "UPDATE"]
      resources: ["clusters"]
    namespaceSelector:
      matchExpressions:
      - key: capsule.clastix.io/tenant
        operator: Exists
  validations:
  - expression: |
      !has(object.spec.enablePDB) ||
      object.spec.enablePDB == false ||
      (has(object.spec.instances) && object.spec.instances >= 2)      
    message: "Set `.spec.enablePDB` to `false` for CNPG Clusters when the number of instances is lower than 2."
    messageExpression: |
      'Set `.spec.enablePDB` to `false` for CNPG Clusters when the number of instances is lower than 2. Current instances: ' +
      string(has(object.spec.instances) ? object.spec.instances : 1)      
    reason: Invalid
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: pdb-cnpg-cluster-validation-binding
spec:
  policyName: pdb-cnpg-cluster-validation
  validationActions: ["Deny"]

Mutate User Namespace

You should enforce the usage of User Namespaces. Most Helm-Charts currently don’t support this out of the box. With Kyverno you can enforce this on Pod level.

---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: tenants-user-namespace
spec:
  rules:
    - name: enforce-no-host-users
      match:
        any:
        - resources:
            kinds:
            - Pod
            namespaceSelector:
              matchExpressions:
              - key: capsule.clastix.io/tenant
                operator: Exists
            # selector:
            #   matchExpressions:
            #     - key: company.com/allow-host-users
            #       operator: NotIn
            #       values:
            #         - "true"
      preconditions:
        all:
        - key: "{{request.operation || 'BACKGROUND'}}"
          operator: AnyIn
          value:
            - CREATE
            - UPDATE
      skipBackgroundRequests: true
      mutate:
        patchStrategicMerge:
          spec:
            hostUsers: false

Note that users still can override this setting by adding the label company.com/allow-host-users=true to their namespace. You can change the label to your needs. This is because NFS does not support user namespaces and you might want to allow this for specific tenants.

Disallow Daemonsets

Tenant’s should not be allowed to create Daemonsets, unless they have dedicated nodes.

---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: deny-daemonset-create
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: ["apps"]
        apiVersions: ["v1"]
        resources: ["daemonsets"]
        operations: ["CREATE"]
        scope: "Namespaced"
  validations:
    - expression: "false"
      message: "Creating DaemonSets is not allowed in this cluster."
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: deny-daemonset-create-binding
spec:
  policyName: deny-daemonset-create
  validationActions: ["Deny"]
  matchResources:
    namespaceSelector:
      matchExpressions:
        - key: capsule.clastix.io/tenant
          operator: Exists

---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: tenant-workload-restrictions
spec:
  validationFailureAction: Enforce
  rules:
  - name: block-daemonset-create
    match:
      any:
      - resources:
          kinds:
          - DaemonSet
          namespaceSelector:
            matchExpressions:
            - key: capsule.clastix.io/tenant
              operator: Exists
    preconditions:
      all:
      - key: "{{ request.operation || 'BACKGROUND' }}"
        operator: Equals
        value: CREATE
    validate:
      message: "Creating DaemonSets is not allowed in this cluster."
      deny:
        conditions:
          any:
          - key: "true"
            operator: Equals
            value: "true"

Enforce EmptDir Requests/Limits

By Defaults emptyDir Volumes do not have any limits. This could lead to a situation, where a tenant fills up the node disk. To avoid this, you can enforce limits on emptyDir volumes. You may also consider restricting the usage of emptyDir with the medium: Memory option, as this could lead to memory exhaustion on the node.

---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: tenant-workload-restrictions
spec:
  rules:
    - name: default-emptydir-sizelimit
      match:
        any:
        - resources:
            kinds:
            - Pod
            namespaceSelector:
              matchExpressions:
              - key: capsule.clastix.io/tenant
                operator: Exists
      mutate:
        foreach:
        - list: "request.object.spec.volumes[]"
          preconditions:
            all:
            - key: "{{element.keys(@)}}"
              operator: AnyIn
              value: emptyDir
            - key: "{{element.emptyDir.sizeLimit || ''}}"
              operator: Equals
              value: ''
          patchesJson6902: |-
            - path: "/spec/volumes/{{elementIndex}}/emptyDir/sizeLimit"
              op: add
              value: 250Mi

Block Ephemeral Containers

Ephemeral containers, enabled by default in Kubernetes 1.23, allow users to use the kubectl debug functionality and attach a temporary container to an existing Pod. This may potentially be used to gain access to unauthorized information executing inside one or more containers in that Pod. This policy blocks the use of ephemeral containers.

---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: block-ephemeral-containers
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      # 1) Regular Pods (ensure spec doesn't carry ephemeralContainers)
      - apiGroups: [""]
        apiVersions: ["v1"]
        resources: ["pods"]
        operations: ["CREATE","UPDATE"]
        scope: "Namespaced"
      # 2) Subresource used by `kubectl debug` to inject ephemeral containers
      - apiGroups: [""]
        apiVersions: ["v1"]
        resources: ["pods/ephemeralcontainers"]
        operations: ["UPDATE","CREATE"]  # UPDATE is typical, CREATE included for future-proofing
        scope: "Namespaced"
  validations:
    # Deny any request that targets the pods/ephemeralcontainers subresource
    - expression: request.subResource != "ephemeralcontainers"
      message: "Ephemeral (debug) containers are not permitted (subresource)."
    # For direct Pod create/update, allow only if the field is absent or empty
    - expression: >
        !has(object.spec.ephemeralContainers) ||
        size(object.spec.ephemeralContainers) == 0        
      message: "Ephemeral (debug) containers are not permitted in Pod specs."
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: block-ephemeral-containers-binding
spec:
  policyName: block-ephemeral-containers
  validationActions: ["Deny"]
  matchResources:
    namespaceSelector:
      matchExpressions:
        - key: capsule.clastix.io/tenant
          operator: Exists

# Source: https://kyverno.io/policies/other/block-ephemeral-containers/block-ephemeral-containers/
---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: block-ephemeral-containers
  annotations:
    policies.kyverno.io/title: Block Ephemeral Containers
    policies.kyverno.io/category: Other
    policies.kyverno.io/severity: medium
    kyverno.io/kyverno-version: 1.6.0
    policies.kyverno.io/minversion: 1.6.0
    kyverno.io/kubernetes-version: "1.23"
    policies.kyverno.io/subject: Pod
    policies.kyverno.io/description: >-
      Ephemeral containers, enabled by default in Kubernetes 1.23, allow users to use the
      `kubectl debug` functionality and attach a temporary container to an existing Pod.
      This may potentially be used to gain access to unauthorized information executing inside
      one or more containers in that Pod. This policy blocks the use of ephemeral containers.      
spec:
  validationFailureAction: Enforce
  background: true
  rules:
  - name: block-ephemeral-containers
    match:
      any:
      - resources:
          kinds:
            - Pod
          namespaceSelector:
            matchExpressions:
            - key: capsule.clastix.io/tenant
              operator: Exists
    validate:
      message: "Ephemeral (debug) containers are not permitted."
      pattern:
        spec:
          X(ephemeralContainers): "null"

QOS Classes

You may consider the upstream policies, depending on your needs:

Certificates

Deny ClusterIssuer in Certificates

Often when working in multi-tenant environments, you want to ensure that tenants are not using ClusterIssuers to issue certificates, but rather use namespaced Issuers within their own namespace. This policy enforces that cert-manager.io/v1/Certificate resources do not reference ClusterIssuers and that the Issuer referenced is in the same namespace as the Certificate.

---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: certificates-only-local-issuer
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: deny-clusterissuer-in-certificates
      match:
        any:
          - resources:
              kinds:
                - cert-manager.io/v1/Certificate
              namespaceSelector:
                matchExpressions:
                - key: capsule.clastix.io/tenant
                  operator: Exists
      validate:
        message: "Certificates must not reference ClusterIssuers; use a namespaced Issuer in the same namespace."
        deny:
          conditions:
            any:
              - key: "{{ request.object.spec.issuerRef.kind || 'Issuer' }}"
                operator: Equals
                value: "ClusterIssuer"

    - name: deny-cross-namespace-issuerref-in-certificates
      match:
        any:
          - resources:
              kinds:
                - cert-manager.io/v1/Certificate
      validate:
        message: "Certificates must reference an Issuer in the same namespace (spec.issuerRef.namespace must be empty or equal to the Certificate namespace)."
        deny:
          conditions:
            any:
              # If issuerRef.namespace is set and differs from the Certificate namespace -> deny
              - key: "{{request.object.spec.issuerRef.namespace || '' }}"
                operator: NotEquals
                value: ""
                # AND also not equal to request namespace
              - key: "{{ request.object.spec.issuerRef.namespace || request.namespace  }}"
                operator: NotEquals
                value: "{{ request.namespace }}"

Deny ClusterIssuer in Gateways

Deny to usage of ClusterIssuers in Gateways by checking for the cert-manager.io/cluster-issuer annotation. This ensures that tenants use namespaced issuer mechanisms instead.

This requires extra permissions to allow Kyverno to read Gateway resources:

admissionController:
  rbac:
    clusterRole:
      extraResources:
       - apiGroups: ["gateway.networking.k8s.io"]
         resources: ["*"]
         verbs: ["get", "list", "watch"]

---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: gateways-deny-cluster-issuer-annotation
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: deny-cert-manager-cluster-issuer-annotation
      match:
        any:
          - resources:
              kinds:
                - gateway.networking.k8s.io/v1/Gateway
              namespaceSelector:
                matchExpressions:
                - key: capsule.clastix.io/tenant
                  operator: Exists
      validate:
        message: "Gateways must not use cert-manager.io/cluster-issuer; use namespaced issuer mechanisms instead."
        deny:
          conditions:
            any:
              - key: "{{ request.object.metadata.annotations.\"cert-manager.io/cluster-issuer\" || '' }}"
                operator: NotEquals
                value: ""

Images

Allowed Registries

---
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  name: solar
spec:
  containerRegistries:
    allowed:
    - "docker.io"
    - "public.ecr.aws"
    - "quay.io"
    - "mcr.microsoft.com"

# Or with a Kyverno Policy. Here the default registry is `docker.io`, when no registry prefix is specified:
---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: restrict-image-registries
  annotations:
    policies.kyverno.io/title: Restrict Image Registries
    policies.kyverno.io/category: Best Practices, EKS Best Practices
    policies.kyverno.io/severity: medium
spec:
  validationFailureAction: Audit
  background: true
  rules:
  - name: validate-registries
    match:
      any:
      - resources:
          kinds:
          - Pod
    validate:
      message: "Using unknown image registry."
      foreach:
      - list: "request.object.spec.initContainers"
        deny:
          conditions:
          - key: '{{images.initContainers."{{element.name}}".registry }}'
            operator: NotIn
            value:
            - "docker.io"
            - "public.ecr.aws"
            - "quay.io"
            - "mcr.microsoft.com"

      - list: "request.object.spec.ephemeralContainers"
        deny:
          conditions:
          - key: '{{images.ephemeralContainers."{{element.name}}".registry }}'
            operator: NotIn
            value:
            - "docker.io"
            - "public.ecr.aws"
            - "quay.io"
            - "mcr.microsoft.com"

      - list: "request.object.spec.containers"
        deny:
          conditions:
          - key: '{{images.containers."{{element.name}}".registry }}'
            operator: NotIn
            value:
            - "docker.io"
            - "public.ecr.aws"
            - "quay.io"
            - "mcr.microsoft.com"

Allowed PullPolicy

Certificate Management

Selective ClusterIssuers

Allow certain ClusterIssuers within Tenants:

---
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: certificates-restrict-clusterissuer-by-tenant-label
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: allow-clusterissuer-only-when-managed-by-matches
      match:
        any:
          - resources:
              kinds:
                - cert-manager.io/v1/Certificate
              namespaceSelector:
                matchExpressions:
                  - key: capsule.clastix.io/tenant
                    operator: Exists
      context:
        - name: certManagedBy
          variable:
            jmesPath: request.object.metadata.labels."capsule.clastix.io/managed-by" || ''
        - name: clusterIssuerManagedBy
          apiCall:
            urlPath: /apis/cert-manager.io/v1/clusterissuers/{{ request.object.spec.issuerRef.name }}
            jmesPath: metadata.labels."company.com/tenant" || ''
            default: ""
      preconditions:
        all:
          - key: "{{ request.object.spec.issuerRef.kind || 'Issuer' }}"
            operator: Equals
            value: ClusterIssuer
          - key: "{{request.operation || 'BACKGROUND'}}"
            operator: AnyIn
            value:
            - CREATE
            - UPDATE
      validate:
        message: >-
          ClusterIssuer is only allowed when the Certificate label
          capsule.clastix.io/managed-by ({{certManagedBy}}) matches the referenced ClusterIssuer label
          company.com/tenant ({{clusterIssuerManagedBy}}).          
        deny:
          conditions:
            any:
              - key: "{{certManagedBy}}"
                operator: Equals
                value: ""
              - key: "{{clusterIssuerManagedBy}}"
                operator: Equals
                value: ""
              - key: "{{certManagedBy}}"
                operator: NotEquals
                value: "{{clusterIssuerManagedBy}}"

2 - Architecture

Architecture references and considerations

Key Decisions

Introducing a new separation of duties can lead to a significant paradigm shift. This has technical implications and may also impact your organizational structure. Therefore, when designing a multi-tenant platform pattern, carefully consider the following aspects. As Cluster Administrator, ask yourself:

🔑 How much ownership can be delegated to Tenant Owners (Platform Users)?

The answer to this question may be influenced by the following aspects:

Are the Cluster Administrators willing to grant permissions to Tenant Owners?
- You might have a problem with know-how and probably your organisation is not yet pushing Kubernetes itself enough as a key strategic platform. The key here is enabling Platform Users through good UX and know-how transfers
Who is responsible for the deployed workloads within the Tenants?
- If Platform Administrators are still handling this, a true “shift left” has not yet been achieved.
Who gets paged during a production outage within a Tenant’s application?
- You’ll need robust monitoring that enables Tenant Owners to clearly understand and manage what’s happening inside their own tenant.
Are your customers technically capable of working directly with the Kubernetes API?
- If not, you may need to build a more user-friendly platform with better UX, for example a multi-tenant ArgoCD setup, or UI layers like Headlamp.

Personas

In Capsule, we introduce a new persona called the Tenant Owner. The goal is to enable Cluster Administrators to delegate tenant management responsibilities to Tenant Owners. Here’s how it works:

Capsule Administrators

Configure Capsule Administrators. The ClusterRoles assigned to Administrators can be configured in the CapsuleConfiguration as well.

They are promoted to Tenant-Owners for all available tenants. Effectively granting them the ability to manage all namespaces within the cluster, across all tenants.

Note: Granting Capsule Administrator rights should be done with caution, as it provides extensive control over the cluster’s multi-tenant environment. When granting Capsule Administrator rights, the entity gets the privileges to create any namespace (also not part of capsule tenants) and the privileges to delete any tenant namespaces.

Capsule Administrators can:

Create and manage namespaces via labels in any tenant.
Create namespaces outside of tenants.
Delete namespaces in any tenant.

Administrators come in handy in bootstrap scenarios or GitOps scenarios where certain users/serviceaccounts need to be able to manage namespaces for all tenants.

Capsule Users

Configure Capsule Users

Any entity which needs to interact with tenants and their namespaces must be defined as a Capsule User. This is where the flexibility of Capsule comes into play. You can define users or groups as Capsule Users, allowing them to create and manage namespaces within any tenant they have access to. If they are not defined as Capsule Users, any interactions will be ignored by Capsule. Often a best practice is to define a single group which identifies all your tenant users. This way you can have one generic group for all your users and then use additional groups to separate responsibilities (e.g. administrators vs normal users).

Only one entry is needed to identify a Capsule User. This is only important for Namespace Admission..

Capsule Users Admission

You can always verify the effective Capsule Users by checking the Configuration Status. As this is variable with Tenant Owners, the status will always show the effective users:

kubectl get capsuleconfiguration default -o jsonpath='{.status.users}' | jq

[
  {
    "kind": "Group",
    "name": "oidc:kubernetes:admin"
  },
  {
    "kind": "Group",
    "name": "projectcapsule.dev"
  },
  {
    "kind": "User",
    "name": "test-user"
  }
]

Tenant Owners

Configure Tenant Owners

Every Tenant Owner must be a Capsule User. By using the TenantOwner CRD this is automatically handeled.

They manage the namespaces within their tenants and perform administrative tasks confined to their tenant boundaries. This delegation allows teams to operate more autonomously while still adhering to organizational policies. Tenant Owners can be used to shift reposnsability of one tenant towards this user group. promoting them to the SPOC of all namespaces within the tenant.

Tenant Owners can:

Create and manage namespaces within their tenant.
Delete namespaces within their tenant.

Capsule provides robust tools to strictly enforce tenant boundaries, ensuring that each tenant operates within its defined limits. This separation of duties promotes both security and efficient resource management.

Layouts

Let’s discuss different Tenant Layouts which could be used . These are just approaches we have seen, however you might also find a combination of these which fits your use-case.

Tenant As A Service

With this approach you essentially just provide your Customers with the Tenant on your cluster. The rest is their responsibility. This concludes to a shared responsibility model. This can be achieved when also the Tenant Owners are responsible for everything they are provisiong within their Tenant’s namespaces.

Resourcepool Dashboard

Scheduling

Workload distribution across your compute infrastructure can be approached in various ways, depending on your specific priorities. Regardless of the use case, it’s essential to preserve maximum flexibility for your platform administrators. This means ensuring that:

Nodes can be drained or deleted at any time.
Cluster updates can be performed at any time.
The number of worker nodes can be scaled up or down as needed.

If your cluster architecture prevents any of these capabilities, or if certain applications block the enforcement of these policies, you should reconsider your approach.

Dedicated

Strong tenant isolation, ensuring that any noisy neighbor effects remain confined within individual tenants (tenant responsibility). This approach may involve higher administrative overhead and costs compared to shared compute. It also provides enhanced security by dedicating nodes to a single customer/application. It is recommended, at a minimum, to separate the cluster’s operator workload from customer workloads.

Dedicated Nodepool

Shared

With this approach you share the nodes amongst all Tenants, therefore giving you more potential for optimizing resources on a node level. It’s a common pattern to separate the controllers needed to power your Distribution (operators) from the actual workload. This ensures smooth operations for the cluster

Overview:

✅ Designed for cost efficiency .
✅ Suitable for applications that typically experience low resource fluctuations and run with multiple replicas.
❌ Not ideal for applications that are not cloud-native ready, as they may adversely affect the operation of other applications or the maintenance of node pools.
❌ Not ideal if strong isolation is required

Shared Nodepool

We provide the concept of ResourcePools or CustomQuotas to manage resources cross namespaces. There’s some further aspects you must think about with shared approaches:

3 - Known Metadata

Labels and Annotations in Capsule Items

Labels

Labels commonly used in Capsule items are listed below. These labels are applied to the corresponding resources by the Capsule controller and can be used for filtering and selection purposes.

`capsule.clastix.io/tenant`

Description	Target Objects	Audience
Established the connection between Object and Tenant. It’s value indicates the owning `Tenant`	* `Namespaces` having a relationship to `Tenants`	`Controller`

`projectcapsule.dev/tenant`

Description	Target Objects	Audience
Established the connection between Object and Tenant. It’s value indicates the owning `Tenant`. Long term replacement for `capsule.clastix.io/tenant` and `capsule.clastix.io/managed-by` labels.	* All namespaced items within a `Tenant` `Namespace` become the corresponding Tenant Label via Mutating Admission.	`Controller`

`capsule.clastix.io/managed-by`

Description	Target Objects	Audience
Established the connection between Object and Tenant. It’s value indicates the owning `Tenant`. Long term replacement for `capsule.clastix.io/tenant`	* All namespaced items within a `Tenant` `Namespace` become the corresponding Tenant Label via Mutating Admission. This label is still added to keep compatibility wiht the Capsule Proxy.	`User`

`projectcapsule.dev/managed-by`

Description	Target Objects
Indicator which controller of capsule or `Custom Resource` is responsible for managing the corresponding Object. Mainly used in Replications to establish that objects are at least managed by one Replications.	* `Any Object being influenced by Replications`	`User`

`projectcapsule.dev/created-by`

`projectcapsule.dev/name`

Description	Target Objects	Audience
Label for tracking internal name or allowing for faster selects. Mainly used to identify relevant rulestatus	* `Tenant Namespaces`	`Controller`

`projectcapsule.dev/cordoned`

Description	Target Objects	Audience
Indicator that a namespace is cordoned (when value equals `true`)	* `Tenant Namespaces`	`User`

`projectcapsule.dev/pool`

Description	Target Objects	Audience
Allocation of Resourcepool via ResourcePoolClaims	* `ResourcePoolClaims`	`User`

4 - Templating

Templating in Capsule Items

Fast Templates

For simple template cases we provide a fast templating engine. With this engine, you can use Go templates syntax to reference Tenant and Namespace fields. There are no operators or anything else supported.

Available fields are:

{{tenant.name}}: The Name of the Tenant
{{namespace}}: The Name of the namespace within the tenant (current context)

Sprout Templating

Our template library is mainly based on the upstream implementation from Sprout. You can find the all available functions here:

https://docs.atom.codes/sprout/registries/list-of-all-registries

We have removed certain functions which could exploit runtime information. Therefor the following functions are not available:

env
expandEnv

Data

You can provide structured data for each Tenant which can be used in templating:

apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
  name: solar
spec:
  data:
    bool: true
    foo: bar
    list:
    - a
    - b
    number: 123
    obj:
      nested: value

Function Library

Custom Functions we provide in our template package.

deterministicUUID

deterministicUUID generates a deterministic, RFC-4122–compliant UUID (version 5 + RFC4122 variant) from a set of input strings. It is designed for use in templates where you need stable, repeatable IDs derived from meaningful inputs (e.g. cluster name, tenant, role name), instead of random UUIDs.

This is especially useful for:

Crossplane / IaC resources that must not change IDs across reconciles

The function takes any number of strings and turns them into a UUID in a fully deterministic way.

What that means in practice:

If you call it twice with the same values, you get the same UUID
If any input changes, the UUID changes too
There is no randomness involved
The output is always a valid UUID

So from the outside, it behaves just like a normal UUID, just deterministic.

deterministicUUID(parts ...string) string

Example usage:

{{ deterministicUUID "cluster-a" "app-123" "tenant-x" "some-role" }}

generateAgeKey

generateAgeKey generates a new age X25519 key pair for use with age. It returns both the private identity and the public recipient key.

This is useful in templates where a resource needs to create an age-compatible encryption identity, for example when generating Kubernetes Secrets that are later used for encrypting or decrypting data.

This is especially useful for:

Bootstrap secrets that need an age identity
Generating encryption keys during initial provisioning
Creating age recipient keys for systems that need to encrypt data for a generated identity

The function does not return a plain string. It returns an object with two fields:

Identity: the private age identity, e.g. AGE-SECRET-KEY-1...
Recipient: the public age recipient, e.g. age1...

What that means in practice:

Each call generates a new key pair
The identity is the private key and must be treated as secret
The recipient is the public key and can be shared with systems that need to encrypt data
The output is not deterministic
Calling this function during every reconcile may rotate the generated key unless the result is persisted

generateAgeKey() any

Example usage:

{{ $key := generateAgeKey }}
apiVersion: v1
kind: Secret
metadata:
  name: age-key
type: Opaque
stringData:
  identity: {{ $key.Identity | quote }}
  recipient: {{ $key.Recipient | quote }}

Output:

apiVersion: v1
kind: Secret
metadata:
  name: age-key
type: Opaque
stringData:
  identity: "AGE-SECRET-KEY-1..."
  recipient: "age1..."

Because this function creates a new random key pair on every call, it should usually only be used when the generated Secret is created once and then reused. For continuously reconciled resources, prefer generating the key in controller logic and persisting it before using it in templates.

generateAgePQKey

generateAgePQKey generates a new age post-quantum hybrid key pair for use with age. It returns both the private identity and the public recipient key.

This is useful in templates where a resource needs to create an age-compatible encryption identity using the newer hybrid recipient format.

This is especially useful for:

Bootstrap secrets that should use age hybrid keys
Generating encryption keys during initial provisioning
Creating public recipient keys for systems that need to encrypt data for a generated hybrid identity
Future-facing age encryption setups where hybrid keys are preferred

The function does not return a plain string. It returns an object with two fields:

Identity: the private age hybrid identity, e.g. AGE-SECRET-KEY-PQ-1...
Recipient: the public age recipient, e.g. age1...

What that means in practice:

Each call generates a new key pair
The identity is the private key and must be treated as secret
The recipient is the public key and can be shared with systems that need to encrypt data
The output is not deterministic
Calling this function during every reconcile may rotate the generated key unless the result is persisted

generateAgePQKey() any

Example Usage:

{{ $key := generateAgePQKey }}
apiVersion: v1
kind: Secret
metadata:
  name: age-pq-key
type: Opaque
stringData:
  identity: {{ $key.Identity | quote }}
  recipient: {{ $key.Recipient | quote }}

Output:

apiVersion: v1
kind: Secret
metadata:
  name: age-pq-key
type: Opaque
stringData:
  identity: "AGE-SECRET-KEY-PQ-1..."
  recipient: "age1..."