Configure the Trustee operator

Now that the Trustee operator is installed, we need to set it up.

Create a route for Trustee

Create a secure route with edge TLS termination for Trustee. External ingress traffic reaches the router pods as HTTPS and passes on to the Trustee pods as HTTP.

Create a route and set the TRUSTEE_HOST variable. This variable will be useful later when we set the peer-pods ConfigMap. In this workshop, we will create an unprotecred http route. This is not for production usage.

Currently, only a route with a valid CA-signed certificate is supported. It is not possible to use a route with self-signed certificate. Therefore, we will create an http unsecure route instead of https and enable insecure_http in the Trustee config to allow such connection. This is not a supported approach for production and just useful for trying everything in a single cluster. 
# Workshop setup
HTTP="http://"
oc apply -f-<<EOF
---
kind: Route
apiVersion: route.openshift.io/v1
metadata:
  namespace: trustee-operator-system
  name: kbs-service
spec:
  to:
    kind: Service
    name: kbs-service
    weight: 100
  port:
      targetPort: kbs-port
  wildcardPolicy: None
---
EOF

# Production setup
# HTTP="https://"
# oc create route edge --service=kbs-service --port kbs-port -n trustee-operator-system

TRUSTEE_ROUTE="$(oc get route -n trustee-operator-system kbs-service \
  -o jsonpath={.spec.host})"

TRUSTEE_HOST=${HTTP}${TRUSTEE_ROUTE}

echo $TRUSTEE_HOST

Example output:

http://kbs-service-trustee-operator-system.apps.rs01nyk5.eastus.aroapp.io

Create the Trustee authentication secret

  1. Create private and public keys.

    mkdir trustee
cd trustee
openssl genpkey -algorithm ed25519 > privateKey
openssl pkey -in privateKey -pubout -out publicKey
cd -

  2. Create a secret with the public key.

    oc create secret generic kbs-auth-public-key --from-file=./trustee/publicKey -n trustee-operator-system

  3. Check that the newly created secret exists.

    oc get secret -n trustee-operator-system

Create Trustee ConfigMap

Create the and apply the Trustee kbs-configmap.yaml ConfigMap.

cat > kbs-configmap.yaml <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: kbs-config-cm
  namespace: trustee-operator-system
data:
  kbs-config.toml: |
    [http_server]
    sockets = ["0.0.0.0:8080"]
    insecure_http = true

    [admin]
    insecure_api = true
    auth_public_key = "/etc/auth-secret/publicKey"

    [attestation_token]
    insecure_key = true
    attestation_token_type = "CoCo"

    [attestation_service]
    type = "coco_as_builtin"
    work_dir = "/opt/confidential-containers/attestation-service"
    policy_engine = "opa"

    [attestation_service.attestation_token_broker]
    type = "Ear"
    policy_dir = "/opt/confidential-containers/attestation-service/policies"

    [attestation_service.attestation_token_config]
    duration_min = 5

    [attestation_service.rvps_config]
    type = "BuiltIn"

    [attestation_service.rvps_config.storage]
    type = "LocalJson"
    file_path = "/opt/confidential-containers/rvps/reference-values/reference-values.json"

    [[plugins]]
    name = "resource"
    type = "LocalFs"
    dir_path = "/opt/confidential-containers/kbs/repository"

    [policy_engine]
    policy_path = "/opt/confidential-containers/opa/policy.rego"
EOF

cat kbs-configmap.yaml
In this example, insicure_http is enabled. This is to allow the test http route set up above. 
oc apply -f kbs-configmap.yaml

The initdata policy

About initdata

This step will be also useful in the OSC operator setup. However, since this policy is part of the attestation and we need the expected PCR to be added in the Trustee reference values, we will create the file now.

The initdata specification provides a flexible way to initialize a CoCo peer pod with sensitive or workload-specific data at runtime, avoiding the need to embed such data in the virtual machine (VM) image. This enhances security by reducing exposure of confidential information and improves flexibility by eliminating custom image builds. For example, initdata can include three configuration settings:

  • An X.509 certificate for secure communication.

  • A cryptographic key for authentication.

  • An optional Kata Agent policy.rego file to enforce runtime behavior when overriding the default Kata Agent policy.

We can apply an initdata configuration by using one of the following methods:

  • Globally by including it in the peer pods config map, setting a cluster-wide default for all pods.

  • For a specific pod when configuring a pod workload object, allowing customization for individual workloads.

    The io.katacontainers.config.runtime.cc_init_data annotation under metadata:annotations: in the pod yaml spec overrides the global INITDATA setting in the peer pods config map for that specific pod. The Kata runtime handles this precedence automatically at pod creation time.

The initdata content configures the following components:

  • Attestation Agent (AA), which verifies the trustworthiness of the peer pod by sending evidence to the Trustee for attestation.

  • Confidential Data Hub (CDH), which manages secrets and secure data access within the peer pod VM.

  • Kata Agent, which enforces runtime policies and manages the lifecycle of the containers inside the pod VM.

Create the initdata policy

In this section, we will create the initdata that will be later set up as global in the OSC operator configmap.

In this policy, we will set the Trustee address in the internal CVM components. We will use TRUSTEE_HOST defined previously when configuring the Trustee.

Why do we add this to the Trustee reference values? Remember that this policy is actually added into the OSC operator Configmap, or injected as pod annotation at deployment time. Both scenarios are happening outside the trusted zone, meaning a rogue admin can simply change these values and connect the CoCo pod to a different Trustee, use insecure http and so on. 
cat > initdata.toml <<EOF
algorithm = "sha256"
version = "0.1.0"

[data]
"aa.toml" = '''
[token_configs]
[token_configs.coco_as]
url = "${TRUSTEE_HOST}"

[token_configs.kbs]
url = "${TRUSTEE_HOST}"
#cert = """
# <cert here>
#"""
'''

"cdh.toml"  = '''
socket = 'unix:///run/confidential-containers/cdh.sock'
credentials = []

[kbc]
name = "cc_kbc"
url = "${TRUSTEE_HOST}"
#kbs_cert = """
# <cert here>
#"""
'''

"policy.rego" = '''
package agent_policy

import future.keywords.in
import future.keywords.if

default AddARPNeighborsRequest := true
default AddSwapRequest := true
default CloseStdinRequest := true
default CopyFileRequest := true
default CreateContainerRequest := true
default CreateSandboxRequest := true
default DestroySandboxRequest := true
default GetMetricsRequest := true
default GetOOMEventRequest := true
default GuestDetailsRequest := true
default ListInterfacesRequest := true
default ListRoutesRequest := true
default MemHotplugByProbeRequest := true
default OnlineCPUMemRequest := true
default PauseContainerRequest := true
default PullImageRequest := true
default RemoveContainerRequest := true
default RemoveStaleVirtiofsShareMountsRequest := true
default ReseedRandomDevRequest := true
default ResumeContainerRequest := true
default SetGuestDateTimeRequest := true
default SetPolicyRequest := true
default SignalProcessRequest := true
default StartContainerRequest := true
default StartTracingRequest := true
default StatsContainerRequest := true
default StopTracingRequest := true
default TtyWinResizeRequest := true
default UpdateContainerRequest := true
default UpdateEphemeralMountsRequest := true
default UpdateInterfaceRequest := true
default UpdateRoutesRequest := true
default WaitProcessRequest := true
default WriteStreamRequest := true

# Enable logs, to see the output of curl
default ReadStreamRequest := true

# Restrict exec
default ExecProcessRequest := false

ExecProcessRequest if {
    input_command = concat(" ", input.process.Args)
    some allowed_command in policy_data.allowed_commands
    input_command == allowed_command
}

# Add allowed commands for exec
policy_data := {
  "allowed_commands": [
        "curl -s http://127.0.0.1:8006/cdh/resource/default/kbsres1/key1"
  ]
}

'''
EOF

cat initdata.toml

Note that if TRUSTEE_HOST uses HTTPS with a CA-signed certificate (necessary for production environments), the certificate has to be added by uncommenting the block in cert under [token_config.kbs] and the one in kbs_cert under [kbc]. Otherwise, like in this demo, we can leave it commented out.

A note on policy.rego

Under policy.rego, you can specify a custom Kata Agent policy. The default policy allows all API calls. For production environments, set ReadStreamRequest and ExecProcessRequest to false to disable the oc exec and oc log APIs, preventing unencrypted data transmission via the control plane. Adjust other true or false values to customize the policy further based on your needs. Note that if ExecProcessRequest is enabled, but ReadStreamRequest is not, the user can still inject commands, but won’t be able to see the output. This does not mean the command won’t be executed.

In this demo, we try something a bit more advanced: in order to securely allow the user to manually perform attestation, we restrict the exec commands to only allow the secret fetching, and nothing else. allowed_commands defines the only commands allowed to be exec’ed into the CoCo pod.

What the above means is that the defined command to curl key1 will work, but any other command will fail. Even oc exec -it pods/your_pod — curl http://127.0.0.1:8006/cdh/resource/default/kbsres1/key2 will not work.

This is extremely useful if the pod has to provide restricted access to an untrusted actor (admin, developer) to for example debug the application logic inside the Confidential Container.

For more information about these policy please check alternative Kata policy.

Measure the policy

Let’s convert the policy in base64 and store it in the INITDATA variable.

INITDATA=$(cat initdata.toml | gzip | base64 -w0)

echo $INITDATA

Now, let’s calculate the expected value of PCR8, which will be given in the reference values to make sure that every CoCo pod actually uses this initdata config.

initial_pcr=0000000000000000000000000000000000000000000000000000000000000000

hash=$(sha256sum initdata.toml | cut -d' ' -f1)

PCR8_HASH=$(echo -n "$initial_pcr$hash" | xxd -r -p | sha256sum | cut -d' ' -f1)

echo $PCR8_HASH

Configure Trustee

You can configure the following values, policies, and secrets for Trustee:

In the sections below, we will elencate how to set up all these options, but for the purpose of the workshop, we will not enforce the image signature verification.

Reference values for the Reference Value Provider Service

Purpose of this resource: In an attestation scenario, the client (CoCo) collects measurements from the running software, the Trusted Execution Environment (TEE) hardware and firmware and it submits a quote with the claims to the Attestation Server (Trustee, what we are setting right now). These measurements must match the trusted digests registered to the Trustee. This process ensures that the confidential VM (CVM) is running the expected software stack and has not been tampered with. By setting reference values, the user effectively defines the trusted digest (expected values) that Trustee expects from a valid client.

You can configure reference values for the Reference Value Provider Service (RVPS) by specifying the trusted digests of your hardware platform.

Red Hat currently ships an official CoCo podVM image together with its measurements. Therefore we simply need to insert the values into rvps-configmap.yaml. Such values ensure that CoCo is running that specific image, with a RH kernel and specific features like initdata policy enabled.

cat > rvps-configmap.yaml <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: rvps-reference-values
  namespace: trustee-operator-system
data:
  reference-values.json: |

    [
     {
        "name": "pcr03",
        "expiration": "2025-12-12T00:00:00Z",
        "hash-value": [
          {
                "alg": "sha256",
                "value": "3d458cfe55cc03ea1f443f1562beec8df51c75e14a9fcf9a7234a13f198e7969"
          }
        ]
     },
     {
        "name": "pcr08",
        "expiration": "2025-12-12T00:00:00Z",
        "hash-value": [
          {
                "alg": "sha256",
                "value": "${PCR8_HASH}"
          }
        ]
     },
     {
        "name": "pcr09",
        "expiration": "2025-12-12T00:00:00Z",
        "hash-value": [
          {
                "alg": "sha256",
                "value": "22e306eac888c8393203858a8b4b7b8f36f3d1434fc4dd044e6b20c6fa43c4d9"
          }
        ]
     },
     {
        "name": "pcr11",
        "expiration": "2025-12-12T00:00:00Z",
        "hash-value": [
          {
                "alg": "sha256",
                "value": "53e58bd6ebb6103c18fd19093cb1bcd0a9235685ad642a6d0981ce8314f5e81d"
          }
        ]
     },
     {
        "name": "pcr12",
        "expiration": "2025-12-12T00:00:00Z",
        "hash-value": [
          {
                "alg": "sha256",
                "value": "267c5142db5118a15e5bd98011bf49bb21e72405ece1d9b1ca7fb27de95ee5b3"
          }
        ]
     }
    ]
EOF

cat rvps-configmap.yaml

Inside reference-values.json field, specify the trusted digests for your hardware platform if required. Otherwise, leave it empty. For the purpose of this workshop, you can leave it empty.

Once the reference values have been added, apply the ConfigMap.

oc apply -f rvps-configmap.yaml

Attestation policy

Purpose of this resource: An attestation policy defines which part of the attestation report sent by the client (CoCo) is important for the Attester (Trustee), and how to compare the report with the reference values.

By default, Trustee has already an attestation policy. You can overwrite the default one by creating your own attestation policy.

The attestation policy follows the Open Policy Agent specification.

The policy below checks the Platform Configuration Register (PCR) values 03, 08, 09, 11, and 12 against the reference values to ensure that the Confidential Containers pod uses the specified restrictive Kata agent policy and that the Red Hat pod VM image has not been altered. The attestation process is successful only if all the values match. For details, see Linux TPM PCR Registry in the UAPI Group Specifications documentation.

cat > attestation-policy.yaml <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: attestation-policy
  namespace: trustee-operator-system
data:
  default.rego: |
    package policy

    import rego.v1
    default executables := 33
    default hardware := 97
    default configuration := 36

    ##### Azure vTPM SNP
    executables := 3 if {
      input.azsnpvtpm.tpm.pcr03 in data.reference.pcr03
      input.azsnpvtpm.tpm.pcr08 in data.reference.pcr08
      input.azsnpvtpm.tpm.pcr09 in data.reference.pcr09
      input.azsnpvtpm.tpm.pcr11 in data.reference.pcr11
      input.azsnpvtpm.tpm.pcr12 in data.reference.pcr12
    }

    hardware := 0 if {
      input.azsnpvtpm
    }

    configuration := 0 if {
      input.azsnpvtpm
    }

    ##### Azure vTPM TDX
    executables := 3 if {
      input.aztdxvtpm.tpm.pcr03 in data.reference.pcr03
      input.aztdxvtpm.tpm.pcr08 in data.reference.pcr08
      input.aztdxvtpm.tpm.pcr09 in data.reference.pcr09
      input.aztdxvtpm.tpm.pcr11 in data.reference.pcr11
      input.aztdxvtpm.tpm.pcr12 in data.reference.pcr12
    }

    hardware := 0 if {
      input.aztdxvtpm
    }

    configuration := 0 if {
      input.aztdxvtpm
    }
EOF

cat attestation-policy.yaml

Once you defined your own policy, apply it.

oc apply -f attestation-policy.yaml

TDX ConfigMap

Purpose of this resource: If your TEE is Intel Trust Domain Extensions (TDX), meaning the instance size you use or plan to use is Standard_DCe*, you must create the following ConfigMap.

Enabling such configmap does not prevent CoCo to use other TEEs.

cat > tdx-config.yaml <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: tdx-config
  namespace: trustee-operator-system
data:
  sgx_default_qcnl.conf: |
    {
      "collateral_service": "https://api.trustedservices.intel.com/sgx/certification/v4/"
    }
EOF

cat tdx-config.yaml

Once tdx-config.yaml is ready, apply the ConfigMap.

oc apply -f tdx-config.yaml

Container image signature verification policy

Purpose of this resource: Sets wether to enforce or not the container image signature verification feature. If enabled, all containers images not signed by the trusted certificate provided in the container image verification secret will not be run.

The Trustee Operator returns this secret to the CoCo CVM components (which will run the CoCo pod) after attestation, to make sure they will perform the intended check. The CVM components will then compare the secret with the actual pod signature to determine whether to run it or not, ensuring that only trusted and authenticated container images are deployed in your environment.

You must in any case create the container image signature verification policy because signature verification is always enabled. If this policy is missing, the pods will not start.

In this workshop, we will use a policy that disables signature verification. In a production environment is of course strongly recommended to enable it. You can use Red Hat Trusted Artifact Signer or other tools to sign container images.

For more information on security policy, see containers-policy.json 5.

  1. Create a security-policy-config.json that enables or disables signature verification.

  2. After security-policy-config.json is created, upload it as a secret with the following command:

oc create secret generic security-policy \
  --from-file=osc=./security-policy-config.json \
  -n trustee-operator-system
Do not alter the secret type, security-policy, or the key, osc.

Note that security-policy will be later used in the KbsConfig

Disable signature verification:

cat > security-policy-config.json <<EOF
{
  "default": [
  {
    "type": "insecureAcceptAnything"
  }],
  "transports": {}
}
EOF

cat security-policy-config.json

Enable signature verification

If you use container image signature verification, you must create a secret that contains the public container image signing key. In other words, if the container is not signed by a trusted signature, it shouldn’t run.

Specify the type $CONTAINER_IMAGE_SIGNATURE_TYPE (for example img-sig), the tag $CONTAINER_IMAGE_SIGNATURE_TAG (for example pub-key), and $CONTAINER_IMAGE_SIGNATURE_PK, the public container image signing key.

CONTAINER_IMAGE_SIGNATURE_TYPE=type
CONTAINER_IMAGE_SIGNATURE_TAG=tag
CONTAINER_IMAGE_SIGNATURE_PK=public_key_file

Create a secret with the following command:

oc create secret generic $CONTAINER_IMAGE_SIGNATURE_TYPE \
    --from-file=$CONTAINER_IMAGE_SIGNATURE_TAG=./$CONTAINER_IMAGE_SIGNATURE_PK \
    -n trustee-operator-system

Note that $CONTAINER_IMAGE_SIGNATURE_TYPE will be later used in the KbsConfig

Then, create security-policy-config.json.

Specify the image repository for $SECURITY_POLICY_TRANSPORT, for example, docker:. For more information, see containers-transports 5.

Specify the container $SECURITY_POLICY_REGISTRY and $SECURITY_POLICY_IMAGE, for example, quay.io and my-image.

Use the previously defined container image signature verification secret tag and type defined as $CONTAINER_IMAGE_SIGNATURE_TYPE and $CONTAINER_IMAGE_SIGNATURE_TAG.

SECURITY_POLICY_TRANSPORT=transport
SECURITY_POLICY_REGISTRY=registry
SECURITY_POLICY_IMAGE=image

Create security-policy-config.json:

cat > security-policy-config.json <<EOF
{
  "default": [
      {
      "type": "insecureAcceptAnything"
      }
  ],
  "transports": {
      "$SECURITY_POLICY_TRANSPORT": {
          "$SECURITY_POLICY_REGISTRY/$SECURITY_POLICY_IMAGE":
          [
              {
                  "type": "sigstoreSigned",
                  "keyPath": "kbs:///default/$CONTAINER_IMAGE_SIGNATURE_TYPE/$CONTAINER_IMAGE_SIGNATURE_TAG"
              }
          ]
      }
  }
}
EOF

cat security-policy-config.json

Resource access policy

Purpose of this resource: Resource policies control which secrets are released and are generally scoped to the workload. They allow the user define which attested workload has access to which resource, to avoid that the wrong client accesses data that it is not supposed to.

In this example below we are creating a simple policy that accepts any request that comes from an attester (client) that does use a TEE. For more information about resource access policies, and how to create stronger ones, look here

cat > resourcepolicy-configmap.yaml <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: resource-policy
  namespace: trustee-operator-system
data:
  policy.rego: |
    package policy
    import rego.v1

    default allow = false
    allow if {
      input["submods"]["cpu"]["ear.status"] == "affirming"
    }
EOF

cat resourcepolicy-configmap.yaml

Once the policy has been implemented, apply the ConfigMap.

oc apply -f resourcepolicy-configmap.yaml

Add a secret to Trustee

Populate Trustee with secret(s) that are then managed by the above policies and if attestation is successful, are sent to the client(s) (CoCo). For example, a Confidential Container image/workload could be encrypted, and the key to decrypt it is stored inside the Trustee and provided only if attestation is successful. In this section, we will show how to add the key into Trustee.

Prerequisites: You have created one or more custom keys. In this workshop, we will also create 2 keys.

Define secret name and values. In this example, the kbsres1 secret has two entries (key1, key2), which the clients retrieve. You can add additional secrets according to your requirements by using the same format.

Create first the key:

openssl rand 128 > key.bin

We will add key1 as a simple string containing the text Confidential_Secret!, and key2 as key.bin.

oc create secret generic kbsres1 \
  --from-literal key1=Confidential_Secret! \
  --from-file key2=key.bin \
  -n trustee-operator-system

Note that kbsres1 will be later used in the KbsConfig

Create the KbsConfig custom resource

To complete Trustee setup, you must create a KbsConfig.

cat > kbsconfig-cr.yaml <<EOF
apiVersion: confidentialcontainers.org/v1alpha1
kind: KbsConfig
metadata:
  labels:
    app.kubernetes.io/name: kbsconfig
    app.kubernetes.io/instance: kbsconfig
    app.kubernetes.io/part-of: trustee-operator
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/created-by: trustee-operator
  name: kbsconfig
  namespace: trustee-operator-system
spec:
  kbsConfigMapName: kbs-config-cm
  kbsAuthSecretName: kbs-auth-public-key
  kbsDeploymentType: AllInOneDeployment
  kbsRvpsRefValuesConfigMapName: rvps-reference-values
  kbsSecretResources: ["kbsres1", "security-policy"]
  kbsResourcePolicyConfigMapName: resource-policy
  kbsAttestationPolicyConfigMapName: attestation-policy
  tdxConfigSpec:
    kbsTdxConfigMapName: tdx-config
  kbsServiceType: NodePort
EOF

cat kbsconfig-cr.yaml

If you did the optional steps:

  • For enforced container image signature verification, extend kbsSecretResources list to also have the value of $CONTAINER_IMAGE_SIGNATURE_TYPE.

  • If you are not planning to use TDX, disable the tdxConfigSpec section.

  • Define kbsServiceType if you created a service type other than the default ClusterIP service to expose applications within the cluster external traffic. You can specify NodePort, LoadBalancer, or ExternalName. In this workshop, we will enable also NodePort.

Once the KbsConfig has been configured, apply it.

oc apply -f kbsconfig-cr.yaml

Verification

Verify the Trustee configuration by checking the Trustee pods and logs.

  1. Check that both pods are up and running:

    oc get pods -n trustee-operator-system

    Expected output:

    NAME                                                   READY   STATUS    RESTARTS   AGE
trustee-deployment-8585f98449-9bbgl                    1/1     Running   0          22m
trustee-operator-controller-manager-5fbd44cd97-55dlh   2/2     Running   0          59m

  2. Check the KBS pod logs

    POD_NAME=$(oc get pods -l app=kbs -o jsonpath='{.items[0].metadata.name}' -n trustee-operator-system)

oc logs -n trustee-operator-system $POD_NAME

    Expected output:

[2024-05-30T13:44:24Z INFO  kbs] Using config file /etc/kbs-config/kbs-config.json
[2024-05-30T13:44:24Z WARN  attestation_service::rvps] No RVPS address provided and will launch a built-in rvps
[2024-05-30T13:44:24Z INFO  attestation_service::token::simple] No Token Signer key in config file, create an ephemeral key and without CA pubkey cert
[2024-05-30T13:44:24Z INFO  api_server] Starting HTTPS server at [0.0.0.0:8080]
[2024-05-30T13:44:24Z INFO  actix_server::builder] starting 4 workers
[2024-05-30T13:44:24Z INFO  actix_server::server] Tokio runtime found; starting in existing Tokio runtime