Install the Trustee operator

In confidential computing environments, attestation is crucial in verifying the trustworthiness of the location where you plan to run your workload or where you plan to send confidential information. Before actually running the workload or transmitting the confidential information, you need to perform attestation.

The Trustee project (previously known as CoCo KBS) includes components deployed on a trusted side and used to verify whether the remote workload is running in a trusted execution environment (TEE). It also verifies that the remote environment uses the expected software and hardware versions.

For an overview of attestation, read our Learn about Confidential Computing Attestation blog series.

For details on attestation flow in Confidential Containers, read our Understanding the Confidential Containers Attestation Flow article.

To summarize, the workflow of attestation for CoCo is the following: a Confidential Container runs in a confidential virtual machine supported by TEEs (which provide memory encryption), but this is not enough to guarantee full confidentiality of data in use. Attestation ensures that the hardware and software running the Confidential Container are not tampered with, and are just "pretending" to do their work. Additionally, in the Trustee workflow, a remote attester doesn’t just check if a Confidential Container is running in a safe environment: it also provides "secrets" back to the container only if the attestation has been successful.

What is a "secret"? Suppose the container image or the workload (AI model, or a generic binary running in the container) is encrypted. This means that at development phase, in a secure environment, some/all components of the container are encrypted, but the key to decrypt is not embedded inside the container. Therefore running such containers without the key will not even start the application, since all files are encrypted blobs.

The key is instead provided to a remote attester (Trustee operator), which holds it as an Openshift Secret. This secret is sent back to the CoCo pod asking for it only if attestation has been successful.

In other words, the workflow is the following:

  1. A Confidential Container is developed in a secure environment. In addition to the traditional development steps, two additional actions are necessary:

    1. The binary/pod image is encrypted with a secure key, which is not embedded inside the container.

    2. The container logic is extended to initially perform a curl http://127.0.0.1:8006/cdh/resource/<path to the Trustee secret> to fetch the secret from Trustee, assuming attestation was successful, and decrypt the application/container.

  2. The Trustee operator is installed and configured in a secure environment, and the secure key used to encrypt the pod is loaded into it.

  3. In the untrusted cluster, OSC is installed and configured to use CoCo and connect with Trustee.

  4. The developed CoCo pod is started. The pod starts in a Confidential VM, which then starts the actual containerized application.

  5. The application performs the curl call to ask for the secret

  6. The Trustee agent running inside the Confidential VM takes care of getting evidence from TEE, OS and various software, prepare an attestation report and send it to the Trustee together with the request for that secret.

  7. Trustee analyzes the report, and if it is matching the expected values, it means the CoCo pod is running in a secure environment. If the policy to provide that specific secret to that specific pod passes, sends back the requested key to the pod.

  8. The pod gets the key, unlocks the binary/container image and starts running as usual.

In this workshop, we will install Trustee in the same cluster that hosts OSC and therefore Confidential Containers. This is not a suitable solution for production environments. Because the Trustee operator contains the refernce values and secrets that are necessary to enstablish the if a Confidential Container is running in a truly safe environment, it should be installed in a separate cluster running in a trusted environment (on prem, for example). Because of the limitations of this ARO workshop, it is not possible to set up two clusters.

Let’s start with installing Trustee and setting it up. In CoCo, connecting Trustee with OSC is mandatory because as we will see later, before even booting a container OSC asks Trustee to verify the container image signature (even if the signature verification is in permissive mode in Trustee). Therefore it is imperative that Trustee is up and running before OSC is fully configured. We will then install OSC and test our first Confidential Container!

There are two ways to install the Trustee operator:

Using the web interface

  1. In the Openshift Container Platform web console, navigate to Operators > OperatorHub.

  2. In the Filter by keyword field, type trustee.

  3. Click on the confidential compute attestation tile.

    01 trustee operatorhub ui

  4. On the Install Operator page, select stable from the list of available Update Channel options.

    02 trustee install ui

  5. Verify that Operator recommended Namespace is selected for Installed Namespace. This installs the Operator in the mandatory trustee-operator-system namespace. If this namespace does not yet exist, it is automatically created.

Attempting to install the Trustee Operator in a namespace other than trustee-operator-system causes the installation to fail.

  1. Verify that Automatic is selected for Update approval. Automatic is the default value, and enables automatic updates to Trustee when a new z-stream release is available.

  2. Click Install.

The Trustee Operator is now installed on your cluster.

Verification

  1. Navigate to Operators > Installed Operators.

  2. Verify that the Trustee Operator is displayed.

    03 trustee install ui

Using the command line

If you don’t want to copy-paste, there is a script that performs all the steps above automatically.

  1. Create and apply a Namespace.yaml manifest file:

    cat > Namespace.yaml <<EOF
apiVersion: v1
kind: Namespace
metadata:
  name: trustee-operator-system
EOF
    oc apply -f Namespace.yaml

  2. Create and apply a OperatorGroup.yaml manifest file:

    cat > OperatorGroup.yaml <<EOF
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: trustee-operator-group
  namespace: trustee-operator-system
spec:
  targetNamespaces:
  - trustee-operator-system
EOF
    oc apply -f OperatorGroup.yaml

  3. Create and apply a Subscription.yaml manifest file:

    cat > Subscription.yaml <<EOF
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: trustee-operator
  namespace: trustee-operator-system
spec:
  channel: stable
  installPlanApproval: Automatic
  name: trustee-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
EOF
    You can change the operator version by changing the startingCSV field in the Subscription.yaml file. 
    oc apply -f Subscription.yaml

Verification

Ensure that the Operator is correctly installed by running the following command and checking that the PHASE is in Succeeded state:

oc get csv -n trustee-operator-system

Example output

NAME                      DISPLAY                            VERSION   REPLACES   PHASE
trustee-operator.v0.4.1   confidential compute attestation   0.4.1                Succeeded

Scripted way

Avoid copy-pasting all steps above and do it in one step: NOTE: You can change the operator version by changing the startingCSV field under the Subscription resource type.

oc apply -f-<<EOF
---
apiVersion: v1
kind: Namespace
metadata:
  name: trustee-operator-system
---
apiVersion: operators.coreos.com/v1
kind: OperatorGroup
metadata:
  name: trustee-operator-group
  namespace: trustee-operator-system
spec:
  targetNamespaces:
  - trustee-operator-system
---
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: trustee-operator
  namespace: trustee-operator-system
spec:
  channel: stable
  installPlanApproval: Automatic
  name: trustee-operator
  source: redhat-operators
  sourceNamespace: openshift-marketplace
EOF

Verification

Ensure that the Operator is correctly installed by running the following command and checking that the PHASE is in Succeeded state:

watch oc get csv -n trustee-operator-system