Understanding Confidential Containers - Workshop on ARO

Red Hat Openshift sandboxed containers provides the capability to run Confidential Containers (CoCo). Confidential Containers are containers deployed within an isolated hardware enclave protecting data and code from privileged users such as cloud or cluster administrators. The CNCF Confidential Containers project is the foundation for the OpenShift CoCo solution. Note that CoCo is an additional feature provided by OpenShift sandboxed containers, and consequently, it’s available through the OpenShift sandboxed containers operator.

Please refer to the Exploring the OpenShift Confidential Containers solution blogpost for more information.

In this workshop, we are also showing another operator, the confidential compute attestation operator (also known as Trustee), which can verify the trustworthiness of TEEs remotely. For more information, please refer to this blogpost.

We will show how to set up the Trustee and OSC operator and run a simple hello-openshift Confidential Container running with the kata-remote runtime class (peer pods solution). This effectively means that the hello-openshift container runs in a separate, confidential, independent virtual machine, and not in the worker node. In another example, we will also show how attestation and secure key retrieval workflow happens between a CoCo pod and Trustee.

The goal of this workshop is to provide the user not only an environment and documentation to test CoCo, but also provide additional explanations on the design choices behind some options and the benefit they bring to the overall user experience. CoCo is designed to bring confidential computing at kubernetes level, making it as simple as possible while preserving all security benefits that confidential computing brings.

Before starting this workshop, we suggest to read the following blogposts to get a background on CoCo:

All blogposts are under this blog series.

Structure of the workshop

This workshop demystifies the CoCo threat model and architecture. We will demonstrate the different roles and knowledge requirements of various personas in a typical Kubernetes environment.

We will show performed by 3 different personas:

  • The operational security (OpSec) persona, which is responsible of the overall security and ensuring that CoCo is actually working and protecting the workload. This persona knows well what CoCo is, studied the whole CoCo theory chapter of this workshop, and knows how to install and configure it (especially Trustee), and has access to Trustee to insert secrets and configure the policies.

  • The cluster admin persona, in charge of maintaining and monitoring the cluster. In the CoCo threat model, the cluster admin is not trusted. This person can potentially see what a traditional pod is doing, read secrets inside the cluster, and manipulate settings and container images.

  • The application developer persona, which has very to no little knowledge of CoCo. He/she doesn’t need to know about the CoCo architecture, what attestation is, and doesn’t need to do any modification in the application to run it with CoCo. In some environment, this persona could very well be completely oblivious to CoCo, if operations like the secret management is handled by the operational security persona. In others, it has to have a very minimal knowledge, for example know that secrets are stored and managed somewhere else (Trustee).

The workshop will show the steps performed by the above users in two phases: setup and deployment.

  1. Setup phase 1: install and configure the trusted cluster (Trustee and secrets)

    1. The operational security persona, responsible of the cluster(s) security, installs and configures Trustee in a trusted and secure cluster, by defining all security policies.

    2. The application developer adds the application-specific secrets to Trustee, and creates the necessary sealed secres (more on this later). As mentioned before, this operation can also be performed by the operational security persona. In this workshop, we will delegate this to the developer, as always relying on the operational security persona could cause bottlenecks for developers.

  2. Setup phase 2: install and configure the untrusted cluster (where CoCo runs)

    1. The cluster admin installs and configures OSC in the untrusted cluster, where the CoCo application will be first checked by Trustee and its policies, and then run.

  3. Deployment phase: the application developer launches various applications

    1. If they behave as expected, and secrets are available, it means the application is running securely despite not trusting the cluster and admin where it runs.

    2. If they don’t, it means something has been compromised or forbidden by security policies. The developer needs the operational security responsible helo to understand what is happening and how to solve the issue.

If you are mostly interested into the developer role and/or want to skip some persona step, at the beginning of each section of the setup phases we will provide a script that automates the steps, so you can also only focus on the deployment. What is important is to not skip any section, but rather execute the scripts to speedup the flow when available.

In this workshop, we will install Trustee in the same cluster that hosts OSC and therefore Confidential Containers. This is not a suitable solution for production environments. Because the Trustee operator contains the reference values and secrets that are necessary to establish if a Confidential Container is running in a truly safe environment, it should be installed in a separate cluster running in a trusted environment (on prem, for example). Because of the limitations of this ARO workshop, it is not possible to set up two clusters. You can assume that every steps done in the setup phase 1, and in general everything running in the trustee-operator-system namespace should run in a trusted, separate environment.

Enable tmux

Since many steps create and require environment variables, it is higly recommended to run the whole workshop on tmux.

If you are running this workshop for the first time, run

tmux

To enter the console. If you are entering again, first list all the existing sessions and ensure there is the one you previously opened:

tmux ls

And then access it:

tmux a