Containers Are Great – But What Are The Risks?
Containers are popular due to their portability and enablement of application reusability. They can be easily deployed to multiple environments and container fleets can be orchestrated to cater for large-scale deployments. This also means if we have a security issue with our containers or container images, we will have that problem at a large scale and potentially across multiple environments.
Hence, we want to look into common risks and how we can address them as part of our DevSecOps approach.
The three main risk categories for containers throughout their life-cycle are:
- A container image can be compromised, for example while it is stored in the container registry. As a result we might create new containers based on the compromised image, which means all containers based on that particular image will be vulnerable.
- Containers can be compromised while they are running. The hacker can then exploit data or credentials on the compromised container.
- Once a container is compromised it can be misused to attack other containers, the container’s host OS, or other hosts.
In this write-up we are focusing on the first risk category: We want to make sure that container images cannot be compromised either before they get uploaded to our container registry or while they are stored in the registry.
How to Address Container Image Risks
Let’s assume we have created a container image that includes a Linux kernel, some software packages, and our bespoke application. This means we have a couple of layers that are deployed on top of each other. This works similar to a GIT repository: every time we perform a commit, it is added to the list of commits. A container image represents a sum of all these commits. Each commit we perform will create a new layer.
From a security perspective this means that each layer adds additional risk. For example: we could have vulnerabilities on the OS level, or miss out on security patches on the software package level, or a developer might accidentally include security keys or passwords to the bespoke application. Additionally, we need to make sure that container images cannot be modified without adequate permissions.
Considering these risks we need a cross-layered approach to validate our image before it turns into a container.
- Container scanning:
There are many container scanning solutions available. Some of them scan the container at runtime. This means we need to initiate the scan as part of our CI/CD pipelines. Once the scan result meets our expectations we can store the image in our container registry. All these steps should be performed automatically.
Most scanning products scan all layers for known vulnerabilities. The scan report will usually categorise the findings by severity level. Some cloud native registry offerings have a built-in image scanning solution, which can scan images during the initial upload and also recurring, to make sure new vulnerabilities are not found in existing images.
- Other scanning tools:
It is recommended to include binary scanning and credential scanning solutions in your CI/CD toolchain. Binary scanning solutions make sure that your third party libraries, such as Java libraries, are not vulnerable. Credential scanning solutions make sure that no one accidentally uploads access keys or passwords.
If we are dealing with a data-centric solution then a PII (personal identifiable information) scanning solution can be helpful to make sure we don’t have sensitive information where it should not be. For example development environments should not include real customer information, but rather dummy data or anonymised data. - Container registry access protection and encryption:
In order to reduce the attack vector, we will limit the physical access to known networks or IP addresses (e.g. private subnets in the cloud or corporate networks).
We also want to make sure that only authorised personnel or systems can upload and download images to and from our registry. Therefore we enforce authentication and authorisation. The transport protocol should be encrypted, which is usually a default setting in cloud based container registry solutions. - Image life-cycle & metadata:
We need to regularly refresh our images to make sure they are up to date and include all the latest security patches. Therefore we need to phase out older images.
We can do this by versioning our images and using metadata to mark older versions as deprecated. Metadata can also be used to describe the status (e.g. production-ready), application team, or purpose (e.g. internal, external). - Feedback loop:
Our image management process should happen as part of the CI/CD pipelines. We need a feedback loop to flag vulnerabilities. This prevents images with critical findings from being used and we can make sure that only non-vulnerable images can be downloaded from the registry.
Container security is a complex topic and the above guidelines should be a good starting point for tackling container security within CI/CD pipelines. Don’t forget to implement ongoing scans of your environments.
