AWS Elastic Container Service (ECS) and Elastic Kubernetes Service (EKS) require provisioning of compute resources to run container workloads.
When these services were initially announced, compute resources were provisioned using EC2. With EC2, customers have to figure out the size of the virtual machine(s) and add it to the pool of compute resources available for container workloads. EC2 instances are either provisioned on-demand or reserved instances are used, and customers have to worry about patching, security, etc.
In 2017, AWS released a feature called Fargate. Fargate under the hood uses EC2 virtual machines or bare metal and AWS manages the lifecycle of the virtual machines. So customers are not on the hook to patch and secure the compute resources. Initially, the cost of Fargate made it less palatable, but AWS subsequently reduced the prices significantly and customers now use Fargate with ECS or EKS.
Containers running inside Fargate can be configured to send log data to popular log aggregation tools. Some compute, memory, network, and telemetry information is also made available to customers.
Customers don’t have to worry about maintaining virtual machines, but they’re still on the hook for creating secure applications and properly packaging containers. Many of Uptycs’ customers are interested in details like:
- OS packages and versions in a container
- Application dependencies and versions (Node, Java, Golang, Python, etc.)
- Process events
- Environment
- File integrity monitoring
- Network activity
If the application running in the container is compromised, an attacker can access databases or other resources the application has access to. With Fargate, the surface area of attack has been reduced as customers are not on the hook to secure virtual machines or bare metal. But container security is still something customers have to address.
For ECS or EKS with EC2, customers run k8sosquery as a DaemonSet. Osquery tables can be used to gather information from the EC2 VM and also from the containers running on it. Customers also get process, file, socket, and other events in near real-time using either Linux auditd or eBPF. Fargate unfortunately does not provide osquery any means to gather details from the host.
The recent version (1.4) of Fargate added support for ptrace. This allows applications running inside the container to use process tracing to get process events. Osquery supported inotify for a while and this can be used to track file events inside the container. Using Uptycs’ AWS integration, AWS Flow Logs can be used to track network activity. To get events and other security insights, osquery needs to be running inside every Fargate container.
Ideally, osquery does not need to be packaged in every container. But Fargate is an exception to this. Below we document two approaches for packaging osquery in application containers. The focus of this post is on ECS with Fargate. We’ll talk about EKS with Fargate in a separate post.
Preparation
There are a few manual one-time actions that need to be performed.
Log in to your Uptycs platform and in the software downloads section, select “Download flags + secret.” You will not need the flags, but the secret must be provisioned into your AWS account.
Figure 1. Download Flags + Secret from Uptycs.
The Uptycs secret from the downloaded file should be saved in either AWS Secrets Manager or Systems Manager Parameter Store so that osquery can use it for enrollment with Uptycs. ECS provides the ability to retrieve secrets from AWS Secrets Manager or Systems Manager Parameter Store and inject them into a container as environment variables. However, we don’t prefer this approach. The osquery process_envs table captures the secrets from the container as plain text. Systems Manager Parameter Store SecureString might be optimal for the Uptycs secret as it is cost effective.
Figure 2. Storing a SecureString in the AWS Systems Manager Parameter Store.
If you already use ECS Task Role (not Task Execution Role), it should be updated to allow osquery to read the Uptycs secret. If not, an IAM role should be created for ECS tasks. A policy like the following can be attached to the existing or new task role to retrieve the Uptycs secret from the Parameter Store.
Figure 3. A sample policy that can be attached to a task role to retrieve the Uptycs secret. (Click to see larger version.)
Build Time
In this approach, osquery is packaged into every ECS task container when the containers are built. This requires changing the Dockerfile. For example, if the Dockerfile to deploy NGINX looks like the following:
Figure 4. Dockerfile example.
It should be modified as follows:
Figure 5. Modified Dockerfile. (Click to see larger version.)
FROMis added before the existingFROMinstruction. Uptycs ECS farquery is added with an osquery alias.- The
COPYinstruction is added to copy /opt/uptycs/osquery from the ecs-farquery image to the final image. - The
ENVinstruction is added to specify your Uptycs SaaS endpoint hostname and the Amazon Resource Names (ARN) where the Uptycs secret is saved. This can be specified in the image or in every task definition as environment variables (see Figure 7, below). ENTRYPOINTis updated to add/opt/uptycs/osquery/bin/entrypoint.shbefore the existing entrypoint.
When creating the task definition:
- Specify the IAM Task Role with a policy that allows osquery to read the secret from the Parameter Store.
- The task definition should also add the
SYS_PTRACEcapability, which allows osquery to capture process events.
Figure 6. Adding SYS_PTRACE, which allows osquery to capture process events.
- If the
ENVinstruction in the Dockerfile was not added forUPTYCS_HOSTNAMEandUPTYCS_SECRET, these should be added to the task definition:
Figure 7. UPTYCS_HOSTNAME and UPTYCS_SECRET being added to the task definition. (Click to see larger version.)
Finally, when creating the service:
- Use 1.4.0 as the
Fargate Platform Version.LATESTdefaults to 1.3.0.
Deploy Time
This approach does not require modifying the Dockerfile, but every task definition needs to be modified as described here.
Add a new non-essential container in containerDefinitions.
Figure 8. Adding a container in containerDefinitions. (Click to see larger version.)
You should modify entryPoint, linuxParameters, environment, and volumesFrom for all existing containers in containerDefinitions as follows:
Figure 9. Modifications in containerDefinitions. (Click to see larger version.)
The first argument in entryPoint should be /opt/uptycs/osquery/bin/entrypoint.sh. This should be followed by the ENTRYPOINT and/or CMD instructions values from the container. If ENTRYPOINT and/or CMD values from the container image are not appended here, the actual container application will not be started.
To add SYS_PTRACE capability, linuxParameter should be used. This lets osquery capture process events inside the container.
UPTYCS_HOME and UPTYCS_SECRET should be passed to the container as environment key values.
volumesFrom should be used to mount /opt/uptycs/osquery from the uptycs/ecs-farquery image.
In addition to updating task definitions, the service should be updated to use 1.4.0 as the Fargate Platform Version. LATEST, which is 1.3.0, does not support the PTRACE capability.
Uptycs
Once the ECS Fargate task is RUNNING, each task will show up as an asset in Uptycs:
Figure 10. Once the ECS Fargate task is active, each task will show up as an asset in Uptycs. (Click to see larger version.)
We recently added two new tables, ecs_metadata and ecs_task, that capture the ECS metadata and task information for each container:
Figure 11. Retrieving task information for an ECS container. (Click to see larger version.)
A flag profile that enables inotify (allow_inotify_file_events) and process events (audit_allow_process_events) can be created and assigned to Fargate containers to capture process and FIM activity.
Caveats:
- ptrace is not the most efficient way to capture process activity, but this is the only mechanism AWS Fargate exposes for tracking process events.
- inotify does not provide PID, but this should be a non-issue because containers typically are not expected to run multiple applications. All activity can be attributed to the container application.
