Securing Workloads with Software Pipeline Detection & Response

Next Gen Workload Detection & Response Through Stronger Correlation of Runtime To Code Commit

Even with the presence of various offline security scanners, as part of your software pipeline, malicious and vulnerable code makes it to production deployment. It is not to say offline security scanners (e.g. Agentless, CI time, etc) are not necessary. However, they are proving insufficient to catch more new waves of attacks that focus on injecting malicious code rather than exploiting known vulnerabilities. Hence there is a need to run time protection and also there is a need to do a very quick RCA when these are caught during runtime. 

Let’s first dive deep into recent attacks that highlight the pain points and then discuss how Uptycs enables quick detection & root cause analysis of these types of attacks.

The Emerging Threat Landscape

The complexity and interconnectedness of modern software development environments have opened new avenues for cyberattacks. Recent incidents highlight the vulnerabilities across different stages of the software supply chain.

These examples underscore the importance of dynamic and comprehensive security measures throughout the software development lifecycle to safeguard against sophisticated and evolving threats.

Recent breaches in the software supply chain reveal how attackers exploit vulnerabilities at different stages of software development, emphasizing the need for robust security measures across the entire lifecycle:

1. SolarWinds Attack: Attackers inserted malicious code into the Orion software updates, compromising thousands of organizations, including U.S. government agencies. This breach led to widespread espionage and data theft, demonstrating the risks of supply chain vulnerabilities.
2. Dependency Confusion: Attackers uploaded malicious packages to public repositories, tricking organizations into incorporating these compromised packages into their software builds. This resulted in unauthorized access and data theft, highlighting the dangers of relying on public dependencies without stringent validation.
3. Fake Dependabot Commits: Attackers mimicked Dependabot, an automated tool for managing dependencies, by creating fake commits. These fake updates could introduce malicious code into software projects, stressing the importance of verifying the authenticity of automated tools and their outputs.
4. NPM Reverse Shell and Data Mining: In this type of attack, malicious NPM packages were used to establish reverse shells, giving attackers control over compromised systems. They also engaged in data mining, extracting sensitive information from the infected environment. This highlights the importance of vetting packages from public repositories and monitoring for suspicious activity.

Solution

In order to solve for these new waves of threats, you need the following:

This evidences the need to solve for some key challenges: 

• Image Provenance - Software install to commit (image provenance) understand entire software process flow that was involved in producing the deployed artifact.

• Runtime Workload SCA - an understanding of the software that went into the image including 3rd party dependencies, licensing, and packages and metadata of what is deployed and running.

• Software Supply Chain Posture - Your supply chain may be secure now but how secure was it during the time of the image build. Potential vulnerabilities may have been introduced due to software supply chain vulnerabilities.

• Continuous Compliance and Hardening - In order to prevent malicious commits from entering the pipeline and addressing software supply chain vulnerabilities in Code repositories, CI pipelines and Registries, it is important to continuously measure and harden your software supply chain.

Example

Detection: First we see a reverse shell via eBPF Detection. We see with deep attribution context that the reverse shell came from execution of ngserver.sh.

This was spawned from a container instance deployed. We can see the container image - this can tell us about what was checked code wise and also if we caught any vulnerabilities during the build phase.

Root Cause Analysis: Next in order to understand the origin of the root cause we go to the container image to see a full code commit to runtime provenance.

Here we can see that this image went through GitHub, Jenkins, Artifactory, and is now deployed in Kubernetes. We also see that each stage of the development process has a violation. We can work backwards to triage the root cause of the reverse shell detection by looking at the code commit. We see that one of the PRs has violations.

When we look at the PR we see that it only has 1 Approved Review while others have 2. Looking at the Repository Branch Protection Rules, it does say that every PR should have 2 Required Approval Reviews. This could be an indication that there has been a bypass and that some malicious code could have been committed.

PR failing the check of having 2 Approved Reviews.

Branch Protection rules requires 2 approved reviews per PR

When we dig deeper into the code review, we can see that malicious code for the reverse shell - the same ngserver that we saw in the detection was checked in. This ngserver itself is leading to the reverse shell.

With provenance, organizations can save significant time in root causing vulnerabilities and instead use that time to assess risk and patch key risks and vulnerabilities in their environment!

With Root Cause Analysis completed, the final step in the Uptycs Blast Radius Mitigation Framework ensures proactive defenses—read on to discover how DevSecOps Guardrails round out this powerful security strategy.

The Uptycs Blast Radius Mitigation Framework is a five-step journey to cloud security resilience. Read the guide to learn more.