Detecting security threats is difficult work, now more so than ever. Our threat intelligence tools are playing catch-up with increasingly sophisticated attack vectors, including polymorphic malware, quick-turn domains and other turn-on-a-dime attack tactics.
Little wonder then that most security professionals place a high priority on real time threat intelligence and detection. In a constantly changing security landscape, rapid response threat intelligence tools are high on the cybersecurity wish list. Agility is important. However, chasing zero-day response time is only part of the solution.
If real time threat intelligence and detection tools are your eyes on the ground now, historical records provide a strategic view for incident investigation. (Tweet this!)
This article explores how you can use Uptycs to generate powerful queries of historical threat data, and why an analysis of past incidents can improve your threat intelligence.
Why Applying Threat Intelligence On Historical Data Improves Detection
As networks become hybridized across cloud and physical infrastructure, preventing malicious activity is an increasingly complex battle on multiple fronts. Not only is it harder to reliably detect an attack in real-time, it’s also common for attackers to cover their tracks.
Heightening this security problem, virtual machines can be created and destroyed at any time. The origin of an attack can be erased from existence before its activity is detected. In these situations, real time threat intelligence alone is of limited use.
Historical data allows you to piece together a clearer image of exactly how an attack occurred. Where real time threat intelligence can only reveal fragments of an attack, historical data permits you to reconstruct exactly what happened, and when. Moreover, historical data lets you uncover malicious activities which may have been completely missed using real time threat detection tools alone.
The Uptycs security analytics platform is unique in allowing security professionals to query both real-time and historical data against threat intelligence feeds, providing a simple and versatile approach to comprehensive threat assessment.
Handling Historical Data & Threat Intelligence At Scale
One of the greatest challenges of scanning historical data is scale. A large network’s historical dataset is enormous. Handling it efficiently requires both extremely robust infrastructure for storing data and a unique system architecture capable of querying vast datasets.
The Uptycs platform overcomes this problem of scale with high volume cloud infrastructure, and efficient query tools. Uptycs uses osquery to collect system state data from connected endpoints and cloud workloads, and then streams that data over a secure TLS protocol, storing it in an exclusive instance.
Uptycs also provides pruned threat intelligence feeds, enabling fast and simple integration with your system telemetry to know not just that a new threat is known, but whether it is or isn’t active in your environment. In this way, Uptycs delivers powerful scalability, and can accommodate historical data from networks of 100,000 machines or more—an impossible task for conventional threat detection tools.
Threat Intelligence Pruning Process
As importantly, Uptycs allows security professionals to generate detailed queries on threat intelligence data. These queries can quickly and efficiently deliver insight into incidents that would otherwise, in practical terms, remain out of analytical reach. This begins with categorization. Uptycs maintains over five million pieces of data on threat intelligence indicators. The data is analyzed for false positives and then broken into seven potential threat categories:
- Malware: All domains and IP addresses associated with malware activities.
- Attack: IP addresses associated with attack activity.
- Anonymizer: Domains and IP addresses associated with anonymization networks.
- Coinminer: Domains and IP addresses associated with crypto currencies, including pool services and websites.
- Phishing: Domains and IP addresses associated with phishing activities.
- DGA: Domains generated using a domain generation algorithm.
- NRD: Newly registered domains.
For every alert triggered during real-time scanning, Uptycs provides the category, description and threat intelligence source name, as shown below.
Categorizing real time threat intelligence in this way allows security professionals to choose exactly what they want to analyze while eliminating the distraction of less contextually relevant data.
Building One Query For Real Time And Historical Data
You can also use these granular threat categories in conjunction with queries that seamlessly blends historical and real time threat intelligence within a given timeframe. Here’s a practical illustration of what that looks like.
Uptycs’ entire threat intelligence database is exposed in a SQL table: upt_threat_indicators. A simple join query can be run on the global historical data, connecting process_open_sockets, dns_lookup_events or socket_events tables with the upt_threat_indicators table. For example, here is a query that will provide detection results from December 28th to the current day:
SELECT ut.indicator, ut.category, ut.description, p.pid, p.remote_address, p.upt_hostname from upt_threat_indicators ut inner join process_open_sockets p on p.remote_address = ut.indicator where p.upt_day >= 20181228 limit 1000;
The results of this query are shown below:
Note that the query sets a limit on the number of days to analyze. It’s advisable to start narrow (maybe a week, first) and then expand in increments as you have a better handle on the amount of data your query will return.
The query can be further refined to limit your analysis to the malware category:
SELECT ut.indicator, ut.category, ut.description, p.pid, p.remote_address, p.upt_hostname from upt_threat_indicators ut inner join process_open_sockets p on p.remote_address = ut.indicator where p.upt_day >= 20181228 and ut.category = 'malware' limit 1000;
Excluding Existing Triggered Alerts
In the above examples, real time threat intelligence is drawn into analysis. This provides the security professional with an overarching view encompassing all available time periods. However, some analysis is better performed when previously detected incidents are excluded; for example, if you need to isolate a historical timeline of previously unknown activities from real time threat detection results.
The following query zeroes in on results not yet detected in alerts:
SELECT ut.INDICATOR,
ut.category,
Ut.description,
p.pid,
p.remote_address,
p.upt_hostname
FROM upt_threat_indicators ut
INNER JOIN process_open_sockets p
ON p.remote_address = ut.INDICATOR
JOIN upt_alerts u ON (p.remote_address <> u.value AND p.upt_asset_id = u.upt_asset_id
AND p.upt_day = CAST(date_format(alert_time, '%Y%m%d') AS INTEGER))
WHERE p.upt_day >= 20181228 and ut.category = 'malware' limit 1000;
The results of this query are shown below:
Real Time Data Needs Historical Context
Using osquery, Uptycs allows security professionals to analyze real time threat intelligence data alongside a large body of invaluable historical system data. As these simple examples demonstrate, the platform offers analytical versatility at any scale. This makes routine analysis of large historical datasets a practical daily reality.
By looking back on threat intelligence data, security professionals are better equipped to stay on top of an increasingly demanding security arena. If you’d like to see interactive threat hunting through osquery in action, sign up for our on-demand webinar or ask us to see it live.
Acknowledgement: Thanks to my colleague Vibhor Kumar in helping me with SQL queries.
