Last updated 23/07/2021
Amazon GuardDuty is an automated threat detection service that continuously monitors for suspicious activity and unauthorized behavior to protect your AWS accounts, workloads, and data stored in Amazon S3. In this post, I’ll share how you can use GuardDuty with its newly enhanced highly-customized machine learning model to better protect your AWS environment from potential threats. The model works at scale and across a broad spectrum of customer use cases, workloads, and operating models. The new findings available to you in GuardDuty allow you to use anomaly detection to generate security-relevant results that are differentiated from unusual but benign, without a lot of false alarms or noise.
The new GuardDuty machine learning model operates on the continuous stream of API invocations that occur in your AWS accounts, based on user activity that is tracked in AWS CloudTrail. The model is based on Variational Autoencoders, which have recently emerged as one of the most successful approaches to model complex data distributions. This allows GuardDuty to learn the probability distribution of API calls invoked by users in your AWS account. Even if a user in your account operates in a certain way for the first time, the model will predict whether it is part of their normal, expected operations. It’s this added probabilistic nature that helps to produce accurate detections of suspicious behavior within your account.
GuardDuty now uses this new machine learning model to identify unusual activity within your accounts, analyze the security relevance of the activity, given the context in which it was invoked, and apply predictive probability to make a final verdict on whether that activity is sufficiently anomalous to warrant investigation. The new model-based threat detections added to GuardDuty will also help you identify the attack tactic associated with the anomalous API invocations, including discovery, initial access, persistence, privilege escalation, defense evasion, credential access, impact, and data exfiltration.
The new GuardDuty findings have shown an average decrease in anomalous account activity alerts by over 50%, while also expanding the monitoring coverage by over 300%. The anomalous user behavior these new findings flag is more likely to represent actual malicious activity compared to the previous anomaly detections in GuardDuty.
Suspicious behavior the machine learning model can now help you detect includes:
If you are an existing GuardDuty customer, then the new machine learning threat detections are already available to you as part of the service. As with all GuardDuty detections, we work to continually expand and improve them, and new detections are enabled by default for all your GuardDuty-enabled AWS accounts, at no additional cost. This means that as you are reading this, the model is tracking API activity by users in your accounts across dozens of AWS services, including Amazon EC2, Amazon Identity and Access Management (IAM), Amazon Simple Storage Service (S3), Amazon DynamoDB, Amazon Elasticsearch Service, Amazon Relational Database Service (RDS), AWS Key Management Service (KMS), Amazon Elastic Container Registry (Amazon ECR), and AWS Secrets Manager.
The new machine learning behavioral detections in GuardDuty give you rich contextual data as part of the GuardDuty findings. The new GuardDuty findings allow you to make quick, on-the-spot decisions about whether to trigger an incident response workflow. You can view this enriched context in the GuardDuty console, and the full detail is included in the finding JSON pushed out through Amazon EventBridge and published to AWS Security Hub.
In the new findings, you can now see a detailed list of all anomalous APIs that were invoked by the same user within a window of several minutes. For example, instead of telling you that IAM user Admin-1 anomalously invoked the S3 ListBuckets API, GuardDuty will tell you that in proximity to that operation the user also anomalously invoked the GetBucketACL, GetBucketPublicAccessBlock, and PutBucketPublicAccessBlock APIs. All the APIs on the list will be grouped by their associated AWS service to make it even easier for you to quickly understand which services were accessed as part of the anomalous activity. The finding will also provide details on which of the anomalously invoked APIs were called successfully and which ones failed, including the error response received. This context can be important as you triage the findings. For example, successful API invocations indicate the user was able to perform the operations and could represent a higher severity incident. On the other hand, if the user’s access was denied, this could indicate compromised credentials with an attacker attempting to identify what access permissions are available.
Each finding will explicitly call out which attributes of the activity were unusual for the specific user, as well as for all other users that operate in the same AWS account to help you identify why the behavior was flagged as highly suspicious. Finally, the finding details also include information on what the expected behavior is for the user, and for all other users that operate in the same AWS account. The expected behavior will be grouped based on its frequency during the profiling period. For example, frequent (daily or weekly), infrequent (a few times a month), or rare (less than once a month).
Let’s walk through an example. As you triage alerts in the GuardDuty console, you see an alert on anomalous discovery related activity shown in Figure 1:
Figure 1: Discovery: IAMUser/AnomalousBehavior finding Anomalous APIs section view
Under the Anomalous APIs section, you see that the user successfully invoked three DynamoDB and RDS-related APIs that are associated with discovery tactics: ListTables, DescribeTables, and DescribeDBSnapshots.
Working your way down the findings detail pane, next you see the Unusual behavior sections shown in Figure 2.
Figure 2: Discovery: IAMUser/AnomalousBehavior finding unusual behavior section view
Analyzing this section, you learn that the activity was conducted from a remote IP associated with a network that is unusual for this user, as well as for all other users that operate in the same AWS account. You can also see that it is unusual for this user to invoke the APIs that were used to list and describe DynamoDB tables and to describe RDS database snapshots. Below the Unusual behavior sections, you see three additional sections that provide additional important details on the activity that are shown in Figure 3.
Figure 3: Discovery: IAMUser/AnomalousBehavior finding additional sections
The Resource affected section helps you answer important questions about the AWS IAM user associated with the activity, including the user name and user type. The Action section allows you to dive deeper into one of the API actions that were part of the activity, including the user agent that was used as part of the activity. Finally, under the Actor section, you can see information on the remote IP used, including the geolocation and associated network.
At this point, this activity already looks highly suspicious, as the actor operated from a network that no other user in your AWS account has operated from, and the APIs invoked are not part of profiled operations. However, there is more you can review before making a final verdict. Diving one layer deeper, you can review what behavior is expected for the user, as well as for the rest of the users in this AWS account. Returning to the Anomalous APIs section (shown in Figure 1), you can choose the Usual APIs button to open up the dialog box shown in Figure 4.
Figure 4: Discovery: IAMUser/AnomalousBehavior usual APIs dialog box
The Usual APIs dialog box shows you a list of the top-20 APIs that are most commonly invoked both by the user (under User Identity on the left-hand side), as well as all other users that operate in your AWS account (under All users in the account on the right-hand side). You can quickly scan the list and see that invoking APIs associated with DynamoDB and RDS is not part of common operations in your AWS account. Rather, this specific user often invokes APIs associated with Amazon GuardDuty, while the rest of the users in your account often invoke APIs associated with S3 buckets and EC2 instances.
To finish the triage process, you check the usual behavior for both the AWS account and the user that invoked the anomalous APIs. You return to the Unusual behavior (Account) section (shown in Figure 2) and choose the Usual behavior button to open up the dialog box shown in Figure 5.
Figure 5: Discovery: IAMUser/AnomalousBehavior Account usual behavior
Then, you go to the unusual behavior (User Identity) section (shown in Figure 2) and choose the Usual behavior button to open up the dialog box shown in Figure 6.
Figure 6: Discovery: IAMUser/AnomalousBehavior User Identity usual behavior
The Usual behavior (Account) dialog box in Figure 5 shows the expected behavior across all users in your account. The Usual behavior (User Identity) dialog box in Figure 6 shows the expected behavior for the user that invoked the anomalous APIs. There are a number of different tabs in each dialog box that provide you with information about various attributes of the expected behavior. Focusing on the commonly used networks, you can see that users in this AWS account, including the specific user that invoked the anomalous APIs, usually operate from the Amazon network, or from networks of common US-based service providers. This provides you with further evidence that the anomalous behavior detected by GuardDuty is highly suspicious.
After you identify that the activity detected is clearly suspicious, you can choose the Investigate with Detective link at the top of the finding detail pane shown in Figure 7, to start an Amazon Detective investigation.
Figure 7: Discovery: IAMUser/AnomalousBehavior launching an Amazon Detective investigation
Amazon Detective complements Amazon GuardDuty by collecting log and event data from sources such as AWS CloudTrail and Amazon Virtual Private Cloud (VPC) Flow Logs. Detective organized the data into an analytics-driven graph model that summarizes resources, user behaviors, and associated interactions observed across all enabled accounts for up to the last 12 months. Detective uses this data to produce tailored visualizations that summarize network and user activity across all your enabled accounts.
Detective can help you quickly answer questions such as the following:
By providing visual analytics and summaries collected from management events and network traffic, Detective helps you identify the root cause of security issues. Being able to quickly see patterns of activity while being able to drill down and understand the details, help you quickly understand how long issues have been going on and what needs to be remediated.
If you are already using Amazon GuardDuty today then no further action is required in order for you to start using this new capability to view the findings in the console. If you have set up GuardDuty to push findings through AWS EventBridge for downstream consumption by workflow tools, then you should verify that your EventBridge rules are configured to deliver these newly added findings.
Topic Related Post
NovelVista Learning Solutions is a professionally managed training organization with specialization in certification courses. The core management team consists of highly qualified professionals with vast industry experience. NovelVista is an Accredited Training Organization (ATO) to conduct all levels of ITIL Courses. We also conduct training on DevOps, AWS Solution Architect associate, Prince2, MSP, CSM, Cloud Computing, Apache Hadoop, Six Sigma, ISO 20000/27000 & Agile Methodologies.
* Your personal details are for internal use only and will remain confidential.
 |
ITIL
Every Weekend |
|---|---|
 |
AWS
Every Weekend |
 |
DevOps
Every Weekend |
 |
PRINCE2
Every Weekend |
