Most AWS teams turn on GuardDuty and assume it will alert when something bad happens. Published tests show that assumption is only partly true. In one, six data-theft techniques run with Stratus Red Team produced zero GuardDuty alerts. In another, GuardDuty caught a stopped CloudTrail trail and S3 ransomware, but did not flag a backdoor access key on an IAM user.

The only way to know what your setup catches is to attack it. This post covers what those tests found, how to run the same kind of test safely in your own sandbox with cloud-firedrill, how to read the results, and how to close the gaps you find.

What did published GuardDuty tests find?

Two public write-ups ran Stratus Red Team against GuardDuty and reported results technique by technique. Together they show a clear pattern: GuardDuty reliably catches loud, signature-style events, and misses quieter data access.

Technique GuardDuty result Time to finding Source
Stop CloudTrail trail Detected 17 minutes Tyler, Apr 2025
S3 ransomware, client-side encryption Detected, with S3 Protection Nearly 1 hour Tyler, Apr 2025
Create access key on an IAM user Not detected n/a Tyler, Apr 2025
Retrieve many Secrets Manager secrets (two variants) Not detected n/a McCarthy, Jul 2024
Retrieve and decrypt SSM parameters Not detected n/a McCarthy, Jul 2024
Share an EBS snapshot Not detected n/a McCarthy, Jul 2024
Share an RDS snapshot Not detected n/a McCarthy, Jul 2024
S3 ransomware, batch deletion Not detected n/a McCarthy, Jul 2024

McCarthy ran his six techniques over VPN from varied locations and saw zero alerts. He notes the result matched a similar simulation by Invictus IR.

Read these results with care. Both are small, single-account tests, and McCarthy’s appeared on the blog of Tracebit, a company that sells a complementary detection product. GuardDuty’s anomaly models also depend on each account’s history, and its machine learning tends to struggle with low-volume, high-impact attacks such as stealing a single critical secret. Your results may differ, which is exactly why you should run your own.

Some gaps are by design. The InstanceCredentialExfiltration finding has documented bypasses, such as using stolen credentials through VPC endpoints. And because findings take minutes to appear, McCarthy estimated that even at the best observed latency, a fast attacker could exfiltrate around 100 GB from S3 before an alert fires.

Why isn’t the GuardDuty Tester enough?

AWS publishes the amazon-guardduty-tester and a guide to testing and evaluating GuardDuty detections. It generates activity designed to produce specific findings, which makes it a good check that GuardDuty is switched on and wired up. In McCarthy’s second experiment, 8 of 11 tester findings fired as expected, with a median latency of 15 minutes.

It answers a different question from the one an attacker poses. The tester shows which findings GuardDuty can produce. It doesn’t show what happens when someone runs the techniques attackers actually use, whether the finding reaches a person, or whether your SCPs block the next step. For that you need to run real techniques and check every layer.

What are Stratus Red Team and cloud-firedrill?

Stratus Red Team is an open source tool from Datadog for emulating common attack techniques directly in your cloud environment. Each technique is small, self-contained and mapped to MITRE ATT&CK, and the technique list now covers AWS, Azure, GCP, Entra ID and Kubernetes. Every technique has the same lifecycle: warm-up creates the prerequisites, detonate performs the attack, revert undoes its side effects, and cleanup removes everything.

Stratus detonates. It doesn’t tell you whether anything noticed. cloud-firedrill is an open source tool that builds on Stratus and adds that half:

  • runs a technique, then polls GuardDuty, Security Hub, CloudTrail, your alert queue or any SIEM for the evidence you said should appear;
  • measures how long each alert took;
  • reports each result as DETECTED, LOGGED, MISSED or PREVENTED;
  • always cleans up, and refuses to run in any account you didn’t name.

How to run your first fire drill

Step 1: Prepare a sandbox account

Use a dedicated sandbox account inside your AWS Organization, never production. Enable GuardDuty there with the same protection plans as production; some findings only exist with an add-on plan. For example, the S3 ransomware detection in one test required the S3 Protection plan. Use short-lived credentials, for example from IAM Identity Center.

Budget for it. GuardDuty is a paid service and the techniques deploy real, short-lived resources, so costs depend on the techniques you run. The starter scenarios use cheap resources only.

Step 2: Install

cloud-firedrill needs Python 3.10 or newer and the Stratus binary on your PATH:

pipx install cloud-firedrill
go install github.com/datadog/stratus-red-team/v2/cmd/[email protected]

Step 3: See what would run

plan validates the scenarios and prints what would happen, without calling AWS:

firedrill plan starter

The built-in starter scenarios need no extra setup. They stop a CloudTrail trail, backdoor an IAM user with a new access key, open an S3 bucket policy to an outside account, share an EBS snapshot, and try an SSE-C write of the kind used in S3 ransomware.

Step 4: Run it

firedrill run starter --expected-account 111111111111 --confirm

Three guards stand between this command and a mistake. The run refuses any account other than the one in --expected-account, refuses accounts whose alias looks like production, and does nothing at all without --confirm. Cleanup runs even when a technique fails.

A full run takes up to about 45 minutes, because cloud-firedrill waits for GuardDuty instead of judging too early. Published latencies range from 15 minutes to nearly an hour.

Step 5: Read the results

firedrill report results.json
Result What it means What to do
DETECTED An alert fired Check the time to detect
LOGGED CloudTrail recorded it, but nobody would have been alerted Add a detection
MISSED Nothing expected appeared Your detection, logging or alert path is broken
PREVENTED A guardrail denied the attack Good

Expect some LOGGED results from the starter set. That is the point: they show what GuardDuty alone does not catch in your account.

Fire drill loop: plan, run, read results; LOGGED or MISSED leads to a new CloudTrail rule and another run, DETECTED or PREVENTED is recorded and scheduled.

Which attack techniques should you test first?

Start with techniques that mirror attacks that have actually happened. This set maps each one to a real incident and to the GuardDuty finding documented for it, so you know what success looks like:

Technique Real-world pattern GuardDuty finding to expect Built-in scenario
aws.defense-evasion.cloudtrail-stop Attackers turning off logging to hide Stealth:IAMUser/CloudTrailLoggingDisabled, no add-on plan needed starter
aws.persistence.iam-backdoor-user New access keys for persistence, as in the 2025 Sysdig intrusion None on its own; GuardDuty only flags it if the key is later used for anomalous activity starter
aws.exfiltration.ec2-share-ebs-snapshot Data stolen by sharing a disk snapshot with an outside account None observed in McCarthy’s test starter
aws.credential-access.ec2-steal-instance-credentials Capital One, 2019: role credentials stolen from EC2 instance metadata InstanceCredentialExfiltration.OutsideAWS or .InsideAWS, depending on where the credentials are used guardduty-gaps
aws.impact.s3-ransomware-client-side-encryption S3 data encrypted and held for ransom AttackSequence:S3/CompromisedData, with S3 Protection guardduty-gaps

The guardduty-gaps scenarios need the sandbox Terraform that ships with cloud-firedrill, which adds an alert path and custom detections. Run firedrill list scenarios to see the built-in files, and stratus show <technique-id> to see exactly what a technique creates.

How to close the gaps

Every missed technique above still left a record in CloudTrail. As one tester put it, where GuardDuty stays quiet you can rely on CloudTrail logs to discover the activity. CloudTrail is also faster: Tracebit’s research puts the average CloudTrail delay at around two and a half minutes, versus 15 minutes or more for GuardDuty.

Write your own rules for the gaps. These map each missed technique to the CloudTrail events it produces:

Missed technique CloudTrail events to alert on
Backdoor access key CreateAccessKey where the caller is not the key’s own user, or any CreateAccessKey in an account that should have no IAM users
Mass secret retrieval Bursts of GetSecretValue or BatchGetSecretValue from one identity
SSM parameter theft GetParameters or GetParametersByPath with decryption, in volume
EBS snapshot sharing ModifySnapshotAttribute adding a create-volume permission for another account
RDS snapshot sharing ModifyDBSnapshotAttribute or ModifyDBClusterSnapshotAttribute adding a restore permission
S3 mass deletion High volumes of DeleteObject or DeleteObjects; this needs S3 data events, which are not available by default

cloud-firedrill’s sandbox Terraform includes EventBridge and Lambda detections for several of these: backdoor access keys, external role trust, shared snapshots and external bucket policies. They publish Security Hub findings that the guardduty-gaps scenarios check for, so a fix is proven by the same test that found the gap.

Make it continuous. Detection breaks silently when someone changes a log pipeline or a rule. Run the scenarios on a schedule in CI, with OIDC instead of stored keys, and compare each run with the last: a technique that was DETECTED last week and LOGGED this week is a regression worth an alert of its own.

Consider deception for the quiet attacks. Low-volume theft, like reading one critical secret, is hard for any anomaly model. Decoy credentials and resources that no legitimate process touches give a deterministic alert on first use, and can be built with commercial tools or your own CloudTrail rules.

Conclusion

GuardDuty is worth running, and the published tests show it catches loud events like a stopped trail or S3 ransomware. They also show it can stay silent on secret theft, snapshot sharing and backdoor keys, and that even its hits arrive minutes to an hour later.

You don’t have to take anyone’s word for it, including this post’s. Run the starter scenarios in a sandbox, write down what came back LOGGED or MISSED, and turn every miss into a CloudTrail rule you test again on the next run.

Sources

Stratus Red Team

GuardDuty test results and testing guidance

Detection references