BEST PRACTICE SERIES

Datadog
Cloud SIEM
BEST PRACTICE SERIES

Datadog Cloud SIEM

Best practices for monitoring AWS CloudTrail logs 3

Blazing trails 4

Understanding AWS CloudTrail audit logs 5

Key CloudTrail audit logs to monitor 12

Collect and analyze CloudTrail logs with Datadog 15

Start monitoring your AWS CloudTrail audit logs 18

Best practices for monitoring GCP audit logs 19

A primer on the Google Cloud hierarchy 20

Understanding Google Cloud Audit Logs 20

Key GCP audit logs to monitor 27

Shipping your audit logs 30

Collect and analyze audit logs with Datadog 31

Start monitoring your Cloud Audit Logs 36

Best practices for monitoring Microsoft Azure platform logs 37

Understanding Azure platform logs 39

Key Azure platform logs to monitor 44

Shipping your Azure platform logs 48

Collect and analyze Azure platform logs with Datadog 48

Start monitoring your Azure platform logs with Datadog 51

Best practices for monitoring Kubernetes security via audit logs 52

A primer on generating audit logs 53

Interpreting your Kubernetes API server audit logs 53

Key Kubernetes audit logs to monitor 55

Monitor Kubernetes security with Datadog 59

Start monitoring your Kubernetes audit logs 60

Best practices for monitoring authentication logs 61

Best practices for writing authentication logs 62

Use authentication logs to detect common security threats 65

Monitor your authentication logs with Datadog 66

Get insight into all of your authentication events 68

 Best practices for monitoring
AWS CloudTrail logs
Justin Massey Engineering teams that build, scale, and manage cloud-based applications
on AWS know that at some point in time, their applications and infrastruc-
Jonathan Epstein ture will be under attack. But as applications expand and new features
are added, securing the full scope of an AWS environment becomes an
increasingly complex task.
To add visibility and auditability, AWS CloudTrail tracks the who, what,
where, and when of activity that occurs in your AWS environment and
records this activity in the form of audit logs. Accordingly, CloudTrail audit
logs contain information that is key to monitoring the actions performed
across your AWS accounts, identifying possible malicious behavior and
surfacing parts of your infrastructure that might not be configured properly.
In this guide, we’ll look at:
– the structure of AWS CloudTrail audit logs
– important CloudTrail logs to monitor
– how Datadog can help you collect and monitor your CloudTrail logs
But first, let’s take a quick look at how AWS CloudTrail organizes and
monitors activity within your AWS accounts.
[Link] Best practices for monitoring AWS CloudTrail logs 4

Blazing trails As previously mentioned, AWS CloudTrail records each instance of activity
(such as API requests and user logins) it detects in your environment as
an event, which is a JSON object that specifies the activity’s particulars,
including the time at which it occurred, who performed the activity, the
resources that were affected by the activity, and more. You can view and
filter all of your events in the Event History page in the AWS CloudTrail
console, where they are available for up to 90 days after they occur.

Most AWS customers use a consolidated trail for all CloudTrail events.
However, you can create an event stream that filters in or out events. For
instance, in order to reduce your log load, you might want to create an
event stream that solely consists of activity related to a certain AWS
service or resource. To do this, you create a trail, or an event stream that
sends events to a chosen AWS S3 bucket as log files. This way, your
events are available according to the retention policy you specify, can be
quickly filtered to find critical issues, and can be alerted on using Amazon
CloudWatch or Amazon Simple Notification Service (SNS).
CloudTrail saves your audit logs in gzip archive form to the S3 bucket that
you specify when creating the trail. The name of the file includes the trail
creator’s account number, the Region in which the log was recorded, and
the month, day, and year when the file was created. For more information
on finding your CloudTrail log files, see the AWS documentation.
By default, trails are Region agnostic; that is, a trail will log relevant events
across every Region. You can create single-Region trails to focus on a single
Region’s activity, but we recommend creating an all-Region trail, as doing
so will give you more visibility and automatically track data from new
Regions as they come online.
[Link] Best practices for monitoring AWS CloudTrail logs 5

You can also set up an organization trail to monitor all of the logs gener-
ated by the AWS accounts within an AWS Organization. AWS Organizations
allows you to centrally manage the access permissions of users in all of
the accounts in the organization, and can be set up at no additional cost.
Organizations are recommended when your team needs to manage many
different AWS accounts by governing your ever-changing environment and
enforce configurations on your primary and member accounts.

Understanding AWS CloudTrail records three different types of events from most AWS
AWS CloudTrail services based on the actions users perform in the AWS Management
audit logs Console, Command Line Interface (CLI), and SDKs/APIs, as well as
automated actions performed by AWS. For a list of services that are
not tracked by CloudTrail, see the AWS documentation. The three event
types are:
– Management events: entries for management and network (control plane)
operations performed on the resources in your AWS account, such as
security group configuration changes, IAM role permission adjustments,
and AWS Virtual Private Cloud (VPC) network alterations.
– Data events: entries for data request operations—such as Get , Delete ,
and Put API commands—performed on an AWS data plane resource.
– Insight events: entries that reflect unusual API activity in your AWS
account in comparison to your historical API usage, such as excessive
API calls in a short frame of time.
As management and data events make up the vast majority of event logs
in CloudTrail, we’ll look at them in more detail. For more information on
using insight events to track and discover anomalies in your AWS data,
see the AWS documentation.

MANAGEMENT EVENTS
Management events include all management operations performed on
resources in your account, as well as most non-API actions. Non-API actions
include logins ( AwsConsoleSignIn ) to the AWS console and automated
service actions like cryptographic key rotations ( AwsServiceEvent ). AWS
CloudTrail logs management events by default.
The sample management event below records a console login, indicated
by the field eventType: AwsConsoleSignIn . It shows that someone with
the userName Alice successfully signed in to the AWS console without
multifactor authentication.
[Link] Best practices for monitoring AWS CloudTrail logs 6

{
"eventVersion": "1.05",
"userIdentity": {
"type": "IAMUser",
"principalId": "AIDABBBBBBBBBBBBBBBBB",
"arn": "arn:aws:iam::111111111111:user/alice",
"accountId": "111111111111",
"userName": "alice"
},
"eventTime": "2020-09-23T09:09:56Z",
"eventSource": "[Link]",
"eventName": "ConsoleLogin",
"awsRegion": "us-east-1",
"sourceIPAddress": "[Link]",
"userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.135
Safari/537.36",
"requestParameters": null,
"responseElements": {
"ConsoleLogin": "Success"
},
"additionalEventData": {
"LoginTo": "[Link]
"MobileVersion": "No",
"MFAUsed": "No"
},
"eventID": "6894a571-9f34-47b8-b75c-5f4ca34f281e",
"eventType": "AwsConsoleSignIn",
"recipientAccountId": "111111111111"
}

DATA EVENTS
Data events provide details on the operations performed on or within a
resource or service, such as AWS IAM roles, Amazon EC2 instances, Amazon
S3 buckets, and AWS Lambda functions. Because they are often high-
volume activities, data events are disabled by default when you create a
trail; you must add the resources or resource types to a trail in order to
track them in AWS CloudTrail.
The below example shows that user Alice successfully performed the
PutObject Amazon S3 operation on a bucket called example-bucket to
upload the file [Link] .
[Link] Best practices for monitoring AWS CloudTrail logs 7

{
"eventVersion": "1.07",
"userIdentity": {
"type": "IAMUser",
"principalId": "AIDAAAAAAAAAAAAAAAAAA",
"arn": "arn:aws:iam::111111111111:user/Alice",
"accountId": "111111111111",
"accessKeyId": "AKIAAAAAAAAAAAAAAAAAA",
"userName": "Alice"
},
"eventTime": "2020-09-22T20:15:25Z",
"eventSource": "[Link]",
"eventName": "PutObject",
"awsRegion": "us-east-1",
"sourceIPAddress": "[Link]",
"userAgent": "[Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.102
Safari/537.36]",
"requestParameters": {
"X-Amz-Date": "20200922T201524Z",
"bucketName": "example-bucket",
"X-Amz-Algorithm": "AWS4-HMAC-SHA256",
"x-amz-acl": "private",
"X-Amz-SignedHeaders": "content-md5;content-type;host;x-amz-acl;
x-amz-storage-class",
"Host": "[Link]",
"X-Amz-Expires": "300",
"key": "[Link]",
"x-amz-storage-class": "STANDARD"
},
"responseElements": null,
"additionalEventData": {
"SignatureVersion": "SigV4",
"CipherSuite": "ECDHE-RSA-AES128-GCM-SHA256",
"bytesTransferredIn": 12,
"AuthenticationMethod": "QueryString",
"x-amz-id-2": "d2UncmUgaGlyaW5nIDopIGh0dHBzOi8vd3d3LmRhdGFkb2docS
5jb20vY2FyZWVycy8K",
"bytesTransferredOut": 0
},
"requestID": "EEEEEEEEEEEEEEEE",
"eventID": "f378e059-d87f-44b7-aee2-7ebfa1beff93",
"readOnly": false,
"resources": [
{
"type": "AWS::S3::Object",
"ARN": "arn:aws:s3:::example-bucket/[Link]"
},
{
"accountId": "111111111111",
"type": "AWS::S3::Bucket",
"ARN": "arn:aws:s3:::example-bucket"
}
],
"eventType": "AwsApiCall",
"managementEvent": false,
"recipientAccountId": "111111111111",
"eventCategory": "Data"
}
[Link] Best practices for monitoring AWS CloudTrail logs 8

INTERPRETING YOUR CLOUDTRAIL LOGS

AWS CloudTrail logs contain invaluable information that lets you monitor
activity across your AWS environment, so it’s important to understand how
to interpret them in order to conduct investigations. In this section, we’ll
do a deep-dive into a sample management event in a CloudTrail log file to
illustrate which fields you should focus on.
CloudTrail log files are written in JSON format, with each event presented
as a single JSON object. Entries of all event types include some of the
same important fields, such as the access key ID of the AWS identity that
performed the action ( userIdentity fields) and the details of the action
performed ( eventName and requestParameters ). Management and
data event entries also provide responseElements fields that help you
determine whether the action was successfully performed.
In the snippet below, we can see that a user named Alice ( userName ) made
a call to create a new user ( eventName ) named Bob ( requestParameters ).

{
"eventVersion": "1.05",
"userIdentity": {
"type": "IAMUser",
"principalId": "AIDAAAAAAAAAAAAAAAAAA",
"arn": "arn:aws:iam::111111111111:user/Alice",
"accountId": "111111111111",
"accessKeyId": "AKIAAAAAAAAAAAAAAAAAA",
"userName": "Alice"
},
"eventTime": "2020-09-21T10:31:20Z",
"eventSource": "[Link]",
"eventName": "CreateUser",
"awsRegion": "us-east-1",
"sourceIPAddress": "[Link]",
"userAgent": "[Link]",
"requestParameters": {
"userName": "bob",
"tags": []
},
"responseElements": {
"user": {
"path": "/",
"userName": "bob",
"userId": "AIDABBBBBBBBBBBBBBBBB ",
"arn": "arn:aws:iam::111111111111:user/bob",
"createDate": "Sep 21, 2020 10:31:20 AM"
}
},
"requestID": "604e7549-4ea4-4185-83b0-acff4e462d27",
"eventID": "600e50af-0a2c-4352-95a8-7b813c744072",
"eventType": "AwsApiCall",
"recipientAccountId": "111111111111"
}
[Link] Best practices for monitoring AWS CloudTrail logs 9

Because the entry returns identification details for the newly created
user ( responseElements ), we know that the command was successfully
performed. Otherwise, the JSON response would have included
an errorCode and errorMessage element, as seen in the AWS
documentation.
Before we look at the most important CloudTrail logs to monitor, it’s
essential to understand the different user identity types defined by
CloudTrail, and how CloudTrail identifies the user who performed an action.

CloudTrail identity types

Every CloudTrail event log contains a userIdentity element that describes
the user or service that performed the action. Within this element, the
type field describes which sort of user or service made the request and
which level of credentials that user or service employed to make the
request. CloudTrail userIdentity types include:
– Root: The request was made with your primary AWS account credentials.
If you set up an alias for your AWS account, that alias will appear here
instead.
– IAMUser: The request was made with the credentials of an IAM user.
– FederatedUser: The request was made by a user with temporary security
credentials provided through a federation token.
– AWSAccount: The request was made by a third-party AWS account.
– AWSService: The request was made by an AWS service account.
Many AWS services use service accounts to perform automated actions
on your behalf.
– AssumedRole: The request was made with temporary credentials obtained
by using the AWS Security Token Service (STS) AssumeRole operation.
While most of these identity types are fairly straightforward, AssumedRoles
obfuscate the name of the user who performed the action. In the following
section, we will look at how AssumeRole calls work in practice, how to
determine the user behind an AssumedRole identity, and how a clever
adversary might use an AssumedRole session to hide their true identity.

Interpreting the initial identity of an ‘AssumedRole’ CloudTrail log

A common practice for a multi-account setup in AWS is to manage all
users in a single AWS account, which we’ll call account A. In turn, a security
best practice is to make sure that IAM users do not have any IAM policies
directly associated with them, and to instead give them temporary creden-
tials to perform actions. We can do this by creating a separate account
(e.g., account B) that contains IAM roles, each of which has a set of allowed
actions that are defined in an IAM policy. We can then allow users in
account A to assume those roles when they need to perform an action.
[Link] Best practices for monitoring AWS CloudTrail logs 10

Let’s say a user in account A wants to list all of the AWS Regions enabled
in account B. First, the user would AssumeRole into a role in account B
that has DescribeRegions permissions, obtain the temporary credentials
returned by the AssumeRole command, and then use them to perform
the command. The CloudTrail log in which a user ( userName: Alice )
from account A ( accountId: 222222222222 ) assumes a role in account B
( accountId: 11111111111 ) would look like this:

{
"eventVersion": "1.05",
"userIdentity": {
"type": "IAMUser",
"principalId": "AIDAAAAAAAAAAAAAAAAAA",
"arn": "arn:aws:iam::222222222222:user/Alice",
"accountId": "222222222222",
"accessKeyId": "AKIAAAAAAAAAAAAAAAAAA",
"userName": "Alice"
},
"eventTime": "2020-09-22T16:23:50Z",
"eventSource": "[Link]",
"eventName": "AssumeRole",
"awsRegion": "us-east-1",
"sourceIPAddress": "[Link]",
"userAgent": "aws-sdk-go/1.16.8 (go1.12.7; linux; amd64)",
"requestParameters": {
"roleArn": "arn:aws:iam::111111111111:role/ExampleRole",
"roleSessionName": "ExampleRoleSession",
"externalId": "ffffffffffffffffffffffffffffffff",
"durationSeconds": 3600
},
"responseElements": {
"credentials": {
"accessKeyId": "ASIADDDDDDDDDDDDDDDD",
"expiration": "Sep 22, 2020 5:23:50 PM",
"sessionToken": "d2UncmUgaGlyaW5nIDopIGh0dHBzOi8vd3d3
LmRhdGFkb2docS5jb20vY2FyZWVycy8K"
},
"assumedRoleUser": {
"assumedRoleId": "AROAEEEEEEEEEEEEEEEEE:ExampleRoleSession",
"arn": "arn:aws:sts::111111111111:assumed-role/ExampleRole/
ExampleRoleSession"
}
},
"requestID": "4da64d92-6130-4355-86f2-1609a6eb53e1",
"eventID": "ffef7974-b1a0-4e88-b27f-0b143965f30c",
"resources": [
{
"accountId": "111111111111",
"type": "AWS::IAM::Role",
"ARN": "arn:aws:iam::111111111111:role/ExampleRole"
}
],
"eventType": "AwsApiCall",
"recipientAccountId": "111111111111",
"sharedEventID": "4f61c867-6a49-4c41-a267-388c38e99866"
}
[Link] Best practices for monitoring AWS CloudTrail logs 11

The AssumeRole command returns an AccessKeyId

( ASIADDDDDDDDDDDDDDDD ) that user Alice can then use to perform
the role’s delegated actions. In the following event log, we can see
that an AssumeRole user uses the access key ASIADDDDDDDDDDDDDDDD
to perform the DescribeRegions operation; we can thus infer that
user Alice used the access key.

{
"eventVersion": "1.05",
"userIdentity": {
"type": "AssumedRole",
"principalId": "AROAEEEEEEEEEEEEEEEEE:ExampleRoleSession",
"arn": "arn:aws:sts::111111111111:assumed-role/ExampleRole/
ExampleRoleSession",
"accountId": "111111111111",
"accessKeyId": "ASIADDDDDDDDDDDDDDDD",
"sessionContext": {
"sessionIssuer": {
"type": "Role",
"principalId": "AROAEEEEEEEEEEEEEEEEE",
"arn": "arn:aws:iam::111111111111:role/ExampleRole",
"accountId": "111111111111",
"userName": "ExampleRole"
},
"webIdFederationData": {},
"attributes": {
"mfaAuthenticated": "false",
"creationDate": "2020-09-22T15:58:31Z"
}
}
},
"eventTime": "2020-09-22T16:26:02Z",
"eventSource": "[Link]",
"eventName": "DescribeRegions",
"awsRegion": "us-east-1",
"sourceIPAddress": "[Link]",
"userAgent": "aws-sdk-go/1.16.8 (go1.12.7; linux; amd64)",
"requestParameters": {
"regionSet": {}
},
"responseElements": null,
"requestID": "0a857cb2-90c4-4f09-9624-1149fb27f8a1",
"eventID": "26fe99a5-8ed5-4923-9cf7-b6cdf96fa5f3",
"eventType": "AwsApiCall",
"recipientAccountId": "111111111111"
}
[Link] Best practices for monitoring AWS CloudTrail logs 12

Controlling AssumedRole session names

A good way to control assumed roles and more easily track users who
perform actions using assumed roles is to stipulate the user’s session
name. To do this, you specify the permissible session names in the trust
policy of the role that will be assumed. For example, the following trust
policy specifies that, in order to assume a role, the user must name their
session after their own username. Otherwise, the AssumeRole command
will fail.

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<AccountNumber>:root"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringLike": {
"sts:RoleSessionName": "${aws:username}"
}
}
}
]
}

With this configuration, you can easily track and filter the actions
performed in each assumed role session—or catch anyone who fails to
provide a valid session name. For more examples of controlling session
names, see the related AWS blog post.

Key CloudTrail IAM policies in AWS are complex; they can potentially provide users with
audit logs permissions to access all resources within an AWS account. This means
to monitor that there is ample opportunity for security misconfigurations that will
inadvertently allow someone to manipulate your environment and give
themselves further access to your assets. By monitoring your audit logs,
you can get a fuller picture of user activity and the ways in which users
interact with your resources—including whether or not they’re authorized
to perform those interactions in the first place.
[Link] Best practices for monitoring AWS CloudTrail logs 13

Attackers often look for overly permissive IAM permissions or misconfi-

gurations across a wide variety of AWS resources, including:
– IAM users/roles
– EC2 instances
– S3 buckets
For example, an S3 bucket might have a policy attached that provides
Read access to all authenticated users, instead of just the authenticated
users within your account. If an attacker were to discover this vulnerability,
they could read all of the data saved within that bucket, potentially
exposing customer or business information.
Your CloudTrail logs contain a reliable record of user activity and, once
you know which logs to focus on, can provide you with all of the information
you need to monitor your environment. The following resource-based
types of logs are particularly important, as they are where the majority
of threats will originate:
– user accounts
– buckets
– networking components
We’ll take a look at some sample logs generated from these resource-based
operations below. When reading event logs, you should always pay attention
to the JSON attributes that can help you spot a possible attack or miscon-
figuration. These include the response of a call (i.e., responseElements ),
the API call that was made (i.e., eventName ), and any identifying informa-
tion, such as the user or role that called the command (i.e., various fields
under userIdentity ).

USER ACCOUNTS
One of the most common ways for an attacker to infiltrate your environment
is by using an exposed AWS Secret Access Key and enumerating the key’s
permissions. If the exposed key has extensive management permissions,
the attacker can proceed to give themselves further permissions while
disabling your security infrastructure. Monitoring your CloudTrail logs for
the following activity can help alert you to attackers as they inspect their
permissions and attempt to maintain persistence in your environment:
– Unauthorized Activity
Unauthorized user activity logs contain the following error message in
the responseElements :
[Link] Best practices for monitoring AWS CloudTrail logs 14

{
[...],
"errorCode": "[Link]",
"errorMessage": "You are not authorized to perform this operation.",
"requestParameters": {
"regionSet": {}
},
"responseElements": null,
"requestID": "0a857cb2-90c4-4f09-9624-1149fb27f8a1",
"eventID": "26fe99a5-8ed5-4923-9cf7-b6cdf96fa5f3",
"eventType": "AwsApiCall",
"recipientAccountId": "111111111111"
}

A single unauthorized activity log is not necessarily indicative of a threat.

For instance, the unauthorized action may have occurred due to a user not
having the permissions needed to view certain AWS console resources.
Or it could be the result of a service attempting to call on a resource for
which it does not have access.
However, if this is the first time an IAM user is receiving an authorization
error, it might be worth investigating what caused the error. It could be the
result of an attacker attempting to use the account or service to gain
further access to your resources. For instance, they might attempt to create
a new user or role as a backdoor into your environment, or expand the IAM
policy already associated with a user or role they’ve gained access to.
– AWS GuardDuty DetectorDeleted
In order to go undetected when performing unauthorized or malicious
actions, an attacker might attempt to disable the Amazon GuardDuty threat
detectors running in your AWS account. It’s always worth investigating
any instances of GuardDuty detector deletion.

BUCKETS
Attackers often target S3 buckets when attempting to breach your environ-
ment. As with user accounts, an attacker might gain access to a bucket’s
contents due to a security misconfiguration or human error. By monitoring
your CloudTrail logs, you can spot the following bucket enumeration and
modification attack techniques.
– AWS S3 Buckets Enumerated
– AWS S3 Bucket Policy Modified
If an attacker gets access to an EC2 instance, the first thing they might do is
enumerate all of the S3 buckets that they have access to from the relevant
instance profile, or attempt to change a bucket’s access policy altogether.
[Link] Best practices for monitoring AWS CloudTrail logs 15

As most automated resources already have direct access to all of the

buckets they need, a ListBuckets or PutBucketPolicy call is usually
worthy of investigation.
– AWS S3 Public Access Block Removed
Similarly, an attempt to remove a public access block attached to an
S3 bucket, is an event that should be investigated. This could be a legiti-
mate user trying to accomplish a task by removing a security control as a
debugging mechanism. Alternatively, it could be an attacker attempt-
ing to open the bucket to the public internet. We recommend investigating
DeleteAccountPublicAccessBlock event logs as soon as possible.

NETWORKING COMPONENTS
Attackers may also attempt to access your environment through a miscon-
figured network resource, such as a VPC, a route table, a network gateway,
a network access control list, or a security group. CloudTrail logs can help
you spot the following types of possible network attacks and take the
proper steps to resolve the breach.
– AWS VPC Created or Modified
– AWS Route Table Created or Modified
– AWS Network Gateway Created or Modified
– AWS Network Access Control List Created or Modified
– AWS Security Group Created or Modified
To check the posture of your networking resources and make sure that
they are securely configured, we recommend using Datadog’s Compliance
Monitoring tool, which scans your AWS environment for misconfigurations
in real-time.

Collect and The benefits of using Datadog as your log monitoring platform for your AWS
analyze infrastructure include:
CloudTrail logs – direct integrations with AWS CloudTrail, Amazon S3, AWS Kinesis Firehose,
with Datadog and Amazon Lambda that streamline the log export process
– automatic field parsing of all AWS CloudTrail logs streaming from your
AWS environment using log processing pipelines
– cost-effective collection and archiving of all CloudTrail logs using
Datadog’s Logging without Limits™
– expanded scope of log context for security and compliance analysis
Once you’ve set up the AWS integration for your services and have
CloudTrail logs streaming into Datadog, you can build custom dashboards
to get a high-level perspective on the health and security of your AWS
[Link] Best practices for monitoring AWS CloudTrail logs 16

environment. And using Datadog’s built-in Threat Detection Rules, you

can detect critical security and operational issues—including the ones we
discussed above—as they occur.

EXPORT YOUR CLOUDTRAIL LOGS TO DATADOG

Exporting CloudTrail logs from AWS to Datadog enables you to analyze
and more deeply contextualize the events recorded with other observability
data from your environment. A simple way of doing this is by using Amazon
Kinesis Data Firehose, a fully managed AWS service that automates the
delivery of your real-time, distributed streaming data to external data
storage and analysis repositories.
Using Kinesis Data Firehose for AWS data delivery comes with a number
of advantages, including near-real-time uploading, serverless data transfor-
mation options, and integrations with the full suite of AWS services. For
instructions on setting up Kinesis Data Firehose for use with Datadog, see
our blog post.

EXPLORE CLOUDTRAIL LOGS IN DATADOG

Once your audit logs are streaming into Datadog’s Log Explorer, you can
easily filter and search them to find the most important logs for your
particular use case. For instance, referring back to the key AWS audit logs
to monitor, you might want to look for events in which a user attempted
to create or change the permissions of a security group. To do so, you
would filter your logs to look for CreateSecurityGroup, AuthorizeSecurity
GroupIngress, or AuthorizeSecurityGroupEgress events.
[Link] Best practices for monitoring AWS CloudTrail logs 17

Beyond filtering your audit logs to find potential problems, you can use
them to build high-level Datadog dashboards with custom data
visualizations. This way, you can get a quick, top-down perspective of
your incoming logs without endlessly filtering through them.

DETECT SECURITY THREATS IN REAL-TIME

To help you catch security threats as they occur, Datadog Cloud SIEM
lets you apply strict Detection Rules to your entire event stream as it is
ingested. Detection Rules are available out of the box and match the attack
techniques enumerated by the MITRE ATT&CK® framework, which covers
many of the critical event types we’ve already discussed. Additionally, if
you’d like to evaluate events based on your environment’s specific needs,
you can create your own rules.
When an incoming event matches one of your Detection Rules, Datadog
creates a Security Signal that can be inspected in the Security Signals
explorer. Security Signals provide context around each trigger event, such
as the username and IP address that initiated the offending action, the
timeline of the event itself, and standardized guidelines for responding to
the threat.
[Link] Best practices for monitoring AWS CloudTrail logs 18

Start moni- In this post, we reviewed how to interpret AWS CloudTrail audit logs:
toring your we looked at how each event type works, outlined best practices for follow-
AWS CloudTrail ing users and roles across multiple logs, and highlighted the most
audit logs important audit logs to investigate. We also walked through how to import
your CloudTrail logs into Datadog using Amazon Kinesis Data Firehose,
as well some of the best ways to use Datadog to triage your logs and catch
security issues as they occur. For more information on monitoring your
AWS audit logs and securing your applications with Datadog, check out our
documentation. And if you’re not already using Datadog, get started now
with a 14-day free trial.
 Best practices for monitoring
GCP audit logs
Justin Massey Google Cloud Platform (GCP) is a suite of cloud computing services for
deploying, managing, and monitoring applications. A critical part of deploy-
Mallory Mooney ing reliable applications is securing your infrastructure. Google Cloud Audit
Logs record the who, where, and when for activity within your environment,
providing a breadcrumb trail that administrators can use to monitor access
and detect potential threats across your resources (e.g., storage buckets,
databases, service accounts, virtual machines). GCP collects audit logs
from all GCP services, so you can get more context around user and service
account activity for security analysis and identify possible vulnerabilities
that you should address before they become bigger issues.
In this guide, we’ll cover:
– the structure of GCP audit logs
– best practices for using audit logs to monitor GCP security
– how to export audit logs from GCP
– how Datadog can help you collect and monitor your GCP audit logs
Before we dive into audit logs, let’s first look at the Google Cloud resource
hierarchy, as it plays a role in interpreting your logs.
[Link] Best practices for monitoring GCP audit logs 20

A primer on A resource hierarchy organizes your Google Cloud resources by organiza-

the Google tion, project, and folder and defines which members have access to what
Cloud hierarchy resources. This hierarchy often mimics the structure of a company, where
certain resources, such as a Google Compute Engine (GCE) instance or
Google Cloud Storage (GCS) bucket for example, are owned by a specific
department within the company.

To grant access to your resources, you can create Cloud IAM policies.
A policy can include members of several different account types, includ-
ing Google accounts and service accounts. These policies propagate
down the hierarchy. In the example above, if you grant a Google group
the editor role for the Engineering folder, their Google account credentials
will automatically give them read and write access to all the resources
in both the Demo and Lab projects. Understanding the hierarchy of your
organization, as well as the roles of individual users and services, will
help you quickly sift through this data in your audit logs and pinpoint any
potential threats to your applications.

Understanding Google Cloud can emit three different types of audit logs for every
Google Cloud organization, folder, and project within your resource hierarchy:
Audit Logs – Admin Activity: entries for API calls or user administrative activity that
changes resource configurations
– System Event: entries for Google system administrative activity that
modifies resource configurations
– Data Access: entries for API calls that read resource configurations or
metadata, or user-level API calls that read or write resource data
[Link] Best practices for monitoring GCP audit logs 21

Most Google Cloud services emit each of these audit log types, enabling
you to view resource activity for different levels of your hierarchy. This
includes G Suite if you are sharing that data with Cloud Logging, so you
can also view the Admin Activity and Data Access audit logs that G Suite
writes at the organizational level.
You can access your logs using GCP console. After logging in, select
Logging then Log Viewer from the navigation menu.

It’s important to note that, while you can see project-level logs in the
con-sole, you can only view organization- and folder-level logs with the
Cloud Logging API. To view all of your audit logs in one place, you can ship
them to Datadog. We’ll cover how to configure Datadog to collect your
audit logs later.
Next, we’ll go through each type of log in more detail.
[Link] Best practices for monitoring GCP audit logs 22

ADMIN ACTIVITY AUDIT LOGS

Admin Activity audit logs record administrative changes to your resource
configuration. This could include events such as a user or service altering
the permissions of a Google Cloud Storage bucket or a user creating a new
service account, as seen in the sample entry below.

{
"protoPayload": {
"@type": "[Link]/[Link]",
"status": {},
"authenticationInfo": {
"principalEmail": "[Link]@[Link]",
"principalSubject": "user:[Link]@[Link]"
},
"requestMetadata": {},
"serviceName": "[Link]",
"methodName": "[Link]",
"authorizationInfo": [
{
"resource": "projects/-/serviceAccounts/123456789012345678901",
"permission": "[Link]",
"granted": true,
"resourceAttributes": {}
}
],
"resourceName": "projects/-/serviceAccounts/123456789012345678901",
"request": {
"@type": "[Link]/[Link].v1.
CreateServiceAccountKeyRequest",
"name": "projects/-/serviceAccounts/
ab6c214d4e4fgh1234567@[Link]",
"private_key_type": 2
},
"response": {},
"insertId": "vwqyhke39pfj",
"resource": {
"type": "service_account",
"labels": {}
},
"timestamp": "2020-05-18T15:09:50.746563735Z",
"severity": "NOTICE",
"logName": "projects/sample-project/logs/[Link].
com%2Factivity",
"receiveTimestamp": "2020-05-18T15:09:51.695275787Z"
}

Users require either the Cloud IAM Logging Viewer or Project Viewer role
to view Admin Activity logs. They are enabled by default and, although you
are not able to configure or disable them, you are not charged for them.
[Link] Best practices for monitoring GCP audit logs 23

SYSTEM EVENT AUDIT LOGS

Google Cloud services generate System Event audit logs when they make
modifications to your resource configurations. For example, you may see
the following audit log entry when GCE live migrates an instance to another
host.

{
"protoPayload": {
"@type": "[Link]/[Link]",
"authenticationInfo": {
"principalEmail": "system@[Link]"
},
"serviceName": "[Link]",
"methodName": "[Link]",
"resourceName": "projects/sample-project/zones/us-central1-a/
instances/gke-default-pool-123",
"request": {
"@type": "[Link]/[Link].
migrateOnHostMaintenance"
}
},
"resource": {
"type": "gce_instance",
"labels": {
"zone": "us-central1-a",
"instance_id": "123456789012345",
"project_id": "sample-project"
}
},
"timestamp": "2020-05-07T19:59:34.633Z",
"severity": "INFO",
"logName": "projects/sample-project/logs/[Link].
com%2Fsystem_event"
}

Like Admin Activity logs, System Event audit logs are enabled by default,
do not incur charges, and require either the Logging Viewer or Project
Viewer role.

DATA ACCESS AUDIT LOGS

Data Access audit logs consist of three sub-types:
– Admin read: reads of service metadata or configuration data
(e.g., listing buckets or nodes within a cluster)
– Data read: reads of data within a service (e.g., listing data within a bucket)
– Data write: writes of data to a service (e.g., writing data to a bucket)
[Link] Best practices for monitoring GCP audit logs 24

Due to the potentially large volume of Data Access logs your environment
can generate, they are not enabled by default. You need to explicitly enable
each sub-type for any services you want to monitor. A service account
retrieving a list of storage buckets, for example, may generate the following
ADMIN_READ Data Access log:

{
"protoPayload": {
"@type": "[Link]/[Link]",
"status": {},
"authenticationInfo": {
"principalEmail": "[Link]",
"serviceAccountKeyName": "//[Link]/projects/
sample-test/serviceAccounts/a1bcd2e3456k9f00934gh@test-bot.
[Link]/keys/e123e0b9dde1234b1d22a4cb123456d51ree"
},
"requestMetadata": {},
"serviceName": "[Link]",
"methodName": "[Link]",
"authorizationInfo": [
{
"permission": "[Link]",
"granted": true,
"resourceAttributes": {}
}
],
"resourceLocation": {},
"insertId": "14suazjdct0e",
"resource": {
"type": "gcs_bucket",
"labels": {
"project_id": "sample-project",
"location": "global",
"bucket_name": ""
}
},
"timestamp": "2020-05-20T13:03:05.396618238Z",
"severity": "INFO",
"logName": "projects/sample-project/logs/cloudaudit.
[Link]%2Fdata_access",
"receiveTimestamp": "2020-05-20T13:03:06.387630414Z"
}

Note that, unlike Admin Activity and System Event logs, GCP charges you
for Data Access logs. Users need either the Cloud IAM Logging Viewer,
Logging Private Logs Viewer, Project Viewer, or Project Owner role to view
these logs.
[Link] Best practices for monitoring GCP audit logs 25

INTERPRETING YOUR AUDIT LOGS

Audit logs include important data for monitoring GCP service activity, and
knowing how to interpret log entries can help you pinpoint potential gaps in
your cloud security policies. Let’s break down a sample access log—which
records a successful call by a service account to list storage buckets—to
show where you can find the most useful information.
All audit logs provide a payload ( protoPayload ) that includes authenti-
cation ( authenticationInfo ) and authorization ( authorizationInfo )
data of the user (or service account) as well as the call ( methodName )
that was made. In the snippet below, you can see that a test-bot service
account made a call to list a group of storage buckets ( [Link].
list ). Note also the @type field that identifies this as an audit log. All
audit logs, regardless of sub-type share this attribute. We’ll come back to
it when we look at shipping logs to an external service.

"protoPayload": {
"@type": "[Link]/[Link]",
"status": {},
"authenticationInfo": {
"principalEmail": "test-bot@example-project.
[Link]",
"serviceAccountKeyName": "..."
},
"requestMetadata": {},
"serviceName": "[Link]",
"methodName": "[Link]",
"authorizationInfo": [
{
"permission": "[Link]",
"granted": true,
"resourceAttributes": {}
}
],
"resourceLocation": {}
}

You can tell which requests are from a user versus a service account
by looking at the domain of the principalEmail . If the domain is
[Link] , the account is a service account. If the domain
is [Link] , it is a Google service performing admin activity. If it is
any other domain, it is a user making the request.
Calls to list storage buckets within a project are typically made by users,
not service accounts. If you see log entries that show service accounts
making these calls then you may need to troubleshoot further by monitor-
ing any other calls the account is making. Or, you can identify which team
within your organization owns the service to confirm that the calls were
necessary.
[Link] Best practices for monitoring GCP audit logs 26

The resource section provides more details about the specific resources
being queried or modified. The snippet below, for example, shows that
the test-bot service account made a call to list all Cloud Storage buckets
( gcs_bucket ) in the project sample-project .

"resource": {
"type": "gcs_bucket",
"labels": {
"project_id": "sample-project",
"location": "global",
"bucket_name": ""
}
}

You can look at the log entry’s logName for information including a suffix
identifying the log sub-type (e.g., Admin Activity, System Access, Data
Access) and where in the hierarchy the request was made. Here, we can
see that this is an Admin Activity log recording a request to a project,
sample-project:

"logName": "projects/sample-project/logs/cloudaudit.
[Link]%2Factivity"

Your GCP resources can produce a large volume of audit logs, making it
difficult to find the logs you need to detect unusual activity or troubleshoot
problems before they turn into security incidents. Next, we’ll highlight
some of the critical log events you should track to make sure your envi-
ronment is secure. Then, we’ll walk through shipping those logs to a
monitoring service like Datadog.
[Link] Best practices for monitoring GCP audit logs 27

Key GCP audit

logs to monitor

Cloud IAM policies are complex and can grant users and service accounts
access to resources at every level of your environment’s hierarchy.
Monitoring audit logs provides a better understanding of who is accessing
a resource, how they are doing it, and whether or not the access was
permitted.
Some common scenarios that lead to your GCP account being compromised
include:
– publicly accessible GCP resources, such
as storage buckets or compute instances
– misconfigured IAM permissions
– mishandled GCP credentials
Attackers often look for these types of vulnerabilities in order to gain
access to your environment. Once they have access, they can modify
GCP services, escalate privileges and create new accounts, and exfiltrate
sensitive data.
As an example of misconfigured permissions, a Cloud Storage IAM policy
for your storage buckets can include the value allAuthenticatedUsers
as a member of the role Storage Object Admin . This would grant all
authenticated users to GCP—not just authenticated users within your
account—the ability to create, delete, and read all objects within the
storage bucket.
GCP provides security guidelines that are mapped to frameworks such
as CIS Benchmarks, which offer baseline best practices for securing your
environment. Your audit logs complement these guidelines and provide
a detailed history of activity, ensuring that you can mitigate potential
threats to your environment. Next, we’ll look at the following key audit
logs you can monitor for your resources:
[Link] Best practices for monitoring GCP audit logs 28

– user and service accounts

– buckets
– Logging sinks and Pub/Subs
For each log type we cover, we will look at some JSON attribute data that
can help you pinpoint the origin of a possible attack. This includes the
status of a call (i.e., [Link] attribute), the
API call that was made (i.e., [Link] attribute),
and the email address for the account that made the API call (i.e., the
[Link] attribute).
We’ll also look at a few CIS Benchmark recommendations for securing
each resource.

USER AND SERVICE ACCOUNTS

Attempts to compromise your environment often start with an attacker
using an exposed API key and enumerating—or listing out— their permis-
sions within the GCP account. Monitoring your audit logs for the following
activity can help you identify attackers attempting to enumerate their
permissions or maintain persistence in your environment.
– Unauthorized User Activity
– Unauthorized Service Account Activity
Unauthorized activity logs will include, as part of their protoPayload ,
an entry showing that the call was denied:

"protoPayload": {
"status": {
"code": 7,
"message": "PERMISSION_DENIED"
}
}

A single denied call for a user or service account does not mean the account
is compromised. For example, it could be a user navigating around the GCP
console who is not permitted to access certain GCP services. Or, if you have
an internal service that is a part of a build pipeline requesting access each
time a build runs, it might be denied due to a misconfigured build job or
IAM permission.
However, if this is the first time a service account is receiving PERMISSION_
DENIED responses, it may be worth investigating why the account is being
denied access. This type of unauthorized activity could be an indicator that
an attacker has access to a compromised user or service account.
[Link] Best practices for monitoring GCP audit logs 29

Once they have access, attackers may then try to create new service
accounts or service account keys in order to create a backdoor into your
environment and maintain persistence.
– Service Account Created
– Service Account Key Created
Logs for new service accounts include an entry for a [Link].
[Link] API call in the methodName JSON attribute.
When a new service account key is created, you will see a log entry for a
[Link] call.
These logs do not always indicate a security threat, but you should
ensure that the calls were legitimate. For example, an attacker may create
a key on a service account with domain-wide delegation enabled. If a
service account with this enabled has the ability to access a G Suite
account, it could be used to leverage permissions within that account,
such as creating new G Suite administrators. Google does not log whether
or not the service account has been delegated domain-wide authority
when creating a Service Account Key; if you notice new key creation
activity, you should investigate further. To protect your service accounts,
CIS recommends ensuring that you do not grant them any admin, editor,
or owner roles.

BUCKETS
Storage buckets are often a component of a breach in public clouds. This
may be due to a misconfigured bucket or an attacker exploiting another
vulnerability to gain access to a storage bucket. Monitoring your Cloud
Audit Logs can detect the following bucket misconfigurations or attacker
techniques.
– Bucket Enumerated by a Service Account
To enumerate their permissions, an attacker will first attempt to use a
compromised service account to list storage buckets. For these types of
events, you will see a service account making a [Link] call.
Service accounts typically do not need to list storage buckets because they
are already configured to access the buckets they need, so if you see this
type of log you should investigate.
– Bucket Exposed to World
If a bucket is publicly accessible, you will see audit log entries that show
the [Link].
member JSON attribute is set to allUsers or allAuthenticatedUsers
and the action is set to ADD :
[Link] Best practices for monitoring GCP audit logs 30

"protoPayload": {
"serviceData": {
"policyDelta": {
"bindingDeltas": [
{
"action": "ADD",
"member": "allUsers",
"role": "roles/[Link]"
}
]
}
}
}

You should review your permission policies any time you see this level of
access to a GCP resource as it could be an indicator that the resource is
accessible to any users outside your network.

LOGGING SINKS AND PUB/SUBS

Your Cloud Audit Logs can also alert you to modifications to a sink or a
Pub/Sub topic or subscription, which is a technique attackers often use
to disable security tools. Changes to one of these sources could disrupt
the flow of logs to an external monitoring or analysis tool, reducing your
visibility into activity in your environment. You can detect this type of
attack by watching your audit logs for the following events:
– Logging Sink Modified
– Pub/Sub Subscriber Modified
– Pub/Sub Topic Deleted
Your Cloud Audit Logs capture these types of activity, enabling you to
triage potential threats to your environment and quickly troubleshoot
problems with the permissions of your users and service accounts.
To get even deeper insights into activity across your GCP environment,
you can easily export these logs to other monitoring and analysis tools.

Shipping your As part of its Operations Suite (formerly Stackdriver), GCP provides Cloud
audit logs Logging for querying and analyzing all of your logs. Cloud Logging uses
sinks for exporting logs to another source. All sinks include an export
destination and a logs query. You can use Cloud Logging sinks to export
your logs to a destination such as cloud storage, a BigQuery dataset,
[Link] Best practices for monitoring GCP audit logs 31

or a Publish Subscribe (Pub/Sub) topic. Cloud Logging compares a sink’s

query against incoming logs and forwards matching entries to the appro-
priate destination.
Pub/Subs provide fast, asynchronous messaging for your applications,
and are recommended for exporting audit logs to an external monitoring
service like Datadog for further analysis. When you add a Pub/Sub topic
as a sink’s export destination, Cloud Logging will send matched logs to
that topic. Pub/Sub topics are message queues that other services can
subscribe to with a Pub/Sub subscription in order to automatically receive
those messages.
With the ability to export audit logs, you can easily incorporate them
into your existing monitoring tools and get deeper insights into potential
security or compliance issues within your environments. Next, we’ll show
you how you can use Datadog to easily collect and analyze these logs.

Collect and Datadog provides turnkey integrations for GCP and G Suite that offer
analyze audit several benefits for collecting and monitoring your logs:
logs with – the G Suite integration simplifies the process for ingesting
Datadog authentication logs
– Datadog automatically parses all Google Cloud and G Suite audit logs
streaming from your GCP environments
– Datadog enriches Google Cloud and G Suite logs with more
contextual information for improved investigations
Once you enable the integrations, you can build custom dashboards
to get a high-level view of log activity and use Datadog’s built-in threat
detection rules to sift through key audit logs and automatically identify
critical security and compliance issues in your environments. First,
we’ll walk through setting up your GCP account to forward Cloud Audit
Logs to Datadog.

EXPORT CLOUD AUDIT LOGS TO DATADOG

As mentioned earlier, creating a sink that forwards logs to a Pub/Sub
topic is the recommended method for exporting Cloud Audit Logs to a
third-party service. The following query will collect audit logs from all
projects and organizations in your environment. Though in this example
we use the Google Cloud Console, you can also use the Cloud Logging API
and gcloud logging command-line tool to export logs.

protoPayload.@type:"[Link]/[Link]"
[Link] Best practices for monitoring GCP audit logs 32

To create a sink with this query, click on the “Create Sink” button in the
console’s Logs Viewer, provide a name for the sink, and select Pub/Sub
as the sink’s service. In the example configuration below, we’ve set the
export destination for the queried logs as a Pub/Sub topic called datadog-
exporter-topic . You can create a new topic by navigating to the Cloud
Pub Sub console of your GCP account.

To send organization and folder logs to Datadog, you will need to use the
Logging API and the gcloud command line utility. You can create sinks for
these logs with the following commands:

$ gcloud logging sinks create {sink-name} [Link]/

projects/{project-id}/topics/{topic-name} --organization=
{organization-id} --log-filter="protoPayload.@type=\"[Link].
com/[Link]\""
$ gcloud logging sinks create {sink-name} [Link]/
projects/{project-id}/topics/{topic-name} --folder={folder-id}
--log-filter="protoPayload.@type=\"[Link]/google.
[Link]\""

After running each of the commands, you will need to grant the newly
generated service account permission with the role of Pub/Sub Publisher
on the topic.
Ideally, it’s best to forward all audit logs to your monitoring service. This
ensures that you do not miss critical events that can negatively affect your
applications. However, it’s important to note that your environment can
generate a lot of Data Access audit logs, and Pub/Sub throughputs are
subject to quota limits. If you are running into those limits, you can split
[Link] Best practices for monitoring GCP audit logs 33

your logs over several topics to break up throughput. You can also create
an alert in Datadog to automatically notify you when you are close to
hitting the quota limits.
Once you create the sink, Cloud Logging will export all new Data Access,
Admin Activity, and System Activity logs to the selected topic. In order
to direct those logs to Datadog, create a new Pub/Sub subscription for
the datadog-exporter-topic topic and add Datadog as a subscriber,
as seen in the example below.

The delivery type uses the Push method to send logs to a Datadog endpoint.
Note that the endpoint requires a Datadog API key, which you can find in
your account’s settings.
With the GCP integration enabled and configured to receive data, you will
start seeing new Cloud Audit Logs in Datadog as GCP services generate
them. If you enabled the G Suite integration, you will also see G Suite logs
streaming alongside your other logs. Datadog’s log processing pipelines
automatically parse properties from your Google Cloud and G Suite audit
logs as tags, which you can use in the Log Explorer to sort and filter all
your logs and search for a subset of them that you need. For example, you
might want to look for accounts that made a call to list buckets within a
specific project.
[Link] Best practices for monitoring GCP audit logs 34

Keep in mind that the methodName attribute in GCP audit logs is

automatically mapped to the Event Name standard attribute in Datadog.

GET A HIGH-LEVEL VIEW OF LOG ACTIVITY

With Datadog’s GCP and G Suite integrations, you can immediately get
deeper insights into log activity and monitor application security and
compliance. You can build dashboards that provide an overview of the
logs streaming into Datadog, as seen in the example below.

You can create facets from attributes in your ingested logs and use them
to build visualizations, such as a list of the top five sources in your envi-
ronment of Admin Activity audit logs.
[Link] Best practices for monitoring GCP audit logs 35

IDENTIFY POTENTIAL SECURITY THREATS IN REAL TIME

Datadog’s Cloud SIEM product provides out-of-the-box detection rules
covering common techniques outlined by the MITRE ATT&CK® framework
for attacking applications running in GCP environments, many of which
can be identified in the key audit logs we looked at earlier. In addition,
these rules are also mapped to compliance frameworks, such as CIS, to
detect changes in your environment that may introduce misconfigurations
in your environment and leave your infrastructure vulnerable.
When audit logs trigger a rule, Datadog creates a security signal. You
can review and triage signals in the Security Signals explorer, where
they’re retained for 15 months. Each signal provides more details about
the activity as well as guidelines for responding.

In the example signal above, you can see that a service account enumer-
ated a storage bucket. The signal includes a sample of the Cloud Audit Log
that triggered the rule as well as the name of the service account that made
the calls.
Datadog’s GCP detection rules can also help you automatically monitor
changes to a Cloud Logging sink or Cloud Pub/Sub topic or subscription,
which could disrupt the flow of logs to your Datadog account.
[Link] Best practices for monitoring GCP audit logs 36

Start monitoring In this post, we looked at GCP audit logs and how they can provide
your Cloud Audit invaluable insight into activity in your environment so that you can
Logs quickly identify possible misconfigurations and threats. We then walked
through some best practices for collecting audit logs as well as how
Datadog can provide deeper visibility into GCP security. You can check
out our documentation for more information on getting started moni-
toring the security of your applications and GCP resources. Or, sign up
for a free trial to start monitoring your applications today.
 Best practices for monitoring
Microsoft Azure platform logs
Justin Massey Microsoft Azure provides a suite of cloud computing services that allow
organizations across every industry to deploy, manage, and monitor full-
Jonathan Epstein scale web applications. As you expand your Azure-based applications,
securing the full scope of your cloud resources becomes an increasingly
complex task. Azure platform logs record the who, what, when, and
where of all user-performed and service account activity within your
Azure environment. Collecting and analyzing Azure platform logs is vital
for monitoring the security of your Azure assets and identifying poten-
tially malicious activity before it can propagate across your system and
cause serious issues.
In this guide, we’ll take a look at:
– the types and structure of Azure platform logs
– key Azure platform logs to monitor
– how Datadog can help you collect and monitor your Azure logs
But, before we jump into Azure logs, it’s important to understand the
organizational hierarchy of your Azure resource directory, as it provides
the foundation for properly interpreting and acting on those logs.
[Link] Best practices for monitoring Microsoft Azure platform logs 38

Azure uses Azure Active Directory (Azure AD) to manage identity and
access management (IAM) across all resources within an organization,
referred to as a tenant. To help structure which users and services have
permission to access resources, Azure AD organizes a tenant’s cloud
resources in a directory structure across four levels: management groups,
subscriptions, resource groups, and resources. These levels act as a
hierarchy, so permissions configured for an entity at a higher level apply
to all sub-resources within that entity. At the very top of each tenant’s
Azure AD hierarchy is a root management group, which controls global
policies applied to the rest of the directory.
Azure AD creates a managed identity for every resource in your directory.
The identity provides access credentials based on that resource’s resource
group and subscription. When a user or resource attempts to perform an
action (such as reading from a storage bucket or creating a VM), it requests
an access token from Azure AD, which is used to authenticate the request.
The request is logged and contains detailed information about the action
performed in your environment.
[Link] Best practices for monitoring Microsoft Azure platform logs 39

Understanding Microsoft Azure generates three categories of platform logs that record
Azure platform different types of actions:
logs – Azure Active Directory reports detail changes made in Azure AD and
login activity.
– Activity logs record operations performed on an Azure resource
(i.e., operations coming from the the control plane), such as creating
a VM. Activity logs also record Service Health events.
– Resource logs capture operations performed within an Azure
resource (i.e., operations coming from the data plane), such as querying
a database or writing to a storage bucket.
In the following sections, we’ll look at each type of log and how you can
differentiate between them by looking at the category field that Azure
includes in all logs.

ACTIVE DIRECTORY REPORTS

Azure Active Directory logs detail all authentication activity related to
your AD users and their access to resources, and thus help you track key
information like which users are using resources, which administrators have
delegated permissions to users, and which users are potential security
concerns. You can view Azure AD logs directly within the AD portal.
There are two subtypes of Azure AD logs: activity logs and security reports.
Activity logs record the actions of users in your organization, such as
sign-ins, as well as all user credential changes. Activity logs recording
user sign-ins will have Sign-In as their Category field, while credential
changes will have Policy .

Azure Active Directory logs track actions like sign-ins, and can be viewed in the Active Directory
portal.

Security reports record any instances of unusual (and potentially malicious)

user activity, such as multiple failed sign-ins or access from a new country.
Similarly, these logs will also have Sign-In or Audi logs as their category
type respectively.
[Link] Best practices for monitoring Microsoft Azure platform logs 40

ACTIVITY LOGS
Azure activity logs (not to be confused with the AD activity log subtype)
record either creates and changes (i.e., PUT , POST , and DELETE
operations) performed on the resources within your Azure subscriptions,
such as starting a virtual machine or editing the configuration of an
Azure Pipeline. Each activity log contains key information on the parti-
culars of the event, including which user (if any) performed the operation,
which resource the operation was attempted on, when the operation
occurred, and whether it was successful. There are multiple categories
of activity logs, but the majority of actions on your subscription-level
assets will be related to configuration changes, so you likely will most
frequently see ‘Administrative’ or ‘Policy’ values in the Category field.
In order to store and access your activity logs, you need to create a
workspace in Log Analytics, Azure’s log management tool. Once you’ve
done this, you can access your logs in the Azure portal using the side
navigation bar, or with the Azure CLI.
The abridged sample below shows an Administrative type activity
log in which Caller: [Link]@[Link]
has successfully updated a network security group ( "authorization.
action": "[Link]/networkSecurityGroups/write" ).
The log’s [Link] field also tells us which subscription
and resource group the network security group is a part of, which can
help us discern which access permissions allowed the user to make
the configuration changes.
[Link] Best practices for monitoring Microsoft Azure platform logs 41

{
"authorization": {
"action": "[Link]/networkSecurityGroups/write",
"scope": "/subscriptions/12345678-1234-1234-1234-123456789101/
resourcegroups/stanfield/providers/[Link]/
networkSecurityGroups/Jumpbox-nsg"
},
"caller": "[Link]@[Link]",
"channels": "Operation",
[...]
"category": {
"value": "Administrative",
"localizedValue": "Administrative"
},
"eventTimestamp": "2021-02-25T14:14:37.4666455Z",
"id": "/subscriptions/12345678-1234-1234-1234-123456789101/
resourcegroups/stanfield/providers/[Link]/
networkSecurityGroups/Jumpbox-nsg/events/
306b539b-397d-4bd1-9364-61f1cb4bbab7/ticks/637498592774666455",
"level": "Informational",
"operationId": "cf187bda-b64d-4221-bae6-d3d71b067fac",
"operationName": {
"value": "[Link]/networkSecurityGroups/write",
"localizedValue": "Create or Update Network Security Group"
},
"resourceGroupName": "stanfield",
"resourceProviderName": {
"value": "[Link]",
"localizedValue": "[Link]"
},
"resourceType": {
"value": "[Link]/networkSecurityGroups",
"localizedValue": "[Link]/networkSecurityGroups"
},
"resourceId": "/subscriptions/12345678-1234-1234-1234-123456789101/
resourcegroups/stanfield/providers/[Link]/
networkSecurityGroups/Jumpbox-nsg",
"status": {
"value": "Succeeded",
"localizedValue": "Succeeded"
},
"subStatus": {
"value": "",
"localizedValue": ""
},
"submissionTimestamp": "2021-02-25T14:15:38.1594455Z",
"subscriptionId": "12345678-1234-1234-1234-123456789101",
"tenantId": "98765432-9876-9876-9876-987654321098",
"properties": {
"eventCategory": "Administrative",
"entity": "/subscriptions/12345678-1234-1234-1234-123456789101/
resourcegroups/stanfield/providers/[Link]/
networkSecurityGroups/Jumpbox-nsg",
"message": "[Link]/networkSecurityGroups/write",
"hierarchy": "98765432-9876-9876-9876-987654321098/
12345678-1234-1234-1234-123456789101"
},
"relatedEvents": []
}
[Link] Best practices for monitoring Microsoft Azure platform logs 42

RESOURCE LOGS
Resource logs detail all of the actions that occur within an existing Azure
resource, such as reads and writes to a vault in Azure Key Vault, or to a
database in Azure SQL Database. Like activity logs, resource logs each
contain a schema of standardized fields that provide key information such
as the ID of the resource in which the request was made (as well as the
IDs of the subscription and resource group to which the resource belongs),
the timestamp of the request, and whether or not the request succeeded.
Like activity logs, you need to create a Log Analytics workspace In order
to store and access your resource logs.
Because there are so many types of resource logs, it’s best to assume
that any log with a Category type not mentioned in the above Azure AD
and activity log sections is a resource log. Resource log categories depend
on the type of resource being modified and are often descriptive of the
“scope” of the actions taken, such as Execution or Request . For a full
list of resource log categories, see Azure’s documentation.
It’s important to note that, unlike Azure AD and activity logs, Azure
services do not emit resource logs by default due to their volume. In order
to enable resource log collection, you need to create a diagnostic setting
for each resource and designate which Log Analytics workspace will
receive its logs. We’ll look at this process later in this post.
The following abridged sample resource log shows that caller: john.
doe@[Link] successfully added a new key
( OperationNameValue: [Link]/vaults/write ) to an
instance of Azure Key Vault.
[Link] Best practices for monitoring Microsoft Azure platform logs 43

{
"authorization": {
"action": "[Link]/vaults/write",
"scope": "/subscriptions/12345678-1234-1234-1234-123456789101/
resourcegroups/stanfield/providers/[Link]/vaults/sekret"
},
"caller": "[Link]@[Link]",
"channels": "Operation",
[...]
"description": "",
"eventDataId": "8912ad72-5a13-4b09-8d0f-d4d62319be5d",
"eventName": {
"value": "EndRequest",
"localizedValue": "End request"
},
"category": {
"value": "Administrative",
"localizedValue": "Administrative"
},
"eventTimestamp": "2021-02-26T18:56:41.6480881Z",
"id": "/subscriptions/12345678-1234-1234-1234-123456789101/
resourcegroups/stanfield/providers/[Link]/vaults/sekret/
events/8912ad72-5a13-4b09-8d0f-d4d62319be5d/ticks/637499626016480881",
"level": "Informational",
"operationId": "5fb0b823-f5d4-4278-98b3-25d84a06a87f",
"operationName": {
"value": "[Link]/vaults/write",
"localizedValue": "[Link]/vaults/write"
},
"resourceGroupName": "stanfield",
"resourceProviderName": {
"value": "[Link]",
"localizedValue": "[Link]"
},
"resourceType": {
"value": "[Link]/vaults",
"localizedValue": "[Link]/vaults"
},
"resourceId": "/subscriptions/12345678-1234-1234-1234-123456789101/
resourcegroups/stanfield/providers/[Link]/vaults/sekret",
"status": {
"value": "Accepted",
"localizedValue": "Accepted"
},
"subStatus": {
"value": "OK",
"localizedValue": "OK (HTTP Status Code: 200)"
},
"submissionTimestamp": "2021-02-26T18:57:36.1579484Z",
"subscriptionId": "12345678-1234-1234-1234-123456789101",
"tenantId": "98765432-9876-9876-9876-987654321098",
"properties": {
"statusCode": "OK",
"serviceRequestId": "6f1140e7-57b3-4002-823c-deb47b0615e7",
"eventCategory": "Administrative",
"entity": "/subscriptions/12345678-1234-1234-1234-123456789101/
resourcegroups/stanfield/providers/[Link]/aults/sekret",
"message": "[Link]/vaults/write",
"hierarchy": "98765432-9876-9876-9876-987654321098/
12345678-1234-1234-1234-123456789101"
},
"relatedEvents": []
}
[Link] Best practices for monitoring Microsoft Azure platform logs 44

INTERPRETING YOUR AZURE PLATFORM LOGS

While each type of Azure log has unique fields, all of them share some
common information that is invaluable for helping you track the actions
occurring in your environment. First a foremost, the caller field is
universal and gives you the identity of the user or service that performed
the logged action. Similarly, as discussed in the sections above, each
log has a category field that helps you determine its log type: AD logs
have Sign-in , Policy , or Audit logs as their category field, activity
logs will most likely have Administrative or Policy as their category
field, and resource logs will have anything else. Again, there are some
exceptions, but this is a useful rule of thumb for interpreting most of your
Azure logs.
Azure activity and resource logs also include information on the resource
that was interacted with, including itsassociated ResourceGroup ,
SubscriptionId , and ResourceId . You can use these values to discern
where the permission configurations attached to a resource are coming
from, and thus how a user or service was able to interact with them based
on their own permission configurations. This way, you can spot users that
have overreaching permissions.
Finally, each log contains the name of the operation performed
( OperationName ), the status of the operation ( ActivityStatus ), the
time at which the request was made ( TimeGenerated ), and a unique
OperationId. You can use the OperationId to track the operation later,
or include the operation’s details in later API calls.
In the following sections, we’ll look at some of the most important Azure
platform logs of each type, and discuss how monitoring them can help you
keep your environment as secure as possible.

Key Azure Because Azure generates platform logs for nearly every action that occurs
platform logs across your directories, it can be difficult to sift through the massive
to monitor quantity of logs and find the ones that are useful for detecting threats.
In this section, we’ll look at some of the most important platform logs
and consider the best approaches to remediating the threats they may
represent. In particular, we’ll cover two main types of logs that can alert
you to different types of potential vulnerabilities:
– authentication logs
– resource-based logs
[Link] Best practices for monitoring Microsoft Azure platform logs 45

AUTHENTICATION LOGS
Your Azure Active Directory and activity logs provide a record of user
activity, including all successful and unsuccessful login events. These
logs are invaluable for detecting suspicious login activity. One example
of this is a brute force attack, in which an attacker repeatedly attempts
to guess a user’s login credentials. When Azure records a failed login
attempt, the resulting Azure AD log will have a category of SignInLog .
If a user’s login attempts fail ( outcome: failure ) multiple times in
a row, you should investigate the rest of the logs' details.
Of course, multiple login attempts are not always indicative of a brute
force attack; the user may have legitimately forgotten their credentials.
But it’s worth determining if the user was compromised and potentially
rotating their credentials. In the following logs, we can see that a user
failed multiple single-factor authentication attempts in a row; however,
because they then successfully logged in with multi-factor authentica-
tion, we know that this was not the work of an attacker.

Another sign of potential nefarious login activity is when a user

(or multiple users) fails to successfully complete multi-factor authen-
tication when logging in. Azure AD logs indicating this always
have an authenticationRequirement: multiFactorAuthentication
field with outcome: failed as a subfield. Again, such logs are not always
indicative of a threat, but it’s worth setting an alert to notify you
whenever they’re produced.

RESOURCE-BASED LOGS
When attempting to gain persistence in your Azure environment,
an attacker will often begin by trying to find instances of the following
types of resources that have overly permissive access policies
attached to them:
– Azure roles including user, Active Directory administrator,
and classic subscription administrator
– Virtual Machines and other compute resources
– Azure Storage objects, such as blobs, files, disks, etc.
– Network security groups
Once an attacker finds a misconfigured resource, they can then read or
modify its contents, exfiltrate its sensitive data, or escalate its access
policies to allow for further exploitation. The most vulnerable resources
[Link] Best practices for monitoring Microsoft Azure platform logs 46

are those that are accessible from the public internet, so you may consider
enabling resource logging on them and closely monitoring the logs that
they emit.
For instance, you might have a network security group that’s open to
inbound traffic from all IP addresses. If the group is mistakenly associated
with a network interface that’s meant for private communications, an
attacker could exploit the misconfiguration and gain further access to your
network. In this case, it’s important to look out for resource logs that record
action: action: [Link]/networkSecurityGroups/write
and sourceAddressPrefix values of [Link] or * , such as the sample
log below.
Similarly, it’s worth inspecting activity logs that detail instances of creating
or modifying security groups or security group rules, as these logs may
represent an attacker delegating overreaching permissions to themselves.
In the following log snippet, user [Link]@[Link]
has initiated an update to a security group ( "operationName": { "value":
"[Link]/networkSecurityGroups/securityRules/write",
"localizedValue": "Create or Update Security Rule" } ) called
Jumpbox-nsg .
Because Azure emits such a large volume of logs, it can be difficult to get a
big-picture perspective of the activity in your environment. For this reason,
it’s best to use a third-party log management solution. In the following
section, we’ll look at how to export Azure logs, and how Datadog can help
you get the most context out of your logging inventory.
[Link] Best practices for monitoring Microsoft Azure platform logs 47

{
"authorization": {
"action": "[Link]/networkSecurityGroups/
securityRules/write",
"scope": "/subscriptions/12345678-1234-1234-1234-123456789101/
resourceGroups/stanfield/providers/[Link]/
networkSecurityGroups/Jumpbox-nsg/securityRules/Port_8080"
},
"caller": "[Link]@[Link]",
"channels": "Operation",
[...]
"description": "",
"eventDataId": "7495ef59-0d7a-41a5-80e2-e033c0356b2a",
"eventName": {
"value": "BeginRequest",
"localizedValue": "BeginRequest"
},
"category": {
"value": "Administrative",
"localizedValue": "Administrative"
},
"eventTimestamp": "2021-02-25T20:36:51.5686671Z",
"id": "/subscriptions/12345678-1234-1234-1234-123456789101/
resourceGroups/stanfield/providers/[Link]/
networkSecurityGroups/Jumpbox-nsg/securityRules/Port_8080/
events/7495ef59-0d7a-41a5-80e2-e033c0356b2a/ticks/637498822115686671",
"level": "Informational",
"operationId": "405fe015-c0e1-41af-8b7d-bf92076d6650",
"operationName": {
"value": "[Link]/networkSecurityGroups/
securityRules/write",
"localizedValue": "Create or Update Security Rule"
},
"resourceGroupName": "stanfield",
"resourceProviderName": {
"value": "[Link]",
"localizedValue": ""[Link]"
},
"resourceType": {
"value": "[Link]/networkSecurityGroups/securityRules",
"localizedValue": "[Link]/networkSecurityGroups/
securityRules"
},
"resourceId": "/subscriptions/12345678-1234-1234-1234-123456789101/
resourceGroups/stanfield/providers/[Link]/
networkSecurityGroups/Jumpbox-nsg/securityRules/Port_8080",
"status": {
"value": "Started",
"localizedValue": "Started"
},
"subStatus": {
"value": "",
"localizedValue": "OK"
},
"submissionTimestamp": "2021-02-25T20:37:51.1436369Z",
"subscriptionId": "12345678-1234-1234-1234-123456789101",
"tenantId": "98765432-9876-9876-9876-987654321098",
"properties": {
"requestbody": "{\"properties\":{\"protocol\":\"*\",\"sourcePortRanges\":null,\
"sourcePortRange\":\"*\",\"sourceAddressPrefixes\":null,\"sourceAddressPrefix
\":\"*\",\"destinationPortRanges\":null,\"destinationPortRange\":\"8080\",\
"destinationAddressPrefixes\":null,\"destinationAddressPrefix\":\"*\",\
"access\":\"Allow\",\"priority\":310,\"direction\":\"Inbound\",\"sourceApplica
tionSecurityGroups\":[],\"destinationApplicationSecurityGroups\":[]},\"id\":\"
/subscriptions/12345678-1234-1234-1234-123456789101/resourceGroups/stanfield/
providers/[Link]/networkSecurityGroups/Jumpbox-nsg/securityRules/
Port_8080\",\"name\":\"Port_8080\"}",
"eventCategory": "6f1140e7-57b3-4002-823c-deb47b0615e7",
"eventCategory": "Administrative",
"entity": "/subscriptions/12345678-1234-1234-1234-123456789101/resourceGroups/
stanfield/providers/[Link]/networkSecurityGroups/Jumpbox-nsg/
securityRules/Port_8080",
"message": "[Link]/networkSecurityGroups/securityRules/write",
"hierarchy": "98765432-9876-9876-9876-987654321098/12345678-1234-1234-1234-
123456789101"
},
"relatedEvents": []
}
[Link] Best practices for monitoring Microsoft Azure platform logs 48

Shipping your While there are multiple ways to ship your platform logs to third-party
Azure platform applications, Azure recommends using Event Hubs, which are distributed
logs data streaming pipelines that can handle the large volume of platform
logs generated by your Azure environment. Event Hubs let you to ship
your logs in near real-time, and using Datadog’s Event Hub integration,
you can easily keep track of the health of your Hubs and make sure that
you’re receiving your logging data as instantaneously as possible.
In the next section, we’ll look at the benefits of using Datadog to manage
and analyze your Azure platform logs, including multiple features that
help you monitor these critical logs and use them to alert you to these
potential threats. We’ll also go over how to set up an Event Hub and
configure it to send your logs to Datadog.

Collect and There are a number of benefits that come with using Datadog as your log
analyze Azure monitoring platform, including:
platform logs – Datadog’s Azure integration that collects data from all of your Azure
with Datadog services, including Virtual Machine, Load Balancer, Event Hub,
and SQL Database
– automatic parsing and enrichment of your Azure platform logs via
Datadog’s log processing pipeline
– cost-effective collection and archiving of all you Azure platform logs
using Logging without Limits™
– built-in security and compliance analysis across all of your logs
Once you’re collecting your Azure platform logs with Datadog, you
can build custom dashboards that visualize log data for a full-picture
perspective of your Azure environment’s activity. Datadog’s built-in
Threat Detection Rules automatically watch your logs for potential
malicious activity, including the key events we’ve discussed, so you
are notified as soon as security and compliance issues occur.

EXPORT YOUR AZURE PLATFORM LOGS TO DATADOG

After you’ve enabled Datadog’s Azure integration, the next step is to
create an Azure Event Hub instance to start streaming your platform logs
to Datadog. Finally, configure your resources' diagnostic setting to send
their logs to the Event Hub instance so that they will be forwarded on to
Datadog.
To simplify this set-up process, Datadog provides an automated script that
creates and configures all of the necessary resources in your Azure account
and links them to a new Event Hub instance that forwards to Datadog.
[Link] Best practices for monitoring Microsoft Azure platform logs 49

To use this script, open Cloud Shell in Azure and run the following two
commands. The first command downloads the resource automation script:

(New-Object [Link]).DownloadFile("[Link]
[Link]/DataDog/datadog-serverless-functions/master/azure/eventhub_
log_forwarder/activity_logs_deploy.ps1", "activity_logs_deploy.ps1")

Use the second command to run that script, substituting <api_key>

with your Datadog API key and <subscription_id> with your Azure
Subscription ID:

./activity_logs_deploy.ps1 -ApiKey <api_key>

-SubscriptionId <subscription_id>

Upon successful configuration, you’ll start to see your Azure platform logs
appear in real time in Datadog’s Log Explorer. Datadog’s log processing
pipeline automatically parses metadata from your Azure platform logs
and uses it to create log attributes, which you can use as tags to quickly
filter, sort, and group your logs by key facets like service, action, user,
subscription, and resource group.

You can sort through all of your ingested Azure platform logs by tag in the Datadog Log Explorer.
[Link] Best practices for monitoring Microsoft Azure platform logs 50

GET A FULL-PICTURE PERSPECTIVE ON LOG ACTIVITY

In order to get a high-level overview of your Azure platform logs, you
can build dashboards that visualize your most important log events.
By filtering on the attributes mentioned above, you can create custom
data visualizations, such as a list of Azure users that are most frequently
failing login attempts.

DETECT SECURITY THREATS IN REAL TIME

To help you catch security threats to your Azure environment as quickly
as possible, Datadog Cloud SIEM lets you apply detection rules to your
Azure platform logs as they are ingested. Out-of-the-box Azure detection
rules match the attack techniques standardized in the MITRE ATT&CK®
framework, which covers some of the important log events we looked at
above. You can also create custom rules to look for malicious behavior that
is specific to your environment.
Whenever an incoming Azure platform log matches the logic of a rule,
Datadog creates a Security Signal. You can browse all of the signals created
against your Azure logs in the Security Signals explorer, where each signal
contains information on the incident, including a timeline of the threat’s
detection and a brief recommendation on how to best remediate it. This
makes it easy to triage threats and focus on the most urgent ones.

Security rules trigger alerts based on the custom logic you provide. In this example, an alert is
triggered whenever an AD user attempts to login without multi-factor authentication.
[Link] Best practices for monitoring Microsoft Azure platform logs 51

Start monitoring In this post, we reviewed how Azure platform logs are organized, how
your Azure the hierarchy of Azure environments affects their contents, and how to
platform logs analyze each type of platform log to get visibility into activity across
with Datadog your Azure services. We also looked at some of the most important logs
to watch in order to secure your environment, as well as how Datadog
can help your organization triage your logs and catch such security issues
as they occur. For more information on the benefits of monitoring your
Azure platform logs with Datadog, check out our Azure documentation.
If you’re not already using Datadog, get started now with a 14-day free trial.
 Best practices for monitoring
Kubernetes security via audit logs
Mallory Mooney Kubernetes continues to be a popular platform for deploying containerized
applications, but securing Kubernetes environments as you scale up is
Justin Massey challenging. Each new container increases your application’s attack surface,
or the number of potential entry points for unauthorized access. Without
complete visibility into every managed container and application request,
you can easily overlook gaps in your application’s security as well as
malicious activity.
Kubernetes audit logs provide a complete record of activity (e.g., the who,
where, when, and how) in your Kubernetes control plane. Monitoring your
audit logs can be invaluable in helping you detect and mitigate miscon-
figurations or abuse of Kubernetes resources before confidential data is
compromised. However, Kubernetes components and services can generate
millions of log events per day, so knowing which logs to focus on is difficult.
In this post, we’ll cover:
– How to interpret Kubernetes audit logs
– Some key audit logs for monitoring Kubernetes cluster security
– How Datadog can help you monitor your audit logs and alert you
to suspicious activity in your environments
First, we’ll briefly look at audit log policies and how Kubernetes uses them
to generate audit logs.
[Link] Best practices for monitoring Kubernetes security via audit logs 53

A primer on When a request comes in to the Kubernetes API server, it can create one
generating of several different audit events such as creating a new pod or service
audit logs account. The server filters these events through an audit policy. An audit
policy is a set of rules that specifies which audit events should be recorded
and where they should be sent, for example to either a JSON log file or an
external API backend for storage.

The scope of cluster activity that Kubernetes will capture with audit logs
depends on your audit policy’s configuration and the levels you set for each
of your resources, so it’s important that the policy collects the data you
need for monitoring Kubernetes security. Otherwise, you may not be able to
easily surface legitimate threats to your applications. For instance, a policy
that doesn’t collect data about pod activity makes it more difficult to know
when an unauthorized account creates new pods with privileged containers.
On the other hand, policies that collect audit events from endpoints not
directly related to cluster activity, such as /healthz or /version , create
noise.

Interpreting In addition to capturing the right audit logs, knowing how to interpret log
your Kubernetes entries is necessary for pinpointing flaws in your environment’s security
API server settings. Let’s break down a sample audit log—which creates a new pod—
audit logs to show you what information is most useful for Kubernetes security
monitoring.
The requestURI and verb attributes show the request path (i.e., which
API endpoint the request targets) and action (e.g., list, create, watch) that
are used to make the request. The API server maps the action to a cor-
responding HTTP method. In the example entry above, you can see that a
user made a POST request to create a new pod in the “default” namespace.
The request path can also capture commands that users run inside a
container via the kubectl exec command, giving you more context for
how a user or service interacts with your application. For example, you
can detect when a user account is passing an ls command to a pod to list
and explore its directories.
[Link] Best practices for monitoring Kubernetes security via audit logs 54

[Link]

ini{
"kind":"Event",
"apiVersion":"[Link]/v1",
"metadata":{ "creationTimestamp":"2020-10-21T21:47:07Z" },
"level":"RequestResponse",
"timestamp":"2020-10-21T21:47:07Z",
"auditID":"20ac14d3-1214-42b8-af3c-31454f6d7dfb",
"stage":"ResponseStarted",
"requestURI":"/api/v1/namespaces/default/pods",
"verb":"create",
"user": {
"username":"[Link]@[Link]",
"groups":[ "system:authenticated" ]
},
"sourceIPs":[ "[Link]" ],
"objectRef": {
"resource":"pods",
"namespace":"default",
"apiVersion":"v1"
},
"requestReceivedTimestamp":"2020-10-21T21:47:07.603214Z",
"stageTimestamp":"2020-10-21T21:47:07.603214Z",
"annotations": {
"[Link]/decision": "allow",
"[Link]/reason": "RBAC: allowed by RoleBinding
'demo/test-account' of Role 'cluster-admin' to ServiceAccount
'demo/test-account'"
}
}

The user , sourceIPs , and [Link]/reason attributes

provide more details about who made the request, their permission groups
within your organization, and the specific RBAC permissions that allowed
the request (e.g., RBAC Role, ClusterRoles). This data can help you audit a
user’s level of access or determine if an account was compromised if it is
making requests from an unknown IP address. It can also help you identify
misconfigurations that could leave your application vulnerable to attacks.
Next, we’ll highlight some of the key types of Kubernetes audit logs
you should monitor to make sure that you have the right security policies
in place and can quickly identify security gaps in your containerized
environment.
[Link] Best practices for monitoring Kubernetes security via audit logs 55

Key Kubernetes There are several techniques an attacker can use to access and modify
audit logs Kubernetes resources, accounts, and services. Many of these techniques
to monitor focus on exposing simple misconfigurations in your Kubernetes environ-
ment or RBAC policies. We’ll look at some important audit logs you can
monitor to easily surface suspicious activity in your cluster and detect a
few of the techniques used in the following areas:
– access to your Kubernetes environment
– changes to Kubernetes resources
– user and service account activity

ACCESS TO YOUR KUBERNETES ENVIRONMENT

The Kubernetes API server manages all requests from users and other
Kubernetes resources in your environment and is one of the first components
an attacker may try to access. With access to the API server, an attacker
will attempt to find and control other resources in your environment, such
as service accounts and pods. Monitoring your audit logs for the following
activity can help you identify vulnerabilities as well as detect an attack
before it escalates to other parts of your Kubernetes environment.
– Anonymous requests allowed
Kubernetes’s default settings allow anonymous requests to the API server.
Additionally, by default each node’s kubelet also allows anonymous
requests. CIS Benchmarks recommends disabling these settings because,
for example, they potentially give attackers full access to the kubelet API
to run commands in a pod, escalate privileges, and more.
[Link] Best practices for monitoring Kubernetes security via audit logs 56

If your API server and kubelets allow anonymous access, the username
and groups attributes in your audit logs will show system:anonymous
and system:unauthenticated , respectively.

[Link]

"user": {
"username":"system:anonymous",
"groups":["system:unauthenticated"]
},
"sourceIPs":[ "[Link]" ],

If you see these types of calls, you should troubleshoot further and
determine if you need to disable this setting for your environment.
To mitigate this, you can disable anonymous access and review your
role-based access control (RBAC) policies to ensure they are configured
correctly.
It’s important to note that managed Kubernetes services (e.g., Google
Kubernetes Engine, Amazon Elastic Kubernetes Service) generally disable
anonymous access, but it is still an important configuration to check.
A sudden increase in anonymous requests—even denied anonymous
requests—could mean that an attacker is probing the kubelet and API
server to discover what kinds of service data they can retrieve.
– Arbitrary execution in pod
– Attempt to get secrets from a cluster,
namespace, or a pod in a namespace
Once attackers know they have access to the kubelet and API server, they
will often start running arbitrary commands in a pod—such as launching
an interactive shell in a pod—or listing secrets via kubectl to see if they
can broaden their access. If you see an unauthorized IP address (e.g.,
in the sourceIPs , username , and group audit log attributes) running
commands in a pod such as listing a pod’s directories (e.g., in the
requestURI and verb attributes), it could be an indicator that an attacker
already has access to your environment and is finding ways to create,
modify, or control Kubernetes resources, which we’ll look at in the next
section.
[Link] Best practices for monitoring Kubernetes security via audit logs 57

CHANGES TO KUBERNETES RESOURCES

Changes to your environment are not always indicative of malicious activity,

but they are still important to monitor, especially if that activity is coming
from an unusual IP address or service account. The following activity is
common when attackers are trying to gain more control over your clusters.
– Pod created in a preconfigured namespace
– Pod created with a sensitive mount
– Pod attached to the host network
Attackers will often look for a service account (or create a new account)
that has the ability to attach pods to the host network or create new pods
in a preconfigured namespace, such as kube-system or kube-public .
This allows them to communicate with other pods in the namespace, create
pods with privileged containers, mount hostPath volumes on a node for
unrestricted access to the host’s file system, or deploy a DaemonSet to
easily control every node in a cluster.
If you notice the same service account or IP address creating new pods
or host mounts on a node, you may need to scale back that account’s
permissions or update your admission controllers—such as a Pod Security
Policy—to limit what pod features can be used. Privileged containers are
particularly valuable to attackers because they provide unfettered access
to other malicious activity, such as privilege escalation.
[Link] Best practices for monitoring Kubernetes security via audit logs 58

ACCOUNT ACTIVITY

Privilege escalation is another technique attackers use to gain control

over Kubernetes resources. Tactics often include attempts to create new
accounts (or modify existing ones) with elevated privileges for full access
to a resource. Monitoring the following activity can help you identify efforts
to modify an account’s permissions or create new accounts with admin
privileges.
– A cluster-admin role attached to an account
The cluster-admin role in RBAC policies gives accounts super-user
access to any resource. As a general rule, accounts should only have the
permissions they need to perform their specific tasks, so most accounts
do not need this level of access. You should follow up to ensure the role
is needed for the account in question if you see audit logs for any service
account that indicate it’s using the cluster-admin role.
You should also monitor for attempts to create a ClusterRoleBinding
that attaches an account to the cluster-admin role. An account with
this role binding has full control over every resource in the cluster and
in all namespaces. If you see this activity in your audit logs, you should
investigate further to ensure that the account isn’t compromised. You can
find all of this information in your audit logs' [Link]/
reason attribute.
– Service account created in a preconfigured namespace
Logs for new service accounts do not always indicate a security threat,
but you should investigate newly created accounts to ensure they are
legitimate, especially if they have a role that allows write privileges or
super-user access. As with your other audit logs, you can find more
[Link] Best practices for monitoring Kubernetes security via audit logs 59

information about the service account in the requestURI attribute, and

look at the user and sourceIPs attributes to determine if the activity is
coming from a known source.

Monitor Datadog offers a built-in Kubernetes audit log integration, so you can
Kubernetes easily track environment activity in real time. To start collecting your
security audit logs, you will need to deploy the Datadog Agent to your Kubernetes
with Datadog environment, then enable log collection.
Once you’re collecting Kubernetes audit logs with Datadog, you can
create facets from attributes in your ingested logs and use them to build
visualizations to explore and focus on the most important log data—check
out some real life examples in Datadog’s KubeCon North America 2019 talk,
Making the Most Out of Kubernetes Audit Logs. And Datadog’s out-of-
the-box Threat Detection Rules automatically monitor your audit logs and
identify critical security and compliance issues in your environments.

IDENTIFY POTENTIAL SECURITY THREATS IN REAL TIME

Datadog’s Detection Rules cover common techniques attackers use to
access Kubernetes environments, many of which can be identified in the
audit logs we looked at earlier. Datadog also provides rules mapped to
compliance frameworks, such as CIS, to detect changes in your Kubernetes
environment that may leave your applications vulnerable.
When audit logs trigger a rule, Datadog creates a Security Signal that
you can review in the Security Signals explorer, where it’s retained for 15
months. Each rule includes a description of the identified vulnerability
and remediation steps, so you can quickly address any issues before they
become more serious.
[Link] Best practices for monitoring Kubernetes security via audit logs 60

In the screenshot above, Datadog detected a user’s attempt to launch an

interactive shell in a running pod via the kubectl exec command. Most
users should not need to run commands in a pod, so you should investigate
further to ensure that the user’s account isn’t compromised.
Datadog also gives you full visibility into all of your Kubernetes resources,
so you can easily investigate a resource that triggered a Security Signal.
For example, you can review the configurations of a new pod to determine
if it has privileged access to your cluster (e.g., a service account with
the cluster-admin role or hostPath volume mounts).

Start monitoring In this post, we looked at some key Kubernetes audit logs and how they
your Kubernetes can help you detect unusual activity in your clusters. We also walked
audit logs through how Datadog can provide better visibility into Kubernetes
environment security. Check out our documentation to learn more about
Datadog’s Kubernetes audit log integration, or sign up for a free trial to
start monitoring your Kubernetes applications today.
 Best practices for monitoring
authentication logs
Justin Massey If you are running a user-facing web application, you likely implement
some form of authentication flow to allow users to log in securely. You may
Maxim Brown even use multiple systems and methods for different purposes or separate
groups of users. For example, employees might use OAuth-based authen-
tication managed by a company-provided Google account to log in to
internal services while customers can use a username and password system
or their own Google credentials. Being able to log, monitor, and analyze all
authentication events is key for identifying security threats and managing
customer records for compliance purposes.
Authentication logs from these different sources and parts of your envi-
ronment might have different formats and be managed by different teams or
implemented using different third-party services like Google, Okta, or Auth0.
This can make meaningful analysis of all authentication activity difficult.
Making sure that your applications write authentication logs that contain
enough information and that use a standard, easily parsable format makes it
easier to perform complex analysis on all authentication logs. For example,
you can quickly identify users or IP addresses associated with the most
failed login attempts. Or, you might want to track trends in login sources
(e.g., username vs. Okta vs. G Suite) and flows (e.g., password vs. OAuth
vs. SAML). A second benefit of standardizing your logs is that it makes it
easier to quickly audit or delete authentication events for specific users for
compliance purposes.
[Link] Best practices for monitoring authentication logs 62

In this post, we’ll look at:

– Best practices for managing and formatting your authentication logs
– Some key security threats that authentication events can help you identify
– How you can use Datadog to get the most out of your authentication logs

Best practices In order to extract the most information from your authentication events
for writing in a way that is useful, you should:
authentication – Make sure to log everything from all authentication flows
logs
– Write logs that include all the information you may need
– Make sure your logs use a standardized, easily parsable format

LOG EVERYTHING
You are probably logging events from authentication flows that you manage
internally, such as when your employees log in using company-provided
Google credentials. But if, for example, your application allows customers
to create accounts and log in with their own Google credentials, which you
don’t manage, you are likely not collecting those logs.
In order to get visibility into all of your authentication activity, you should
make sure you log events for all login flows at the application level. This
ensures that you’re logging everything, eliminating gaps in your monitoring
coverage. It also gives you more control over how you log authentication
events and what data you are collecting.

INCLUDE ENOUGH INFORMATION

Logs that don’t include all the data you might need about an authentication
event may not be very useful. For example:

2020-01-01 12:00:01 John Doe logged in

This log provides the “who” (John Doe), the “what” (logged in), and the
“when” (2020-01-01 12:00:01) of the authentication event. But it doesn’t
give you information like “how” (e.g., did John use a username and pass-
word, or did he log in with his Google account?) or “where” (e.g., what IP
address did John log in from?). You would also need a separate log event
to indicate a login failure. Without this data, you can’t, for example, monitor
trends in login sources and methods, or identify potential authentication
attacks. Next, let’s look at a log that includes that information:
[Link] Best practices for monitoring authentication logs 63

2020-01-01 12:00:01 google oauth login success by John Doe from [Link]

This log provides more details about the event that you can use to perform
complex analysis more easily. By logging all authentication events at
the application level, you can ensure that your logs contain this level of
information.

LOG USING A STANDARD, PARSABLE FORMAT

Even if you’re collecting detailed logs for all authentication events, if they
all are written as simple strings, parsing them or searching for the logs you
need is cumbersome. You can instead have your application write logs in a
key-value format, using = as a separator. Using this format means that a
key-value parser, such as Datadog’s Grok Parser can easily process them.
Let’s take the log we saw above:

2020-01-01 12:00:01 google oauth login success by John Doe from [Link]

If we log this using a key-value format, it might look like the following:

INFO 2020-01-01 12:00:01 [Link]="John Doe" [Link]=authentication

[Link]="google oauth" [Link]=success [Link]=[Link]

Datadog can then parse this as the following JSON:

{
"usr": {
"id": "John Doe"
},
"evt": {
"category": "authentication",
"name": "google oauth",
"outcome": "success",
},
"network": {
"client": {
"ip": "[Link]"
}
}
}
[Link] Best practices for monitoring authentication logs 64

Using the same format across all of your authentication logs means you
can easily use these attributes to slice and dice log data to view exactly
the information you need. For example, you can easily look for which users
( [Link] ) have the highest number of failed logins ( [Link]:failure ).
A key-value format also makes it easy to add custom attributes to logs.
For example, you might want each log to include a reCAPTCHA v3 score
to identify possible bot activity. Another important point is to use quotes
to wrap any attribute values that may contain spaces. This ensures that
you capture the full value in a way that is easily parsable.
It’s important to use a standard naming convention for the attributes in
your logs to ensure that you can search and aggregate data across all of
them, regardless of where they come from. We recommend making sure
your authentication logs include the following standard attributes:
– [Link]
– [Link]
– [Link]
– [Link]
– [Link]

[Link]
This attribute identifies the user who is requesting authentication. You
should make sure that the value for [Link] is a username or email address
rather than any unique identifier you might use to identify users in your
database. You should also make sure to include a unique username even if
the user does not have one in your database. This allows you to track which
user IDs are failing to login, which can be used to detect an attack.

[Link]
Setting this to authentication makes it easy to search for authentication
events and filter them out from the rest of your event logs.

[Link]
The [Link] should include more detailed information about the source
and method of authentication, such as whether it used SAML ( evt.
name="saml" ) or was a Google login using OAuth ( [Link]="google
oauth" ).

[Link]
This should be set to either success or failure . You can use this attribute
to easily look for patterns in login failures across your applications.
[Link] Best practices for monitoring authentication logs 65

[Link]
This is the IP address of the user or client who is requesting authentication.
Recording this provides insight into where requests are coming from.

Use authentica- Now that you are collecting and parsing key data out of your authentication
tion logs to logs, you can use them to detect possible security threats. For example,
detect common if you see a significant number of failed login attempts from a single user
security threats within a short period of time, it could indicate a brute force attack.
If those failed login attempts are followed by a successful one, it could be
a successful account takeover that you should investigate immediately.
Another common authentication attack technique that you can easily look
for in your logs is credential stuffing. Credential stuffing is when an attacker
mixes and matches breached login credentials to try to match a real user
account. In order to detect this type of attack, you can look for logins using
multiple [Link] values all coming from the same [Link]

Multiple logins from one IP address using several different user IDs could indicate a credential
stuffing attack.

Next, we’ll look at how Datadog can help you automatically detect
malicious activity like these and other attacks by monitoring your
authentication logs.
[Link] Best practices for monitoring authentication logs 66

Monitor With Datadog Cloud SIEM, you can easily monitor your authentica-tion logs
your authenti- and get alerted to possible attacks or other suspicious activity.
cation logs You can create custom dashboards to visualize key authentication data
with Datadog like counts of logins by source and outcome. This provides you with a high-
level view of activity across your entire user base and helps you see trends
in how users are logging in and the top sources of failed logins, so you can
identify suspicious spikes that you need to investigate.

Create dashboards to visualize and track trends in authentication activity.

Datadog also provides out-of-the-box dashboards, such as the IP

investigation dashboard and User investigation dashboard. These correlate
key data from your authentication logs with relevant data from the rest of
your environment to assist your investigations.

Datadog includes turn-key Detection Rules that scan 100 percent of your
ingested logs in real time for common attacker techniques. If any log trig-
gers one of these rules, it generates a Security Signal that includes key data
about the event, such as the type of attack detected and suggestions on
a response strategy. You can easily view, filter, and sort all of your Security
Signals in the explorer to triage them and see where to focus your efforts.
[Link] Best practices for monitoring authentication logs 67

For signals triggered from the Credential Stuffing Attack Detection

Rule, Datadog includes an out-of-the-box runbook designed to help you
with response and remediation. The interactive runbook guides you through
investigation strategies alongside graphs of your own logs. You can save
a copy of the runbook and set the the time frame, document your investi-
gation in markdown, and easily share it with teammates for comment.

Datadog ingests and analyzes all of your logs, ensuring that you can
detect threats across your entire environment. You can archive any logs
that you don’t want to index, and then quickly rehydrate them in the
future for investigations, audits, and compliance purposes.
[Link] Best practices for monitoring authentication logs 68

Get insight In this post we looked at some best practices for managing authentication
into all of your logs that can help you easily track and analyze user activity and identify
authentication security threats across your environment. With Datadog, you can get even
events more insight into the security of your applications as well as the health
and performance of the rest of your stack, all from a unified platform.
If you’re not a Datadog user, sign up for a free trial to start monitoring your
authentication logs today.