/

Console

• English
• Deutsch
• Español
• Español – América Latina
• Français
• Indonesia
• Italiano
• Português
• Português – Brasil
• 中文 – 简体
• 中文 – 繁體
• 日本語
• 한국어

Sign in

• Cloud Logging

Start free

Configure log buckets

This document describes how to create and manage Cloud Logging buckets using the Google Cloud console, the Google Cloud CLI, and the Logging API. It also provides instructions for creating and managing log buckets at the Google Cloud project level. You can't create log buckets at the folder or organization level; however, Cloud Logging automatically creates _Default and _Required log buckets at the folder and organization level for you.

You can upgrade log buckets to use Observability Analytics. Observability Analytics lets you run SQL queries on your log data, helping you troubleshoot application, security, and networking issues.

To use BigQuery to analyze your log data, you have two choices:

• Upgrade a log bucket to use Observability Analytics and then create a linked BigQuery dataset. In this scenario, Logging stores your log data but BigQuery can read the log data.

• Export your log entries to BigQuery. In this scenario, you must create a sink, BigQuery stores and manages the data, and you have the option to use partitioned tables.

When your log data is available to BigQuery, you can join your log data with other data stored in BigQuery, and you can access this data from other tools like Data Studio and Looker.

For a conceptual overview of buckets, see Routing and storage overview: Log buckets.

This document doesn't describe how to create a log bucket that uses a customer-managed encryption key (CMEK). If you are interested in that topic, then see Configure CMEK for logs storage.

Before you begin

To get started with buckets, do the following:

• Configure your Google Cloud project:

  1. Verify that billing is enabled for your Google Cloud project.

  2. To get the permissions that you need to create, upgrade, and link a log bucket, ask your administrator to grant you the Logs Configuration Writer (roles/logging.configWriter) IAM role on your project. For more information about granting roles, see Manage access to projects, folders, and organizations.

     You might also be able to get the required permissions through custom roles or other predefined roles.

     For the full list of permissions and roles, see Access control with IAM.

  3. Optional: To use BigQuery to view the data stored in a log bucket, do the following:
     1. Make sure that the BigQuery API is enabled. You can verify that the API is enabled by listing available services.
     2. Your Identity and Access Management role includes the permissions that let you create a linked dataset. For more information, see Permissions for linked BigQuery datasets.
• Understand the supported regions in which you can store your logs.

• Select the tab for how you plan to use the samples on this page:

  Console

  When you use the Google Cloud console to access Google Cloud services and APIs, you don't need to set up authentication.

  gcloud

  In the Google Cloud console, activate Cloud Shell.

  Activate Cloud Shell

  At the bottom of the Google Cloud console, a Cloud Shell session starts and displays a command-line prompt. Cloud Shell is a shell environment with the Google Cloud CLI already installed and with values already set for your current project. It can take a few seconds for the session to initialize.

  REST

  To use the REST API samples on this page in a local development environment, you use the credentials you provide to the gcloud CLI.

  Install the Google Cloud CLI.

  If you're using an external identity provider (IdP), you must first sign in to the gcloud CLI with your federated identity.

  For more information, see Authenticate for using REST in the Google Cloud authentication documentation.

• If you plan to use the Google Cloud CLI or Cloud Logging API to create or manage your log buckets, then understand the LogBucket formatting requirements.

Create a bucket

You can create a maximum of 100 buckets per Google Cloud project. You can't create log buckets in folders or organizations.

To create a user-defined log bucket for your Google Cloud project, do the following:

gcloud logging buckets create BUCKET_ID --location=LOCATION --enable-analytics --async OPTIONAL_FLAGS

gcloud logging buckets create my-bucket --location global --description "My first bucket"

gcloud logging buckets create my-upgraded-bucket --location global \
      --description "My first upgraded bucket" \
      --enable-analytics --retention-days=45

After creating a bucket, create a sink to route log entries to your bucket and configure log views to control who can access the logs in your new bucket and which logs are accessible to them. You can also update the bucket to configure custom retention and restricted fields.

Track volume of logs stored in log buckets

The Logs Storage page in the Google Cloud console tracks the volume of logs data stored in log buckets:

In the Google Cloud console, go to the Logs Storage page:

Go to Logs Storage

If you use the search bar to find this page, then select the result whose subheading is Logging.

The Logs Storage page displays a summary of statistics for your Google Cloud project:

The following statistics are reported:

• Current month ingestion: The amount of logs data that your Google Cloud project has stored in log buckets since the first day of the current calendar month.

• Previous month ingestion: The amount of logs data that your Google Cloud project stored in log buckets in the last calendar month.

• Projected ingestion by EOM: The estimated amount of logs data that your Google Cloud project will store in log buckets by the end of the current calendar month, based on current usage.

• Current month billable storage: The amount of logs data that has been retained for over 30 days that is billed.

The previous statistics don't include logs in the _Required bucket. The logs in that bucket can't be excluded or disabled.

The Log Router page in the Google Cloud console gives you tools that you can use to minimize any charges for storing logs in log buckets or for storage that exceeds your monthly allotment. You can do the following:

• Disable logs from being stored at the bucket level.
• Exclude certain log entries from being stored in log buckets.

For more information, see Manage sinks.

Manage buckets

This section describes how to manage your log buckets using the Google Cloud CLI or the Google Cloud console.

Update a bucket

To update the properties of your bucket, such as the description or retention period, do the following:

gcloud logging buckets update BUCKET_ID --location=LOCATION UPDATED_ATTRIBUTES

gcloud logging buckets update my-bucket --location=global --description "Updated description"

Upgrade a bucket to use Observability Analytics

After you upgrade a bucket to use Observability Analytics, any new log entries that arrive are available to analyze in the Observability Analytics interface. You can't undo an upgrade operation on a bucket.

To upgrade an existing bucket to use Observability Analytics, the following restrictions apply:

• The log bucket was created at the Google Cloud project level.
• The log bucket is unlocked unless it is the _Required bucket.

• There aren't pending updates to the bucket.

gcloud logging buckets update BUCKET_ID --location=LOCATION --enable-analytics --async

Create a linked BigQuery dataset

When you want to use the capabilities of BigQuery to analyze your log data, upgrade a log bucket to use Observability Analytics, and then create a linked dataset. With this configuration, Logging stores your log data but BigQuery can read the log data.

gcloud logging links create LINK_ID --bucket=BUCKET_ID --location=LOCATION

gcloud logging links create mylink --bucket=my-bucket --location=global

Lock a bucket

When you lock a bucket against updates, you also lock the bucket's retention policy. After a retention policy is locked, you can't delete the bucket until every log entry in the bucket has fulfilled the bucket's retention period. If you want to prevent the accidental deletion of a project that contains a locked log bucket, then add a lien to the project. To learn more, see Protecting projects with liens.

To prevent anyone from updating or deleting a log bucket, lock the bucket. To lock the bucket, do the following:

gcloud logging buckets update BUCKET_ID --location=LOCATION --locked

gcloud logging buckets update my-bucket --location=global --locked

List buckets

To list the log buckets associated with a Google Cloud project, and to see details such as retention settings, do the following:

gcloud logging buckets list

gcloud logging buckets describe _Default --location=global

View a bucket's details

To view the details of a single log bucket, do the following:

gcloud logging buckets describe _Default --location=global

Delete a bucket

You can delete log buckets that satisfy one of the following:

• The log bucket is unlocked.
• The log bucket is locked and all log entries in the log bucket have fulfilled the bucket's retention period.

You can't delete a log bucket that is locked against updates when that log bucket stores log entries that haven't fulfilled the bucket's retention period.

After you issue the delete command, the log bucket transitions to the DELETE_REQUESTED state, and it stays in that state for 7 days. During this time period, Logging continues to route logs to the log bucket. You can stop routing logs to the log bucket by deleting or modifying the log sinks that route log entries to the bucket.

You can't create a new log bucket that uses the same name as a log bucket that is in the DELETE_REQUESTED state.

To delete a log bucket, do the following:

gcloud logging buckets delete BUCKET_ID --location=LOCATION

Restore a deleted bucket

You can restore, or undelete, a log bucket that's in the pending deletion state. To restore a log bucket, do the following:

gcloud logging buckets undelete BUCKET_ID --location=LOCATION

Alert on monthly log bytes stored in log buckets

To create an alerting policy, on the Logs Storage page in the Google Cloud console, click add_alert Create usage alert. This button opens the Create alerting policy page in Monitoring, and populates the metric type field with logging.googleapis.com/billing/bytes_ingested.

To create an alerting policy that triggers when the number of log bytes written to your log buckets exceeds your user-defined limit for Cloud Logging, use the following settings.

For more information about alerting policies, see Alerting overview.

Write to a bucket

You don't directly write logs to a log bucket. Rather, you write logs to Google Cloud resource: a Google Cloud project, folder, or organization. The sinks in the parent resource then route the logs to destinations, including log buckets. A sink routes logs to a log bucket destination when the logs match the sink's filter and the sink has permission to route the logs to the log bucket.

Read from a bucket

Each log bucket has a set of log views. To read logs from a log bucket, you need access to a log view on the log bucket. Log views let you grant a user access to only a subset of the logs stored in a log bucket. For information about how to configure log views, and how to grant access to specific log views, see Configure log views on a log bucket.

To read logs from a log bucket, do the following:

gcloud logging read LOG_FILTER --bucket=BUCKET_ID --location=LOCATION --view=LOG_VIEW_ID

For detailed information about the filtering syntax, see Logging query language.

Configure custom retention

When you create a log bucket, you can customize the period that Cloud Logging stores logs in the bucket. You can also configure the retention period for the _Default log bucket when that log bucket is in a project.

You can't change the retention period of the _Required log bucket. You also can't extend the retention period of a log bucket that is in a folder or organization. However, you can create a log sink to route folder- or organization-level log entries to a log bucket that is in a project, and then configure the retention period of that log bucket.

If you shorten the retention period of a log bucket, then there is a 7-day grace period in which expired logs aren't deleted. You can't query or view the expired logs. However, during this 7-day period, you can restore full access by extending the bucket's retention period. Logs stored during the grace period count toward your retention costs.

Retention enforcement is an eventually-consistent process. If you write log entries to a log bucket when the log entries are older than the bucket's retention period, then you might be able to briefly see these log entries. For example, if you send log entries that are 10 days old to a log bucket with a retention period of 7 days, then those log entries are stored and then eventually purged. These log entries don't contribute to your retention costs. They do contribute to your storage costs. To minimize your storage costs, don't write log entries that are older than your bucket's retention period.

To update the retention period for a custom log bucket or for the _Default log bucket that is in a project, do the following:

gcloud logging buckets update BUCKET_ID  --location=LOCATION --retention-days=RETENTION_DAYS

gcloud logging buckets update _Default --location=global --retention-days=365

If you extend a bucket's retention period, then the retention rules apply going forward and not retroactively. Logs can't be recovered after the applicable retention period ends.

Asynchronous API methods

The response of an asynchronous method like projects.locations.buckets.createAsync is an Operation object.

Applications that call an asynchronous API method should poll the operation.get endpoint until the value of the Operation.done field is true:

• When done is false, the operation is in progress.

  To refresh the status information, send a GET request to the operation.get endpoint.

• When done is true, the operation is complete and either the error or response field is set:

  • error: When set, the asynchronous operation failed. The value of this field is a Status object that contains a gRPC error code and an error message.
  • response: When set, the asynchronous operation completed successfully, and the value reflects the result.

To poll an asynchronous command by using the Google Cloud CLI, run the following command:

gcloud logging operations describe OPERATION_ID --location=LOCATION --project=PROJECT_ID

For more information, see gcloud logging operations describe.

Troubleshoot common issues

If you encounter problems when using log buckets, refer to the following troubleshooting steps and answers to common questions.

Why can't I delete this bucket?

If you're trying to delete a bucket, do the following:

• Verify that you have the correct permissions to delete the bucket. For the list of the permissions that you need, see Access control with IAM.

• Determine whether the bucket is locked by listing the bucket's attributes. If the bucket is locked, check the bucket's retention period. You can't delete a locked bucket until all of the logs in the bucket have fulfilled the bucket's retention period.

• Verify that the log bucket doesn't have a linked BigQuery dataset. You can't delete a log bucket with a linked dataset.

  The following error is shown in response to a delete command on a log bucket that has a linked dataset:

  FAILED_PRECONDITION: This bucket is used for advanced analytics and has an active link. The link must be deleted first before deleting the bucket
  

  To list the links associated with a log bucket, run the gcloud logging links list command or run the projects.locations.buckets.links.list API method.

Which service accounts are routing logs to my bucket?

To determine if any service accounts have IAM permissions to route logs to your bucket, do the following:

1. In the Google Cloud console, go to the IAM page:

   Go to IAM

   If you use the search bar to find this page, then select the result whose subheading is IAM & Admin.

2. From the Permissions tab, view by Roles. You see a table with all the IAM roles and principals associated with your Google Cloud project.

3. In the table's Filter text box filter_list, enter Logs Bucket Writer.

   You see any principals with the Logs Bucket Writer role. If a principal is a service account, its ID contains the string gserviceaccount.com.

4. Optional: If you want to remove a service account from being able to route logs to your Google Cloud project, select the check box check_box_outline_blank for the service account and click Remove.

Why do I see logs for a Google Cloud project even though I excluded them from my _Default sink?

You might be viewing logs in a log bucket in a centralized Google Cloud project, which aggregates logs from across your organization.

If you're using the Logs Explorer to access these logs and see logs that you excluded from the _Default sink, then your view might be set to the Google Cloud project level.

To fix this issue, select Log view in the Refine scope menu and then select the log view associated with the _Default bucket in your Google Cloud project. You shouldn't see the excluded logs anymore.

What's next

For information on the log bucket API methods, refer to the LogBucket reference documentation.

If you manage an organization or a folder, then you can specify the location of the _Default and _Required log buckets of child resources. You can also configure whether log buckets use CMEK and the behavior of the _Default log sink. For more information, see Configure default settings for organizations and folders.

For information on addressing common use cases with log buckets, see the following topics:

• Aggregate and store your organization's logs.

• Multi-tenant logging on GKE.

• Regionalize your logs.