Console
Start free

Apache Kafka trigger

The Apache Kafka trigger is a Connector Event trigger that lets you invoke an integration based on the Apache Kafka events that you've subscribed to in your Apache Kafka connection.

Before you begin

If you plan to create or configure a new connection for the Apache Kafka trigger, make sure that you have the following Identity and Access Management (IAM) role on the project:

Grant the following IAM roles to the service account that you want to use for the Apache Kafka trigger:

Add the Apache Kafka trigger

To add a Apache Kafka trigger to your integration, perform the following steps:

  1. In the Google Cloud console, go to the Application Integration page.

    Go to Application Integration

  2. In the navigation menu, click Integrations

    The Integrations List page appears listing all the integrations available in the Google Cloud project.

  3. Select an existing integration or click Create integration to create a new one.

    If you are creating a new integration:

    1. Enter a name and description in the Create Integration pane.
    2. Select a region for the integration.
    3. Select a service account for the integration. You can change or update the service account details of an integration any time from the Integration summary pane in the integration toolbar.
    4. Click Create.

    The newly created integration opens in the integration editor.

  4. In the integration editor navigation bar, click Triggers to view the list of available triggers.
  5. Click and place the Apache Kafka trigger element in the integration editor.

    6. To configure the Apache Kafka trigger, you can either use an existing Apache Kafka connection available in Integration Connectors, or create a new Apache Kafka connection using the inline connection creation option.

Configure Apache Kafka trigger using an existing connection

You can configure the Apache Kafka trigger using an existing Apache Kafka connection in Integration Connectors. Note that the Apache Kafka trigger is an Connector Event trigger, therefore you can only use a Apache Kafka connection with event subscription enabled to configure the trigger.

For information about how to configure a Apache Kafka trigger using a new Apache Kafka connection, see Configure Apache Kafka trigger using a new connection

To configure a Apache Kafka trigger using an existing Apache Kafka connection, perform the following steps:

  1. Click the Apache Kafka trigger element in the integration editor to open the trigger configuration pane.
  2. Click Configure trigger.
  3. Provide the following configuration details in the Connector Event Trigger Editor page:
    1. Region: Select the region of your Apache Kafka connection.
    2. Connection: Select the Apache Kafka connection that you want to use.

      Application Integration only displays those Apache Kafka connections that are active and have an event subscription enabled.

    3. Topic name: Enter the topic name. Note: As per Apache Kafka documentation, the maximum length for a topic name is 249 characters.
    4. Consumer group ID: Optionally, enter the consumer group ID that specifies the name of the consumer group to which the Kafka consumer belongs. The maximum length for a consumer group ID is generally 255 characters.
    5. Partition: Enter the partition if you want to subscribe to a particular partition. Partitions are a way to break up a topic into multiple parts and store each part on a different node in the cluster.
    6. Initial Offset: Select the offset reset strategy:
      • If you select earliest, your connection consumes unconsumed messages including any messages produced before the lifetime of the consumer group.
      • If you select latest, your connection consumes messages that are produced after the consumer group is created.
    7. Service Account: Select a service account with the required IAM roles for the Apache Kafka trigger.
  4. Click Done.

Configure Apache Kafka trigger using a new connection

To configure the Apache Kafka trigger using a new Apache Kafka connection, perform the following steps:

  1. Click the Apache Kafka trigger element in the integration editor to open the trigger configuration pane.
  2. Click Configure Apache Kafka trigger.
  3. Skip the Region field.
  4. Click Connection and select the Create Connection option from the list. The Create Connection pane appears.
  5. In the Location step, choose the location for the new Apache Kafka connection:
    1. Region: Select a region from the list.
    2. Click Next.
  6. In the Connection Details step, provide details about the new Apache Kafka connection:
    1. Connector version: Choose an available version of the Apache Kafka connector from the list.
    2. Connection Name: Enter a name for the Apache Kafka connection.
    3. (Optional) Description: Enter a description for the connection.
    4. (Optional) Enable Cloud Logging: Select this checkbox to store all log data of the connection.
    5. Service Account: Select a service account with the required IAM roles for the Apache Kafka connection.
    6. The Enable event subscription, entity and actions option is selected by default for the Apache Kafka connection.
    7. Type Detection Scheme: Select MessageOnly.
    8. Registry Service: The Schema Registry service used for working with topic schemas.
    9. Registry Type: Type of the schema specified for the a specific topic.
    10. Registry Version: Version of the schema read from RegistryUrl for the specified topic.
    11. Registry User: Username or Access Key value to authorize with the server specified in RegistryUrl.
    12. Registry Password: Secret Manager Secret containing the password/secret Key value to authorize with the server specified in RegistryUrl.
    13. Optionally, configure the Connection node settings:

      • Minimum number of nodes: Enter the minimum number of connection nodes.
      • Maximum number of nodes: Enter the maximum number of connection nodes.

      A node is a unit (or replica) of a connection that processes transactions. More nodes are required to process more transactions for a connection and conversely, fewer nodes are required to process fewer transactions. To understand how the nodes affect your connector pricing, see Pricing for connection nodes. If you don't enter any values, by default the minimum nodes are set to 2 (for better availability) and the maximum nodes are set to 50.

    14. Optionally, click + ADD LABEL to add a label to the Connection in the form of a key and value pair.
    15. Enable SSL: This field sets whether SSL is enabled.
    16. Click Next.
  7. In the Destinations section, enter details of the Kafka bootstrap server you want to connect to.
    1. Destination Type: Select a Destination Type.
      • To specify the destination hostname or IP address, select Host address and enter the address in the Host 1 field.
      • To establish a private connection, select Host Address and add the endpoint attachment created for the SAP Gateway using the HTTPS protocol.

      If you want to establish a public connection to your backend systems with additional security, you can consider configuring static outbound IP addresses for your connections, and then configure your firewall rules to allowlist only the specific static IP addresses.

      To specify additional Kafka bootstrap servers, click + ADD DESTINATION.

    2. Click Next.
  8. In the Authentication section, enter the authentication details.
    1. Select an Authentication type and enter the relevant details.

      The following authentication types are supported by the Apache Kafka connection:

      • Username and password
        • Username: The Apache Kafka username to use for the connection.
        • Password: Secret Manager Secret containing the password associated with the Apache Kafka username.
        • Auth Scheme: The scheme used for authentication.

          The following Auth Schemes are supported by the Apache Kafka connection:

          • Plain
          • SCRAM-SHA-1
          • SCRAM-SHA-256
      • Not Available

        If you want to use anonymous login, select Not Available.

    2. Click Next.
  9. Enter the dead-letter configuration. If you configure dead-letter, the connection writes the unprocessed events to the specified Pub/Sub topic. Enter the following details:
    1. Dead-letter project ID: The Google Cloud project ID where you have configured the dead-letter Pub/Sub topic.
    2. Dead-letter topic: The Pub/Sub topic where you want to write the details of the unprocessed event.
  10. Click Next.
  11. Review: Review your connection and authentication details.
  12. Click Create.

Trigger output

The Apache Kafka trigger takes a couple of minutes to complete the event subscription configuration. You can view the status of your event subscription in the trigger configuration pane under Event Subscription details.

The Apache Kafka trigger indicates the status of an event subscription using the following states:

In addition to the event subscription status, the Event Subscription details section also displays other details such as connection region, connection name, and event subscription name.

Trigger output variable

For each event, the Apache Kafka trigger generates a ConnectorEventPayload output variable (in JSON format) which you can use in your downstream tasks. The output variable will have the payload from your Apache Kafka event. For example: 

  {
    "message": "{\"key\":\"18\",\"topic\":\"test\",\"value\":\"hello world\",\"partition\":0,\"offset\":12,\"headers\":\"{}\",\"timestamp\":1712755076824,\"serializedMessagesize\":14}",
    "contentType": "text/plain"
 }

View event subscriptions

To view and manage all the event subscriptions associated with a connection in Integration Connectors, do the following:

  1. Go to Integration Connectors > Connections page.

    Go to the Connections page

  2. Click the connection for which you want to view the subscriptions.
  3. Click the Event subscriptions tab.

    This displays all the event subscriptions for the connection.

Edit Apache Kafka trigger

You can edit a Apache Kafka trigger to change or update the connection configuration and event subscription details.

To edit a Apache Kafka trigger, perform the following steps:

  1. Click the Apache Kafka trigger element in the integration editor to open the trigger configuration pane.
  2. Click Configure Apache Kafka trigger.
  3. Do the following in the Connector Event Trigger Editor page:
    1. To retain the previously configured event subscription, click Retain, else click Delete.
    2. Update the connection configuration and event subscription details as desired.
    3. Click Done.

    4. You can view the updated connection and event subscription details in the trigger configuration pane under Event Subscription details.

Quotas and limits

For information about quotas and limits, see Quotas and limits.

What's next

  • Learn about all tasks and triggers.
    • Learn how to test and publish an integration.
  • Learn about error handling.
  • Learn about integration execution logs.