Internet network endpoint groups overview

Cloud Load Balancing supports proxying traffic to external backends outside Google Cloud. To define an external backend for a load balancer, you use a resource called an internet network endpoint group (NEG).

You can use this type of deployment when you want to serve content from an external backend, but you want your Google Cloud load balancer to be the frontend. This lets you do the following:

Figure 1 shows an external Application Load Balancer with multiple backend types, one of which is an external backend configured with an internet NEG.

Internet network endpoint groups in load balancing.
Figure 1. Internet network endpoint groups in load balancing (click to enlarge).

Internet NEG backends are supported with various global and regional load balancers. Depending on your load balancer (global or regional), internet NEG support differs with respect to DNS, health checking, available number of endpoints, server certificate validation, and traffic routing behaviors.

The sections that follow explain how external backends are used with Cloud Load Balancing. If you want to use an external backend with Cloud Service Mesh, see Cloud Service Mesh with internet network endpoint groups.

Terminology

The following terms are sometimes used interchangeably because they have the same or similar meanings:

This document uses the term external backend except when referring to the internet NEG API resource.

Load balancer components

This section describes the load balancing architecture and resources required to configure a load balancer with an external backend. The load balancer requires special configuration only for the backend service. The frontend configuration is the same as any other load balancer.

The following figures show the Google Cloud resources required to set up a load balancer with an external backend.

Global external

This figure shows the Google Cloud resources required to set up a global external Application Load Balancer with an external backend.

Global external Application Load Balancer with external backend.
Global external Application Load Balancer with external backend (click to enlarge).

Regional external

This figure shows the Google Cloud resources required to set up a regional external Application Load Balancer with an external backend.

Regional external Application Load Balancer with an external backend.
Regional external Application Load Balancer with an external backend (click to enlarge).

Regional internal

This figure shows the Google Cloud resources required to set up a regional internal Application Load Balancer with an external backend.

Regional internal Application Load Balancer with an external backend.
Regional internal Application Load Balancer with an external backend (click to enlarge).

This figure shows the Google Cloud resources required to set up a regional internal proxy Network Load Balancer with an external backend.

Regional internal proxy Network Load Balancer with an external backend.
Regional internal proxy Network Load Balancer with an external backend (click to enlarge).

You can only use internet NEGs on the Premium network service tier.

Frontend configuration

No special frontend configuration is required for creating a load balancer with an internet NEG backend. Forwarding rules are used to route traffic by IP address, port, and protocol to a target proxy. The target proxy then terminates connections from clients. Additionally, Envoy-based load balancers require a proxy-only subnet.

Application Load Balancers also use URL maps to set up URL-based routing of requests to the appropriate backend services.

For more details about each of these components, refer to the architecture sections of the respective load balancer:

Internet NEG

An internet NEG is a resource used to define an external backend for the load balancer. There are two types of internet NEGs: global internet NEG and regional internet NEG. They differ in the scope (global versus regional) and behavior. The external backend referenced by a global internet NEG must be reachable exclusively over the internet; it can't be reachable over Cloud VPN or Cloud Interconnect. If the external backend references a Google API or service, the service must be reachable through TCP port 80 or 443 by using the HTTP, HTTPS, or HTTP/2 protocol.

There are two ways to configure the external endpoint referenced by the NEG: INTERNET_FQDN_PORT or INTERNET_IP_PORT. If the INTERNET_IP_PORT format is chosen, only a public internet routable IP address can be used; if the INTERNET_FQDN_PORT format is chosen the FQDN can be resolved to either a public internet routable IP address or to a private IP address depending on the scope of the endpoint: regional or global.

Global internet NEGs are powered by Google Front End (GFE) technologies. They use a fixed set of fixed IP addresses for egress traffic to all clients. They support only one endpoint per NEG and one internet NEG per backend service.

The following table describes how different load balancers support global internet NEGs.

Load balancers Endpoint type Endpoint definition Scope Health checks Server certificate validation
  • Global external Application Load Balancer
  • Classic Application Load Balancer

INTERNET_FQDN_PORT

A publicly resolvable fully qualified domain name and an optional port. For example, backend.example.com:443.

The domain name must be resolvable by Google's public DNS infrastructure.

Only one endpoint per NEG is allowed.

Global Not supported Always validated against public CA certificates.

INTERNET_IP_PORT

A publicly routable IP address and an optional port. For example, 8.8.8.8:4431.

The IP address can't be an RFC 1918 address.

Only one endpoint per NEG is allowed.

Not validated.

If you don't specify a port while adding the endpoint, the default port of the NEG is used. If you didn't specify a default port for the NEG, the well-known port for your backend protocol is used (80 for HTTP and 443 for HTTPS and HTTP/2).

Regional internet NEGs are powered by managed Envoy proxies. Each NEG can have multiple endpoints, and a backend service can include multiple internet NEGs.

For egress traffic, you can configure Cloud NAT gateways to set the source IP addresses. Alternatively, you can route traffic using learned routes within your VPC network. While Cloud NAT isn't required for this routing method, it is supported.

The following table describes how different load balancers support regional internet NEGs.

Load balancers Endpoint type Endpoint definition Scope Health checks Server certificate validation
  • Regional external Application Load Balancer
  • Regional internal Application Load Balancer
  • Regional external proxy Network Load Balancer
  • Regional internal proxy Network Load Balancer

INTERNET_FQDN_PORT

Either a publicly or privately resolvable fully qualified domain name and an optional port. For example, backend.example.com:443*.

The domain name resolution process follows the Cloud DNS name resolution order process.

A maximum of 256 endpoints per NEG are allowed.

Regional Envoy distributed health checks Not validated. Set up Backend Authenticated TLS to perform certificate validation.

INTERNET_IP_PORT

Only a publicly routable IP address and an optional port. For example, 8.8.8.8:4432.

The IP address can't be an RFC 1918 address.

A maximum of 256 endpoints per NEG are allowed.

Not validated. Set up Backend Authenticated TLS to perform certificate validation.

* With regional internet NEGs, you are required to specify a port. You can specify a default port while creating the NEG, or you can specify a port each time you add an endpoint to the NEG, or you can do both. If you don't specify a port while adding an endpoint, the default port of the NEG is used.

DNS resolution for regional INTERNET_FQDN_PORT endpoints

If your domain is resolvable over the internet, no other configuration is needed to set up DNS. However, if you're resolving private FQDNs, you'll need to configure Cloud DNS to facilitate DNS resolution. The name must be hosted on Cloud DNS or be resolvable using DNS forwarding from Cloud DNS to an on-premises DNS or DNS peering if referencing a Private DNS zone in another VPC network.

Start by creating a Cloud DNS zone to host the DNS records in your project. Then add the DNS records to it. For specific configuration steps, see the Cloud DNS documentation. The Cloud DNS resolution order is detailed at Name resolution order.

If you're using Shared VPC, note the specific network requirements. You can also use other features of Cloud DNS, such as forwarding zones, to fetch records from an on-premises DNS server.

IP address resolution for global INTERNET_FQDN_PORT endpoints

When an INTERNET_FQDN_PORT endpoint points to a DNS record that returns multiple IP addresses, the IP address is resolved in the following way:

For more information about the IP ranges and locations used by Google's DNS resolver infrastructure, see the Google public DNS documentation. Names that can't be resolved by the public DNS system aren't usable as external backends.

IP address resolution for regional INTERNET_FQDN_PORT endpoints

Regional internet NEGs support domain name resolution using both Cloud DNS and Google's public DNS.

If the DNS server returns multiple IP addresses, Envoy load balances traffic among the returned IP addresses based on the configured load balancing algorithm (round robin, least request, and so on). The list of endpoints is periodically refreshed based on DNS TTL. You can configure retry policies to force Envoy to attempt to connect to another IP address if one fails.

Backend service

Backend services provide configuration information to the load balancer. Load balancers use the information in a backend service to direct incoming traffic to one or more attached backends.

To set up a load balancer with an external backend, you configure the backend service with an internet NEG backend. When you add an internet NEG to a backend service, the following considerations apply, depending on the scope of the NEG:

Health checks

Your health check configuration varies depending on the type of load balancer:

Regional external Application Load Balancer, regional internal Application Load Balancer, regional external proxy Network Load Balancer, and regional internal proxy Network Load Balancer. Health checks are optional. Health check probes for these load balancers originate from the proxy-only subnet and are then NAT-translated (by using Cloud NAT) to either pre-reserved IP addresses or auto-allocated NAT IP addresses. For details, see Regional NEGs: Use a Cloud NAT gateway.

Distributed Envoy health checks are created by using the same Google Cloud console, gcloud CLI, and API processes as centralized health checks. No other configuration is required.

Points to note:

Enable the external backend to receive requests

Configure your external backend to allow traffic from Google Cloud.

This procedure depends on the scope of the NEG: global or regional.

Global NEGs: Allowlist default Google egress IP addresses

If you're using a global internet NEG, you must add the IP address ranges that Google uses to send requests to external backends to an allowlist. To look up the IP addresses to be added to an allowlist, query the _cloud-eoips.googleusercontent.com DNS TXT record by using a tool like dig or nslookup.

For an example, see Allow the external backend to receive traffic from Google Cloud.

Regional NEGs: Use a Cloud NAT gateway

If you're using a regional internet NEG, you'll first set up a Cloud NAT gateway to allocate a set of IP address ranges from where Google Cloud traffic should originate.

The gateway endpoint should be of type ENDPOINT_TYPE_MANAGED_PROXY_LB.

The Cloud NAT gateway can be configured to either automatically allocate external IP addresses based on demand or use a manually pre-reserved set of external IP addresses.

Dynamic port allocation is supported for regional internet NEGs. IP addresses can be shared by proxies, thus fully utilized.

This Cloud NAT configuration applies to the entire proxy-only subnet. Internet traffic associated with all the regional Envoy-based load balancers in the region share the same NAT gateway.

NAT gateways configured on proxy-only subnets don't support logging and monitoring. That is, the --enable-logging and --log-filter flags aren't supported.

Authenticate requests to the external backend

This section applies only to Application Load Balancers.

To authenticate requests sent to your external backend, you can do one of the following:

Logs

Requests proxied to an external backend are logged to Cloud Logging in the same way that requests for other backends are logged.

For more information, see the following:

If you enable Cloud CDN for an external Application Load Balancer using an external backend, cache hits are also logged.

Header processing with external backends

Different load balancers might handle header processing in different ways when they proxy requests to an external backend. Review the following information to understand how your load balancer type might process HTTP headers.

Global external Application Load Balancers and classic Application Load Balancers

When a global external Application Load Balancer or a classic Application Load Balancer proxies requests to an external backend, it adjusts HTTP headers in the following ways:

Regional external Application Load Balancers and regional internal Application Load Balancers

There is no special header processing for Envoy-based load balancers using internet NEGs. To learn how Envoy typically processes headers, see HTTP header manipulation.

Limitations

Quotas and limits

For information about quotas and limits, see the NEG backends quota table and the Endpoints per NEG quota table.

Pricing

For information about internet egress rates to internet NEG endpoints, see Premium Tier pricing.

If you configured a Cloud NAT gateway to map your regional Envoy-based load balancer's proxy-only subnet, see Network pricing: Cloud NAT.

For any other information about Cloud Load Balancing pricing and regional internet NEG charges, see Network pricing: Cloud Load Balancing.

What's next