Console
Start free

Troubleshooting using the serial console

This page describes how to enable interactive access to an instance's serial console to debug boot and networking issues, troubleshoot malfunctioning instances, interact with the GRand Unified Bootloader (GRUB), and perform other troubleshooting tasks.

A Compute Engine instance has four virtual serial ports. Interacting with a serial port is similar to using a terminal window, in that input and output is entirely in text mode and there is no graphical interface or mouse support. The instance's operating system, BIOS, and other system-level entities often write output to the serial ports, and can accept input such as commands or answers to prompts. Typically, these system-level entities use the first serial port (port 1) and serial port 1 is often referred to as the serial console.

If you only need to view serial port output without issuing any commands to the serial console, you can call the getSerialPortOutput method or use Cloud Logging to read information that your instance has written to its serial port; see Viewing serial port logs. However, if you run into problems accessing your instance through SSH or need to troubleshoot an instance that is not fully booted, you can enable interactive access to the serial console, which lets you connect to and interact with any of your instance's serial ports. For example, you can directly run commands and respond to prompts in the serial port.

When you enable or disable the serial port, you can use any Boolean value that is accepted by the metadata server. For more information, see Boolean values.

Before you begin

Permissions required for this task

To perform this task, you must have the following permissions:

Enabling interactive access on the serial console

Enable interactive serial console access for individual compute instances or for an entire project.

Enabling access for a project

Enabling interactive serial console access on a project enables access for all compute instances that are part of that project.

By default, interactive serial port access is disabled. You can also explicitly disable it by setting the serial-port-enable key to FALSE. In either case, any per-instance setting overrides the project-level setting or the default setting.

Console

  1. In the Google Cloud console, go to the Metadata page.

    Go to Metadata

  2. Click Edit to edit metadata entries.
  3. Add a new entry that uses the key serial-port-enable and value TRUE.
  4. Save your changes.

gcloud

Using the Google Cloud CLI, enter the project-info add-metadata command as follows:

gcloud compute project-info add-metadata \
    --metadata serial-port-enable=TRUE

REST

In the API, make a request to the projects().setCommonInstanceMetadata method, providing the serial-port-enable key with a value of TRUE:

{
 "fingerprint": "FikclA7UBC0=",
 "items": [
  {
   "key": "serial-port-enable",
   "value": "TRUE"
  }
 ]
}

Enabling access for a compute instance

Enable interactive serial console access for a specific instance. A per-instance setting, if it exists, overrides any project-level setting. You can also disable access for a specific instance, even if access is enabled on the project level, by setting serial-port-enable to FALSE, instead of TRUE. Similarly, you can enable access for one or more instances even if it is disabled for the project, explicitly or by default.

Console

  1. In the Google Cloud console, go to the VM instances page.

    Go to the VM instances page

  2. Click the instance you want to enable access for.
  3. Click Edit.
  4. Under the Remote access section, toggle the Enable connecting to serial ports checkbox.
  5. Save your changes.

gcloud

Using the Google Cloud CLI, enter the instances add-metadata command, replacing instance-name with the name of your instance.

gcloud compute instances add-metadata instance-name \
    --metadata serial-port-enable=TRUE

REST

In the API, make a request to the instances().setMetadata method with the serial-port-enable key and a value of TRUE:

POST https://compute.googleapis.com/compute/v1/projects/myproject/zones/us-central1-a/instances/example-instance/setMetadata
{
 "fingerprint": "zhma6O1w2l8=",
 "items": [
  {
   "key": "serial-port-enable",
   "value": "TRUE"
  }
 ]
}

Configure serial console for a bare metal instance

For bare metal instances, increase the bit rate, also known as baud rate, for the serial console to 115,200 bps (~11.5kB/sec). Using a slower speed can cause garbled or missing console output.

Bootloader configuration varies between operating systems and OS versions. Refer to the OS distributor's documentation for instructions.

If modifying the bit rate on the command line for the current session, use a command similar to the following:

console=ttyS0,115200

If modifying the GRUB configuration, use a command similar to the following:

serial --speed=115200

Make sure that you update the actual bootloader configuration. This can be done with update-grub, grub2-mkconfig, or a similar command.

Connecting to a serial console

Compute Engine offers regional serial console gateways for each Google Cloud region. After enabling interactive access for a compute instance's serial console, you can connect to a regional serial console.

The serial console authenticates users with SSH keys. Specifically, you must add your public SSH key to the project or instance metadata and store your private key on the local machine from which you want to connect. The gcloud CLI and the Google Cloud console automatically add SSH keys to the project for you. If you are using a third-party client, you might need to add SSH keys manually.

If you are using a third-party client, you can additionally validate the connection using the serial console's host keys. When you use the Google Cloud CLI to connect, host key authentication is done automatically on your behalf.

Console

To connect to a compute instance's regional serial console, do the following:

  1. In the Google Cloud console, go to the VM instances page.

    Go to the VM instances page

  2. Click the instance you want to connect to.
  3. Under Details, click Connect to serial console to connect on the default port (port 1).
  4. If you want to connect to another serial port, click the down arrow next to the Connect to serial console button and change the port number accordingly.
  5. For Windows instances, open the drop-down menu next to the button and connect to Port 2 to access the serial console.

gcloud

To connect to a compute instance's regional serial console, use the gcloud compute connect-to-serial-port command:

gcloud compute connect-to-serial-port INSTANCE_NAME \
    --port=PORT_NUMBER

Replace the following:

  • INSTANCE_NAME: the name of the compute instance whose serial console you want to connect to.

  • PORT_NUMBER: the port number you want to connect. For Linux instances, use 1, for Windows instances, use 2. To learn more about port numbers, see Understanding serial port numbering.

Other SSH clients

You can connect to an instance's serial console using other third-party SSH clients, as long as the client lets you connect to TCP port 9600. Before you connect, you can optionally validate the connection using the serial console's host keys.

To connect to the regional serial console of a compute instance, run one of the following commands, depending on your instance's OS:

  • To connect to a Linux instance:

    ssh -i PRIVATE_SSH_KEY_FILE -p 9600 PROJECT_ID.ZONE.INSTANCE_NAME.USERNAME.OPTIONS@REGION-ssh-serialport.googleapis.com

  • To connect to a Windows instance:

    ssh -i PRIVATE_SSH_KEY_FILE -p 9600 PROJECT_ID.ZONE.INSTANCE_NAME.USERNAME.OPTIONS.port=2@REGION-ssh-serialport.googleapis.com

Replace the following:

  • PRIVATE_SSH_KEY_FILE: The private SSH key for the compute instance.
  • PROJECT_ID: The project ID for this compute instance.
  • ZONE: The zone of the compute instance.
  • REGION: The region of the compute instance.
  • INSTANCE_NAME: The name of the compute instance.
  • USERNAME: The username you are using to connect to your instance. Typically, this is the username on your local machine.
  • OPTIONS: Additional options you can specify for this connection. For example, you can specify a certain serial port and specify any advanced option. The port number can be 1 through 4, inclusively. To learn more about port numbers, see understanding serial port numbering. If omitted, you will connect to serial port 1.

If you are having trouble connecting using a third-party SSH client, you can run the gcloud compute connect-to-serial-port command with the --dry-run command-line option to see the SSH command that it would have run on your behalf. Then you can compare the options with the command you are using.

Validate third-party SSH client connections

When you use a third-party SSH client that isn't the Google Cloud CLI, we recommend that you verify that you're protected against impersonation or man-in-the-middle attacks by checking Google's Serial Port SSH host key. To set up your system to check the SSH host key, complete the following steps:

  1. Download the SSH host key for the serial console you will be using:

  2. Open your known hosts file, generally located at ~/.ssh/known_hosts.

  3. Add the contents of the SSH host key, with the server's hostname prepended to the key. For example, if the us-central1 server key contains the line ssh-rsa AAAAB3NzaC1yc..., then ~/.ssh/known_hosts should have a line like this:

    [us-central1-ssh-serialport.googleapis.com]:9600 ssh-rsa AAAAB3NzaC1yc...

For security reasons, Google might occasionally change the Google Serial Port SSH host key. If your client fails to authenticate the server key, immediately end the connection attempt and complete the earlier steps to download a new Google Serial Port SSH host key.

If, after updating the host key, you continue to receive a host authentication error from your client, stop attempts to connect to the serial port and contact Google support. Don't provide any credentials over a connection where host authentication has failed.

Disconnecting from the serial console

To disconnect from the serial console, follow the instructions for the method you used to connect.

Console

In the Google Cloud console, disconnect from the serial console by doing the following:

  1. Close the browser window or tab that contains the serial console connection.

gcloud

In the Google Cloud CLI, disconnect from the serial console by doing the following:

  1. Press the ENTER key.
  2. Type ~. (tilde, followed by a period).

Other SSH clients

In other SSH clients, disconnect from the serial console by doing the following:

  1. Press the ENTER key.
  2. Type ~. (tilde, followed by a period).

In the Google Cloud CLI, or using SSH, you can discover other commands by typing ~?. You can also examine the man page for SSH with the following command:

man ssh

Don't try to disconnect using any of the following methods:

Connecting to a serial console with a login prompt

If you are trying to troubleshoot an issue with a compute instance that has booted completely or trying to troubleshoot an issue that occurs after compute instance has booted past single user mode, you might be prompted for login information when trying to access the serial console.

By default, Google-supplied Linux system images are not configured to allow password-based logins for local users. However, Google-supplied Windows images are configured to allow password-based logins for local users.

If your compute instance is running an image that is preconfigured with serial port logins, you need to set up a local password on the compute instance so that you can sign in to the serial console, if prompted. You can set up a local password after connecting to the compute instance or by using a start-up script.

Setting up a local password using a startup script

You can use a startup script to set up a local password that lets you connect to the serial console during or after instance creation.

To set up a local password in an existing compute instance, select one of the following options:

Linux

  1. In the Google Cloud console, go to the VM instances page.

    Go to VM instances

  2. In the Name column, click the name of the compute instance for which you want to add a local password.

    The details page of the compute instance opens.

  3. Click Edit.

    The page to edit the details of the compute instance opens.

  4. In the Metadata > Automation section, do the following:

    1. If the compute instance has an existing startup script, then remove it and store the script somewhere safe.

    2. Add the following startup script:

      #!/bin/bash
useradd USERNAME
echo 'USERNAME:PASSWORD' | chpasswd
usermod -aG google-sudoers USERNAME

      Replace the following:

      • USERNAME: the username that you want to add.

      • PASSWORD: the password for the username. As some operating systems require minimal password length and complexity, specify a password as follows:

        • Use at least 12 characters.

        • Use a mix of upper and lower case letters, numbers, and symbols.

  5. Click Save.

    The details page of the compute instance opens.

  6. Click Reset.

  7. Connect to the serial console.

  8. When prompted, enter your login information.

Windows

  1. In the Google Cloud console, go to the VM instances page.

    Go to VM instances

  2. In the Name column, click the name of the compute instance for which you want to add a local password.

    The details page of the compute instance opens.

  3. Click Edit.

    The page to edit the details of the compute instance opens.

  4. In the Metadata section, do the following:

    1. If the compute instance has an existing startup script, then store the script somewhere safe, and then, to delete the script, click Delete item.

    2. Click Add item.

    3. In the Key field, enter windows-startup-script-cmd.

    4. In the Value field, enter the following script:

      net user USERNAME PASSWORD /ADD /Y
net localgroup administrators USERNAME /ADD

      Replace the following:

      • USERNAME: the username that you want to add.

      • PASSWORD: the password for the username. As some operating systems require minimal password length and complexity, specify a password as follows:

        • Use at least 12 characters.

        • Use a mix of upper and lower case letters, numbers, and symbols.

  5. Click Save.

    The details page of the compute instance opens.

  6. Click Reset.

  7. Connect to the serial console.

  8. When prompted, enter your login information.

After the user has been created, replace the startup script with the startup script that you stored in this section.

Setting up a local password using passwd on the compute instance

The following instructions describe how to set up a local password for a user on a compute instance so that the user can log on to the serial console of that compute instance by using the specified password.

  1. Connect to the compute instance. Replace INSTANCE_NAME with the name of your instance.

    gcloud compute ssh INSTANCE_NAME

  2. On the compute instance, create a local password with the following command. This action sets a password for the user that you are logged in as.

    sudo passwd $(whoami)

  3. Follow the prompts to create a password.

  4. Next, log out of the instance and connect to the serial console.

  5. Enter your login information when prompted.

Setting up a login on other serial ports

Login prompts are enabled on port 1 by default on all Linux public images that use systemd service management. For Windows images, Login prompts are enabled on port 2 by default and managed by Device Manager. However, port 1 can often be overwhelmed by logging data and other information being printed to the port. As an alternative, you can choose to enable a login prompt on another port, such as port 2 (ttyS1), by executing the following command on your compute instance:

Linux

For Linux operating systems using systemd:

  • Enable the service temporarily until the next reboot:

    sudo systemctl start serial-getty@ttyS1.service

  • Enable the service permanently, starting with the next reboot:

    sudo systemctl enable serial-getty@ttyS1.service

Windows

For Windows operating systems:

  • Open Command Prompt as Administrator

  • Change the EMS Port from COM2 to COM1:

    bcdedit /emssettings EMSPORT:1 EMSBAUDRATE:9600

  • Reboot the compute instance

Understanding serial port numbering

Each virtual machine instance has four serial ports. For consistency with the getSerialPortOutput API, each port is numbered 1 through 4. Linux and other similar systems number their serial ports 0 through 3. For example, on many operating system images, the corresponding devices are /dev/ttyS0 through /dev/ttyS3. Windows refers to serial ports as COM1 through COM4. To connect to what Windows considers COM3 and Linux considers ttyS2, you would specify port 3. Use the following table to help you figure out which port you want to connect to.

Virtual machine instance serial ports Standard Linux serial ports Windows COM ports
1 /dev/ttyS0 COM1
2 /dev/ttyS1 COM2
3 /dev/ttyS2 COM3
4 /dev/ttyS3 COM4

Note that many Linux images use port 1 (/dev/ttyS0) for logging messages from the kernel and system programs.

Sending a serial break

The Magic SysRq key feature lets you perform low-level tasks regardless of the system's state. For example, you can sync file systems, reboot the instance, end processes, and unmount file systems using the Magic SysRq key feature.

To send a Magic SysRq command using a simulated serial break:

  1. Press the ENTER key.
  2. Type ~B (tilde, followed by uppercase B).
  3. Type the Magic SysRq command.

Viewing serial console audit logs

Compute Engine provides audit logs to track who has connected and disconnected from an instance's serial console. To view logs, you must have permissions for the Logs Viewer or be a project viewer or editor.

  1. In the Google Cloud console, go to the Logs Explorer page.

    Go to Logs Explorer

  2. Expand the drop-down menu and select GCE VM Instance.
  3. In the search bar, type ssh-serialport.googleapis.com and press Enter.
  4. A list of audit logs appears. The logs describe connections and disconnections from a serial console. Expand any of the entries to get more information.

For any of the audit logs, you can:

  1. Expand the protoPayload property.
  2. Look for methodName to see activity this log applies to (either a connection or disconnection request). For example, if this log tracks a disconnection from the serial console, the method name would say "google.ssh-serialport.v1.disconnect". Similarly, a connection log would say "google.ssh-serialport.v1.connect". An audit log entry is recorded at the beginning and end of each session on the serial console.

There are different audit log properties for different log types. For example, audit logs relating to connections have properties that are specific to connection logs, while audit logs for disconnections have their own set of properties. There are certain audit log properties that are also shared between both log types.

All serial console logs

The following table provides audit log properties and their values for all serial console logs: Property Value requestMetadata.callerIp The IP address and port number from which the connection originated.

serviceName ssh-serialport.googleapis.com

resourceName A string containing the project ID, zone, instance name, and serial port number to indicate which serial console this pertains to. For example, projects/myproject/zones/us-east1-a/instances/example-instance/SerialPort/2 is port number 2, also known as COM2 or /dev/ttyS1, for the instance example-instance.

resource.labels Properties identifying the instance ID, zone, and project ID.

timestamp A timestamp indicating when the session began or ended.

severity NOTICE

operation.id An ID string uniquely identifying the session; you can use this to associate a disconnect entry with the corresponding connection entry.

operation.producer ssh-serialport.googleapis.com

Connection logs

The following table provides audit log properties and their values specific for connection logs: Property Value methodName google.ssh-serialport.v1.connect

status.message Connection succeeded.

request.serialConsoleOptions Any options that were specified with the request, including the serial port number.

request.@type type.googleapis.com/google.compute.SerialConsoleSessionBegin

request.username The username specified for this request. This is used to select the public key to match.

operation.first TRUE

status.code For successful connection requests, a status.code value of google.rpc.Code.OK indicates that the operation completed successfully without any errors. Because the enum value for this property is 0, the status.code property is not displayed. However, any code that checks for a status.code value of google.rpc.Code.OK will work as expected.

Disconnection logs

The following table provides audit log properties and their values specific for disconnection logs:

Property Value
methodName google.ssh-serialport.v1.disconnect
response.duration The amount of time, in seconds, that the session lasted.
response.@type type.googleapis.com/google.compute.SerialConsoleSessionEnd
operation.last TRUE

Failed connection logs

When a connection fails, Compute Engine creates an audit log entry. A failed connection log looks very similar to a successful connection entry, but has the following properties to indicate a failed connection.

Property Value
severity ERROR
status.code

The canonical Google API error code that best describes the error. The following are possible error codes that might appear:
status.message The human-readable message for this entry.

Disabling interactive serial console access

You can disable interactive serial console access by changing metadata on the specific instance or project, or by setting an Organization Policy that disables interactive serial console access to all compute instances for one or more projects that are part of the organization.

Disabling interactive serial console on a particular instance or project

Project owners and editors, as well as users who have been granted the compute.instanceAdmin.v1 role, can disable access to the serial console by changing the metadata on the particular instance or project. Similar to enabling serial console access, set the serial-port-enable metadata to FALSE:

serial-port-enable=FALSE

For example, using the Google Cloud CLI, you can apply this metadata to a specific instance like so:

gcloud compute instances add-metadata instance-name \
    --metadata=serial-port-enable=FALSE

To apply the metadata to the project:

gcloud compute project-info add-metadata \
    --metadata=serial-port-enable=FALSE

Disabling interactive serial console access through Organization Policy

If you have been granted the orgpolicy.policyAdmin role on the organization, you can set an organization policy that prevents interactive access to the serial console, regardless of whether interactive serial console access is enabled on the metadata server. After the organization policy is set, the policy effectively overrides the serial-port-enable metadata key, and no users of the organization or project can enable interactive serial console access. By default, this constraint is set to FALSE.

The constraint for disabling interactive serial console access is as follows:

compute.disableSerialPortAccess

Complete the following instructions to set this policy on the organization. After setting up a policy, you can grant exemptions on a per-project basis.

gcloud

To set the policy using the Google Cloud CLI, run the resource-manager enable-enforce command. Replace organization-id with your organization ID. For example, 1759840282.

gcloud resource-manager org-policies enable-enforce \
    --organization organization-id compute.disableSerialPortAccess

REST

To set a policy in the API, make a POST request to the following URL. Replace organization-name with your organization name. For example, organizations/1759840282.

 POST https://cloudresourcemanager.googleapis.com/v1/organization-name:setOrgPolicy

The request body should contain a policy object with the following constraint:

"constraint": "constraints/compute.disableSerialPortAccess"

For example:

 {
   "policy":
   {
     "booleanPolicy":
     {
       "enforced": TRUE
     },
     "constraint": "constraints/compute.disableSerialPortAccess"
   }
 }

The policy is immediately effective, so any projects under the organization immediately stop allowing interactive access to the serial console.

To temporarily disable the policy, use the disable-enforce command:

gcloud resource-manager org-policies disable-enforce \
    --organization organization-id compute.disableSerialPortAccess

Alternatively, you can make an API request where the request body sets the enforced parameter to FALSE:

{
  "policy":
  {
    "booleanPolicy":
    {
      "enforced": FALSE
    },
    "constraint": "constraints/compute.disableSerialPortAccess"
  }
}

Setting the organization policy at the project level

You can set the same organizational policy on a per-project basis. This overrides the setting at the organization level.

gcloud

To turn off enforcement of this policy for a specific project. Replace project-id with your project ID.

gcloud resource-manager org-policies disable-enforce \
    --project project-id compute.disableSerialPortAccess

You can turn on enforcement of this policy by using the enable-enforce command with the same values.

REST

In the API, make a POST request to the following URL to enable interactive serial console access for the project, replacing project-id with the project ID:

POST https://cloudresourcemanager.googleapis.com/v1/projects/project-id:setOrgPolicy

The request body should contain a policy object with the following constraint:

"constraint": "constraints/compute.disableSerialPortAccess"

For example:

{
  "policy":
  {
    "booleanPolicy":
    {
      "enforced": FALSE
    },
    "constraint": "constraints/compute.disableSerialPortAccess"
  }
}

Troubleshooting Web UI connections

If you encounter an "SSH authentication failed" error or are redirected to a generic SSH error page when using the Connect to serial console button, check the following:

Tips and tricks

Advanced options

You can also use the following advanced options with the serial port.

Controlling max connections

You can set the max-connections property to control how many concurrent connections can be made to this serial port at a time. The default and maximum number of connections is 5. For example:

gcloud compute connect-to-serial-port instance-name \
    --port port-number \
    --extra-args max-connections=3
 
ssh -i private-ssh-key-file -p 9600 project-id.zone.instance-name.username.max-connections=3@ssh-serialport.googleapis.com

Setting replay options

By default, each time you connect to the serial console, you will receive a replay of the last 10 lines of data, regardless of whether the last 10 lines have been seen by another SSH client. You can change this setting and control how many and which lines are returned by setting the following options:

With the Google Cloud CLI, append the following to your connect-to-serial-port command, where N is the specified number of lines (or bytes or absolute byte index, depending on which replay option you are selecting):

--extra-args replay-lines=N

If you are using a third-party SSH client, provide this option in your SSH command:

ssh -i private-ssh-key-file -p 9600 myproject.us-central1-f.example-instance.jane.port=3.replay-lines=N@ssh-serialport.googleapis.com

You can also use a combination of these options as well. For example:

replay-lines=N and replay-bytes=new
Replay the specified number of lines OR replay all output not previously sent to any client, whichever is larger. The first client to connect with this flag combination will see all the output that has been sent to the serial port, and clients that connect subsquently will only see the last N lines. Examples:
gcloud compute connect-to-serial-port instance-name--port port-number --extra-args replay-lines=N,replay-bytes=new
 
ssh -i private-ssh-key-file -p 9600 project-id.zone.instance-name.username.replay-lines=N.replay-bytes=new@ssh-serialport.googleapis.com
replay-lines=N and replay-bytes=M
Replay lines up to, but not more than, the number of lines or bytes described by these flags, whichever is less. This option won't replay more than N or M bytes.
gcloud compute connect-to-serial-port instance-name--port port-number --extra-args replay-lines=N,replay-bytes=M
 
ssh -i private-ssh-key-file -p 9600 project-id.zone.instance-name.username.replay-lines=N.replay-bytes=M@ssh-serialport.googleapis.com

Handling dropped output

The most recent 1 MiB of output for each serial port is always available and generally, your SSH client shouldn't miss any output from the serial port. If, for some reason, your SSH client stops accepting output for a period of time but does not disconnect, and more than 1 MiB of new data is produced, your SSH client might miss some output. When your SSH client is not accepting data fast enough to keep up with the output on the serial console port, you can set the on-dropped-output property to determine how the console behaves.

Set any of the following applicable options with this property:

For example:

gcloud compute connect-to-serial-port instance-name \
    --port port-number \
    --extra-args on-dropped-output=ignore
 
ssh -i private-ssh-key-file -p 9600 project-id.zone.instance-name.username.on-dropped-output=ignore@ssh-serialport.googleapis.com

Enabling disconnect using exit or logout commands

You can enable disconnecting on exit or logout commands by setting the on-dtr-low property to disconnect when you connect to the serial console.

On the Google Cloud CLI, append the following flag to your connect-to-serial-port command:

--extra-args on-dtr-low=disconnect

If you are using a third-party SSH client, provide this option in your SSH command:

ssh -i private-ssh-key-file -p 9600 myproject.us-central1-f.example-instance.jane.port=3.on-dtr-low=disconnect@ssh-serialport.googleapis.com

Enabling the disconnect option might cause your instance to disconnect one or more times when you are rebooting the instance because the operating system resets the serial ports while booting up.

The default setting for the on-dtr-low option is none. If you use the default setting none, you can reboot your instance without being disconnected from the serial console, but the console won't disconnect through normal means such as exit or logout commands, or normal key combinations like Ctrl+D.

What's next