Console
Start free

Import from BigQuery

You can create data stores from BigQuery tables in two ways:

The following table compares the two ways that you can import BigQuery data into Gemini Enterprise data stores.

One-time ingestion Periodic ingestion
Generally available (GA). Public preview.
Data must be refreshed manually. Data updates automatically every 1, 3, or 5 days. Data cannot be manually refreshed.
Gemini Enterprise creates a single data store from one table in a BigQuery. Gemini Enterprise creates a data connector for a BigQuery dataset and a data store (called an entity data store) for each table specified. For each data connector, the tables must have the same data type (for example, structured) and be in the same BigQuery dataset.
Data from multiple tables can be combined in one data store by first ingesting data from one table and then more data from another source or BigQuery table. Because manual data import is not supported, the data in an entity data store can only be sourced from one BigQuery table.
Data source access control is supported. Data source access control is not supported. The imported data can contain access controls but these controls won't be respected.
You can create a data store using either the Google Cloud console or the API. You must use the console to create data connectors and their entity data stores.
CMEK-compliant. CMEK-compliant.

Before you begin

To import data from a source Google Cloud project that's different from the Google Cloud project with the Gemini Enterprise data store, grant the following Identity and Access Management (IAM) roles to the service-PROJECT_NUMBER@gcp-sa-discoveryengine.iam.gserviceaccount.com service account in the project that contains the Gemini Enterprise data store:

Import once from BigQuery

To ingest data from a BigQuery table, use the following steps to create a data store and ingest data using either the Google Cloud console or the API.

Before importing your data, review Prepare data for ingesting.

Console

To use the Google Cloud console to ingest data from BigQuery, follow these steps:

  1. In the Google Cloud console, go to the Gemini Enterprise page.

    Gemini Enterprise

  2. Go to the Data Stores page.

  3. Click Create Data Store.

  4. On the Select a data source page, select BigQuery.

  5. Select what kind of data you are importing.

  6. Click One time.

  7. In the BigQuery path field, click Browse, select a table that you have prepared for ingesting, and then click Select. Alternatively, enter the table location directly in the BigQuery path field.

  8. Click Continue.

  9. If you are doing one-time import of structured data:

    1. Map fields to key properties.

    2. If there are important fields missing from the schema, use Add new field to add them.

      For more information, see About auto-detect and edit.

    3. Click Continue.

  10. Choose a region for your data store.

  11. Enter a name for your data store.

  12. Click Create.

  13. To check the status of your ingestion, go to the Data Stores page and click your data store name to see details about it on its Data page. When the status column on the Activity tab changes from In progress to Import completed, the ingestion is complete.

    Depending on the size of your data, ingestion can take several minutes to several hours.

REST

To use the command line to create a data store and import data from BigQuery, follow these steps.

  • Create a data store.

    curl -X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
-H "X-Goog-User-Project: PROJECT_ID" \
"https://discoveryengine.googleapis.com/v1/projects/PROJECT_ID/locations/global/collections/default_collection/dataStores?dataStoreId=DATA_STORE_ID" \
-d '{
  "displayName": "DATA_STORE_DISPLAY_NAME",
  "industryVertical": "GENERIC",
  "solutionTypes": ["SOLUTION_TYPE_SEARCH"]
}'

    Replace the following:

    Optional: If you're uploading unstructured data and want to configure document parsing or to turn on document chunking for RAG, specify the documentProcessingConfig object and include it in your data store creation request. Configuring an OCR parser for PDFs is recommended if you're ingesting scanned PDFs. For how to configure parsing or chunking options, see Parse and chunk documents.

    • Import data from BigQuery.

    If you defined a schema, make sure the data conforms to that schema.

    curl -X POST \
-H "Authorization: Bearer $(gcloud auth print-access-token)" \
-H "Content-Type: application/json" \
"https://discoveryengine.googleapis.com/v1/projects/PROJECT_ID/locations/global/collections/default_collection/dataStores/DATA_STORE_ID/branches/0/documents:import" \
-d '{
  "bigquerySource": {
    "projectId": "PROJECT_ID",
    "datasetId":"DATASET_ID",
    "tableId": "TABLE_ID",
    "dataSchema": "DATA_SCHEMA",
    "aclEnabled": "BOOLEAN"
  },
  "reconciliationMode": "RECONCILIATION_MODE",
  "autoGenerateIds": "AUTO_GENERATE_IDS",
  "idField": "ID_FIELD",
  "errorConfig": {
    "gcsPrefix": "ERROR_DIRECTORY"
  }
}'

    Replace the following:

  • PROJECT_ID: the ID of your project.
  • DATA_STORE_ID: the ID of the data store.
  • DATASET_ID: the ID of the BigQuery dataset.
  • TABLE_ID: the ID of the BigQuery table.
  • DATA_SCHEMA: optional. Values are document and custom. The default is document.
  • ERROR_DIRECTORY: optional. A Cloud Storage directory for error information about the import—for example, gs://<your-gcs-bucket>/directory/import_errors. Google recommends leaving this field empty to let Gemini Enterprise automatically create a temporary directory.
  • RECONCILIATION_MODE: optional. Values are FULL and INCREMENTAL. Default is INCREMENTAL. Specifying INCREMENTAL causes an incremental refresh of data from BigQuery to your data store. This does an upsert operation, which adds new documents and replaces existing documents with updated documents with the same ID. Specifying FULL causes a full rebase of the documents in your data store. In other words, new and updated documents are added to your data store, and documents that are not in BigQuery are removed from your data store. The FULL mode is helpful if you want to automatically delete documents that you no longer need.

  • AUTO_GENERATE_IDS: optional. Specifies whether to automatically generate document IDs. If set to true, document IDs are generated based on a hash of the payload. Note that generated document IDs might not remain consistent over multiple imports. If you auto-generate IDs over multiple imports, Google highly recommends setting reconciliationMode to FULL to maintain consistent document IDs.

    Specify autoGenerateIds only when bigquerySource.dataSchema is set to custom. Otherwise an INVALID_ARGUMENT error is returned. If you don't specify autoGenerateIds or set it to false, you must specify idField. Otherwise the documents fail to import.

    • ID_FIELD: optional. Specifies which fields are the document IDs. For BigQuery source files, idField indicates the name of the column in the BigQuery table that contains the document IDs.

    Specify idField only when: (1) bigquerySource.dataSchema is set to custom, and (2) auto_generate_ids is set to false or is unspecified. Otherwise an INVALID_ARGUMENT error is returned.

    The value of the BigQuery column name must be of string type, must be between 1 and 63 characters, and must conform to RFC-1034. Otherwise, the documents fail to import.

    Connect to BigQuery with periodic syncing

    Before importing your data, review Prepare data for ingesting.

    The following procedure describes how to create a BigQuery data store that periodically syncs data from a BigQuery dataset. If your dataset has multiple tables, you can add them to the BigQuery data store you are creating. Each table you add is referred to as an entity. Gemini Enterprise creates a separate data store for each entity. Therefore, when you create the data store using the Google Cloud console, you get a collection of data stores representing these ingested data entities.

    Data from the dataset is synced periodically to the entity data stores. You can specify synchronization daily, every three days, or every five days.

    Console

    To create a data store that periodically syncs data from a BigQuery dataset to Gemini Enterprise, follow these steps:

    1. In the Google Cloud console, go to the Gemini Enterprise page.

      Gemini Enterprise

    2. In the navigation menu, click Data Stores.

    3. Click Create Data Store.

    4. On the Source page, select BigQuery.

    5. Select the kind of data that you are importing.

    6. Click Periodic.

    7. Select the Sync frequency, how often you want the Gemini Enterprise connector to sync with the BigQuery dataset. You can change the frequency later.

    8. In the BigQuery dataset path field, click Browse, select the dataset that contains the tables that you have prepared for ingesting. Alternatively, enter the table location directly in the BigQuery path field. The format for the path is projectname.datasetname.

    9. In the Tables to sync field, click Browse, and then select a table that contains the data that you want for your data store.

    10. If there are additional tables in the dataset that you want to use for data stores, click Add table and specify those tables too.

    11. Click Continue.

    12. Choose a region for your data store, enter a name for your data connector, and click Create.

      You have now created a data connector, which will periodically sync data with the BigQuery dataset. And, you have created one or more entity data stores. The data stores have the same names as the BigQuery tables.

    13. To check the status of your ingestion, go to the Data Stores page and click your data connector name to see details about it on its Data page > Data ingestion activity tab. When the status column on the Activity tab changes from In progress to succeeded, the first ingestion is complete.

      Depending on the size of your data, ingestion can take several minutes to several hours.

    After you set up your data source and import data the first time, the data store syncs data from that source at a frequency that you select during setup. About an hour after the data connector is created, the first sync occurs. The next sync then occurs around 24 hours, 72 hours, or 120 hours later.

    Next steps