Skip to main content
Last updated on

Data Export Overview

The data export feature writes query result sets or Apache Doris table data to a specified storage system in a specified file format. It is commonly used for result set downloads and cross-system data exchange.

Data Export vs. Data Backup

Both data export and data backup can output data from Apache Doris to external storage, but they target different scenarios. The following table compares the core differences between the two:

Comparison DimensionData ExportData BackupFinal storage locationHDFS, object storage, local file systemHDFS, object storageData formatOpen formats such as Parquet, ORC, CSVApache Doris internal storage formatExecution speedMedium (requires reading data and converting to the target format)Fast (uploads Doris data files directly, no parsing or conversion required)FlexibilityAllows flexible export scope definition through SQLSupports only table-level full backupTypical use scenariosResult set download, data exchange between different systemsData backup, data migration between Apache Doris clusters

Choosing an Export Method

Apache Doris provides the following three data export methods, each suited to different export needs:

Capability Comparison of the Three Methods

The following table compares the three export methods across execution mode, SQL capabilities, concurrency, and supported formats, helping you choose quickly:

Comparison DimensionSELECT INTO OUTFILEEXPORTMySQL DUMPSynchronous / AsynchronousSynchronousAsynchronous (check progress with SHOW EXPORT after submission)SynchronousSupports arbitrary SQLSupportedNot supportedNot supportedExport specified partitionsSupportedSupportedNot supportedExport specified tabletsSupportedNot supportedNot supportedConcurrent exportSupported, high concurrency (limited by single-node operators such as ORDER BY)Supported, high concurrency (Tablet-level concurrency)Not supported, single-threaded exportSupported export formatsParquet, ORC, CSVParquet, ORC, CSVMySQL Dump proprietary formatSupports external tablesSupportedPartially supportedNot supportedSupports viewsSupportedSupportedSupportedSupported export locationsS3, HDFSS3, HDFSLOCAL

Applicable Scenarios

SELECT INTO OUTFILE

Suitable for the following scenarios:

For detailed usage, see SELECT INTO OUTFILE.

EXPORT

Suitable for the following scenarios:

For detailed usage, see Export Asynchronous Export.

MySQL Dump

Suitable for the following scenarios:

For detailed usage, see MySQL Dump.

Column Type Mapping for Exported Files

The Parquet and ORC file formats have their own data type definitions, and Apache Doris automatically converts internal data types to the corresponding Parquet/ORC types during export. The CSV format has no type definitions; all data is output as text.

The mappings between Apache Doris data types and the ORC and Parquet formats are listed below.

ORC Type Mapping

Doris TypeORC TypebooleanbooleantinyinttinyintsmallintsmallintintintbigintbigintlargeIntstringdatestringdatev2stringdatetimestringdatetimev2timestampfloatfloatdoubledoublechar / varchar / stringstringdecimaldecimalstructstructmapmaparrayarrayjsonstringvariantstringbitmapbinaryquantile_statebinaryhllbinary

Parquet Type Mapping

When Apache Doris exports to the Parquet file format, it first converts the in-memory Doris data to the Arrow in-memory format, and then Arrow writes it out to the Parquet file. The mappings are as follows:

Doris TypeArrow TypeParquet Physical TypeParquet Logical TypebooleanbooleanBOOLEANtinyintint8INT32INT_8smallintint16INT32INT_16intint32INT32INT_32bigintint64INT64INT_64largeIntutf8BYTE_ARRAYUTF8dateutf8BYTE_ARRAYUTF8datev2date32INT32DATEdatetimeutf8BYTE_ARRAYUTF8datetimev2timestampINT96 / INT64TIMESTAMP(MICROS/MILLIS/SECONDS)floatfloat32FLOATdoublefloat64DOUBLEchar / varchar / stringutf8BYTE_ARRAYUTF8decimaldecimal128FIXED_LEN_BYTE_ARRAYDECIMAL(scale, precision)structstructParquet GroupmapmapParquet MaparraylistParquet Listjsonutf8BYTE_ARRAYUTF8variantutf8BYTE_ARRAYUTF8bitmapbinaryBYTE_ARRAYquantile_statebinaryBYTE_ARRAYhllbinaryBYTE_ARRAY
note

In versions 2.1.11 and 3.0.7, the parquet.enable_int96_timestamps property is supported to specify whether the Doris datetimev2 type is stored as INT96 or INT64 in Parquet. The default is INT96. INT96 has been deprecated in the Parquet standard and is used only for compatibility with legacy systems (such as Hive versions before 4.0).

Report issue
Doris Homepage
Ask Questions on Discussion
Chat on Slack
Chat on Discord
On This Page