TPC-DS Benchmark
TPC-DS (Transaction Processing Performance Council Decision Support Benchmark) is a benchmark test that focuses on decision support and aims to evaluate the performance of data warehousing and analytics systems. It was developed by the Transaction Processing Performance Council (TPC) organization to compare the capabilities of different systems in handling complex queries and large-scale data analysis.
The design goal of TPC-DS is to simulate complex decision support workloads in the real world. It tests the performance of systems through a series of complex queries and data operations, including joins, aggregations, sorting, filtering, subqueries, and more. These query patterns cover various scenarios ranging from simple to complex, such as report generation, data mining, and OLAP (Online Analytical Processing).
This document mainly introduces the performance of Doris on the TPC-DS 1000G test set.
On 99 queries on the TPC-DS standard test data set, we conducted a comparison test based on Apache Doris 2.1.7-rc03 and Apache Doris 2.0.15.1 versions.
1. Hardware Environmentβ
| Hardware | Configuration Instructions |
|---|---|
| Number of Machines | 4 Aliyun Virtual Machine (1FEοΌ3BEs) |
| CPU | Intel Xeon (Ice Lake) Platinum 8369B 32C |
| Memory | 128G |
| Disk | Enterprise SSD (PL0) |
2. Software Environmentβ
- Doris Deployed 3BEs and 1FE
- Kernel Version: Linux version 5.15.0-101-generic
- OS version: Ubuntu 20.04 LTS (Focal Fossa)
- Doris software version: Apache Doris 2.1.7-rc03, Apache Doris 2.0.15.1
- JDK: openjdk version "1.8.0_352-352"
3. Test Data Volumeβ
The TPC-DS 1000G data generated by the simulation of the entire test are respectively imported into Apache Doris 2.1.7-rc03 and Apache Doris 2.0.15.1 for testing. The following is the relevant description and data volume of the table.
| TPC-DS Table Name | Rows |
|---|---|
| customer_demographics | 1,920,800 |
| reason | 65 |
| warehouse | 20 |
| date_dim | 73,049 |
| catalog_sales | 1,439,980,416 |
| call_center | 42 |
| inventory | 783,000,000 |
| catalog_returns | 143,996,756 |
| household_demographics | 7,200 |
| customer_address | 6,000,000 |
| income_band | 20 |
| catalog_page | 30,000 |
| item | 300,000 |
| web_returns | 71,997,522 |
| web_site | 54 |
| promotion | 1,500 |
| web_sales | 720,000,376 |
| store | 1,002 |
| web_page | 3,000 |
| time_dim | 86,400 |
| store_returns | 287,999,764 |
| store_sales | 2,879,987,999 |
| ship_mode | 20 |
| customer | 12,000,000 |
4. Test SQLβ
TPC-DS 99 test query statements : TPC-DS-Query-SQL
5. Test Resultsβ
Here we use Apache Doris 2.1.7-rc03 and Apache Doris 2.0.15.1 for comparative testing. In the test, we use Query Time(ms) as the main performance indicator. The test results are as follows: (Apache Doris 2.0.15.1 q78 q79 failed to execute due to lack of latest memory optimization and was removed when calculating the total sum)
| Query | Apache Doris 2.1.7-rc03 (ms) | Apache Doris 2.0.15.1 (ms) |
|---|---|---|
| query01 | 630 | 890 |
| query02 | 4930 | 6930 |
| query03 | 360 | 460 |
| query04 | 11070 | 42320 |
| query05 | 620 | 15360 |
| query06 | 220 | 1020 |
| query07 | 550 | 750 |
| query08 | 330 | 670 |
| query09 | 6830 | 7550 |
| query10 | 370 | 2900 |
| query11 | 6960 | 27380 |
| query12 | 100 | 80 |
| query13 | 790 | 2860 |
| query14 | 13470 | 42340 |
| query15 | 510 | 940 |
| query16 | 520 | 550 |
| query17 | 1310 | 2650 |
| query18 | 560 | 820 |
| query19 | 200 | 400 |
| query20 | 100 | 190 |
| query21 | 80 | 80 |
| query22 | 2300 | 3070 |
| query23 | 38240 | 75260 |
| query24 | 8340 | 26580 |
| query25 | 780 | 1190 |
| query26 | 200 | 220 |
| query27 | 530 | 750 |
| query28 | 5940 | 7400 |
| query29 | 940 | 1250 |
| query30 | 270 | 490 |
| query31 | 1890 | 2530 |
| query32 | 60 | 70 |
| query33 | 350 | 450 |
| query34 | 750 | 1380 |
| query35 | 1370 | 8970 |
| query36 | 530 | 570 |
| query37 | 60 | 60 |
| query38 | 7520 | 8710 |
| query39 | 560 | 1010 |
| query40 | 150 | 180 |
| query41 | 50 | 40 |
| query42 | 100 | 140 |
| query43 | 1150 | 1960 |
| query44 | 2020 | 3220 |
| query45 | 430 | 960 |
| query46 | 1250 | 2760 |
| query47 | 2660 | 5790 |
| query48 | 630 | 2570 |
| query49 | 730 | 800 |
| query50 | 1640 | 2200 |
| query51 | 6430 | 6270 |
| query52 | 110 | 160 |
| query53 | 250 | 490 |
| query54 | 1280 | 7790 |
| query55 | 110 | 160 |
| query56 | 290 | 410 |
| query57 | 1480 | 3510 |
| query58 | 240 | 550 |
| query59 | 7760 | 11870 |
| query60 | 380 | 490 |
| query61 | 540 | 670 |
| query62 | 740 | 1560 |
| query63 | 210 | 460 |
| query64 | 5790 | 6840 |
| query65 | 4900 | 7960 |
| query66 | 480 | 810 |
| query67 | 27320 | 46110 |
| query68 | 1600 | 2380 |
| query69 | 380 | 800 |
| query70 | 3480 | 5330 |
| query71 | 460 | 790 |
| query72 | 3160 | 5390 |
| query73 | 660 | 1250 |
| query74 | 5990 | 16450 |
| query75 | 4610 | 8410 |
| query76 | 1590 | 2950 |
| query77 | 300 | 480 |
| query78 | 17970 | - |
| query79 | 3040 | - |
| query80 | 570 | 910 |
| query81 | 460 | 760 |
| query82 | 270 | 330 |
| query83 | 220 | 290 |
| query84 | 130 | 110 |
| query85 | 520 | 470 |
| query86 | 760 | 1220 |
| query87 | 800 | 8760 |
| query88 | 5560 | 9690 |
| query89 | 430 | 750 |
| query90 | 150 | 400 |
| query91 | 150 | 120 |
| query92 | 40 | 40 |
| query93 | 2440 | 2670 |
| query94 | 340 | 310 |
| query95 | 350 | 1810 |
| query96 | 660 | 1680 |
| query97 | 5020 | 14990 |
| query98 | 190 | 330 |
| query99 | 1560 | 3230 |
| Total | 261320 | 507380 |
6. Environmental Preparationβ
Please refer to the official document to install and deploy Doris to obtain a normal running Doris cluster (at least 1 FE 1 BE, 1 FE 3 BE is recommended).
7. Data Preparationβ
7.1 Download and Install TPC-DS Data Generation Toolβ
Execute the following script to download and compile the tpcds-tools tool.
sh bin/build-tpcds-dbgen.sh
7.2 Generating the TPC-DS Test Setβ
Execute the following script to generate the TPC-H dataset:
sh bin/gen-tpcds-data.sh -s 1000
Note 1: Check the script help via
sh gen-tpcds-data.sh -h.Note 2: The data will be generated under the
tpcds-data/directory with the suffix.dat. The total file size is about 1000GB and may need a few minutes to an hour to generate.Note 3: A standard test data set of 100G is generated by default.
7.3 Create Tableβ
7.3.1 Prepare the doris-cluster.conf Fileβ
Before import the script, you need to write the FEβs ip port and other information in the doris-cluster.conf file.
The file is located under ${DORIS_HOME}/tools/tpcds-tools/conf/ .
The content of the file includes FE's ip, HTTP port, user name, password and the DB name of the data to be imported:
# Any of FE host
export FE_HOST='127.0.0.1'
# http_port in fe.conf
export FE_HTTP_PORT=8030
# query_port in fe.conf
export FE_QUERY_PORT=9030
# Doris username
export USER='root'
# Doris password
export PASSWORD=''
# The database where TPC-H tables located
export DB='tpcds'
Execute the Following Script to Generate and Create TPC-H Tableβ
sh bin/create-tpcds-tables.sh -s 1000
Or copy the table creation statement in create-tpcds-tables and excute it in Doris.
7.4 Import Dataβ
Please perform data import with the following command:
sh bin/load-tpcds-data.sh
7.5 Query Testβ
7.5.1 Executing Query Scriptsβ
Execute the above test SQL or execute the following command
sh bin/run-tpcds-queries.sh -s 1000
7.5.2 Single SQL Executionβ
You can also retrieve the latest SQL from the code repository. The address for the latest test query statements of TPC-DS.
