Blog/Release Notes

Apache Doris 3.0.1 just released

Apache DorisAugust 23, 2024

Dear community members, the Apache Doris 3.0.1 version was officially released on August 23, 2024, featuring updates and improvements in compute-storage decoupling, lakehouse, semi-structured data analysis, asynchronous materialized views, and more.

Quick Download: https://doris.apache.org/download/

GitHub Release: https://github.com/apache/doris/releases

Behavior Changes

Query Optimizer

• Added the variable use_max_length_of_varchar_in_ctas to control the length behavior of VARCHAR type when executing CREATE TABLE AS SELECT (CTAS) operations. #37069

  • This variable is set to true by default.

  • When set to true, if the VARCHAR type column originates from a table, the derived length is used; otherwise, the maximum length is used.

  • When set to false, the VARCHAR type will always use the derived length.

• All data types will now be displayed in lowercase to maintain compatibility with MySQL format. #38012

• Multiple query statements in the same query request must now be separated by semicolons. #38670

Query Execution

• The default number of parallel tasks after shuffle operations in the cluster is set to 100, which will improve query stability and concurrent processing capability in large clusters. #38196

Storage

• The default value of trash_file_expire_time_sec has been changed from 86400 seconds to 0 seconds, which means that if files are deleted by mistake and the FE trash is cleared, the data cannot be recovered.

• The table attribute enable_mow_delete_on_delete_predicate (introduced in version 3.0.0) has been renamed to enable_mow_light_delete.

• Explicit transactions are now prohibited from performing delete operations on tables with written data.

• Heavy schema change operations are prohibited on tables with auto-increment fields.

New Features

Job Scheduling

• Optimized the execution logic of internal scheduling jobs, decoupling the strong association between start time and immediate execution parameters. Now, tasks can be created with a specified start time or selected for immediate execution, without conflict, enhancing scheduling flexibility. #36805

Compute-Storage Decoupled

• Supports dynamic modification of the upper limit for file cache usage. #37484

• Recycler now supports object storage rate limiting and server-side rate limiting retry functionality. #37663 #37680

Lakehouse

• Added the session variable serde_dialect to set the output format for complex types. #37039

• SQL interception now supports external tables.

  • For more information, refer to the documentation on SQL Interception.

• Insert overwrite now supports Iceberg tables. #37191

Asynchronous Materialized Views

• Supports partition roll-up and build at the hourly level. #37678

• Supports atomic replacement of asynchronous materialized view definition statements. #36749

• Transparent rewriting now supports Insert statements. #38115

• Transparent rewriting now supports the VARIANT type. #37929

Query Execution

• The group concat function now supports DISTINCT and ORDER BY options. #38744

Semi-Structured Data Management

• The ES Catalog now maps nested or object types in Elasticsearch to the JSON type in Doris. #37101

• Added the MULTI_MATCH function, which supports matching keywords across multiple fields and can leverage inverted indexes to accelerate searches. #37722

• Added the explode_json_object function, which can unfold objects in JSON data into multiple rows. #36887

• Inverted indexes now support memtable advancement, requiring index construction only once during multi-replica writes, reducing CPU consumption and improving performance. #35891

• Added MATCH_PHRASE support for positive slop, e.g., msg MATCH_PHRASE 'a b 2+' can match instances containing words a and b with a slop of no more than two, and a preceding b; regular slop without the final + does not guarantee this order. #36356

Other

• Added the FE parameter skip_audit_user_list, where user operations specified in this configuration will not be recorded in the audit log. #38310

  • For more information, refer to the documentation on Audit Plugin.

Improvements

Storage

• Reduced the likelihood of write failures caused by disk balancing within a single BE. #38000

• Decreased memory consumption by the memtable limiter. #37511

• Moved old partitions to the FE trash during partition replacement operations. #36361

• Optimized memory consumption during compaction. #37099

• Added a session variable to control audit logs for JDBC PreparedStatement, with default setting to not print. #38419

• Optimized the logic for selecting BEs for group commits. #35558

• Improved the performance of column updates. #38487

• Optimized the use of delete bitmap cache. #38761

• Added a configuration to control query affinity during hot and cold tiering. #37492

Compute-Storage Decoupled

• Implemented automatic retries when encountering object storage server rate limiting. #37199

• Adapted the number of threads for memtable flush in the compute-storage decoupled mode. #38789

• Added Azure as a compile option to support compilation in environments without Azure support.

• Optimized the observability of object storage access rate limiting. #38294

• Allowed the file cache TTL queue to perform LRU eviction, enhancing TTL queue usability. #37312

• Optimized the number of balance writeeditlog IO operations in the storage and compute separation mode. #37787

• Improved table creation speed in the storage and compute separation mode by sending tablet creation requests in batches. #36786

• Optimized read failures caused by potential inconsistencies in the local file cache through backoff retries. #38645

Lakehouse

• Optimized memory statistics for Parquet/ORC format read and write operations. #37234

• Trino Connector Catalog now supports predicate pushdown. #37874

• Added a session variable enable_count_push_down_for_external_table to control whether to enable count(*) pushdown optimization for external tables. #37046

• Optimized the read logic for Hudi snapshot reads, returning an empty set when the snapshot is empty, consistent with Spark behavior. #37702

• Improved the read performance of partition columns for Hive tables. #37377

Asynchronous Materialized Views

• Improved transparent rewrite plan speed by 20%. #37197

• Eliminated roll-up during transparent rewrite if the group key satisfies data uniqueness for better nested matching. #38387

• Transparent rewrite now performs better aggregation elimination to improve the matching success rate of nested materialized views. #36888

MySQL Compatibility

• Now correctly populates the database name, table name, and original name in the MySQL protocol result columns. #38126

• Supported the hint format /*+ func(value) */. #37720

Query Optimizer

• Significantly improved the plan speed for complex queries. #38317

• Adaptively chose whether to perform bucket shuffle based on the number of data buckets to avoid performance degradation in extreme cases. #36784

• Optimized the cost estimation logic for SEMI / ANTI JOIN. #37951 #37060

• Supported pushing Limit down to the first stage of aggregation to improve performance. #34853

• Partition pruning now supports filter conditions containing the date_trunc or date function. #38025 #38743

• SQL cache now supports query scenarios that include user variables. #37915

• Optimized error messages for invalid aggregation semantics. #38122

Query Execution

• Adapted AggState compatibility from 2.1 to 3.x and fixed Coredump issues. #37104

• Refactored the strategy selection for local shuffle without Join. #37282

• Modified the scanner for internal table queries to be asynchronous to prevent stalling during such queries. #38403

• Optimized the block merge process during Hash table construction for Join operators. #37471

• Optimized the duration of lock holding for MultiCast. #37462

• Optimized gRPC keepAliveTime and added link monitoring to reduce the probability of query failure due to RPC errors. #37304

• Cleaned up all dirty pages in jemalloc when memory limits were exceeded. #37164

• Optimized the processing performance of aes_encrypt/decrypt functions for constant types. #37194

• Optimized the processing performance of the json_extract function for constant data. #36927

• Optimized the processing performance of the ParseUrl function for constant data. #36882

Semi-Structured Data Management

• Bitmap indexes now default to using inverted indexes, with enable_create_bitmap_index_as_inverted_index set to true by default. #36692

• In the compute-storage decoupled mode, DESC can now view sub-columns of VARIANT type. #38143

• Removed the step of checking file existence during inverted index queries to reduce access latency to remote storage. #36945

• Complex types ARRAY / MAP / STRUCT now support replace_if_not_null for AGG tables. #38304

• Escape characters for JSON data are now supported. #37176 #37251

• Inverted index queries now behave consistently on MOW tables and DUP tables. #37428

• Optimized the performance of inverted index acceleration for IN queries. #37395

• Reduced unnecessary memory allocation during TOPN queries to improve performance. #37429

• When creating an inverted index with tokenization, the support_phrase option is now automatically enabled to accelerate match_phrase series phrase queries. #37949

Other

• Audit log now can record SQL types. #37790

• Added support for information_schema.processlist to show all FE. #38701

• Cached ranger's atamask and rowpolicy to accelerate query efficiency. #37723

• Optimized metadata management in job manager to release locks immediately after modifying metadata, reducing lock holding time. #38162

Bug Fixes

Upgrade

• Fix the issue where mtmv load fails during upgrade from version 2.1. #38799

• Resolve the issue where null_type cannot be found during the upgrade to version 2.1. #39373

• Address the compatibility issue with permission persistence during the upgrade from version 2.1 to 3.0. #39288

Load

• Fix the issue where parsing fails when the newline character is surrounded by delimiters in CSV format parsing. #38347

• Resolve potential exception issues when FE forwards group commit. #38228 #38265

• Group commit now supports the new optimizer. #37002

• Fix the issue where group commit reports data errors when JDBC setNull is used. #38262

• Optimize the retry logic for group commit when encountering delete bitmap lock errors. #37600

• Resolve the issue where routine load cannot use CSV delimiters and escape characters. #38402

• Fix the issue where routine load job names with mixed case cannot be displayed. #38523

• Optimize the logic for actively recovering routine load during FE master-slave switching. #37876

• Resolve the issue where routine load pauses when all data in Kafka is expired. #37288

• Fix the issue where show routine load returns empty results. #38199

• Resolve the memory leak issue during multi-table stream import in routine load. #38255

• Fix the issue where stream load does not return the error URL. #38325

• Resolve potential load channel leak issues. #38031 #37500

• Fix the issue where no error may be reported when importing fewer segments than expected. #36753

• Resolve the load stream leak issue. #38912

• Optimize the impact of offline nodes on import operations. #38198

• Fix the issue where transactions do not end when inserting into empty data. #38991

Storage

01 Backup and Restoration

• Fix the issue where tables cannot be written after backup and restoration. #37089

• Resolve the issue where view database names are incorrect after backup and restoration. #37412

02 Compaction

• Fix the issue where cumu compaction handles delete errors incorrectly during ordered data compression. #38742

• Resolve the issue of duplicate keys in aggregate tables caused by sequential compression optimization. #38224

• Fix the issue where compression operations cause coredump in large wide tables. #37960

• Resolve the compression starvation issue caused by inaccurate concurrent statistics of compression tasks. #37318

03 MOW Unique Key

• Resolve the issue of inconsistent data between replicas caused by cumulative compression deletion of delete sign. #37950

• MOW delete now uses partial column updates with the new optimizer. #38751

• Fix the potential duplicate key issue in MOW tables under compute-storage decoupled. #39018

• Resolve the issue where MOW unique and duplicate tables cannot modify column order. #37067

• Fix the potential data correctness issue caused by segcompaction. #37760

• Resolve the potential memory leak issue during column updates. #37706

04 Other

• Fix the small probability of exceptions in TOPN queries. #39119 #39199

• Resolve the issue where auto-increment IDs may duplicate during FE restart. #37306

• Fix the potential queuing issue in the delete operation priority queue. #37169

• Optimize the delete retry logic. #37363

• Resolve the issue with bucket = 0 in table creation statements under the new optimizer. #38971

• Fix the issue where FE reports success incorrectly when image generation fails. #37508

• Resolve the issue where using the wrong nodename during FE offline nodes may cause inconsistent FE members. #37987

• Fix the issue where CCR partition addition may fail. #37295

• Resolve the int32 overflow issue in inverted index files. #38891

• Fix the issue where TRUNCATE TABLE failure may cause BE to fail to go offline. #37334

• Resolve the issue where publish cannot continue due to null pointers. #37724 #37531

• Fix the potential coredump issue when manually triggering disk migration. #37712

Compute-Storage Decoupled

• Fixed the issue where show create table might display the file_cache_ttl_seconds attribute twice. #38052

• Fixed the issue where segment Footer TTL was not set correctly after setting file cache TTL. #37485

• Fixed the issue where file cache might cause coredump due to massive conversion of cache types. #38518

• Fixed the potential file descriptor (fd) leak in file cache. #38051

• Fixed the issue where schema change Job overwriting compaction Job prevented base tablet compaction from completing normally. #38210

• Fixed the potential inaccuracy of base compaction score due to data race. #38006

• Fixed the issue where error messages from imports might not be uploaded correctly to object storage. #38359

• Fixed the inconsistency in return information between compute-storage decoupled mode and storage and compute integration mode for 2PC imports. #38076

• Fix the issue where incorrect file size setting during file cache warm-up leads to coredump. #38939

• Fixed the issue where partial column updates did not correctly dequeue delete operations. #37151

• Fixed compatibility issues with permission persistence in compute-storage decoupled mode. #38136 #37708

• Fixed the issue where observer did not retry correctly when encountering a -230 error. #37625

• Fixed the issue where show load with conditions did not perform correct analysis. #37656

• Fixed the issue where show streamload in compute-storage decoupled mode caused BE coredump. #37903

• Fixed the issue where copy into did not correctly verify column names in strict mode. #37650

• Fixed the issue where multi-stream imports into a single table lacked permissions. #38878

• Fixed the potential overflow issue in getVersionUpdateTimeMs. #38074

• Fixed the issue where FE azure blob list was not implemented correctly. #37986

• Fixed the issue where inaccurate azure blob recycling time calculation prevented recycling. #37535

• Fixed the issue where inverted index files were not deleted in compute-storage decoupled mode. #38306

Lakehouse

• Fixed the issue with reading binary data from Oracle Catalog. #37078

• Fixed the potential deadlock issue when acquiring external table metadata in multi-FE scenarios. #37756

• Fixed the issue where JNI scanner failure caused BE nodes to crash. #37697

• Fixed the issue with slow reading of date types from Trino Connector Catalog. #37266

• Optimized kerberos authentication logic for Hive Catalog. #37301

• Fixed the issue where region attributes might be parsed incorrectly when parsing MinIO properties. #37249

• Fixed the issue where creating too many FileSystems by FE caused memory leaks. #36954

• Fixed the issue with reading incorrect time zone information from Paimon. #37716

• Fixed the potential thread leak issue caused by Hive write-back operations. #36990

• Fixed the null pointer issue caused by enabling Hive metastore event synchronization. #38421

• Fixed the issue where error messages were unclear or caused stalling when creating catalogs. #37551

• Fixed the issue where reading Hive text format tables behaved differently from Hive. #37638

• Fixed the logic error when switching between catalogs and databases. #37828

MySQL Compatibility

• Fixed the issue where certain flags in the MySQL protocol were set incorrectly when SSL was enabled. #38086

Asynchronous Materialized Views

• Fixed the issue where construction might fail when the base table had a very large number of partitions. #37589

• Fixed the issue where nested materialized views incorrectly performed full table refreshes even when partition refreshes were possible. #38698

• Fixed the issue where partition refresh could not handle the simultaneous existence of valid and invalid dependencies when analyzing partition dependencies. #38367

• Fixed the issue where the final result containing NULL type might cause asynchronous materialized views to fail. #37019

• Fixed the planning error that might occur during transparent rewriting when both synchronous and asynchronous materialized views with the same name were present. #37311

Synchronous Materialized Views

• The rewritten synchronous materialized views now can correctly perform partition pruning. #38527

• When rewriting synchronous materialized views, those with unready data are no longer selected. #38148

Query Optimizer

• Fixed the deadlock issue that might occur when queries and delete operations are performed simultaneously. #38660

• Fixed the issue where bucket pruning might incorrectly prune on decimal column buckets. #37889

• Fixed the issue where planning might be incorrect when mark join participates in join reorder. #39152

• Fixed the issue where the result is incorrect when the correlation condition of a correlated subquery is not a simple column. #37644

• Fixed the issue where partition pruning cannot correctly handle or expressions. #38897

• Fixed the planning error that might occur when optimizing the execution order of JOIN and AGG. #37343

• Fixed the issue where str_to_date performs incorrect constant folding calculations on datev1 types. #37360

• Fixed the issue where the ACOS function's constant folding returns non-NaN values. #37932

• Fixed the occasional planning error: "The children format needs to be [WhenClause+, DefaultValue?]". #38491

• Fixed the issue where planning might be incorrect when the projection includes window functions and there is both the original column and its alias. #38166

• Fixed the issue where planning might report an error when the aggregation parameter contains a lambda expression. #37109

• Fixed the insert error that might occur in extreme cases: "MultiCastDataSink cannot be cast to DataStreamSink". #38526

• Fixed the issue where the new optimizer does not correctly handle char(0)/varchar(0) when creating a table. #38427

• Fixed the incorrect behavior of char(255) toSql. #37340

• Fixed the issue where the nullable attribute within the agg_state type might lead to planning errors. #37489

• Fixed the issue where row count statistics are inaccurate during mark Join. #38270

Query Execution

• Fixed issues where the Pipeline execution engine was stuck, causing queries to not end, in multiple scenarios. #38657, #38206, #38885, #38151, #37297

• Fixed the coredump issue caused by NULL and non-NULL columns during set difference calculations. #38750

• Fixed the error when using the DECIMAL type with pure decimals in delete statements. #37801

• Fixed the issue where the width_bucket function returned incorrect results. #37892

• Fixed the query error when a single row of data was very large and the result set was also large (exceeding 2GB). #37990

• Fixed the coredump issue caused by incorrect release of rpc connections during single-replica imports. #38087

• Fixed the coredump issue caused by processing NULL values with the foreach function. #37349

• Fixed the issue where stddev returned incorrect results for DECIMALV2 types. #38731

• Fixed the slow performance of bitmap union calculations. #37816

• Fixed the issue where RowsProduced for aggregation operators was not set in the profile. #38271

• Fixed the overflow issue when calculating the number of buckets for the hash table under hash join. #37193, #37493

• Fixed the inaccurate recording of the jemalloc cache memory tracker. #37464

• Added the enable_stacktrace configuration option, allowing users to control whether exception stacks are output in BE logs. #37713

• Fixed the issue where Arrow Flight SQL did not work correctly when enable_parallel_result_sink was set to false. #37779

• Fixed the incorrect use of colocate Join. #37361, #37729

• Fixed the calculation overflow issue of the round function on DECIMAL128 types. #37733, #38106

• Fixed the coredump issue when passing a const string to the sleep function. #37681

• Increased the queue length for audit logs, solving the issue where audit logs could not be recorded normally under high concurrency scenarios with thousands of concurrent connections. #37786

• Fixed the issue where creating a workload group caused too many threads, leading to BE coredump. #38096

• Fixed the coredump issue caused by the MULTI_MATCH_ANY function. #37959

• Fixed the transaction rollback issue caused by insert overwrite auto partition. #38103

• Fixed the issue where the TimeUtils formatter did not use the correct time zone. #37465

• Fixed the issue where results were incorrect under constant folding scenarios for week/yearweek. #37376

• Fixed the issue where the convert_tz function returned incorrect results. #37358, #38764

• Fixed the coredump issue when using the collect_set function with window functions. #38234

• Fixed the coredump issue caused by percentile_approx during rolling upgrades. #39321

• Fixed the coredump issue caused by the mod function when encountering abnormal input. #37999

• Fixed the issue where the hash table was not fully built when the broadcast join probe started running. #37643

• Fixed the issue where executing the same expression in multithreaded environments might lead to incorrect results for Java UDFs. #38612

• Fixed the overflow issue caused by incorrect return types of the conv function. #38001

• Fixed the issue where the json_replace function returned incorrect types. #3701

• Fixed the issue where the nullable attribute setting was unreasonable for the percentile aggregation function. #37330

• Fixed the issue where the results of the histogram function were unstable. #38608

• Fixed the issue where task state was displayed incorrectly in the profile. #38082

• Fixed the issue where some queries were incorrectly canceled when the system just started. #37662

Semi-Structured Data Management

• Fix some issues with time series compression. #39170 #39176

• Fix the issue of incorrect index size statistics during compression. #37232

• Fix the potential incorrect matching of ultra-long strings without tokenization in inverted indexes. #37679 #38218

• Fix the high memory usage issue of array_range and array_with_const functions when dealing with large data volumes. #38284 #37495

• Fix the potential coredump issue when selecting columns of ARRAY / MAP / STRUCT types. #37936

• Fix the import failure issue caused by simdjson parsing errors when specifying jsonpath in Stream Load. #38490

• Fix the exception handling issue when there are duplicate keys in JSON data. #38146

• Fix the potential query error after DROP INDEX. #37646

• Fix the error return issue in row merging checks during index compression. #38732

• Inverted index v2 format now supports renaming columns. #38079

• Fix the coredump issue when the MATCH function matches an empty string without an index. #37947

• Fix the handling of NULL values in inverted indexes. #37921 #37842 #38741

• Fix the incorrect row_store_page_size after FE restart. #38240

Other

• Fix the timezone configuration issue. The default timezone is no longer fixed at UTC+8 and is now obtained from system configuration. #37294

• Fix the class conflict issue when using ranger due to multiple JSR specification implementations. #37575

• Fix the potential uninitialized field issue in some BE code. #37403

• Fix the error in delete statements for random distributed tables. #37985

• Fix the incorrect requirement for alter_priv permission on the base table when creating a synchronized materialized view. #38011

• Fix the issue of not authenticating resources when used in TVF. #36928

Credits

Thanks all who contribute to this release:

@133tosakarin, @924060929, @AshinGau, @Baymine, @BePPPower, @BiteTheDDDDt, @ByteYue, @CalvinKirs, @Ceng23333, @DarvenDuan, @FreeOnePlus, @Gabriel39, @HappenLee, @JNSimba, @Jibing-Li, @KassieZ, @Lchangliang, @LiBinfeng-01, @Mryange, @SWJTU-ZhangLei, @TangSiyang2001, @Tech-Circle-48, @Vallishp, @Yukang-Lian, @Yulei-Yang, @airborne12, @amorynan, @bobhan1, @cambyzju, @cjj2010, @csun5285, @dataroaring, @deardeng, @eldenmoon, @englefly, @feiniaofeiafei, @felixwluo, @freemandealer, @gavinchou, @ghkang98, @hello-stephen, @hubgeter, @hust-hhb, @jacktengg, @kaijchen, @kaka11chen, @keanji-x, @liaoxin01, @liutang123, @luwei16, @luzhijing, @lxr599, @morningman, @morrySnow, @mrhhsg, @mymeiyi, @platoneko, @qidaye, @qzsee, @seawinde, @shuke987, @sollhui, @starocean999, @suxiaogang223, @w41ter, @wangbo, @wangshuo128, @whutpencil, @wsjz, @wuwenchi, @wyxxxcat, @xiaokang, @xiedeyantu, @xinyiZzz, @xy720, @xzj7019, @yagagagaga, @yiguolei, @yujun777, @z404289981, @zclllyybb, @zddr, @zfr9527, @zhangbutao, @zhangstar333, @zhannngchen, @zhiqiang-hhhh, @zjj, @zy-kkk, @zzzxl1993