ClickHouse v20.6.3.28 Release Notes

  • Backward Incompatible Change

    • โšก๏ธ When upgrading from versions older than 20.5, if rolling update is performed and cluster contains both versions 20.5 or greater and less than 20.5, if ClickHouse nodes with old versions are restarted and old version has been started up in presence of newer versions, it may lead to Part ... intersects previous part errors. To prevent this error, first install newer clickhouse-server packages on all cluster nodes and then do restarts (so, when clickhouse-server is restarted, it will start up with the new version).

    ๐Ÿ†• New Feature

    • โž• Added an initial implementation of EXPLAIN query. Syntax: EXPLAIN SELECT .... This fixes #1118. #11873 (Nikolai Kochetov).
    • โž• Added storage RabbitMQ. #11069 (Kseniia Sumarokova).
    • Implemented PostgreSQL-like ILIKE operator for #11710. #12125 (Mike).
    • Supported RIGHT and FULL JOIN with SET join_algorithm = 'partial_merge'. Only ALL strictness is allowed (ANY, SEMI, ANTI, ASOF are not). #12118 (Artem Zuikov).
    • โž• Added a function initializeAggregation to initialize an aggregation based on a single value. #12109 (Guillaume Tassery).
    • ๐Ÿ‘Œ Supported ALTER TABLE ... [ADD|MODIFY] COLUMN ... FIRST #4006. #12073 (Winter Zhang).
    • โž• Added function parseDateTimeBestEffortUS. #12028 (flynn).
    • ๐Ÿ‘Œ Support format ORC for output (was supported only for input). #11662 (Kruglov Pavel).

    ๐Ÿ› Bug Fix

    • ๐Ÿšš Fixed aggregate function any(x) is found inside another aggregate function in query error with SET optimize_move_functions_out_of_any = 1 and aliases inside any(). #13419 (Artem Zuikov).
    • ๐Ÿ›  Fixed PrettyCompactMonoBlock for clickhouse-local. Fixed extremes/totals with PrettyCompactMonoBlock. This fixes #7746. #13394 (Azat Khuzhin).
    • ๐Ÿ›  Fixed possible error Totals having transform was already added to pipeline in case of a query from delayed replica. #13290 (Nikolai Kochetov).
    • ๐Ÿ›  The server may crash if user passed specifically crafted arguments to the function h3ToChildren. This fixes #13275. #13277 (alexey-milovidov).
    • ๐Ÿ›  Fixed potentially low performance and slightly incorrect result for uniqExact, topK, sumDistinct and similar aggregate functions called on Float types with NaN values. It also triggered assert in debug build. This fixes #12491. #13254 (alexey-milovidov).
    • ๐Ÿ›  Fixed function if with nullable constexpr as cond that is not literal NULL. Fixes #12463. #13226 (alexey-milovidov).
    • ๐Ÿ›  Fixed assert in arrayElement function in case of array elements are Nullable and array subscript is also Nullable. This fixes #12172. #13224 (alexey-milovidov).
    • ๐Ÿ›  Fixed DateTime64 conversion functions with constant argument. #13205 (Azat Khuzhin).
    • ๐Ÿ›  Fixed wrong index analysis with functions. It could lead to pruning wrong parts, while reading from MergeTree tables. Fixes #13060. Fixes #12406. #13081 (Anton Popov).
    • ๐Ÿ›  Fixed error Cannot convert column because it is constant but values of constants are different in source and result for remote queries which use deterministic functions in scope of query, but not deterministic between queries, like now(), now64(), randConstant(). Fixes #11327. #13075 (Nikolai Kochetov).
    • ๐Ÿ›  Fixed unnecessary limiting for the number of threads for selects from local replica. #12840 (Nikolai Kochetov).
    • ๐Ÿ›  Fixed rare bug when ALTER DELETE and ALTER MODIFY COLUMN queries executed simultaneously as a single mutation. Bug leads to an incorrect amount of rows in count.txt and as a consequence incorrect data in part. Also, fix a small bug with simultaneous ALTER RENAME COLUMN and ALTER ADD COLUMN. #12760 (alesapin).
    • ๐Ÿ›  Fixed CAST(Nullable(String), Enum()). #12745 (Azat Khuzhin).
    • ๐Ÿ›  Fixed a performance with large tuples, which are interpreted as functions in IN section. The case when user write WHERE x IN tuple(1, 2, ...) instead of WHERE x IN (1, 2, ...) for some obscure reason. #12700 (Anton Popov).
    • Fixed memory tracking for input_format_parallel_parsing (by attaching thread to group). #12672 (Azat Khuzhin).
    • ๐Ÿ›  Fixed bloom filter index with const expression. This fixes #10572. #12659 (Winter Zhang).
    • ๐Ÿ›  Fixed SIGSEGV in StorageKafka when broker is unavailable (and not only). #12658 (Azat Khuzhin).
    • โž• Added support for function if with Array(UUID) arguments. This fixes #11066. #12648 (alexey-milovidov).
    • ๐Ÿ›  CREATE USER IF NOT EXISTS now doesn't throw exception if the user exists. This fixes #12507. #12646 (Vitaly Baranov).
    • ๐Ÿ‘ Better exception message in disk access storage. #12625 (alesapin).
    • ๐Ÿ›  The function groupArrayMoving* was not working for distributed queries. It's result was calculated within incorrect data type (without promotion to the largest type). The function groupArrayMovingAvg was returning integer number that was inconsistent with the avg function. This fixes #12568. #12622 (alexey-milovidov).
    • ๐Ÿ›  Fixed lack of aliases with function any. #12593 (Anton Popov).
    • ๐Ÿ›  Fixed race condition in external dictionaries with cache layout which can lead server crash. #12566 (alesapin).
    • โœ‚ Remove data for Distributed tables (blocks from async INSERTs) on DROP TABLE. #12556 (Azat Khuzhin).
    • Fixed bug which lead to broken old parts after ALTER DELETE query when enable_mixed_granularity_parts=1. Fixes #12536. #12543 (alesapin).
    • ๐Ÿ‘ Better exception for function in with invalid number of arguments. #12529 (Anton Popov).
    • ๐Ÿ›  Fixing race condition in live view tables which could cause data duplication. #12519 (vzakaznikov).
    • ๐Ÿ›  Fixed performance issue, while reading from compact parts. #12492 (Anton Popov).
    • ๐Ÿ›  Fixed backwards compatibility in binary format of AggregateFunction(avg, ...) values. This fixes #12342. #12486 (alexey-milovidov).
    • ๐Ÿ›  Fixed SETTINGS parse after FORMAT. #12480 (Azat Khuzhin).
    • ๐Ÿ›  Fixed the deadlock if text_log is enabled. #12452 (alexey-milovidov).
    • ๐Ÿ›  Fixed overflow when very large LIMIT or OFFSET is specified. This fixes #10470. This fixes #11372. #12427 (alexey-milovidov).
    • ๐Ÿ›  Fixed possible segfault if StorageMerge. This fixes #12054. #12401 (tavplubix).
    • โช Reverted change introduced in #11079 to resolve #12098. #12397 (Mike).
    • โž• Additional check for arguments of bloom filter index. This fixes #11408. #12388 (alexey-milovidov).
    • ๐Ÿ›  Avoid exception when negative or floating point constant is used in WHERE condition for indexed tables. This fixes #11905. #12384 (alexey-milovidov).
    • ๐Ÿ‘ Allowed to CLEAR column even if there are depending DEFAULT expressions. This fixes #12333. #12378 (alexey-milovidov).
    • ๐Ÿ›  Fix TOTALS/ROLLUP/CUBE for aggregate functions with -State and Nullable arguments. This fixes #12163. #12376 (alexey-milovidov).
    • ๐Ÿ›  Fixed error message and exit codes for ALTER RENAME COLUMN queries, when RENAME is not allowed. Fixes #12301 and #12303. #12335 (alesapin).
    • ๐Ÿ›  Fixed very rare race condition in ReplicatedMergeTreeQueue. #12315 (alexey-milovidov).
    • ๐Ÿ‘ป When using codec Delta or DoubleDelta with non fixed width types, exception with code LOGICAL_ERROR was returned instead of exception with code BAD_ARGUMENTS (we ensure that exceptions with code logical error never happen). This fixes #12110. #12308 (alexey-milovidov).
    • ๐Ÿ›  Fixed order of columns in WITH FILL modifier. Previously order of columns of ORDER BY statement wasn't respected. #12306 (Anton Popov).
    • ๐Ÿ”€ Avoid "bad cast" exception when there is an expression that filters data by virtual columns (like _table in Merge tables) or by "index" columns in system tables such as filtering by database name when querying from system.tables, and this expression returns Nullable type. This fixes #12166. #12305 (alexey-milovidov).
    • ๐Ÿ›  Fixed TTL after renaming column, on which depends TTL expression. #12304 (Anton Popov).
    • ๐Ÿ›  Fixed SIGSEGV if there is an message with error in the middle of the batch in Kafka Engine. #12302 (Azat Khuzhin).
    • ๐Ÿ›  Fixed the situation when some threads might randomly hang for a few seconds during DNS cache updating. #12296 (tavplubix).
    • ๐Ÿ›  Fixed typo in setting name. #12292 (alexey-milovidov).
    • ๐Ÿ‘‰ Show error after TrieDictionary failed to load. #12290 (Vitaly Baranov).
    • ๐Ÿ›  The function arrayFill worked incorrectly for empty arrays that may lead to crash. This fixes #12263. #12279 (alexey-milovidov).
    • ๐Ÿ›  Implement conversions to the common type for LowCardinality types. This allows to execute UNION ALL of tables with columns of LowCardinality and other columns. This fixes #8212. This fixes #4342. #12275 (alexey-milovidov).
    • ๐Ÿ›  Fixed the behaviour on reaching redirect limit in request to S3 storage. #12256 (ianton-ru).
    • ๐Ÿ›  Fixed the behaviour when during multiple sequential inserts in StorageFile header for some special types was written more than once. This fixed #6155. #12197 (Nikita Mikhaylov).
    • ๐Ÿ›  Fixed logical functions for UInt8 values when they are not equal to 0 or 1. #12196 (Alexander Kazakov).
    • Cap max_memory_usage* limits to the process resident memory. #12182 (Azat Khuzhin).
    • ๐Ÿ›  Fix dictGet arguments check during GROUP BY injective functions elimination. #12179 (Azat Khuzhin).
    • ๐Ÿ›  Fixed the behaviour when SummingMergeTree engine sums up columns from partition key. Added an exception in case of explicit definition of columns to sum which intersects with partition key columns. This fixes #7867. #12173 (Nikita Mikhaylov).
    • ๐Ÿ‘ Don't split the dictionary source's table name into schema and table name itself if ODBC connection doesn't support schema. #12165 (Vitaly Baranov).
    • ๐Ÿ›  Fixed wrong logic in ALTER DELETE that leads to deleting of records when condition evaluates to NULL. This fixes #9088. This closes #12106. #12153 (alexey-milovidov).
    • ๐Ÿ›  Fixed transform of query to send to external DBMS (e.g. MySQL, ODBC) in presense of aliases. This fixes #12032. #12151 (alexey-milovidov).
    • ๐Ÿ›  Fixed bad code in redundant ORDER BY optimization. The bug was introduced in #10067. #12148 (alexey-milovidov).
    • ๐Ÿ›  Fixed potential overflow in integer division. This fixes #12119. #12140 (alexey-milovidov).
    • ๐Ÿ›  Fixed potential infinite loop in greatCircleDistance, geoDistance. This fixes #12117. #12137 (alexey-milovidov).
    • ๐Ÿšš Normalize "pid" file handling. In previous versions the server may refuse to start if it was killed without proper shutdown and if there is another process that has the same pid as previously runned server. Also pid file may be removed in unsuccessful server startup even if there is another server running. This fixes #3501. #12133 (alexey-milovidov).
    • ๐Ÿ›  Fixed bug which leads to incorrect table metadata in ZooKeepeer for ReplicatedVersionedCollapsingMergeTree tables. Fixes #12093. #12121 (alesapin).
    • ๐ŸŒฒ Avoid "There is no query" exception for materialized views with joins or with subqueries attached to system logs (system.query_log, metric_log, etc) or to engine=Buffer underlying table. #12120 (filimonov).
    • ๐Ÿ›  Fixed handling dependency of table with ENGINE=Dictionary on dictionary. This fixes #10994. This fixes #10397. #12116 (Vitaly Baranov).
    • ๐Ÿ›  Format Parquet now properly works with LowCardinality and LowCardinality(Nullable) types. Fixes #12086, #8406. #12108 (Nikolai Kochetov).
    • ๐Ÿ›  Fixed performance for selects with UNION caused by wrong limit for the total number of threads. Fixes #12030. #12103 (Nikolai Kochetov).
    • ๐Ÿ›  Fixed segfault with -StateResample combinators. #12092 (Anton Popov).
    • Fixed empty result_rows and result_bytes metrics in system.quey_log for selects. Fixes #11595. #12089 (Nikolai Kochetov).
    • ๐Ÿ›  Fixed unnecessary limiting the number of threads for selects from VIEW. Fixes #11937. #12085 (Nikolai Kochetov).
    • ๐Ÿ›  Fixed SIGSEGV in StorageKafka on DROP TABLE. #12075 (Azat Khuzhin).
    • ๐Ÿ›  Fixed possible crash while using wrong type for PREWHERE. Fixes #12053, #12060. #12060 (Nikolai Kochetov).
    • ๐Ÿ›  Fixed error Cannot capture column for higher-order functions with Tuple(LowCardinality) argument. Fixes #9766. #12055 (Nikolai Kochetov).
    • ๐Ÿ›  Fixed constraints check if constraint is a constant expression. This fixes #11360. #12042 (alexey-milovidov).
    • ๐Ÿ›  Fixed wrong result and potential crash when invoking function if with arguments of type FixedString with different sizes. This fixes #11362. #12021 (alexey-milovidov).

    ๐Ÿ‘Œ Improvement

    • ๐Ÿ‘ Allowed to set JOIN kind and type in more standard way: LEFT SEMI JOIN instead of SEMI LEFT JOIN. For now both are correct. #12520 (Artem Zuikov).
    • lifetime_rows/lifetime_bytes for Buffer engine. #12421 (Azat Khuzhin).
    • ๐Ÿ‘ป Write the detail exception message to the client instead of 'MySQL server has gone away'. #12383 (BohuTANG).
    • Allows to change a charset which is used for printing grids borders. Available charsets are following: UTF-8, ASCII. Setting output_format_pretty_grid_charset enables this feature. #12372 (Sabyanin Maxim).
    • ๐Ÿ‘Œ Supported MySQL 'SELECT DATABASE()' #9336 2. Add MySQL replacement query integration test. #12314 (BohuTANG).
    • โž• Added KILL QUERY [connection_id] for the MySQL client/driver to cancel the long query, issue #12038. #12152 (BohuTANG).
    • โž• Added support for %g (two digit ISO year) and %G (four digit ISO year) substitutions in formatDateTime function. #12136 (vivarum).
    • โž• Added 'type' column in system.disks. #12115 (ianton-ru).
    • Improved REVOKE command: now it requires grant/admin option for only access which will be revoked. For example, to execute REVOKE ALL ON *.* FROM user1 now it doesn't require to have full access rights granted with grant option. Added command REVOKE ALL FROM user1 - it revokes all granted roles from user1. #12083 (Vitaly Baranov).
    • โž• Added replica priority for load_balancing (for manual prioritization of the load balancing). #11995 (Azat Khuzhin).
    • ๐Ÿ“‡ Switched paths in S3 metadata to relative which allows to handle S3 blobs more easily. #11892 (Vladimir Chebotarev).

    ๐ŸŽ Performance Improvement

    • Improved performace of 'ORDER BY' and 'GROUP BY' by prefix of sorting key (enabled with optimize_aggregation_in_order setting, disabled by default). #11696 (Anton Popov).
    • โšก๏ธ Removed injective functions inside uniq*() if set optimize_injective_functions_inside_uniq=1. #12337 (Ruslan Kamalov).
    • ๐ŸŽ Index not used for IN operator with literals", performance regression introduced around v19.3. This fixes "#10574. #12062 (nvartolomei).
    • Implemented single part uploads for DiskS3 (experimental feature). #12026 (Vladimir Chebotarev).

    Experimental Feature

    • Added new in-memory format of parts in MergeTree-family tables, which stores data in memory. Parts are written on disk at first merge. Part will be created in in-memory format if its size in rows or bytes is below thresholds min_rows_for_compact_part and min_bytes_for_compact_part. Also optional support of Write-Ahead-Log is available, which is enabled by default and is controlled by setting in_memory_parts_enable_wal. #10697 (Anton Popov).

    ๐Ÿ— Build/Testing/Packaging Improvement

    • Implement AST-based query fuzzing mode for clickhouse-client. See this label for the list of issues we recently found by fuzzing. Most of them were found by this tool, and a couple by SQLancer and 00746_sql_fuzzy.pl. #12111 (Alexander Kuzmenkov).
    • โž• Add new type of tests based on Testflows framework. #12090 (vzakaznikov).
    • โž• Added S3 HTTPS integration test. #12412 (Pavel Kovalenko).
    • ๐ŸŒฒ Log sanitizer trap messages from separate thread. This will prevent possible deadlock under thread sanitizer. #12313 (alexey-milovidov).
    • โœ… Now functional and stress tests will be able to run with old version of clickhouse-test script. #12287 (alesapin).
    • โœ‚ Remove strange file creation during build in orc. #12258 (Nikita Mikhaylov).
    • ๐Ÿณ Place common docker compose files to integration docker container. #12168 (Ilya Yatsishin).
    • ๐Ÿ›  Fix warnings from CodeQL. CodeQL is another static analyzer that we will use along with clang-tidy and PVS-Studio that we use already. #12138 (alexey-milovidov).
    • ๐Ÿ— Minor CMake fixes for UNBUNDLED build. #12131 (Matwey V. Kornilov).
    • โž• Added a showcase of the minimal Docker image without using any Linux distribution. #12126 (alexey-milovidov).
    • โฌ†๏ธ Perform an upgrade of system packages in the clickhouse-server docker image. #12124 (Ivan Blinkov).
    • โž• Add UNBUNDLED flag to system.build_options table. Move skip lists for clickhouse-test to clickhouse repo. #12107 (alesapin).
    • ๐Ÿ”’ Regular check by Anchore Container Analysis security analysis tool that looks for CVE in clickhouse-server Docker image. Also confirms that Dockerfile is buildable. Runs daily on master and on pull-requests to Dockerfile. #12102 (Ivan Blinkov).
    • ๐Ÿ”’ Daily check by GitHub CodeQL security analysis tool that looks for CWE. #12101 (Ivan Blinkov).
    • โšก๏ธ Install ca-certificates before the first apt-get update in Dockerfile. #12095 (Ivan Blinkov).