ClickHouse v20.6.3.28 Release Notes
-
Backward Incompatible Change
- โก๏ธ When upgrading from versions older than 20.5, if rolling update is performed and cluster contains both versions 20.5 or greater and less than 20.5, if ClickHouse nodes with old versions are restarted and old version has been started up in presence of newer versions, it may lead to
Part ... intersects previous part
errors. To prevent this error, first install newer clickhouse-server packages on all cluster nodes and then do restarts (so, when clickhouse-server is restarted, it will start up with the new version).
๐ New Feature
- โ Added an initial implementation of
EXPLAIN
query. Syntax:EXPLAIN SELECT ...
. This fixes #1118. #11873 (Nikolai Kochetov). - โ Added storage
RabbitMQ
. #11069 (Kseniia Sumarokova). - Implemented PostgreSQL-like
ILIKE
operator for #11710. #12125 (Mike). - Supported RIGHT and FULL JOIN with
SET join_algorithm = 'partial_merge'
. Only ALL strictness is allowed (ANY, SEMI, ANTI, ASOF are not). #12118 (Artem Zuikov). - โ Added a function
initializeAggregation
to initialize an aggregation based on a single value. #12109 (Guillaume Tassery). - ๐ Supported
ALTER TABLE ... [ADD|MODIFY] COLUMN ... FIRST
#4006. #12073 (Winter Zhang). - โ Added function
parseDateTimeBestEffortUS
. #12028 (flynn). - ๐ Support format
ORC
for output (was supported only for input). #11662 (Kruglov Pavel).
๐ Bug Fix
- ๐ Fixed
aggregate function any(x) is found inside another aggregate function in query
error withSET optimize_move_functions_out_of_any = 1
and aliases insideany()
. #13419 (Artem Zuikov). - ๐ Fixed
PrettyCompactMonoBlock
for clickhouse-local. Fixed extremes/totals withPrettyCompactMonoBlock
. This fixes #7746. #13394 (Azat Khuzhin). - ๐ Fixed possible error
Totals having transform was already added to pipeline
in case of a query from delayed replica. #13290 (Nikolai Kochetov). - ๐ The server may crash if user passed specifically crafted arguments to the function
h3ToChildren
. This fixes #13275. #13277 (alexey-milovidov). - ๐ Fixed potentially low performance and slightly incorrect result for
uniqExact
,topK
,sumDistinct
and similar aggregate functions called on Float types with NaN values. It also triggered assert in debug build. This fixes #12491. #13254 (alexey-milovidov). - ๐ Fixed function if with nullable constexpr as cond that is not literal NULL. Fixes #12463. #13226 (alexey-milovidov).
- ๐ Fixed assert in
arrayElement
function in case of array elements are Nullable and array subscript is also Nullable. This fixes #12172. #13224 (alexey-milovidov). - ๐ Fixed
DateTime64
conversion functions with constant argument. #13205 (Azat Khuzhin). - ๐ Fixed wrong index analysis with functions. It could lead to pruning wrong parts, while reading from
MergeTree
tables. Fixes #13060. Fixes #12406. #13081 (Anton Popov). - ๐ Fixed error
Cannot convert column because it is constant but values of constants are different in source and result
for remote queries which use deterministic functions in scope of query, but not deterministic between queries, likenow()
,now64()
,randConstant()
. Fixes #11327. #13075 (Nikolai Kochetov). - ๐ Fixed unnecessary limiting for the number of threads for selects from local replica. #12840 (Nikolai Kochetov).
- ๐ Fixed rare bug when
ALTER DELETE
andALTER MODIFY COLUMN
queries executed simultaneously as a single mutation. Bug leads to an incorrect amount of rows incount.txt
and as a consequence incorrect data in part. Also, fix a small bug with simultaneousALTER RENAME COLUMN
andALTER ADD COLUMN
. #12760 (alesapin). - ๐ Fixed
CAST(Nullable(String), Enum())
. #12745 (Azat Khuzhin). - ๐ Fixed a performance with large tuples, which are interpreted as functions in
IN
section. The case when user writeWHERE x IN tuple(1, 2, ...)
instead ofWHERE x IN (1, 2, ...)
for some obscure reason. #12700 (Anton Popov). - Fixed memory tracking for
input_format_parallel_parsing
(by attaching thread to group). #12672 (Azat Khuzhin). - ๐ Fixed bloom filter index with const expression. This fixes #10572. #12659 (Winter Zhang).
- ๐ Fixed
SIGSEGV
inStorageKafka
when broker is unavailable (and not only). #12658 (Azat Khuzhin). - โ Added support for function
if
withArray(UUID)
arguments. This fixes #11066. #12648 (alexey-milovidov). - ๐
CREATE USER IF NOT EXISTS
now doesn't throw exception if the user exists. This fixes #12507. #12646 (Vitaly Baranov). - ๐ Better exception message in disk access storage. #12625 (alesapin).
- ๐ The function
groupArrayMoving*
was not working for distributed queries. It's result was calculated within incorrect data type (without promotion to the largest type). The functiongroupArrayMovingAvg
was returning integer number that was inconsistent with theavg
function. This fixes #12568. #12622 (alexey-milovidov). - ๐ Fixed lack of aliases with function
any
. #12593 (Anton Popov). - ๐ Fixed race condition in external dictionaries with cache layout which can lead server crash. #12566 (alesapin).
- โ Remove data for Distributed tables (blocks from async INSERTs) on DROP TABLE. #12556 (Azat Khuzhin).
- Fixed bug which lead to broken old parts after
ALTER DELETE
query whenenable_mixed_granularity_parts=1
. Fixes #12536. #12543 (alesapin). - ๐ Better exception for function
in
with invalid number of arguments. #12529 (Anton Popov). - ๐ Fixing race condition in live view tables which could cause data duplication. #12519 (vzakaznikov).
- ๐ Fixed performance issue, while reading from compact parts. #12492 (Anton Popov).
- ๐ Fixed backwards compatibility in binary format of
AggregateFunction(avg, ...)
values. This fixes #12342. #12486 (alexey-milovidov). - ๐ Fixed SETTINGS parse after FORMAT. #12480 (Azat Khuzhin).
- ๐ Fixed the deadlock if
text_log
is enabled. #12452 (alexey-milovidov). - ๐ Fixed overflow when very large
LIMIT
orOFFSET
is specified. This fixes #10470. This fixes #11372. #12427 (alexey-milovidov). - ๐ Fixed possible segfault if
StorageMerge
. This fixes #12054. #12401 (tavplubix). - โช Reverted change introduced in #11079 to resolve #12098. #12397 (Mike).
- โ Additional check for arguments of bloom filter index. This fixes #11408. #12388 (alexey-milovidov).
- ๐ Avoid exception when negative or floating point constant is used in WHERE condition for indexed tables. This fixes #11905. #12384 (alexey-milovidov).
- ๐ Allowed to
CLEAR
column even if there are dependingDEFAULT
expressions. This fixes #12333. #12378 (alexey-milovidov). - ๐ Fix
TOTALS/ROLLUP/CUBE
for aggregate functions with-State
andNullable
arguments. This fixes #12163. #12376 (alexey-milovidov). - ๐ Fixed error message and exit codes for
ALTER RENAME COLUMN
queries, whenRENAME
is not allowed. Fixes #12301 and #12303. #12335 (alesapin). - ๐ Fixed very rare race condition in
ReplicatedMergeTreeQueue
. #12315 (alexey-milovidov). - ๐ป When using codec
Delta
orDoubleDelta
with non fixed width types, exception with codeLOGICAL_ERROR
was returned instead of exception with codeBAD_ARGUMENTS
(we ensure that exceptions with code logical error never happen). This fixes #12110. #12308 (alexey-milovidov). - ๐ Fixed order of columns in
WITH FILL
modifier. Previously order of columns ofORDER BY
statement wasn't respected. #12306 (Anton Popov). - ๐ Avoid "bad cast" exception when there is an expression that filters data by virtual columns (like
_table
inMerge
tables) or by "index" columns in system tables such as filtering by database name when querying fromsystem.tables
, and this expression returnsNullable
type. This fixes #12166. #12305 (alexey-milovidov). - ๐ Fixed
TTL
after renaming column, on which depends TTL expression. #12304 (Anton Popov). - ๐ Fixed SIGSEGV if there is an message with error in the middle of the batch in
Kafka
Engine. #12302 (Azat Khuzhin). - ๐ Fixed the situation when some threads might randomly hang for a few seconds during
DNS
cache updating. #12296 (tavplubix). - ๐ Fixed typo in setting name. #12292 (alexey-milovidov).
- ๐ Show error after
TrieDictionary
failed to load. #12290 (Vitaly Baranov). - ๐ The function
arrayFill
worked incorrectly for empty arrays that may lead to crash. This fixes #12263. #12279 (alexey-milovidov). - ๐ Implement conversions to the common type for
LowCardinality
types. This allows to execute UNION ALL of tables with columns of LowCardinality and other columns. This fixes #8212. This fixes #4342. #12275 (alexey-milovidov). - ๐ Fixed the behaviour on reaching redirect limit in request to
S3
storage. #12256 (ianton-ru). - ๐ Fixed the behaviour when during multiple sequential inserts in
StorageFile
header for some special types was written more than once. This fixed #6155. #12197 (Nikita Mikhaylov). - ๐ Fixed logical functions for UInt8 values when they are not equal to 0 or 1. #12196 (Alexander Kazakov).
- Cap max_memory_usage* limits to the process resident memory. #12182 (Azat Khuzhin).
- ๐ Fix dictGet arguments check during
GROUP BY
injective functions elimination. #12179 (Azat Khuzhin). - ๐ Fixed the behaviour when
SummingMergeTree
engine sums up columns from partition key. Added an exception in case of explicit definition of columns to sum which intersects with partition key columns. This fixes #7867. #12173 (Nikita Mikhaylov). - ๐ Don't split the dictionary source's table name into schema and table name itself if ODBC connection doesn't support schema. #12165 (Vitaly Baranov).
- ๐ Fixed wrong logic in
ALTER DELETE
that leads to deleting of records when condition evaluates to NULL. This fixes #9088. This closes #12106. #12153 (alexey-milovidov). - ๐ Fixed transform of query to send to external DBMS (e.g. MySQL, ODBC) in presense of aliases. This fixes #12032. #12151 (alexey-milovidov).
- ๐ Fixed bad code in redundant ORDER BY optimization. The bug was introduced in #10067. #12148 (alexey-milovidov).
- ๐ Fixed potential overflow in integer division. This fixes #12119. #12140 (alexey-milovidov).
- ๐ Fixed potential infinite loop in
greatCircleDistance
,geoDistance
. This fixes #12117. #12137 (alexey-milovidov). - ๐ Normalize "pid" file handling. In previous versions the server may refuse to start if it was killed without proper shutdown and if there is another process that has the same pid as previously runned server. Also pid file may be removed in unsuccessful server startup even if there is another server running. This fixes #3501. #12133 (alexey-milovidov).
- ๐ Fixed bug which leads to incorrect table metadata in ZooKeepeer for ReplicatedVersionedCollapsingMergeTree tables. Fixes #12093. #12121 (alesapin).
- ๐ฒ Avoid "There is no query" exception for materialized views with joins or with subqueries attached to system logs (system.query_log, metric_log, etc) or to engine=Buffer underlying table. #12120 (filimonov).
- ๐ Fixed handling dependency of table with ENGINE=Dictionary on dictionary. This fixes #10994. This fixes #10397. #12116 (Vitaly Baranov).
- ๐ Format
Parquet
now properly works withLowCardinality
andLowCardinality(Nullable)
types. Fixes #12086, #8406. #12108 (Nikolai Kochetov). - ๐ Fixed performance for selects with
UNION
caused by wrong limit for the total number of threads. Fixes #12030. #12103 (Nikolai Kochetov). - ๐ Fixed segfault with
-StateResample
combinators. #12092 (Anton Popov). - Fixed empty
result_rows
andresult_bytes
metrics insystem.quey_log
for selects. Fixes #11595. #12089 (Nikolai Kochetov). - ๐ Fixed unnecessary limiting the number of threads for selects from
VIEW
. Fixes #11937. #12085 (Nikolai Kochetov). - ๐ Fixed SIGSEGV in StorageKafka on DROP TABLE. #12075 (Azat Khuzhin).
- ๐ Fixed possible crash while using wrong type for
PREWHERE
. Fixes #12053, #12060. #12060 (Nikolai Kochetov). - ๐ Fixed error
Cannot capture column
for higher-order functions withTuple(LowCardinality)
argument. Fixes #9766. #12055 (Nikolai Kochetov). - ๐ Fixed constraints check if constraint is a constant expression. This fixes #11360. #12042 (alexey-milovidov).
- ๐ Fixed wrong result and potential crash when invoking function
if
with arguments of typeFixedString
with different sizes. This fixes #11362. #12021 (alexey-milovidov).
๐ Improvement
- ๐ Allowed to set
JOIN
kind and type in more standard way:LEFT SEMI JOIN
instead ofSEMI LEFT JOIN
. For now both are correct. #12520 (Artem Zuikov). - lifetime_rows/lifetime_bytes for Buffer engine. #12421 (Azat Khuzhin).
- ๐ป Write the detail exception message to the client instead of 'MySQL server has gone away'. #12383 (BohuTANG).
- Allows to change a charset which is used for printing grids borders. Available charsets are following: UTF-8, ASCII. Setting
output_format_pretty_grid_charset
enables this feature. #12372 (Sabyanin Maxim). - ๐ Supported MySQL 'SELECT DATABASE()' #9336 2. Add MySQL replacement query integration test. #12314 (BohuTANG).
- โ Added
KILL QUERY [connection_id]
for the MySQL client/driver to cancel the long query, issue #12038. #12152 (BohuTANG). - โ Added support for
%g
(two digit ISO year) and%G
(four digit ISO year) substitutions informatDateTime
function. #12136 (vivarum). - โ Added 'type' column in system.disks. #12115 (ianton-ru).
- Improved
REVOKE
command: now it requires grant/admin option for only access which will be revoked. For example, to executeREVOKE ALL ON *.* FROM user1
now it doesn't require to have full access rights granted with grant option. Added commandREVOKE ALL FROM user1
- it revokes all granted roles fromuser1
. #12083 (Vitaly Baranov). - โ Added replica priority for load_balancing (for manual prioritization of the load balancing). #11995 (Azat Khuzhin).
- ๐ Switched paths in S3 metadata to relative which allows to handle S3 blobs more easily. #11892 (Vladimir Chebotarev).
๐ Performance Improvement
- Improved performace of 'ORDER BY' and 'GROUP BY' by prefix of sorting key (enabled with
optimize_aggregation_in_order
setting, disabled by default). #11696 (Anton Popov). - โก๏ธ Removed injective functions inside
uniq*()
ifset optimize_injective_functions_inside_uniq=1
. #12337 (Ruslan Kamalov). - ๐ Index not used for IN operator with literals", performance regression introduced around v19.3. This fixes "#10574. #12062 (nvartolomei).
- Implemented single part uploads for DiskS3 (experimental feature). #12026 (Vladimir Chebotarev).
Experimental Feature
- Added new in-memory format of parts in
MergeTree
-family tables, which stores data in memory. Parts are written on disk at first merge. Part will be created in in-memory format if its size in rows or bytes is below thresholdsmin_rows_for_compact_part
andmin_bytes_for_compact_part
. Also optional support of Write-Ahead-Log is available, which is enabled by default and is controlled by settingin_memory_parts_enable_wal
. #10697 (Anton Popov).
๐ Build/Testing/Packaging Improvement
- Implement AST-based query fuzzing mode for clickhouse-client. See this label for the list of issues we recently found by fuzzing. Most of them were found by this tool, and a couple by SQLancer and
00746_sql_fuzzy.pl
. #12111 (Alexander Kuzmenkov). - โ Add new type of tests based on Testflows framework. #12090 (vzakaznikov).
- โ Added S3 HTTPS integration test. #12412 (Pavel Kovalenko).
- ๐ฒ Log sanitizer trap messages from separate thread. This will prevent possible deadlock under thread sanitizer. #12313 (alexey-milovidov).
- โ
Now functional and stress tests will be able to run with old version of
clickhouse-test
script. #12287 (alesapin). - โ Remove strange file creation during build in
orc
. #12258 (Nikita Mikhaylov). - ๐ณ Place common docker compose files to integration docker container. #12168 (Ilya Yatsishin).
- ๐ Fix warnings from CodeQL.
CodeQL
is another static analyzer that we will use along withclang-tidy
andPVS-Studio
that we use already. #12138 (alexey-milovidov). - ๐ Minor CMake fixes for UNBUNDLED build. #12131 (Matwey V. Kornilov).
- โ Added a showcase of the minimal Docker image without using any Linux distribution. #12126 (alexey-milovidov).
- โฌ๏ธ Perform an upgrade of system packages in the
clickhouse-server
docker image. #12124 (Ivan Blinkov). - โ Add
UNBUNDLED
flag tosystem.build_options
table. Move skip lists forclickhouse-test
to clickhouse repo. #12107 (alesapin). - ๐ Regular check by Anchore Container Analysis security analysis tool that looks for CVE in
clickhouse-server
Docker image. Also confirms thatDockerfile
is buildable. Runs daily onmaster
and on pull-requests toDockerfile
. #12102 (Ivan Blinkov). - ๐ Daily check by GitHub CodeQL security analysis tool that looks for CWE. #12101 (Ivan Blinkov).
- โก๏ธ Install
ca-certificates
before the firstapt-get update
in Dockerfile. #12095 (Ivan Blinkov).
- โก๏ธ When upgrading from versions older than 20.5, if rolling update is performed and cluster contains both versions 20.5 or greater and less than 20.5, if ClickHouse nodes with old versions are restarted and old version has been started up in presence of newer versions, it may lead to