ClickHouse v20.5.2.7

« Changelog History

ClickHouse v20.5.2.7 Release Notes

• Backward Incompatible Change

  • Return non-Nullable result from COUNT(DISTINCT), and uniq aggregate functions family. If all passed values are NULL, return zero instead. This improves SQL compatibility. #11661 (alexey-milovidov).
  • Added a check for the case when user-level setting is specified in a wrong place. User-level settings should be specified in users.xml inside <profile> section for specific user profile (or in <default> for default settings). The server won't start with exception message in log. This fixes #9051. If you want to skip the check, you can either move settings to the appropriate place or add <skip_check_for_incorrect_settings>1</skip_check_for_incorrect_settings> to config.xml. #11449 (alexey-milovidov).
  • The setting input_format_with_names_use_header is enabled by default. It will affect parsing of input formats -WithNames and -WithNamesAndTypes. #10937 (alexey-milovidov).
  • 👉 Remove experimental_use_processors setting. It is enabled by default. #10924 (Nikolai Kochetov).
  • 🐎 Update zstd to 1.4.4. It has some minor improvements in performance and compression ratio. If you run replicas with different versions of ClickHouse you may see reasonable error messages Data after merge is not byte-identical to data on another replicas. with explanation. These messages are Ok and you should not worry. This change is backward compatible but we list it here in changelog in case you will wonder about these messages. #10663 (alexey-milovidov).
  • Added a check for meaningless codecs and a setting allow_suspicious_codecs to control this check. This closes #4966. #10645 (alexey-milovidov).
  • 👀 Several Kafka setting changes their defaults. See #11388.
  • ⚡️ When upgrading from versions older than 20.5, if rolling update is performed and cluster contains both versions 20.5 or greater and less than 20.5, if ClickHouse nodes with old versions are restarted and old version has been started up in presence of newer versions, it may lead to Part ... intersects previous part errors. To prevent this error, first install newer clickhouse-server packages on all cluster nodes and then do restarts (so, when clickhouse-server is restarted, it will start up with the new version).

  🆕 New Feature

  • TTL DELETE WHERE and TTL GROUP BY for automatic data coarsening and rollup in tables. #10537 (expl0si0nn).
  • 🚚 Implementation of PostgreSQL wire protocol. #10242 (Movses).
  • ➕ Added system tables for users, roles, grants, settings profiles, quotas, row policies; added commands SHOW USER, SHOW [CURRENT|ENABLED] ROLES, SHOW SETTINGS PROFILES. #10387 (Vitaly Baranov).
  • 👌 Support writes in ODBC Table function #10554 (ageraab). #10901 (tavplubix).
  • Add query performance metrics based on Linux perf_events (these metrics are calculated with hardware CPU counters and OS counters). It is optional and requires CAP_SYS_ADMIN to be set on clickhouse binary. #9545 Andrey Skobtsov. #11226 (Alexander Kuzmenkov).
  • 👍 Now support NULL and NOT NULL modifiers for data types in CREATE query. #11057 (Павел Потемкин).
  • ➕ Add ArrowStream input and output format. #11088 (hcz).
  • 👌 Support Cassandra as external dictionary source. #4978 (favstovol).
  • ➕ Added a new layout direct which loads all the data directly from the source for each query, without storing or caching data. #10622 (Artem Streltsov).
  • Added new complex_key_direct layout to dictionaries, that does not store anything locally during query execution. #10850 (Artem Streltsov).
  • ➕ Added support for MySQL style global variables syntax (stub). This is needed for compatibility of MySQL protocol. #11832 (alexey-milovidov).
  • ➕ Added syntax highligting to clickhouse-client using replxx. #11422 (Tagir Kuskarov).
  • minMap and maxMap functions were added. #11603 (Ildus Kurbangaliev).
  • Add the system.asynchronous_metric_log table that logs historical metrics from system.asynchronous_metrics. #11588 (Alexander Kuzmenkov).
  • ➕ Add functions extractAllGroupsHorizontal(haystack, re) and extractAllGroupsVertical(haystack, re). #11554 (Vasily Nemkov).
  • ➕ Add SHOW CLUSTER(S) queries. #11467 (hexiaoting).
  • ➕ Add netloc function for extracting network location, similar to urlparse(url), netloc in python. #11356 (Guillaume Tassery).
  • ➕ Add 2 more virtual columns for engine=Kafka to access message headers. #11283 (filimonov).
  • Add _timestamp_ms virtual column for Kafka engine (type is Nullable(DateTime64(3))). #11260 (filimonov).
  • ➕ Add function randomFixedString. #10866 (Andrei Nekrashevich).
  • ➕ Add function fuzzBits that randomly flips bits in a string with given probability. #11237 (Andrei Nekrashevich).
  • 👍 Allow comparison of numbers with constant string in comparison operators, IN and VALUES sections. #11647 (alexey-milovidov).
  • Add round_robin load_balancing mode. #11645 (Azat Khuzhin).
  • Add cast_keep_nullable setting. If set CAST(something_nullable AS Type) return Nullable(Type). #11733 (Artem Zuikov).
  • Added column position to system.columns table and column_position to system.parts_columns table. It contains ordinal position of a column in a table starting with 1. This closes #7744. #11655 (alexey-milovidov).
  • 👍 ON CLUSTER support for SYSTEM {FLUSH DISTRIBUTED,STOP/START DISTRIBUTED SEND}. #11415 (Azat Khuzhin).
  • ➕ Add system.distribution_queue table. #11394 (Azat Khuzhin).
  • 👌 Support for all format settings in Kafka, expose some setting on table level, adjust the defaults for better performance. #11388 (filimonov).
  • ➕ Add port function (to extract port from URL). #11120 (Azat Khuzhin).
  • Now dictGet* functions accept table names. #11050 (Vitaly Baranov).
  • The clickhouse-format tool is now able to format multiple queries when the -n argument is used. #10852 (Darío).
  • 🔧 Possibility to configure proxy-resolver for DiskS3. #10744 (Pavel Kovalenko).
  • 👉 Make pointInPolygon work with non-constant polygon. PointInPolygon now can take Array(Array(Tuple(..., ...))) as second argument, array of polygon and holes. #10623 (Alexey Ilyukhov) #11421 (Alexey Ilyukhov).
  • Added move_ttl_info to system.parts in order to provide introspection of move TTL functionality. #10591 (Vladimir Chebotarev).
  • Possibility to work with S3 through proxies. #10576 (Pavel Kovalenko).
  • ➕ Add NCHAR and NVARCHAR synonims for data types. #11025 (alexey-milovidov).
  • Resolved #7224: added FailedQuery, FailedSelectQuery and FailedInsertQuery metrics to system.events table. #11151 (Nikita Orlov).
  • ➕ Add more jemalloc statistics to system.asynchronous_metrics, and ensure that we see up-to-date values for them. #11748 (Alexander Kuzmenkov).
  • 👍 Allow to specify default S3 credentials and custom auth headers. #11134 (Grigory Pervakov).
  • ➕ Added new functions to import/export DateTime64 as Int64 with various precision: to-/fromUnixTimestamp64Milli/-Micro/-Nano. #10923 (Vasily Nemkov).
  • 👍 Allow specifying mongodb:// URI for MongoDB dictionaries. #10915 (Alexander Kuzmenkov).
  • OFFSET keyword can now be used without an affiliated LIMIT clause. #10802 (Guillaume Tassery).
  • ➕ Added system.licenses table. This table contains licenses of third-party libraries that are located in contrib directory. This closes #2890. #10795 (alexey-milovidov).
  • 🆕 New function function toStartOfSecond(DateTime64) -> DateTime64 that nullifies sub-second part of DateTime64 value. #10722 (Vasily Nemkov).
  • ➕ Add new input format JSONAsString that accepts a sequence of JSON objects separated by newlines, spaces and/or commas. #10607 (Kruglov Pavel).
  • 👍 Allowed to profile memory with finer granularity steps than 4 MiB. Added sampling memory profiler to capture random allocations/deallocations. #10598 (alexey-milovidov).
  • 👍 SimpleAggregateFunction now also supports sumMap. #10000 (Ildus Kurbangaliev).
  • 👌 Support ALTER RENAME COLUMN for the distributed table engine. Continuation of #10727. Fixes #10747. #10887 (alesapin).

  🐛 Bug Fix

  • 🛠 Fix UBSan report in Decimal parse. This fixes #7540. #10512 (alexey-milovidov).
  • 🛠 Fix potential floating point exception when parsing DateTime64. This fixes #11374. #11875 (alexey-milovidov).
  • 🛠 Fix rare crash caused by using Nullable column in prewhere condition. #11895 #11608 #11869 (Nikolai Kochetov).
  • 🔀 Don't allow arrayJoin inside higher order functions. It was leading to broken protocol synchronization. This closes #3933. #11846 (alexey-milovidov).
  • 🛠 Fix wrong result of comparison of FixedString with constant String. This fixes #11393. This bug appeared in version 20.4. #11828 (alexey-milovidov).
  • 🛠 Fix wrong result for if with NULLs in condition. #11807 (Artem Zuikov).
  • 🛠 Fix using too many threads for queries. #11788 (Nikolai Kochetov).
  • Fixed Scalar doesn't exist exception when using WITH <scalar subquery> ... in SELECT ... FROM merge_tree_table ... https://github.com/ClickHouse/ClickHouse/issues/11621. #11767 (Amos Bird).
  • Fix unexpected behaviour of queries like SELECT *, xyz.* which were success while an error expected. #11753 (hexiaoting).
  • 📇 Now replicated fetches will be cancelled during metadata alter. #11744 (alesapin).
  • 📇 Parse metadata stored in zookeeper before checking for equality. #11739 (Azat Khuzhin).
  • 🛠 Fixed LOGICAL_ERROR caused by wrong type deduction of complex literals in Values input format. #11732 (tavplubix).
  • 🛠 Fix ORDER BY ... WITH FILL over const columns. #11697 (Anton Popov).
  • 🛠 Fix very rare race condition in SYSTEM SYNC REPLICA. If the replicated table is created and at the same time from the separate connection another client is issuing SYSTEM SYNC REPLICA command on that table (this is unlikely, because another client should be aware that the table is created), it's possible to get nullptr dereference. #11691 (alexey-milovidov).
  • Pass proper timeouts when communicating with XDBC bridge. Recently timeouts were not respected when checking bridge liveness and receiving meta info. #11690 (alexey-milovidov).
  • 🛠 Fix LIMIT n WITH TIES usage together with ORDER BY statement, which contains aliases. #11689 (Anton Popov).
  • 🛠 Fix possible Pipeline stuck for selects with parallel FINAL. Fixes #11636. #11682 (Nikolai Kochetov).
  • 🛠 Fix error which leads to an incorrect state of system.mutations. It may show that whole mutation is already done but the server still has MUTATE_PART tasks in the replication queue and tries to execute them. This fixes #11611. #11681 (alesapin).
  • 🛠 Fix syntax hilite in CREATE USER query. #11664 (alexey-milovidov).
  • ➕ Add support for regular expressions with case-insensitive flags. This fixes #11101 and fixes #11506. #11649 (alexey-milovidov).
  • ✂ Remove trivial count query optimization if row-level security is set. In previous versions the user get total count of records in a table instead filtered. This fixes #11352. #11644 (alexey-milovidov).
  • 🛠 Fix bloom filters for String (data skipping indices). #11638 (Azat Khuzhin).
  • Without -q option the database does not get created at startup. #11604 (giordyb).
  • 🛠 Fix error Block structure mismatch for queries with sampling reading from Buffer table. #11602 (Nikolai Kochetov).
  • 🛠 Fix wrong exit code of the clickhouse-client, when exception.code() % 256 == 0. #11601 (filimonov).
  • 🛠 Fix race conditions in CREATE/DROP of different replicas of ReplicatedMergeTree. Continue to work if the table was not removed completely from ZooKeeper or not created successfully. This fixes #11432. #11592 (alexey-milovidov).
  • 🛠 Fix trivial error in log message about "Mark cache size was lowered" at server startup. This closes #11399. #11589 (alexey-milovidov).
  • 🛠 Fix error Size of offsets doesn't match size of column for queries with PREWHERE column in (subquery) and ARRAY JOIN. #11580 (Nikolai Kochetov).
  • 🛠 Fixed rare segfault in SHOW CREATE TABLE Fixes #11490. #11579 (tavplubix).
  • 🛠 All queries in HTTP session have had the same query_id. It is fixed. #11578 (tavplubix).
  • 🐳 Now clickhouse-server docker container will prefer IPv6 checking server aliveness. #11550 (Ivan Starkov).
  • Fix the error Data compressed with different methods that can happen if min_bytes_to_use_direct_io is enabled and PREWHERE is active and using SAMPLE or high number of threads. This fixes #11539. #11540 (alexey-milovidov).
  • Fix shard_num/replica_num for <node> (breaks use_compact_format_in_distributed_parts_names). #11528 (Azat Khuzhin).
  • Fix async INSERT into Distributed for prefer_localhost_replica=0 and w/o internal_replication. #11527 (Azat Khuzhin).
  • 🛠 Fix memory leak when exception is thrown in the middle of aggregation with -State functions. This fixes #8995. #11496 (alexey-milovidov).
  • Fix Pipeline stuck exception for INSERT SELECT FINAL where SELECT (max_threads>1) has multiple streams but INSERT has only one (max_insert_threads==0). #11455 (Azat Khuzhin).
  • 🛠 Fix wrong result in queries like select count() from t, u. #11454 (Artem Zuikov).
  • 🛠 Fix return compressed size for codecs. #11448 (Nikolai Kochetov).
  • 🛠 Fix server crash when a column has compression codec with non-literal arguments. Fixes #11365. #11431 (alesapin).
  • 🛠 Fix potential uninitialized memory read in MergeTree shutdown if table was not created successfully. #11420 (alexey-milovidov).
  • 🛠 Fix crash in JOIN over LowCarinality(T) and Nullable(T). #11380. #11414 (Artem Zuikov).
  • 🛠 Fix error code for wrong USING key. #11373. #11404 (Artem Zuikov).
  • 🛠 Fixed geohashesInBox with arguments outside of latitude/longitude range. #11403 (Vasily Nemkov).
  • 👍 Better errors for joinGet() functions. #11389 (Artem Zuikov).
  • 🛠 Fix possible Pipeline stuck error for queries with external sort and limit. Fixes #11359. #11366 (Nikolai Kochetov).
  • ✂ Remove redundant lock during parts send in ReplicatedMergeTree. #11354 (alesapin).
  • 🛠 Fix support for \G (vertical output) in clickhouse-client in multiline mode. This closes #9933. #11350 (alexey-milovidov).
  • 🛠 Fix potential segfault when using Lazy database. #11348 (alexey-milovidov).
  • 🛠 Fix crash in direct selects from Join table engine (without JOIN) and wrong nullability. #11340 (Artem Zuikov).
  • 🛠 Fix crash in quantilesExactWeightedArray. #11337 (Nikolai Kochetov).
  • 📇 Now merges stopped before change metadata in ALTER queries. #11335 (alesapin).
  • Make writing to MATERIALIZED VIEW with setting parallel_view_processing = 1 parallel again. Fixes #10241. #11330 (Nikolai Kochetov).
  • 🛠 Fix visitParamExtractRaw when extracted JSON has strings with unbalanced { or [. #11318 (Ewout).
  • 🛠 Fix very rare race condition in ThreadPool. #11314 (alexey-milovidov).
  • 🛠 Fix insignificant data race in clickhouse-copier. Found by integration tests. #11313 (alexey-milovidov).
  • 🛠 Fix potential uninitialized memory in conversion. Example: SELECT toIntervalSecond(now64()). #11311 (alexey-milovidov).
  • 🛠 Fix the issue when index analysis cannot work if a table has Array column in primary key and if a query is filtering by this column with empty or notEmpty functions. This fixes #11286. #11303 (alexey-milovidov).
  • Fix bug when query speed estimation can be incorrect and the limit of min_execution_speed may not work or work incorrectly if the query is throttled by max_network_bandwidth, max_execution_speed or priority settings. Change the default value of timeout_before_checking_execution_speed to non-zero, because otherwise the settings min_execution_speed and max_execution_speed have no effect. This fixes #11297. This fixes #5732. This fixes #6228. Usability improvement: avoid concatenation of exception message with progress bar in clickhouse-client. #11296 (alexey-milovidov).
  • 🛠 Fix crash when SET DEFAULT ROLE is called with wrong arguments. This fixes https://github.com/ClickHouse/ClickHouse/issues/10586. #11278 (Vitaly Baranov).
  • 🛠 Fix crash while reading malformed data in Protobuf format. This fixes https://github.com/ClickHouse/ClickHouse/issues/5957, fixes https://github.com/ClickHouse/ClickHouse/issues/11203. #11258 (Vitaly Baranov).
  • 🛠 Fixed a bug when cache dictionary could return default value instead of normal (when there are only expired keys). This affects only string fields. #11233 (Nikita Mikhaylov).
  • 🛠 Fix error Block structure mismatch in QueryPipeline while reading from VIEW with constants in inner query. Fixes #11181. #11205 (Nikolai Kochetov).
  • 🛠 Fix possible exception Invalid status for associated output. #11200 (Nikolai Kochetov).
  • Now primary.idx will be checked if it's defined in CREATE query. #11199 (alesapin).
  • 🛠 Fix possible error Cannot capture column for higher-order functions with Array(Array(LowCardinality)) captured argument. #11185 (Nikolai Kochetov).
  • 🛠 Fixed S3 globbing which could fail in case of more than 1000 keys and some backends. #11179 (Vladimir Chebotarev).
  • 🔀 If data skipping index is dependent on columns that are going to be modified during background merge (for SummingMergeTree, AggregatingMergeTree as well as for TTL GROUP BY), it was calculated incorrectly. This issue is fixed by moving index calculation after merge so the index is calculated on merged data. #11162 (Azat Khuzhin).
  • 🛠 Fix for the hang which was happening sometimes during DROP of table engine=Kafka (or during server restarts). #11145 (filimonov).
  • 🛠 Fix excessive reserving of threads for simple queries (optimization for reducing the number of threads, which was partly broken after changes in pipeline). #11114 (Azat Khuzhin).
  • ✂ Remove logging from mutation finalization task if nothing was finalized. #11109 (alesapin).
  • 🛠 Fixed deadlock during server startup after update with changes in structure of system log tables. #11106 (alesapin).
  • 🛠 Fixed memory leak in registerDiskS3. #11074 (Pavel Kovalenko).
  • 🚚 Fix error No such name in Block::erase() when JOIN appears with PREWHERE or optimize_move_to_prewhere makes PREWHERE from WHERE. #11051 (Artem Zuikov).
  • 🛠 Fixes the potential missed data during termination of Kafka engine table. #11048 (filimonov).
  • 🛠 Fixed parseDateTime64BestEffort argument resolution bugs. #10925. #11038 (Vasily Nemkov).
  • 🛠 Now it's possible to ADD/DROP and RENAME the same one column in a single ALTER query. Exception message for simultaneous MODIFY and RENAME became more clear. Partially fixes #10669. #11037 (alesapin).
  • 🛠 Fixed parsing of S3 URLs. #11036 (Vladimir Chebotarev).
  • 🛠 Fix memory tracking for two-level GROUP BY when there is a LIMIT. #11022 (Azat Khuzhin).
  • 🛠 Fix very rare potential use-after-free error in MergeTree if table was not created successfully. #10986 (alexey-milovidov).
  • 🛠 Fix metadata (relative path for rename) and data (relative path for symlink) handling for Atomic database. #10980 (Azat Khuzhin).
  • 🛠 Fix server crash on concurrent ALTER and DROP DATABASE queries with Atomic database engine. #10968 (tavplubix).
  • 🛠 Fix incorrect raw data size in method getRawData(). #10964 (Igr).
  • 🛠 Fix incompatibility of two-level aggregation between versions 20.1 and earlier. This incompatibility happens when different versions of ClickHouse are used on initiator node and remote nodes and the size of GROUP BY result is large and aggregation is performed by a single String field. It leads to several unmerged rows for a single key in result. #10952 (alexey-milovidov).
  • Avoid sending partially written files by the DistributedBlockOutputStream. #10940 (Azat Khuzhin).
  • 🛠 Fix crash in SELECT count(notNullIn(NULL, [])). #10920 (Nikolai Kochetov).
  • 🛠 Fix for the hang which was happening sometimes during DROP of table engine=Kafka (or during server restarts). #10910 (filimonov).
  • Now it's possible to execute multiple ALTER RENAME like a TO b, c TO a. #10895 (alesapin).
  • 🛠 Fix possible race which could happen when you get result from aggregate function state from multiple thread for the same column. The only way (which I found) it can happen is when you use finalizeAggregation function while reading from table with Memory engine which stores AggregateFunction state for quanite* function. #10890 (Nikolai Kochetov).
  • 🛠 Fix backward compatibility with tuples in Distributed tables. #10889 (Anton Popov).
  • 🛠 Fix SIGSEGV in StringHashTable (if such key does not exist). #10870 (Azat Khuzhin).
  • 🛠 Fixed WATCH hangs after LiveView table was dropped from database with Atomic engine. #10859 (tavplubix).
  • 🛠 Fixed bug in ReplicatedMergeTree which might cause some ALTER on OPTIMIZE query to hang waiting for some replica after it become inactive. #10849 (tavplubix).
  • ⚡️ Now constraints are updated if the column participating in CONSTRAINT expression was renamed. Fixes #10844. #10847 (alesapin).
  • 🛠 Fix potential read of uninitialized memory in cache dictionary. #10834 (alexey-milovidov).
  • 🛠 Fix columns order after Block::sortColumns() (also add a test that shows that it affects some real use case - Buffer engine). #10826 (Azat Khuzhin).
  • 🛠 Fix the issue with ODBC bridge when no quoting of identifiers is requested. This fixes #7984. #10821 (alexey-milovidov).
  • 🛠 Fix UBSan and MSan report in DateLUT. #10798 (alexey-milovidov).
  • 🛠 Make use of src_type for correct type conversion in key conditions. Fixes #6287. #10791 (Andrew Onyshchuk).
  • 🐎 Get rid of old libunwind patches. https://github.com/ClickHouse-Extras/libunwind/commit/500aa227911bd185a94bfc071d68f4d3b03cb3b1#r39048012 This allows to disable -fno-omit-frame-pointer in clang builds that improves performance at least by 1% in average. #10761 (Amos Bird).
  • 🛠 Fix avgWeighted when using floating-point weight over multiple shards. #10758 (Baudouin Giard).
  • Fix parallel_view_processing behavior. Now all insertions into MATERIALIZED VIEW without exception should be finished if exception happened. Fixes #10241. #10757 (Nikolai Kochetov).
  • 🛠 Fix combinator -OrNull and -OrDefault when combined with -State. #10741 (hcz).
  • 🛠 Fix crash in generateRandom with nested types. Fixes #10583. #10734 (Nikolai Kochetov).
  • 🛠 Fix data corruption for LowCardinality(FixedString) key column in SummingMergeTree which could have happened after merge. Fixes #10489. #10721 (Nikolai Kochetov).
  • 🛠 Fix usage of primary key wrapped into a function with 'FINAL' modifier and 'ORDER BY' optimization. #10715 (Anton Popov).
  • 🛠 Fix possible buffer overflow in function h3EdgeAngle. #10711 (alexey-milovidov).
  • 🛠 Fix disappearing totals. Totals could have being filtered if query had had join or subquery with external where condition. Fixes #10674. #10698 (Nikolai Kochetov).
  • 🛠 Fix atomicity of HTTP insert. This fixes #9666. #10687 (Andrew Onyshchuk).
  • 🛠 Fix multiple usages of IN operator with the identical set in one query. #10686 (Anton Popov).
  • Fixed bug, which causes http requests stuck on client close when readonly=2 and cancel_http_readonly_queries_on_client_close=1. Fixes #7939, #7019, #7736, #7091. #10684 (tavplubix).
  • 🛠 Fix order of parameters in AggregateTransform constructor. #10667 (palasonic1).
  • Fix the lack of parallel execution of remote queries with distributed_aggregation_memory_efficient enabled. Fixes #10655. #10664 (Nikolai Kochetov).
  • 🛠 Fix possible incorrect number of rows for queries with LIMIT. Fixes #10566, #10709. #10660 (Nikolai Kochetov).
  • 🛠 Fix bug which locks concurrent alters when table has a lot of parts. #10659 (alesapin).
  • 🛠 Fix nullptr dereference in StorageBuffer if server was shutdown before table startup. #10641 (alexey-milovidov).
  • ⚡️ Fix predicates optimization for distributed queries (enable_optimize_predicate_expression=1) for queries with HAVING section (i.e. when filtering on the server initiator is required), by preserving the order of expressions (and this is enough to fix), and also force aggregator use column names over indexes. Fixes: #10613, #11413. #10621 (Azat Khuzhin).
  • Fix optimize_skip_unused_shards with LowCardinality. #10611 (Azat Khuzhin).
  • 🛠 Fix segfault in StorageBuffer when exception on server startup. Fixes #10550. #10609 (tavplubix).
  • On SYSTEM DROP DNS CACHE query also drop caches, which are used to check if user is allowed to connect from some IP addresses. #10608 (tavplubix).
  • 🛠 Fixed incorrect scalar results inside inner query of MATERIALIZED VIEW in case if this query contained dependent table. #10603 (Nikolai Kochetov).
  • 🛠 Fixed handling condition variable for synchronous mutations. In some cases signals to that condition variable could be lost. #10588 (Vladimir Chebotarev).
  • 🛠 Fixes possible crash createDictionary() is called before loadStoredObject() has finished. #10587 (Vitaly Baranov).
  • 🛠 Fix error the BloomFilter false positive must be a double number between 0 and 1 #10551. #10569 (Winter Zhang).
  • 🛠 Fix SELECT of column ALIAS which default expression type different from column type. #10563 (Azat Khuzhin).
  • Implemented comparison between DateTime64 and String values (just like for DateTime). #10560 (Vasily Nemkov).
  • 🛠 Fix index corruption, which may accur in some cases after merge compact parts into another compact part. #10531 (Anton Popov).
  • ⚡️ Disable GROUP BY sharding_key optimization by default (optimize_distributed_group_by_sharding_key had been introduced and turned of by default, due to trickery of sharding_key analyzing, simple example is if in sharding key) and fix it for WITH ROLLUP/CUBE/TOTALS. #10516 (Azat Khuzhin).
  • Fixes: #10263 (after that PR dist send via INSERT had been postponing on each INSERT) Fixes: #8756 (that PR breaks distributed sends with all of the following conditions met (unlikely setup for now I guess): internal_replication == false, multiple local shards (activates the hardlinking code) and distributed_storage_policy (makes link(2) fails on EXDEV)). #10486 (Azat Khuzhin).
  • Fixed error with "max_rows_to_sort" limit. #10268 (alexey-milovidov).
  • Get dictionary and check access rights only once per each call of any function reading external dictionaries. #10928 (Vitaly Baranov).

  👌 Improvement

  • Apply TTL for old data, after ALTER MODIFY TTL query. This behaviour is controlled by setting materialize_ttl_after_modify, which is enabled by default. #11042 (Anton Popov).
  • When parsing C-style backslash escapes in string literals, VALUES and various text formats (this is an extension to SQL standard that is endemic for ClickHouse and MySQL), keep backslash if unknown escape sequence is found (e.g. \% or \w) that will make usage of LIKE and match regular expressions more convenient (it's enough to write name LIKE 'used\_cars' instead of name LIKE 'used\\_cars') and more compatible at the same time. This fixes #10922. #11208 (alexey-milovidov).
  • 🛠 When reading Decimal value, cut extra digits after point. This behaviour is more compatible with MySQL and PostgreSQL. This fixes #10202. #11831 (alexey-milovidov).
  • 👍 Allow to DROP replicated table if the metadata in ZooKeeper was already removed and does not exist (this is also the case when using TestKeeper for testing and the server was restarted). Allow to RENAME replicated table even if there is an error communicating with ZooKeeper. This fixes #10720. #11652 (alexey-milovidov).
  • Slightly improve diagnostic of reading decimal from string. This closes #10202. #11829 (alexey-milovidov).
  • 🛠 Fix sleep invocation in signal handler. It was sleeping for less amount of time than expected. #11825 (alexey-milovidov).
  • (Only Linux) OS related performance metrics (for CPU and I/O) will work even without CAP_NET_ADMIN capability. #10544 (Alexander Kazakov).
  • ➕ Added hostname as an alias to function hostName. This feature was suggested by Victor Tarnavskiy from Yandex.Metrica. #11821 (alexey-milovidov).
  • ➕ Added support for distributed DDL (update/delete/drop partition) on cross replication clusters. #11703 (Nikita Mikhaylov).
  • 🐳 Emit warning instead of error in server log at startup if we cannot listen one of the listen addresses (e.g. IPv6 is unavailable inside Docker). Note that if server fails to listen all listed addresses, it will refuse to startup as before. This fixes #4406. #11687 (alexey-milovidov).
  • 🐳 Default user and database creation on docker image starting. #10637 (Paramtamtam).
  • 🖨 When multiline query is printed to server log, the lines are joined. Make it to work correct in case of multiline string literals, identifiers and single-line comments. This fixes #3853. #11686 (alexey-milovidov).
  • Multiple names are now allowed in commands: CREATE USER, CREATE ROLE, ALTER USER, SHOW CREATE USER, SHOW GRANTS and so on. #11670 (Vitaly Baranov).
  • ➕ Add support for distributed DDL (UPDATE/DELETE/DROP PARTITION) on cross replication clusters. #11508 (frank lee).
  • 💻 Clear password from command line in clickhouse-client and clickhouse-benchmark if the user has specified it with explicit value. This prevents password exposure by ps and similar tools. #11665 (alexey-milovidov).
  • 🖨 Don't use debug info from ELF file if it doesn't correspond to the running binary. It is needed to avoid printing wrong function names and source locations in stack traces. This fixes #7514. #11657 (alexey-milovidov).
  • 📜 Return NULL/zero when value is not parsed completely in parseDateTimeBestEffortOrNull/Zero functions. This fixes #7876. #11653 (alexey-milovidov).
  • Skip empty parameters in requested URL. They may appear when you write http://localhost:8123/?&a=b or http://localhost:8123/?a=b&&c=d. This closes #10749. #11651 (alexey-milovidov).
  • 👍 Allow using groupArrayArray and groupUniqArrayArray as SimpleAggregateFunction. #11650 (Volodymyr Kuznetsov).
  • 👍 Allow comparison with constant strings by implicit conversions when analysing index conditions on other types. This may close #11630. #11648 (alexey-milovidov).
  • 0️⃣ https://github.com/ClickHouse/ClickHouse/pull/7572#issuecomment-642815377 Support config default HTTPHandlers. #11628 (Winter Zhang).
  • Make more input formats to work with Kafka engine. Fix the issue with premature flushes. Fix the performance issue when kafka_num_consumers is greater than number of partitions in topic. #11599 (filimonov).
  • Improve multiple_joins_rewriter_version=2 logic. Fix unknown columns error for lambda aliases. #11587 (Artem Zuikov).
  • 👍 Better exception message when cannot parse columns declaration list. This closes #10403. #11537 (alexey-milovidov).
  • ⚡️ Improve enable_optimize_predicate_expression=1 logic for VIEW. #11513 (Artem Zuikov).
  • ➕ Adding support for PREWHERE in live view tables. #11495 (vzakaznikov).
  • ⚡️ Automatically update DNS cache, which is used to check if user is allowed to connect from an address. #11487 (tavplubix).
  • ⚡️ OPTIMIZE FINAL will force merge even if concurrent merges are performed. This closes #11309 and closes #11322. #11346 (alexey-milovidov).
  • 🖨 Suppress output of cancelled queries in clickhouse-client. In previous versions result may continue to print in terminal even after you press Ctrl+C to cancel query. This closes #9473. #11342 (alexey-milovidov).
  • ⚡️ Now history file is updated after each query and there is no race condition if multiple clients use one history file. This fixes #9897. #11453 (Tagir Kuskarov).
  • 👍 Better log messages in while reloading configuration. #11341 (alexey-milovidov).
  • ✂ Remove trailing whitespaces from formatted queries in clickhouse-client or clickhouse-format in some cases. #11325 (alexey-milovidov).
  • Add setting "output_format_pretty_max_value_width". If value is longer, it will be cut to avoid output of too large values in terminal. This closes #11140. #11324 (alexey-milovidov).
  • 👍 Better exception message in case when there is shortage of memory mappings. This closes #11027. #11316 (alexey-milovidov).
  • 👌 Support (U)Int8, (U)Int16, Date in ASOF JOIN. #11301 (Artem Zuikov).
  • Support kafka_client_id parameter for Kafka tables. It also changes the default client.id used by ClickHouse when communicating with Kafka to be more verbose and usable. #11252 (filimonov).
  • 👻 Keep the value of DistributedFilesToInsert metric on exceptions. In previous versions, the value was set when we are going to send some files, but it is zero, if there was an exception and some files are still pending. Now it corresponds to the number of pending files in filesystem. #11220 (alexey-milovidov).
  • ➕ Add support for multi-word data type names (such as DOUBLE PRECISION and CHAR VARYING) for better SQL compatibility. #11214 (Павел Потемкин).
  • Provide synonyms for some data types. #10856 (Павел Потемкин).
  • 0️⃣ The query log is now enabled by default. #11184 (Ivan Blinkov).
  • 👉 Show authentication type in table system.users and while executing SHOW CREATE USER query. #11080 (Vitaly Baranov).
  • ✂ Remove data on explicit DROP DATABASE for Memory database engine. Fixes #10557. #11021 (tavplubix).
  • 🔊 Set thread names for internal threads of rdkafka library. Make logs from rdkafka available in server logs. #10983 (Azat Khuzhin).
  • 👌 Support for unicode whitespaces in queries. This helps when queries are copy-pasted from Word or from web page. This fixes #10896. #10903 (alexey-milovidov).
  • 👍 Allow large UInt types as the index in function tupleElement. #10874 (hcz).
  • Respect prefer_localhost_replica/load_balancing on INSERT into Distributed. #10867 (Azat Khuzhin).
  • Introduce min_insert_block_size_rows_for_materialized_views, min_insert_block_size_bytes_for_materialized_views settings. This settings are similar to min_insert_block_size_rows and min_insert_block_size_bytes, but applied only for blocks inserted into MATERIALIZED VIEW. It helps to control blocks squashing while pushing to MVs and avoid excessive memory usage. #10858 (Azat Khuzhin).
  • 🛠 Get rid of exception from replicated queue during server shutdown. Fixes #10819. #10841 (alesapin).
  • 🛠 Ensure that varSamp, varPop cannot return negative results due to numerical errors and that stddevSamp, stddevPop cannot be calculated from negative variance. This fixes #10532. #10829 (alexey-milovidov).
  • 👍 Better DNS exception message. This fixes #10813. #10828 (alexey-milovidov).
  • 🔄 Change HTTP response code in case of some parse errors to 400 Bad Request. This fix #10636. #10640 (alexey-milovidov).
  • 🖨 Print a message if clickhouse-client is newer than clickhouse-server. #10627 (alexey-milovidov).
  • ➕ Adding support for INSERT INTO [db.]table WATCH query. #10498 (vzakaznikov).
  • 👍 Allow to pass quota_key in clickhouse-client. This closes #10227. #10270 (alexey-milovidov).

  🐎 Performance Improvement

  • 👍 Allow multiple replicas to assign merges, mutations, partition drop, move and replace concurrently. This closes #10367. #11639 (alexey-milovidov) #11795 (alexey-milovidov).
  • Optimization of GROUP BY with respect to table sorting key, enabled with optimize_aggregation_in_order setting. #9113 (dimarub2000).
  • Selects with final are executed in parallel. Added setting max_final_threads to limit the number of threads used. #10463 (Nikolai Kochetov).
  • 👌 Improve performance for INSERT queries via INSERT SELECT or INSERT with clickhouse-client when small blocks are generated (typical case with parallel parsing). This fixes #11275. Fix the issue that CONSTRAINTs were not working for DEFAULT fields. This fixes #11273. Fix the issue that CONSTRAINTS were ignored for TEMPORARY tables. This fixes #11274. #11276 (alexey-milovidov).
  • Optimization that eliminates min/max/any aggregators of GROUP BY keys in SELECT section, enabled with optimize_aggregators_of_group_by_keys setting. #11667 (xPoSx). #11806 (Azat Khuzhin).
  • 🚚 New optimization that takes all operations out of any function, enabled with optimize_move_functions_out_of_any #11529 (Ruslan).
  • 👌 Improve performance of clickhouse-client in interactive mode when Pretty formats are used. In previous versions, significant amount of time can be spent calculating visible width of UTF-8 string. This closes #11323. #11323 (alexey-milovidov).
  • Improved performance for queries with ORDER BY and small LIMIT (less, then max_block_size). #11171 (Albert Kidrachev).
  • ➕ Add runtime CPU detection to select and dispatch the best function implementation. Add support for codegeneration for multiple targets. This closes #1017. #10058 (DimasKovas).
  • 0️⃣ Enable mlock of clickhouse binary by default. It will prevent clickhouse executable from being paged out under high IO load. #11139 (alexey-milovidov).
  • 👉 Make queries with sum aggregate function and without GROUP BY keys to run multiple times faster. #10992 (alexey-milovidov).
  • 🚚 Improving radix sort (used in ORDER BY with simple keys) by removing some redundant data moves. #10981 (Arslan Gumerov).
  • 🔀 Sort bigger parts of the left table in MergeJoin. Buffer left blocks in memory. Add partial_merge_join_left_table_buffer_bytes setting to manage the left blocks buffers sizes. #10601 (Artem Zuikov).
  • Remove duplicate ORDER BY and DISTINCT from subqueries, this optimization is enabled with optimize_duplicate_order_by_and_distinct #10067 (Mikhail Malafeev).
  • This feature eliminates functions of other keys in GROUP BY section, enabled with optimize_group_by_function_keys #10051 (xPoSx).
  • New optimization that takes arithmetic operations out of aggregate functions, enabled with optimize_arithmetic_operations_in_aggregate_functions #10047 (Ruslan).
  • 🐎 Use HTTP client for S3 based on Poco instead of curl. This will improve performance and lower memory usage of s3 storage and table functions. #11230 (Pavel Kovalenko).
  • 🛠 Fix Kafka performance issue related to reschedules based on limits, which were always applied. #11149 (filimonov).
  • Enable percpu_arena:percpu for jemalloc (This will reduce memory fragmentation due to thread pool). #11084 (Azat Khuzhin).
  • ⚡️ Optimize memory usage when reading a response from an S3 HTTP client. #11561 (Pavel Kovalenko).
  • 🐎 Adjust the default Kafka settings for better performance. #11388 (filimonov).

  Experimental Feature

  • ➕ Add data type Point (Tuple(Float64, Float64)) and Polygon (Array(Array(Tuple(Float64, Float64))). #10678 (Alexey Ilyukhov).
  • ➕ Add's a hasSubstr function that allows for look for subsequences in arrays. Note: this function is likely to be renamed without further notice. #11071 (Ryad Zenine).
  • Added OpenCL support and bitonic sort algorithm, which can be used for sorting integer types of data in single column. Needs to be build with flag -DENABLE_OPENCL=1. For using bitonic sort algorithm instead of others you need to set bitonic_sort for Setting's option special_sort and make sure that OpenCL is available. This feature does not improve performance or anything else, it is only provided as an example and for demonstration purposes. It is likely to be removed in near future if there will be no further development in this direction. #10232 (Ri).

  🏗 Build/Testing/Packaging Improvement

  • Enable clang-tidy for programs and utils. #10991 (alexey-milovidov).
  • ✂ Remove dependency on tzdata: do not fail if /usr/share/zoneinfo directory does not exist. Note that all timezones work in ClickHouse even without tzdata installed in system. #11827 (alexey-milovidov).
  • ➕ Added MSan and UBSan stress tests. Note that we already have MSan, UBSan for functional tests and "stress" test is another kind of tests. #10871 (alexey-milovidov).
  • 🏗 Print compiler build id in crash messages. It will make us slightly more certain about what binary has crashed. Added new function buildId. #11824 (alexey-milovidov).
  • ➕ Added a test to ensure that mutations continue to work after FREEZE query. #11820 (alexey-milovidov).
  • ✅ Don't allow tests with "fail" substring in their names because it makes looking at the tests results in browser less convenient when you type Ctrl+F and search for "fail". #11817 (alexey-milovidov).
  • ✂ Removes unused imports from HTTPHandlerFactory. #11660 (Bharat Nallan).
  • ➕ Added a random sampling of instances where copier is executed. It is needed to avoid Too many simultaneous queries error. Also increased timeout and decreased fault probability. #11573 (Nikita Mikhaylov).
  • 🛠 Fix missed include. #11525 (Matwey V. Kornilov).
  • 🏗 Speed up build by removing old example programs. Also found some orphan functional test. #11486 (alexey-milovidov).
  • 🏗 Increase ccache size for builds in CI. #11450 (alesapin).
  • ✅ Leave only unit_tests_dbms in deb build. #11429 (Ilya Yatsishin).
  • 🚀 Update librdkafka to version 1.4.2. #11256 (filimonov).
  • 🔨 Refactor CMake build files. #11390 (Ivan).
  • 🛠 Fix several flaky integration tests. #11355 (alesapin).
  • ➕ Add support for unit tests run with UBSan. #11345 (alexey-milovidov).
  • Remove redundant timeout from integration test test_insertion_sync_fails_with_timeout. #11343 (alesapin).
  • 👍 Better check for hung queries in clickhouse-test. #11321 (alexey-milovidov).
  • 🏗 Emit a warning if server was build in debug or with sanitizers. #11304 (alexey-milovidov).
  • ✅ Now clickhouse-test check the server aliveness before tests run. #11285 (alesapin).
  • Fix potentially flacky test 00731_long_merge_tree_select_opened_files.sh. It does not fail frequently but we have discovered potential race condition in this test while experimenting with ThreadFuzzer: #9814 See link for the example. #11270 (alexey-milovidov).
  • 👷 Repeat test in CI if curl invocation was timed out. It is possible due to system hangups for 10+ seconds that are typical in our CI infrastructure. This fixes #11267. #11268 (alexey-milovidov).
  • ➕ Add a test for Join table engine from @donmikel. This closes #9158. #11265 (alexey-milovidov).
  • 🛠 Fix several non significant errors in unit tests. #11262 (alesapin).
  • Now parts of linker command for cctz library will not be shuffled with other libraries. #11213 (alesapin).
  • Split /programs/server into actual program and library. #11186 (Ivan).
  • 👌 Improve build scripts for protobuf & gRPC. #11172 (Vitaly Baranov).
  • 🐎 Enable performance test that was not working. #11158 (alexey-milovidov).
  • ✅ Create root S3 bucket for tests before any CH instance is started. #11142 (Pavel Kovalenko).
  • ➕ Add performance test for non-constant polygons. #11141 (alexey-milovidov).
  • Fixing 00979_live_view_watch_continuous_aggregates test. #11024 (vzakaznikov).
  • ➕ Add ability to run zookeeper in integration tests over tmpfs. #11002 (alesapin).
  • 👷 Wait for odbc-bridge with exponential backoff. Previous wait time of 200 ms was not enough in our CI environment. #10990 (alexey-milovidov).
  • 🛠 Fix non-deterministic test. #10989 (alexey-milovidov).
  • ➕ Added a test for empty external data. #10926 (alexey-milovidov).
  • ✅ Database is recreated for every test. This improves separation of tests. #10902 (alexey-milovidov).
  • ➕ Added more asserts in columns code. #10833 (alexey-milovidov).
  • 👍 Better cooperation with sanitizers. Print information about query_id in the message of sanitizer failure. #10832 (alexey-milovidov).
  • 🛠 Fix obvious race condition in "Split build smoke test" check. #10820 (alexey-milovidov).
  • 🛠 Fix (false) MSan report in MergeTreeIndexFullText. The issue first appeared in #9968. #10801 (alexey-milovidov).
  • ➕ Add MSan suppression for MariaDB Client library. #10800 (alexey-milovidov).
  • GRPC make couldn't find protobuf files, changed make file by adding the right link. #10794 (mnkonkova).
  • ⚠ Enable extra warnings (-Weverything) for base, utils, programs. Note that we already have it for the most of the code. #10779 (alexey-milovidov).
  • ⚠ Suppressions of warnings from libraries was mistakenly declared as public in #10396. #10776 (alexey-milovidov).
  • ⏪ Restore a patch that was accidentially deleted in #10396. #10774 (alexey-milovidov).
  • 🛠 Fix performance tests errors, part 2. #10773 (alexey-milovidov).
  • 🛠 Fix performance test errors. #10766 (alexey-milovidov).
  • ⚡️ Update cross-builds to use clang-10 compiler. #10724 (Ivan).
  • ⚡️ Update instruction to install RPM packages. This was suggested by Denis (TG login @ldviolet) and implemented by Arkady Shejn. #10707 (alexey-milovidov).
  • Trying to fix tests/queries/0_stateless/01246_insert_into_watch_live_view.py test. #10670 (vzakaznikov).
  • Fixing and re-enabling 00979_live_view_watch_continuous_aggregates.py test. #10658 (vzakaznikov).
  • 🛠 Fix OOM in ASan stress test. #10646 (alexey-milovidov).
  • 🛠 Fix UBSan report (adding zero to nullptr) in HashTable that appeared after migration to clang-10. #10638 (alexey-milovidov).
  • ✂ Remove external call to ld (bfd) linker during tzdata processing in compile time. #10634 (alesapin).
  • 👍 Allow to use lld to link blobs (resources). #10632 (alexey-milovidov).
  • 🛠 Fix UBSan report in LZ4 library. #10631 (alexey-milovidov). See also https://github.com/lz4/lz4/issues/857
  • ⚡️ Update LZ4 to the latest dev branch. #10630 (alexey-milovidov).
  • ➕ Added auto-generated machine-readable file with the list of stable versions. #10628 (alexey-milovidov).
  • 🛠 Fix capnproto version check for capnp::UnalignedFlatArrayMessageReader. #10618 (Matwey V. Kornilov).
  • ✅ Lower memory usage in tests. #10617 (alexey-milovidov).
  • 🛠 Fixing hard coded timeouts in new live view tests. #10604 (vzakaznikov).
  • ✅ Increasing timeout when opening a client in tests/queries/0_stateless/helpers/client.py. #10599 (vzakaznikov).
  • 🏗 Enable ThinLTO for clang builds, continuation of https://github.com/ClickHouse/ClickHouse/pull/10435. #10585 (Amos Bird).
  • ➕ Adding fuzzers and preparing for oss-fuzz integration. #10546 (kyprizel).
  • 🛠 Fix FreeBSD build. #10150 (Ivan).
  • ➕ Add new build for query tests using pytest framework. #10039 (Ivan).

• All Versions
• Previous v20.5.1.3833
• Next v20.5.4.40
• Latest Version