ClickHouse v21.3 Release Notes

Release Date: 2021-03-12 // 3 months ago
  • Backward Incompatible Change

    • πŸ”€ Now it's not allowed to create MergeTree tables in old syntax with table TTL because it's just ignored. Attach of old tables is still possible. #20282 (alesapin).
    • Now all case-insensitive function names will be rewritten to their canonical representations. This is needed for projection query routing (the upcoming feature). #20174 (Amos Bird).
    • πŸ›  Fix creation of TTL in cases, when its expression is a function and it is the same as ORDER BY key. Now it's allowed to set custom aggregation to primary key columns in TTL with GROUP BY. Backward incompatible: For primary key columns, which are not in GROUP BY and aren't set explicitly now is applied function any instead of max, when TTL is expired. Also if you use TTL with WHERE or GROUP BY you can see exceptions at merges, while making rolling update. #15450 (Anton Popov).

    πŸ†• New Feature

    • Add file engine settings: engine_file_empty_if_not_exists and engine_file_truncate_on_insert. #20620 (M0r64n).
    • βž• Add aggregate function deltaSum for summing the differences between consecutive rows. #20057 (Russ Frank).
    • New event_time_microseconds column in system.part_log table. #20027 (Bharat Nallan).
    • βž• Added timezoneOffset(datetime) function which will give the offset from UTC in seconds. This close #issue:19850. #19962 (keenwolf).
    • Add setting insert_shard_id to support insert data into specific shard from distributed table. #19961 (flynn).
    • ⚑️ Function reinterpretAs updated to support big integers. Fixes #19691. #19858 (Maksim Kita).
    • βž• Added Server Side Encryption Customer Keys (the x-amz-server-side-encryption-customer-(key/md5) header) support in S3 client. See the link. Closes #19428. #19748 (Vladimir Chebotarev).
    • βž• Added implicit_key option for executable dictionary source. It allows to avoid printing key for every record if records comes in the same order as the input keys. Implements #14527. #19677 (Maksim Kita).
    • Add quota type query_selects and query_inserts. #19603 (JackyWoo).
    • βž• Add function extractTextFromHTML #19600 (zlx19950903), (alexey-milovidov).
    • Tables with MergeTree* engine now have two new table-level settings for query concurrency control. Setting max_concurrent_queries limits the number of concurrently executed queries which are related to this table. Setting min_marks_to_honor_max_concurrent_queries tells to apply previous setting only if query reads at least this number of marks. #19544 (Amos Bird).
    • βž• Added file function to read file from user_files directory as a String. This is different from the file table function. This implements #issue:18851. #19204 (keenwolf).

    Experimental feature

    • βž• Add experimental Replicated database engine. It replicates DDL queries across multiple hosts. #16193 (tavplubix).
    • Introduce experimental support for window functions, enabled with allow_experimental_window_functions = 1. This is a preliminary, alpha-quality implementation that is not suitable for production use and will change in backward-incompatible ways in future releases. Please see the documentation for the list of supported features. #20337 (Alexander Kuzmenkov).
    • βž• Add the ability to backup/restore metadata files for DiskS3. #18377 (Pavel Kovalenko).

    🐎 Performance Improvement

    • Hedged requests for remote queries. When setting use_hedged_requests enabled (off by default), allow to establish many connections with different replicas for query. New connection is enabled in case existent connection(s) with replica(s) were not established within hedged_connection_timeout or no data was received within receive_data_timeout. Query uses the first connection which send non empty progress packet (or data packet, if allow_changing_replica_until_first_data_packet); other connections are cancelled. Queries with max_parallel_replicas > 1 are supported. #19291 (Kruglov Pavel). This allows to significantly reduce tail latencies on very large clusters.
    • βž• Added support for PREWHERE (and enable the corresponding optimization) when tables have row-level security expressions specified. #19576 (Denis Glazachev).
    • The setting distributed_aggregation_memory_efficient is enabled by default. It will lower memory usage and improve performance of distributed queries. #20599 (alexey-milovidov).
    • πŸ‘Œ Improve performance of GROUP BY multiple fixed size keys. #20472 (alexey-milovidov).
    • πŸ‘Œ Improve performance of aggregate functions by more strict aliasing. #19946 (alexey-milovidov).
    • πŸ”’ Speed up reading from Memory tables in extreme cases (when reading speed is in order of 50 GB/sec) by simplification of pipeline and (consequently) less lock contention in pipeline scheduling. #20468 (alexey-milovidov).
    • 🐎 Partially reimplement HTTP server to make it making less copies of incoming and outgoing data. It gives up to 1.5 performance improvement on inserting long records over HTTP. #19516 (Ivan).
    • βž• Add compress setting for Memory tables. If it's enabled the table will use less RAM. On some machines and datasets it can also work faster on SELECT, but it is not always the case. This closes #20093. Note: there are reasons why Memory tables can work slower than MergeTree: (1) lack of compression (2) static size of blocks (3) lack of indices and prewhere... #20168 (alexey-milovidov).
    • πŸ‘ Slightly better code in aggregation. #20978 (alexey-milovidov).
    • βž• Add back intDiv/modulo specializations for better performance. This fixes #21293 . The regression was introduced in https://github.com/ClickHouse/ClickHouse/pull/18145 . #21307 (Amos Bird).
    • Do not squash blocks too much on INSERT SELECT if inserting into Memory table. In previous versions inefficient data representation was created in Memory table after INSERT SELECT. This closes #13052. #20169 (alexey-milovidov).
    • πŸ›  Fix at least one case when DataType parser may have exponential complexity (found by fuzzer). This closes #20096. #20132 (alexey-milovidov).
    • Parallelize SELECT with FINAL for single part with level > 0 when do_not_merge_across_partitions_select_final setting is 1. #19375 (Kruglov Pavel).
    • Fill only requested columns when querying system.parts and system.parts_columns. Closes #19570. #21035 (Anmol Arora).
    • Perform algebraic optimizations of arithmetic expressions inside avg aggregate function. close #20092. #20183 (flynn).

    πŸ‘Œ Improvement

    • πŸ›  Case-insensitive compression methods for table functions. Also fixed LZMA compression method which was checked in upper case. #21416 (Vladimir Chebotarev).
    • βž• Add two settings to delay or throw error during insertion when there are too many inactive parts. This is useful when server fails to clean up parts quickly enough. #20178 (Amos Bird).
    • πŸ‘ Provide better compatibility for mysql clients. 1. mysql jdbc 2. mycli. #21367 (Amos Bird).
    • Forbid to drop a column if it's referenced by materialized view. Closes #21164. #21303 (flynn).
    • MySQL dictionary source will now retry unexpected connection failures (Lost connection to MySQL server during query) which sometimes happen on SSL/TLS connections. #21237 (Alexander Kazakov).
    • πŸ“œ Usability improvement: more consistent DateTime64 parsing: recognize the case when unix timestamp with subsecond resolution is specified as scaled integer (like 1111111111222 instead of 1111111111.222). This closes #13194. #21053 (alexey-milovidov).
    • Do only merging of sorted blocks on initiator with distributed_group_by_no_merge. #20882 (Azat Khuzhin).
    • When loading config for mysql source ClickHouse will now randomize the list of replicas with the same priority to ensure the round-robin logics of picking mysql endpoint. This closes #20629. #20632 (Alexander Kazakov).
    • Function 'reinterpretAs(x, Type)' renamed into 'reinterpret(x, Type)'. #20611 (Maksim Kita).
    • πŸ‘Œ Support vhost for RabbitMQ engine #20576. #20596 (Kseniia Sumarokova).
    • πŸ‘Œ Improved serialization for data types combined of Arrays and Tuples. Improved matching enum data types to protobuf enum type. Fixed serialization of the Map data type. Omitted values are now set by default. #20506 (Vitaly Baranov).
    • πŸ›  Fixed race between execution of distributed DDL tasks and cleanup of DDL queue. Now DDL task cannot be removed from ZooKeeper if there are active workers. Fixes #20016. #20448 (tavplubix).
    • πŸ‘‰ Make FQDN and other DNS related functions work correctly in alpine images. #20336 (filimonov).
    • Do not allow early constant folding of explicitly forbidden functions. #20303 (Azat Khuzhin).
    • Implicit conversion from integer to Decimal type might succeeded if integer value doe not fit into Decimal type. Now it throws ARGUMENT_OUT_OF_BOUND. #20232 (tavplubix).
    • Lockless SYSTEM FLUSH DISTRIBUTED. #20215 (Azat Khuzhin).
    • Normalize count(constant), sum(1) to count(). This is needed for projection query routing. #20175 (Amos Bird).
    • πŸ‘Œ Support all native integer types in bitmap functions. #20171 (Amos Bird).
    • ⚑️ Updated CacheDictionary, ComplexCacheDictionary, SSDCacheDictionary, SSDComplexKeyDictionary to use LRUHashMap as underlying index. #20164 (Maksim Kita).
    • πŸ”§ The setting access_management is now configurable on startup by providing CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT, defaults to disabled (0) which was the prior value. #20139 (Marquitos).
    • πŸ›  Fix toDateTime64(toDate()/toDateTime()) for DateTime64 - Implement DateTime64 clamping to match DateTime behaviour. #20131 (Azat Khuzhin).
    • Quota improvements: SHOW TABLES is now considered as one query in the quota calculations, not two queries. SYSTEM queries now consume quota. Fix calculation of interval's end in quota consumption. #20106 (Vitaly Baranov).
    • πŸ‘Œ Supports path IN (set) expressions for system.zookeeper table. #20105 (小路).
    • πŸ‘‰ Show full details of MaterializeMySQL tables in system.tables. #20051 (Stig Bakken).
    • πŸ›  Fix data race in executable dictionary that was possible only on misuse (when the script returns data ignoring its input). #20045 (alexey-milovidov).
    • The value of MYSQL_OPT_RECONNECT option can now be controlled by "opt_reconnect" parameter in the config section of mysql replica. #19998 (Alexander Kazakov).
    • If user calls JSONExtract function with Float32 type requested, allow inaccurate conversion to the result type. For example the number 0.1 in JSON is double precision and is not representable in Float32, but the user still wants to get it. Previous versions return 0 for non-Nullable type and NULL for Nullable type to indicate that conversion is imprecise. The logic was 100% correct but it was surprising to users and leading to questions. This closes #13962. #19960 (alexey-milovidov).
    • βž• Add conversion of block structure for INSERT into Distributed tables if it does not match. #19947 (Azat Khuzhin).
    • Improvement for the system.distributed_ddl_queue table. Initialize MaxDDLEntryID to the last value after restarting. Before this PR, MaxDDLEntryID will remain zero until a new DDLTask is processed. #19924 (Amos Bird).
    • πŸ‘‰ Show MaterializeMySQL tables in system.parts. #19770 (Stig Bakken).
    • βž• Add separate config directive for Buffer profile. #19721 (Azat Khuzhin).
    • 🚚 Move conditions that are not related to JOIN to WHERE clause. #18720. #19685 (hexiaoting).
    • Add ability to throttle INSERT into Distributed based on amount of pending bytes for async send (bytes_to_delay_insert/max_delay_to_insert and bytes_to_throw_insert settings for Distributed engine has been added). #19673 (Azat Khuzhin).
    • πŸ›  Fix some rare cases when write errors can be ignored in destructors. #19451 (Azat Khuzhin).
    • πŸ–¨ Print inline frames in stack traces for fatal errors. #19317 (Ivan).

    πŸ› Bug Fix

    • πŸ›  Fix redundant reconnects to ZooKeeper and the possibility of two active sessions for a single clickhouse server. Both problems introduced in #14678. #21264 (alesapin).
    • πŸ›  Fix error Bad cast from type ... to DB::ColumnLowCardinality while inserting into table with LowCardinality column from Values format. Fixes #21140 #21357 (Nikolai Kochetov).
    • πŸ›  Fix a deadlock in ALTER DELETE mutations for non replicated MergeTree table engines when the predicate contains the table itself. Fixes #20558. #21477 (alesapin).
    • πŸ›  Fix SIGSEGV for distributed queries on failures. #21434 (Azat Khuzhin).
    • πŸ›  Now ALTER MODIFY COLUMN queries will correctly affect changes in partition key, skip indices, TTLs, and so on. Fixes #13675. #21334 (alesapin).
    • πŸ‘‰ Fix bug with join_use_nulls and joining TOTALS from subqueries. This closes #19362 and #21137. #21248 (vdimir).
    • πŸ›  Fix crash in EXPLAIN for query with UNION. Fixes #20876, #21170. #21246 (flynn).
    • πŸ”€ Now mutations allowed only for table engines that support them (MergeTree family, Memory, MaterializedView). Other engines will report a more clear error. Fixes #21168. #21183 (alesapin).
    • πŸ›  Fixes #21112. Fixed bug that could cause duplicates with insert query (if one of the callbacks came a little too late). #21138 (Kseniia Sumarokova).
    • Fix input_format_null_as_default take effective when types are nullable. This fixes #21116 . #21121 (Amos Bird).
    • πŸ›  fix bug related to cast Tuple to Map. Closes #21029. #21120 (hexiaoting).
    • πŸ›  Fix the metadata leak when the Replicated*MergeTree with custom (non default) ZooKeeper cluster is dropped. #21119 (fastio).
    • πŸ›  Fix type mismatch issue when using LowCardinality keys in joinGet. This fixes #21114. #21117 (Amos Bird).
    • fix default_replica_path and default_replica_name values are useless on Replicated(*)MergeTree engine when the engine needs specify other parameters. #21060 (mxzlxy).
    • Out of bound memory access was possible when formatting specifically crafted out of range value of type DateTime64. This closes #20494. This closes #20543. #21023 (alexey-milovidov).
    • Block parallel insertions into storage join. #21009 (vdimir).
    • πŸ›  Fixed behaviour, when ALTER MODIFY COLUMN created mutation, that will knowingly fail. #21007 (Anton Popov).
    • ⚑️ Closes #9969. Fixed Brotli http compression error, which reproduced for large data sizes, slightly complicated structure and with json output format. Update Brotli to the latest version to include the "fix rare access to uninitialized data in ring-buffer". #20991 (Kseniia Sumarokova).
    • πŸ›  Fix 'Empty task was returned from async task queue' on query cancellation. #20881 (Azat Khuzhin).
    • πŸ›  USE database; query did not work when using MySQL 5.7 client to connect to ClickHouse server, it's fixed. Fixes #18926. #20878 (tavplubix).
    • πŸ›  Fix usage of -Distinct combinator with -State combinator in aggregate functions. #20866 (Anton Popov).
    • πŸ›  Fix subquery with union distinct and limit clause. close #20597. #20610 (flynn).
    • πŸ›  Fixed inconsistent behavior of dictionary in case of queries where we look for absent keys in dictionary. #20578 (Nikita Mikhaylov).
    • πŸ›  Fix the number of threads for scalar subqueries and subqueries for index (after #19007 single thread was always used). Fixes #20457, #20512. #20550 (Nikolai Kochetov).
    • πŸ›  Fix crash which could happen if unknown packet was received from remove query (was introduced in #17868). #20547 (Azat Khuzhin).
    • βž• Add proper checks while parsing directory names for async INSERT (fixes SIGSEGV). #20498 (Azat Khuzhin).
    • πŸ›  Fix function transform does not work properly for floating point keys. Closes #20460. #20479 (flynn).
    • πŸ›  Fix infinite loop when propagating WITH aliases to subqueries. This fixes #20388. #20476 (Amos Bird).
    • πŸ›  Fix abnormal server termination when http client goes away. #20464 (Azat Khuzhin).
    • Fix LOGICAL_ERROR for join_use_nulls=1 when JOIN contains const from SELECT. #20461 (Azat Khuzhin).
    • πŸ›  Check if table function view is used in expression list and throw an error. This fixes #20342. #20350 (Amos Bird).
    • Avoid invalid dereference in RANGE_HASHED() dictionary. #20345 (Azat Khuzhin).
    • πŸ‘‰ Fix null dereference with join_use_nulls=1. #20344 (Azat Khuzhin).
    • πŸ›  Fix incorrect result of binary operations between two constant decimals of different scale. Fixes #20283. #20339 (Maksim Kita).
    • πŸ›  Fix too often retries of failed background tasks for ReplicatedMergeTree table engines family. This could lead to too verbose logging and increased CPU load. Fixes #20203. #20335 (alesapin).
    • Restrict to DROP or RENAME version column of *CollapsingMergeTree and ReplacingMergeTree table engines. #20300 (alesapin).
    • πŸ›  Fixed the behavior when in case of broken JSON we tried to read the whole file into memory which leads to exception from the allocator. Fixes #19719. #20286 (Nikita Mikhaylov).
    • πŸ›  Fix exception during vertical merge for MergeTree table engines family which don't allow to perform vertical merges. Fixes #20259. #20279 (alesapin).
    • πŸ›  Fix rare server crash on config reload during the shutdown. Fixes #19689. #20224 (alesapin).
    • πŸ›  Fix CTE when using in INSERT SELECT. This fixes #20187, fixes #20195. #20211 (Amos Bird).
    • πŸ›  Fixes #19314. #20156 (Ivan).
    • πŸ›  fix toMinute function to handle special timezone correctly. #20149 (keenwolf).
    • πŸ›  Fix server crash after query with if function with Tuple type of then/else branches result. Tuple type must contain Array or another complex type. Fixes #18356. #20133 (alesapin).
    • The MongoDB table engine now establishes connection only when it's going to read data. ATTACH TABLE won't try to connect anymore. #20110 (Vitaly Baranov).
    • πŸ›  Bugfix in StorageJoin. #20079 (vdimir).
    • πŸ›  Fix the case when calculating modulo of division of negative number by small divisor, the resulting data type was not large enough to accomodate the negative result. This closes #20052. #20067 (alexey-milovidov).
    • ⚑️ MaterializeMySQL: Fix replication for statements that update several tables. #20066 (HΓ₯vard KvΓ₯len).
    • 🐳 Prevent "Connection refused" in docker during initialization script execution. #20012 (filimonov).
    • EmbeddedRocksDB is an experimental storage. Fix the issue with lack of proper type checking. Simplified code. This closes #19967. #19972 (alexey-milovidov).
    • πŸ›  Fix a segfault in function fromModifiedJulianDay when the argument type is Nullable(T) for any integral types other than Int32. #19959 (PHO).
    • πŸ›  BloomFilter index crash fix. Fixes #19757. #19884 (Maksim Kita).
    • πŸ›  Deadlock was possible if system.text_log is enabled. This fixes #19874. #19875 (alexey-milovidov).
    • πŸ›  Fix starting the server with tables having default expressions containing dictGet(). Allow getting return type of dictGet() without loading dictionary. #19805 (Vitaly Baranov).
    • πŸ›  Fix clickhouse-client abort exception while executing only select. #19790 (taiyang-li).
    • πŸ›  Fix a bug that moving pieces to destination table may failed in case of launching multiple clickhouse-copiers. #19743 (madianjun).
    • πŸ›  Background thread which executes ON CLUSTER queries might hang waiting for dropped replicated table to do something. It's fixed. #19684 (yiguolei).

    πŸ— Build/Testing/Packaging Improvement

    • πŸ‘ Allow to build ClickHouse with AVX-2 enabled globally. It gives slight performance benefits on modern CPUs. Not recommended for production and will not be supported as official build for now. #20180 (alexey-milovidov).
    • πŸ›  Fix some of the issues found by Coverity. See #19964. #20010 (alexey-milovidov).
    • πŸ‘ Allow to start up with modified binary under gdb. In previous version if you set up breakpoint in gdb before start, server will refuse to start up due to failed integrity check. #21258 (alexey-milovidov).
    • βž• Add a test for different compression methods in Kafka. #21111 (filimonov).
    • Fixed port clash from test_storage_kerberized_hdfs test. #19974 (Ilya Yatsishin).
    • 🐳 Print stdout and stderr to log when failed to start docker in integration tests. Before this PR there was a very short error message in this case which didn't help to investigate the problems. #20631 (Vitaly Baranov).

Previous changes from v21.2.2.8

  • Backward Incompatible Change

    • Bitwise functions (bitAnd, bitOr, etc) are forbidden for floating point arguments. Now you have to do explicit cast to integer. #19853 (Azat Khuzhin).
    • Forbid lcm/gcd for floats. #19532 (Azat Khuzhin).
    • πŸ›  Fix memory tracking for OPTIMIZE TABLE/merges; account query memory limits and sampling for OPTIMIZE TABLE/merges. #18772 (Azat Khuzhin).
    • πŸ‘€ Disallow floating point column as partition key, see #18421. #18464 (hexiaoting).
    • πŸ‘ Excessive parenthesis in type definitions no longer supported, example: Array((UInt8)).

    πŸ†• New Feature

    • βž• Added PostgreSQL table engine (both select/insert, with support for multidimensional arrays), also as table function. Added PostgreSQL dictionary source. Added PostgreSQL database engine. #18554 (Kseniia Sumarokova).
    • πŸ‘ Data type Nested now supports arbitrary levels of nesting. Introduced subcolumns of complex types, such as size0 in Array, null in Nullable, names of Tuple elements, which can be read without reading of whole column. #17310 (Anton Popov).
    • βž• Added Nullable support for FlatDictionary, HashedDictionary, ComplexKeyHashedDictionary, DirectDictionary, ComplexKeyDirectDictionary, RangeHashedDictionary. #18236 (Maksim Kita).
    • Adds a new table called system.distributed_ddl_queue that displays the queries in the DDL worker queue. #17656 (Bharat Nallan).
    • βž• Added support of mapping LDAP group names, and attribute values in general, to local roles for users from ldap user directories. #17211 (Denis Glazachev).
    • πŸ‘Œ Support insert into table function cluster, and for both table functions remote and cluster, support distributing data across nodes by specify sharding key. Close #16752. #18264 (flynn).
    • βž• Add function decodeXMLComponent to decode characters for XML. Example: SELECT decodeXMLComponent('Hello,"world"!') #17659. #18542 (nauta).
    • βž• Added functions parseDateTimeBestEffortUSOrZero, parseDateTimeBestEffortUSOrNull. #19712 (Maksim Kita).
    • βž• Add sign math function. #19527 (flynn).
    • βž• Add information about used features (functions, table engines, etc) into system.query_log. #18495. #19371 (Kseniia Sumarokova).
    • πŸ‘ Function formatDateTime support the %Q modification to format date to quarter. #19224 (Jianmei Zhang).
    • πŸ‘Œ Support MetaKey+Enter hotkey binding in play UI. #19012 (sundyli).
    • βž• Add three functions for map data type: 1. mapContains(map, key) to check weather map.keys include the second parameter key. 2. mapKeys(map) return all the keys in Array format 3. mapValues(map) return all the values in Array format. #18788 (hexiaoting).
    • βž• Add log_comment setting related to #18494. #18549 (Zijie Lu).
    • βž• Add support of tuple argument to argMin and argMax functions. #17359 (Ildus Kurbangaliev).
    • πŸ‘Œ Support EXISTS VIEW syntax. #18552 (Du Chuan).
    • βž• Add SELECT ALL syntax. closes #18706. #18723 (flynn).

    🐎 Performance Improvement

    • Faster parts removal by lowering the number of stat syscalls. This returns the optimization that existed while ago. More safe interface of IDisk. This closes #19065. #19086 (alexey-milovidov).
    • Aliases declared in WITH statement are properly used in index analysis. Queries like WITH column AS alias SELECT ... WHERE alias = ... may use index now. #18896 (Amos Bird).
    • Add optimize_alias_column_prediction (on by default), that will: - Respect aliased columns in WHERE during partition pruning and skipping data using secondary indexes; - Respect aliased columns in WHERE for trivial count queries for optimize_trivial_count; - Respect aliased columns in GROUP BY/ORDER BY for optimize_aggregation_in_order/optimize_read_in_order. #16995 (sundyli).
    • Speed up aggregate function sum. Improvement only visible on synthetic benchmarks and not very practical. #19216 (alexey-milovidov).
    • 🐎 Update libc++ and use another ABI to provide better performance. #18914 (Danila Kutenin).
    • Rewrite sumIf() and sum(if()) function to countIf() function when logically equivalent. #17041 (flynn).
    • Use a connection pool for S3 connections, controlled by the s3_max_connections settings. #13405 (Vladimir Chebotarev).
    • βž• Add support for zstd long option for better compression of string columns to save space. #17184 (ygrek).
    • 🚀 Slightly improve server latency by removing access to configuration on every connection. #19863 (alexey-milovidov).
    • πŸ”’ Reduce lock contention for multiple layers of the Buffer engine. #19379 (Azat Khuzhin).
    • πŸ‘Œ Support splitting Filter step of query plan into Expression + Filter pair. Together with Expression + Expression merging optimization (#17458) it may delay execution for some expressions after Filter step. #19253 (Nikolai Kochetov).

    πŸ‘Œ Improvement

    • πŸ›  SELECT count() FROM table now can be executed if only one any column can be selected from the table. This PR fixes #10639. #18233 (Vitaly Baranov).
    • πŸ›  Set charset to utf8mb4 when interacting with remote MySQL servers. Fixes #19795. #19800 (alexey-milovidov).
    • πŸ‘ S3 table function now supports auto compression mode (autodetect). This closes #18754. #19793 (Vladimir Chebotarev).
    • Correctly output infinite arguments for formatReadableTimeDelta function. In previous versions, there was implicit conversion to implementation specific integer value. #19791 (alexey-milovidov).
    • Table function S3 will use global region if the region can't be determined exactly. This closes #10998. #19750 (Vladimir Chebotarev).
    • In distributed queries if the setting async_socket_for_remote is enabled, it was possible to get stack overflow at least in debug build configuration if very deeply nested data type is used in table (e.g. Array(Array(Array(...more...)))). This fixes #19108. This change introduces minor backward incompatibility: excessive parenthesis in type definitions no longer supported, example: Array((UInt8)). #19736 (alexey-milovidov).
    • βž• Add separate pool for message brokers (RabbitMQ and Kafka). #19722 (Azat Khuzhin).
    • Fix rare max_number_of_merges_with_ttl_in_pool limit overrun (more merges with TTL can be assigned) for non-replicated MergeTree. #19708 (alesapin).
    • πŸ“œ Dictionary: better error message during attribute parsing. #19678 (Maksim Kita).
    • βž• Add an option to disable validation of checksums on reading. Should never be used in production. Please do not expect any benefits in disabling it. It may only be used for experiments and benchmarks. The setting only applicable for tables of MergeTree family. Checksums are always validated for other table engines and when receiving data over network. In my observations there is no performance difference or it is less than 0.5%. #19588 (alexey-milovidov).
    • πŸ‘Œ Support constant result in function multiIf. #19533 (Maksim Kita).
    • Enable function length/empty/notEmpty for datatype Map, which returns keys number in Map. #19530 (taiyang-li).
    • βž• Add --reconnect option to clickhouse-benchmark. When this option is specified, it will reconnect before every request. This is needed for testing. #19872 (alexey-milovidov).
    • πŸ‘Œ Support using the new location of .debug file. This fixes #19348. #19520 (Amos Bird).
    • πŸ“œ toIPv6 function parses IPv4 addresses. #19518 (Bharat Nallan).
    • Add http_referer field to system.query_log, system.processes, etc. This closes #19389. #19390 (alexey-milovidov).
    • πŸ‘Œ Improve MySQL compatibility by making more functions case insensitive and adding aliases. #19387 (Daniil Kondratyev).
    • βž• Add metrics for MergeTree parts (Wide/Compact/InMemory) types. #19381 (Azat Khuzhin).
    • πŸ‘ Allow docker to be executed with arbitrary uid. #19374 (filimonov).
    • πŸ›  Fix wrong alignment of values of IPv4 data type in Pretty formats. They were aligned to the right, not to the left. This closes #19184. #19339 (alexey-milovidov).
    • Allow change max_server_memory_usage without restart. This closes #18154. #19186 (alexey-milovidov).
    • πŸ›  The exception when function bar is called with certain NaN argument may be slightly misleading in previous versions. This fixes #19088. #19107 (alexey-milovidov).
    • πŸ›  Explicitly set uid / gid of clickhouse user & group to the fixed values (101) in clickhouse-server images. #19096 (filimonov).
    • πŸ›  Fixed PeekableReadBuffer: Memory limit exceed error when inserting data with huge strings. Fixes #18690. #18979 (tavplubix).
    • 🐳 Docker image: several improvements for clickhouse-server entrypoint. #18954 (filimonov).
    • βž• Add normalizeQueryKeepNames and normalizedQueryHashKeepNames to normalize queries without masking long names with ?. This helps better analyze complex query logs. #18910 (Amos Bird).
    • Check per-block checksum of the distributed batch on the sender before sending (without reading the file twice, the checksums will be verified while reading), this will avoid stuck of the INSERT on the receiver (on truncated .bin file on the sender). Avoid reading .bin files twice for batched INSERT (it was required to calculate rows/bytes to take squashing into account, now this information included into the header, backward compatible is preserved). #18853 (Azat Khuzhin).
    • πŸ›  Fix issues with RIGHT and FULL JOIN of tables with aggregate function states. In previous versions exception about cloneResized method was thrown. #18818 (templarzq).
    • βž• Added prefix-based S3 endpoint settings. #18812 (Vladimir Chebotarev).
    • βž• Add [UInt8, UInt16, UInt32, UInt64] arguments types support for bitmapTransform, bitmapSubsetInRange, bitmapSubsetLimit, bitmapContains functions. This closes #18713. #18791 (sundyli).
    • Allow CTE (Common Table Expressions) to be further aliased. Propagate CSE (Common Subexpressions Elimination) to subqueries in the same level when enable_global_with_statement = 1. This fixes #17378 . This fixes https://github.com/ClickHouse/ClickHouse/pull/16575#issuecomment-753416235 . #18684 (Amos Bird).
    • ⚑️ Update librdkafka to v1.6.0-RC2. Fixes #18668. #18671 (filimonov).
    • πŸ›  In case of unexpected exceptions automatically restart background thread which is responsible for execution of distributed DDL queries. Fixes #17991. #18285 (εΎη‚˜).
    • ⚑️ Updated AWS C++ SDK in order to utilize global regions in S3. #17870 (Vladimir Chebotarev).
    • Added support for WITH ... [AND] [PERIODIC] REFRESH [interval_in_sec] clause when creating LIVE VIEW tables. #14822 (vzakaznikov).
    • πŸ”€ Restrict MODIFY TTL queries for MergeTree tables created in old syntax. Previously the query succeeded, but actually it had no effect. #19064 (Anton Popov).

    πŸ› Bug Fix

    • πŸ›  Fix index analysis of binary functions with constant argument which leads to wrong query results. This fixes #18364. #18373 (Amos Bird).
    • πŸ›  Fix starting the server with tables having default expressions containing dictGet(). Allow getting return type of dictGet() without loading dictionary. #19805 (Vitaly Baranov).
    • πŸ›  Fix server crash after query with if function with Tuple type of then/else branches result. Tuple type must contain Array or another complex type. Fixes #18356. #20133 (alesapin).
    • ⚑️ MaterializeMySQL (experimental feature): Fix replication for statements that update several tables. #20066 (HΓ₯vard KvΓ₯len).
    • 🐳 Prevent "Connection refused" in docker during initialization script execution. #20012 (filimonov).
    • EmbeddedRocksDB is an experimental storage. Fix the issue with lack of proper type checking. Simplified code. This closes #19967. #19972 (alexey-milovidov).
    • πŸ›  Fix a segfault in function fromModifiedJulianDay when the argument type is Nullable(T) for any integral types other than Int32. #19959 (PHO).
    • The function greatCircleAngle returned inaccurate results in previous versions. This closes #19769. #19789 (alexey-milovidov).
    • πŸ›  Fix rare bug when some replicated operations (like mutation) cannot process some parts after data corruption. Fixes #19593. #19702 (alesapin).
    • πŸ›  Background thread which executes ON CLUSTER queries might hang waiting for dropped replicated table to do something. It's fixed. #19684 (yiguolei).
    • πŸ›  Fix wrong deserialization of columns description. It makes INSERT into a table with a column named \ impossible. #19479 (alexey-milovidov).
    • Mark distributed batch as broken in case of empty data block in one of files. #19449 (Azat Khuzhin).
    • πŸ›  Fixed very rare bug that might cause mutation to hang after DROP/DETACH/REPLACE/MOVE PARTITION. It was partially fixed by #15537 for the most cases. #19443 (tavplubix).
    • πŸ›  Fix possible error Extremes transform was already added to pipeline. Fixes #14100. #19430 (Nikolai Kochetov).
    • πŸ›  Fix default value in join types with non-zero default (e.g. some Enums). Closes #18197. #19360 (vdimir).
    • Do not mark file for distributed send as broken on EOF. #19290 (Azat Khuzhin).
    • Fix leaking of pipe fd for async_socket_for_remote. #19153 (Azat Khuzhin).
    • πŸ›  Fix infinite reading from file in ORC format (was introduced in #10580). Fixes #19095. #19134 (Nikolai Kochetov).
    • πŸ›  Fix issue in merge tree data writer which can lead to marks with bigger size than fixed granularity size. Fixes #18913. #19123 (alesapin).
    • πŸ›  Fix startup bug when clickhouse was not able to read compression codec from LowCardinality(Nullable(...)) and throws exception Attempt to read after EOF. Fixes #18340. #19101 (alesapin).
    • πŸ›  Simplify the implementation of tupleHammingDistance. Support for tuples of any equal length. Fixes #19029. #19084 (Nikolai Kochetov).
    • πŸ‘‰ Make sure groupUniqArray returns correct type for argument of Enum type. This closes #17875. #19019 (alexey-milovidov).
    • πŸ›  Fix possible error Expected single dictionary argument for function if use function ignore with LowCardinality argument. Fixes #14275. #19016 (Nikolai Kochetov).
    • πŸ›  Fix inserting of LowCardinality column to table with TinyLog engine. Fixes #18629. #19010 (Nikolai Kochetov).
    • πŸ›  Fix minor issue in JOIN: Join tries to materialize const columns, but our code waits for them in other places. #18982 (Nikita Mikhaylov).
    • 🚚 Disable optimize_move_functions_out_of_any because optimization is not always correct. This closes #18051. This closes #18973. #18981 (alexey-milovidov).
    • πŸ›  Fix possible exception QueryPipeline stream: different number of columns caused by merging of query plan's Expression steps. Fixes #18190. #18980 (Nikolai Kochetov).
    • πŸ›  Fixed very rare deadlock at shutdown. #18977 (tavplubix).
    • πŸ›  Fixed rare crashes when server run out of memory. #18976 (tavplubix).
    • πŸ›  Fix incorrect behavior when ALTER TABLE ... DROP PART 'part_name' query removes all deduplication blocks for the whole partition. Fixes #18874. #18969 (alesapin).
    • πŸ›  Fixed issue #18894 Add a check to avoid exception when long column alias('table.column' style, usually auto-generated by BI tools like Looker) equals to long table name. #18968 (Daniel Qin).
    • Fix error Task was not found in task queue (possible only for remote queries, with async_socket_for_remote = 1). #18964 (Nikolai Kochetov).
    • πŸ›  Fix bug when mutation with some escaped text (like ALTER ... UPDATE e = CAST('foo', 'Enum8(\'foo\' = 1') serialized incorrectly. Fixes #18878. #18944 (alesapin).
    • ATTACH PARTITION will reset mutations. #18804. #18935 (fastio).
    • πŸ›  Fix issue with bitmapOrCardinality that may lead to nullptr dereference. This closes #18911. #18912 (sundyli).
    • πŸ›  Fixed Attempt to read after eof error when trying to CAST NULL from Nullable(String) to Nullable(Decimal(P, S)). Now function CAST returns NULL when it cannot parse decimal from nullable string. Fixes #7690. #18718 (Winter Zhang).
    • πŸ›  Fix data type convert issue for MySQL engine. #18124 (bo zeng).
    • πŸ›  Fix clickhouse-client abort exception while executing only select. #19790 (taiyang-li).

    πŸ— Build/Testing/Packaging Improvement