« Changelog History


ClickHouse v21.8 Release Notes

Release Date: 2021-08-12 // over 2 years ago

  • ⬆️ Upgrade Notes

    • 🌲 New version is using Map data type for system logs tables (system.query_log, system.query_thread_log, system.processes, system.opentelemetry_span_log). These tables will be auto-created with new data types. Virtual columns are created to support old queries. Closes #18698. #23934, #25773 (hexiaoting, sundy-li, Maksim Kita). If you want to downgrade from version 21.8 to older versions, you will need to cleanup system tables with logs manually. Look at /var/lib/clickhouse/data/system/*_log.

    🆕 New Features

    • ➕ Add support for a part of SQL/JSON standard. #24148 (l1tsolaiki, Kseniia Sumarokova).
    • Collect common system metrics (in system.asynchronous_metrics and system.asynchronous_metric_log) on CPU usage, disk usage, memory usage, IO, network, files, load average, CPU frequencies, thermal sensors, EDAC counters, system uptime; also added metrics about the scheduling jitter and the time spent collecting the metrics. It works similar to atop in ClickHouse and allows access to monitoring data even if you have no additional tools installed. Close #9430. #24416 (alexey-milovidov, Yegor Levankov).
    • ➕ Add MaterializedPostgreSQL table engine and database engine. This database engine allows replicating a whole database or any subset of database tables. #20470 (Kseniia Sumarokova).
    • ➕ Add new functions leftPad(), rightPad(), leftPadUTF8(), rightPadUTF8(). #26075 (Vitaly Baranov).
    • ➕ Add the FIRST keyword to the ADD INDEX command to be able to add the index at the beginning of the indices list. #25904 (xjewer).
    • Introduce system.data_skipping_indices table containing information about existing data skipping indices. Close #7659. #25693 (Dmitry Novik).
    • ➕ Add bin/unbin functions. #25609 (zhaoyu).
    • 👌 Support Map and UInt128, Int128, UInt256, Int256 types in mapAdd and mapSubtract functions. #25596 (Ildus Kurbangaliev).
    • 👌 Support DISTINCT ON (columns) expression, close #25404. #25589 (Zijie Lu).
    • ➕ Add an ability to reset a custom setting to default and remove it from the table's metadata. It allows rolling back the change without knowing the system/config's default. Closes #14449. #17769 (xjewer).
    • 💻 Render pipelines as graphs in Web UI if EXPLAIN PIPELINE graph = 1 query is submitted. #26067 (alexey-milovidov).

    🐎 Performance Improvements

    • Compile aggregate functions. Use option compile_aggregate_expressions to enable it. #24789 (Maksim Kita).
    • 👌 Improve latency of short queries that require reading from tables with many columns. #26371 (Anton Popov).

    👌 Improvements

    • 🌲 Use Map data type for system logs tables (system.query_log, system.query_thread_log, system.processes, system.opentelemetry_span_log). These tables will be auto-created with new data types. Virtual columns are created to support old queries. Closes #18698. #23934, #25773 (hexiaoting, sundy-li, Maksim Kita).
    • For a dictionary with a complex key containing only one attribute, allow not wrapping the key expression in tuple for functions dictGet, dictHas. #26130 (Maksim Kita).
    • Implement function bin/hex from AggregateFunction states. #26094 (zhaoyu).
    • 👌 Support arguments of UUID type for empty and notEmpty functions. UUID is empty if it is all zeros (nil UUID). Closes #3446. #25974 (zhaoyu).
    • Add support for SET SQL_SELECT_LIMIT in MySQL protocol. Closes #17115. #25972 (Kseniia Sumarokova).
    • 📚 More instrumentation for network interaction: add counters for recv/send bytes; add gauges for recvs/sends. Added missing documentation. Close #5897. #25962 (alexey-milovidov).
    • 🚚 Add setting optimize_move_to_prewhere_if_final. If query has FINAL, the optimization move_to_prewhere will be enabled only if both optimize_move_to_prewhere and optimize_move_to_prewhere_if_final are enabled. Closes #8684. #25940 (Kseniia Sumarokova).
    • 👍 Allow complex quoted identifiers of JOINed tables. Close #17861. #25924 (alexey-milovidov).
    • ➕ Add support for Unicode (e.g. Chinese, Cyrillic) components in Nested data types. Close #25594. #25923 (alexey-milovidov).
    • Allow quantiles* functions to work with aggregate_functions_null_for_empty. Close #25892. #25919 (alexey-milovidov).
    • 👍 Allow parameters for parametric aggregate functions to be arbitrary constant expressions (e.g., 1 + 2), not just literals. It also allows using the query parameters (in parameterized queries like {param:UInt8}) inside parametric aggregate functions. Closes #11607. #25910 (alexey-milovidov).
    • 📜 Correctly throw the exception on the attempt to parse an invalid Date. Closes #6481. #25909 (alexey-milovidov).
    • Support for multiple includes in configuration. It is possible to include users configuration, remote server configuration from multiple sources. Simply place <include /> element with from_zk, from_env or incl attribute, and it will be replaced with the substitution. #24404 (nvartolomei).
    • 👌 Support for queries with a column named "null" (it must be specified in back-ticks or double quotes) and ON CLUSTER. Closes #24035. #25907 (alexey-milovidov).
    • 👌 Support LowCardinality, Decimal, and UUID for JSONExtract. Closes #24606. #25900 (Kseniia Sumarokova).
    • Convert history file from readline format to replxx format. #25888 (Azat Khuzhin).
    • 🛠 Fix an issue which can lead to intersecting parts after DROP PART or background deletion of an empty part. #25884 (alesapin).
    • 👍 Better handling of lost parts for ReplicatedMergeTree tables. Fixes rare inconsistencies in ReplicationQueue. Fixes #10368. #25820 (alesapin).
    • 👍 Allow starting clickhouse-client with unreadable working directory. #25817 (ianton-ru).
    • 🛠 Fix "No available columns" error for Merge storage. #25801 (Azat Khuzhin).
    • 👍 MySQL Engine now supports the exchange of column comments between MySQL and ClickHouse. #25795 (Storozhuk Kostiantyn).
    • 🛠 Fix inconsistent behaviour of GROUP BY constant on empty set. Closes #6842. #25786 (Kseniia Sumarokova).
    • 🔀 Cancel already running merges in partition on DROP PARTITION and TRUNCATE for ReplicatedMergeTree. Resolves #17151. #25684 (tavplubix).
    • 👌 Support ENUM` data type for MaterializeMySQL. #25676 (Storozhuk Kostiantyn).
    • 👌 Support materialized and aliased columns in JOIN, close #13274. #25634 (Vladimir C).
    • 🛠 Fix possible logical race condition between ALTER TABLE ... DETACH and background merges. #25605 (Azat Khuzhin).
    • 👉 Make NetworkReceiveElapsedMicroseconds metric to correctly include the time spent waiting for data from the client to INSERT. Close #9958. #25602 (alexey-milovidov).
    • 👌 Support TRUNCATE TABLE for S3 and HDFS. Close #25530. #25550 (Kseniia Sumarokova).
    • 👌 Support for dynamic reloading of config to change number of threads in pool for background jobs execution (merges, mutations, fetches). #25548 (Nikita Mikhaylov).
    • 👍 Allow extracting of non-string element as string using JSONExtract. This is for #25414. #25452 (Amos Bird).
    • 👌 Support regular expression in Database argument for StorageMerge. Close #776. #25064 (flynn).
    • 💻 Web UI: if the value looks like a URL, automatically generate a link. #25965 (alexey-milovidov).
    • 👉 Make sudo service clickhouse-server start to work on systems with systemd like Centos 8. Close #14298. Close #17799. #25921 (alexey-milovidov).

    🐛 Bug Fixes

    • 🛠 Fix incorrect SET ROLE in some cases. #26707 (Vitaly Baranov).
    • 🛠 Fix potential nullptr dereference in window functions. Fix #25276. #26668 (Alexander Kuzmenkov).
    • 🛠 Fix incorrect function names of groupBitmapAnd/Or/Xor. Fix #26557 (Amos Bird).
    • 🛠 Fix crash in RabbitMQ shutdown in case RabbitMQ setup was not started. Closes #26504. #26529 (Kseniia Sumarokova).
    • 🛠 Fix issues with CREATE DICTIONARY query if dictionary name or database name was quoted. Closes #26491. #26508 (Maksim Kita).
    • 🛠 Fix broken name resolution after rewriting column aliases. Fix #26432. #26475 (Amos Bird).
    • 🔀 Fix infinite non-joined block stream in partial_merge_join close #26325. #26374 (Vladimir C).
    • 🛠 Fix possible crash when login as dropped user. Fix #26073. #26363 (Vitaly Baranov).
    • Fix optimize_distributed_group_by_sharding_key for multiple columns (leads to incorrect result w/ optimize_skip_unused_shards=1/allow_nondeterministic_optimize_skip_unused_shards=1 and multiple columns in sharding key expression). #26353 (Azat Khuzhin).
    • CAST from Date to DateTime (or DateTime64) was not using the timezone of the DateTime type. It can also affect the comparison between Date and DateTime. Inference of the common type for Date and DateTime also was not using the corresponding timezone. It affected the results of function if and array construction. Closes #24128. #24129 (Maksim Kita).
    • 🛠 Fixed rare bug in lost replica recovery that may cause replicas to diverge. #26321 (tavplubix).
    • 🛠 Fix zstd decompression in case there are escape sequences at the end of internal buffer. Closes #26013. #26314 (Kseniia Sumarokova).
    • 🛠 Fix logical error on join with totals, close #26017. #26250 (Vladimir C).
    • Remove excessive newline in thread_name column in system.stack_trace table. Fix #24124. #26210 (alexey-milovidov).
    • 🛠 Fix joinGet with LowCarinality columns, close #25993. #26118 (Vladimir C).
    • 🛠 Fix possible crash in pointInPolygon if the setting validate_polygons is turned off. #26113 (alexey-milovidov).
    • 🛠 Fix throwing exception when iterate over non-existing remote directory. #26087 (ianton-ru).
    • 🛠 Fix rare server crash because of abort in ZooKeeper client. Fixes #25813. #26079 (alesapin).
    • 🛠 Fix wrong thread count estimation for right subquery join in some cases. Close #24075. #26052 (Vladimir C).
    • 🛠 Fixed incorrect sequence_id in MySQL protocol packets that ClickHouse sends on exception during query execution. It might cause MySQL client to reset connection to ClickHouse server. Fixes #21184. #26051 (tavplubix).
    • 🛠 Fix possible mismatched header when using normal projection with PREWHERE. Fix #26020. #26038 (Amos Bird).
    • 🛠 Fix formatting of type Map with integer keys to JSON. #25982 (Anton Popov).
    • 🛠 Fix possible deadlock during query profiler stack unwinding. Fix #25968. #25970 (Maksim Kita).
    • 🛠 Fix crash on call dictGet() with bad arguments. #25913 (Vitaly Baranov).
    • 🛠 Fixed scram-sha-256 authentication for PostgreSQL engines. Closes #24516. #25906 (Kseniia Sumarokova).
    • 🛠 Fix extremely long backoff for background tasks when the background pool is full. Fixes #25836. #25893 (alesapin).
    • 🛠 Fix ARM exception handling with non default page size. Fixes #25512, #25044, #24901, #23183, #20221, #19703, #19028, #18391, #18121, #17994, #12483. #25854 (Maksim Kita).
    • Fix sharding_key from column w/o function for remote() (before select * from remote('127.1', system.one, dummy) leads to Unknown column: dummy, there are only columns . error). #25824 (Azat Khuzhin).
    • 🛠 Fixed Not found column ... and Missing column ... errors when selecting from MaterializeMySQL. Fixes #23708, #24830, #25794. #25822 (tavplubix).
    • Fix optimize_skip_unused_shards_rewrite_in for non-UInt64 types (may select incorrect shards eventually or throw Cannot infer type of an empty tuple or Function tuple requires at least one argument). #25798 (Azat Khuzhin).
    • 🛠 Fix rare bug with DROP PART query for ReplicatedMergeTree tables which can lead to error message Unexpected merged part intersecting drop range. #25783 (alesapin).
    • 🛠 Fix bug in TTL with GROUP BY expression which refuses to execute TTL after first execution in part. #25743 (alesapin).
    • 👍 Allow StorageMerge to access tables with aliases. Closes #6051. #25694 (Kseniia Sumarokova).
    • 🛠 Fix slow dict join in some cases, close #24209. #25618 (Vladimir C).
    • 🛠 Fix ALTER MODIFY COLUMN of columns, which participates in TTL expressions. #25554 (Anton Popov).
    • 🛠 Fix assertion in PREWHERE with non-UInt8 type, close #19589. #25484 (Vladimir C).
    • 🛠 Fix some fuzzed msan crash. Fixes #22517. #26428 (Nikolai Kochetov).
    • ⚡️ Update chown cmd check in clickhouse-server docker entrypoint. It fixes error 'cluster pod restart failed (or timeout)' on kubernetes. #26545 (Ky Li).