« Changelog History


ClickHouse v20.7.2.30 Release Notes

Release Date: 2020-08-31 // over 3 years ago

  • Backward Incompatible Change

    • Function modulo (operator %) with at least one floating point number as argument will calculate remainder of division directly on floating point numbers without converting both arguments to integers. It makes behaviour compatible with most of DBMS. This also applicable for Date and DateTime data types. Added alias mod. This closes #7323. #12585 (alexey-milovidov).
    • 🗄 Deprecate special printing of zero Date/DateTime values as 0000-00-00 and 0000-00-00 00:00:00. #12442 (alexey-milovidov).
    • 🛠 The function groupArrayMoving* was not working for distributed queries. It's result was calculated within incorrect data type (without promotion to the largest type). The function groupArrayMovingAvg was returning integer number that was inconsistent with the avg function. This fixes #12568. #12622 (alexey-milovidov).
    • ➕ Add sanity check for MergeTree settings. If the settings are incorrect, the server will refuse to start or to create a table, printing detailed explanation to the user. #13153 (alexey-milovidov).
    • Protect from the cases when user may set background_pool_size to value lower than number_of_free_entries_in_pool_to_execute_mutation or number_of_free_entries_in_pool_to_lower_max_size_of_merge. In these cases ALTERs won't work or the maximum size of merge will be too limited. It will throw exception explaining what to do. This closes #10897. #12728 (alexey-milovidov).
    • ⚡️ When upgrading from versions older than 20.5, if rolling update is performed and cluster contains both versions 20.5 or greater and less than 20.5, if ClickHouse nodes with old versions are restarted and old version has been started up in presence of newer versions, it may lead to Part ... intersects previous part errors. To prevent this error, first install newer clickhouse-server packages on all cluster nodes and then do restarts (so, when clickhouse-server is restarted, it will start up with the new version).

    🆕 New Feature

    • ⚡️ Polygon dictionary type that provides efficient "reverse geocoding" lookups - to find the region by coordinates in a dictionary of many polygons (world map). It is using carefully optimized algorithm with recursive grids to maintain low CPU and memory usage. #9278 (achulkov2).
    • ➕ Added support of LDAP authentication for preconfigured users ("Simple Bind" method). #11234 (Denis Glazachev).
    • Introduce setting alter_partition_verbose_result which outputs information about touched parts for some types of ALTER TABLE ... PARTITION ... queries (currently ATTACH and FREEZE). Closes #8076. #13017 (alesapin).
    • ➕ Add bayesAB function for bayesian-ab-testing. #12327 (achimbab).
    • ➕ Added system.crash_log table into which stack traces for fatal errors are collected. This table should be empty. #12316 (alexey-milovidov).
    • ➕ Added http headers X-ClickHouse-Database and X-ClickHouse-Format which may be used to set default database and output format. #12981 (hcz).
    • ➕ Add minMap and maxMap functions support to SimpleAggregateFunction. #12662 (Ildus Kurbangaliev).
    • Add setting allow_non_metadata_alters which restricts to execute ALTER queries which modify data on disk. Disabled be default. Closes #11547. #12635 (alesapin).
    • 👍 A function formatRow is added to support turning arbitrary expressions into a string via given format. It's useful for manipulating SQL outputs and is quite versatile combined with the columns function. #12574 (Amos Bird).
    • ➕ Add FROM_UNIXTIME function for compatibility with MySQL, related to 12149. #12484 (flynn).
    • Allow Nullable types as keys in MergeTree tables if allow_nullable_key table setting is enabled. Closes #5319. #12433 (Amos Bird).
    • Integration with COS. #12386 (fastio).
    • ➕ Add mapAdd and mapSubtract functions for adding/subtracting key-mapped values. #11735 (Ildus Kurbangaliev).

    🐛 Bug Fix

    • 🛠 Fix premature ON CLUSTER timeouts for queries that must be executed on a single replica. Fixes #6704, #7228, #13361, #11884. #13450 (alesapin).
    • 🛠 Fix crash in mark inclusion search introduced in #12277. #14225 (Amos Bird).
    • 🛠 Fix race condition in external dictionaries with cache layout which can lead server crash. #12566 (alesapin).
    • 🛠 Fix visible data clobbering by progress bar in client in interactive mode. This fixes #12562 and #13369 and #13584 and fixes #12964. #13691 (alexey-milovidov).
    • 🛠 Fixed incorrect sorting order for LowCardinality columns when ORDER BY multiple columns is used. This fixes #13958. #14223 (Nikita Mikhaylov).
    • Removed hardcoded timeout, which wrongly overruled query_wait_timeout_milliseconds setting for cache-dictionary. #14105 (Nikita Mikhaylov).
    • 🛠 Fixed wrong mount point in extra info for Poco::Exception: no space left on device. #14050 (tavplubix).
    • Fix wrong query optimization of select queries with DISTINCT keyword when subqueries also have DISTINCT in case optimize_duplicate_order_by_and_distinct setting is enabled. #13925 (Artem Zuikov).
    • 🛠 Fixed potential deadlock when renaming Distributed table. #13922 (tavplubix).
    • 🛠 Fix incorrect sorting for FixedString columns when ORDER BY multiple columns is used. Fixes #13182. #13887 (Nikolai Kochetov).
    • 🛠 Fix potentially lower precision of topK/topKWeighted aggregations (with non-default parameters). #13817 (Azat Khuzhin).
    • 🛠 Fix reading from MergeTree table with INDEX of type SET fails when compared against NULL. This fixes #13686. #13793 (Amos Bird).
    • 🛠 Fix step overflow in function range(). #13790 (Azat Khuzhin).
    • 🛠 Fixed Directory not empty error when concurrently executing DROP DATABASE and CREATE TABLE. #13756 (alexey-milovidov).
    • ➕ Add range check for h3KRing function. This fixes #13633. #13752 (alexey-milovidov).
    • 🛠 Fix race condition between DETACH and background merges. Parts may revive after detach. This is continuation of #8602 that did not fix the issue but introduced a test that started to fail in very rare cases, demonstrating the issue. #13746 (alexey-milovidov).
    • Fix logging Settings.Names/Values when log_queries_min_type greater than QUERY_START. #13737 (Azat Khuzhin).
    • 🛠 Fix incorrect message in clickhouse-server.init while checking user and group. #13711 (ylchou).
    • 🚚 Do not optimize any(arrayJoin()) to arrayJoin() under optimize_move_functions_out_of_any. #13681 (Azat Khuzhin).
    • 🛠 Fixed possible deadlock in concurrent ALTER ... REPLACE/MOVE PARTITION ... queries. #13626 (tavplubix).
    • 🛠 Fixed the behaviour when sometimes cache-dictionary returned default value instead of present value from source. #13624 (Nikita Mikhaylov).
    • 🛠 Fix secondary indices corruption in compact parts (compact parts is an experimental feature). #13538 (Anton Popov).
    • 🛠 Fix wrong code in function netloc. This fixes #13335. #13446 (alexey-milovidov).
    • 🛠 Fix error in parseDateTimeBestEffort function when unix timestamp was passed as an argument. This fixes #13362. #13441 (alexey-milovidov).
    • 🛠 Fix invalid return type for comparison of tuples with NULL elements. Fixes #12461. #13420 (Nikolai Kochetov).
    • 🚚 Fix wrong optimization caused aggregate function any(x) is found inside another aggregate function in query error with SET optimize_move_functions_out_of_any = 1 and aliases inside any(). #13419 (Artem Zuikov).
    • 🛠 Fix possible race in StorageMemory. #13416 (Nikolai Kochetov).
    • 🛠 Fix empty output for Arrow and Parquet formats in case if query return zero rows. It was done because empty output is not valid for this formats. #13399 (hcz).
    • 🛠 Fix select queries with constant columns and prefix of primary key in ORDER BY clause. #13396 (Anton Popov).
    • 🛠 Fix PrettyCompactMonoBlock for clickhouse-local. Fix extremes/totals with PrettyCompactMonoBlock. Fixes #7746. #13394 (Azat Khuzhin).
    • 🛠 Fixed deadlock in system.text_log. #12452 (alexey-milovidov). It is a part of #12339. This fixes #12325. #13386 (Nikita Mikhaylov).
    • 🛠 Fixed File(TSVWithNames*) (header was written multiple times), fixed clickhouse-local --format CSVWithNames* (lacks header, broken after #12197), fixed clickhouse-local --format CSVWithNames* with zero rows (lacks header). #13343 (Azat Khuzhin).
    • 🛠 Fix segfault when function groupArrayMovingSum deserializes empty state. Fixes #13339. #13341 (alesapin).
    • Throw error on arrayJoin() function in JOIN ON section. #13330 (Artem Zuikov).
    • 👉 Fix crash in LEFT ASOF JOIN with join_use_nulls=1. #13291 (Artem Zuikov).
    • 🛠 Fix possible error Totals having transform was already added to pipeline in case of a query from delayed replica. #13290 (Nikolai Kochetov).
    • 🛠 The server may crash if user passed specifically crafted arguments to the function h3ToChildren. This fixes #13275. #13277 (alexey-milovidov).
    • 🛠 Fix potentially low performance and slightly incorrect result for uniqExact, topK, sumDistinct and similar aggregate functions called on Float types with NaN values. It also triggered assert in debug build. This fixes #12491. #13254 (alexey-milovidov).
    • 🛠 Fix assertion in KeyCondition when primary key contains expression with monotonic function and query contains comparison with constant whose type is different. This fixes #12465. #13251 (alexey-milovidov).
    • Return passed number for numbers with MSB set in function roundUpToPowerOfTwoOrZero(). It prevents potential errors in case of overflow of array sizes. #13234 (Azat Khuzhin).
    • 🛠 Fix function if with nullable constexpr as cond that is not literal NULL. Fixes #12463. #13226 (alexey-milovidov).
    • 🛠 Fix assert in arrayElement function in case of array elements are Nullable and array subscript is also Nullable. This fixes #12172. #13224 (alexey-milovidov).
    • 🛠 Fix DateTime64 conversion functions with constant argument. #13205 (Azat Khuzhin).
    • 🛠 Fix parsing row policies from users.xml when names of databases or tables contain dots. This fixes #5779, #12527. #13199 (Vitaly Baranov).
    • 🛠 Fix access to redis dictionary after connection was dropped once. It may happen with cache and direct dictionary layouts. #13082 (Anton Popov).
    • 🛠 Fix wrong index analysis with functions. It could lead to some data parts being skipped when reading from MergeTree tables. Fixes #13060. Fixes #12406. #13081 (Anton Popov).
    • 🛠 Fix error Cannot convert column because it is constant but values of constants are different in source and result for remote queries which use deterministic functions in scope of query, but not deterministic between queries, like now(), now64(), randConstant(). Fixes #11327. #13075 (Nikolai Kochetov).
    • 🛠 Fix crash which was possible for queries with ORDER BY tuple and small LIMIT. Fixes #12623. #13009 (Nikolai Kochetov).
    • 🛠 Fix Block structure mismatch error for queries with UNION and JOIN. Fixes #12602. #12989 (Nikolai Kochetov).
    • Corrected merge_with_ttl_timeout logic which did not work well when expiration affected more than one partition over one time interval. (Authored by @excitoon). #12982 (Alexander Kazakov).
    • 🛠 Fix columns duplication for range hashed dictionary created from DDL query. This fixes #10605. #12857 (alesapin).
    • 🛠 Fix unnecessary limiting for the number of threads for selects from local replica. #12840 (Nikolai Kochetov).
    • 🛠 Fix rare bug when ALTER DELETE and ALTER MODIFY COLUMN queries executed simultaneously as a single mutation. Bug leads to an incorrect amount of rows in count.txt and as a consequence incorrect data in part. Also, fix a small bug with simultaneous ALTER RENAME COLUMN and ALTER ADD COLUMN. #12760 (alesapin).
    • Wrong credentials being used when using clickhouse dictionary source to query remote tables. #12756 (sundyli).
    • 🛠 Fix CAST(Nullable(String), Enum()). #12745 (Azat Khuzhin).
    • 🛠 Fix performance with large tuples, which are interpreted as functions in IN section. The case when user writes WHERE x IN tuple(1, 2, ...) instead of WHERE x IN (1, 2, ...) for some obscure reason. #12700 (Anton Popov).
    • Fix memory tracking for input_format_parallel_parsing (by attaching thread to group). #12672 (Azat Khuzhin).
    • 🚚 Fix wrong optimization optimize_move_functions_out_of_any=1 in case of any(func(<lambda>)). #12664 (Artem Zuikov).
    • 🛠 Fixed #10572 fix bloom filter index with const expression. #12659 (Winter Zhang).
    • 🛠 Fix SIGSEGV in StorageKafka when broker is unavailable (and not only). #12658 (Azat Khuzhin).
    • ➕ Add support for function if with Array(UUID) arguments. This fixes #11066. #12648 (alexey-milovidov).
    • 🛠 CREATE USER IF NOT EXISTS now doesn't throw exception if the user exists. This fixes #12507. #12646 (Vitaly Baranov).
    • ⚡️ Exception There is no supertype... can be thrown during ALTER ... UPDATE in unexpected cases (e.g. when subtracting from UInt64 column). This fixes #7306. This fixes #4165. #12633 (alexey-milovidov).
    • 🛠 Fix possible Pipeline stuck error for queries with external sorting. Fixes #12617. #12618 (Nikolai Kochetov).
    • 🛠 Fix error Output of TreeExecutor is not sorted for OPTIMIZE DEDUPLICATE. Fixes #11572. #12613 (Nikolai Kochetov).
    • 🛠 Fix the issue when alias on result of function any can be lost during query optimization. #12593 (Anton Popov).
    • ✂ Remove data for Distributed tables (blocks from async INSERTs) on DROP TABLE. #12556 (Azat Khuzhin).
    • Now ClickHouse will recalculate checksums for parts when file checksums.txt is absent. Broken since #9827. #12545 (alesapin).
    • Fix bug which lead to broken old parts after ALTER DELETE query when enable_mixed_granularity_parts=1. Fixes #12536. #12543 (alesapin).
    • 🛠 Fixing race condition in live view tables which could cause data duplication. LIVE VIEW is an experimental feature. #12519 (vzakaznikov).
    • 🛠 Fix backwards compatibility in binary format of AggregateFunction(avg, ...) values. This fixes #12342. #12486 (alexey-milovidov).
    • 🛠 Fix crash in JOIN with dictionary when we are joining over expression of dictionary key: t JOIN dict ON expr(dict.id) = t.id. Disable dictionary join optimisation for this case. #12458 (Artem Zuikov).
    • 🛠 Fix overflow when very large LIMIT or OFFSET is specified. This fixes #10470. This fixes #11372. #12427 (alexey-milovidov).
    • kafka: fix SIGSEGV if there is a message with error in the middle of the batch. #12302 (Azat Khuzhin).

    👌 Improvement

    • 🔊 Keep smaller amount of logs in ZooKeeper. Avoid excessive growing of ZooKeeper nodes in case of offline replicas when having many servers/tables/inserts. #13100 (alexey-milovidov).
    • Now exceptions forwarded to the client if an error happened during ALTER or mutation. Closes #11329. #12666 (alesapin).
    • ➕ Add QueryTimeMicroseconds, SelectQueryTimeMicroseconds and InsertQueryTimeMicroseconds to system.events, along with system.metrics, processes, query_log, etc. #13028 (ianton-ru).
    • ➕ Added SelectedRows and SelectedBytes to system.events, along with system.metrics, processes, query_log, etc. #12638 (ianton-ru).
    • Added current_database information to system.query_log. #12652 (Amos Bird).
    • 👍 Allow TabSeparatedRaw as input format. #12009 (hcz).
    • 👍 Now joinGet supports multi-key lookup. #12418 (Amos Bird).
    • 👍 Allow *Map aggregate functions to work on Arrays with NULLs. Fixes #13157. #13225 (alexey-milovidov).
    • 📜 Avoid overflow in parsing of DateTime values that will lead to negative unix timestamp in their timezone (for example, 1970-01-01 00:00:00 in Moscow). Saturate to zero instead. This fixes #3470. This fixes #4172. #12443 (alexey-milovidov).
    • 👍 AvroConfluent: Skip Kafka tombstone records - Support skipping broken records #13203 (Andrew Onyshchuk).
    • 🛠 Fix wrong error for long queries. It was possible to get syntax error other than Max query size exceeded for correct query. #13928 (Nikolai Kochetov).
    • 🛠 Fix data race in lgamma function. This race was caught only in tsan, no side effects really happened. #13842 (Nikolai Kochetov).
    • 🛠 Fix a 'Week'-interval formatting for ATTACH/ALTER/CREATE QUOTA-statements. #13417 (vladimir-golovchenko).
    • Now broken parts are also reported when encountered in compact part processing. Compact parts is an experimental feature. #13282 (Amos Bird).
    • 🛠 Fix assert in geohashesInBox. This fixes #12554. #13229 (alexey-milovidov).
    • 🛠 Fix assert in parseDateTimeBestEffort. This fixes #12649. #13227 (alexey-milovidov).
    • Minor optimization in Processors/PipelineExecutor: breaking out of a loop because it makes sense to do so. #13058 (Mark Papadakis).
    • 👌 Support TRUNCATE table without TABLE keyword. #12653 (Winter Zhang).
    • 🛠 Fix explain query format overwrite by default. This fixes #12541. #12541 (BohuTANG).
    • 👍 Allow to set JOIN kind and type in more standad way: LEFT SEMI JOIN instead of SEMI LEFT JOIN. For now both are correct. #12520 (Artem Zuikov).
    • Changes default value for multiple_joins_rewriter_version to 2. It enables new multiple joins rewriter that knows about column names. #12469 (Artem Zuikov).
    • ➕ Add several metrics for requests to S3 storages. #12464 (ianton-ru).
    • 0️⃣ Use correct default secure port for clickhouse-benchmark with --secure argument. This fixes #11044. #12440 (alexey-milovidov).
    • ⏪ Rollback insertion errors in Log, TinyLog, StripeLog engines. In previous versions insertion error lead to inconsisent table state (this works as documented and it is normal for these table engines). This fixes #12402. #12426 (alexey-milovidov).
    • Implement RENAME DATABASE and RENAME DICTIONARY for Atomic database engine - Add implicit {uuid} macro, which can be used in ZooKeeper path for ReplicatedMergeTree. It works with CREATE ... ON CLUSTER ... queries. Set show_table_uuid_in_table_create_query_if_not_nil to true to use it. - Make ReplicatedMergeTree engine arguments optional, /clickhouse/tables/{uuid}/{shard}/ and {replica} are used by default. Closes #12135. - Minor fixes. - These changes break backward compatibility of Atomic database engine. Previously created Atomic databases must be manually converted to new format. Atomic database is an experimental feature. #12343 (tavplubix).
    • 🚚 Separated AWSAuthV4Signer into different logger, removed excessive AWSClient: AWSClient from log messages. #12320 (Vladimir Chebotarev).
    • 👍 Better exception message in disk access storage. #12625 (alesapin).
    • 👍 Better exception for function in with invalid number of arguments. #12529 (Anton Popov).
    • 🛠 Fix error message about adaptive granularity. #12624 (alesapin).
    • 🛠 Fix SETTINGS parse after FORMAT. #12480 (Azat Khuzhin).
    • 🔀 If MergeTree table does not contain ORDER BY or PARTITION BY, it was possible to request ALTER to CLEAR all the columns and ALTER will stuck. Fixed #7941. #12382 (alexey-milovidov).
    • Avoid re-loading completion from the history file after each query (to avoid history overlaps with other client sessions). #13086 (Azat Khuzhin).

    🐎 Performance Improvement

    • Lower memory usage for some operations up to 2 times. #12424 (alexey-milovidov).
    • ⚡️ Optimize PK lookup for queries that match exact PK range. #12277 (Ivan Babrou).
    • ⚡️ Slightly optimize very short queries with LowCardinality. #14129 (Anton Popov).
    • 🐎 Slightly improve performance of aggregation by UInt8/UInt16 keys. #13091 and #13055 (alexey-milovidov).
    • Push down LIMIT step for query plan (inside subqueries). #13016 (Nikolai Kochetov).
    • Parallel primary key lookup and skipping index stages on parts, as described in #11564. #12589 (Ivan Babrou).
    • Converting String-type arguments of function "if" and "transform" into enum if set optimize_if_transform_strings_to_enum = 1. #12515 (Artem Zuikov).
    • Replaces monotonic functions with its argument in ORDER BY if set optimize_monotonous_functions_in_order_by=1. #12467 (Artem Zuikov).
    • Add order by optimization that rewrites ORDER BY x, f(x) with ORDER by x if set optimize_redundant_functions_in_order_by = 1. #12404 (Artem Zuikov).
    • 👍 Allow pushdown predicate when subquery contains WITH clause. This fixes #12293 #12663 (Winter Zhang).
    • 👌 Improve performance of reading from compact parts. Compact parts is an experimental feature. #12492 (Anton Popov).
    • Attempt to implement streaming optimization in DiskS3. DiskS3 is an experimental feature. #12434 (Vladimir Chebotarev).

    🏗 Build/Testing/Packaging Improvement