ClickHouse changelog

Changelog History

Page 2

• v21.2.2.8 Changes

  February 07, 2021

  Backward Incompatible Change

  • Bitwise functions (bitAnd, bitOr, etc) are forbidden for floating point arguments. Now you have to do explicit cast to integer. #19853 (Azat Khuzhin).
  • Forbid lcm/gcd for floats. #19532 (Azat Khuzhin).
  • 🛠 Fix memory tracking for OPTIMIZE TABLE/merges; account query memory limits and sampling for OPTIMIZE TABLE/merges. #18772 (Azat Khuzhin).
  • 👀 Disallow floating point column as partition key, see #18421. #18464 (hexiaoting).
  • 👍 Excessive parenthesis in type definitions no longer supported, example: Array((UInt8)).

  🆕 New Feature

  • ➕ Added PostgreSQL table engine (both select/insert, with support for multidimensional arrays), also as table function. Added PostgreSQL dictionary source. Added PostgreSQL database engine. #18554 (Kseniia Sumarokova).
  • 👍 Data type Nested now supports arbitrary levels of nesting. Introduced subcolumns of complex types, such as size0 in Array, null in Nullable, names of Tuple elements, which can be read without reading of whole column. #17310 (Anton Popov).
  • ➕ Added Nullable support for FlatDictionary, HashedDictionary, ComplexKeyHashedDictionary, DirectDictionary, ComplexKeyDirectDictionary, RangeHashedDictionary. #18236 (Maksim Kita).
  • Adds a new table called system.distributed_ddl_queue that displays the queries in the DDL worker queue. #17656 (Bharat Nallan).
  • ➕ Added support of mapping LDAP group names, and attribute values in general, to local roles for users from ldap user directories. #17211 (Denis Glazachev).
  • 👌 Support insert into table function cluster, and for both table functions remote and cluster, support distributing data across nodes by specify sharding key. Close #16752. #18264 (flynn).
  • ➕ Add function decodeXMLComponent to decode characters for XML. Example: SELECT decodeXMLComponent('Hello,"world"!') #17659. #18542 (nauta).
  • ➕ Added functions parseDateTimeBestEffortUSOrZero, parseDateTimeBestEffortUSOrNull. #19712 (Maksim Kita).
  • ➕ Add sign math function. #19527 (flynn).
  • ➕ Add information about used features (functions, table engines, etc) into system.query_log. #18495. #19371 (Kseniia Sumarokova).
  • 👍 Function formatDateTime support the %Q modification to format date to quarter. #19224 (Jianmei Zhang).
  • 👌 Support MetaKey+Enter hotkey binding in play UI. #19012 (sundyli).
  • ➕ Add three functions for map data type: 1. mapContains(map, key) to check weather map.keys include the second parameter key. 2. mapKeys(map) return all the keys in Array format 3. mapValues(map) return all the values in Array format. #18788 (hexiaoting).
  • ➕ Add log_comment setting related to #18494. #18549 (Zijie Lu).
  • ➕ Add support of tuple argument to argMin and argMax functions. #17359 (Ildus Kurbangaliev).
  • 👌 Support EXISTS VIEW syntax. #18552 (Du Chuan).
  • ➕ Add SELECT ALL syntax. closes #18706. #18723 (flynn).

  🐎 Performance Improvement

  • Faster parts removal by lowering the number of stat syscalls. This returns the optimization that existed while ago. More safe interface of IDisk. This closes #19065. #19086 (alexey-milovidov).
  • Aliases declared in WITH statement are properly used in index analysis. Queries like WITH column AS alias SELECT ... WHERE alias = ... may use index now. #18896 (Amos Bird).
  • Add optimize_alias_column_prediction (on by default), that will: - Respect aliased columns in WHERE during partition pruning and skipping data using secondary indexes; - Respect aliased columns in WHERE for trivial count queries for optimize_trivial_count; - Respect aliased columns in GROUP BY/ORDER BY for optimize_aggregation_in_order/optimize_read_in_order. #16995 (sundyli).
  • Speed up aggregate function sum. Improvement only visible on synthetic benchmarks and not very practical. #19216 (alexey-milovidov).
  • 🐎 Update libc++ and use another ABI to provide better performance. #18914 (Danila Kutenin).
  • Rewrite sumIf() and sum(if()) function to countIf() function when logically equivalent. #17041 (flynn).
  • Use a connection pool for S3 connections, controlled by the s3_max_connections settings. #13405 (Vladimir Chebotarev).
  • ➕ Add support for zstd long option for better compression of string columns to save space. #17184 (ygrek).
  • 🚤 Slightly improve server latency by removing access to configuration on every connection. #19863 (alexey-milovidov).
  • 🔒 Reduce lock contention for multiple layers of the Buffer engine. #19379 (Azat Khuzhin).
  • 👌 Support splitting Filter step of query plan into Expression + Filter pair. Together with Expression + Expression merging optimization (#17458) it may delay execution for some expressions after Filter step. #19253 (Nikolai Kochetov).

  👌 Improvement

  • 🛠 SELECT count() FROM table now can be executed if only one any column can be selected from the table. This PR fixes #10639. #18233 (Vitaly Baranov).
  • 🛠 Set charset to utf8mb4 when interacting with remote MySQL servers. Fixes #19795. #19800 (alexey-milovidov).
  • 👍 S3 table function now supports auto compression mode (autodetect). This closes #18754. #19793 (Vladimir Chebotarev).
  • Correctly output infinite arguments for formatReadableTimeDelta function. In previous versions, there was implicit conversion to implementation specific integer value. #19791 (alexey-milovidov).
  • Table function S3 will use global region if the region can't be determined exactly. This closes #10998. #19750 (Vladimir Chebotarev).
  • In distributed queries if the setting async_socket_for_remote is enabled, it was possible to get stack overflow at least in debug build configuration if very deeply nested data type is used in table (e.g. Array(Array(Array(...more...)))). This fixes #19108. This change introduces minor backward incompatibility: excessive parenthesis in type definitions no longer supported, example: Array((UInt8)). #19736 (alexey-milovidov).
  • ➕ Add separate pool for message brokers (RabbitMQ and Kafka). #19722 (Azat Khuzhin).
  • Fix rare max_number_of_merges_with_ttl_in_pool limit overrun (more merges with TTL can be assigned) for non-replicated MergeTree. #19708 (alesapin).
  • 📜 Dictionary: better error message during attribute parsing. #19678 (Maksim Kita).
  • ➕ Add an option to disable validation of checksums on reading. Should never be used in production. Please do not expect any benefits in disabling it. It may only be used for experiments and benchmarks. The setting only applicable for tables of MergeTree family. Checksums are always validated for other table engines and when receiving data over network. In my observations there is no performance difference or it is less than 0.5%. #19588 (alexey-milovidov).
  • 👌 Support constant result in function multiIf. #19533 (Maksim Kita).
  • Enable function length/empty/notEmpty for datatype Map, which returns keys number in Map. #19530 (taiyang-li).
  • ➕ Add --reconnect option to clickhouse-benchmark. When this option is specified, it will reconnect before every request. This is needed for testing. #19872 (alexey-milovidov).
  • 👌 Support using the new location of .debug file. This fixes #19348. #19520 (Amos Bird).
  • 📜 toIPv6 function parses IPv4 addresses. #19518 (Bharat Nallan).
  • Add http_referer field to system.query_log, system.processes, etc. This closes #19389. #19390 (alexey-milovidov).
  • 👌 Improve MySQL compatibility by making more functions case insensitive and adding aliases. #19387 (Daniil Kondratyev).
  • ➕ Add metrics for MergeTree parts (Wide/Compact/InMemory) types. #19381 (Azat Khuzhin).
  • 👍 Allow docker to be executed with arbitrary uid. #19374 (filimonov).
  • 🛠 Fix wrong alignment of values of IPv4 data type in Pretty formats. They were aligned to the right, not to the left. This closes #19184. #19339 (alexey-milovidov).
  • Allow change max_server_memory_usage without restart. This closes #18154. #19186 (alexey-milovidov).
  • 🛠 The exception when function bar is called with certain NaN argument may be slightly misleading in previous versions. This fixes #19088. #19107 (alexey-milovidov).
  • 🛠 Explicitly set uid / gid of clickhouse user & group to the fixed values (101) in clickhouse-server images. #19096 (filimonov).
  • 🛠 Fixed PeekableReadBuffer: Memory limit exceed error when inserting data with huge strings. Fixes #18690. #18979 (tavplubix).
  • 🐳 Docker image: several improvements for clickhouse-server entrypoint. #18954 (filimonov).
  • ➕ Add normalizeQueryKeepNames and normalizedQueryHashKeepNames to normalize queries without masking long names with ?. This helps better analyze complex query logs. #18910 (Amos Bird).
  • Check per-block checksum of the distributed batch on the sender before sending (without reading the file twice, the checksums will be verified while reading), this will avoid stuck of the INSERT on the receiver (on truncated .bin file on the sender). Avoid reading .bin files twice for batched INSERT (it was required to calculate rows/bytes to take squashing into account, now this information included into the header, backward compatible is preserved). #18853 (Azat Khuzhin).
  • 🛠 Fix issues with RIGHT and FULL JOIN of tables with aggregate function states. In previous versions exception about cloneResized method was thrown. #18818 (templarzq).
  • ➕ Added prefix-based S3 endpoint settings. #18812 (Vladimir Chebotarev).
  • ➕ Add [UInt8, UInt16, UInt32, UInt64] arguments types support for bitmapTransform, bitmapSubsetInRange, bitmapSubsetLimit, bitmapContains functions. This closes #18713. #18791 (sundyli).
  • Allow CTE (Common Table Expressions) to be further aliased. Propagate CSE (Common Subexpressions Elimination) to subqueries in the same level when enable_global_with_statement = 1. This fixes #17378 . This fixes https://github.com/ClickHouse/ClickHouse/pull/16575#issuecomment-753416235 . #18684 (Amos Bird).
  • ⚡️ Update librdkafka to v1.6.0-RC2. Fixes #18668. #18671 (filimonov).
  • 🛠 In case of unexpected exceptions automatically restart background thread which is responsible for execution of distributed DDL queries. Fixes #17991. #18285 (徐炘).
  • ⚡️ Updated AWS C++ SDK in order to utilize global regions in S3. #17870 (Vladimir Chebotarev).
  • Added support for WITH ... [AND] [PERIODIC] REFRESH [interval_in_sec] clause when creating LIVE VIEW tables. #14822 (vzakaznikov).
  • 🔀 Restrict MODIFY TTL queries for MergeTree tables created in old syntax. Previously the query succeeded, but actually it had no effect. #19064 (Anton Popov).

  🐛 Bug Fix

  • 🛠 Fix index analysis of binary functions with constant argument which leads to wrong query results. This fixes #18364. #18373 (Amos Bird).
  • 🛠 Fix starting the server with tables having default expressions containing dictGet(). Allow getting return type of dictGet() without loading dictionary. #19805 (Vitaly Baranov).
  • 🛠 Fix server crash after query with if function with Tuple type of then/else branches result. Tuple type must contain Array or another complex type. Fixes #18356. #20133 (alesapin).
  • ⚡️ MaterializeMySQL (experimental feature): Fix replication for statements that update several tables. #20066 (Håvard Kvålen).
  • 🐳 Prevent "Connection refused" in docker during initialization script execution. #20012 (filimonov).
  • EmbeddedRocksDB is an experimental storage. Fix the issue with lack of proper type checking. Simplified code. This closes #19967. #19972 (alexey-milovidov).
  • 🛠 Fix a segfault in function fromModifiedJulianDay when the argument type is Nullable(T) for any integral types other than Int32. #19959 (PHO).
  • The function greatCircleAngle returned inaccurate results in previous versions. This closes #19769. #19789 (alexey-milovidov).
  • 🛠 Fix rare bug when some replicated operations (like mutation) cannot process some parts after data corruption. Fixes #19593. #19702 (alesapin).
  • 🛠 Background thread which executes ON CLUSTER queries might hang waiting for dropped replicated table to do something. It's fixed. #19684 (yiguolei).
  • 🛠 Fix wrong deserialization of columns description. It makes INSERT into a table with a column named \ impossible. #19479 (alexey-milovidov).
  • Mark distributed batch as broken in case of empty data block in one of files. #19449 (Azat Khuzhin).
  • 🛠 Fixed very rare bug that might cause mutation to hang after DROP/DETACH/REPLACE/MOVE PARTITION. It was partially fixed by #15537 for the most cases. #19443 (tavplubix).
  • 🛠 Fix possible error Extremes transform was already added to pipeline. Fixes #14100. #19430 (Nikolai Kochetov).
  • 🛠 Fix default value in join types with non-zero default (e.g. some Enums). Closes #18197. #19360 (vdimir).
  • Do not mark file for distributed send as broken on EOF. #19290 (Azat Khuzhin).
  • Fix leaking of pipe fd for async_socket_for_remote. #19153 (Azat Khuzhin).
  • 🛠 Fix infinite reading from file in ORC format (was introduced in #10580). Fixes #19095. #19134 (Nikolai Kochetov).
  • 🛠 Fix issue in merge tree data writer which can lead to marks with bigger size than fixed granularity size. Fixes #18913. #19123 (alesapin).
  • 🛠 Fix startup bug when clickhouse was not able to read compression codec from LowCardinality(Nullable(...)) and throws exception Attempt to read after EOF. Fixes #18340. #19101 (alesapin).
  • 🛠 Simplify the implementation of tupleHammingDistance. Support for tuples of any equal length. Fixes #19029. #19084 (Nikolai Kochetov).
  • 👉 Make sure groupUniqArray returns correct type for argument of Enum type. This closes #17875. #19019 (alexey-milovidov).
  • 🛠 Fix possible error Expected single dictionary argument for function if use function ignore with LowCardinality argument. Fixes #14275. #19016 (Nikolai Kochetov).
  • 🛠 Fix inserting of LowCardinality column to table with TinyLog engine. Fixes #18629. #19010 (Nikolai Kochetov).
  • 🛠 Fix minor issue in JOIN: Join tries to materialize const columns, but our code waits for them in other places. #18982 (Nikita Mikhaylov).
  • 🚚 Disable optimize_move_functions_out_of_any because optimization is not always correct. This closes #18051. This closes #18973. #18981 (alexey-milovidov).
  • 🛠 Fix possible exception QueryPipeline stream: different number of columns caused by merging of query plan's Expression steps. Fixes #18190. #18980 (Nikolai Kochetov).
  • 🛠 Fixed very rare deadlock at shutdown. #18977 (tavplubix).
  • 🛠 Fixed rare crashes when server run out of memory. #18976 (tavplubix).
  • 🛠 Fix incorrect behavior when ALTER TABLE ... DROP PART 'part_name' query removes all deduplication blocks for the whole partition. Fixes #18874. #18969 (alesapin).
  • 🛠 Fixed issue #18894 Add a check to avoid exception when long column alias('table.column' style, usually auto-generated by BI tools like Looker) equals to long table name. #18968 (Daniel Qin).
  • Fix error Task was not found in task queue (possible only for remote queries, with async_socket_for_remote = 1). #18964 (Nikolai Kochetov).
  • 🛠 Fix bug when mutation with some escaped text (like ALTER ... UPDATE e = CAST('foo', 'Enum8(\'foo\' = 1') serialized incorrectly. Fixes #18878. #18944 (alesapin).
  • ATTACH PARTITION will reset mutations. #18804. #18935 (fastio).
  • 🛠 Fix issue with bitmapOrCardinality that may lead to nullptr dereference. This closes #18911. #18912 (sundyli).
  • 🛠 Fixed Attempt to read after eof error when trying to CAST NULL from Nullable(String) to Nullable(Decimal(P, S)). Now function CAST returns NULL when it cannot parse decimal from nullable string. Fixes #7690. #18718 (Winter Zhang).
  • 🛠 Fix data type convert issue for MySQL engine. #18124 (bo zeng).
  • 🛠 Fix clickhouse-client abort exception while executing only select. #19790 (taiyang-li).

  🏗 Build/Testing/Packaging Improvement

  • ⚙ Run SQLancer (logical SQL fuzzer) in CI. #19006 (Ilya Yatsishin).
  • ✅ Query Fuzzer will fuzz newly added tests more extensively. This closes #18916. #19185 (alexey-milovidov).
  • ↔ Integrate with Big List of Naughty Strings for better fuzzing. #19480 (alexey-milovidov).
  • ➕ Add integration tests run with MSan. #18974 (alesapin).
  • 🛠 Fixed MemorySanitizer errors in cyrus-sasl and musl. #19821 (Ilya Yatsishin).
  • Insuffiient arguments check in positionCaseInsensitiveUTF8 function triggered address sanitizer. #19720 (alexey-milovidov).
  • ✂ Remove --project-directory for docker-compose in integration test. Fix logs formatting from docker container. #19706 (Ilya Yatsishin).
  • ✅ Made generation of macros.xml easier for integration tests. No more excessive logging from dicttoxml. dicttoxml project is not active for 5+ years. #19697 (Ilya Yatsishin).
  • Allow to explicitly enable or disable watchdog via environment variable CLICKHOUSE_WATCHDOG_ENABLE. By default it is enabled if server is not attached to terminal. #19522 (alexey-milovidov).
  • 👍 Allow building ClickHouse with Kafka support on arm64. #19369 (filimonov).
  • 👍 Allow building librdkafka without ssl. #19337 (filimonov).
  • 🏗 Restore Kafka input in FreeBSD builds. #18924 (Alexandre Snarskii).
  • 🛠 Fix potential nullptr dereference in table function VALUES. #19357 (alexey-milovidov).
  • 🛠 Avoid UBSan reports in arrayElement function, substring and arraySum. Fixes #19305. Fixes #19287. This closes #19336. #19347 (alexey-milovidov).

• v21.1.3.32 Changes

  February 03, 2021

  🐛 Bug Fix

  • 🛠 BloomFilter index crash fix. Fixes #19757. #19884 (Maksim Kita).
  • 🛠 Fix crash when pushing down predicates to union distinct subquery. This fixes #19855. #19861 (Amos Bird).
  • 🛠 Fix filtering by UInt8 greater than 127. #19799 (Anton Popov).
  • In previous versions, unusual arguments for function arrayEnumerateUniq may cause crash or infinite loop. This closes #19787. #19788 (alexey-milovidov).
  • 🛠 Fixed stack overflow when using accurate comparison of arithmetic type with string type. #19773 (tavplubix).
  • 🛠 Fix crash when nested column name was used in WHERE or PREWHERE. Fixes #19755. #19763 (Nikolai Kochetov).
  • 🛠 Fix a segmentation fault in bitmapAndnot function. Fixes #19668. #19713 (Maksim Kita).
  • Some functions with big integers may cause segfault. Big integers is experimental feature. This closes #19667. #19672 (alexey-milovidov).
  • 🛠 Fix wrong result of function neighbor for LowCardinality argument. Fixes #10333. #19617 (Nikolai Kochetov).
  • 🛠 Fix use-after-free of the CompressedWriteBuffer in Connection after disconnect. #19599 (Azat Khuzhin).
  • 🔀 DROP/DETACH TABLE table ON CLUSTER cluster SYNC query might hang, it's fixed. Fixes #19568. #19572 (tavplubix).
  • Query CREATE DICTIONARY id expression fix. #19571 (Maksim Kita).
  • Fix SIGSEGV with merge_tree_min_rows_for_concurrent_read/merge_tree_min_bytes_for_concurrent_read=0/UINT64_MAX. #19528 (Azat Khuzhin).
  • 🛠 Buffer overflow (on memory read) was possible if addMonth function was called with specifically crafted arguments. This fixes #19441. This fixes #19413. #19472 (alexey-milovidov).
  • Uninitialized memory read was possible in encrypt/decrypt functions if empty string was passed as IV. This closes #19391. #19397 (alexey-milovidov).
  • 🛠 Fix possible buffer overflow in Uber H3 library. See https://github.com/uber/h3/issues/392. This closes #19219. #19383 (alexey-milovidov).
  • Fix system.parts _state column (LOGICAL_ERROR when querying this column, due to incorrect order). #19346 (Azat Khuzhin).
  • 🛠 Fixed possible wrong result or segfault on aggregation when Materialized View and its target table have different structure. Fixes #18063. #19322 (tavplubix).
  • 🛠 Fix error Cannot convert column now64() because it is constant but values of constants are different in source and result. Continuation of #7156. #19316 (Nikolai Kochetov).
  • 🛠 Fix bug when concurrent ALTER and DROP queries may hang while processing ReplicatedMergeTree table. #19237 (alesapin).
  • 🛠 Fixed There is no checkpoint error when inserting data through http interface using Template or CustomSeparated format. Fixes #19021. #19072 (tavplubix).
  • Disable constant folding for subqueries on the analysis stage, when the result cannot be calculated. #18446 (Azat Khuzhin).
  • 🚚 Mutation might hang waiting for some non-existent part after MOVE or REPLACE PARTITION or, in rare cases, after DETACH or DROP PARTITION. It's fixed. #15537 (tavplubix).

• v21.1.2.15 Changes

  January 18, 2021

  Backward Incompatible Change

  • The setting input_format_null_as_default is enabled by default. #17525 (alexey-milovidov).
  • Check settings constraints for profile settings from config. Server will fail to start if users.xml contain settings that do not meet corresponding constraints. #18486 (tavplubix).
  • Restrict ALTER MODIFY SETTING from changing storage settings that affects data parts (write_final_mark and enable_mixed_granularity_parts). #18306 (Amos Bird).
  • Set insert_quorum_parallel to 1 by default. It is significantly more convenient to use than "sequential" quorum inserts. But if you rely to sequential consistency, you should set the setting back to zero. #17567 (alexey-milovidov).
  • ✂ Remove sumburConsistentHash function. This closes #18120. #18656 (alexey-milovidov).
  • ✂ Removed aggregate functions timeSeriesGroupSum, timeSeriesGroupRateSum because a friend of mine said they never worked. This fixes #16869. If you have luck using these functions, write a email to [email protected]. #17423 (alexey-milovidov).
  • Prohibit toUnixTimestamp(Date()) (before it just returns UInt16 representation of Date). #17376 (Azat Khuzhin).
  • 👍 Allow using extended integer types (Int128, Int256, UInt256) in avg and avgWeighted functions. Also allow using different types (integer, decimal, floating point) for value and for weight in avgWeighted function. This is a backward-incompatible change: now the avg and avgWeighted functions always return Float64 (as documented). Before this change the return type for Decimal arguments was also Decimal. #15419 (Mike).
  • Expression toUUID(N) no longer works. Replace with toUUID('00000000-0000-0000-0000-000000000000'). This change is motivated by non-obvious results of toUUID(N) where N is non zero.
  • 👀 SSL Certificates with incorrect "key usage" are rejected. In previous versions they are used to work. See #19262.
  • ⚡️ incl references to substitutions file (/etc/metrika.xml) were removed from the default config (<remote_servers>, <zookeeper>, <macros>, <compression>, <networks>). If you were using substitutions file and were relying on those implicit references, you should put them back manually and explicitly by adding corresponding sections with incl="..." attributes before the update. See #18740 (alexey-milovidov).

  🆕 New Feature

  • Implement gRPC protocol in ClickHouse. #15111 (Vitaly Baranov).
  • 👍 Allow to use multiple zookeeper clusters. #17070 (fastio).
  • Implemented REPLACE TABLE and CREATE OR REPLACE TABLE queries. #18521 (tavplubix).
  • 0️⃣ Implement UNION DISTINCT and treat the plain UNION clause as UNION DISTINCT by default. Add a setting union_default_mode that allows to treat it as UNION ALL or require explicit mode specification. #16338 (flynn).
  • ➕ Added function accurateCastOrNull. This closes #10290. Add type conversions in x IN (subquery) expressions. This closes #10266. #16724 (Maksim Kita).
  • 👍 IP Dictionary supports IPv4 / IPv6 types directly. #17571 (vdimir).
  • 👍 IP Dictionary supports key fetching. Resolves #18241. #18480 (vdimir).
  • 👍 Add *.zst compression/decompression support for data import and export. It enables using *.zst in file() function and Content-encoding: zstd in HTTP client. This closes #16791 . #17144 (Abi Palagashvili).
  • ➕ Added mannWitneyUTest, studentTTest and welchTTest aggregate functions. Refactored rankCorr a bit. #16883 (Nikita Mikhaylov).
  • ➕ Add functions countMatches/countMatchesCaseInsensitive. #17459 (Azat Khuzhin).
  • Implement countSubstrings()/countSubstringsCaseInsensitive()/countSubstringsCaseInsensitiveUTF8() (Count the number of substring occurrences). #17347 (Azat Khuzhin).
  • 🌲 Add information about used databases, tables and columns in system.query_log. Add query_kind and normalized_query_hash fields. #17726 (Amos Bird).
  • Add a setting optimize_on_insert. When enabled, do the same transformation for INSERTed block of data as if merge was done on this block (e.g. Replacing, Collapsing, Aggregating...). This setting is enabled by default. This can influence Materialized View and MaterializeMySQL behaviour (see detailed description). This closes #10683. #16954 (Kruglov Pavel).
  • Kerberos Authenticaiton for HDFS. #16621 (Ilya Golshtein).
  • 👌 Support SHOW SETTINGS statement to show parameters in system.settings. SHOW CHANGED SETTINGS and LIKE/ILIKE clause are also supported. #18056 (Jianmei Zhang).
  • 👍 Function position now supports POSITION(needle IN haystack) synax for SQL compatibility. This closes #18701. ... #18779 (Jianmei Zhang).
  • Now we have a new storage setting max_partitions_to_read for tables in the MergeTree family. It limits the max number of partitions that can be accessed in one query. A user setting force_max_partition_limit is also added to enforce this constraint. #18712 (Amos Bird).
  • Add query_id column to system.part_log for inserted parts. Closes #10097. #18644 (flynn).
  • 👍 Allow create table as select with columns specification. Example CREATE TABLE t1 (x String) ENGINE = Memory AS SELECT 1;. #18060 (Maksim Kita).
  • ➕ Added arrayMin, arrayMax, arrayAvg aggregation functions. #18032 (Maksim Kita).
  • Implemented ATTACH TABLE name FROM 'path/to/data/' (col1 Type1, ... query. It creates new table with provided structure and attaches table data from provided directory in user_files. #17903 (tavplubix).
  • ➕ Add mutation support for StorageMemory. This closes #9117. #15127 (flynn).
  • 👌 Support syntax EXISTS DATABASE name. #18458 (Du Chuan).
  • Support builtin function isIPv4String && isIPv6String like MySQL. #18349 (Du Chuan).
  • Add a new setting insert_distributed_one_random_shard = 1 to allow insertion into multi-sharded distributed table without any distributed key. #18294 (Amos Bird).
  • Add settings min_compress_block_size and max_compress_block_size to MergeTreeSettings, which have higher priority than the global settings and take effect when they are set. close 13890. #17867 (flynn).
  • ➕ Add support for 64bit roaring bitmaps. #17858 (Andy Yang).
  • ⚡️ Extended OPTIMIZE ... DEDUPLICATE syntax to allow explicit (or implicit with asterisk/column transformers) list of columns to check for duplicates on. ... #17846 (Vasily Nemkov).
  • ➕ Added functions toModifiedJulianDay, fromModifiedJulianDay, toModifiedJulianDayOrNull, and fromModifiedJulianDayOrNull. These functions convert between Proleptic Gregorian calendar date and Modified Julian Day number. #17750 (PHO).
  • ➕ Add ability to use custom TLD list: added functions firstSignificantSubdomainCustom, cutToFirstSignificantSubdomainCustom. #17748 (Azat Khuzhin).
  • ➕ Add support for PROXYv1 protocol to wrap native TCP interface. Allow quotas to be keyed by proxy-forwarded IP address (applied for PROXYv1 address and for X-Forwarded-For from HTTP interface). This is useful when you provide access to ClickHouse only via trusted proxy (e.g. CloudFlare) but want to account user resources by their original IP addresses. This fixes #17268. #17707 (alexey-milovidov).
  • 👍 Now clickhouse-client supports opening EDITOR to edit commands. Alt-Shift-E. #17665 (Amos Bird).
  • ➕ Add function encodeXMLComponent to escape characters to place string into XML text node or attribute. #17659 (nauta).
  • 🛠 Introduce DETACH TABLE/VIEW ... PERMANENTLY syntax, so that after restarting the table does not reappear back automatically on restart (only by explicit request). The table can still be attached back using the short syntax ATTACH TABLE. Implements #5555. Fixes #13850. #17642 (filimonov).
  • ➕ Add asynchronous metrics on total amount of rows, bytes and parts in MergeTree tables. This fix #11714. #17639 (flynn).
  • Add settings limit and offset for out-of-SQL pagination: #16176 They are useful for building APIs. These two settings will affect SELECT query as if it is added like select * from (your_original_select_query) t limit xxx offset xxx;. #17633 (hexiaoting).
  • 🏗 Provide a new aggregator combinator : -SimpleState to build SimpleAggregateFunction types via query. It's useful for defining MaterializedView of AggregatingMergeTree engine, and will benefit projections too. #16853 (Amos Bird).
  • ➕ Added queries-file parameter for clickhouse-client and clickhouse-local. #15930 (Maksim Kita).
  • ➕ Added query parameter for clickhouse-benchmark. #17832 (Maksim Kita).
  • 👍 EXPLAIN AST now support queries other then SELECT. #18136 (taiyang-li).

  Experimental Feature

  • ➕ Added functions for calculation of minHash and simHash of text n-grams and shingles. They are intended for semi-duplicate search. Also functions bitHammingDistance and tupleHammingDistance are added. #7649 (flynn).
  • ➕ Add new data type Map. See #1841. First version for Map only supports String type of key and value. #15806 (hexiaoting).
  • 📜 Implement alternative SQL parser based on ANTLR4 runtime and generated from EBNF grammar. #11298 (Ivan).

  🐎 Performance Improvement

  • 🆕 New IP Dictionary implementation with lower memory consumption, improved performance for some cases, and fixed bugs. #16804 (vdimir).
  • Parallel formatting for data export. #11617 (Nikita Mikhaylov).
  • 🔧 LDAP integration: Added verification_cooldown parameter in LDAP server connection configuration to allow caching of successful "bind" attempts for configurable period of time. #15988 (Denis Glazachev).
  • ➕ Add --no-system-table option for clickhouse-local to run without system tables. This avoids initialization of DateLUT that may take noticeable amount of time (tens of milliseconds) at startup. #18899 (alexey-milovidov).
  • 🐎 Replace PODArray with PODArrayWithStackMemory in AggregateFunctionWindowFunnelData to improve windowFunnel function performance. #18817 (flynn).
  • 🔀 Don't send empty blocks to shards on synchronous INSERT into Distributed table. This closes #14571. #18775 (alexey-milovidov).
  • ⚡️ Optimized read for StorageMemory. #18052 (Maksim Kita).
  • 🐎 Using Dragonbox algorithm for float to string conversion instead of ryu. This improves performance of float to string conversion significantly. #17831 (Maksim Kita).
  • Speedup IPv6CIDRToRange implementation. #17569 (vdimir).
  • Add remerge_sort_lowered_memory_bytes_ratio setting (If memory usage after remerge does not reduced by this ratio, remerge will be disabled). #17539 (Azat Khuzhin).
  • 👌 Improve performance of AggregatingMergeTree with SimpleAggregateFunction(String) in PK. #17109 (Azat Khuzhin).
  • Now the -If combinator is devirtualized, and count is properly vectorized. It is for this PR. #17043 (Amos Bird).
  • 🛠 Fix performance of reading from Merge tables over huge number of MergeTree tables. Fixes #7748. #16988 (Anton Popov).
  • 👌 Improved performance of function repeat. #16937 (satanson).
  • 🐎 Slightly improved performance of float parsing. #16809 (Maksim Kita).
  • ➕ Add possibility to skip merged partitions for OPTIMIZE TABLE ... FINAL. #15939 (Kruglov Pavel).
  • Integrate with fast_float from Daniel Lemire to parse floating point numbers. #16787 (Maksim Kita). It is not enabled, because performance its performance is still lower than rough float parser in ClickHouse.
  • Fix max_distributed_connections (affects prefer_localhost_replica = 1 and max_threads != max_distributed_connections). #17848 (Azat Khuzhin).
  • Adaptive choice of single/multi part upload when sending data to S3. Single part upload is controlled by a new setting max_single_part_upload_size. #17934 (Pavel Kovalenko).
  • 👌 Support for async tasks in PipelineExecutor. Initial support of async sockets for remote queries. #17868 (Nikolai Kochetov).
  • 🚚 Allow to use optimize_move_to_prewhere optimization with compact parts, when sizes of columns are unknown. #17330 (Anton Popov).

  👌 Improvement

  • 🌲 Avoid deadlock when executing INSERT SELECT into itself from a table with TinyLog or Log table engines. This closes #6802. This closes #18691. This closes #16812. This closes #14570. #15260 (alexey-milovidov).
  • 👌 Support SHOW CREATE VIEW name syntax like MySQL. #18095 (Du Chuan).
  • All queries of type Decimal * Float or vice versa are allowed, including aggregate ones (e.g. SELECT sum(decimal_field * 1.1) or SELECT dec_col * float_col), the result type is Float32 or Float64. #18145 (Mike).
  • 👌 Improved minimal Web UI: add history; add sharing support; avoid race condition of different requests; add request in-flight and ready indicators; add favicon; detect Ctrl+Enter if textarea is not in focus. #17293 #17770 (alexey-milovidov).
  • clickhouse-server didn't send close request to ZooKeeper server. #16837 (alesapin).
  • Avoid server abnormal termination in case of too low memory limits (max_memory_usage = 1 / max_untracked_memory = 1). #17453 (Azat Khuzhin).
  • 🛠 Fix non-deterministic result of windowFunnel function in case of same timestamp for different events. #18884 (Fuwang Hu).
  • 🐳 Docker: Explicitly set uid / gid of clickhouse user & group to the fixed values (101) in clickhouse-server Docker images. #19096 (filimonov).
  • Asynchronous INSERTs to Distributed tables: Two new settings (by analogy with MergeTree family) has been added: - fsync_after_insert - Do fsync for every inserted. Will decreases performance of inserts. - fsync_directories - Do fsync for temporary directory (that is used for async INSERT only) after all operations (writes, renames, etc.). #18864 (Azat Khuzhin).
  • 🐳 SYSTEM KILL command started to work in Docker. This closes #18847. #18848 (alexey-milovidov).
  • Expand macros in the zk path when executing FETCH PARTITION. #18839 (fastio).
  • Apply ALTER TABLE <replicated_table> ON CLUSTER MODIFY SETTING ... to all replicas. Because we don't replicate such alter commands. #18789 (Amos Bird).
  • 👍 Allow column transformer EXCEPT to accept a string as regular expression matcher. This resolves #18685 . #18699 (Amos Bird).
  • 🛠 Fix SimpleAggregateFunction in SummingMergeTree. Now it works like AggregateFunction. In previous versions values were summed together regardless to the aggregate function. This fixes #18564 . #8052. #18637 (Amos Bird). Another fix of using SimpleAggregateFunction in SummingMergeTree. This fixes #18676 . #18677 (Amos Bird).
  • 🛠 Fixed assertion error inside allocator in case when last argument of function bar is NaN. Now simple ClickHouse's exception is being thrown. This fixes #17876. #18520 (Nikita Mikhaylov).
  • 🛠 Fix usability issue: no newline after exception message in some tools. #18444 (alexey-milovidov).
  • ➕ Add ability to modify primary and partition key column type from LowCardinality(Type) to Type and vice versa. Also add an ability to modify primary key column type from EnumX to IntX type. Fixes #5604. #18362 (alesapin).
  • Implement untuple field access. #18133. #18309 (hexiaoting).
  • 👍 Allow to parse Array fields from CSV if it is represented as a string containing array that was serialized as nested CSV. Example: "[""Hello"", ""world"", ""42"""" TV""]" will parse as ['Hello', 'world', '42" TV']. Allow to parse array in CSV in a string without enclosing braces. Example: "'Hello', 'world', '42"" TV'" will parse as ['Hello', 'world', '42" TV']. #18271 (alexey-milovidov).
  • 🔀 Make better adaptive granularity calculation for merge tree wide parts. #18223 (alesapin).
  • Now clickhouse install could work on Mac. The problem was that there is no procfs on this platform. #18201 (Nikita Mikhaylov).
  • 👍 Better hints for SHOW ... query syntax. #18183 (Du Chuan).
  • 👍 Array aggregation arrayMin, arrayMax, arraySum, arrayAvg support for Int128, Int256, UInt256. #18147 (Maksim Kita).
  • ➕ Add disk to Set and Join storage settings. #18112 (Grigory Pervakov).
  • 🔀 Access control: Now table function merge() requires current user to have SELECT privilege on each table it receives data from. This PR fixes #16964. #18104 #17983 (Vitaly Baranov).
  • Temporary tables are visible in the system tables system.tables and system.columns now only in those session where they have been created. The internal database _temporary_and_external_tables is now hidden in those system tables; temporary tables are shown as tables with empty database with the is_temporary flag set instead. #18014 (Vitaly Baranov).
  • 🛠 Fix clickhouse-client rendering issue when the size of terminal window changes. #18009 (Amos Bird).
  • ⬇ Decrease log verbosity of the events when the client drops the connection from Warning to Information. #18005 (filimonov).
  • 📇 Forcibly removing empty or bad metadata files from filesystem for DiskS3. S3 is an experimental feature. #17935 (Pavel Kovalenko).
  • Access control: allow_introspection_functions=0 prohibits usage of introspection functions but doesn't prohibit giving grants for them anymore (the grantee will need to set allow_introspection_functions=1 for himself to be able to use that grant). Similarly allow_ddl=0 prohibits usage of DDL commands but doesn't prohibit giving grants for them anymore. #17908 (Vitaly Baranov).
  • Usability improvement: hints for column names. #17112. #17857 (fastio).
  • ➕ Add diagnostic information when two merge tables try to read each other's data. #17854 (徐炘).
  • 🐳 Let the possibility to override timeout value for running script using the ClickHouse docker image. #17818 (Guillaume Tassery).
  • 🔧 Check system log tables' engine definition grammar to prevent some configuration errors. Notes that this grammar check is not semantical, that means such mistakes as non-existent columns / expression functions would be not found out util the table is created. #17739 (Du Chuan).
  • ✂ Removed exception throwing at RabbitMQ table initialization if there was no connection (it will be reconnecting in the background). #17709 (Kseniia Sumarokova).
  • Do not ignore server memory limits during Buffer flush. #17646 (Azat Khuzhin).
  • 🆓 Switch to patched version of RocksDB (from ClickHouse-Extras) to fix use-after-free error. #17643 (Nikita Mikhaylov).
  • ➕ Added an offset to exception message for parallel parsing. This fixes #17457. #17641 (Nikita Mikhaylov).
  • Don't throw "Too many parts" error in the middle of INSERT query. #17566 (alexey-milovidov).
  • 👍 Allow query parameters in UPDATE statement of ALTER query. Fixes #10976. #17563 (alexey-milovidov).
  • Query obfuscator: avoid usage of some SQL keywords for identifier names. #17526 (alexey-milovidov).
  • 👷 Export current max ddl entry executed by DDLWorker via server metric. It's useful to check if DDLWorker hangs somewhere. #17464 (Amos Bird).
  • Export asynchronous metrics of all servers current threads. It's useful to track down issues like this. #17463 (Amos Bird).
  • Include dynamic columns like MATERIALIZED / ALIAS for wildcard query when settings asterisk_include_materialized_columns and asterisk_include_alias_columns are turned on. #17462 (Ken Chen).
  • 👍 Allow specifying TTL to remove old entries from system log tables, using the <ttl> attribute in config.xml. #17438 (Du Chuan).
  • 👀 Now queries coming to the server via MySQL and PostgreSQL protocols have distinctive interface types (which can be seen in the interface column of the tablesystem.query_log): 4 for MySQL, and 5 for PostgreSQL, instead of formerly used 1 which is now used for the native protocol only. #17437 (Vitaly Baranov).
  • 🛠 Fix parsing of SETTINGS clause of the INSERT ... SELECT ... SETTINGS query. #17414 (Azat Khuzhin).
  • Correctly account memory in RadixSort. #17412 (Nikita Mikhaylov).
  • ➕ Add eof check in receiveHello in server to prevent getting Attempt to read after eof exception. #17365 (Kruglov Pavel).
  • Avoid possible stack overflow in bigint conversion. Big integers are experimental. #17269 (flynn).
  • 🛠 Now set indices will work with GLOBAL IN. This fixes #17232 , #5576 . #17253 (Amos Bird).
  • Add limit for http redirects in request to S3 storage (s3_max_redirects). #17220 (ianton-ru).
  • 🔀 When -OrNull combinator combined -If, -Merge, -MergeState, -State combinators, we should put -OrNull in front. #16935 (flynn).
  • 👌 Support HTTP proxy and HTTPS S3 endpoint configuration. #16861 (Pavel Kovalenko).
  • ➕ Added proper authentication using environment, ~/.aws and AssumeRole for S3 client. #16856 (Vladimir Chebotarev).
  • ➕ Add more OpenTelemetry spans. Add an example of how to export the span data to Zipkin. #16535 (Alexander Kuzmenkov).
  • Cache dictionaries: Completely eliminate callbacks and locks for acquiring them. Keys are not divided into "not found" and "expired", but stored in the same map during query. #14958 (Nikita Mikhaylov).
  • Fix never worked fsync_part_directory/fsync_after_insert/in_memory_parts_insert_sync (experimental feature). #18845 (Azat Khuzhin).
  • 👍 Allow using Atomic engine for nested database of MaterializeMySQL engine. #14849 (tavplubix).

  🐛 Bug Fix

  • 🛠 Fix the issue when server can stop accepting connections in very rare cases. #17542 (Amos Bird, alexey-milovidov).
  • 🛠 Fix index analysis of binary functions with constant argument which leads to wrong query results. This fixes #18364. #18373 (Amos Bird).
  • 🛠 Fix possible wrong index analysis when the types of the index comparison are different. This fixes #17122. #17145 (Amos Bird).
  • 🔀 Disable write with AIO during merges because it can lead to extremely rare data corruption of primary key columns during merge. #18481 (alesapin).
  • 🔀 Restrict merges from wide to compact parts. In case of vertical merge it led to broken result part. #18381 (Anton Popov).
  • 🛠 Fix possible incomplete query result while reading from MergeTree* in case of read backoff (message <Debug> MergeTreeReadPool: Will lower number of threads in logs). Was introduced in #16423. Fixes #18137. #18216 (Nikolai Kochetov).
  • 🛠 Fix use after free bug in rocksdb library. #18862 (sundyli).
  • 🛠 Fix infinite reading from file in ORC format (was introduced in #10580). Fixes #19095. #19134 (Nikolai Kochetov).
  • 🛠 Fix bug in merge tree data writer which can lead to marks with bigger size than fixed granularity size. Fixes #18913. #19123 (alesapin).
  • 🛠 Fix startup bug when clickhouse was not able to read compression codec from LowCardinality(Nullable(...)) and throws exception Attempt to read after EOF. Fixes #18340. #19101 (alesapin).
  • 🔀 Restrict MODIFY TTL queries for MergeTree tables created in old syntax. Previously the query succeeded, but actually it had no effect. #19064 (Anton Popov).
  • 👉 Make sure groupUniqArray returns correct type for argument of Enum type. This closes #17875. #19019 (alexey-milovidov).
  • 🛠 Fix possible error Expected single dictionary argument for function if use function ignore with LowCardinality argument. Fixes #14275. #19016 (Nikolai Kochetov).
  • 🛠 Fix inserting of LowCardinality column to table with TinyLog engine. Fixes #18629. #19010 (Nikolai Kochetov).
  • Join tries to materialize const columns, but our code wants them in other places. #18982 (Nikita Mikhaylov).
  • 🚚 Disable optimize_move_functions_out_of_any because optimization is not always correct. This closes #18051. This closes #18973. #18981 (alexey-milovidov).
  • 🛠 Fix possible exception QueryPipeline stream: different number of columns caused by merging of query plan's Expression steps. Fixes #18190. #18980 (Nikolai Kochetov).
  • 🛠 Fixed very rare deadlock at shutdown. #18977 (tavplubix).
  • 🛠 Fix incorrect behavior when ALTER TABLE ... DROP PART 'part_name' query removes all deduplication blocks for the whole partition. Fixes #18874. #18969 (alesapin).
  • Attach partition should reset the mutation. #18804. #18935 (fastio).
  • 🛠 Fix issue with bitmapOrCardinality that may lead to nullptr dereference. This closes #18911. #18912 (sundyli).
  • 🛠 Fix possible hang at shutdown in clickhouse-local. This fixes #18891. #18893 (alexey-milovidov).
  • 🛠 Queries for external databases (MySQL, ODBC, JDBC) were incorrectly rewritten if there was an expression in form of x IN table. This fixes #9756. #18876 (alexey-milovidov).
  • 🛠 Fix *If combinator with unary function and Nullable types. #18806 (Azat Khuzhin).
  • Fix the issue that asynchronous distributed INSERTs can be rejected by the server if the setting network_compression_method is globally set to non-default value. This fixes #18741. #18776 (alexey-milovidov).
  • 🛠 Fixed Attempt to read after eof error when trying to CAST NULL from Nullable(String) to Nullable(Decimal(P, S)). Now function CAST returns NULL when it cannot parse decimal from nullable string. Fixes #7690. #18718 (Winter Zhang).
  • 🛠 Fix minor issue with logging. #18717 (sundyli).
  • 🛠 Fix removing of empty parts in ReplicatedMergeTree tables, created with old syntax. Fixes #18582. #18614 (Anton Popov).
  • 🛠 Fix previous bug when date overflow with different values. Strict Date value limit to "2106-02-07", cast date > "2106-02-07" to value 0. #18565 (hexiaoting).
  • ➕ Add FixedString data type support for replication from MySQL. Replication from MySQL is an experimental feature. This patch fixes #18450 Also fixes #6556. #18553 (awesomeleo).
  • 🛠 Fix possible Pipeline stuck error while using ORDER BY after subquery with RIGHT or FULL join. #18550 (Nikolai Kochetov).
  • 🛠 Fix bug which may lead to ALTER queries hung after corresponding mutation kill. Found by thread fuzzer. #18518 (alesapin).
  • 📜 Proper support for 12AM in parseDateTimeBestEffort function. This fixes #18402. #18449 (vladimir-golovchenko).
  • 🛠 Fixed value is too short error when executing toType(...) functions (toDate, toUInt32, etc) with argument of type Nullable(String). Now such functions return NULL on parsing errors instead of throwing exception. Fixes #7673. #18445 (tavplubix).
  • 🛠 Fix the unexpected behaviour of SHOW TABLES. #18431 (fastio).
  • 🛠 Fix -SimpleState combinator generates incompatible arugment type and return type. #18404 (Amos Bird).
  • 🛠 Fix possible race condition in concurrent usage of Set or Join tables and selects from system.tables. #18385 (alexey-milovidov).
  • Fix filling table system.settings_profile_elements. This PR fixes #18231. #18379 (Vitaly Baranov).
  • 🛠 Fix possible crashes in aggregate functions with combinator Distinct, while using two-level aggregation. Fixes #17682. #18365 (Anton Popov).
  • 🛠 Fixed issue when clickhouse-odbc-bridge process is unreachable by server on machines with dual IPv4/IPv6 stack; Fixed issue when ODBC dictionary updates are performed using malformed queries and/or cause crashes of the odbc-bridge process; Possibly closes #14489. #18278 (Denis Glazachev).
  • 🛠 Access control: SELECT count() FROM table now can be executed if the user has access to at least single column from a table. This PR fixes #10639. #18233 (Vitaly Baranov).
  • 🛠 Access control: SELECT JOIN now requires the SELECT privilege on each of the joined tables. This PR fixes #17654. #18232 (Vitaly Baranov).
  • 🛠 Fix key comparison between Enum and Int types. This fixes #17989. #18214 (Amos Bird).
  • 🛠 Replication from MySQL (experimental feature). Fixes #18186 Fixes #16372 Fix unique key convert issue in MaterializeMySQL database engine. #18211 (Winter Zhang).
  • 🛠 Fix inconsistency for queries with both WITH FILL and WITH TIES #17466. #18188 (hexiaoting).
  • 🛠 Fix inserting a row with default value in case of parsing error in the last column. Fixes #17712. #18182 (Jianmei Zhang).
  • 🛠 Fix Unknown setting profile error on attempt to set settings profile. #18167 (tavplubix).
  • 🛠 Fix error when query MODIFY COLUMN ... REMOVE TTL doesn't actually remove column TTL. #18130 (alesapin).
  • Fixed std::out_of_range: basic_string in S3 URL parsing. #18059 (Vladimir Chebotarev).
  • 🛠 Fix comparison of DateTime64 and Date. Fixes #13804 and #11222. ... #18050 (Vasily Nemkov).
  • 🛠 Replication from MySQL (experimental feature): Fixes #15187 Fixes #17912 support convert MySQL prefix index for MaterializeMySQL. #17944 (Winter Zhang).
  • 🔧 When server log rotation was configured using logger.size parameter with numeric value larger than 2^32, the logs were not rotated properly. This is fixed. #17905 (Alexander Kuzmenkov).
  • Trivial query optimization was producing wrong result if query contains ARRAY JOIN (so query is actually non trivial). #17887 (sundyli).
  • 🛠 Fix possible segfault in topK aggregate function. This closes #17404. #17845 (Maksim Kita).
  • WAL (experimental feature): Do not restore parts from WAL if in_memory_parts_enable_wal is disabled. #17802 (detailyang).
  • 👻 Exception message about max table size to drop was displayed incorrectly. #17764 (alexey-milovidov).
  • 🛠 Fixed possible segfault when there is not enough space when inserting into Distributed table. #17737 (tavplubix).
  • 🛠 Fixed problem when ClickHouse fails to resume connection to MySQL servers. #17681 (Alexander Kazakov).
  • 🐧 Windows: Fixed Function not implemented error when executing RENAME query in Atomic database with ClickHouse running on Windows Subsystem for Linux. Fixes #17661. #17664 (tavplubix).
  • 🛠 In might be determined incorrectly if cluster is circular- (cross-) replicated or not when executing ON CLUSTER query due to race condition when pool_size > 1. It's fixed. #17640 (tavplubix).
  • 🛠 Fix empty system.stack_trace table when server is running in daemon mode. #17630 (Amos Bird).
  • 🔀 Exception fmt::v7::format_error can be logged in background for MergeTree tables. This fixes #17613. #17615 (alexey-milovidov).
  • 🛠 When clickhouse-client is used in interactive mode with multiline queries, single line comment was erronously extended till the end of query. This fixes #13654. #17565 (alexey-milovidov).
  • 🛠 Fix alter query hang when the corresponding mutation was killed on the different replica. Fixes #16953. #17499 (alesapin).
  • 🛠 Fix issue with memory accounting when mark cache size was underestimated by clickhouse. It may happen when there are a lot of tiny files with marks. #17496 (alesapin).
  • Fix ORDER BY with enabled setting optimize_redundant_functions_in_order_by. #17471 (Anton Popov).
  • 🛠 Fix duplicates after DISTINCT which were possible because of incorrect optimization. Fixes #17294. #17296 (li chengxiang). #17439 (Nikolai Kochetov).
  • 🛠 Fixed high CPU usage in background tasks of *MergeTree tables. #17416 (tavplubix).
  • 🛠 Fix possible crash while reading from JOIN table with LowCardinality types. Fixes #17228. #17397 (Nikolai Kochetov).
  • 🛠 Replication from MySQL (experimental feature): Fixes #16835 try fix miss match header with MySQL SHOW statement. #17366 (Winter Zhang).
  • 🛠 Fix nondeterministic functions with predicate optimizer. This fixes #17244. #17273 (Winter Zhang).
  • 🛠 Fix possible Unexpected packet Data received from client error for Distributed queries with LIMIT. #17254 (Azat Khuzhin).
  • 🛠 Fix set index invalidation when there are const columns in the subquery. This fixes #17246. #17249 (Amos Bird).
  • clickhouse-copier: Fix for non-partitioned tables #15235. #17248 (Qi Chen).
  • 🛠 Fixed possible not-working mutations for parts stored on S3 disk (experimental feature). #17227 (Pavel Kovalenko).
  • 🐛 Bug fix for funciton fuzzBits, related issue: #16980. #17051 (hexiaoting).
  • Fix optimize_distributed_group_by_sharding_key for query with OFFSET only. #16996 (Azat Khuzhin).
  • 🛠 Fix queries from Merge tables over Distributed tables with JOINs. #16993 (Azat Khuzhin).
  • 🛠 Fix order by optimization with monotonic functions. Fixes #16107. #16956 (Anton Popov).
  • 🛠 Fix incorrect comparison of types DateTime64 with different scales. Fixes #16655 ... #16952 (Vasily Nemkov).
  • Fix optimization of group by with enabled setting optimize_aggregators_of_group_by_keys and joins. Fixes #12604. #16951 (Anton Popov).
  • Minor fix in SHOW ACCESS query. #16866 (tavplubix).
  • Fix the behaviour with enabled optimize_trivial_count_query setting with partition predicate. #16767 (Azat Khuzhin).
  • 🛠 Return number of affected rows for INSERT queries via MySQL wire protocol. Previously ClickHouse used to always return 0, it's fixed. Fixes #16605. #16715 (Winter Zhang).
  • Fix inconsistent behavior caused by select_sequential_consistency for optimized trivial count query and system tables. #16309 (Hao Chen).
  • Throw error when REPLACE column transformer operates on non existing column. #16183 (hexiaoting).
  • 👻 Throw exception in case of not equi-join ON expression in RIGH|FULL JOIN. #15162 (Artem Zuikov).

  🏗 Build/Testing/Packaging Improvement

  • ➕ Add simple integrity check for ClickHouse binary. It allows to detect corruption due to faulty hardware (bit rot on storage media or bit flips in RAM). #18811 (alexey-milovidov).
  • 🔄 Change OpenSSL to BoringSSL. It allows to avoid issues with sanitizers. This fixes #12490. This fixes #17502. This fixes #12952. #18129 (alexey-milovidov).
  • Simplify Sys/V init script. It was not working on Ubuntu 12.04 or older. #17428 (alexey-milovidov).
  • Multiple improvements in ./clickhouse install script. #17421 (alexey-milovidov).
  • 👍 Now ClickHouse can pretend to be a fake ZooKeeper. Currently, storage implementation is just stored in-memory hash-table, and server partially support ZooKeeper protocol. #16877 (alesapin).
  • 🛠 Fix dead list watches removal for TestKeeperStorage (a mock for ZooKeeper). #18065 (alesapin).
  • ➕ Add SYSTEM SUSPEND command for fault injection. It can be used to faciliate failover tests. This closes #15979. #18850 (alexey-milovidov).
  • 🏗 Generate build id when ClickHouse is linked with lld. It's appeared that lld does not generate it by default on my machine. Build id is used for crash reports and introspection. #18808 (alexey-milovidov).
  • 🛠 Fix shellcheck errors in style check. #18566 (Ilya Yatsishin).
  • ⚡️ Update timezones info to 2020e. #18531 (alesapin).
  • 🛠 Fix codespell warnings. Split style checks into separate parts. Update style checks docker image. #18463 (Ilya Yatsishin).
  • 📄 Automated check for leftovers of conflict markers in docs. #18332 (alexey-milovidov).
  • ✅ Enable Thread Fuzzer for stateless tests flaky check. #18299 (alesapin).
  • Do not use non thread-safe function strerror. #18204 (alexey-milovidov).
  • ⚡️ Update anchore/scan-action@main workflow action (was moved from master to main). #18192 (Stig Bakken).
  • ✅ Now clickhouse-test does DROP/CREATE databases with a timeout. #18098 (alesapin).
  • ✅ Enable experimental support for Pytest framework for stateless tests. #17902 (Ivan).
  • 🐳 Now we use the fresh docker daemon version in integration tests. #17671 (alesapin).
  • 🏗 Send info about official build, memory, cpu and free disk space to Sentry if it is enabled. Sentry is opt-in feature to help ClickHouse developers. This closes #17279. #17543 (alexey-milovidov).
  • There was an uninitialized variable in the code of clickhouse-copier. #17363 (Nikita Mikhaylov).
  • ✅ Fix one MSan report from #17309. #17344 (Nikita Mikhaylov).
  • 🛠 Fix for the issue with IPv6 in Arrow Flight library. See the comments for details. #16664 (Zhanna).
  • ➕ Add a library that replaces some libc functions to traps that will terminate the process. #16366 (alexey-milovidov).
  • 🔊 Provide diagnostics in server logs in case of stack overflow, send error message to clickhouse-client. This closes #14840. #16346 (alexey-milovidov).
  • ✅ Now we can run almost all stateless functional tests in parallel. #15236 (alesapin).
  • 🛠 Fix corruption in librdkafka snappy decompression (was a problem only for gcc10 builds, but official builds uses clang already, so at least recent official releases are not affected). #18053 (Azat Khuzhin).
  • 🖨 If server was terminated by OOM killer, print message in log. #13516 (alexey-milovidov).
  • 🛠 PODArray: Avoid call to memcpy with (nullptr, 0) arguments (Fix UBSan report). This fixes #18525. #18526 (alexey-milovidov).
  • 👷 Minor improvement for path concatenation of zookeeper paths inside DDLWorker. #17767 (Bharat Nallan).
  • 👍 Allow to reload symbols from debug file. This PR also fixes a build-id issue. #17637 (Amos Bird).
  • ✅ TestFlows: fixes to LDAP tests that fail due to slow test execution. #18790 (vzakaznikov).
  • ⚡️ TestFlows: Merging requirements for AES encryption functions. Updating aes_encryption tests to use new requirements. Updating TestFlows version to 1.6.72. #18221 (vzakaznikov).
  • ⚡️ TestFlows: Updating TestFlows version to the latest 1.6.72. Re-generating requirements.py. #18208 (vzakaznikov).
  • ⚡️ TestFlows: Updating TestFlows README.md to include "How To Debug Why Test Failed" section. #17808 (vzakaznikov).
  • ✅ TestFlows: tests for RBAC ACCESS MANAGEMENT privileges. #17804 (MyroTk).
  • ⚡️ TestFlows: RBAC tests for SHOW, TRUNCATE, KILL, and OPTIMIZE. - Updates to old tests. - Resolved comments from #https://github.com/ClickHouse/ClickHouse/pull/16977. #17657 (MyroTk).
  • ✅ TestFlows: Added RBAC tests for ATTACH, CREATE, DROP, and DETACH. #16977 (MyroTk).

• v20.11.3.3 Changes

  November 13, 2020

  🐛 Bug Fix

  • 🛠 Fix rare silent crashes when query profiler is on and ClickHouse is installed on OS with glibc version that has (supposedly) broken asynchronous unwind tables for some functions. This fixes #15301. This fixes #13098. #16846 (alexey-milovidov).

• v20.11.2.1 Changes

  November 11, 2020

  Backward Incompatible Change

  • 0️⃣ If some profile was specified in distributed_ddl config section, then this profile could overwrite settings of default profile on server startup. It's fixed, now settings of distributed DDL queries should not affect global server settings. #16635 (tavplubix).
  • Restrict to use of non-comparable data types (like AggregateFunction) in keys (Sorting key, Primary key, Partition key, and so on). #16601 (alesapin).
  • Remove ANALYZE and AST queries, and make the setting enable_debug_queries obsolete since now it is the part of full featured EXPLAIN query. #16536 (Ivan).
  • Aggregate functions boundingRatio, rankCorr, retention, timeSeriesGroupSum, timeSeriesGroupRateSum, windowFunnel were erroneously made case-insensitive. Now their names are made case sensitive as designed. Only functions that are specified in SQL standard or made for compatibility with other DBMS or functions similar to those should be case-insensitive. #16407 (alexey-milovidov).
  • 👉 Make rankCorr function return nan on insufficient data https://github.com/ClickHouse/ClickHouse/issues/16124. #16135 (hexiaoting).
  • ⚡️ When upgrading from versions older than 20.5, if rolling update is performed and cluster contains both versions 20.5 or greater and less than 20.5, if ClickHouse nodes with old versions are restarted and old version has been started up in presence of newer versions, it may lead to Part ... intersects previous part errors. To prevent this error, first install newer clickhouse-server packages on all cluster nodes and then do restarts (so, when clickhouse-server is restarted, it will start up with the new version).

  🆕 New Feature

  • ➕ Added support of LDAP as a user directory for locally non-existent users. #12736 (Denis Glazachev).
  • ➕ Add system.replicated_fetches table which shows currently running background fetches. #16428 (alesapin).
  • Added setting date_time_output_format. #15845 (Maksim Kita).
  • ➕ Added minimal web UI to ClickHouse. #16158 (alexey-milovidov).
  • 👍 Allows to read/write Single protobuf message at once (w/o length-delimiters). #15199 (filimonov).
  • Added initial OpenTelemetry support. ClickHouse now accepts OpenTelemetry traceparent headers over Native and HTTP protocols, and passes them downstream in some cases. The trace spans for executed queries are saved into the system.opentelemetry_span_log table. #14195 (Alexander Kuzmenkov).
  • 👍 Allow specify primary key in column list of CREATE TABLE query. This is needed for compatibility with other SQL dialects. #15823 (Maksim Kita).
  • Implement OFFSET offset_row_count {ROW | ROWS} FETCH {FIRST | NEXT} fetch_row_count {ROW | ROWS} {ONLY | WITH TIES} in SELECT query with ORDER BY. This is the SQL-standard way to specify LIMIT. #15855 (hexiaoting).
  • 🌲 errorCodeToName function - return variable name of the error (useful for analyzing query_log and similar). system.errors table - shows how many times errors has been happened (respects system_events_show_zero_values). #16438 (Azat Khuzhin).
  • ➕ Added function untuple which is a special function which can introduce new columns to the SELECT list by expanding a named tuple. #16242 (Nikolai Kochetov, Amos Bird).
  • Now we can provide identifiers via query parameters. And these parameters can be used as table objects or columns. #16594 (Amos Bird).
  • ➕ Added big integers (UInt256, Int128, Int256) and UUID data types support for MergeTree BloomFilter index. Big integers is an experimental feature. #16642 (Maksim Kita).
  • ➕ Add farmFingerprint64 function (non-cryptographic string hashing). #16570 (Jacob Hayes).
  • Add log_queries_min_query_duration_ms, only queries slower then the value of this setting will go to query_log/query_thread_log (i.e. something like slow_query_log in mysql). #16529 (Azat Khuzhin).
  • 🐳 Ability to create a docker image on the top of Alpine. Uses precompiled binary and glibc components from ubuntu 20.04. #16479 (filimonov).
  • ➕ Added toUUIDOrNull, toUUIDOrZero cast functions. #16337 (Maksim Kita).
  • Add max_concurrent_queries_for_all_users setting, see #6636 for use cases. #16154 (nvartolomei).
  • Add a new option print_query_id to clickhouse-client. It helps generate arbitrary strings with the current query id generated by the client. Also print query id in clickhouse-client by default. #15809 (Amos Bird).
  • ➕ Add tid and logTrace functions. This closes #9434. #15803 (flynn).
  • ➕ Add function formatReadableTimeDelta that format time delta to human readable string ... #15497 (Filipe Caixeta).
  • ➕ Added disable_merges option for volumes in multi-disk configuration. #13956 (Vladimir Chebotarev).

  Experimental Feature

  • New functions encrypt, aes_encrypt_mysql, decrypt, aes_decrypt_mysql. These functions are working slowly, so we consider it as an experimental feature. #11844 (Vasily Nemkov).

  🐛 Bug Fix

  • Mask password in data_path in the system.distribution_queue. #16727 (Azat Khuzhin).
  • Fix IN operator over several columns and tuples with enabled transform_null_in setting. Fixes #15310. #16722 (Anton Popov).
  • The setting max_parallel_replicas worked incorrectly if the queried table has no sampling. This fixes #5733. #16675 (alexey-milovidov).
  • Fix optimize_read_in_order/optimize_aggregation_in_order with max_threads > 0 and expression in ORDER BY. #16637 (Azat Khuzhin).
  • 0️⃣ Calculation of DEFAULT expressions was involving possible name collisions (that was very unlikely to encounter). This fixes #9359. #16612 (alexey-milovidov).
  • Fix query_thread_log.query_duration_ms unit. #16563 (Azat Khuzhin).
  • 🛠 Fix a bug when using MySQL Master -> MySQL Slave -> ClickHouse MaterializeMySQL Engine. MaterializeMySQL is an experimental feature. #16504 (TCeason).
  • 🛠 Specifically crafted argument of round function with Decimal was leading to integer division by zero. This fixes #13338. #16451 (alexey-milovidov).
  • 🛠 Fix DROP TABLE for Distributed (racy with INSERT). #16409 (Azat Khuzhin).
  • 🛠 Fix processing of very large entries in replication queue. Very large entries may appear in ALTER queries if table structure is extremely large (near 1 MB). This fixes #16307. #16332 (alexey-milovidov).
  • 🛠 Fixed the inconsistent behaviour when a part of return data could be dropped because the set for its filtration wasn't created. #16308 (Nikita Mikhaylov).
  • 🛠 Fix dictGet in sharding_key (and similar places, i.e. when the function context is stored permanently). #16205 (Azat Khuzhin).
  • 🛠 Fix the exception thrown in clickhouse-local when trying to execute OPTIMIZE command. Fixes #16076. #16192 (filimonov).
  • 🛠 Fixes #15780 regression, e.g. indexOf([1, 2, 3], toLowCardinality(1)) now is prohibited but it should not be. #16038 (Mike).
  • 🛠 Fix bug with MySQL database. When MySQL server used as database engine is down some queries raise Exception, because they try to get tables from disabled server, while it's unnecessary. For example, query SELECT ... FROM system.parts should work only with MergeTree tables and don't touch MySQL database at all. #16032 (Kruglov Pavel).
  • 0️⃣ Now exception will be thrown when ALTER MODIFY COLUMN ... DEFAULT ... has incompatible default with column type. Fixes #15854. #15858 (alesapin).
  • 🛠 Fixed IPv4CIDRToRange/IPv6CIDRToRange functions to accept const IP-column values. #15856 (vladimir-golovchenko).

  👌 Improvement

  • 🛠 Treat INTERVAL '1 hour' as equivalent to INTERVAL 1 HOUR, to be compatible with Postgres and similar. This fixes #15637. #15978 (flynn).
  • 📜 Enable parsing enum values by their numeric ids for CSV, TSV and JSON input formats. #15685 (vivarum).
  • Better read task scheduling for JBOD architecture and MergeTree storage. New setting read_backoff_min_concurrency which serves as the lower limit to the number of reading threads. #16423 (Amos Bird).
  • ➕ Add missing support for LowCardinality in Avro format. #16521 (Mike).
  • ↪ Workaround for use S3 with nginx server as proxy. Nginx currenty does not accept urls with empty path like http://domain.com?delete, but vanilla aws-sdk-cpp produces this kind of urls. This commit uses patched aws-sdk-cpp version, which makes urls with "/" as path in this cases, like http://domain.com/?delete. #16814 (ianton-ru).
  • 👍 Better diagnostics on parse errors in input data. Provide row number on Cannot read all data errors. #16644 (alexey-milovidov).
  • 🛠 Make the behaviour of minMap and maxMap more desireable. It will not skip zero values in the result. Fixes #16087. #16631 (Ildus Kurbangaliev).
  • 👍 Better update of ZooKeeper configuration in runtime. #16630 (sundyli).
  • Apply SETTINGS clause as early as possible. It allows to modify more settings in the query. This closes #3178. #16619 (alexey-milovidov).
  • Now event_time_microseconds field stores in Decimal64, not UInt64. #16617 (Nikita Mikhaylov).
  • Now paratmeterized functions can be used in APPLY column transformer. #16589 (Amos Bird).
  • 👌 Improve scheduling of background task which removes data of dropped tables in Atomic databases. Atomic databases do not create broken symlink to table data directory if table actually has no data directory. #16584 (tavplubix).
  • Subqueries in WITH section (CTE) can reference previous subqueries in WITH section by their name. #16575 (Amos Bird).
  • Add current_database into system.query_thread_log. #16558 (Azat Khuzhin).
  • 👍 Allow to fetch parts that are already committed or outdated in the current instance into the detached directory. It's useful when migrating tables from another cluster and having N to 1 shards mapping. It's also consistent with the current fetchPartition implementation. #16538 (Amos Bird).
  • 🛠 Multiple improvements for RabbitMQ: Fixed bug for #16263. Also minimized event loop lifetime. Added more efficient queues setup. #16426 (Kseniia Sumarokova).
  • 🛠 Fix debug assertion in quantileDeterministic function. In previous version it may also transfer up to two times more data over the network. Although no bug existed. This fixes #15683. #16410 (alexey-milovidov).
  • ➕ Add TablesToDropQueueSize metric. It's equal to number of dropped tables, that are waiting for background data removal. #16364 (tavplubix).
  • 👍 Better diagnostics when client has dropped connection. In previous versions, Attempt to read after EOF and Broken pipe exceptions were logged in server. In new version, it's information message Client has dropped the connection, cancel the query.. #16329 (alexey-milovidov).
  • Add total_rows/total_bytes (from system.tables) support for Set/Join table engines. #16306 (Azat Khuzhin).
  • 🔀 Now it's possible to specify PRIMARY KEY without ORDER BY for MergeTree table engines family. Closes #15591. #16284 (alesapin).
  • If there is no tmp folder in the system (chroot, misconfigutation etc) clickhouse-local will create temporary subfolder in the current directory. #16280 (filimonov).
  • ➕ Add support for nested data types (like named tuple) as sub-types. Fixes #15587. #16262 (Ivan).
  • Support for database_atomic_wait_for_drop_and_detach_synchronously/NO DELAY/SYNC for DROP DATABASE. #16127 (Azat Khuzhin).
  • Add allow_nondeterministic_optimize_skip_unused_shards (to allow non deterministic like rand() or dictGet() in sharding key). #16105 (Azat Khuzhin).
  • Fix memory_profiler_step/max_untracked_memory for queries via HTTP (test included). Fix the issue that adjusting this value globally in xml config does not help either, since those settings are not applied anyway, only default (4MB) value is used. Fix query_id for the most root ThreadStatus of the http query (by initializing QueryScope after reading query_id). #16101 (Azat Khuzhin).
  • Now it's allowed to execute ALTER ... ON CLUSTER queries regardless of the <internal_replication> setting in cluster config. #16075 (alesapin).
  • 🛠 Fix rare issue when clickhouse-client may abort on exit due to loading of suggestions. This fixes #16035. #16047 (alexey-milovidov).
  • ➕ Add support of cache layout for Redis dictionaries with complex key. #15985 (Anton Popov).
  • Fix query hang (endless loop) in case of misconfiguration (connections_with_failover_max_tries set to 0). #15876 (Azat Khuzhin).
  • 🔄 Change level of some log messages from information to debug, so information messages will not appear for every query. This closes #5293. #15816 (alexey-milovidov).
  • ✂ Remove MemoryTrackingInBackground* metrics to avoid potentially misleading results. This fixes #15684. #15813 (alexey-milovidov).
  • ➕ Add reconnects to zookeeper-dump-tree tool. #15711 (alexey-milovidov).
  • 👍 Allow explicitly specify columns list in CREATE TABLE table AS table_function(...) query. Fixes #9249 Fixes #14214. #14295 (tavplubix).

  🐎 Performance Improvement

  • 🔀 Do not merge parts across partitions in SELECT FINAL. #15938 (Kruglov Pavel).
  • 👌 Improve performance of -OrNull and -OrDefault aggregate functions. #16661 (alexey-milovidov).
  • 👌 Improve performance of quantileMerge. In previous versions it was obnoxiously slow. This closes #1463. #16643 (alexey-milovidov).
  • 👌 Improve performance of logical functions a little. #16347 (alexey-milovidov).
  • 👌 Improved performance of merges assignment in MergeTree table engines. Shouldn't be visible for the user. #16191 (alesapin).
  • 📜 Speedup hashed/sparse_hashed dictionary loading by preallocating the hash table. #15454 (Azat Khuzhin).
  • Now trivial count optimization becomes slightly non-trivial. Predicates that contain exact partition expr can be optimized too. This also fixes #11092 which returns wrong count when max_parallel_replicas > 1. #15074 (Amos Bird).

  🏗 Build/Testing/Packaging Improvement

  • ➕ Add flaky check for stateless tests. It will detect potentially flaky functional tests in advance, before they are merged. #16238 (alesapin).
  • 👉 Use proper version for croaring instead of amalgamation. #16285 (sundyli).
  • 👌 Improve generation of build files for ya.make build system (Arcadia). #16700 (alexey-milovidov).
  • ➕ Add MySQL BinLog file check tool for MaterializeMySQL database engine. MaterializeMySQL is an experimental feature. #16223 (Winter Zhang).
  • 🏁 Check for executable bit on non-executable files. People often accidentially commit executable files from Windows. #15843 (alexey-milovidov).
  • Check for #pragma once in headers. #15818 (alexey-milovidov).
  • 🛠 Fix illegal code style &vector[idx] in libhdfs3. This fixes libcxx debug build. See also https://github.com/ClickHouse-Extras/libhdfs3/pull/8 . #15815 (Amos Bird).
  • 🛠 Fix build of one miscellaneous example tool on Mac OS. Note that we don't build examples on Mac OS in our CI (we build only ClickHouse binary), so there is zero chance it will not break again. This fixes #15804. #15808 (alexey-milovidov).
  • Simplify Sys/V init script. #14135 (alexey-milovidov).
  • Added boost::program_options to db_generator in order to increase its usability. This closes #15940. #15973 (Nikita Mikhaylov).

• v20.10.4.1 Changes

  November 13, 2020

  🐛 Bug Fix

  • 🛠 Fix rare silent crashes when query profiler is on and ClickHouse is installed on OS with glibc version that has (supposedly) broken asynchronous unwind tables for some functions. This fixes #15301. This fixes #13098. #16846 (alexey-milovidov).
  • Fix IN operator over several columns and tuples with enabled transform_null_in setting. Fixes #15310. #16722 (Anton Popov).
  • This will fix optimize_read_in_order/optimize_aggregation_in_order with max_threads>0 and expression in ORDER BY. #16637 (Azat Khuzhin).
  • 🚚 Now when parsing AVRO from input the LowCardinality is removed from type. Fixes #16188. #16521 (Mike).
  • Fix rapid growth of metadata when using MySQL Master -> MySQL Slave -> ClickHouse MaterializeMySQL Engine, and slave_parallel_worker enabled on MySQL Slave, by properly shrinking GTID sets. This fixes #15951. #16504 (TCeason).
  • 🛠 Fix DROP TABLE for Distributed (racy with INSERT). #16409 (Azat Khuzhin).
  • 🛠 Fix processing of very large entries in replication queue. Very large entries may appear in ALTER queries if table structure is extremely large (near 1 MB). This fixes #16307. #16332 (alexey-milovidov).
  • 🛠 Fix bug with MySQL database. When MySQL server used as database engine is down some queries raise Exception, because they try to get tables from disabled server, while it's unnecessary. For example, query SELECT ... FROM system.parts should work only with MergeTree tables and don't touch MySQL database at all. #16032 (Kruglov Pavel).

  👌 Improvement

  • ↪ Workaround for use S3 with nginx server as proxy. Nginx currenty does not accept urls with empty path like http://domain.com?delete, but vanilla aws-sdk-cpp produces this kind of urls. This commit uses patched aws-sdk-cpp version, which makes urls with "/" as path in this cases, like http://domain.com/?delete. #16813 (ianton-ru).

• v20.10.3.30 Changes

  October 28, 2020

  Backward Incompatible Change

  • Make multiple_joins_rewriter_version obsolete. Remove first version of joins rewriter. #15472 (Artem Zuikov).
  • Change default value of format_regexp_escaping_rule setting (it's related to Regexp format) to Raw (it means - read whole subpattern as a value) to make the behaviour more like to what users expect. #15426 (alexey-milovidov).
  • Add support for nested multiline comments /* comment /* comment */ */ in SQL. This conforms to the SQL standard. #14655 (alexey-milovidov).
  • Added MergeTree settings (max_replicated_merges_with_ttl_in_queue and max_number_of_merges_with_ttl_in_pool) to control the number of merges with TTL in the background pool and replicated queue. This change breaks compatibility with older versions only if you use delete TTL. Otherwise, replication will stay compatible. You can avoid incompatibility issues if you update all shard replicas at once or execute SYSTEM STOP TTL MERGES until you finish the update of all replicas. If you'll get an incompatible entry in the replication queue, first of all, execute SYSTEM STOP TTL MERGES and after ALTER TABLE ... DETACH PARTITION ... the partition where incompatible TTL merge was assigned. Attach it back on a single replica. #14490 (alesapin).
  • ⚡️ When upgrading from versions older than 20.5, if rolling update is performed and cluster contains both versions 20.5 or greater and less than 20.5, if ClickHouse nodes with old versions are restarted and old version has been started up in presence of newer versions, it may lead to Part ... intersects previous part errors. To prevent this error, first install newer clickhouse-server packages on all cluster nodes and then do restarts (so, when clickhouse-server is restarted, it will start up with the new version).

  🆕 New Feature

  • 🔀 Background data recompression. Add the ability to specify TTL ... RECOMPRESS codec_name for MergeTree table engines family. #14494 (alesapin).
  • ➕ Add parallel quorum inserts. This closes #15601. #15601 (Latysheva Alexandra).
  • Settings for additional enforcement of data durability. Useful for non-replicated setups. #11948 (Anton Popov).
  • When duplicate block is written to replica where it does not exist locally (has not been fetched from replicas), don't ignore it and write locally to achieve the same effect as if it was successfully replicated. #11684 (alexey-milovidov).
  • 👍 Now we support WITH <identifier> AS (subquery) ... to introduce named subqueries in the query context. This closes #2416. This closes #4967. #14771 (Amos Bird).
  • Introduce enable_global_with_statement setting which propagates the first select's WITH statements to other select queries at the same level, and makes aliases in WITH statements visible to subqueries. #15451 (Amos Bird).
  • Secure inter-cluster query execution (with initial_user as current query user). #13156 (Azat Khuzhin). #15551 (Azat Khuzhin).
  • 🚚 Add the ability to remove column properties and table TTLs. Introduced queries ALTER TABLE MODIFY COLUMN col_name REMOVE what_to_remove and ALTER TABLE REMOVE TTL. Both operations are lightweight and executed at the metadata level. #14742 (alesapin).
  • ➕ Added format RawBLOB. It is intended for input or output a single value without any escaping and delimiters. This closes #15349. #15364 (alexey-milovidov).
  • ➕ Add the reinterpretAsUUID function that allows to convert a big-endian byte string to UUID. #15480 (Alexander Kuzmenkov).
  • Implement force_data_skipping_indices setting. #15642 (Azat Khuzhin).
  • Add a setting output_format_pretty_row_numbers to numerate the result in Pretty formats. This closes #15350. #15443 (flynn).
  • ➕ Added query obfuscation tool. It allows to share more queries for better testing. This closes #15268. #15321 (alexey-milovidov).
  • ➕ Add table function null('structure'). #14797 (vxider).
  • ➕ Added formatReadableQuantity function. It is useful for reading big numbers by human. #14725 (Artem Hnilov).
  • ➕ Add format LineAsString that accepts a sequence of lines separated by newlines, every line is parsed as a whole as a single String field. #14703 (Nikita Mikhaylov), #13846 (hexiaoting).
  • ➕ Add JSONStrings format which output data in arrays of strings. #14333 (hcz).
  • ➕ Add support for "Raw" column format for Regexp format. It allows to simply extract subpatterns as a whole without any escaping rules. #15363 (alexey-milovidov).
  • Allow configurable NULL representation for TSV output format. It is controlled by the setting output_format_tsv_null_representation which is \N by default. This closes #9375. Note that the setting only controls output format and \N is the only supported NULL representation for TSV input format. #14586 (Kruglov Pavel).
  • 👌 Support Decimal data type for MaterializedMySQL. MaterializedMySQL is an experimental feature. #14535 (Winter Zhang).
  • ➕ Add new feature: SHOW DATABASES LIKE 'xxx'. #14521 (hexiaoting).
  • ➕ Added a script to import (arbitrary) git repository to ClickHouse as a sample dataset. #14471 (alexey-milovidov).
  • Now insert statements can have asterisk (or variants) with column transformers in the column list. #14453 (Amos Bird).
  • New query complexity limit settings max_rows_to_read_leaf, max_bytes_to_read_leaf for distributed queries to limit max rows/bytes read on the leaf nodes. Limit is applied for local reads only, excluding the final merge stage on the root node. #14221 (Roman Khavronenko).
  • Allow user to specify settings for ReplicatedMergeTree* storage in <replicated_merge_tree> section of config file. It works similarly to <merge_tree> section. For ReplicatedMergeTree* storages settings from <merge_tree> and <replicated_merge_tree> are applied together, but settings from <replicated_merge_tree> has higher priority. Added system.replicated_merge_tree_settings table. #13573 (Amos Bird).
  • ➕ Add mapPopulateSeries function. #13166 (Ildus Kurbangaliev).
  • 👌 Supporting MySQL types: decimal (as ClickHouse Decimal) and datetime with sub-second precision (as DateTime64). #11512 (Vasily Nemkov).
  • Introduce event_time_microseconds field to system.text_log, system.trace_log, system.query_log and system.query_thread_log tables. #14760 (Bharat Nallan).
  • Add event_time_microseconds to system.asynchronous_metric_log & system.metric_log tables. #14514 (Bharat Nallan).
  • Add query_start_time_microseconds field to system.query_log & system.query_thread_log tables. #14252 (Bharat Nallan).

  🐛 Bug Fix

  • 🛠 Fix the case when memory can be overallocated regardless to the limit. This closes #14560. #16206 (alexey-milovidov).
  • 🛠 Fix executable dictionary source hang. In previous versions, when using some formats (e.g. JSONEachRow) data was not feed to a child process before it outputs at least something. This closes #1697. This closes #2455. #14525 (alexey-milovidov).
  • 🛠 Fix double free in case of exception in function dictGet. It could have happened if dictionary was loaded with error. #16429 (Nikolai Kochetov).
  • 🛠 Fix group by with totals/rollup/cube modifers and min/max functions over group by keys. Fixes #16393. #16397 (Anton Popov).
  • Fix async Distributed INSERT with prefer_localhost_replica=0 and internal_replication. #16358 (Azat Khuzhin).
  • 🛠 Fix a very wrong code in TwoLevelStringHashTable implementation, which might lead to memory leak. #16264 (Amos Bird).
  • 🛠 Fix segfault in some cases of wrong aggregation in lambdas. #16082 (Anton Popov).
  • 🛠 Fix ALTER MODIFY ... ORDER BY query hang for ReplicatedVersionedCollapsingMergeTree. This fixes #15980. #16011 (alesapin).
  • 📜 MaterializedMySQL (experimental feature): Fix collate name & charset name parser and support length = 0 for string type. #16008 (Winter Zhang).
  • 👍 Allow to use direct layout for dictionaries with complex keys. #16007 (Anton Popov).
  • Prevent replica hang for 5-10 mins when replication error happens after a period of inactivity. #15987 (filimonov).
  • 🛠 Fix rare segfaults when inserting into or selecting from MaterializedView and concurrently dropping target table (for Atomic database engine). #15984 (tavplubix).
  • 🛠 Fix ambiguity in parsing of settings profiles: CREATE USER ... SETTINGS profile readonly is now considered as using a profile named readonly, not a setting named profile with the readonly constraint. This fixes #15628. #15982 (Vitaly Baranov).
  • MaterializedMySQL (experimental feature): Fix crash on create database failure. #15954 (Winter Zhang).
  • 🛠 Fixed DROP TABLE IF EXISTS failure with Table ... doesn't exist error when table is concurrently renamed (for Atomic database engine). Fixed rare deadlock when concurrently executing some DDL queries with multiple tables (like DROP DATABASE and RENAME TABLE) - Fixed DROP/DETACH DATABASE failure with Table ... doesn't exist when concurrently executing DROP/DETACH TABLE. #15934 (tavplubix).
  • 🛠 Fix incorrect empty result for query from Distributed table if query has WHERE, PREWHERE and GLOBAL IN. Fixes #15792. #15933 (Nikolai Kochetov).
  • 🛠 Fixes #12513: difference expressions with same alias when query is reanalyzed. #15886 (Winter Zhang).
  • 🛠 Fix possible very rare deadlocks in RBAC implementation. #15875 (Vitaly Baranov).
  • 🛠 Fix exception Block structure mismatch in SELECT ... ORDER BY DESC queries which were executed after ALTER MODIFY COLUMN query. Fixes #15800. #15852 (alesapin).
  • MaterializedMySQL (experimental feature): Fix select count() inaccuracy. #15767 (tavplubix).
  • 🛠 Fix some cases of queries, in which only virtual columns are selected. Previously Not found column _nothing in block exception may be thrown. Fixes #12298. #15756 (Anton Popov).
  • 🛠 Fix drop of materialized view with inner table in Atomic database (hangs all subsequent DROP TABLE due to hang of the worker thread, due to recursive DROP TABLE for inner table of MV). #15743 (Azat Khuzhin).
  • 🚚 Possibility to move part to another disk/volume if the first attempt was failed. #15723 (Pavel Kovalenko).
  • 🛠 Fix error Cannot find column which may happen at insertion into MATERIALIZED VIEW in case if query for MV containes ARRAY JOIN. #15717 (Nikolai Kochetov).
  • Fixed too low default value of max_replicated_logs_to_keep setting, which might cause replicas to become lost too often. Improve lost replica recovery process by choosing the most up-to-date replica to clone. Also do not remove old parts from lost replica, detach them instead. #15701 (tavplubix).
  • 🛠 Fix rare race condition in dictionaries and tables from MySQL. #15686 (alesapin).
  • 🛠 Fix (benign) race condition in AMQP-CPP. #15667 (alesapin).
  • 🛠 Fix error Cannot add simple transform to empty Pipe which happened while reading from Buffer table which has different structure than destination table. It was possible if destination table returned empty result for query. Fixes #15529. #15662 (Nikolai Kochetov).
  • 🔀 Proper error handling during insert into MergeTree with S3. MergeTree over S3 is an experimental feature. #15657 (Pavel Kovalenko).
  • 🛠 Fixed bug with S3 table function: region from URL was not applied to S3 client configuration. #15646 (Vladimir Chebotarev).
  • 🛠 Fix the order of destruction for resources in ReadFromStorage step of query plan. It might cause crashes in rare cases. Possibly connected with #15610. #15645 (Nikolai Kochetov).
  • Subtract ReadonlyReplica metric when detach readonly tables. #15592 (sundyli).
  • 🛠 Fixed Element ... is not a constant expression error when using JSON* function result in VALUES, LIMIT or right side of IN operator. #15589 (tavplubix).
  • 👻 Query will finish faster in case of exception. Cancel execution on remote replicas if exception happens. #15578 (Azat Khuzhin).
  • 🛠 Prevent the possibility of error message Could not calculate available disk space (statvfs), errno: 4, strerror: Interrupted system call. This fixes #15541. #15557 (alexey-milovidov).
  • 🛠 Fix Database <db> doesn't exist. in queries with IN and Distributed table when there's no database on initiator. #15538 (Artem Zuikov).
  • 🚚 Mutation might hang waiting for some non-existent part after MOVE or REPLACE PARTITION or, in rare cases, after DETACH or DROP PARTITION. It's fixed. #15537 (tavplubix).
  • 🛠 Fix bug when ILIKE operator stops being case insensitive if LIKE with the same pattern was executed. #15536 (alesapin).
  • 🛠 Fix Missing columns errors when selecting columns which absent in data, but depend on other columns which also absent in data. Fixes #15530. #15532 (alesapin).
  • Throw an error when a single parameter is passed to ReplicatedMergeTree instead of ignoring it. #15516 (nvartolomei).
  • 🛠 Fix bug with event subscription in DDLWorker which rarely may lead to query hangs in ON CLUSTER. Introduced in #13450. #15477 (alesapin).
  • Report proper error when the second argument of boundingRatio aggregate function has a wrong type. #15407 (detailyang).
  • 🛠 Fixes #15365: attach a database with MySQL engine throws exception (no query context). #15384 (Winter Zhang).
  • 🛠 Fix the case of multiple occurrences of column transformers in a select query. #15378 (Amos Bird).
  • 🛠 Fixed compression in S3 storage. #15376 (Vladimir Chebotarev).
  • 🛠 Fix bug where queries like SELECT toStartOfDay(today()) fail complaining about empty time_zone argument. #15319 (Bharat Nallan).
  • 🛠 Fix race condition during MergeTree table rename and background cleanup. #15304 (alesapin).
  • 🛠 Fix rare race condition on server startup when system logs are enabled. #15300 (alesapin).
  • 🛠 Fix hang of queries with a lot of subqueries to same table of MySQL engine. Previously, if there were more than 16 subqueries to same MySQL table in query, it hang forever. #15299 (Anton Popov).
  • 🛠 Fix MSan report in QueryLog. Uninitialized memory can be used for the field memory_usage. #15258 (alexey-milovidov).
  • 🛠 Fix 'Unknown identifier' in GROUP BY when query has JOIN over Merge table. #15242 (Artem Zuikov).
  • 🛠 Fix instance crash when using joinGet with LowCardinality types. This fixes https://github.com/ClickHouse/ClickHouse/issues/15214. #15220 (Amos Bird).
  • 🛠 Fix bug in table engine Buffer which doesn't allow to insert data of new structure into Buffer after ALTER query. Fixes #15117. #15192 (alesapin).
  • Adjust Decimal field size in MySQL column definition packet. #15152 (maqroll).
  • Fixes Data compressed with different methods in join_algorithm='auto'. Keep LowCardinality as type for left table join key in join_algorithm='partial_merge'. #15088 (Artem Zuikov).
  • ⚡️ Update jemalloc to fix percpu_arena with affinity mask. #15035 (Azat Khuzhin). #14957 (Azat Khuzhin).
  • 🛠 We already use padded comparison between String and FixedString (https://github.com/ClickHouse/ClickHouse/blob/master/src/Functions/FunctionsComparison.h#L333). This PR applies the same logic to field comparison which corrects the usage of FixedString as primary keys. This fixes https://github.com/ClickHouse/ClickHouse/issues/14908. #15033 (Amos Bird).
  • If function bar was called with specifically crafted arguments, buffer overflow was possible. This closes #13926. #15028 (alexey-milovidov).
  • 🛠 Fixed Cannot rename ... errno: 22, strerror: Invalid argument error on DDL query execution in Atomic database when running clickhouse-server in Docker on Mac OS. #15024 (tavplubix).
  • 🛠 Fix crash in RIGHT or FULL JOIN with join_algorith='auto' when memory limit exceeded and we should change HashJoin with MergeJoin. #15002 (Artem Zuikov).
  • Now settings number_of_free_entries_in_pool_to_execute_mutation and number_of_free_entries_in_pool_to_lower_max_size_of_merge can be equal to background_pool_size. #14975 (alesapin).
  • 🛠 Fix to make predicate push down work when subquery contains finalizeAggregation function. Fixes #14847. #14937 (filimonov).
  • 🛠 Publish CPU frequencies per logical core in system.asynchronous_metrics. This fixes https://github.com/ClickHouse/ClickHouse/issues/14923. #14924 (Alexander Kuzmenkov).
  • 📇 MaterializedMySQL (experimental feature): Fixed .metadata.tmp File exists error. #14898 (Winter Zhang).
  • 🛠 Fix the issue when some invocations of extractAllGroups function may trigger "Memory limit exceeded" error. This fixes #13383. #14889 (alexey-milovidov).
  • 🛠 Fix SIGSEGV for an attempt to INSERT into StorageFile with file descriptor. #14887 (Azat Khuzhin).
  • 🛠 Fixed segfault in cache dictionary #14837. #14879 (Nikita Mikhaylov).
  • 🛰 MaterializedMySQL (experimental feature): Fixed bug in parsing MySQL binlog events, which causes Attempt to read after eof and Packet payload is not fully read in MaterializeMySQL database engine. #14852 (Winter Zhang).
  • 🛠 Fix rare error in SELECT queries when the queried column has DEFAULT expression which depends on the other column which also has DEFAULT and not present in select query and not exists on disk. Partially fixes #14531. #14845 (alesapin).
  • 🛠 Fix a problem where the server may get stuck on startup while talking to ZooKeeper, if the configuration files have to be fetched from ZK (using the from_zk include option). This fixes #14814. #14843 (Alexander Kuzmenkov).
  • 🛠 Fix wrong monotonicity detection for shrunk Int -> Int cast of signed types. It might lead to incorrect query result. This bug is unveiled in #14513. #14783 (Amos Bird).
  • 👯 Replace column transformer should replace identifiers with cloned ASTs. This fixes https://github.com/ClickHouse/ClickHouse/issues/14695 . #14734 (Amos Bird).
  • 🛠 Fixed missed default database name in metadata of materialized view when executing ALTER ... MODIFY QUERY. #14664 (tavplubix).
  • 🛠 Fix bug when ALTER UPDATE mutation with Nullable column in assignment expression and constant value (like UPDATE x = 42) leads to incorrect value in column or segfault. Fixes #13634, #14045. #14646 (alesapin).
  • 🛠 Fix wrong Decimal multiplication result caused wrong decimal scale of result column. #14603 (Artem Zuikov).
  • 🛠 Fix function has with LowCardinality of Nullable. #14591 (Mike).
  • Cleanup data directory after Zookeeper exceptions during CreateQuery for StorageReplicatedMergeTree Engine. #14563 (Bharat Nallan).
  • 🛠 Fix rare segfaults in functions with combinator -Resample, which could appear in result of overflow with very large parameters. #14562 (Anton Popov).
  • 🛠 Fix a bug when converting Nullable(String) to Enum. Introduced by https://github.com/ClickHouse/ClickHouse/pull/12745. This fixes https://github.com/ClickHouse/ClickHouse/issues/14435. #14530 (Amos Bird).
  • 🛠 Fixed the incorrect sorting order of Nullable column. This fixes #14344. #14495 (Nikita Mikhaylov).
  • 🛠 Fix currentDatabase() function cannot be used in ON CLUSTER ddl query. #14211 (Winter Zhang).
  • 🛰 MaterializedMySQL (experimental feature): Fixed Packet payload is not fully read error in MaterializeMySQL database engine. #14696 (BohuTANG).

  👌 Improvement

  • 0️⃣ Enable Atomic database engine by default for newly created databases. #15003 (tavplubix).
  • ➕ Add the ability to specify specialized codecs like Delta, T64, etc. for columns with subtypes. Implements #12551, fixes #11397, fixes #4609. #15089 (alesapin).
  • Dynamic reload of zookeeper config. #14678 (sundyli).
  • Now it's allowed to execute ALTER ... ON CLUSTER queries regardless of the <internal_replication> setting in cluster config. #16075 (alesapin).
  • 👍 Now joinGet supports multi-key lookup. Continuation of #12418. #13015 (Amos Bird).
  • 🔀 Wait for DROP/DETACH TABLE to actually finish if NO DELAY or SYNC is specified for Atomic database. #15448 (tavplubix).
  • Now it's possible to change the type of version column for VersionedCollapsingMergeTree with ALTER query. #15442 (alesapin).
  • Unfold {database}, {table} and {uuid} macros in zookeeper_path on replicated table creation. Do not allow RENAME TABLE if it may break zookeeper_path after server restart. Fixes #6917. #15348 (tavplubix).
  • The function now allows an argument with timezone. This closes 15264. #15285 (flynn).
  • 🐳 Do not allow connections to ClickHouse server until all scripts in /docker-entrypoint-initdb.d/ are executed. #15244 (Aleksei Kozharin).
  • ➕ Added optimize setting to EXPLAIN PLAN query. If enabled, query plan level optimisations are applied. Enabled by default. #15201 (Nikolai Kochetov).
  • 👻 Proper exception message for wrong number of arguments of CAST. This closes #13992. #15029 (alexey-milovidov).
  • ➕ Add option to disable TTL move on data part insert. #15000 (Pavel Kovalenko).
  • Ignore key constraints when doing mutations. Without this pull request, it's not possible to do mutations when force_index_by_date = 1 or force_primary_key = 1. #14973 (Amos Bird).
  • 👍 Allow to drop Replicated table if previous drop attempt was failed due to ZooKeeper session expiration. This fixes #11891. #14926 (alexey-milovidov).
  • 🛠 Fixed excessive settings constraint violation when running SELECT with SETTINGS from a distributed table. #14876 (Amos Bird).
  • Provide a load_balancing_first_offset query setting to explicitly state what the first replica is. It's used together with FIRST_OR_RANDOM load balancing strategy, which allows to control replicas workload. #14867 (Amos Bird).
  • 👉 Show subqueries for SET and JOIN in EXPLAIN result. #14856 (Nikolai Kochetov).
  • 👍 Allow using multi-volume storage configuration in storage Distributed. #14839 (Pavel Kovalenko).
  • Construct query_start_time and query_start_time_microseconds from the same timespec. #14831 (Bharat Nallan).
  • Support for disabling persistency for StorageJoin and StorageSet, this feature is controlled by setting disable_set_and_join_persistency. And this PR solved issue #6318. #14776 (vxider).
  • Now COLUMNS can be used to wrap over a list of columns and apply column transformers afterwards. #14775 (Amos Bird).
  • ➕ Add merge_algorithm to system.merges table to improve merging inspections. #14705 (Amos Bird).
  • 🛠 Fix potential memory leak caused by zookeeper exists watch. #14693 (hustnn).
  • 👍 Allow parallel execution of distributed DDL. #14684 (Azat Khuzhin).
  • ➕ Add QueryMemoryLimitExceeded event counter. This closes #14589. #14647 (fastio).
  • 🛠 Fix some trailing whitespaces in query formatting. #14595 (Azat Khuzhin).
  • ClickHouse treats partition expr and key expr differently. Partition expr is used to construct an minmax index containing related columns, while primary key expr is stored as an expr. Sometimes user might partition a table at coarser levels, such as partition by i / 1000. However, binary operators are not monotonic and this PR tries to fix that. It might also benifit other use cases. #14513 (Amos Bird).
  • ➕ Add an option to skip access checks for DiskS3. s3 disk is an experimental feature. #14497 (Pavel Kovalenko).
  • Speed up server shutdown process if there are ongoing S3 requests. #14496 (Pavel Kovalenko).
  • 👻 SYSTEM RELOAD CONFIG now throws an exception if failed to reload and continues using the previous users.xml. The background periodic reloading also continues using the previous users.xml if failed to reload. #14492 (Vitaly Baranov).
  • 👍 For INSERTs with inline data in VALUES format in the script mode of clickhouse-client, support semicolon as the data terminator, in addition to the new line. Closes https://github.com/ClickHouse/ClickHouse/issues/12288. #13192 (Alexander Kuzmenkov).
  • 👌 Support custom codecs in compact parts. #12183 (Anton Popov).

  🐎 Performance Improvement

  • 0️⃣ Enable compact parts by default for small parts. This will allow to process frequent inserts slightly more efficiently (4..100 times). #11913 (alexey-milovidov).
  • 👌 Improve quantileTDigest performance. This fixes #2668. #15542 (Kruglov Pavel).
  • Significantly reduce memory usage in AggregatingInOrderTransform/optimize_aggregation_in_order. #15543 (Azat Khuzhin).
  • Faster 256-bit multiplication. #15418 (Artem Zuikov).
  • 👌 Improve performance of 256-bit types using (u)int64_t as base type for wide integers. Original wide integers use 8-bit types as base. #14859 (Artem Zuikov).
  • 🔀 Explicitly use a temporary disk to store vertical merge temporary data. #15639 (Grigory Pervakov).
  • 🌲 Use one S3 DeleteObjects request instead of multiple DeleteObject in a loop. No any functionality changes, so covered by existing tests like integration/test_log_family_s3. #15238 (ianton-ru).
  • 🛠 Fix DateTime <op> DateTime mistakenly choosing the slow generic implementation. This fixes https://github.com/ClickHouse/ClickHouse/issues/15153. #15178 (Amos Bird).
  • 👌 Improve performance of GROUP BY key of type FixedString. #15034 (Amos Bird).
  • Only mlock code segment when starting clickhouse-server. In previous versions, all mapped regions were locked in memory, including debug info. Debug info is usually splitted to a separate file but if it isn't, it led to +2..3 GiB memory usage. #14929 (alexey-milovidov).
  • ClickHouse binary become smaller due to link time optimization.

  🏗 Build/Testing/Packaging Improvement

  • 🏗 Now we use clang-11 for production ClickHouse build. #15239 (alesapin).
  • 🏗 Now we use clang-11 to build ClickHouse in CI. #14846 (alesapin).
  • 🐧 Switch binary builds (Linux, Darwin, AArch64, FreeDSD) to clang-11. #15622 (Ilya Yatsishin).
  • ✅ Now all test images use llvm-symbolizer-11. #15069 (alesapin).
  • 👍 Allow to build with llvm-11. #15366 (alexey-milovidov).
  • Switch from clang-tidy-10 to clang-tidy-11. #14922 (alexey-milovidov).
  • 0️⃣ Use LLVM's experimental pass manager by default. #15608 (Danila Kutenin).
  • 🌐 Don't allow any C++ translation unit to build more than 10 minutes or to use more than 10 GB or memory. This fixes #14925. #15060 (alexey-milovidov).
  • 🐎 Make performance test more stable and representative by splitting test runs and profile runs. #15027 (alexey-milovidov).
  • 🐎 Attempt to make performance test more reliable. It is done by remapping the executable memory of the process on the fly with madvise to use transparent huge pages - it can lower the number of iTLB misses which is the main source of instabilities in performance tests. #14685 (alexey-milovidov).
  • Convert to python3. This closes #14886. #15007 (Azat Khuzhin).
  • ✅ Fail early in functional tests if server failed to respond. This closes #15262. #15267 (alexey-milovidov).
  • 👍 Allow to run AArch64 version of clickhouse-server without configs. This facilitates #15174. #15266 (alexey-milovidov).
  • 👌 Improvements in CI docker images: get rid of ZooKeeper and single script for test configs installation. #15215 (alesapin).
  • 🛠 Fix CMake options forwarding in fast test script. Fixes error in #14711. #15155 (alesapin).
  • ➕ Added a script to perform hardware benchmark in a single command. #15115 (alexey-milovidov).
  • Splitted huge test test_dictionaries_all_layouts_and_sources into smaller ones. #15110 (Nikita Mikhaylov).
  • 🛠 Maybe fix MSan report in base64 (on servers with AVX-512). This fixes #14006. #15030 (alexey-milovidov).
  • ✅ Reformat and cleanup code in all integration test *.py files. #14864 (Bharat Nallan).
  • 🛠 Fix MaterializeMySQL empty transaction unstable test case found in CI. #14854 (Winter Zhang).
  • 🏗 Attempt to speed up build a little. #14808 (alexey-milovidov).
  • 🏗 Speed up build a little by removing unused headers. #14714 (alexey-milovidov).
  • 🛠 Fix build failure in OSX. #14761 (Winter Zhang).
  • 0️⃣ Enable ccache by default in cmake if it's found in OS. #14575 (alesapin).
  • 🏗 Control CI builds configuration from the ClickHouse repository. #14547 (alesapin).
  • 🚚 In CMake files: - Moved some options' descriptions' parts to comments above. - Replace 0 -> OFF, 1 -> ON in options default values. - Added some descriptions and links to docs to the options. - Replaced FUZZER option (there is another option ENABLE_FUZZING which also enables same functionality). - Removed ENABLE_GTEST_LIBRARY option as there is ENABLE_TESTS. See the full description in PR: #14711 (Mike).
  • 👉 Make binary a bit smaller (~50 Mb for debug version). #14555 (Artem Zuikov).
  • 👉 Use std::filesystem::path in ConfigProcessor for concatenating file paths. #14558 (Bharat Nallan).
  • 🛠 Fix debug assertion in bitShiftLeft() when called with negative big integer. #14697 (Artem Zuikov).

• v20.9.5.5 Changes

  November 13, 2020

  🐛 Bug Fix

  • 🛠 Fix rare silent crashes when query profiler is on and ClickHouse is installed on OS with glibc version that has (supposedly) broken asynchronous unwind tables for some functions. This fixes #15301. This fixes #13098. #16846 (alexey-milovidov).
  • 🚚 Now when parsing AVRO from input the LowCardinality is removed from type. Fixes #16188. #16521 (Mike).
  • Fix rapid growth of metadata when using MySQL Master -> MySQL Slave -> ClickHouse MaterializeMySQL Engine, and slave_parallel_worker enabled on MySQL Slave, by properly shrinking GTID sets. This fixes #15951. #16504 (TCeason).
  • 🛠 Fix DROP TABLE for Distributed (racy with INSERT). #16409 (Azat Khuzhin).
  • 🛠 Fix processing of very large entries in replication queue. Very large entries may appear in ALTER queries if table structure is extremely large (near 1 MB). This fixes #16307. #16332 (alexey-milovidov).
  • 🛠 Fixed the inconsistent behaviour when a part of return data could be dropped because the set for its filtration wasn't created. #16308 (Nikita Mikhaylov).
  • 🛠 Fix bug with MySQL database. When MySQL server used as database engine is down some queries raise Exception, because they try to get tables from disabled server, while it's unnecessary. For example, query SELECT ... FROM system.parts should work only with MergeTree tables and don't touch MySQL database at all. #16032 (Kruglov Pavel).

• v20.9.4.76 Changes

  October 29, 2020

  🐛 Bug Fix

  • 🛠 Fix double free in case of exception in function dictGet. It could have happened if dictionary was loaded with error. #16429 (Nikolai Kochetov).
  • 🛠 Fix group by with totals/rollup/cube modifers and min/max functions over group by keys. Fixes #16393. #16397 (Anton Popov).
  • Fix async Distributed INSERT w/ prefer_localhost_replica=0 and internal_replication. #16358 (Azat Khuzhin).
  • 🛠 Fix a very wrong code in TwoLevelStringHashTable implementation, which might lead to memory leak. I'm suprised how this bug can lurk for so long.... #16264 (Amos Bird).
  • 🛠 Fix the case when memory can be overallocated regardless to the limit. This closes #14560. #16206 (alexey-milovidov).
  • 🛠 Fix ALTER MODIFY ... ORDER BY query hang for ReplicatedVersionedCollapsingMergeTree. This fixes #15980. #16011 (alesapin).
  • 🛠 Fix collate name & charset name parser and support length = 0 for string type. #16008 (Winter Zhang).
  • 👍 Allow to use direct layout for dictionaries with complex keys. #16007 (Anton Popov).
  • Prevent replica hang for 5-10 mins when replication error happens after a period of inactivity. #15987 (filimonov).
  • 🛠 Fix rare segfaults when inserting into or selecting from MaterializedView and concurrently dropping target table (for Atomic database engine). #15984 (tavplubix).
  • 🛠 Fix ambiguity in parsing of settings profiles: CREATE USER ... SETTINGS profile readonly is now considered as using a profile named readonly, not a setting named profile with the readonly constraint. This fixes https://github.com/ClickHouse/ClickHouse/issues/15628. #15982 (Vitaly Baranov).
  • 🛠 Fix a crash when database creation fails. #15954 (Winter Zhang).
  • 🛠 Fixed DROP TABLE IF EXISTS failure with Table ... doesn't exist error when table is concurrently renamed (for Atomic database engine). Fixed rare deadlock when concurrently executing some DDL queries with multiple tables (like DROP DATABASE and RENAME TABLE) Fixed DROP/DETACH DATABASE failure with Table ... doesn't exist when concurrently executing DROP/DETACH TABLE. #15934 (tavplubix).
  • 🛠 Fix incorrect empty result for query from Distributed table if query has WHERE, PREWHERE and GLOBAL IN. Fixes #15792. #15933 (Nikolai Kochetov).
  • 🛠 Fix possible deadlocks in RBAC. #15875 (Vitaly Baranov).
  • 🛠 Fix exception Block structure mismatch in SELECT ... ORDER BY DESC queries which were executed after ALTER MODIFY COLUMN query. Fixes #15800. #15852 (alesapin).
  • 🛠 Fix select count() inaccuracy for MaterializeMySQL. #15767 (tavplubix).
  • 🛠 Fix some cases of queries, in which only virtual columns are selected. Previously Not found column _nothing in block exception may be thrown. Fixes #12298. #15756 (Anton Popov).
  • Fixed too low default value of max_replicated_logs_to_keep setting, which might cause replicas to become lost too often. Improve lost replica recovery process by choosing the most up-to-date replica to clone. Also do not remove old parts from lost replica, detach them instead. #15701 (tavplubix).
  • 🛠 Fix error Cannot add simple transform to empty Pipe which happened while reading from Buffer table which has different structure than destination table. It was possible if destination table returned empty result for query. Fixes #15529. #15662 (Nikolai Kochetov).
  • 🛠 Fixed bug with globs in S3 table function, region from URL was not applied to S3 client configuration. #15646 (Vladimir Chebotarev).
  • 🛠 Decrement the ReadonlyReplica metric when detaching read-only tables. This fixes https://github.com/ClickHouse/ClickHouse/issues/15598. #15592 (sundyli).
  • Throw an error when a single parameter is passed to ReplicatedMergeTree instead of ignoring it. #15516 (nvartolomei).

  👌 Improvement

  • Now it's allowed to execute ALTER ... ON CLUSTER queries regardless of the <internal_replication> setting in cluster config. #16075 (alesapin).
  • Unfold {database}, {table} and {uuid} macros in ReplicatedMergeTree arguments on table creation. #16160 (tavplubix).

• v20.9.3.45 Changes

  October 09, 2020

  🐛 Bug Fix

  • 🛠 Fix error Cannot find column which may happen at insertion into MATERIALIZED VIEW in case if query for MV containes ARRAY JOIN. #15717 (Nikolai Kochetov).
  • 🛠 Fix race condition in AMQP-CPP. #15667 (alesapin).
  • 🛠 Fix the order of destruction for resources in ReadFromStorage step of query plan. It might cause crashes in rare cases. Possibly connected with #15610. #15645 (Nikolai Kochetov).
  • 🛠 Fixed Element ... is not a constant expression error when using JSON* function result in VALUES, LIMIT or right side of IN operator. #15589 (tavplubix).
  • 🛠 Prevent the possibility of error message Could not calculate available disk space (statvfs), errno: 4, strerror: Interrupted system call. This fixes #15541. #15557 (alexey-milovidov).
  • Significantly reduce memory usage in AggregatingInOrderTransform/optimize_aggregation_in_order. #15543 (Azat Khuzhin).
  • 🚚 Mutation might hang waiting for some non-existent part after MOVE or REPLACE PARTITION or, in rare cases, after DETACH or DROP PARTITION. It's fixed. #15537 (tavplubix).
  • 🛠 Fix bug when ILIKE operator stops being case insensitive if LIKE with the same pattern was executed. #15536 (alesapin).
  • 🛠 Fix Missing columns errors when selecting columns which absent in data, but depend on other columns which also absent in data. Fixes #15530. #15532 (alesapin).
  • 🛠 Fix bug with event subscription in DDLWorker which rarely may lead to query hangs in ON CLUSTER. Introduced in #13450. #15477 (alesapin).
  • Report proper error when the second argument of boundingRatio aggregate function has a wrong type. #15407 (detailyang).
  • 🛠 Fix bug where queries like SELECT toStartOfDay(today()) fail complaining about empty time_zone argument. #15319 (Bharat Nallan).
  • 🛠 Fix race condition during MergeTree table rename and background cleanup. #15304 (alesapin).
  • 🛠 Fix rare race condition on server startup when system.logs are enabled. #15300 (alesapin).
  • 🛠 Fix MSan report in QueryLog. Uninitialized memory can be used for the field memory_usage. #15258 (alexey-milovidov).
  • 🛠 Fix instance crash when using joinGet with LowCardinality types. This fixes https://github.com/ClickHouse/ClickHouse/issues/15214. #15220 (Amos Bird).
  • 🛠 Fix bug in table engine Buffer which doesn't allow to insert data of new structure into Buffer after ALTER query. Fixes #15117. #15192 (alesapin).
  • Adjust decimals field size in mysql column definition packet. #15152 (maqroll).
  • 🛠 Fixed Cannot rename ... errno: 22, strerror: Invalid argument error on DDL query execution in Atomic database when running clickhouse-server in docker on Mac OS. #15024 (tavplubix).
  • 🛠 Fix to make predicate push down work when subquery contains finalizeAggregation function. Fixes #14847. #14937 (filimonov).
  • 🛠 Fix a problem where the server may get stuck on startup while talking to ZooKeeper, if the configuration files have to be fetched from ZK (using the from_zk include option). This fixes #14814. #14843 (Alexander Kuzmenkov).

  👌 Improvement

  • Now it's possible to change the type of version column for VersionedCollapsingMergeTree with ALTER query. #15442 (alesapin).

• « First
• ‹ Prev
• 1
• 2
• 3
• 4
• 5
• 6
• Next ›
• Last »