ClickHouse v21.11

« Changelog History

ClickHouse v21.11 Release Notes

• Backward Incompatible Change

  • 🔄 Change order of json_path and json arguments in SQL/JSON functions (to be consistent with the standard). Closes #30449. #30474 (Kseniia Sumarokova).
  • Remove MergeTree table setting write_final_mark. It will be always true. #30455 (Kseniia Sumarokova). No actions required, all tables are compatible with the new version.
  • 🚚 Function bayesAB is removed. Please help to return this function back, refreshed. This closes #26233. #29934 (alexey-milovidov).
  • This is relevant only if you already started using the experimental clickhouse-keeper support. Now ClickHouse Keeper snapshots compressed with ZSTD codec by default instead of custom ClickHouse LZ4 block compression. This behavior can be turned off with compress_snapshots_with_zstd_format coordination setting (must be equal on all quorum replicas). Backward incompatibility is quite rare and may happen only when new node will send snapshot (happens in case of recovery) to the old node which is unable to read snapshots in ZSTD format. #29417 (alesapin).

  🆕 New Feature

  • New asynchronous INSERT mode allows to accumulate inserted data and store it in a single batch in background. On client it can be enabled by setting async_insert for INSERT queries with data inlined in query or in separate buffer (e.g. for INSERT queries via HTTP protocol). If wait_for_async_insert is true (by default) the client will wait until data will be flushed to table. On server-side it controlled by the settings async_insert_threads, async_insert_max_data_size and async_insert_busy_timeout_ms. Implements #18282. #27537 (Anton Popov). #20557 (Ivan). Notes on performance: with asynchronous inserts you can do up to around 10 000 individual INSERT queries per second, so it is still recommended to insert in batches if you want to achieve performance up to millions inserted rows per second.
  • ➕ Add interactive mode for clickhouse-local. So, you can just run clickhouse-local to get a command line ClickHouse interface without connecting to a server and process data from files and external data sources. Also merge the code of clickhouse-client and clickhouse-local together. Closes #7203. Closes #25516. Closes #22401. #26231 (Kseniia Sumarokova).
  • ➕ Added support for executable (scriptable) user defined functions. These are UDFs that can be written in any programming language. #28803 (Maksim Kita).
  • 👍 Allow predefined connections to external data sources. This allows to avoid specifying credentials or addresses while using external data sources, they can be referenced by names instead. Closes #28367. #28577 (Kseniia Sumarokova).
  • ➕ Added INFORMATION_SCHEMA database with SCHEMATA, TABLES, VIEWS and COLUMNS views to the corresponding tables in system database. Closes #9770. #28691 (tavplubix).
  • 👌 Support EXISTS (subquery). Closes #6852. #29731 (Kseniia Sumarokova).
  • 🌲 Session logging for audit. Logging all successful and failed login and logout events to a new system.session_log table. #22415 (Vasily Nemkov) (Vitaly Baranov).
  • 👌 Support multidimensional cosine distance and euclidean distance functions; L1, L2, Lp, Linf distances and norms. Scalar product on tuples and various arithmetic operators on tuples. This fully closes #4509 and even more. #27933 (Alexey Boykov).
  • ➕ Add support for compression and decompression for INTO OUTFILE and FROM INFILE (with autodetect or with additional optional parameter). #27135 (Filatenkov Artur).
  • ➕ Add CORS (Cross Origin Resource Sharing) support with HTTP OPTIONS request. It means, now Grafana will work with serverless requests without a kludges. Closes #18693. #29155 (Filatenkov Artur).
  • 👍 Queries with JOIN ON now supports disjunctions (OR). #21320 (Ilya Golshtein).
  • ➕ Added function tokens. That allow to split string into tokens using non-alpha numeric ASCII characters as separators. #29981 (Maksim Kita). Added function ngrams to extract ngrams from text. Closes #29699. #29738 (Maksim Kita).
  • ➕ Add functions for Unicode normalization: normalizeUTF8NFC, normalizeUTF8NFD, normalizeUTF8NFKC, normalizeUTF8NFKD functions. #28633 (darkkeks).
  • 🔊 Streaming consumption of application log files in ClickHouse with FileLog table engine. It's like Kafka or RabbitMQ engine but for append-only and rotated logs in local filesystem. Closes #6953. #25969 (flynn) (Kseniia Sumarokova).
  • ➕ Add CapnProto output format, refactor CapnProto input format. #29291 (Kruglov Pavel).
  • 👍 Allow to write number in query as binary literal. Example SELECT 0b001;. #29304 (Maksim Kita).
  • ➕ Added hashed_array dictionary type. It saves memory when using dictionaries with multiple attributes. Closes #30236. #30242 (Maksim Kita).
  • ➕ Added JSONExtractKeys function. #30056 (Vitaly).
  • ➕ Add a function getOSKernelVersion - it returns a string with OS kernel version. #29755 (Memo).
  • ➕ Added MD4 and SHA384 functions. MD4 is an obsolete and insecure hash function, it can be used only in rare cases when MD4 is already being used in some legacy system and you need to get exactly the same result. #29602 (Nikita Tikhomirov).
  • HSTS can be enabled for Clickhouse HTTP server by setting hsts_max_age in configuration file with a positive number. #29516 (凌涛).
  • 👍 Huawei OBS Storage support. Closes #24294. #29511 (kevin wan).
  • 🆕 New function mapContainsKeyLike to get the map that key matches a simple regular expression. #29471 (凌涛). New function mapExtractKeyLike to get the map only kept elements matched specified pattern. #30793 (凌涛).
  • Implemented ALTER TABLE x MODIFY COMMENT. #29264 (Vasily Nemkov).
  • ➕ Adds H3 inspection functions that are missing from ClickHouse but are available via the H3 api: https://h3geo.org/docs/api/inspection. #29209 (Bharat Nallan).
  • 👍 Allow non-replicated ALTER TABLE FETCH and ATTACH in Replicated databases. #29202 (Kevin Michel).
  • Added a setting output_format_csv_null_representation: This is the same as output_format_tsv_null_representation but is for CSV output. #29123 (PHO).
  • ➕ Added function zookeeperSessionUptime() which returns uptime of current ZooKeeper session in seconds. #28983 (tavplubix).
  • Implements the h3ToGeoBoundary function. #28952 (Ivan Veselov).
  • ➕ Add aggregate function exponentialMovingAverage that can be used as window function. This closes #27511. #28914 (alexey-milovidov).
  • Allow to include subcolumns of table columns into DESCRIBE query result (can be enabled by setting describe_include_subcolumns). #28905 (Anton Popov).
  • Executable, ExecutablePool added option send_chunk_header. If this option is true then chunk rows_count with line break will be sent to client before chunk. #28833 (Maksim Kita).
  • 🛠 tokenbf_v1 and ngram support Map with key of String of FixedSring type. It enhance data skipping in query with map key filter. sql CREATE TABLE map_tokenbf ( row_id UInt32, map Map(String, String), INDEX map_tokenbf map TYPE ngrambf_v1(4,256,2,0) GRANULARITY 1 ) Engine=MergeTree() Order by id With table above, the query select * from map_tokebf where map['K']='V' will skip the granule that doesn't contain key A . Of course, how many rows will skipped is depended on the granularity and index_granularity you set. #28511 (凌涛).
  • Send profile events from server to client. New packet type ProfileEvents was introduced. Closes #26177. #28364 (Dmitry Novik).
  • 🛠 Bit shift operations for FixedString and String data types. This closes #27763. #28325 (小路).
  • 👌 Support adding / deleting tables to replication from PostgreSQL dynamically in database engine MaterializedPostgreSQL. Support alter for database settings. Closes #27573. #28301 (Kseniia Sumarokova).
  • ➕ Added function accurateCastOrDefault(x, T). Closes #21330. Authors @taiyang-li. #23028 (Maksim Kita).
  • ➕ Add Function toUUIDOrDefault, toUInt8/16/32/64/256OrDefault, toInt8/16/32/64/128/256OrDefault, which enables user defining default value(not null) when string parsing is failed. #21330 (taiyang-li).

  🐎 Performance Improvement

  • 🔀 Background merges can be preempted by each other and they are scheduled with appropriate priorities. Now long running merges won't prevent short merges to proceed. This is needed for a better scheduling and controlling of merges execution. It reduces the chances to get "too many parts" error. #22381. #25165 (Nikita Mikhaylov). Added an ability to execute more merges and mutations than the number of threads in background pool. Merges and mutations will be executed step by step according to their sizes (lower is more prioritized). The ratio of the number of tasks to threads to execute is controlled by a setting background_merges_mutations_concurrency_ratio, 2 by default. #29140 (Nikita Mikhaylov).
  • 👍 Allow to use asynchronous reads for remote filesystems. Lower the number of seeks while reading from remote filesystems. It improves performance tremendously and makes the experimental web and s3 disks to work faster than EBS under certain conditions. #29205 (Kseniia Sumarokova). In the meantime, the web disk type (static dataset hosted on a web server) is graduated from being experimental to be production ready.
  • Queries with INTO OUTFILE in clickhouse-client will use multiple threads. Fix the issue with flickering progress-bar when using INTO OUTFILE. This closes #30873. This closes #30872. #30886 (alexey-milovidov).
  • ⬇️ Reduce amount of redundant compressed data read from disk for some types SELECT queries (only for MergeTree engines family). #30111 (alesapin).
  • ✂ Remove some redundant seek calls while reading compressed blocks in MergeTree table engines family. #29766 (alesapin).
  • 👉 Make url table function to process multiple URLs in parallel. This closes #29670 and closes #29671. #29673 (alexey-milovidov).
  • Improve performance of aggregation in order of primary key (with enabled setting optimize_aggregation_in_order). #30266 (Anton Popov).
  • Now clickhouse is using DNS cache while communicating with external S3. #29999 (alesapin).
  • ➕ Add support for pushdown of IS NULL/IS NOT NULL to external databases (i.e. MySQL). #29463 (Azat Khuzhin). Transform isNull/isNotNull to IS NULL/IS NOT NULL (for external dbs, i.e. MySQL). #29446 (Azat Khuzhin).
  • SELECT queries from Dictionary tables will use multiple threads. #30500 (Maksim Kita).
  • 👌 Improve performance for filtering (WHERE operation) of Decimal columns. #30431 (Jun Jin).
  • ✂ Remove branchy code in filter operation with a better implementation with popcnt/ctz which have better performance. #29881 (Jun Jin).
  • 👌 Improve filter bytemask generator (used for WHERE operator) function all in one with SSE/AVX2/AVX512 instructions. Note that by default ClickHouse is only using SSE, so it's only relevant for custom builds. #30014 (jasperzhu). #30670 (jasperzhu).
  • 👌 Improve the performance of SUM aggregate function of Nullable floating point numbers. #28906 (Raúl Marín).
  • Speed up part loading process with multiple disks are in use. The idea is similar to https://github.com/ClickHouse/ClickHouse/pull/16423 . Prod env shows improvement: 24 min -> 16 min . #28363 (Amos Bird).
  • ⬇️ Reduce default settings for S3 multipart upload part size to lower memory usage. #28679 (ianton-ru).
  • Speed up bitmapAnd function. #28332 (dddounaiking).
  • ✂ Removed sub-optimal mutation notifications in StorageMergeTree when merges are still going. #27552 (Vladimir Chebotarev).
  • 🐎 Attempt to improve performance of string comparison. #28767 (alexey-milovidov).
  • Primary key index and partition filter can work in tuple. #29281 (凌涛).
  • If query has multiple quantile aggregate functions with the same arguments but different level parameter, they will be fused together and executed in one pass if the setting optimize_syntax_fuse_functions is enabled. #26657 (hexiaoting).
  • ⚡️ Now min-max aggregation over the first expression of primary key is optimized by projection. This is for #329. #29918 (Amos Bird).

  Experimental Feature

  • ➕ Add ability to change nodes configuration (in .xml file) for ClickHouse Keeper. #30372 (alesapin).
  • ➕ Add sparkbar aggregate function. This closes #26175. #27481 (小路). Note: there is one flaw in this function, the behaviour will be changed in future releases.

  👌 Improvement

  • 👍 Allow user to change log levels without restart. #29586 (Nikolay Degterinsky).
  • ✅ Multiple improvements for SQL UDF. Queries for manipulation of SQL User Defined Functions now support ON CLUSTER clause. Example CREATE FUNCTION test_function ON CLUSTER 'cluster' AS x -> x + 1;. Closes #30666. #30734 (Maksim Kita). Support CREATE OR REPLACE, CREATE IF NOT EXISTS syntaxes. #30454 (Maksim Kita). Added DROP IF EXISTS support. Example DROP FUNCTION IF EXISTS test_function. #30437 (Maksim Kita). Support lambdas. Example CREATE FUNCTION lambda_function AS x -> arrayMap(element -> element * 2, x);. #30435 (Maksim Kita). Support SQL user defined functions for clickhouse-local. #30179 (Maksim Kita).
  • Enable per-query memory profiler (set to memory_profiler_step = 4MiB) globally. #29455 (Azat Khuzhin).
  • Added columns data_compressed_bytes, data_uncompressed_bytes, marks_bytes into system.data_skipping_indices. Added columns secondary_indices_compressed_bytes, secondary_indices_uncompressed_bytes, secondary_indices_marks_bytes into system.parts. Closes #29697. #29896 (Maksim Kita).
  • ➕ Add table alias to system.tables and database alias to system.databases #29677. #29882 (kevin wan).
  • Correctly resolve interdependencies between tables on server startup. Closes #8004, closes #15170. #28373 (tavplubix).
  • Avoid error "Division by zero" when denominator is Nullable in functions divide, intDiv and modulo. Closes #22621. #28352 (Kruglov Pavel).
  • 👍 Allow to parse values of Date data type in text formats as YYYYMMDD in addition to YYYY-MM-DD. This closes #30870. #30871 (alexey-milovidov).
  • 💻 Web UI: render bars in table cells. #29792 (alexey-milovidov).
  • 👉 User can now create dictionaries with comments: CREATE DICTIONARY ... COMMENT 'vaue' ... #29899 (Vasily Nemkov). Users now can set comments to database in CREATE DATABASE statement ... #29429 (Vasily Nemkov).
  • Introduce compiled_expression_cache_elements_size setting. If you will ever want to use this setting, you will already know what it does. #30667 (Maksim Kita).
  • 👍 clickhouse-format now supports option --query. In previous versions you have to pass the query to stdin. #29325 (凌涛).
  • 👌 Support ALTER TABLE for tables in Memory databases. Memory databases are used in clickhouse-local. #30866 (tavplubix).
  • 👍 Arrays of all serializable types are now supported by arrayStringConcat. #30840 (Nickita Taranov).
  • 👀 ClickHouse now will account docker/cgroups limitations to get system memory amount. See #25662. #30574 (Pavel Medvedev).
  • Fetched table structure for PostgreSQL database is more reliable now. #30477 (Kseniia Sumarokova).
  • 👍 Full support of positional arguments in GROUP BY and ORDER BY. #30433 (Kseniia Sumarokova).
  • 👍 Allow extracting non-string element as string using JSONExtractString. This is for pull/25452#issuecomment-927123287. #30426 (Amos Bird).
  • ➕ Added an ability to use FINAL clause in SELECT queries from GraphiteMergeTree. #30360 (Nikita Mikhaylov).
  • Minor improvements in replica cloning and enqueuing fetch for broken parts, that should avoid extremely rare hanging of GET_PART entries in replication queue. #30346 (tavplubix).
  • 👍 Allow symlinks to files in user_files directory for file table function. #30309 (Kseniia Sumarokova).
  • 🛠 Fixed comparison of Date32 with Date, DateTime, DateTime64 and String. #30219 (liang.huang).
  • 👍 Allow to remove SAMPLE BY expression from MergeTree tables (ALTER TABLE <table> REMOVE SAMPLE BY). #30180 (Anton Popov).
  • Now Keeper (as part of clickhouse-server) will start asynchronously if it can connect to some other node. #30170 (alesapin).
  • 👍 Now clickhouse-client supports native multi-line editing. #30143 (Amos Bird).
  • polygon dictionaries (reverse geocoding): added support for reading the dictionary content with SELECT query method if setting store_polygon_key_column = true. Closes #30090. #30142 (Maksim Kita).
  • ➕ Add ClickHouse logo to Play UI. #29674 (alexey-milovidov).
  • 👍 Better exception message while reading column from Arrow-supported formats like Arrow, ArrowStream, Parquet and ORC. This closes #29926. #29927 (alexey-milovidov).
  • 🛠 Fix data-race between flush and startup in Buffer tables. This can appear in tests. #29930 (Azat Khuzhin).
  • 🛠 Fix lock-order-inversion between DROP TABLE for DatabaseMemory and LiveView. Live View is an experimental feature. Memory database is used in clickhouse-local. #29929 (Azat Khuzhin).
  • 🛠 Fix lock-order-inversion between periodic dictionary reload and config reload. #29928 (Azat Khuzhin).
  • ⚡️ Update zoneinfo files to 2021c. #29925 (alexey-milovidov).
  • ➕ Add ability to configure retries and delays between them for clickhouse-copier. #29921 (Azat Khuzhin).
  • Add shutdown_wait_unfinished_queries server setting to allowing waiting for running queries up to shutdown_wait_unfinished time. This is for #24451. #29914 (Amos Bird).
  • Add ability to trace peak memory usage (with new trace_type in system.trace_log - MemoryPeak). #29858 (Azat Khuzhin).
  • PostgreSQL foreign tables: Added partitioned table prefix 'p' for the query for fetching replica identity index. #29828 (Shoh Jahon).
  • Apply max_untracked_memory/memory_profiler_step/memory_profiler_sample_probability during mutate/merge to profile memory usage during merges. #29681 (Azat Khuzhin).
  • Query obfuscator: clickhouse-format --obfuscate now works with more types of queries. #29672 (alexey-milovidov).
  • 🛠 Fixed the issue: clickhouse-format --obfuscate cannot process queries with embedded dictionaries (functions regionTo...). #29667 (alexey-milovidov).
  • 🛠 Fix incorrect Nullable processing of JSON functions. This fixes #29615 . Mark as improvement because https://github.com/ClickHouse/ClickHouse/pull/28012 is not released. #29659 (Amos Bird).
  • 🐧 Increase listen_backlog by default (to match default in newer linux kernel). #29643 (Azat Khuzhin).
  • Reload dictionaries, models, user defined executable functions if servers config dictionaries_config, models_config, user_defined_executable_functions_config changes. Closes #28142. #29529 (Maksim Kita).
  • Get rid of pointless restriction on projection name. Now projection name can start with tmp_. #29520 (Amos Bird).
  • Fixed There is no query or query context has expired error in mutations with nested subqueries. Do not allow subqueries in mutation if table is replicated and allow_nondeterministic_mutations setting is disabled. #29495 (tavplubix).
  • Apply config changes to max_concurrent_queries during runtime (no need to restart). #29414 (Raúl Marín).
  • Added setting use_skip_indexes. #29405 (Maksim Kita).
  • ➕ Add support for FREEZEing in-memory parts (for backups). #29376 (Mo Xuan).
  • Pass through initial query_id for clickhouse-benchmark (previously if you run remote query via clickhouse-benchmark, queries on shards will not be linked to the initial query via initial_query_id). #29364 (Azat Khuzhin).
  • Skip indexes tokenbf_v1 and ngrambf_v1: added support for Array data type with key of String of FixedString type. #29280 (Maksim Kita). Skip indexes tokenbf_v1 and ngrambf_v1 added support for Map data type with key of String of FixedString type. Author @lingtaolf. #29220 (Maksim Kita).
  • 👍 Function has: added support for Map data type. #29267 (Maksim Kita).
  • ➕ Add compress_logs settings for clickhouse-keeper which allow to compress clickhouse-keeper logs (for replicated state machine) in ZSTD . Implements: #26977. #29223 (alesapin).
  • Add a setting external_table_strict_query - it will force passing the whole WHERE expression in queries to foreign databases even if it is incompatible. #29206 (Azat Khuzhin).
  • Disable projections when ARRAY JOIN is used. In previous versions projection analysis may break aliases in array join. #29139 (Amos Bird).
  • 👌 Support more types in MsgPack input/output format. #29077 (Kruglov Pavel).
  • 👍 Allow to input and output LowCardinality columns in ORC input/output format. #29062 (Kruglov Pavel).
  • Select from system.distributed_ddl_queue might show incorrect values, it's fixed. #29061 (tavplubix).
  • Correct behaviour with unknown methods for HTTP connection. Solves #29050. #29057 (Filatenkov Artur).
  • 🔊 clickhouse-keeper: Fix bug in clickhouse-keeper-converter which can lead to some data loss while restoring from ZooKeeper logs (not snapshot). #29030 (小路). Fix bug in clickhouse-keeper-converter which can lead to incorrect ZooKeeper log deserialization. #29071 (小路).
  • 🛠 Apply settings from CREATE ... AS SELECT queries (fixes: #28810). #28962 (Azat Khuzhin).
  • 🚚 Respect default database setting for ALTER TABLE ... ON CLUSTER ... REPLACE/MOVE PARTITION FROM/TO ... #28955 (anneji-dev).
  • gRPC protocol: Allow change server-side compression from client. #28953 (Vitaly Baranov).
  • 👻 Skip "no data" exception when reading thermal sensors for asynchronous metrics. This closes #28852. #28882 (alexey-milovidov).
  • 🛠 Fixed logical race condition that might cause Dictionary not found error for existing dictionary in rare cases. #28853 (tavplubix).
  • 😌 Relax nested function for If-combinator check (but forbid nested identical combinators). #28828 (Azat Khuzhin).
  • 🛠 Fix possible uncaught exception during server termination. #28761 (Azat Khuzhin).
  • 🔀 Forbid cleaning of tmp directories that can be used by an active mutation/merge if mutation/merge is extraordinarily long. #28760 (Azat Khuzhin).
  • Allow optimization optimize_arithmetic_operations_in_aggregate_functions = 1 when alias is used. #28746 (Amos Bird).
  • Implement detach_not_byte_identical_parts setting for ReplicatedMergeTree, that will detach instead of remove not byte-identical parts (after mege/mutate). #28708 (Azat Khuzhin).
  • Implement max_suspicious_broken_parts_bytes setting for MergeTree (to limit total size of all broken parts, default is 1GiB). #28707 (Azat Khuzhin).
  • Enable expanding macros in RabbitMQ table settings. #28683 (Vitaly Baranov).
  • ⏪ Restore the possibility to read data of a table using the Log engine in multiple threads. #28125 (Vitaly Baranov).
  • 🛠 Fix misbehavior of NULL column handling in JSON functions. This fixes #27930. #28012 (Amos Bird).
  • 👍 Allow to set the size of Mark/Uncompressed cache for skip indices separately from columns. #27961 (Amos Bird).
  • 👍 Allow to mix JOIN with USING with other JOIN types. #23881 (darkkeks).
  • ⚡️ Update aws-sdk submodule for throttling in Yandex Cloud S3. #30646 (ianton-ru).
  • 🛠 Fix releasing query ID and session ID at the end of query processing while handing gRPC call. #29954 (Vitaly Baranov).
  • 🛠 Fix shutdown of AccessControlManager to fix flaky test. #29951 (Vitaly Baranov).
  • 🛠 Fix failed assertion in reading from HDFS. Update libhdfs3 library to be able to run in tests in debug. Closes #29251. Closes #27814. #29276 (Kseniia Sumarokova).

  🏗 Build/Testing/Packaging Improvement

  • ➕ Add support for FreeBSD builds for Aarch64 machines. #29952 (MikaelUrankar).
  • Recursive submodules are no longer needed for ClickHouse. #30315 (alexey-milovidov).
  • 🏗 ClickHouse can be statically built with Musl. This is added as experiment, it does not support building odbc-bridge, library-bridge, integration with CatBoost and some libraries. #30248 (alexey-milovidov).
  • 🍎 Enable Protobuf, Arrow, ORC, Parquet for AArch64 and Darwin (macOS) builds. This closes #29248. This closes #28018. #30015 (alexey-milovidov).
  • ➕ Add cross-build for PowerPC (powerpc64le). This closes #9589. Enable support for interaction with MySQL for AArch64 and PowerPC. This closes #26301. #30010 (alexey-milovidov).
  • Leave only required files in cross-compile toolchains. Include them as submodules (earlier they were downloaded as tarballs). #29974 (alexey-milovidov).
  • 📜 Implemented structure-aware fuzzing approach in ClickHouse for select statement parser. #30012 (Paul).
  • Turning on experimental constexpr expressions evaluator for clang to speed up template code compilation. #29668 (myrrc).
  • ➕ Add ability to compile using newer version fo glibc without using new symbols. #29594 (Azat Khuzhin).
  • ⬇️ Reduce Debug build binary size by clang optimization option. #28736 (flynn).
  • 👷 Now all images for CI will be placed in the separate dockerhub repo. #28656 (alesapin).
  • 👌 Improve support for build with clang-13. #28046 (Sergei Semin).
  • ➕ Add ability to print raw profile events to clickhouse-client (This can be useful for debugging and for testing). #30064 (Azat Khuzhin).
  • ➕ Add time dependency for clickhouse-server unit (systemd and sysvinit init). #28891 (Azat Khuzhin).
  • Reload stacktrace cache when symbol is reloaded. #28137 (Amos Bird).

  🐛 Bug Fix

  • 🛠 Functions for case-insensitive search in UTF-8 strings like positionCaseInsensitiveUTF8 and countSubstringsCaseInsensitiveUTF8 might find substrings that actually does not match in very rare cases, it's fixed. #30663 (tavplubix).
  • 🛠 Fix reading from empty file on encrypted disk. #30494 (Vitaly Baranov).
  • Fix transformation of disjunctions chain to IN (controlled by settings optimize_min_equality_disjunction_chain_length) in distributed queries with settings legacy_column_name_of_tuple_literal = 0. #28658 (Anton Popov).
  • 👍 Allow using a materialized column as the sharding key in a distributed table even if insert_allow_materialized_columns=0:. #28637 (Vitaly Baranov).
  • 🛠 Fix ORDER BY ... WITH FILL with set TO and FROM and no rows in result set. #30888 (Anton Popov).
  • 🛠 Fix set index not used in AND/OR expressions when there are more than two operands. This fixes #30416 . #30887 (Amos Bird).
  • 🛠 Fix crash when projection with hashing function is materialized. This fixes #30861 . The issue is similar to https://github.com/ClickHouse/ClickHouse/pull/28560 which is a lack of proper understanding of the invariant of header's emptyness. #30877 (Amos Bird).
  • 🛠 Fixed ambiguity when extracting auxiliary ZooKeeper name from ZooKeeper path in ReplicatedMergeTree. Previously server might fail to start with Unknown auxiliary ZooKeeper name if ZooKeeper path contains a colon. Fixes #29052. Also it was allowed to specify ZooKeeper path that does not start with slash, but now it's deprecated and creation of new tables with such path is not allowed. Slashes and colons in auxiliary ZooKeeper names are not allowed too. #30822 (tavplubix).
  • Clean temporary directory when localBackup failed by some reason. #30797 (ianton-ru).
  • 🛠 Fixed a race condition between REPLACE/MOVE PARTITION and background merge in non-replicated MergeTree that might cause a part of moved/replaced data to remain in partition. Fixes #29327. #30717 (tavplubix).
  • 🛠 Fix PREWHERE with WHERE in case of always true PREWHERE. #30668 (Azat Khuzhin).
  • 🛠 Limit push down optimization could cause a error Cannot find column. Fixes #30438. #30562 (Nikolai Kochetov).
  • ➕ Add missing parenthesis for isNotNull/isNull rewrites to IS [NOT] NULL (fixes queries that has something like isNotNull(1)+isNotNull(2)). #30520 (Azat Khuzhin).
  • 🛠 Fix deadlock on ALTER with scalar subquery to the same table, close #30461. #30492 (Vladimir C).
  • 🛠 Fixed segfault which might happen if session expired during execution of REPLACE PARTITION. #30432 (tavplubix).
  • 🛠 Queries with condition like IN (subquery) could return incorrect result in case if aggregate projection applied. Fixed creation of sets for projections. #30310 (Amos Bird).
  • 🛠 Fix column alias resolution of JOIN queries when projection is enabled. This fixes #30146. #30293 (Amos Bird).
  • 🛠 Fix some deficiency in replaceRegexpAll function. #30292 (Memo).
  • 🛠 Fix ComplexKeyHashedDictionary, ComplexKeySparseHashedDictionary parsing preallocate option from layout config. #30246 (Maksim Kita).
  • 🛠 Fix [I]LIKE function. Closes #28661. #30244 (Nikolay Degterinsky).
  • 🛠 Fix crash with shortcircuit and lowcardinality in multiIf. #30243 (Raúl Marín).
  • FlatDictionary, HashedDictionary fix bytes_allocated calculation for nullable attributes. #30238 (Maksim Kita).
  • 👍 Allow identifiers starting with numbers in multiple joins. #30230 (Vladimir C).
  • Fix reading from MergeTree with max_read_buffer_size = 0 (when the user wants to shoot himself in the foot) (can lead to exceptions Can't adjust last granule, LOGICAL_ERROR, or even data loss). #30192 (Azat Khuzhin).
  • Fix pread_fake_async/pread_threadpool with min_bytes_to_use_direct_io. #30191 (Azat Khuzhin).
  • 🛠 Fix INSERT SELECT incorrectly fills MATERIALIZED column based of Nullable column. #30189 (Azat Khuzhin).
  • 👌 Support nullable arguments in function initializeAggregation. #30177 (Anton Popov).
  • 🛠 Fix error Port is already connected for queries with GLOBAL IN and WITH TOTALS. Only for 21.9 and 21.10. #30086 (Nikolai Kochetov).
  • 🛠 Fix race between MOVE PARTITION and merges/mutations for MergeTree. #30074 (Azat Khuzhin).
  • ✂ Dropped Memory database might reappear after server restart, it's fixed (#29795). Also added force_remove_data_recursively_on_drop setting as a workaround for Directory not empty error when dropping Ordinary database (because it's not possible to remove data leftovers manually in cloud environment). #30054 (tavplubix).
  • 🛠 Fix crash of sample by tuple(), closes #30004. #30016 (flynn).
  • try to close issue: #29965. #29976 (hexiaoting).
  • 🛠 Fix possible data-race between FileChecker and StorageLog/StorageStripeLog. #29959 (Azat Khuzhin).
  • 🛠 Fix data-race between LogSink::writeMarks() and LogSource in StorageLog. #29946 (Azat Khuzhin).
  • 🛠 Fix potential resource leak of the concurrent query limit of merge tree tables introduced in https://github.com/ClickHouse/ClickHouse/pull/19544. #29879 (Amos Bird).
  • 🛠 Fix system tables recreation check (fails to detect changes in enum values). #29857 (Azat Khuzhin).
  • MaterializedMySQL: Fix an issue where if the connection to MySQL was lost, only parts of a transaction could be processed. #29837 (Håvard Kvålen).
  • 🛠 Avoid Timeout exceeded: elapsed 18446744073.709553 seconds error that might happen in extremely rare cases, presumably due to some bug in kernel. Fixes #29154. #29811 (tavplubix).
  • 🛠 Fix bad cast in ATTACH TABLE ... FROM 'path' query when non-string literal is used instead of path. It may lead to reading of uninitialized memory. #29790 (alexey-milovidov).
  • 🛠 Fix concurrent access to LowCardinality during GROUP BY (in combination with Buffer tables it may lead to troubles). #29782 (Azat Khuzhin).
  • Fix incorrect GROUP BY (multiple rows with the same keys in result) in case of distributed query when shards had mixed versions <= 21.3 and >= 21.4, GROUP BY key had several columns all with fixed size, and two-level aggregation was activated (see group_by_two_level_threshold and group_by_two_level_threshold_bytes). Fixes #29580. #29735 (Nikolai Kochetov).
  • Fixed incorrect behaviour of setting materialized_postgresql_tables_list at server restart. Found in #28529. #29686 (Kseniia Sumarokova).
  • Condition in filter predicate could be lost after push-down optimisation. #29625 (Nikolai Kochetov).
  • 🛠 Fix JIT expression compilation with aliases and short-circuit expression evaluation. Closes #29403. #29574 (Maksim Kita).
  • 🛠 Fix rare segfault in ALTER MODIFY query when using incorrect table identifier in DEFAULT expression like x.y.z... Fixes #29184. #29573 (alesapin).
  • 🛠 Fix nullptr deference for GROUP BY WITH TOTALS HAVING (when the column from HAVING wasn't selected). #29553 (Azat Khuzhin).
  • Avoid deadlocks when reading and writting on Join table engine tables at the same time. #29544 (Raúl Marín).
  • 🛠 Fix bug in check pathStartsWith becuase there was bug with the usage of std::mismatch: The behavior is undefined if the second range is shorter than the first range.. #29531 (Kseniia Sumarokova).
  • In ODBC bridge add retries for error Invalid cursor state. It is a retriable error. Closes #29473. #29518 (Kseniia Sumarokova).
  • 🛠 Fixed incorrect table name parsing on loading of Lazy database. Fixes #29456. #29476 (tavplubix).
  • 🛠 Fix possible Block structure mismatch for subqueries with pushed-down HAVING predicate. Fixes #29010. #29475 (Nikolai Kochetov).
  • 🛠 Fix Logical error Cannot capture columns in functions greatest/least. Closes #29334. #29454 (Kruglov Pavel).
  • ✅ RocksDB table engine: fix race condition during multiple DB opening (and get back some tests that triggers the problem on CI). #29393 (Azat Khuzhin).
  • 🛠 Fix replicated access storage not shutting down cleanly when misconfigured. #29388 (Kevin Michel).
  • ✂ Remove window function nth_value as it is not memory-safe. This closes #29347. #29348 (alexey-milovidov).
  • 🛠 Fix vertical merges of projection parts. This fixes #29253 . This PR also fixes several projection merge/mutation issues introduced in https://github.com/ClickHouse/ClickHouse/pull/25165. #29337 (Amos Bird).
  • 🛠 Fix hanging DDL queries on Replicated database while adding a new replica. #29328 (Kevin Michel).
  • ⏱ Fix connection timeouts (send_timeout/receive_timeout). #29282 (Azat Khuzhin).
  • 🛠 Fix possible Table columns structure in ZooKeeper is different from local table structure exception while recreating or creating new replicas of ReplicatedMergeTree, when one of table columns have default expressions with case-insensitive functions. #29266 (Anton Popov).
  • Send normal Database doesn't exist error (UNKNOWN_DATABASE) to the client (via TCP) instead of Attempt to read after eof (ATTEMPT_TO_READ_AFTER_EOF). #29229 (Azat Khuzhin).
  • 🛠 Fix segfault while inserting into column with type LowCardinality(Nullable) in Avro input format. #29132 (Kruglov Pavel).
  • 🔧 Do not allow to reuse previous credentials in case of inter-server secret (Before INSERT via Buffer/Kafka to Distributed table with interserver secret configured for that cluster, may re-use previously set user for that connection). #29060 (Azat Khuzhin).
  • Handle any_join_distinct_right_table_keys when join with dictionary, close #29007. #29014 (Vladimir C).
  • 🛠 Fix "Not found column ... in block" error, when join on alias column, close #26980. #29008 (Vladimir C).
  • 🛠 Fix the number of threads used in GLOBAL IN subquery (it was executed in single threads since #19414 bugfix). #28997 (Nikolai Kochetov).
  • 🛠 Fix bad optimizations of ORDER BY if it contains WITH FILL. This closes #28908. This closes #26049. #28910 (alexey-milovidov).
  • 🛠 Fix higher-order array functions (SIGSEGV for arrayCompact/ILLEGAL_COLUMN for arrayDifference/arrayCumSumNonNegative) with consts. #28904 (Azat Khuzhin).
  • 🛠 Fix waiting for mutation with mutations_sync=2. #28889 (Azat Khuzhin).
  • 🛠 Fix queries to external databases (i.e. MySQL) with multiple columns in IN ( i.e. (k,v) IN ((1, 2)) ). #28888 (Azat Khuzhin).
  • 🛠 Fix bug with LowCardinality in short-curcuit function evaluation. Closes #28884. #28887 (Kruglov Pavel).
  • 🛠 Fix reading of subcolumns from compact parts. #28873 (Anton Popov).
  • 🛠 Fixed a race condition between DROP PART and REPLACE/MOVE PARTITION that might cause replicas to diverge in rare cases. #28864 (tavplubix).
  • 🛠 Fix expressions compilation with short circuit evaluation. #28821 (Azat Khuzhin).
  • 🛠 Fix extremely rare case when ReplicatedMergeTree replicas can diverge after hard reboot of all replicas. The error looks like Part ... intersects (previous|next) part .... #28817 (alesapin).
  • 👍 Better check for connection usability and also catch any exception in RabbitMQ shutdown just in case. #28797 (Kseniia Sumarokova).
  • 🛠 Fix benign race condition in ReplicatedMergeTreeQueue. Shouldn't be visible for user, but can lead to subtle bugs. #28734 (alesapin).
  • 🛠 Fix possible crash for SELECT with partially created aggregate projection in case of exception. #28700 (Amos Bird).
  • 🛠 Fix the coredump in the creation of distributed tables, when the parameters passed in are wrong. #28686 (Zhiyong Wang).
  • ➕ Add Settings.Names, Settings.Values aliases for system.processes table. #28685 (Vitaly).
  • 👌 Support for S2 Geometry library: Fix the number of arguments required by s2RectAdd and s2RectContains functions. #28663 (Bharat Nallan).
  • 🛠 Fix invalid constant type conversion when Nullable or LowCardinality primary key is used. #28636 (Amos Bird).
  • 🛠 Fix "Column is not under aggregate function and not in GROUP BY" with PREWHERE (Fixes: #28461). #28502 (Azat Khuzhin).

• All Versions
• Previous v21.10
• Next v21.12
• Latest Version