TileDB v2.12.0 Release Notes
-
Disk Format
- Added Delete commit file to format specification.
- โ Added XOR filter type #3383.
๐ New features
๐ Support for DELETE query type, providing the capability to non-destructively (until consolidation with purge) delete data from array from query timestamp forward DELETE feature pull-requests
- Deletes: legacy reader process deletes. #3387
- Deletes: implement delete strategy. #3337
- Deletes: refactored readers process deletes. #3374
- Deletes: adding support for commits consolidation. #3378
- Dense reader: adding num tiles to stats. #3434
- Opt-in core-to-REST-server instrumentation #3432
- Deletes: implement consolidation. #3402
- Deletes: adding examples. #3437
- Deletes: implement serialization. #3450
- Deletes consolidation: switch from marker hashes to condition indexes. #3451
- Deletes: adding purge option for consolidation. #3458
- Deletes: disallow in middle of consolidated fragment with no timestamps. #3470
- Implement delete_fragments API for removing all fragments within a specified time range #3400
- Implement XOR Filter #3383
- ๐ Fragment info serialization support for
tiledb://
URIs #3530 - โ Add support for global order writes to
tiledb://
URIs #3393
API Changes
Config parameters
- โ Add new config
rest.curl.buffersize
for settingCURLOPT_BUFFERSIZE
. #3440
C API
- Add
tiledb_group_get_is_relative_uri_by_name
#3550 - Adding experimental API for getting relevant fragments,
tiledb_query_get_relevant_fragment_num
. #3413 - Add
tiledb_query_get_relevant_fragment_num
for experimental API to get relevant fragments. #3413 - โ Add include policy for non-TileDB headers in the C API #3414
๐ Improvements
๐ Performance
- ๐ Sparse global order reader: merge algorithm optimization. #3331
- ๐ Azure: parallelize remove_dir. #3357
- ๐จ Sparse refactored readers, mark empty fragments as fully loaded early. #3394
- โฌ๏ธ Reduce the number of requests in dir_size #3382
- VFS: Adding option to disable batching for
read_tiles
. #3421 - Avoid duplicate string_view creation in
compute_results_count_sparse_string_range
#3491 - 0๏ธโฃ Memory tracker: using the correct type for setting default budget. #3509
- ๐ Sparse global order reader: compute hilbert vals before filtering tiles. #3497
- Avoid string copy in Dictionary Encoding decompression. #3490
๐ Defects removed
- โ Add check that Dict/RLE for strings is the first filter in the pipeline; allow use w/ other filters #3510
- โ Remove stale declarations from query #3565
- ๐ Fix SC-19287: segfault due to deref nonexistent filestore key #3359
- Demonstrating mingw handle leakage (tiledb_unit extract) #3362
- ๐ Fix and regression test for SC-19240 #3360
- ๐ Don't try to consolidate empty array; fixes SC-19516 #3389
- ๐ Fixes check for experimental schema features to be current version #3404
- Prevent possible compiler dependent errors serializing groups #3399
- ๐ use stoul() to correctly parse (32bit unsigned values) experimental version numbers cross-platform #3410
- Fix deserialize to set array_schema_all_ into array object #3363
- #3430 #3431
- โ Removes unneeded fabs causing warning on clang #3484
- Dictionary encoding should handle zero length strings #3493
- ๐ Fix empty metadata after array open/query submit #3495
- Handle filter_from_capnp FilterType::NONE case #3516
- ๐ Sparse global order reader: incomplete reads when hitting memory limits. #3518
- ๐ Fixes silent failure for mismatched layout and bad layout/array combos on query #3521
- Correct defect in source of keying material #3529
- Rework delete_fragments API #3505
- ๐ Fix segfault after schema evolution when reading using
TILEDB_UNORDERED
#3528 - ๐ Fix SC-21741, array evolve via REST #3532
- Don't fetch Array data in Controller until the Array is fully open #3538
- ๐ Do not allow creation of sparse array with zero capacity. #3546
- ๐ Tile metadata: fixing for ordered writes. #3527
Internal
- Array consistency controller #3130
- ๐ Experimental build format versioning #3364
- โก๏ธ Implementation of
DataBlock
,DataBlock
allocator,join
view, and updates toSource
andSink
. #3366 - Implemented basic platform library #3420
- โ Adds Edge and simple Node classes to the TileDB task graph library. #3453
- โ Adds attribute ranges to Subarray for internal usage #3520
- โ Add list of point ranges to Subarray #3502
- โ Add support for new array open REST call #3339
- Added
Config::must_find
marker for use with newConfig::get
signature. ThrowsStatus_ConfigError
if value cannot be found. #3482
๐ท Build system changes
- โ Add abseil/absl to build via ExternalProject_Add #3454
- โ Add Crc32c to tiledb build via ExternalProject_Add #3455
- ๐ Enable superbuild libcurl to support zstd #3469
- ๐ณ Adjust example dockerfile so layers can be cached better #3488
Full Changelog: