TileDB v2.12.0 Release Notes

  • Disk Format

    ๐Ÿ†• New features

    • ๐Ÿ‘Œ Support for DELETE query type, providing the capability to non-destructively (until consolidation with purge) delete data from array from query timestamp forward DELETE feature pull-requests

      • Deletes: legacy reader process deletes. #3387
      • Deletes: implement delete strategy. #3337
      • Deletes: refactored readers process deletes. #3374
      • Deletes: adding support for commits consolidation. #3378
      • Dense reader: adding num tiles to stats. #3434
      • Opt-in core-to-REST-server instrumentation #3432
      • Deletes: implement consolidation. #3402
      • Deletes: adding examples. #3437
      • Deletes: implement serialization. #3450
      • Deletes consolidation: switch from marker hashes to condition indexes. #3451
      • Deletes: adding purge option for consolidation. #3458
      • Deletes: disallow in middle of consolidated fragment with no timestamps. #3470
    • Implement delete_fragments API for removing all fragments within a specified time range #3400
    • Implement XOR Filter #3383
    • ๐Ÿ‘ Fragment info serialization support for tiledb:// URIs #3530
    • โž• Add support for global order writes to tiledb:// URIs #3393

    API Changes

    Config parameters

    • โž• Add new config rest.curl.buffersize for setting CURLOPT_BUFFERSIZE. #3440

    C API

    • Add tiledb_group_get_is_relative_uri_by_name #3550
    • Adding experimental API for getting relevant fragments, tiledb_query_get_relevant_fragment_num. #3413
    • Add tiledb_query_get_relevant_fragment_num for experimental API to get relevant fragments. #3413
    • โž• Add include policy for non-TileDB headers in the C API #3414

    ๐Ÿ‘Œ Improvements

    ๐ŸŽ Performance

    • ๐Ÿ”€ Sparse global order reader: merge algorithm optimization. #3331
    • ๐Ÿšš Azure: parallelize remove_dir. #3357
    • ๐Ÿ”จ Sparse refactored readers, mark empty fragments as fully loaded early. #3394
    • โฌ‡๏ธ Reduce the number of requests in dir_size #3382
    • VFS: Adding option to disable batching for read_tiles. #3421
    • Avoid duplicate string_view creation in compute_results_count_sparse_string_range #3491
    • 0๏ธโƒฃ Memory tracker: using the correct type for setting default budget. #3509
    • ๐Ÿ“œ Sparse global order reader: compute hilbert vals before filtering tiles. #3497
    • Avoid string copy in Dictionary Encoding decompression. #3490

    ๐Ÿšš Defects removed

    • โž• Add check that Dict/RLE for strings is the first filter in the pipeline; allow use w/ other filters #3510
    • โœ‚ Remove stale declarations from query #3565
    • ๐Ÿ›  Fix SC-19287: segfault due to deref nonexistent filestore key #3359
    • Demonstrating mingw handle leakage (tiledb_unit extract) #3362
    • ๐Ÿ›  Fix and regression test for SC-19240 #3360
    • ๐Ÿ›  Don't try to consolidate empty array; fixes SC-19516 #3389
    • ๐Ÿ›  Fixes check for experimental schema features to be current version #3404
    • Prevent possible compiler dependent errors serializing groups #3399
    • ๐Ÿ“œ use stoul() to correctly parse (32bit unsigned values) experimental version numbers cross-platform #3410
    • Fix deserialize to set array_schema_all_ into array object #3363
    • #3430 #3431
    • โœ‚ Removes unneeded fabs causing warning on clang #3484
    • Dictionary encoding should handle zero length strings #3493
    • ๐Ÿ›  Fix empty metadata after array open/query submit #3495
    • Handle filter_from_capnp FilterType::NONE case #3516
    • ๐Ÿ“œ Sparse global order reader: incomplete reads when hitting memory limits. #3518
    • ๐Ÿ›  Fixes silent failure for mismatched layout and bad layout/array combos on query #3521
    • Correct defect in source of keying material #3529
    • Rework delete_fragments API #3505
    • ๐Ÿ›  Fix segfault after schema evolution when reading using TILEDB_UNORDERED #3528
    • ๐Ÿ›  Fix SC-21741, array evolve via REST #3532
    • Don't fetch Array data in Controller until the Array is fully open #3538
    • ๐Ÿ“œ Do not allow creation of sparse array with zero capacity. #3546
    • ๐Ÿ“‡ Tile metadata: fixing for ordered writes. #3527

    Internal

    • Array consistency controller #3130
    • ๐Ÿ— Experimental build format versioning #3364
    • โšก๏ธ Implementation of DataBlock, DataBlock allocator, join view, and updates to Source and Sink. #3366
    • Implemented basic platform library #3420
    • โž• Adds Edge and simple Node classes to the TileDB task graph library. #3453
    • โž• Adds attribute ranges to Subarray for internal usage #3520
    • โž• Add list of point ranges to Subarray #3502
    • โž• Add support for new array open REST call #3339
    • Added Config::must_find marker for use with new Config::get signature. Throws Status_ConfigError if value cannot be found. #3482

    ๐Ÿ‘ท Build system changes

    • โž• Add abseil/absl to build via ExternalProject_Add #3454
    • โž• Add Crc32c to tiledb build via ExternalProject_Add #3455
    • ๐Ÿ‘ Enable superbuild libcurl to support zstd #3469
    • ๐Ÿณ Adjust example dockerfile so layers can be cached better #3488

    Full Changelog: