« Changelog History


LZ4 v1.8.1.2 Release Notes

Release Date: 2018-01-14 // over 6 years ago

  • LZ4 v1.8.1 most visible new feature is its support for Dictionary compression.
    This was already somewhat possible, but in a complex way, requiring knowledge of internal working.
    👌 Support is now more formally added on the API side within lib/lz4frame_static.h. It's early days, and this new API is tagged "experimental" for the time being.

    👌 Support is also added in the command line utility lz4, using the new command -D, implemented by @felixhandte. The behavior of this command is identical to zstd, should you be already familiar.

    🏗 lz4 doesn't specify how to build a dictionary. All it says is that it can be any file up to 64 KB.
    🏗 This approach is compatible with zstd dictionary builder, which can be instructed to create a 64 KB dictionary with this command :

    zstd --train dirSamples/* -o dictName --maxdict=64KB

    🐎 LZ4 v1.8.1 also offers improved performance at ultra settings (levels 10+).
    📜 These levels receive a new code, called optimal parser, available in lib/lz4_opt.h.
    🐎 Compared with previous version, the new parser uses less memory (from 384KB to 256KB), performs faster, compresses a little bit better (not much, as it was already close to theoretical limit), and resists pathological patterns which could destroy performance (see #339),

    💻 For comparison, here are some quick benchmark using LZ4 v1.8.0 on my laptop with silesia.tar :

    ./lz4 -b9e12 -v ~/dev/bench/silesia.tar
***LZ4 command line interface 64-bits v1.8.0, by Yann Collet***
Benchmarking levels from 9 to 12
 9#silesia.tar : 211984896 -> 77897777 (2.721), 24.2 MB/s ,2401.8 MB/s
10#silesia.tar : 211984896 -> 77852187 (2.723), 16.9 MB/s ,2413.7 MB/s
11#silesia.tar : 211984896 -> 77435086 (2.738), 7.1 MB/s ,2425.7 MB/s
12#silesia.tar : 211984896 -> 77274453 (2.743), 3.3 MB/s ,2390.0 MB/s

    and now using LZ4 v1.8.1 :

    ./lz4 -b9e12 -v ~/dev/bench/silesia.tar
***LZ4 command line interface 64-bits v1.8.1, by Yann Collet***
Benchmarking levels from 9 to 12
 9#silesia.tar : 211984896 -> 77890594 (2.722), 24.4 MB/s ,2405.2 MB/s
10#silesia.tar : 211984896 -> 77859538 (2.723), 19.3 MB/s ,2476.0 MB/s
11#silesia.tar : 211984896 -> 77369725 (2.740), 10.1 MB/s ,2478.4 MB/s
12#silesia.tar : 211984896 -> 77270146 (2.743), 3.7 MB/s ,2508.3 MB/s

    📜 The new parser is also directly compatible with lower compression levels, which brings additional benefits :

    • Compatibility with LZ4_*_destSize() variant, which reverses the logic by trying to fit as much data as possible into a predefined limited size buffer.
    • Compatibility with Dictionary compression, as it uses the same tables as regular HC mode

    In the future, this compatibility will also allow dynamic on-the-fly change of compression level, but such feature is not implemented at this stage.

    🚀 The release also provides a set of small bug fixes and improvements, listed below :

    • perf : faster and stronger ultra modes (levels 10+)
    • perf : slightly faster compression and decompression speed
    • perf : fix bad degenerative case, reported by @c-morgenstern
    • 🛠 fix : decompression failed when using a combination of extDict + low memory address (#397), reported and fixed by Julian Scheid (@jscheid)
    • 👍 cli : support for dictionary compression (-D), by Felix Handte @felixhandte
    • cli : fix : lz4 -d --rm preserves timestamp (#441)
    • cli : fix : do not modify /dev/null permission as root, by @aliceatlas
    • api : new dictionary api in lib/lz4frame_static.h
    • 👍 api : _destSize() variant supported for all compression levels
    • 🏗 build : make and make test compatible with parallel build -jX, reported by @mwgamera
    • 🏗 build : can control LZ4LIB_VISIBILITY macro, by @mikir
    • install: fix man page directory (#387), reported by Stuart Cardall (@itoffshore)

    🛠 Note : v1.8.1.2 is the same as v.1.8.1, with the version number fixed in source code, as notified by Po-Chuan Hsieh (@sunpoet).