LZ4 v220.127.116.11 Release NotesRelease Date: 2018-01-14 // about 3 years ago
LZ4 v1.8.1 most visible new feature is its support for Dictionary compression.
This was already somewhat possible, but in a complex way, requiring knowledge of internal working.
👌 Support is now more formally added on the API side within
lib/lz4frame_static.h. It's early days, and this new API is tagged "experimental" for the time being.
lz4doesn't specify how to build a dictionary. All it says is that it can be any file up to 64 KB.
🏗 This approach is compatible with
zstddictionary builder, which can be instructed to create a 64 KB dictionary with this command :
zstd --train dirSamples/* -o dictName --maxdict=64KB
🐎 LZ4 v1.8.1 also offers improved performance at ultra settings (levels 10+).
📜 These levels receive a new code, called optimal parser, available in
🐎 Compared with previous version, the new parser uses less memory (from 384KB to 256KB), performs faster, compresses a little bit better (not much, as it was already close to theoretical limit), and resists pathological patterns which could destroy performance (see #339),
💻 For comparison, here are some quick benchmark using LZ4 v1.8.0 on my laptop with
./lz4 -b9e12 -v ~/dev/bench/silesia.tar ***LZ4 command line interface 64-bits v1.8.0, by Yann Collet*** Benchmarking levels from 9 to 12 9#silesia.tar : 211984896 -> 77897777 (2.721), 24.2 MB/s ,2401.8 MB/s 10#silesia.tar : 211984896 -> 77852187 (2.723), 16.9 MB/s ,2413.7 MB/s 11#silesia.tar : 211984896 -> 77435086 (2.738), 7.1 MB/s ,2425.7 MB/s 12#silesia.tar : 211984896 -> 77274453 (2.743), 3.3 MB/s ,2390.0 MB/s
and now using LZ4 v1.8.1 :
./lz4 -b9e12 -v ~/dev/bench/silesia.tar ***LZ4 command line interface 64-bits v1.8.1, by Yann Collet*** Benchmarking levels from 9 to 12 9#silesia.tar : 211984896 -> 77890594 (2.722), 24.4 MB/s ,2405.2 MB/s 10#silesia.tar : 211984896 -> 77859538 (2.723), 19.3 MB/s ,2476.0 MB/s 11#silesia.tar : 211984896 -> 77369725 (2.740), 10.1 MB/s ,2478.4 MB/s 12#silesia.tar : 211984896 -> 77270146 (2.743), 3.7 MB/s ,2508.3 MB/s
📜 The new parser is also directly compatible with lower compression levels, which brings additional benefits :
- Compatibility with
LZ4_*_destSize()variant, which reverses the logic by trying to fit as much data as possible into a predefined limited size buffer.
- Compatibility with Dictionary compression, as it uses the same tables as regular HC mode
In the future, this compatibility will also allow dynamic on-the-fly change of compression level, but such feature is not implemented at this stage.
🚀 The release also provides a set of small bug fixes and improvements, listed below :
- perf : faster and stronger ultra modes (levels 10+)
- perf : slightly faster compression and decompression speed
- perf : fix bad degenerative case, reported by @c-morgenstern
- 🛠 fix : decompression failed when using a combination of extDict + low memory address (#397), reported and fixed by Julian Scheid (@jscheid)
- 👍 cli : support for dictionary compression (
-D), by Felix Handte @felixhandte
- cli : fix :
lz4 -d --rmpreserves timestamp (#441)
- cli : fix : do not modify
/dev/nullpermission as root, by @aliceatlas
- api : new dictionary api in
- 👍 api :
_destSize()variant supported for all compression levels
- 🏗 build :
make testcompatible with parallel build
-jX, reported by @mwgamera
- 🏗 build : can control
LZ4LIB_VISIBILITYmacro, by @mikir
- install: fix man page directory (#387), reported by Stuart Cardall (@itoffshore)
🛠 Note :
v18.104.22.168is the same as
v.1.8.1, with the version number fixed in source code, as notified by Po-Chuan Hsieh (@sunpoet).
- Compatibility with