Changelog History
-
v1.27.1612021 Changes
December 05, 2016🚀 Long time since the previous release (I hope you enjoy github master / trunk which we keep stable!) with lots of changes:
- 🔊 Common "wdt> " prefix in all the logging (helps finding/filtering wdt specific logs when embedded in another service)
- Progress report during file discovery
- Throttler changes working with the resource controller and can be lowered in the middle of a transfer and improved fairness.
- 👍 Multiple applications support within one process, Wdt options not forced to be a singleton, apply to a tree of sub object below the application wdt object
- 👍 Allow receiver to specify its --hostname
- 🆕 New --fsync option to fsync after the last block of each file is written.
- Incremental transfers (-enable_download_resumption) improved reporting
- Wdt URI encoding/decoding is now binary safe (transferid can be any binary data)
- 🛠 Fixed disk mode (files transferred serially and not cut in blocks) >2Gb errors; integer overflow
- Fewer dependencies. Uses std::make_uniq (need cxx14)
- Receiver now drives Sender's download resumption through the URI
- Benchmark generation (bench/) code
- 🆕 New more efficient Varint encoding and less likely to have memory corruption
- 🐛 Bug fixes (see commits) like listening sockets staying open after receiver timeout
- 🚀 Misc changes like: Release/build improvements, Code quality improvements like constructor cleanups (#16)
Enjoy!
-
v1.26.1603040 Changes
March 05, 2016- Option to use fadvise(POSIX_FADVISE_DONTNEED) to avoid taking too much cache/buffers on sender/read path
- 🛠 other fixes/improvements
- ✅ Simpler/shorter tests
-
v1.26.1602090 Changes
February 09, 2016🚀 All of the goodies in v1.25 plus:
- Support for deletion during resumption through the delete_extra_files - if a subsequent snapshot has fewer files than were already sent before, delete those first. This is useful for use cases like rocksdb/myrocks where after compaction some files disappear
- ⏪ Dynamic ports (available ports) is now the default, you don't need to specify -start_port 0 anymore for the receiver side of wdt, if you want to revert to fixed ports, use -static_ports or set a non 0 start_port explicitly. This should make it easier for wdt transfers using pipe to connect the receiver and sender to always succeed even if another transfer is already ongoing on the same destination
-
v1.25.1602051 Changes
February 08, 2016🚀 Long time since the previous release (but there is always github master!) - lot's of changes:
🆕 New features/major changes:
👍 Much better command line interface to match the need of the encryption key transmission securely, "-" argument for the sender reads the connection url from stdin, which the receiver emits on stdout. So the 2 can be plugged through a pipe (securely using ssh - which will only be used for secure key exchange, everything else blazing fast and encrypted through WDT's own socket)
ssh dsthost wdt -directory destdir | ssh srchost wdt -directory srcdir -# Or more complex:(ssh localhost wdt -fork -start_port 0 -directory /tmp/dst1; ls *.h ) | wdt -manifest - -
0️⃣ Aes GCM crypto support (requires a recent openssl version) : is now the default - it does both encryption and validates the integrity in 1 pass. We also support this during error handling only resuming at the last block that was successfully validated.
Socket buffer size can be set explicitly (using -send_buffer_size and -receive_buffer_size)
🐛 Bug fixes:
- 👌 improved TransferLogManager locking (fail fast when can't be acquired) and detection of deleted containing directory/log file
- ResourceController double create, wdtSend can cancel previous / preexisting transfer
- with opening files during discovery
Misc:
- Consistent use of "Mbytes" throughout WDT for 1024*1024 = 1048576 bytes (aka Mebibyte (MiB)
- ✅ Testing/path improvements
- Removing direct use of WdtOptions::get across the code base - now the options can be different for different sender/receivers in the same process.
-
v1.23.1511300 Changes
December 01, 2015🚀 Big release !
🆕 New features:
- 👍 WDT now supports encryption (aes128)
The receiver is generating a one time 128bits secret, encoded in the URL. This secret needs to be transmitted securely (at facebook we use secure thrift for this but you can use ssh for instance). Note that we do not try to authenticate and only encrypt.
- Other improvements
Also estimate the time it may take for a slow receiver to consume the enqueued bytes to avoid timing out waiting for the ack to the end of transfer
🚧 There is a new https://github.com/facebook/wdt/blob/master/Wdt.h simpler API for C++ (which is work in progress but shows the direction we are headed)
- 🐛 Bug fixes:
file descriptor (in FileInfo api) wasn't used when download resumption was also on, locking , error code summary, ...
- Known issues:
Issue #102
-
v1.22.1510270 Changes
October 28, 2015🔨 Important bug fixes (notification change in big refactor was incorrect. issue with long running mode,...)
🔄 Changed code layout, moved to <wdt/X.h> from "X.h"
⚠ Clang-3.7 warnings
Summarize the error codes for the sender (more specific than ERROR/OK, for instance ABORT is differentiated from CONN_ERROR)
-
v1.22.1510210 Changes
October 21, 2015New Features
🔨 1. Refactored the code to split Sender and Receiver into clearly identifiable single threaded code and multi-threaded part (SenderThread / ReceiverThread).
- Wdt now has a thread controller, with thread primitives such as Barrier and Funnel Executors which made the writing of certain features of sender and receiver easier like doing process version mismatch, or sending file chunks in download resumption.
- wdt_max_send_test.sh has been updated and is more robust. This test can be used to benchmark wdt and detect regressions.
🐛 Bugs fixed
- A sender thread could fail on a connection and have a file in its thread transfer history that has not been acknowledged, however it wouldn't stop other threads to finish their part of the transfer even on getting the checkpoint, since the last source would not return to queue. This would result in the final status of transfers being error. 🌲 2. Transfer log manager had a writer thread that would write empty buffer to the transfer log file.
- gcc4.9 SIOPF fix for one our static strings in GFlags.
-
v1.21.1510050 Changes
October 06, 2015🔄 Changed the retry logic to be based on lack of progress instead of being arbitrarily attributed to files retry counts
🛠 Fixed serious int overflow in lseek result handling - files between 2 and 4g (and 6 and 8g and 10-12...) ie whose size has bit 31 set and being set using block mode would fail
✂ Removed the "e"xit command on receiver (dated from old times where the receiver would run forever by default)
🛠 Download resumption (TransferLogManager), thread history, checkpoints cleanup and important bug fixes
-
v1.20.1509241 Changes
September 26, 2015Really fixed URL backward compatibility (use -url_backward_compatibility=true if you have a wdt v20 talking to v18 or earlier)
new -exit_on_bad_flags=false if you want to continue despite bad command line (useful for forward compatibility)
shorter urls (wdt://host:port?num_port=4) and correct escaping of ipv6 [addr]
0️⃣ wdt receiver will now not overwrite any file by default unless you add -overwrite or use download resumption
🛠 misc bug fixes and cleanup
-
v1.19.1509200 Changes
September 21, 2015🔄 Changes since previous tag (Sept 16) :
✅ 1. Added fd to FileInfo. If fd, is provided, we use that fd and do not open the file. I am currently using pread always. I have to more perf test for pread. Max send is really slow in my new dev server, so need to test for perf in some other machine. 🛠 2. If connection url was provided, sender ignored file list. Fixed that and
➕ added a test for file info list ✅ 3. Added an option in e2e simple test to test o_direct mode- Simplifying o_direct code. We should never define macros named O_DIRECT.