« Changelog History


MeTA v1.3.7 Release Notes

  • 🐛 Bug fixes

    • 🛠 Fix inconsistent behavior of utf::segmenter (and thus icu_tokenizer) for different locales. Thanks @CanoeFZH and @tng-konrad for helping debug this!

    ✨ Enhancements

    • 👍 Allow for specifying the language and country for locale generation in setting up utf::segmenter (and thus icu_tokenizer)
    • 👍 Allow for suppression of <s> and </s> tags within icu_tokenizer, mostly useful for information retrieval experiments with unigram words. Thanks @husseinhazimeh for the suggestion!
    • ➕ Add a default-unigram-chain filter chain preset which is suitable for information retrieval experiments using unigram words. Thanks @husseinhazimeh for the suggestion!