tokenizers

crates Free

Rust

Provides an implementation of today's most used tokenizers, with a focus on performances and versatility.

Found on Devadex — the discovery index for independent software the big search engines bury. More from crates.