Jtokkit

JTokkit, short for Java Tokenizer Kit, is a Java tokenizer library designed for use with OpenAI models. It was written to give the JVM ecosystem the same capability that the tiktoken library provides for Python, and its main everyday use is counting the tokens a piece of text will consume before a request is sent to a model. Developers create an encoding registry, pick an encoding or ask for the one that matches a specific OpenAI model, and then call encode and decode on the text. The library implements the r50k base, p50k base, p50k edit, cl100k base and o200k base encodings. It has zero dependencies, supports Java 8 and above, and its registry and encoding classes are thread safe so they can be shared across components. Custom encodings can be added either by implementing the encoding interface and registering it, or by supplying new parameters to the existing byte pair encoding algorithm. The maintainers report that it runs two to three times faster than a comparable tokenizer, with the benchmark published in the repository. JTokkit is free and open source under the MIT License and is distributed as a Maven dependency, with Gradle also supported. Documentation lives on the project site and in the JavaDoc. It is maintained by Knuddels.