GitHub

SpeechTokenizer: Unified Speech Tokenizer for Speech Language Models

This is the code for the SpeechTokenizer presented in the SpeechTokenizer: Unified Speech Tokenizer for Speech Language Models. SpeechTokenizer is a unified speech tokenizer for speech language models ...
IEEE

TokenFlow: Unified Image Tokenizer for Multimodal Understanding and Generation

Abstract: We present TokenFlow, a novel unified image tokenizer that bridges the long-standing gap between multimodal understanding and generation. Prior research attempt to employ a single ...
GitHub

Tokenizer for language models.

Tokenize text for Llama, Gemini, GPT-4, DeepSeek, Mistral and many others; in the web, on the client and any platform. Kitoken can load and convert many existing tokenizer formats. Every supported ...