Was ist Token

Definition

A token is the fundamental semantic unit (e.g., a word, a syllable, or a single character) into which text is segmented for mathematical processing by a Large Language Model.

Explanation

In the context of Large Language Models (LLMs), text processing doesn't occur with complete words but rather through what's known as tokenization. An average English word typically corresponds to about 1.3 tokens, while German, with its propensity for longer compound words, often requires more tokens per word. Given that the pricing and context windows of commercial SaaS AI models (e.g., billed per 1,000 tokens) are directly based on this unit, a clear understanding of tokens is economically crucial for B2B businesses. Efficient prompt utilization and optimizing token consumption are key to keeping operational costs low for AI-powered CRM integrations or support bots.