Inside the LLM Prompt Pipeline
- When you submit a prompt, the model breaks the text into tokens (sub‑word pieces), assigns each token an ID, and this token count—not word count—determines the length limits.
- Each token ID is transformed into a high‑dimensional embedding vector, placing semantically similar words (e.g., “king” and “queen”) close together in a learned meaning space.