What is Token?Token kya hai?

🧸 Think of it likeAise samjho

Like the syllables of a word: an LLM reads text as small pieces, not whole words.Shabd ke akshar samooh jaisa: LLM text ko chhote tukdon mein padhta hai, poore shabdon mein nahi.

📘 ExplainedVistaar se

A tokenizer splits text into tokens; in English one token is about three quarters of a word, and Indian languages often need more tokens for the same meaning. Cost, speed and the context window all depend on token counts, so shorter prompts and outputs save money.Tokenizer text ko tokens mein todta hai; English mein ek token lagbhag teen chauthai shabd hai, aur Indian languages ko aksar same matlab ke liye zyaada tokens chahiye. Cost, speed aur context window sab token counts pe depend karte hain, isliye chhote prompts aur outputs paise bachaate hain.

💡 Real exampleAsli example

'Unbelievable' may split into 'un', 'believ', 'able': three tokens.'Unbelievable' 'un', 'believ', 'able' mein toot sakta hai: teen tokens.

📖 Learn it step by stepStep by step seekho

• What is an LLM? Tokens, context and temperatureLLM kya hai? Tokens, context aur temperature
• Calling an LLM API: keys, requests and costsLLM API call karna: keys, requests aur costs

🔗 Related termsJude hue shabd

• Context windowThe maximum amount of text (in tokens) an LLM can consider at once, including its answer.Ek baar mein LLM jitna text (tokens mein) dekh sakta hai uski seema, uske jawaab samet.
• LLMA very large neural network trained on huge amounts of text to predict the next word, which lets it write, answer and reason.Bahut bada neural network jo bahut saare text pe agla shabd predict karne ke liye train hua hai, jisse woh likh, jawaab de aur soch sakta hai.