Token Counting - Complete Guide
Overview
Tokens are the fundamental unit of measurement for AI processing in the AIInSync Chatbot platform. Understanding how tokens are counted helps you monitor usage, manage your account, and optimize your chatbot's performance.
The 1,500 Token "Smart Cap" đĄī¸
To protect your account from accidental exhaustion during data setup, we apply a fixed cap to initial data uploads:
- Document Uploads: Capped at 1,500 tokens per file.
- URL Scrapes: Capped at 1,500 tokens per URL.
- Full Content Access: Even though you are only billed for 1,500 tokens, we still store the entire content of your files. This ensures your AI can search through every word while keeping your costs predictable.
Token Usage by Operation
1. Knowledge Base & Crawling
- Document/URL Upload: Capped at 1,500 tokens + a fixed 100 token processing fee.
- Website Crawl (Start): A flat fee of 10,000 tokens per crawl operation (to cover automated spider resources).
- Website Crawl (Pages): Actual tokens per page + a fixed 100 token completion fee.
- Item Deletion: A fixed 100 tokens per item removed (covers index cleanup).
2. Conversational Chat (AI Usage)
Every message you send triggers an AI processing chain:
- Intent Classification: Actual AI tokens spent to determine what the user wants.
- AI Response (Basic): Input (Query + Prompt) + Output (Response).
- AI Response (RAG): Input (Query + Document Data) + Output.
- Conversation History: If Memory is enabled, previous messages are sent back to the AI, increasing input tokens slightly with each turn.
3. Vector Store (AI Search Index)
Creating a search index requires the AI to read every single word:
- Index Creation: Based on the ACTUAL sum of all tokens in your knowledge base (no cap).
- Index Update: Tokens added + Tokens removed (total effort to sync).
How to Manage Your Tokens Well đĄ
â
Efficiency Tips (Save Tokens)
- Use Direct Intents: For common FAQs (like "Hours of Operation"), set up a direct response. This bypasses the AI generation entirely and saves tokens.
- Disable Memory for Simple Bots: If your bot only answers one-off questions, turning Memory OFF prevents past history from inflating your costs.
- Clean your Knowledge Base: Delete outdated or duplicate documents before updating your Vector Store.
- Use Subdomain Filters: When crawling a website, exclude pages like "Privacy Policy" or "Terms of Service" to avoid wasting tokens on non-essential data.
â Common "Token Wasters" (Avoid)
- Frequent Full Recreates: Only recreate your vector store if more than 30% of your data has changed. For small changes, use the "Update" feature.
- Infinite Loops: Avoid system prompts that encourage the AI to be overly repetitive or long-winded.
- Unfiltered Crawls: Crawling an entire domain without filters can quickly consume thousands of tokens on irrelevant pages.
Monitoring Usage
Where to Check
1.
Dashboard Overview: See your "Tokens Used This Month" at a glance.
2.
Project Page: Check the "Manage & Track" section for a per-chatbot breakdown.
3.
Analytics: Visit the Analytics page for a deep dive into daily usage trends.
If you reach your monthly limit, your chatbot will pause. For urgent limit increases or custom configurations, please call +1 (661) 625-7028.
