TokToken

TokToken (beta release)

Right now, your AI coding agent reads entire files just to find one function. That wastes tokens, money, and context window. TokToken fixes this.

Real numbers from Redis (727 files, 45K symbols, indexed in 0.9s):

TokToken scales well. Check it on the Linux kernel (65K files, 7.4M symbols) indexes in ~170 seconds:

One command indexes your codebase. Your agent searches symbols, traces imports, and retrieves only the code it needs -- 88-99% fewer tokens on every operation.

toktoken index:create            # index your project (once)
toktoken search:symbols "auth"   # find symbols by name
toktoken inspect:symbol <id>     # retrieve just the code that matters

Works with Claude Code, Cursor, Windsurf, Copilot, Gemini, Codex, and any MCP-compatible agent. Single binary, zero dependencies, nothing written inside your project.

Quick setup: tell your AI agent to read docs/LLM.md -- it will install and configure TokToken (almost) autonomously. For the rest of you, humans, keep reading.

Features

• 49 languages via universal-ctags + 16 custom parsers (see docs/LANGUAGES.md)
• Import graph: cross-file dependency tracking with find:importers, find:references, find:callers, and inspect:dependencies
• FTS5 search with relevance scoring, cascading query strategies, and token-budget-aware result slicing
• Incremental indexing using content hashing -- re-indexes only changed files, including single-file reindex
• Dead code detection: find symbols with no callers or importers across the codebase
• Blast radius analysis: trace all files and symbols affected by a change to a given file or symbol
• Circular import detection: identify import cycles in the dependency graph
• Multi-symbol bundles: retrieve context bundles for multiple symbols in a single call with markdown output option
• Scope-filtered search: restrict symbol and text search to a subtree or file set
• Centrality-based ranking: symbols ranked by import-graph centrality in addition to FTS relevance
• MCP server (toktoken serve) for native IDE integration with tiered tool loading
• GitHub repo indexing (toktoken index:github owner/repo) -- index any public repo without cloning
• Smart filtering: excludes non-code files (CSS, HTML) and vendored directories by default, with selective --include override
• Security: symlink escape detection, secret pattern filtering, binary exclusion
• Token savings tracking: cumulative metrics via the stats command
• Auto-update: --self-update with SHA-256 verification and atomic binary replacement
• Cross-platform: Linux (x64/ARM64/ARMv7), macOS (Intel/Apple Silicon), Windows (x64)
• Single static binary: no runtime requirements beyond universal-ctags (see below)
• No project pollution: all data stored under ~/.cache/toktoken/, nothing written inside your project

Prerequisites

TokToken requires universal-ctags (NOT exuberant-ctags) for symbol extraction. Static linking of ctags is planned but not yet implemented.

Verify installation: ctags --version must show Universal Ctags, not Exuberant Ctags.

Quick Start

PATH setup: Ensure ~/.local/bin is in your PATH. If not, add export PATH="$HOME/.local/bin:$PATH" to your ~/.bashrc or ~/.zshrc.

# Install (Linux x86_64 example)
mkdir -p ~/.local/bin
curl -fsSL https://github.com/mauriziofonte/toktoken/releases/latest/download/toktoken-linux-x86_64 \
  -o ~/.local/bin/toktoken && chmod +x ~/.local/bin/toktoken

# Index a project
cd /path/to/your/project
toktoken index:create

# Search for symbols (-k filters by kind, -c enables compact output)
# Note: by default, TokToken excludes non-code files (CSS, HTML, SVG) and
# vendored subdirectories. Use -f / --full to include everything.
# Markdown files are always indexed (documentation kinds: chapter, section, subsection).
toktoken search:symbols "auth" -ck class,method,function

# Full-text search grouped by file
toktoken search:text "TODO" -g file

# Inspect a specific symbol
toktoken inspect:symbol "src/Auth.php::Auth.login#method"

# File outline (cheaper than reading the whole file)
toktoken inspect:outline src/Auth.php

# Update index after edits
toktoken index:update

Commands

Options

All options have a long form (--option). Most also have a single-letter short form (-x). Short boolean flags can be combined: -cn is equivalent to --compact --count. Value flags accept attached (-l10) or separate (-l 10) values.

Indexing options

Used with index:create, index:update, and index:github.

Search options

Used with search:symbols and search:text.

Inspect options

Used with inspect:file, inspect:tree, and inspect:bundle.

Global options

Available for all commands.

Examples

# Search symbols, compact + count only
toktoken search:symbols "auth" -cn

# Search functions in Python, limit 20, compact
toktoken search:symbols "parse" -ck function -L python -l20

# Full-text search grouped by file, 3 lines of context
toktoken search:text "TODO" -g file -C3

# Index with max 10k files, ignore vendor and dist
toktoken index:create -m10000 -i vendor -i dist

# Index with vendor/ included (e.g. Symfony/Laravel projects)
toktoken index:create --include vendor

# Index everything (disable smart filter)
toktoken index:create -f

# Tree with depth 2, compact
toktoken inspect:tree -cd2

# All boolean flags combined
toktoken search:symbols "init" -cnrs
# equivalent to: --compact --count --regex --case-sensitive

Diagnostic Mode

The --diagnostic / -X flag enables structured JSONL output on stderr during indexing. Each line is a self-contained JSON object with a timestamp, phase, event type, and payload. Designed for performance analysis and debugging.

# Index with diagnostics, capture to file
toktoken index:create -X 2>/tmp/diagnostics.jsonl

# Pretty-print events
jq . /tmp/diagnostics.jsonl

Events are organized by phase:

Example event:

{"ts":2.145,"ph":"worker","ev":"done","wid":3,"files":4096,"symbols":312847,"elapsed_ms":21453}

All timestamps (ts) are seconds since pipeline start. Worker events include wid (worker ID). Memory snapshots report vm_kb and rss_kb.

MCP Server

TokToken runs as a Model Context Protocol server for native integration with AI IDEs:

toktoken serve

Claude Code

# User-scoped (available across all projects)
claude mcp add-json --scope user toktoken '{"command":"toktoken","args":["serve"]}'

# Or project-scoped (shared via .mcp.json)
claude mcp add-json --scope project toktoken '{"command":"toktoken","args":["serve"]}'

# Verify
claude mcp list

VS Code / GitHub Copilot

VS Code uses a different format. The top-level key is "servers", not "mcpServers".

Create .vscode/mcp.json in your project root:

{
  "servers": {
    "toktoken": {
      "command": "toktoken",
      "args": ["serve"]
    }
  }
}

See docs/setup/copilot-vscode.md for user-scoped setup and requirements.

Other MCP Clients

For Cursor, Windsurf, Gemini CLI, Gemini Code Assist, and Claude Desktop, add the following to the platform's MCP config file:

{
  "mcpServers": {
    "toktoken": {
      "command": "toktoken",
      "args": ["serve"]
    }
  }
}

Exposes 27 tools: codebase_detect, index_create, index_update, index_file, index_github, search_symbols, search_text, search_cooccurrence, search_similar, inspect_outline, inspect_symbol, inspect_file, inspect_tree, inspect_bundle, inspect_dependencies, inspect_hierarchy, inspect_cycles, inspect_blast_radius, find_importers, find_references, find_callers, find_dead, suggest, stats, projects_list, cache_clear, help.

AI Agent Setup

For LLMs: read docs/LLM.md for complete setup and integration instructions. This file is designed to be consumed directly by AI agents to autonomously configure TokToken for the user's environment.

https://raw.githubusercontent.com/mauriziofonte/toktoken/main/docs/LLM.md

Supported platforms: Claude Code, Cursor, Windsurf, Gemini CLI, Codex CLI, and any agent that can execute shell commands.

Building from Source

Requires: CMake >= 3.16, GCC >= 9 or Clang >= 10.

# Debug (with ASan + UBSan)
cmake -B build -DCMAKE_BUILD_TYPE=Debug
cmake --build build -j$(nproc)

# Release
cmake -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc)

# Static binary (Linux)
cmake -B build -DCMAKE_BUILD_TYPE=Release -DTT_STATIC=ON
cmake --build build -j$(nproc)

# Windows (MSYS2/MinGW64 shell)
cmake -B build -G "Unix Makefiles" -DCMAKE_BUILD_TYPE=Debug
cmake --build build -j$(nproc)

# Run tests
./build/test_unit
./build/test_integration
./build/test_e2e

See CONTRIBUTING.md for development workflow, coding standards, and test conventions.

Storage

TokToken stores all data outside the project directory. Nothing is written inside your codebase.

Cache directory

The cache directory holds indexes, logs, GitHub clones, and the update-check cache.

Home directory resolution: on Unix, $HOME; on Windows, %USERPROFILE% with SHGetFolderPathW(CSIDL_PROFILE) fallback.

~/.cache/toktoken/
    projects/
        <hash>/                 Project directory (hash = first 12 hex chars of xxhash of canonical path)
            db.sqlite           SQLite database (schema v4, WAL mode)
            db.sqlite-wal       WAL journal (auto-managed by SQLite)
            db.sqlite-shm       Shared memory file (auto-managed by SQLite)
    gh-repos/
        <owner>/<repo>/         Shallow clone of GitHub repo (via index:github)
    logs/
        mcp.jsonl               MCP tool call + lifecycle log (append-only JSONL)
    UPSTREAM_VERSION            Cached latest release version (plain text, 12h refresh via curl)

Database contents (per project, in db.sqlite):

Configuration files

Optional, not created by default:

Global config supports all sections (index, logging). Project config supports only the index section. See CONFIGURATION.md for the full reference.

Migration from v0.3.x

Prior to v0.4.0, the cache directory was ~/.cache/.toktoken/ (dot-prefixed). On first access, TokToken atomically renames the old directory to ~/.cache/toktoken/. If another process holds the old path (e.g. a concurrent MCP server), the rename is deferred to the next invocation. No data is lost.

Cleanup

cache:clear deletes the current project's index database. cache:clear --all --force removes all TokToken data (indexes, GitHub clones, logs).

Documentation

License

AGPL-3.0 -- Copyright (c) 2026 Maurizio Fonte

Commercial use without AGPL-3.0 copyleft obligations requires a separate license. Contact: [email protected]

See THIRD_PARTY_NOTICES.md for vendored dependency licenses.