1. The Core Bottleneck

Traditional LLM inference engines struggle with agentic workloads characterized by multi-step reasoning, dynamic tool invocation, and deep context recycling. Monolithic runtime designs tightly couple scheduling logic with execution kernels, creating high thread contention and forcing costly codebase recompilations during business logic iterations.

TokenSpeed re-architects this paradigm. The control plane is implemented as a C++ finite-state machine, leveraging static type systems to enforce request lifecycles and memory ownership safety at compile time. Meanwhile, the execution plane uses Python to maintain rapid iteration velocity without introducing concurrency vulnerabilities in core scheduling.

💡 Core Architecture Insight: By enforcing state machine invariants through the C++ type system at compile time, TokenSpeed eliminates runtime overhead for resource validation and request lifecycle tracking.

2. Architecture and Data Flow

The runtime pipeline separates CPU-side request handling from tensor execution. The SMG-integrated AsyncLLM ingests asynchronous requests, passing them through the compile-time-verified C++ control plane before dispatching tasks to the pluggable kernel registry.

[ AsyncLLM / SMG ] ---> [ C++ Control Plane (FSM) ] ---> [ Pluggable Kernel Registry ]
                                    │
                                    ▼
                        [ Python Execution Plane ] ---> [ Heterogeneous Hardware (GB300/MI355X) ]

This design establishes clear engineering boundaries. The C++ layer maintains state transitions and memory safety without business logic clutter, while the Python layer organizes execution graphs. Communication occurs via a stable public API, ensuring hardware accelerator kernels can be updated independently.

3. Technical Comparison

Dimension TokenSpeed Traditional Paradigms Alternative Frameworks Production Benefits
Control Plane C++ Type-Safe FSM Dynamic Runtime Allocation Monolithic Scheduling Zero runtime memory dangling risks
Kernel Management Pluggable Registry & Public API Hardcoded Kernel Binding Static Engine Compilation Heterogeneous accelerator swaps in hours
Agentic Workload Native MLA & Long Context Throughput-Optimized Batching General Batching Doubled throughput on multi-step agents
Ecosystem LightSeek Non-Profit Foundation Commercial Vendor Backing Single-Vendor Lock-in Long-term neutrality and cross-platform support

TokenSpeed shifts focus away from raw batch throughput toward agentic latency profiles. Delivering 685 tokens/s per user on NVIDIA GB300 confirms the validity of moving state validation to the compilation phase.

4. Getting Started

Deploying TokenSpeed in a production environment requires matching CUDA drivers and dependency specifications. Execute the following sequence to launch a minimal serving instance.

# Clone the repository and navigate to the working directory
git clone https://github.com/lightseekorg/tokenspeed.git
cd tokenspeed

# Install build dependencies and Python execution environment
pip install --upgrade pip
pip install -e .

# Launch an inference server instance with Qwen3.8-Flash-Next
python -m tokenspeed.entrypoints.async_llm \
    --model-path /models/Qwen3.8-Flash-Next \
    --tensor-parallel-size 8 \
    --max-num-seqs 256 \
    --port 8000

Once launched, the engine initializes the control plane FSM and exposes an OpenAI-compatible API endpoint, automatically handling request lifecycles and cache reuse.

5. Production Gotchas

Deploying this inference engine at scale requires careful attention to toolchain versions and memory reservation parameters to prevent runtime instability.

⚠️ Gotcha: Toolchain Version Mismatch: Because the C++ control plane relies heavily on modern template metaprogramming and strict typing, compiling with GCC versions older than 13 or Clang below 16 will trigger build failures. Use the official container images for production builds.

⚠️ Gotcha: KV Cache OOM Risks: Agentic workloads involving multi-turn tool outputs can rapidly exhaust GPU memory if context length parameters are unbounded. Always explicitly configure --max-num-seqs and memory reservation flags based on load testing peaks.