DeepInside DeepGEMM: Unpacking DeepSeek's Unified Tensor Core Library for SM90 and SM100

1. The Core Bottleneck: What Engineering Deadlock Does It Break?

Modern large language models scaling mixture-of-experts (MoE) and low-precision quantization (FP8/FP4) training and inference frequently hit a wall of template bloat and customization bottlenecks in low-level operator libraries. Maintaining heterogeneous operators forces developers to confront immense compilation overheads from massive CUTLASS class-template hierarchies and runtime crashes caused by rigid environment configurations. GEMMs, fused MoEs, MQA scoring, and communication overlap logics are often scattered across isolated repositories without a cohesive memory access pipeline.

DeepGEMM eliminates these historical burdens with a unified codebase. It strips away deep dependencies on massive template frameworks, consolidating core LLM computation primitives into a lean collection of C++/CUDA code. The system relies entirely on DeepJIT for runtime dynamic compilation, avoiding any offline CUDA source builds during installation. This aggressive design hands hardware control back to developers, maximizing matrix multiplication throughput for NVIDIA SM90 and SM100 architectures.

💡 Architectural Core Insight: By discarding redundant template abstractions and introducing minimalist runtime compilation, DeepGEMM reduces underlying hardware control to pure memory layout constraints and TMA alignment.

2. Core Architecture and Data Flow Analysis

DeepGEMM's execution flow is driven by a Python frontend that dispatches computation requests directly to the underlying runtime via a lightweight C++ extension. The framework bypasses multi-layer abstraction overheads, interfacing directly with NVIDIA Hopper and Blackwell hardware features. Data is aligned via TMA (Tensor Memory Accelerator) during the loading phase before entering the DeepJIT engine for kernel assembly and immediate dispatch.

[ Python Frontend ] ---> [ C++ Extension / DeepJIT ] ---> [ Runtime JIT Engine ]
                                                                   │
                                                                   ▼
[ SM90/SM100 Hardware ] <--- [ TMA Aligned Memory Layout ] <--- [ Optimized CUDA Kernels ]

Regarding memory layout, asymmetric quantization imposes rigorous engineering constraints. The SM90 architecture strictly enforces an NT memory layout for inputs, requiring left scaling factors to maintain a TMA-aligned transposed arrangement in FP32 format. Conversely, the SM100 architecture fully supports NT, TN, NN, and TT layouts, upgrading scaling factors to packed UE8M0 format (packing four UE8M0 values into a single torch.int). This strict control over memory layouts eliminates implicit transposition overhead during data movement.

3. Technical Selection and Hardware Benchmarking

These comparisons demonstrate that DeepGEMM abandons the bloated general-purpose template route in favor of a lean, hardware-extremizing strategy. For teams pursuing absolute throughput, bypassing offline compilation time and breaking free from black-box libraries provides a decisive edge in building production-grade LLM infrastructure.

4. Hands-on Geek Guide: Building a Minimal Closed Loop

This section demonstrates how to clone dependencies, initialize the environment, and execute a basic FP8 GEMM computation on nodes equipped with NVIDIA SM90 (e.g., H800) or SM100 architectures.

Prerequisites: - NVIDIA SM90 or SM100 architecture GPU - Python 3.8 or higher - Compilers with C++20 <format> standard library support - CUDA Toolkit 12.9 or higher - PyTorch 2.3 or higher

Execute the following commands to clone the repository and initialize submodules:

# Must clone recursively to include CUTLASS submodule
git clone --recursive [email protected]:deepseek-ai/DeepGEMM.git
cd DeepGEMM

# Link essential include files and build the lightweight C++ extension
cat develop.sh
./develop.sh

# Execute global installation script
cat install.sh
./install.sh

Here is a minimal production-grade Python script invoking native dense FP8 GEMM. Every key tensor parameter and dimension alignment is annotated:

import torch
import deep_gemm

# Define matrix dimensions: M, N, K must satisfy hardware alignment constraints
m, n, k = 4096, 4096, 4096

# Create left input matrix A and cast to FP8 format
a = torch.randn((m, k), device='cuda', dtype=torch.float16).to(torch.float8_e4m3fn)
# Create right transposed matrix B matching DeepGEMM's NT layout requirement
b = torch.randn((n, k), device='cuda', dtype=torch.float16).to(torch.float8_e4m3fn).t()

# Create FP32 left scaling factor as required by SM90 architecture
# Production code must ensure scaling factor has TMA alignment and transposed layout
a_sf = torch.randn((m, k // 128), device='cuda', dtype=torch.float32)
b_sf = torch.randn((n, k // 128), device='cuda', dtype=torch.float32)

# Initialize bias or accumulation matrix C (can pass zero tensor here)
c = torch.zeros((m, n), device='cuda', dtype=torch.bfloat16)

# Invoke DeepGEMM's standard non-grouped FP8 NT GEMM kernel
# Computation formula: D = C + A @ B.T
output = deep_gemm.fp8_gemm_nt(a, b, a_sf, b_sf, c, torch.bfloat16)

print("Execution completed. Output tensor shape:", output.shape)
print("Output tensor dtype:", output.dtype)

Running this script yields output tensor shape torch.Size([4096, 4096]) with torch.bfloat16 dtype, compiled dynamically at runtime.

5. Production Gotchas and Deployment Pitfalls

Deploying DeepGEMM in production clusters often triggers subtle crashes caused by hardware alignment and data preprocessing oversights. Two critical engineering traps require rigorous mitigation.

⚠️ Pitfall Warning [Input Tensor Layout and Type Mismatch]: Under the SM90 architecture, input matrices are strictly restricted to an NT memory layout, with scaling factors required to be in FP32 with a TMA transposed arrangement. Meanwhile, the SM100 architecture mandates packed UE8M0 formats (four values packed into a single torch.int). Passing standard PyTorch row-major tensors or untransposed scaling factors directly will trigger segmentation faults or numerical corruption. The solution is to handle transposition and scaling factor packaging inside dedicated prior kernels before calling fp8_gemm.

⚠️ Pitfall Warning [Performance Degradation from Implicit PyTorch Utilities]: Although the library includes auxiliary PyTorch utility functions for input transposition or FP8 casting, these pure PyTorch implementations lack rigorous pipeline optimization. Invoking these utilities directly inside high-throughput production inference paths introduces severe memory bandwidth bottlenecks. The solution is to fuse data input and FP8 casting logic directly into custom prior CUDA kernels, ensuring pointers passed to DeepGEMM satisfy memory contiguity and alignment conditions out-of-the-box.