Deploying Kimi K3 on a B200 SageMaker cluster: a runbook for Kara

This post is for Kara, who asked for a straight answer. Kimi K3 is a 2.8 trillion parameter open-weight model from Moonshot AI. Its MXFP4 weights are roughly 1.4 TB, and the practical minimum for self-hosting is a multi-node cluster of NVIDIA B200 or GB200 GPUs. Below is the deployment path we would follow on Amazon SageMaker HyperPod.

The goal is to serve the model with an OpenAI-compatible /v1/chat/completionsendpoint, using vLLM or SGLang as the inference engine, across a cluster of B200 nodes connected by high-bandwidth RDMA or NVLink. This is not a single-GPU workload. Even a full DGX B200 is not enough for the dense checkpoint; you need multiple nodes.

What you are actually provisioning

Start with a SageMaker HyperPod cluster backed by NVIDIA B200 instances. The exact instance SKU will depend on what AWS has released in your region, but you are looking for a multi-node GPU cluster with the following properties:

• At least two nodes of 8× B200 each, for 16 GPUs total. This is the working minimum.
• RDMA or NVLink-scale interconnect between nodes. Do not attempt this over vanilla ENA.
• Aggregate GPU memory of at least 1.6 TB to hold the MXFP4 weights, KV cache, and overhead.
• A shared parallel filesystem or high-throughput S3 staging path for the 1.4 TB checkpoint.

For a production-grade deployment, Moonshot recommends a supernode-class configuration with 64 or more accelerators. On AWS, that translates to SageMaker HyperPod using P6e-GB200 UltraServers or an equivalent B200-backed fleet. Plan for reserved capacity; on-demand availability for B200/GB200 clusters is tight.

Stage the checkpoint

Download the open weights from the Moonshot Hugging Face repository (moonshotai/Kimi-K3). The full checkpoint is approximately 1.56 TB on disk and ~1.4 TB of effective weights in MXFP4. Stage it in one of two places:

• S3 + FSx for Lustre: Keep the canonical weights in S3 and mount them via FSx for Lustre on the HyperPod cluster. This is the most common pattern for multi-node inference because it gives every node the same read path.
• Local NVMe cache per node: If your nodes have large local NVMe, download once to node-local storage and reuse it across restarts. This avoids pulling 1.4 TB over the network on every pod reschedule.

Do not expect to stream the weights from Hugging Face directly into a SageMaker endpoint at boot time. The download is too large and the endpoint health check will time out. Pre-stage everything before creating the endpoint.

Choose the inference engine

Both vLLM and SGLang have day-0 support for Kimi K3. We default to vLLM for production because its prefix caching and continuous batching are well exercised, but SGLang is a perfectly valid choice if your team already runs it. The engine must support:

• Tensor parallelism across the GPUs inside a node.
• Pipeline parallelism across nodes, or expert parallelism for the MoE layers.
• Kimi Delta Attention (KDA), the hybrid linear attention mechanism used by K3.
• MXFP4 weight loading and the K3-specific model architecture.

Use a container image that ships the correct vLLM or SGLang revision. Do not reuse a generic PyTorch image and pip-install the engine at runtime; the build is fragile and the model metadata parser needs the exact version that knows about KimiK3ForConditionalGeneration.

Configure the serve command

A representative multi-node vLLM launch looks like this. The exact parallelism dimensions depend on your node count and GPU topology, so treat the numbers as a starting point rather than a copy-paste guarantee:

python -m vllm.entrypoints.openai.api_server \
  --model moonshotai/Kimi-K3 \
  --tensor-parallel-size 8 \
  --pipeline-parallel-size 2 \
  --dtype mxfp4 \
  --max-model-len 131072 \
  --gpu-memory-utilization 0.92 \
  --enable-prefix-caching \
  --trust-remote-code

For two 8×B200 nodes, tensor-parallel-size 8 andpipeline-parallel-size 2 is a reasonable first topology. If you expand to four nodes, you can increase pipeline parallelism or add expert parallelism for the MoE layers. The K3 model card and vLLM docs publish verified recipes for specific node counts; use them before inventing your own topology.

Expose the SageMaker endpoint

On SageMaker HyperPod, you create an InferenceEndpointConfig that points to your model artifact, container image, and cluster. The critical fields are:

• modelName: Kimi-K3
• modelSourceConfig: the Hugging Face model ID or your S3/FSx path
• instanceType / cluster: your B200-backed HyperPod cluster
• invocationEndpoint: v1/chat/completions

SageMaker will poll GET /ping during startup. Because K3 takes a long time to load weights and compile CUDA graphs, the endpoint can appear stuck for several minutes. Make sure your container returns 200 from /ping as soon as the server process is alive, even if the model is not fully warmed up yet. The actual first request will be slow; that is expected.

Verify before you announce it is ready

Run a small validation suite after the endpoint reaches InService:

• Send a short prompt and confirm the response parses and the tokens are coherent.
• Send a long-context prompt to exercise the KV cache and attention path.
• Load-test with a small concurrency sweep to find the throughput knee.
• Check GPU memory utilization on every node; it should be high but not at 100%.
• Confirm RDMA counters show traffic across nodes, not just intra-node.

Cost and operational reality

A B200/GB200 cluster at this scale is expensive. Self-hosting K3 only makes sense if your monthly token volume is high enough to beat the Moonshot API pricing, or if you have a hard data-residency requirement. Otherwise, use the Kimi API Platform and let Moonshot run the supernode.

If you do operate the cluster yourself, plan for: weight updates when Moonshot releases revisions, checkpoint integrity checks before every endpoint creation, and a fast rollback image because a 1.4 TB weight load does not recover quickly from a bad configuration.

Kara, that is the honest shape of it. The hardware is real, the model is real, and the work is mostly weight staging and parallelism bookkeeping.