AIMA AMD395 Qwen3.6 35B Windows Engine — Native Windows Inference

Why Build a Native Windows Engine

The unified memory on the AMD Ryzen AI Max+ 395 lets 35B-class BF16 models fit on a single Windows workstation, but there is a long gap between "it boots" and "it runs as a service": model loading, long contexts, operator scheduling, streaming interfaces, process lifecycle, and overload protection all have to be solved together.

This project keeps the scope tight: only Qwen3.6-35B-A3B BF16, only AMD395, only Windows 11, and only the batch size 1 inference path—made deep, fast, and product-ready.

Product Positioning

AIMA AMD395 Qwen3.6 35B Windows Engine is a model- and hardware-specific native inference engine, not a generic graph runtime.

• Target hardware: AMD Ryzen AI Max+ 395 / Radeon 8060S (gfx1151)
• Target model: Qwen3.6-35B-A3B BF16
• Target OS: Windows 11 x64, ROCm HIP SDK 7.1
• Execution mode: batch size 1, up to 512-token deterministic greedy decode
• Service interface: OpenAI-compatible Completions, Chat Completions, and SSE streaming
• License: Apache License 2.0

Model weights and AMD runtime libraries are not distributed with the repository; users must obtain them separately and match the pinned runtime environment.

Performance Optimizations

Compressing the generic path into a model-specific data path

The inference core is implemented in C/C++/HIP, with AOT kernels precompiled for gfx1151. The runtime assembles CK attention, Triton selected-MoE, AITER/FLA GDN, and host BF16 helper paths according to the model structure, keeping Python or generic frameworks off the timed hot path.

Users do not need to hand-pick a benchmark shape for continuous input lengths. The engine internally decomposes any prompt into optimized q8192 tiles, q1024, and tail paths, then executes through the 40-layer model.

Resident model with transactional prefix reuse

The model, execution plan, and cache are held by a resident provider. Compatible requests reuse the longest token prefix via copy-on-write; a new cache snapshot is committed only after inference succeeds, so failed requests cannot pollute existing cache.

This optimization is most visible with long contexts: a cold 131,072-token request has a TTFT of 108.563 s; when reusing a 131,072-token prefix and appending 1,024 tokens, the TTFT is 8.592 s, about 1/12.6 of the former. The prefix throughput here is computed against the full effective prompt as reuse throughput and should not be equated directly with cold-prefill compute throughput.

Service layer that does not mask native performance

The Rust service layer handles HTTP, SSE, OpenAI schema, queueing, and lifecycle, while the native provider owns model memory and all timed inference paths. The hardware path is fixed at batch 1, and a bounded FIFO queue provides explicit 429 / 503 overload semantics so waiting requests do not consume resources indefinitely.

Measured Performance

The data below comes from real model tests of v1.0.0 on Windows 11, AMD Ryzen AI Max+ 395, Qwen3.6-35B-A3B BF16. TTFT excludes the ~19.94 s model-and-engine loading time.

* The prefix row shows reuse throughput calculated over the full effective prompt.

At the 8K release threshold, the final results versus the acceptance boundary are:

• prefill 2,126.19 tok/s, 41.1% above the 1,506.41 tok/s lower bound
• decode 30.55 tok/s, 8.5% above the 28.17 tok/s lower bound
• TTFT 3.853 s, 8.0% below the 4.187 s upper bound

The full 12-entry performance record is in the repo’s performance notes and machine-readable release matrix.

Correctness Is Not a Footnote to Performance

Optimized results must also pass external BF16 reference checks:

• The first token at 8K selects the same token as the reference service, with an absolute logit difference of 0.125
• Both 131K cold and long-prefix requests completed 512 / 512 token-by-token consistency checks
• MMLU-Pro on both sides is 7,486 / 12,032 (62.2174%)
• Projection mismatch between candidate and reference across all 12,032 questions is 0

The repository publishes sanitized performance, correctness, OpenAI API acceptance, and per-question MMLU-Pro evidence so the results can be audited rather than taken as screenshot peak numbers.

Capabilities and Usage Limits

The resident qrt service provides:

• /v1/models, /v1/completions, /v1/chat/completions
• JSON and per-token SSE streaming
• Streaming function tool calling and tool result continuation
• Continuous, arbitrary input token length
• Prefix cache seed, resident hit, copy-on-write, and contamination isolation
• start, status, stop lifecycle and graceful shutdown

The public HTTP profile supports a total context limit of 262,144 tokens. Non-loopback listeners require an API key by default; model weights, AMD runtime DLLs, and target drivers must be prepared separately. Installation and full commands are in the repo’s README.

Author and Repository Relationship

Created and maintained by Guan Jiawei / Jiawei Guan (@skyguan92).

• Personal original upstream: skyguan92/AIMA-AMD395-Qwen36-35B-Windows-Engine
• Organizational fork and primary website showcase: Approaching-AI/AIMA-AMD395-Qwen36-35B-Windows-Engine