AIMA Intel PTL Qwen3.6 35B U4 Engine: Specialized Intel GPU Inference

Why Build a Dedicated Intel Engine

General inference frameworks are built to serve many models, hardware variants, and batch shapes, so the real bottlenecks of a fixed model on a fixed device often remain at a "generally usable" level.

This project does the opposite: it locks down the Intel PTL target machine, the Qwen3.6-35B-A3B OpenVINO U4 model, and the same-machine stock OpenVINO GPU baseline, then finds bottlenecks with oracle, roofline, and layer-by-layer validation. Every optimization must answer two questions at once: how much faster it runs than stock, and whether the model behavior changed.

Product Positioning

AIMA Intel PTL Qwen3.6 35B U4 Engine is a batch-size-1, single-machine, single-model OpenVINO GPU-specialized inference engine plus a resident HTTP service.

• Target hardware: Intel PTL CLS DVT2 / Core Ultra X7 358H / Intel Arc B390 GPU
• Target memory: 64 GB-class LPDDR
• Target system: Ubuntu 24.04.4 LTS
• Target model: Qwen3.6-35B-A3B OpenVINO U4
• Service interface: OpenAI-style Models, Completions, Chat Completions, and Responses
• License: Apache License 2.0

This is not a performance promise for other Intel GPUs, models, precisions, or batch sizes.

What the Performance Optimization Did

Same-Machine Stock Baseline, Not a Different Environment

The performance baseline is the unmodified stock OpenVINO GPU U4 worker on the same machine. Candidate and stock use isolated plugin, cache, and config paths; each case runs at least 8 interleaved ABBA blocks, followed by 20,000 bootstrap resamples on the paired median ratios.

What is reported is not a cherry-picked set of single-run ratios, but the one-sided 95% lower confidence bound (LCB). Each bucket also includes three prompt classes (filler, prefill-shape, and long-context sentinel), and the worst class decides whether that bucket passes.

Two GPU-Specialized Paths: Short and Long Context

The engine specializes the OpenVINO GPU stack around the locked model and uses separate profiles for short and long contexts. Optimizations cover model-specific OpenCL/Level Zero execution, dynamic quantization, the LM head, operator fusion, memory layout, and reuse; the service then routes ordinary inputs to the smallest fitting bucket automatically, so callers do not need to pad or select internal shapes.

Tying the Performance Gate to the Correctness Gate

Every candidate is run through 512-token greedy generation against the stock worker and a per-position teacher-forced full-vocabulary comparison. Any first-token divergence, top-1 agreement below 0.99, or out-of-bounds KLD blocks the performance results from an official release.

v0.1.0 Official Performance Results

The official matrix covers 7 prompt buckets with 3 prompt classes each, totaling 21 / 21 cases; each case generates 512 tokens.

In the most conservative reading, across all official buckets and prompt classes, the candidate's prefill speedup lower bound over the same-machine stock OpenVINO GPU is at least 1.479×, and the decode speedup lower bound is at least 1.592×. Long-context gains are larger: the 128K bucket's worst lower bounds reach 1.813× and 1.890×.

Full 21-case data, confidence intervals, jitter, and memory records are in the Performance and Correctness Report.

Correctness and Stability Gates

The performance gain did not come at the cost of final model behavior:

• 10,752 / 10,752 greedy tokens are bit-for-bit identical to the stock OpenVINO GPU on a per-token basis
• teacher-forced top-1 agreement minimum is 1.0
• maximum KLD is 0.004836565, below the 0.005 threshold
• all 7 long-context sentinels pass retrieval on both candidate and stock
• all 336 jitter records pass, with a maximum decode TPOT P95/P50 of 1.162728
• no OOM or memory-guard events in 712 memory observations

These metrics measure whether the specialized engine preserves the locked model's behavior relative to stock OpenVINO, not the model's own knowledge or capability scores.

Current Release Boundary: Performance Numbers and the Fix Release Must Be Viewed Separately

The released v0.1.0 passed the exact-bucket official matrix above, but exposed a physical/logical buffer layout inconsistency in the LM head for near-bucket long prompts in real service: inputs of 16,380, 32,758, 65,519, and 131,037 tokens would fail during the final prefill output conversion. Therefore, do not treat v0.1.0 as a stable version for arbitrary-length long-context traffic.

The repository's current v0.1.1 release candidate fixes this issue and has passed:

• all 4 near-bucket long prompts now return HTTP 200
• 131,072-token maximum input validation passes
• fast service tests 67 / 67, real HTTP smoke 18 / 18
• 16,380-token full-vocabulary comparison top-1 8 / 8, maximum KLD 0.000092598
• the fixed long plugin can be rebuilt bit-for-bit from source

But the new plugin fingerprint has not yet completed the full 21-case output512 ABBA8 successor gate, so the v0.1.1 candidate does not inherit v0.1.0's 1.48× / 1.59× performance claims. Fix evidence and exact boundaries are in the near-boundary machine-readable record.

Service Capabilities

The resident service supports:

• /v1/models, /v1/completions, /v1/chat/completions, /v1/responses
• JSON and SSE streaming
• function tools, structured output, and Responses state
• bounded prefix-state reuse, queuing, timeouts, cancellation, and graceful shutdown
• bearer authentication, readiness, Prometheus metrics, and structured logging
• input up to 131,072 tokens, output up to 512 tokens, with no silent truncation

Production startup validates the exact fingerprint of the locked model, plugin, OpenVINO/GenAI runtime, and custom config. The model is about 19.7 GB and is not included in the repository or release assets. Deployment steps are in the service documentation.

Author and Repository Relationship

The project was created and is maintained by Jiawei Guan / 关嘉伟 (@skyguan92).

• Personal original upstream: skyguan92/AIMA-intel-plk-qwen36-35b-u4-engine
• Organization fork and main website showcase version: Approaching-AI/AIMA-intel-plk-qwen36-35b-u4-engine