Can You Run an ONNX Model on the RK3588 NPU? RKNN Compatibility Checklist
Short answer: usually yes, but not by copying a raw .onnx file to the RK3588 and assuming the NPU will execute it. Rockchip's documented path is to use RKNN-Toolkit2 on a development computer to convert the model into .rknn format, then run that model on the board through RKNN-Toolkit-Lite2 or the RKNN Runtime.
Whether your model is actually deployable depends on more than its file format. You must verify four separate gates:
- the ONNX model is valid and produces the expected baseline output;
- RKNN-Toolkit2 can compile the exact graph for
rk3588; - the converted model produces acceptable outputs on the board;
- the complete application meets its accuracy, latency, memory, and stability targets.
A successful conversion only passes the second gate. It does not prove the other three.
Why an ONNX File Is Not an NPU Compatibility Guarantee
ONNX is an open model format designed to improve interoperability between AI frameworks and tools. It describes a computation graph, operators, and data types. It does not require every hardware backend to implement every operator, data type, shape pattern, or opset version.
RKNN-Toolkit2 has its own operator support and compiler restrictions. A graph may contain a familiar operator but still use an unsupported attribute, tensor rank, input type, or dynamic behavior. Custom layers, control flow, unusual post-processing, and export-specific graph patterns need particular attention.
Do not assume an unsupported operator will transparently fall back to the CPU. Depending on the model and software design, conversion may fail or the deployment may require a graph rewrite, a supported alternative, a custom operator, model partitioning, or an application-managed CPU path. The conversion log and the resulting application are the evidence.
The Official RKNN Deployment Path
Rockchip's current RKNN software stack separates model preparation from board-side inference:
| Stage | Where it runs | What it proves |
|---|---|---|
| Export and validate ONNX | Training or development environment | The exported model loads and reproduces the expected baseline behavior |
| Convert with RKNN-Toolkit2 | Development computer | The compiler accepts the graph for the rk3588 target and produces an RKNN model |
| Run with Lite2 or Runtime | RK3588 board | The board can load and execute the RKNN model with the selected firmware, driver, and Runtime |
| Validate the complete pipeline | Target application on the board | Preprocessing, inference, post-processing, memory, latency, accuracy, and recovery meet project requirements |
The official RKNN Model Zoo is a useful starting point because it provides conversion and inference examples for model families including classification, detection, segmentation, pose, OCR, and audio. A Model Zoo example proves that its documented model and workflow were supported under its stated conditions. It does not automatically validate a custom fork, a differently exported graph, or your complete camera pipeline.
Gate 1: Check the ONNX Model Before Conversion
Start with a known-good ONNX baseline. If the source model is already ambiguous, RKNN conversion will make troubleshooting harder.
Record the following before opening RKNN-Toolkit2:
- the exact model name, training revision, and ONNX file hash;
- the ONNX opset and exporter version;
- every input name, shape, data type, layout, and allowed range;
- whether shapes are fixed or dynamic;
- resize, crop, normalization, channel order, and color format;
- output names, shapes, decoding logic, thresholds, anchors, and non-maximum suppression;
- a representative validation set and the baseline accuracy or output tolerance.
Run the ONNX model in its source validation environment and save representative inputs and outputs. This baseline lets you tell the difference between an export problem, an RKNN conversion problem, and an application integration problem.
Check operators and opset together
Compare the graph with Rockchip's current RKNN Toolkit2 operator support list and compiler restrictions. Do not check operator names alone. Review any restrictions on shapes, attributes, batch size, inputs, or data types that apply to your graph.
Rockchip's current public support list is aligned to ONNX opset 19, while the RKNN-Toolkit2 changelog also documents support work across earlier opsets and releases. This is not a promise that every graph exported with a listed opset will compile. Test the exact file with the exact toolkit release you intend to use.
Treat dynamic shapes as a design decision
Fixed input shapes are normally the simpler deployment path. RKNN has limited dynamic-shape support, but support depends on the model, toolkit, and board-side Runtime version. If your application needs several resolutions, define the required shape set and verify it explicitly. Do not replace a real dynamic requirement with a fixed input simply to make conversion succeed unless the application can accept that change.
Gate 2: Inspect the RKNN Build, Not Just the Output File
Configure the converter for the rk3588 target and preserve the complete build log. Warnings about graph rewrites, unsupported operations, type conversion, or quantization are engineering inputs, not console noise.
-
Target platform: confirm that the build is for
rk3588, not a different RKNPU generation. - Input configuration: verify shapes, layouts, mean values, standard deviations, and channel order against the ONNX baseline.
- Precision path: decide whether FP16, INT8, or another supported path is appropriate for the exact model and accuracy target.
- Quantization data: if using INT8, use representative calibration inputs from the real operating distribution. Calibration data is not a substitute for a separate validation set.
- Compiler output: review every warning and save the toolkit version, configuration, scripts, and generated model together.
If INT8 accuracy is unacceptable, Rockchip's Model Zoo FAQ points to hybrid quantization or quantization-aware training as possible approaches. These are options to test, not guaranteed repairs. Always compare the converted model against the saved baseline on representative data.
Gate 3: Match the Board Runtime and Firmware
The host converter is only one part of the stack. The board also uses an RKNPU driver, RKNN Runtime library, and, for connected debugging, rknn_server. A model built with a newer toolkit can behave incorrectly when the board image contains an older or incompatible runtime stack.
For the deployment record, save the RKNN-Toolkit2 version, Runtime or Lite2 version, firmware or OS image, RKNPU driver, target architecture, exact RKNN model, and application build.
Rockchip's Model Zoo states that its demos are verified against the corresponding current RKNPU SDK and warns that older SDK versions may produce incorrect inference results or performance. Obtain firmware and Runtime updates from the board vendor because images and board-specific integrations are not necessarily interchangeable.
Gate 4: Prove Correctness Before Measuring Speed
Loading an RKNN model and receiving output tensors is not enough. Compare board-side results with the ONNX baseline before optimizing performance.
- match input preprocessing byte for byte where practical;
- verify output tensor order, shape, type, and scale;
- match decoding, sigmoid, thresholds, anchors, and non-maximum suppression;
- measure task-level accuracy against the agreed tolerance;
- check failures on edge cases, not only a visually successful demo image.
Rockchip's Model Zoo FAQ documents examples where incorrect post-processing, missing sigmoid behavior, anchor mismatch, fixed-shape exports, or quantization can change YOLO results. That is why a working demo cannot validate a swapped-in custom model without rechecking its export and post-processing.
Only after correctness passes should you measure performance. Keep NPU-only inference time separate from end-to-end application time. Rockchip notes that Model Zoo performance figures generally exclude preprocessing and post-processing. A production test should include capture or decode, resize and color conversion, memory transfers, inference, post-processing, application logic, and output.
Measure on the actual board and record sustained latency, throughput, memory use, CPU/NPU load, temperature, cooling, power mode, test duration, and competing processes. This article does not publish an FPS result for YY3588 because no controlled, reproducible model test was supplied for this guide.
A Practical Compatibility Decision Table
| Current evidence | Decision | Next action |
|---|---|---|
| Exact model matches a maintained RKNN example, uses supported graph patterns, has defined fixed inputs, and reproduces correct board outputs | Strong candidate, not production-approved | Run the complete application and sustained acceptance test |
| Conversion succeeds, but accuracy, preprocessing, post-processing, or board Runtime has not been matched | Compatibility unknown | Compare outputs with the ONNX baseline and align the software stack |
| Graph uses dynamic shapes, custom layers, unusual control flow, or operators with restrictions | Proof of concept required | Test the graph early; evaluate rewrites, custom operations, partitioning, or another runtime path |
| INT8 conversion succeeds but exceeds the allowed accuracy loss | Does not pass the precision target | Recheck calibration; evaluate FP16, hybrid quantization, QAT, or a model change |
| NPU inference is fast but the complete camera application misses latency or stability targets | System does not pass | Profile capture, decode, preprocessing, copies, post-processing, memory, thermals, and application load |
| Model cannot be converted without unacceptable changes | RK3588 NPU path is not proven | Keep a CPU/GPU or another accelerator platform in evaluation |
Where the Youyeetoo YY3588 Fits
The youyeetoo YY3588 is an RK3588 platform with a vendor-listed NPU capability of up to 6 TOPS. Its official Wiki provides the board's current system images, RKNN resources, interfaces, and development entry points.
Those facts make YY3588 a candidate host for an RKNN-compatible model. They do not prove that a specific ONNX model will convert, preserve accuracy, or meet a target FPS. The 6 TOPS rating is a hardware capability, not a model compatibility or application performance guarantee.
Before asking for a compatibility recommendation or ordering a prototype, prepare this minimum package:
- the exact ONNX model or a shareable representative graph;
- opset, exporter, input shapes, data types, and dynamic-shape requirements;
- preprocessing and post-processing code;
- representative calibration and validation inputs;
- baseline outputs and allowed accuracy difference;
- target OS, latency, sustained throughput, memory, and camera requirements;
- any operators, licenses, or model files that cannot be shared.
Next step: With that information, the useful next step is a model-specific feasibility check and prototype test, not a promise based on the .onnx extension or the NPU's TOPS rating. Contact youyeetoo with the minimum package above.
For the broader hardware decision, read Affordable SBCs for Real-Time Computer Vision: What to Check Before You Buy. If you are deciding between the ARM/RKNN path and an existing x86 software stack, see RK3588 vs Intel N100 for an Embedded Project.
Frequently Asked Questions
Can RK3588 run an ONNX model directly?
The documented Rockchip workflow normally converts ONNX to RKNN on a development computer, then runs the .rknn model through RKNN-Toolkit-Lite2 or RKNN Runtime on the RK3588 board. Do not assume that copying a raw ONNX file to the board uses the NPU.
Does a successful RKNN conversion prove the model is supported?
It proves that the compiler produced an RKNN file for that configuration. You still need to compare outputs and task accuracy, verify the board Runtime and firmware, and test the complete application under representative conditions.
Should I use INT8 or FP16?
Choose by the exact model, supported path, accuracy tolerance, memory target, and measured application result. INT8 requires representative calibration data and post-conversion accuracy validation. If accuracy loss is unacceptable, test a supported alternative such as FP16, hybrid quantization, QAT, or a revised model.
Are dynamic input shapes supported?
RKNN has limited dynamic-shape support, but it is not equivalent to unrestricted dynamic execution. Verify the permitted shapes, model restrictions, and required board Runtime version for your exact deployment. Fixed shapes are usually easier when the application can use them honestly.
Does an unsupported operator automatically run on the CPU?
Do not rely on automatic CPU fallback as a general rule. The result may be a conversion failure or may require graph changes, a custom operator, model partitioning, or an application-managed CPU path. Test and document the actual behavior.
Sources
Facts and software behavior in this guide were checked on July 27, 2026 against the following first-party resources:
- ONNX project overview
- Rockchip RKNN-Toolkit2
- RKNN Toolkit2 ONNX operator support list
- Rockchip RKNN Model Zoo
- Rockchip RKNN Model Zoo FAQ
- Youyeetoo YY3588 Wiki
Toolchain versions and support lists change. Recheck the current Rockchip release and the board image before freezing a production deployment.