This article applies to the part number: H8-M.2
“The Hailo-8 M.2 Module is an AI accelerator module for AI applications, compatible with NGFF M.2 form factor M, B+M and A+E keys. The AI module is based on the 26 tera-operations per second(TOPS) Hailo-8 AI processor with high power efficiency. The M.2 AI accelerator features a full PCIe Gen-3.0 2-lane interface (4-lane in M-key module), delivering unprecedented AI performance for edge devices. The M.2 module can be plugged into an existing edge device with M.2 socket to execute in real-time and with low power deep neural network inferencing for a broad range of market segments. Leveraging Hailo’s comprehensive Dataflow Compiler and its support for standard AI frameworks, customers can easily port their Neural Network models to the Hailo-8 and introduce high-performance AI products to the market quickly.”
Hailo-8 Overview
Getting starting with your Hailo AI Device
Hailo Developer Zone
Go to https://hailo.ai/authorization/ to log in or set up an account for access to software downloads like their PCIe drivers, Model Zoo, Integration Tool, and other application and documentation from the Hailo team. The Developer Zone is also where customers should go to request support directly from Hailo.
Installation Instructions
Hailo provided installation documentation in their Developer Zone portal. Login and access there documentation here:
https://hailo.ai/developer-zone/documentation/hailort-v4-17-0/?sp_referrer=install/install.html#installation
Ubuntu 20.04 and 22.04 LTS
Note from the HailoRT User Guide version 4.17.0: The PCIe driver is not signed. In some systems it means that secure boot has to be disabled to load the driver. Users who wish to use secure boot while HailoRT PCIe driver is not signed, can use MOK (Machine-Owner Key), which can be used for (locally) signing third-party drivers. Please notice this is an advanced feature and is recommended for users who are familiar with the process of locally-signing drivers.
PCIe driver and Runtime (HailoRT)
Install the dependencies on the host:
sudo apt-get update && \ sudo apt-get install -y build-essential bison flex curl cmake dkms libelf-dev
Download the relevant .deb files for your platform from the Hailo Developer Zone to install the Hailo PCIe Driver and HailoRT, and copy them to the host. Install the packages using dpkg by running the following command:
sudo dpkg --install hailort_<version>_$(dpkg --print-architecture).deb hailort-pcie-driver_<version>_all.deb
Accept the prompt regarding using DKMS to compile the driver. DKMS is used to build the driver for the running kernel.
Restart the machine.
Verify the Installation using the hailortcli command line utility that is installed with HailoRT:
hailortcli scan
This will display the installed Hailo devices with their PCIe hardware address, with a terminal output similar to:
Hailo Devices: [-] Device: 0000:01:00.0
Other Linux Distributions
While not expressly supported, the Hailo Developer Zone includes instructions for compiling the PCIe driver from source, with a list of dependencies included.
Windows
Windows installer for PCIe driver is available for download in the Hailo Developer Zone.
Validation Testing
Overview
The Hailo Integration Tool is used to characterize, validate and approve target platforms’ compatibility with modules assembled with Hailo-8 AI accelerator. The integration tool supports several types of tests, with different goals:
Hailo Integration Tool
- Characterization tests – designed to evaluate the properties and performance of target platforms with modules assembled with Hailo-8 AI accelerator, and to identify any issues or that may affect their performance and ensure that platforms meet the required specifications and perform optimally in their intended applications
- Production tests – designed to ensure that the target platforms with modules assembled with Hailo-8 AI accelerator, meet the required specifications and perform as intended, in a production line. All tests are marked as characterization/production/both.
When run, all boards connected to the device will have the tests run on them. Verify the available devices with lspci | grep Hailo
Logging
In case of a failure, the list of errors can also be viewed in the .hailo_integration_tool.log file.
Installation
Instructions for installing the integration tool are available in the user guide provided by Hailo
Test Configuration JSON Structure
General Configuration
| Name | Description | Default | Notes |
|---|---|---|---|
continue_to_next_test_upon_failure | if a test fails, all tests are stopped | false | When using multiple modules all tests on each defined module (from the run_modules list) run concurrently |
user_comment | Optional parameter, a text string inserted with this parameter will appear in the report header. | Can be used as a label for the report | |
run_modules | Indicates which modules should run, according to PCIe slot In a case where the user chooses to run the tool on a subset of connectedmodules, it should be indicated as:<BB:DD.F> … <BB:DD.F> | [] (empty string), implicitly defines test plan to run on all devices. You can use [“all”] as well to explicitly define this. | Example value: “01:00.0” |
Test Run List
List of supported tests with execution assignment of true/false. In a case where a test is missing from the list, “false” is assumed as default. The selection is set for all the running modules.
- Thermal Test
- Stress Test
- PCIe Configuration Test
- Performance Test
- Over Current Test
- Inference Difference Test
- PCIe Throughput Test
Example Configuration File
{
"general_configuration": {
"continue_to_next_test_upon_failure": false,
"user_comment": "Put your test report label here",
"run_modules": [],
"version": "1.17.0"
},
"test_list": {
"thermal_test": false,
"stress_test": false,
"pcie_configuration_test": true,
"performance_test": false,
"overcurrent_test": false,
"inference_difference_test": false,
"pcie_throughput_test": false
},
"thermal_test_parameters": [
{
"parameters": {},
"selected_pcie_id": []
}
],
"stress_test_parameters": [
{
"parameters": {
"tinf_prediction_run": false
},
"selected_pcie_id": []
}
],
"pcie_configuration_test_parameters": [
{
"parameters": {},
"selected_pcie_id": []
}
],
"pcie_throughput_test_parameters": {}
}
Do not delete the parameter objects even if a task is not being run, as it is part of the schema that is checked when running the integration tool. If the schema is incorrect, the tool will exit with an error before starting testing.
Tests
The tests outlined below are intended to offer verification tests with a users specific application.
Thermal Configuration Test
Purpose: Characterization
The tool confirms that the user’s thermal solution is sufficient up to a maximal power consumption of 7W and ambient temperature, or otherwise it exits with an error. Review the Integration Tool User Guide for more information about how the test works.
Parameters:
There are no configuration parameters that need to be defined for this test. Hailo recommends that this test be run first before moving on to further testing.
Stress Test
Purpose: Characterization
The tool confirms the system’s capability to run the model for a long duration, without exceeding thermal/current limit or run-time errors. The tool can also estimate the temperature after a full thermal stabilization
Parameters:
| Name | Description | Default | Notes |
|---|---|---|---|
tested_hef | Optional parameter, used to indicate the neural network to be executed during the stress test. There are default neural networks defined for use with this test, but a custom network may be used. To use a custom neural network, provide the file path to the .hef formatted file for this parameter. | “57.hef” | |
expected_fps | This parameter is mandatory if using the tested_hef parameter. | type: int | |
duration | Indicating the required duration of the stress test in minutes | 1440 | If tinf_prediction_run value is true, this parameter is ignored. |
tinf_prediction_run | Optional parameter Indicating if the tool should estimate the thermal stabilization temperature | false | type: bool |
PCIe Configuration Test
Purpose: Characterization, Production
Parameters:
| Name | Description | Default | Notes |
|---|---|---|---|
expected_pcie_generation | Expected PCIe generation | 3 | Valid values: {1, 2, 3} |
expected_pcie_lanes_number | Expected PCIe number of lanes | 4 | Valid values: {1, 2, 4} |
Performance Test
Purpose: Characterization
Hailo suggests running the performance test after the thermal solution has been validated by executing the thermal test
The tool runs several selected networks on the host and then presents a report on the host’s performance with reference to other selected platforms. The tool outputs a pass\fail message, indicating if the tool has noticed any error during the test. When the test finishes with no error – the test
passes.
The reference platforms are outlined in the Integration Tool User Guide.
Parameters:
There are no configuration parameters that need to be defined for this test.
Over Current Test
Purpose: Characterization
Hailo suggests running the over current test after the thermal solution has been validated by executing the thermal test
Verify that the platform is capable of driving the current consumption of the Hailo-8 modules’ current limit or indicate the last verified current consumption level for which the platform had properly operated.
Parameters:
There are no configuration parameters that need to be defined for this test.
Inference Difference Test
Purpose: Production
Verify and validate the board’s inference flow correctness, using pre-defined data comparison to known expected data.
The test runs inference using pre-defined data and model (HEF file – resnet_v1_50.hef) and compares the output to the expected output reference.
Parameters:
There are no configuration parameters that need to be defined for this test.
PCIe Throughput Test
Purpose: Characterization
Measure the PCIe throughput for all tested modules and verify the accumulated throughput rate is above the defined threshold.
Parameters:
| Name | Description | Default | Notes |
|---|---|---|---|
threshold | Defined threshold for PCIe throughput in Gbps. | 20 | Not applicable per module, this parameter is defined once, for all tested modules |
Developer Tools
Python
Pre-compiled wheels are distributed through their “developer zone” as opposed to a package manager like PyPi
GStreamer
GStreamer multimedia framework is a framework for creating streaming media applications. GStreamer’s development framework makes it possible to write any type of streaming multimedia application. The GStreamer framework is designed to make it easy to write applications that handle audio or video or both. It isn’t restricted to audio and video and can process any kind of data flow. The framework is based on plugins that will provide various codecs and other functionality. The plugins can be linked and arranged in a pipeline. This pipeline defines the flow of the data. The GStreamer core function is to provide a framework for plugins, data flow, and media type handling/negotiation. It also provides an API to write applications using the various plugins.
Hailo’s application framework for optimized execution of video-processing pipelines (TAPPAS) is GStreamer based.
HailoNet
HailoNet GStreamer element Infers data using the Hailo8 chip.
HailoNet is a bin element which contains a hailosend element, a hailorecv element and a queue between them. The hailosend element is responsible for sending the data received from the HailoNet’s sink to the Hailo-8 device for inference. Inference is done via the infer() API. The hailorecv element will read the output buffers from the device and attach them as metadata to the source frame that inferred them. That is why the HailoNet has only one source, even in cases where the HEF has more than one output layer.
- The data is inferred according to the selected HEF (
hef-path). - Selecting a specific PCIe device can be done with the
device-idproperty. - If the pipeline contains a single HailoNet then the single HailoNet will be activated by default, otherwise each HailoNet must explicitly specify if it is active or not by setting the
is_activeproperty. This property can also be used for switching the activated networks on a specific device during runtime. - For multi-context networks the
batch-sizeproperty can be used to specify the batch size. - In case the HEF contains multiple networks, a specific network must be specified for inference using the
net-nameproperty.
OnLogic Support Strategy
Windows
Windows customers will receive systems with the PCIe driver for the H8 card pre-configured from the factory when ordering Windows 11 or Windows 10 IoT
Linux
The customer is expected to install the driver themselves when it arrives, whether it is pre-loaded Ubuntu or a custom installation with their distribution of choice. Hailo provides a .deb package that has been validated for Ubuntu 22.04 and 20.04 LTS, which leverages DKMS to build the kernel driver for the kernel that is in-use.