In the previous article, we introduced how to realize fast object detection and tracking by using RZ/A2M with unique DRP technology integrated for industrial automation. Now, let's see how RZ/A2M's DRP detects five different barcode encoding formats within 8ms.
A two-dimensional barcode is a type of barcode. A common two-dimensional barcode is a QR Code, and QR is short for Quick Response. It has become a very popular encoding method on mobile devices in recent years. It can store more information as well as more data type than traditional Bar Code. In addition to this common QR Code, there are also 2D barcodes in other encoding formats, such as Micro QR Code, Data Matrix, Aztec, and Han Xin codes. Therefore, the scanner is required to be able to recognize a variety of 2D barcodes with different code types. A common operation is to call different decoders cyclically and use different decoders to detect and decode one by one, but the problem is that the decoding time varies and the decoding time is too long. It will greatly improve the decoding efficiency if the QR code encoding format can be quickly identified before decoding the formats, and then call the corresponding decoder in a targeted manner.
Renesas’ RZ/A2M microprocessor can solve this problem perfectly with its unique Dynamically Configurable Processor (DRP). It can quickly identify 5 encoding formats including QR, Micro QR, Data Matrix, Aztec, and Han Xin code in 8ms, when data is received from a 1280x720 (1 million) camera. Let’s see how DRP is realized below.
Following are five types of QR codes as examples. We will use the image processing acceleration function of DRP to identify the characteristics of each encoding format
We will extract the barcode symbols from the image data that was input by the MIPI interface through nine steps.
Among the nine steps, most processing steps are processed by the DRP hardware acceleration module (in blue), and a small number of processes are processed by the CPU (in green). During this period, we can see the flexibility and seamless matching connecting between hardware DRP processing and software CPU processing.
Step 1
In order to speed up the image processing, we first load a Bayer2GrayThinning DRP acceleration library to downscale the sample image resolution to 1/4 of the original size and then convert the image format from Bayer to grayscale. The brightness level of each frame of the image is determined and then fed back to the CMOS sensor to adjust the automatic exposure parameters
Since the Bayer2GrayThinning library only occupies one tile of DRP hardware resources, in this step, the multi-tile parallel processing feature of DRP is used to divide each frame of 1280x768 image into six 1280x128 regions, which are parallelized by six DRP processing units. In this way, the processing speed can be increased by six times, and the pixel-level processing inside each processing unit is also parallelized.
Utilizing these two parallel features of DRP, we can complete the processing of one frame of image in 1280x768 resolution in 0.6ms, including 0.2ms DRP library loading time and 0.4ms image data processing time. After processing in Step 1, we can get a 320x192 grayscale image.
Step 2
After getting the grayscale image, we load a binarization adaptive DRP library and convert the image into a black and white image in 0x00 and 0xFF. This DRP library can dynamically calculate the binarization threshold based on the average gray value of each 8x8 pixel area, which can effectively solve the problem of how to choose the binarization threshold in scenes with different brightness.
We can set the parameters of the binary conversion and control the output image to be output in reverse mode, that is, the black part of the two-dimensional barcode image is output as white, so that it is convenient to extract the contour information of the white part in the next step.
The total processing time of Step 2 is about 0.9ms, including 0.6ms DRP library loading time and 0.3ms image data processing time.
Step 3
After obtaining the binarized image, we load a Dilate DRP library to implement the expansion function. The basic implementation principle is as follows;
- Scan the input image with a 3x3 block with a step size of 1 pixel
- The new data output at the center of the scan window is the maximum value within the range of 3x3
- Realization effect: Expand the white area and swallow the enclosed black area
- The number of iterations can be configured, the more times the larger the white extension range
In the expanded image, the area containing the QR code data is basically connected to form a large area containing the entire QR code.
The total processing time is about 0.4ms, including 0.2ms DRP library loading time and 0.2ms image data processing time.
Step 4
At this point, by calling OpenCV’s FindContour method for dilated image, we can get the outer contours of all the highlighted parts in this frame of the image and the smallest bounding rectangle of each contour.
We will determine the aspect ratio, area, location, and other information of the contour one by one and filter out the contour most likely to contain the QR code.
If they don’t match, we will sample the next frame.
This step is implemented by the CPU and will take about 4ms.
The rectangle with a rotation angle in the figure is the outline of the QR code area. Later we will adjust this rotated rectangle to a graph with an inclination of 0 degrees for finer feature detection.
The coordinates of the four vertices of the smallest bounding rectangle will be converted into the coordinates of the original input image, which are used to crop the area containing the QR code from the original image.
Step 5
To ensure the accuracy of the identification feature extraction of the QR code, we will crop the original QR code image in Bayer format from the MIPI input buffer.
At this time, we load a bayer2gray_cropping DRP library to crop the image from the MIPI buffer. The cropping position is the coordinates of the four vertices of the smallest bounding rectangle in Step 4 in the MIPI buffer.
This bayer2gray_cropping only occupies the hardware resources of one tile and has the ability to run six tiles in parallel, so we will use 6 Tiles DRP for parallel processing.
It will take about 0.6ms for this step, the loading time of DRP is 0.25ms, and the execution time is about 0.35ms
Step 6
The left side of the figure below is a cropped grayscale image with a rotation angle, in which four green vertices can be obtained from the contour information in Step 4.
In this step, we use the CPU to implement a simple Keystone and zoom function to map the images in the four green vertices to the buffer on the right. The keystone correction here is simply to extract or repeat the original pixels without generating new points.
Therefore, it is a lossy but fast algorithm.
A more accurate algorithm can use the affine function in DRP, but it will consume more time.
Step 7
After getting an image without rotation, we repeat the binarization adaptive processing in Step 2 to obtain a binarization image without rotation angle.
It takes about 0.5ms for this step.
Step 8
Load the FindContour DRP library at this time to detect the contours of all the highlighted parts in the two-dimensional code image after binarization, and the detection result is the blue rectangular area in the image.
FindContour DRP library will output the location and size data of each contour.
All contour information will be saved for the next step 2d barcode symbols detection.
This process takes about 1.4ms in total, including 0.4ms for DRP library loading and 1.0ms for processing.
Step 9
At this point, we have obtained the coordinate information of all contours in the QR code image. We can easily recognize if they are QR code, Micro QR code, Han Xin code, or Aztec code by the corresponding position of the contour relative to the QR code, whether it is a square contour, contains a symmetrical sub-contour, or contains an eccentric sub-contour.
For Data Matrix detection, we can take advantage of the feature that DRP does not consume CPU load when DRP is running. During the DRP processing in step 8 (1ms DRP execution period), the CPU will calculate the sum statistics of each row and column of the image at the same time, so that enables DRP and CPU run in parallel
Let us review the following nine steps of processing:
We can complete the detection of five different coded QR code types within 8ms, mainly due to:
- Implementation of the DRP library with parallel processing capabilities
- Implementation of multiple DRP libraries at the same time to process multiple areas of a frame of the image at the same time
- During DRP processing, run multiple CPU operations in parallel.
- Seamlessly implement Hardware DRP acceleration processing and OpenCV software processing
At this time, if the cropped two-dimensional code image is handed over to the subsequent corresponding decoder for decoding, it will greatly shorten the processing time of multi-type two-dimensional code decoding.
You can see the result of the real-time detection of a fast-rotating and transform-encoding image in the following video.
