An Effective Method for the Recognition and Verification of Bangladeshi Vehicle Digital Number Plates

: In this study, an effective and automatic technique has been proposed to detect, recognize and verify Bangladeshi number plates. There are four steps in the proposed method: Pre-processing, extraction and segmentation, recognition and verification of the number plate. Image contrast-enhancing and tilt correction techniques are used in the proposed method. Different techniques like edge detection, morphological operations, bounding box method and connected component analysis are used for the extraction and segmentation of number plates and their characters. The template matching method is used for recognizing the characters. The authentication and verification process for a vehicle is the distinctive feature of the proposed method. We have developed a dataset of 500 Bangladeshi number plate images for our research. A rich template dataset and a cloud database containing vehicle details have also been developed for number plate character recognition and vehicle verification purposes, respectively. The proposed method achieves detection, extraction, segmentation and recognition accuracies of 96.8, 94.8, 98.3 and 97.6%, respectively. It is observed that the proposed technique outperformed many other existing techniques.


Introduction
Bangladesh is one of the most densely populated countries in the world. The total population of Bangladesh was nearly 165 million in 2020 according to the data of the (UNSD, 2021). This huge population leads to an increase in the total number of vehicles over the years. According to the data of the (BRTA, 2019), the total number of registered motor vehicles was 40 lakhs in 2019. Besides, the number of unregistered vehicles is also increasing day by day. The huge number of motor vehicles is a threat to the traffic management system for a densely populated country like Bangladesh. It increases crimes like car theft, over-speeding, disobeying traffic rules, excessive overtaking. Along with these incidents, another issue that bothers the traffic management authority is that many car-owners do not renew the taxtoken of their registered car after the expiry date. Hence, the government of Bangladesh loses a handsome amount of taxes each year (BRTA, 2019).
Due to these reasons, it is imperative to develop a system that can automatically detect, recognize and verify Bangla number plates. This process is done manually in our country to date. It is laborious and time-consuming for the traffic police to stop every vehicle and check their registration and tax-token status manually. For any kind of discrepancies, the traffic police have to look at the BRTA's offline database for checking car details. Identifying fraud registrations and fraud taxtoken papers are also challenging task for them. This manual process can be replaced by an automatic system that can perform the above tasks in an appropriate and effective way. But very few researches have been done in the Bangladeshi number plate detection and recognition system. As of today, no system has been developed to automatically verify the Bangladeshi vehicle registration and tax-token status.
In this study, a novel technique based on morphological image processing and template-matching has been introduced for the automatic detection, extraction, recognition and verification of Bangladeshi vehicle number plates. There are four major parts in the proposed method-(i) pre-processing, (ii) extraction and segmentation, (iii) recognition and (iv) verification of number plates. In the pre-processing stage, the input image is resized and converted to a gray-scale image.
After that, the gray-scale image is being fed into the contrast enhancement and tilt correction process. The extraction part emphasizes plate area extraction and character segmentation. The Sobel operator is applied to detect edges following a series of morphological operations like filling, dilation and erosion for the localization of the number plate. The number plate is then extracted and the Otsu method is applied to binarize it. The connected component analysis and bounding box methods are used to segment the characters within the plate boundary. The template matching approach is applied to recognize the characters once all of them are extracted. After recognizing them, they are stored in a text file. At the verification stage, the text file is compared with a cloud database containing the registered vehicle details. If the extracted number plate is matched with one of the stored number plates in the cloud database, the vehicle is marked as registered. After verifying the registration status of the car, the proposed system extracts owner details from the database and checks whether the tax-token of the car has validity or not. The contributions in the proposed method are: 1) This research will help the researchers to perform research in the field of automatic detection, recognition and verification of Bangladeshi car number plates. We have developed a rich collection of Bangla templates containing numerical, alphabets and words. This dataset will help the researchers to work efficiently in this field. 2) BRTA can use the proposed number plate verification technique to automatically detect the cars that are not registered or tax-token date is expired by using their cloud database. This will help to collect registration fees and taxes. 3) The proposed verification technique can be helpful to traffic management authorities also for finding a car that is stolen or used for a malicious purpose. The text file can easily be compared with the cloud database of BRTA containing reported car number plate details. 4) This system can also be used in automatic car parking management, toll collection and for enforcing laws by the Bangladesh police.

Related Works
An automatic method for the detection, extraction and recognition of Bangladeshi analog number plates and their characters is proposed in Mashuk et al. (2010). The Sobel operator is used for detecting the edges of the license plates. Morphological operations and the bounding box technique are used for the localization of the license plates and segmentation of the characters, respectively. For the recognition of the characters, Back-Propagation Neural Network (BPNN) is applied in their proposed method. Amin et al. (2014) have proposed an automatic technique to recognize the Bangladeshi analog number plates. In their proposed method the Sobel edge detector is used for finding out the edges of the plates. A contour property-based Bangladeshi license plate recognition system is proposed by Abedin et al. (2017). They have implemented their system in the Python Open CV environment. The bounding box technique and Convolutional Neural Network (CNN) are used for the segmentation and recognition of the characters. The method proposed in Chandra et al. (2017) has four steps for recognizing the Bangladeshi license plates. These steps are detection and extraction of license plates and segmentation and recognition of the characters. There are 400 images of Bangladeshi car license plates in their dataset. The Sobel operator and bounding box techniques are used for edge detection and plate area extraction, respectively. They have used Radon and Affine transformation for tilt correction. For recognizing the numbers and letters, two different CNN-based classifiers are used in their proposed method. A morphological operation and template matching-based technique is proposed in Hossain et al. (2018) for Bangladeshi license plate area localization and character recognition. In their proposed method the Sobel operator is used for edge detection and the boundary box method is used for character segmentation. Their dataset contains 50 images of Bangladeshi license plates. The proposed method has achieved a number plate detection and character recognition accuracies of 94 and 95.74%, respectively. Rabbani et al. (2018) have proposed an automatic technique based on morphological operation and CNN for the localization of the license plates and recognition of Bangla characters. The connected component analysis is used for segmenting the characters. They have worked with 100 Bangladeshi license plate images and achieve detection, segmentation and recognition accuracies of 93.78, 95.45 and 97.03%, respectively. Islam et al. (2020a) have introduced a morphological operation and histogram analysis-based method for Bangladeshi license plate localization. Connected component analysis and bounding box techniques are used for segmenting the characters. They have used SVM for recognizing the characters. Hossain et al. (2021) have proposed machine learning-based Bangladeshi number plate recognition system. They have used the connected component analysis technique and morphological image processing method to detect and extract the number plate area. A CNN-based model is also proposed to recognize the characters. An automatic system to detect and recognize Bangladeshi number plates is proposed by Alam et al. (2021). In their proposed system, a super-resolution technique is applied to the input image for getting a high-resolution image. The bounding box technique and CNN are applied to segment and recognize the characters, respectively. Islam et al. (2021) have proposed an automatic system to recognize the Bangladeshi number plates. This system applies morphological image processing to the input images for extracting the number plate area. The character recognition is done by using the template matching technique. Their dataset contains 250 Bangladeshi number plate images.
An automatic system is proposed in Chang et al. (2004) to detect and identify the Arabic license plates. In the proposed method, convolution kernel and morphological operation are used for detecting the edges and localizing the plate area. They have used BPNN for recognizing the characters. The dataset contains 65 images of Arabic license plates. This system provides a success rate of 89 and 93% in number plate extraction and character recognition, respectively. Hung and Hsieh (2010) have proposed a system that is installed in a moving vehicle and can take photos of any random car. The wavelet transformation and projection method are used for detecting the license plate and segmenting the characters. The BPNN is used to recognize the characters.
The image database used in their research contains 257 car number plate images. Their proposed system has been tested with these images and gains a recognition accuracy of 88.71%. An algorithm has been introduced in Wen et al. (2011) for recognizing Japanese car number plates. In this study, an improved Bernsen algorithm and an SVM-based algorithm are proposed, respectively, for removing the shadows in the image and recognizing the characters. They have used connected component analysis for segmenting the characters. Their proposed method provides an overall recognition success rate of 93.54%. Islam et al. (2020b) have proposed a morphological image processing and template matching-based technique to localize the number plate and recognize the characters of English number plates. The Sobel operator and the bounding box methods are used to detect the edges and segment the characters, respectively. Singh and Kumar (2019) have developed an automatic Hindi number plate recognition technique. In this research, the connected component labelling process is used for character segmentation and the Histogram of Oriented Gradient (HOG) technique is used for classifying the Hindi characters. An Artificial Neural Network (ANN) model is also proposed for classifying the Hindi numbers and alphabets.

Properties of Bangladeshi Digital Number Plates
In this part, the specific properties of Bangladeshi car number plates are described. BRTA has introduced the retro-reflective Bangla digital license plates in 2012. They have made it mandatory to use digital number plates in vehicles since 2016 (Saif et al., 2019). For this reason, we have considered digital number plates only in our research. The properties of Bangladeshi digital number plates are mentioned below: 1) There are two lines in Bangladeshi number plates as shown in Fig. 1(a). The first line contains the 'city name' for which the car is registered, 'metro' if the car is registered for a metropolitan area and 'class of the vehicle' which can vary depending upon whether it is a private car or motorcycle or truck, etc. The second line is the serial number of the vehicle. The first two numerical denotes the class of the vehicle and the last four is the vehicle number.  (Islam et al., 2020a). The permitted numerical are ০, ১, ২, ৩, ৪, ৫, ৬, ৭, ৮, ৯ which are equivalent to 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 (Islam et al., 2020a).
3) The green background number plates and white background number plates are used for the commercial vehicles and private vehicles, respectively, as shown in Fig. 1(b). The border of both types of vehicles is black. 4) 'Matra' is a unique feature of the Bangla language. It connects the Bangla letters through a distinctive horizontal line running along the tops of these letters (Rabbani et al., 2018). The Bangla alphabets joined by a continuous 'matra' are counted as a single word. In Fig. 1(a), 'ঢাকা' (Dhaka) and 'মমট্রা' (Metro) are the two words where each of them is connected through a 'matra'.

Datasets
For the smooth running of the research, three types of datasets are prepared. The first one is the testing image dataset. The second dataset is the template dataset. It is prepared for character recognition purposes. A cloud dataset containing the car details has also been prepared using 'Google Spread Sheet' for authentication and verification purposes.

Testing Image Dataset
We have prepared a dataset of 500 images of Bangladeshi car license plates. The dataset contains both the green background and white background number plate images. The images are taken at different illumination conditions, diverse angles and varying distances and stored in JPEG format.

Template Dataset
To recognize the characters, a dataset of Bangla alphabetic, numeric and word templates has been prepared. The background of the templates is black with white colour spaced characters. The size of the templates is 42 pixels in height and 24 pixels in width. Both the horizontal and vertical resolution of the templates is 96 dpi. Some of the templates are shown in Fig. 2.

Car Details Dataset
A cloud dataset containing the information of the vehicles has been prepared for this research using 'Google Spread Sheet'. This cloud database contains information about vehicle license plate number, owner name, address, National Identification Number (NID) and tax-token issue and expiry date in the 1 st , 2 nd , 3 rd , 4 th , 5 th and 6 th column, respectively. As the vehicle and its owner's information are sensitive and private, the exact information about the owner's name, address, NID and vehicle tax-token issue and expiry date will not be disclosed.

Proposed Methodology
The proposed method consists of four steps. At the first step, the input image is pre-processed. The second step has two sub-steps. They are extraction of the license plate area from the input image and then segmentation of the characters from the extracted license plate. The third step is the character recognition stage. Verification of the vehicle is done at the fourth step. These steps are depicted elaborately in Fig. 3 and the flow chart of the proposed method is shown in Fig. 4.

Pre-Processing of the Captured Image
Images captured by a high-resolution camera are fed into the system. At first, the input image is resized into 300 pixels in height and 500 pixels in width. This RGB image is then transformed to a gray-scale image by using Eq. 1 (Islam et al., 2021): (1) The contrast of the gray-scale image is enhanced to minimize the light intensity variations by using the MATLAB function imadjust(). The effect of this contrast enhancement technique is shown in Fig. 5.
The contrast-enhanced gray-scale image is then gone through a tilt correction process. First, the image is filtered by a low pass filter named 'wiener2' of size 5 by 5 to remove the adaptive noise. Then, edge detection is performed by using the 'Canny edge detector'. The Radon transform is applied to the edge detected image for finding out the tilt angle of the image (Chandra et al., 2017). After that, affine transformation is applied to correct the tilt angle. This method can correct up to 45° tilt angle. Some tilt corrected images are shown in Fig. 6.

Extraction and Segmentation of Number Plate
This is one of the important stages of an automatic license plate recognition system. This part of the proposed system performs two major tasks-localization and extraction of the number plate area and segmentation of the number plate characters. The steps of the extraction and segmentation process are given below.  Step 1: Detect edges of the tilt corrected gray-scale image by using the Sobel operator.
Step 2: Apply the mentioned morphological operations to the edge detected image -filling, dilation, filling again and erosion (7 times).
Step 3: Apply the 'bounding box' technique to the eroded image for the localization of the plate area.
Step 4: Extract the plate by using the 'crop' method taking the gray-scale image as input.
Step 5: Convert the extracted gray-scale plate image to binary using the Otsu method.
Step 6: Resize the plate to 240 pixels in height keeping the width unchanged.
Step 7: Removal of dust and unnecessary objects from the resized plate image.
Step 8: Split the license plate equally row-wise into two parts.
Step 9: Apply 'connected component analysis' and 'bounding box' techniques for the segmentation of upper part characters from left to right.
Step 10: Extract the segmented characters and resized them as 42 pixels in height and 24 pixels in width.

a. Localization and Extraction of the Number Plate Area
First, we need to localize the number plate area for extracting the number plate. The edges of the tilt corrected gray-scale image are detected by using the Sobel operator as shown in Fig. 7(a). This edge detected image defines the license plate area. The Sobel operator uses two 3×3 kernels. The vertical components and the horizontal components of the gradients are calculated by the first kernel and second kernel presented in Eq. 2 and 3, respectively (Islam et al., 2020a): After detecting the edges of the gray-scale image, several morphological operations like filling, dilation and erosion are applied as shown in Fig. 7(b-e). At the filling stage, white pixels replace the black areas inside the edges. At the dilation stage, the edges are expanded. A thicker edge line is found at this stage. The dilated image is then filled up again with white pixels. At the erosion stage, the small white pixel areas are removed from the boundary (Albashir et al., 2020). This stage is repeated seven times. These morphological operations roughly localize the license plate area.
For the exact localization of the plate area, the 'bounding box' technique is applied to the eroded image. This technique detects a sudden change in pixel values from low to high in the eroded image and creates the smallest possible rectangles (Hossain et al., 2018). After that, the license plate area is cropped by using the 'imcrop' function of MATLAB taking the gray-scale image as input. The final cropped license plate is shown in Fig. 7(f). The Otsu method (Otsu, 1979) is then used to turn the image found in the previous step into a binary image as shown in Fig. 7(g). This approach begins the conversion process by computing a threshold value, T using Eq. 4: Then the binarized license plate image is resized to 240 pixels in height. The width of the image remains unchanged.
The resized image is then fed into the dust and unnecessary object removal process. The dust and unnecessary objects are removed, respectively, by using the 'strel' and 'bwareaopen' functions of MATLAB (Hossain et al., 2018).

b. Segmentation of the Number Plate Characters
After the extraction of the number plate, the proposed method starts to perform the character segmentation process. At first, the double line Bangladeshi number plate is divided equally row-wise into two portions. The upper portion contains the words and letters. The lower portion contains the digits. The 'connected component analysis' technique is used to sequentially find the location of the words, letters and digits from left to right of the upper and lower portion. For the segmentation of characters, the 'bounding box' technique is used. This technique marks the segments with yellow coloured rectangles. The segmented characters are then extracted and resized to 42 pixels in height and 24 pixels in width. The segmentation process is expressed in Algorithm 1.  The segmentation algorithm takes unnecessary object removed image, I as input. After computing the size of the image, it divides the image, I row-wise equally. The upper and lower portions are named Image1 and Image2, respectively. The algorithm is then applied connected component analysis by 'bwlabel' function to Image1. This step finds the location of words and letters in the upper portion. Then the bounding box is applied through the 'regionprops' function to segment each connected component and mark the segments with yellow colour rectangles. Then the algorithm extracts the segments and resized them. The same procedure is applied to Image2 also for extracting the digits.

Algorithm 1: Segmentation of characters
The segmented characters from some sample extracted license plates are depicted in Fig. 8.

Recognition of Characters
Character recognition is one of the important steps towards an automatic number plate recognition system. The segmented characters found from the segmentation stage are fed into the recognition module of the In this research, the template matching technique is used to recognize the characters. We have developed six templates for each of the above-mentioned Bangla characters. The steps of the recognition process are given below.
Character recognition process: Step-1: Correlate each of the extracted characters of the upper portion with all the templates of the template dataset.
Step-2: If, for a template, correlation coefficient > the threshold value (0.45), then recognize that template as a valid probable character.
Step-3: Find the template with the maximum correlation coefficient value and save it as text in a string.
Step-5: Save them in the same string Step-6: Save the plate number in a text file.
To find the maximum correlation coefficient, 'mc' among the templates, we have used Eq. 5 (Islam et al., 2021) Where: 'c' denotes the correlation coefficient between a segmented character and a template, rmn and smn are the image matrices, r and s are the average of rmn and smn, respectively. After recognizing the characters, they are stored in the same string. The final output is saved in a text file named 'number_Plate'. The recognition of segmented characters from the previous stage is shown in Fig. 9.

Verification of the Vehicle
As the population of Bangladesh is increasing rapidly, it is a must demand to build smart cities that can accommodate this large population. The proposed Bangladeshi Vehicle Number Plate Recognition and Verification System (BVNPRVS) can help smart cities to integrate technologies like automatic traffic control, smart parking management system, automatic toll collection, etc. (Alam et al., 2021). A concept to integrate the proposed BVNPRVS with the automatic systems of smart cities is shown in Fig. 10. The traffic management authority and law enforcement forces can use the proposed BVNPRVS to fetch required information about a vehicle automatically from a cloud database by using the cameras installed in the city or mobile phone camera of him/her.
The proposed vehicle verification steps are given below. Verification Process: Step-1: Connect with the cloud database Step-2: Match the text file generated in the recognition stage with the first column of the cloud database containing the license plate numbers. The output message boxes generated by the proposed verification system for step 3, step 6 and step 7 are depicted in Fig. 11, 12 and 13, respectively.

Results
In this research, the simulation is performed in MATLAB 2019a software. The computing environment consists of an operating system of 64 bit Windows 10 Pro, Intel Core i7 CPU, 1TB hard disk, 500 GB SSD and 8 GB RAM. We have used a GPU of Gigabyte RX 6900 XT with 16 GB RAM to execute the processes faster. The captured number plate features are mentioned in Table 1.
The proposed method achieves an accuracy of 96.8% while detecting the number plates. The system detects 484 Bangladeshi number plates out of 500. The proposed method successfully extracts 459 number plates out of 484 detected number plates. This system achieves an accuracy of 98.3% while segmenting the characters of 459 extracted number plates. The system also provides an accuracy of 97.6% while recognizing the characters of the 451 segmented number plates. Some of the number plates from the testing image dataset contain a single word ('city name') and some of them contain two words ('city name' and 'metro'). Each number plate contains a letter and six digits representing the vehicle class and serial number, respectively. The proposed method achieves accuracies of 98.3, 98.4 and 98%, respectively, while recognizing words, letters and digits. The proposed system is successfully authenticated and verified all the recognized Bangladeshi number plates. An overview of the accuracies of different stages is shown in Table 2.

Discussion
The proposed method can detect and recognize both commercial and private vehicles having green and white backgrounds, respectively. The proposed system is capable of correcting a maximum of 45° rotation/tilt angle. If the tilt angle is more than 45°, the system discards the image. The proposed system fails if the image is too much blurry, has too much noisy background, has a very low resolution or the plate is broken. Some of the examples are shown in Fig. 14.
It is found from the discussion that the tilt correction process of the proposed method is quite efficient. The proposed method can also extract the number plate from images affected by blurriness, brightness and other environmental conditions. The proposed segmentation algorithm efficiently segments the characters of the number plates. In our research, we have used a total of 198 templates for 33 Bangla characters. As the number of templates used for the recognition module is high, the recognition accuracy of the proposed system is also high. The proposed verification method doesn't import data to the system rather it will directly search for the recognized license plate number from the cloud database and extract the vehicle details. For this reason, the system takes less time and acts fast to verify a vehicle. A comparison between the proposed method and different existing methods is presented in Table 3. It can be observed that the number of sample images used for this research is significantly higher than other researches. We have achieved higher accuracy in every stage of the process. None of the research has done the number plate authentication and verification process. The total processing time taken by the proposed method from the detection stage to the verification stage is also significantly less than the other existing methods. So, it can be said that our proposed method outperforms different existing methods.   No. of Images -----------------------------Stages Accuracy (

Conclusion
In this research, a noble method has been proposed to detect, recognize and verify Bangladeshi vehicle number plates. In this method, the captured images are pre-processed before being fed into the extraction and segmentation module. The extraction and segmentation processes are done by using a series of methods such as edge detection, morphological operations, bounding box technique and connected component analysis. The proposed segmentation algorithm provides an accuracy of 98.3% while segmenting the characters. The template matching technique is used to recognize the segmented characters. The recognition accuracy of the proposed method is 97.6%.
The system has successfully authenticated and verified all the recognized number plates. Later, the quantitative analysis shows that the proposed method outperformed different other recent methods in this field. The limitation of the proposed method is that the system is failed to detect and recognize some number plate images which are too much noisy or tilted by more than 45°. In the future, we will work on these to update the system that can eliminate the limitations. the manuscript and no ethical issues are involved.