Research Article Open Access

A CNN-KNN Based Recognition of Online Handwritten Symbols within Physics Expressions Using Contour-Based Bounding Box (CBBS) Segmentation Technique

Ujwala Kolte1, Sachin Naik1 and Vidya Kumbhar2
  • 1 Symbiosis Institute of Computer Studies and Research, Symbiosis International (Deemed University), Pune, India
  • 2 Symbiosis Institute of Geoinformatics, Symbiosis International (Deemed University), Pune, India

Abstract

The task of recognizing symbols poses a significant challenge owing to the wide variability in human handwriting. Complexity in terms of the structural representation of symbols used in physics expressions is a major challenge in the recognition process The emergence of online handwriting, fueled by the widespread adoption of handheld digital devices, particularly in educational contexts, highlights the critical importance of precise symbol recognition, especially in the teaching and learning process. In contemporary literature, there is a notable emphasis on LaTex sequencing, symbol recognition and parsing. However, deep learning continues to yield promising results in this domain. The convenience of user input provides benefits to e-learning applications. In this study, we propose three approaches for the recognition of physics symbols within physics expressions (1) A proposed Java user interface for taking input from the user, as convenience of user input provides benefits to e-learning applications. (2) Contour-based bounding box segmentation algorithm, which deals with broken symbols within physics expressions. (3) For recognition, we propose a Convolution Neural Network-K-Nearest Neighbor (CNN-KNN) recognition model, as CNN plays an important role in extracting features, which are further provided as input to the K-NN classifier using the dropout method. Combining these three approaches into a symbol recognition model provides state-of-arts results. Handwritten physics symbols were collected from 20 different writers and each writer has written 5 types of physics expressions under different categories like electric flux, Maxwell’s equations, inductance and pointing vector and moment of Interia. There were 25 classes identified from the 780 samples collected from the users. The recognition rate is identified using (1) Using CNN model, which shows an accuracy of 91.48 and (2) Using the proposed hybrid CNN-KNN model the accuracy reported is 98.06.

References

Artieres, T., Marukatat, S., & Gallinari, P. (2007). Online Handwritten Shape Recognition Using Segmental Hidden Markov Models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 29(2), 205–217. https://doi.org/10.1109/tpami.2007.38
Baró, A., Riba, P., Calvo-Zaragoza, J., & Fornés, A. (2019). From Optical Music Recognition to Handwritten Music Recognition: A baseline. Pattern Recognition Letters, 123, 1–8. https://doi.org/10.1016/j.patrec.2019.02.029
Bresler, M., Prua, D., & Hlavác, V. (2013). Modeling Flowchart Structure Recognition as a Max-Sum Problem. 2013 12th International Conference on Document Analysis and Recognition, 1215–1219. https://doi.org/10.1109/icdar.2013.246
Coquenet, D., Chatelain, C., & Paquet, T. (2023). End-to-End Handwritten Paragraph Text Recognition Using a Vertical Attention Network. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(1), 508–524. https://doi.org/10.1109/tpami.2022.3144899
Huang, B., Du, C., Zhang, Y., & Kechadi, M. T. (2006). A Hybrid HMM-SVM Method for Online Handwriting Symbol Recognition. Sixth International Conference on Intelligent Systems Design and Applications, 887–891. https://doi.org/10.1109/isda.2006.61
Jeon, H.-K., & Yang, C.-S. (2021). Enhancement of Ship Type Classification from a Combination of CNN and KNN. Electronics, 10(10), 1169. https://doi.org/10.3390/electronics10101169
Keshari, B., & Watt, S. (2007). Hybrid Mathematical Symbol Recognition Using Support Vector Machines. Ninth International Conference on Document Analysis and Recognition (ICDAR 2007), 859–863. https://doi.org/10.1109/icdar.2007.4377037
Kherdekar, V. A., Naik, S. A., & Bafna, P. (2023). A Segmentation-based Token Identification for Recognition of Audio Mathematical Expression. International Journal of Advanced Computer Science and Applications, 14(7), 298–303. https://doi.org/10.14569/ijacsa.2023.0140733
Lai, S.-H., Lepetit, V., Nishino, K., & Sato, Y. (Eds.). (2017). Computer Vision – ACCV 2016 (Vol. 10112). Springer Cham. https://doi.org/10.1007/978-3-319-54184-6
Le, A. D., Nguyen, H. D., Indurkhya, B., & Nakagawa, M. (2019). Stroke Order Normalization for Improving Recognition of Online Handwritten Mathematical Expressions. International Journal on Document Analysis and Recognition (IJDAR), 22(1), 29–39. https://doi.org/10.1007/s10032-019-00315-2
Li, Z., Wang, X., Liu, Y., Jin, L., Huang, Y., & Ding, K. (2024). Improving Handwritten Mathematical Expression Recognition via Similar Symbol Distinguishing. IEEE Transactions on Multimedia, 26, 90–102. https://doi.org/10.1109/tmm.2023.3260648
Long, J., Shelhamer, E., & Darrell, T. (2015). Fully Convolutional Networks for Semantic Segmentation. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 3431–3440. https://doi.org/10.1109/cvpr.2015.7298965
Michael, J., Labahn, R., Grüning, T., & Zöllner, J. (2019). Evaluating Sequence-to-Sequence Models for Handwritten Text Recognition. 2019 International Conference on Document Analysis and Recognition (ICDAR), 1286–1293. https://doi.org/10.1109/icdar.2019.00208
Miyao, H., & Maruyama, M. (2007). An Online Handwritten Music Symbol Recognition System. International Journal of Document Analysis and Recognition (IJDAR), 9(1), 49–58. https://doi.org/10.1007/s10032-006-0026-9
Nguyen, C. T., Truong, T.-N., Nguyen, H. T., & Nakagawa, M. (2021). Global Context for Improving Recognition of Online Handwritten Mathematical Expressions. In J. Lladós, D. Lopresti, & S. Uchida (Eds.), Document Analysis and Recognition – ICDAR 2021 (Vol. 12822, pp. 617–631). Springer, Cham. https://doi.org/10.1007/978-3-030-86331-9_40
Ptucha, R., Petroski Such, F., Pillai, S., Brockler, F., Singh, V., & Hutkowski, P. (2019). Intelligent Character Recognition Using Fully Convolutional Neural Networks. Pattern Recognition, 88, 604–613. https://doi.org/10.1016/j.patcog.2018.12.017
Putri, D. U. K., Pratomo, D. N., & Azhari, A. (2023). Hybrid Convolutional Neural Networks-Support Vector Machine Classifier with Dropout for Javanese Character Recognition. TELKOMNIKA (Telecommunication Computing Electronics and Control), 21(2), 346–353. https://doi.org/10.12928/telkomnika.v21i2.24266
Qiu, Q., Tan, Y., Ma, K., Tian, M., Xie, Z., & Tao, L. (2023). Geological Symbol Recognition on Geological Map Using Convolutional Recurrent Neural Network with Augmented Data. Ore Geology Reviews, 153, 105262. https://doi.org/10.1016/j.oregeorev.2022.105262
Sanida, T., Sideris, A., Tsiktsiris, D., & Dasygenis, M. (2022). Lightweight Neural Network for COVID-19 Detection from Chest X-ray Images Implemented on an Embedded System. Technologies, 10(2), 37. https://doi.org/10.3390/technologies10020037
Sejuti, Z. A., & Islam, M. S. (2023). A Hybrid CNN–KNN Approach for Identification of COVID-19 with 5-Fold Cross Validation. Sensors International, 4, 100229. https://doi.org/10.1016/j.sintl.2023.100229
Shuvo, S. N., Hasan, F., Ahmed, M. U., Hossain, S. A., & Abujar, S. (2021). MathNET: Using CNN Bangla Handwritten Digit, Mathematical Symbols, and Trigonometric Function Recognition. In S. Borah, R. Pradhan, N. Dey, & P. Gupta (Eds.), Soft Computing Techniques and Applications (Vol. 1248, pp. 515–523). Springer, Singapore. https://doi.org/10.1007/978-981-15-7394-1_47
Smith, W. A. P., & Pillatt, T. (2023). You Only Look for a Symbol Once: An Object Detector for Symbols and Regions in Documents. In G. A. Fink, R. Jain, K. Kise, & R. Zanibbi (Eds.), Document Analysis and Recognition - ICDAR 2023 (Vol. 14191, pp. 227–243). Springer, Cham. https://doi.org/10.1007/978-3-031-41734-4_14
Sundaram, S., & Ramakrishnan, A. G. (2014). Performance enhancement of online handwritten Tamil symbol recognition with reevaluation techniques. Pattern Analysis and Applications, 17(3), 587–609. https://doi.org/10.1007/s10044-013-0353-7
Tang, P., Hui, S. C., & Fu, C.-W. (2013). Online chemical symbol recognition for handwritten chemical expression recognition. 2013 IEEE/ACIS 12th International Conference on Computer and Information Science (ICIS), 535–540. https://doi.org/10.1109/icis.2013.6607894
Urala, K. B., Ramakrishnan, A. G., & Mohamed, S. (2014). Recognition of open vocabulary, online handwritten pages in Tamil script. 2014 International Conference on Signal Processing and Communications (SPCOM), 1–6. https://doi.org/10.1109/spcom.2014.6984002
Xin, G., Cuiyun, L., Jihong, P., & Weixin, X. (2003). HMM based online hand-drawn graphic symbol recognition. 6th International Conference on Signal Processing, 2002, 1067–1070. https://doi.org/10.1109/icosp.2002.1179973
Zhu, R., & Wang, Y. (2012). Application of Improved Median Filter on Image Processing. Journal of Computers, 7(4), 838–841. https://doi.org/10.4304/jcp.7.4.838-841

Journal of Computer Science
Volume 20 No. 7, 2024, 783-792

DOI: https://doi.org/10.3844/jcssp.2024.783.792

Submitted On: 21 December 2023 Published On: 7 May 2024

How to Cite: Kolte, U., Naik, S. & Kumbhar, V. (2024). A CNN-KNN Based Recognition of Online Handwritten Symbols within Physics Expressions Using Contour-Based Bounding Box (CBBS) Segmentation Technique. Journal of Computer Science, 20(7), 783-792. https://doi.org/10.3844/jcssp.2024.783.792

  • 1,567 Views
  • 813 Downloads
  • 0 Citations

Download

Keywords

  • CNN
  • Contour Based Bounding Box Segmentation (CBBS)
  • K-NN
  • Physics Expression
  • Symbol Recognition