AI-Guided Anatomical Landmark and Abnormality Detection for Autonomous Endoscopy Examination

Md Shakhawat Hossain; Md Shakhawat Hossain; Munim Ahmed; Md Sahilur Rahman; Mahreen Tabassum; Fariha Karim; Md Aulad Hossain; Razib Hayat Khan; Razib Hayat Khan; M. M. Mahbubul Syeed; M. M. Mahbubul Syeed; Mohammad Faisal Uddin; Mohammad Faisal Uddin

doi:10.3844/jcssp.2024.858.871

Abstract

Endoscopy is the routine medical procedure to observe tumors in the human Gastrointestinal (GI) tract by inserting an endoscope, a thin, flexible, tube-like instrument with a light source and camera. Traditionally, an endoscopist performs the endoscopy, orients the endoscope within these structures and navigates this through the help of familiar anatomical landmarks to reach the abnormalities and mark them. Identifying landmarks and abnormalities is critical for the maneuver and the success of endoscopy, which is related to the patient’s comfort, injury and accurate diagnosis. The manual naked-eye-observation maneuver and examination are highly challenging, take a long time and often cause discomfort to the patients and the endoscopists. As a result, several AI-based landmark detection methods have been proposed recently to facilitate autonomous endoscopy examination. However, these methods lack accuracy and consider only limited landmarks. This study presents a Data-efficient image Transformer (DeiT)-based method to detect anatomical landmarks and anomalies for autonomous endoscopy. The proposed method detected 23 landmarks and anomalies from the entire GI tract with 99% accuracy and precision, outperforming the state-of-the-art (91%). Moreover, this method took only 0.045 sec to identify a landmark. The phi coefficient (0.997) indicated a strong positive association between the proposed method and clinical ground truth. Strong association, high accuracy and rapid speed ensured the reliability of the proposed method for autonomous endoscopy examination.

References

Aabakken, L., Barkun, A. N., Cotton, P. B., Fedorov, E., Fujino, M. A., Ivanova, E., Kudo, S., Kuznetzov, K., De Lange, T., Matsuda, K., Moine, O., Rembacken, B., Rey, J., Romagnuolo, J., Rösch, T., Sawhney, M., Yao, K., & Waye, J. D. (2014). Standardized endoscopic reporting. Journal of Gastroenterology and Hepatology, 29(2), 234–240. https://doi.org/10.1111/jgh.12489

Aliyi, S., Dese, K., & Raj, H. (2023). Detection of gastrointestinal tract disorders using deep learning methods from colonoscopy images and videos. Scientific African, 20, e01628. https://doi.org/10.1016/j.sciaf.2023.e01628

Ayyoubi Nezhad, S., Khatibi, T., & Sohrabi, M. (2022). Proposing Novel Data Analytics Method for Anatomical Landmark Identification from Endoscopic Video Frames. Journal of Healthcare Engineering, 2022(1), 8151177. https://doi.org/10.1155/2022/8151177

Borgli, H., Thambawita, V., Smedsrud, P. H., Hicks, S., Jha, D., Eskeland, S. L., Randel, K. R., Pogorelov, K., Lux, M., Nguyen, D. T. D., Johansen, D., Griwodz, C., Stensland, H. K., Garcia-Ceja, E., Schmidt, P. T., Hammer, H. L., Riegler, M. A., Halvorsen, P., & De Lange, T. (2020). HyperKvasir, a comprehensive multi-class image and video dataset for gastrointestinal endoscopy. Scientific Data, 7(1), 283. https://doi.org/10.1038/s41597-020-00622-y

Bour, A., Castillo-Olea, C., Garcia-Zapirain, B., & Zahia, S. (2019). Automatic colon polyp classification using Convolutional Neural Network: A Case Study at Basque Country. 2019 IEEE International Symposium on Signal Processing and Information Technology (ISSPIT), 1–5. https://doi.org/10.1109/isspit47144.2019.9001816

Che, K., Ye, C., Yao, Y., Ma, N., Zhang, R., Wang, J., & Meng, M. Q. H. (2021). Deep learning-based biological anatomical landmark detection in colonoscopy videos. ArXiv, 2108.02948. https://doi.org/10.48550/arXiv.2108.02948

Chicco, D., & Jurman, G. (2020). The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation. BMC Genomics, 21, 1–13. https://doi.org/10.1186/s12864-019-6413-7

Del Moral, P., Nowaczyk, S., & Pashami, S. (2022). Why Is Multiclass Classification Hard? IEEE Access, 10, 80448–80462. https://doi.org/10.1109/access.2022.3192514

Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., & Fei-Fei, L. (2009). ImageNet: A large-scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, 248–255. https://doi.org/10.1109/cvprw.2009.5206848

Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., & Houlsby, N. (2020). An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. ArXiv, 2010.11929. https://doi.org/10.48550/arXiv.2010.11929

Ferlay, J., Soerjomataram, I., Dikshit, R., Eser, S., Mathers, C., Rebelo, M., Parkin, D. M., Forman, D., & Bray, F. (2015). Cancer incidence and mortality worldwide: Sources, methods and major patterns in GLOBOCAN 2012. International Journal of Cancer, 136(5), E359–E386. https://doi.org/10.1002/ijc.29210

He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 770–778. https://doi.org/10.1109/cvpr.2016.90

Hirasawa, T., Aoyama, K., Tanimoto, T., Ishihara, S., Shichijo, S., Ozawa, T., Ohnishi, T., Fujishiro, M., Matsuo, K., Fujisaki, J., & Tada, T. (2018). Application of artificial intelligence using a convolutional neural network for detecting gastric cancer in endoscopic images. Gastric Cancer, 21, 653–660. https://doi.org/10.1007/s10120-018-0793-2

Hossain, M. S., Nakamura, T., Kimura, F., Yagi, Y., & Yamaguchi, M. (2018). Practical image quality evaluation for whole slide imaging scanner. Biomedical Imaging and Sensing Conference, 203–206. https://doi.org/10.1117/12.2316764

Hossain, M. S., Rahman, M. M., Syeed, M. M., Uddin, M. F., Hasan, M., Hossain, M. A., Ksibi, A., Jamjoom, M. M., Ullah, Z., & Samad, M. A. (2023). DeepPoly: Deep Learning-Based Polyps Segmentation and Classification for Autonomous Colonoscopy Examination. IEEE Access, 11, 95889–95902. https://doi.org/10.1109/access.2023.3310541

Hossain, M. S., Syeed, M. M., Fatema, K., & Uddin, M. F. (2022). The Perception of Health Professionals in Bangladesh toward the Digitalization of the Health Sector. International Journal of Environmental Research and Public Health, 19(20), 13695. https://doi.org/10.3390/ijerph192013695

Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017). Densely Connected Convolutional Networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 4700–4708. https://doi.org/10.1109/cvpr.2017.243

Iwagami, H., Ishihara, R., Aoyama, K., Fukuda, H., Shimamoto, Y., Kono, M., Nakahira, H., Matsuura, N., Shichijo, S., Kanesaka, T., Kanzaki, H., Ishii, T., Nakatani, Y., & Tada, T. (2021). Artificial intelligence for the detection of esophageal and esophagogastric junctional adenocarcinoma. Journal of Gastroenterology and Hepatology, 36(1), 131–136. https://doi.org/10.1111/jgh.15136

Kaminski, M. F., Regula, J., Kraszewska, E., Polkowski, M., Wojciechowska, U., Didkowska, J., Zwierko, M., Rupinski, M., Nowacki, M. P., & Butruk, E. (2010). Quality Indicators for Colonoscopy and the Risk of Interval Cancer. New England Journal of Medicine, 362(19), 1795–1803. https://doi.org/10.1056/nejmoa0907667

Luo, H., Xu, G., Li, C., He, L., Luo, L., Wang, Z., Jing, B., Deng, Y., Jin, Y., Li, Y., Li, B., Tan, W., He, C., Seeruttun, S. R., Wu, Q., Huang, J., Huang, D., Chen, B., Lin, S. B., … Xu, R. H. (2019). Real-time artificial intelligence for detection of upper gastrointestinal cancer by endoscopy: A multicentre, case-control, diagnostic study. The Lancet Oncology, 20(12), 1645–1654. https://doi.org/10.1016/s1470-2045(19)30637-0

Misawa, M., Kudo, S. E., Mori, Y., Hotta, K., Ohtsuka, K., Matsuda, T., Saito, S., Kudo, T., Baba, T., Ishida, F., Itoh, H., Oda, M., & Mori, K. (2021). Development of a computer-aided detection system for colonoscopy and a publicly accessible large colonoscopy video database (with video). Gastrointestinal Endoscopy, 93(4), 960–967. https://doi.org/10.1016/j.gie.2020.07.060

Nishitha, R., Amalan, S., Sharma, S., Preejith, S. P., & Sivaprakasam, M. (2022). Image Quality Assessment for Interdependent Image Parameters Using a Score-Based Technique for Endoscopy Applications. 2022 IEEE International Symposium on Medical Measurements and Applications (MeMeA), 1–6. https://doi.org/10.1109/memea54994.2022.9856448

Ozawa, T., Ishihara, S., Fujishiro, M., Kumagai, Y., Shichijo, S., & Tada, T. (2020). Automated endoscopic detection and classification of colorectal polyps using convolutional neural networks. Therapeutic Advances in Gastroenterology, 13, 175628482091065. https://doi.org/10.1177/1756284820910659

Pogorelov, K., Randel, K. R., Griwodz, C., Eskeland, S. L., de Lange, T., Johansen, D., Spampinato, C., Dang-Nguyen, D.-T., Lux, M., Schmidt, P. T., Riegler, M., & Halvorsen, P. (2017). KVASIR: A Multi-Class Image Dataset for Computer Aided Gastrointestinal Disease Detection. Proceedings of the 8th ACM on Multimedia Systems Conference, 164–169. https://doi.org/10.1145/3083187.3083212

Selvaraju, R. R., Cogswell, M., Das, A., Vedantam, R., Parikh, D., & Batra, D. (2017). Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization. Proceedings of the IEEE International Conference on Computer Vision, 618–626. https://doi.org/10.1109/iccv.2017.74

Shakhawat, H. M., Nakamura, T., Kimura, F., Yagi, Y., & Yamaguchi, M. (2020). Automatic Quality Evaluation of Whole Slide Images for the Practical Use of Whole Slide Imaging Scanner. ITE Transactions on Media Technology and Applications, 8(4), 252–268. https://doi.org/10.3169/mta.8.252

Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. ArXiv, 1409.1556. https://doi.org/10.48550/arXiv.1409.1556

Suzuki, H., Yoshitaka, T., Yoshio, T., & Tada, T. (2021). Artificial intelligence for cancer detection of the upper gastrointestinal tract. Digestive Endoscopy, 33(2), 254–262. https://doi.org/10.1111/den.13897

Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015). Going deeper with convolutions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 1–9. https://doi.org/10.1109/cvpr.2015.7298594

Tan, M., & Le, Q. (2019). Efficientnet: Rethinking model scaling for convolutional neural networks. Proceedings of the 36th International Conference on Machine Learning, 6105–6114.

Tomar, N. K., Jha, D., Ali, S., Johansen, H. D., Johansen, D., Riegler, M. A., & Halvorsen, P. (2021). DDANet: Dual Decoder Attention Network for Automatic Polyp Segmentation. In A. Del Bimbo, R. Cucchiara, S. Sclaroff, G. Maria Farinella, T. Mei, M. Bertini, H. Jair Escalante, & R. Vezzani (Eds.), Pattern Recognition. ICPR International Workshops and Challenges (1st ed., Vol. 12668, pp. 307–314). Springer, Cham. https://doi.org/10.1007/978-3-030-68793-9_23

Touvron, H., Sablayrolles, A., Douze, M., Cord, M., & Jegou, H. (2021). Grafit: Learning fine-grained image representations with coarse labels. Proceedings of the IEEE/CVF International Conference on Computer Vision, 874–884. https://doi.org/10.1109/iccv48922.2021.00091

Tran, T.-H., Nguyen, P.-T., Tran, D.-H., Manh, X.-H., Vu, D.-H., Ho, N.-K., Do, K.-L., Nguyen, V.-T., Nguyen, L.-T., Dao, V.-H., & Vu, H. (2021). Classification of anatomical landmarks from upper gastrointestinal endoscopic images. 2021 8th NAFOSTED Conference on Information and Computer Science (NICS), 278–283. https://doi.org/10.1109/nics54270.2021.9701513

AI-Guided Anatomical Landmark and Abnormality Detection for Autonomous Endoscopy Examination

Abstract

References

Download

Keywords