Automatic Skin Lesion Diagnosis and Medical ReportGeneration Based on Image Captioning

Abstract

Captioning or textual description of the visual content of imagesinvolves generating meaningful words and sentences to describe the contentof an image. This work lies at the crossroads of Natural LanguageProcessing (NLP) and computer vision. When dealing with medical imagesand especially skin lesions, it goes beyond simple classification to generatedetailed textual reports describing the skin lesion's conditioncomprehensively. Such reports are crucial for supporting clinical diagnosisand decision-making. The novelty of this study lies in the creation of thefirst dataset specifically designed for skin lesion captioning, generated usingexpert-validated descriptions based on the ABCDE rules. Our approachintegrates the VGG16 architecture for feature extraction and LSTM fortextual description generation. The proposed method was evaluated on thePH2 dataset and achieved a BLEU-1 score of 0.50, demonstrating itspromise for aiding dermatological diagnosis.