Recognition of esophagitis in endoscopic images using transfer learning

INTRODUCTION

Esophagitis is an inflammatory and damaging process of the esophageal mucosa,that can be the outcome of different pathological processes,which share,however,the same clinical presentation:retrosternal pain,dysphagia,odynophagia and heartburn[1,2].Different pathological processes may lead to esophagitis,with possible etiologies embracing gastroesophageal reflux disease (GERD),infectious processes,in eosinophilic esophagitis,medications or even radiation.In extreme cases,it can result in stenosis,fistulization and esophageal perforation[3].These complications,however,may be prevented with precoce diagnosis.

The girl had to ride for a great many days, and for a long and wearisome time, before she got there; but at last she did arrive, and then she asked the East Wind if he could tell her the way to the Prince who dwelt east of the sun and west of the moon

After wandering about for several days until he was weary and hungry, he at last succeeded in finding a way out of the forest, and soon came to a wide and rapid river, which he followed, hoping to find some means of crossing it, and it happened that as the sun rose the next morning he saw something of a dazzling whiteness moored76 out in the middle of the stream

Esophagitis can be suspected based on the clinical history,with a confirmation performed through endoscopic visualization.The differentiation of its etiopathogenesis may be determined from endoscopic and histological study of the esophagus.The endoscopic presentation of eosinophilic esophagitis is characterized by exudates,strictures and concentric rings.In colonization by

there are small and diffuse yellow-white plaques;in cytomegalovirus infection there are large ulcerations;Herpes Virus,in turn,may cause multiple small ulcerations[3-5].GERD,on the other hand,has a better-defined endoscopic classification with the Los Angeles classification,which has four gradations based on the presence,size and distribution of esophageal[6].

Machine learning,main exponent of artificial intelligence,has gained space and attention in healthcare and medical research,especially after the development and validation by Beam and Kohane[7] and Gulshan

[8] of a deep learning algorithm capable of detecting the presence of diabetic retinopathy in studies of the retina[7,8].In the context of esophagitis,the use of machine learning,especially deep learning,may be considered to determine,among others,the presence of esophageal lesions compatible with esophagitis.

Deep learning - which comprehends deep artificial neural network-based algorithms capable of learning from large amounts of data - is considered the state of the art in the field of artificial intelligence for computer vision[9].Among the possible uses of such applications,there is the binary classification of images according to the presence or absence of a given finding.In these cases,a dataset comprising examples of the image type to be classified is divided into two distinct subsets:one to train the model (from which the weights will be learned) and the other to evaluate its performance[10].It is important that the two subsets obtained are representative,in terms of labels proportion,of the original dataset.

Traditionally,algorithms for deep learning use large volumes of data for training.However,obtaining databases large enough to accurately train them can prove to be a highly expensive process.As a way of mitigating this situation,one can choose to apply a technique called transfer learning,which is based on the use of external data to perform a training step mentioned above[10].The use of this technique makes it possible to obtain a scale of pretrained weights in computational models for analyzing,among others,medical images.It should be noted,however,that the use of pretrained weights does not exempt the need to carry out a training stage with data that are representative of the base to be tested,with this second training step (called fine tuning) aiming to improve,principally,the deep layers of the algorithm in order to obtain results with greater accuracy[11].

This study aims to develop a supervised deep learning model using a fine-tuned transfer learning dense convolutional neural network (DCNN) to recognize,in a binary way,the presence of changes compatible with esophagitis in images from endoscopic studies.Thus,it seeks to contribute to the advancement and methodological improvement of a cost-effective and accurate automated technology for the diagnosis of esophagitis,optimizing the management of patients who present this condition.

MATERIALS AND METHODS

Data acquisition

Endoscopic images of 1932 patients with a diagnosis of esophagitis and 1663 patients without any pathological diagnosis (in both cases being z line the image topography) were obtained from the publicly available KSAVIR Dataset[12] and HyperKSAVIR Dataset[13].Were included in this study the images in both datasets labeled as “normal z line” and the images labeled as “esophagitis”.From these data,we set out to develop a binary deep learning classifier using the DenseNet-201 architecture,a densely connected convolutional network which connects each layer to every other layer in a feedforward fashion[14],pretrained on the ImageNet image set.

Down came Jack my Hedgehog with his cock, and told the old King he would show him the right way if he would solemnly promise to give him the first thing he met in front of his royal castle

The top layer of the DenseNet-201 architecture was not included in our model,and its output (that is,the output of the final convolutional block) was converted from a 4 dimensional to a 2 dimensional tensor using global average pooling.As the final layer,we added a dense layer with one unit and sigmoid activation.The structure of the final deep neural network predictive model is summarized in Table 1,and its architecture is illustrated in Figure 1.

Model development,training,and validation

For this purpose,the images were converted to arrays of dimension 256 × 305 × 3,whose units were rescaled using the

preprocessor,and divided into training set (80%) and test set (20%).The training set (

=2876) was divided in batches of size 16 and used to train,throughout 80 epochs,the transfer learning based neural network whose structure is shown in Table 1.The test set (

=719) was used to evaluate the model according to the following metrics:accuracy,sensitivity,specificity,and area under the receiver operating characteristic curve (AUC).

Convolutional neural networks with transfer learning for automated analysis of endoscopic images,as proposed in this study,may be incorporated into daily practice as a clinical decision support tool - screening abnormalities and indicating the need for further specialized evaluation or double checking medical reports.This application would add value especially in contexts of scarce resources,in which the number of endoscopists is limited and they are often poorly trained - increasing,thus,the likelihood of diagnostic errors.Moreover,it is especially promising as an adjunct tool to telemedicine,favoring rural and remote areas.

Then a big fire was lit again, and the mother said: Just sit down there, children, and if you re tired you can sleep a bit; we re going into the forest to cut down wood, and in the evening when we re finished we ll come back to fetch you

By recognizing changes in the mucosa of the esophageal Z-line,the binary transfer learning classifier presented in this study aims to demonstrate the effectiveness of these algorithms to differentiate endoscopic images of the same topography with and without changes characteristic of esophagitis.Unlike other studies that aimed at automatic detection of anatomical landmarks and diverse diseases affecting different anatomical sites using the KVASIR database[19-22],we employed state-of-the-art deep learning to specifically target Z-line related changes,bringing great accuracy to its analysis.

Ethical disclosure

As previously stated,all the imaging data was obtained from the public datasets KSAVIR Dataset[12] and HyperKSAVIR Dataset[13] that were released for both educational and research purposes.Therefore,it was not necessary to submit this study to the ethics committee,being in accordance with all the established precepts by the Committee on Publication Ethics.

RESULTS

And each time her sons saw Eric they brightened like the sun and stars in the sky. She had never seen them so happy. Her heart twitched22 as though it were being strummed. She began to fall in love.

So they were married, and in the middle of all the festivities and rejoicings the bride s father came home and was not a little surprised at finding his daughter celebrating her wedding

In the test set,which was designated to model evaluation,the classifier achieved 93.32% accuracy,93.18% sensitivity,93.46% specificity and a 0.96 AUC.The confusion matrix between true labels and labels predicted by the model is presented in Figure 3,while its receiver operating characteristic curve is presented in Figure 4.

In order to identify the imagery aspects related to the predictive decision,it is possible to plot heatmaps that indicate,colorimetrically,the areas with the greatest influence on the prediction.Examples of such heatmaps for esophagitis images contained in the test set are shown in Figure 5.

DISCUSSION

This study understands that transfer learning associated with DCNN has great potential to aid and improve the quality and rate of esophagitis diagnosis through endoscopic imaging.Improving workflow,providing faster preliminary reports,relieving the burden of the increasing patient population associated with the intensive and repetitive mechanical work is some of the promises of the integration of CNN-based algorithms to medical practice[15].

Once the mark of at least 93% in the parameters of accuracy,sensitivity,and specificity has been reached,we were able to demonstrate the potential of these algorithms to assist in the premature recognition of pathological predecessor endoscopic abnormalities,and as a consequence,to intervene positively in the management of these.Thus,the use of DCNN with pretrained and fine-tuned weights proves to be a good strategy for predictive modeling of this type (and potentially other types) of medical images.In addition,adopting the classification approach combined with the subsequent plotting of heat maps associated with the classificatory decision gives greater explainability to the model.

In the meantime the two sisters came back in full dress, and when they saw that the handsome man had fallen to the share of the youngest, and heard that he was Bearskin, they ran out full of anger and rage

The adopted methodology is schematically summarized in Figure 2.All steps of the predictive model development were performed in Python (version 3.6.9),using Keras library.

However,as it is well settled in applications of deep learning in medical image analytics[23],a major limitation of the technical capability of the proposed classifier is the lack of large-scale labeled data.As already shown by Sun

[24],the performance on artificial intelligence in visual tasks increases logarithmically based on volume of training data size.Coupled with this factor,we cannot ensure how the binary classifier would behave in patients with the presence of other diseases.In both cases,however,training on more plural datasets should optimize performance on the parameters evaluated.

Concerning the predictive behavior towards other possible esophageal Z-line abnormalities,assuming that the algorithm was able to differentiate with high accuracy normal images from images with different degrees of inflammation - and consequently different mucosal lesion configurations - it is reasonable to assume that other esophageal lesions would be differentiated from the healthy aspect,and thus categorized together with the esophagitis images.Among the possible clinical differential situations,esophageal and esophagogastric junction cancers are of particular relevance.Upper endoscopies are considered by the Society of Thoracic Surgeons and the National Comprehensive Cancer Network as the initial diagnostic evaluation to exclude esophageal cancer[25];although techniques such as chromoendoscopy and narrow band imaging are often used to increase the sensitivity of detection of lesions suggestive of malignancy,traditional endoscopic imaging still plays an important role in the investigational flowchart,and can demonstrate suspicious findings incidentally[26].

In view of this,in order to extend the clinical utility of our proposed algorithm to the investigation of potentially malignant endoscopic findings,two main approaches are possible:(1) Propose an adaptation of the model to multiclass classification and,to this end,retrain the model including endoscopic images of esophageal cancer,fine-tuning,if necessary,only the final layers,making appropriate changes in the final dense layer and in the loss function to accommodate 3 classes (thus,the final layer would now have 3 neurons with softmax activation function,and the sparse categorical crossentropy loss function would be adopted);and (2) Preserve the binary classification structure,but proposing to change the labels for normal and abnormal findings (thus,the model would be used to triage any endoscopic abnormalities,ranging from inflammatory findings to lesions suggestive of malignancy) and,for this purpose,retrain the model including endoscopic images representative of other types of lesions (including neoplastic lesions).In either situation,the incorporation of images representative of lesions suspicious for malignancy would be necessary,and the weights derived from training with normal endoscopic images and with esophagitis findings already performed would be used (same domain finetuning).

The model was trained using Adam optimizer with a learning rate of 0.0001 and applying binary cross entropy loss function.All layers of the DenseNet architecture incorporated in the model were set as trainable (that is,we fine-tuned all weights).

School was finally out for the holidays and we were going Christmas shopping. Not for us, not even for friends, but for a little girl named Lauren. Lauren is a poverty-stricken five-year-old, and my family and our friends were buying Christmas presents for her that her family could not afford.

CONCLUSION

The use of deep learning,especially the transfer learning technique,has great potential field for the analysis of clinical images,including endoscopic records.Observing this great potential,this paper applied such technique,associated with retraining of all layers,to classify,with a 93.3% accuracy,esophageal mucosa images obtained from endoscopic studies according to the presence or absence of esophagitis.It then becomes evident the potential of transfer learning with fine-tuning for the analysis of images obtained by endoscopic method and recognition of esophageal lesions.

In view of this,it is opportune to raise new studies involving transfer learning for the analysis of related data,with the aim of improving,disseminating and validating its use for the daily routine of clinical practice.Furthermore,the composition and dissemination high-quality endoscopic image sets representative of various clinical conditions (especially esophageal cancer,given its high clinical and epidemiological relevance) is essential for new studies to be developed and algorithms already proposed to be improved.

ARTICLE HIGHLIGHTS

Research background

Computer vision allied with deep learning,especially through the use of deep convolutional neural networks,has been increasingly employed in the automation of medical image analysis.Among these are endoscopic images,which are of great importance in the evaluation of a number of gastroenterological diseases.

Research motivation

Endoscopic findings constitute the diagnostic definition for esophagitis,a multietiological condition with significant impacts on quality of life and the possibility of evolution to a series of complications.Automating the identification of findings suggestive of esophageal inflammation using artificial intelligence could add great value to the evaluation and management of this clinical condition.

Seeing all this, the four dragons felt very sorry, for they knew the Jade Emperor only cared about pleasure, and never took the people to heart. They could only rely on themselves to relieve the people of their miseries. But how to do it?

In consistency with findings described by Wimmer

[16],when they established the potential of the association of transfer learning with CNN in the classification of endoscopic images,previously used focused on celiac disease,or also described by Song

[17] when they reported a deep learningbased model with the ability to histologically classify polyps with a higher accuracy than trained endoscopists,the performance of our algorithmic model reaffirms the potential of deep learning for computer vision in the field of gastrointestinal diagnostics.In line with the mentioned studies,our study demonstrates the already defended potential of CNN-based artificial intelligence systems to diagnose esophageal disease,and can contribute with methodological insights for the development and improvement of such systems[18].

Research objectives

To identify whether a densely connected convolutional neural network with pre-trained and fine-tuned weights is able to binary classify esophageal Z-line endoscopic images according to the presence or absence of esophagitis.

Research methods

Research results

The proposed model was able to diagnose esophagitis in the validation set with sensitivity of 93.18 and specificity of 93.46,demonstrating the feasibility of using deep transfer learning to discriminate normal from damaged mucosa in endoscopic images of the same anatomical segment.It remains to be investigated whether,by means of a more diverse set of images,this technique can be proposed to identify different types of esophageal abnormalities,and potentially in other organs.

Research conclusions

Convolutional neural networks with transfer learning for automated analysis of endoscopic images,as proposed in this study,demonstrate potential for incorporation into clinical practice as a clinical decision support tool,mainly benefiting scarce resources settings.

Research perspectives

Sets of endoscopic images representative of various clinical conditions should be published,in order to allow the findings of this study to be externally validated and for new models with different classificatory approaches to emerge.

Endoscopic images of 1932 patients with a diagnosis of esophagitis and 1663 patients were splitted in training (80%) and test (20%) and used to develop and evaluate a binary deep learning classifier built using a pre-trained DenseNet-201 architecture.The classifier model performance was evaluated in the test set according to accuracy,sensitivity,specificity and area under the receiver operating characteristic curve.

Caires Silveira E proceeded the data collection/entry,performed data analysis and data interpretation,developed the proposed predictive model and participated in preparation and review of manuscript;Santos Corrêa CF and Madureira Silva L participated in preparation of manuscript and wrote the literature analysis/search;Mattos Pretti S and Almeida Santos B participated in review of manuscript;Freire de Melo F designed the research and participated in review of manuscript.

For this study,there was no need for an appraisal by an ethics committee,since only publicly available anonymized data were used.

The present manuscript used anonymous images to produce its analyzes and results,in a method that obeys the norms of medical bioethics.Thus,there was no direct or even indirect contact between researchers and patients,with no necessity for "Signed Informed Consent Form" to carry out our study.

The authors have no financial relationships to disclose.

No additional data are available.

Two of the shells did not contain any explosives in them, but the third had some paper with a message where the explosives would have been. It took a while to find a translator to read the message; it was in Czech and was probably placed in the cannon shell in the Skoda armament plant in Prague, Czechoslovakia. The majority of the Czechoslovakian people resented German control, but many were forced to work in factories supporting the Nazi19 war effort. The message read, THIS IS OUR WAY OF HELPING20 YOU.

This article is an open-access article that was selected by an in-house editor and fully peer-reviewed by external reviewers.It is distributed in accordance with the Creative Commons Attribution NonCommercial (CC BYNC 4.0) license,which permits others to distribute,remix,adapt,build upon this work non-commercially,and license their derivative works on different terms,provided the original work is properly cited and the use is noncommercial.See:https://creativecommons.org/Licenses/by-nc/4.0/

The extent of Holly s disappointment over the incident didn t become evident until the following evening when she came downstairs to say good night. Well, I guess we know how much my best friend thinks of me, huh, Mom? Her attempt at a breezy（） tone failed miserably6.

Brazil

Elena Caires Silveira 0000-0003-3470-9205;Caio Fellipe Santos Corrêa 0000-0002-5271-6911;Leonardo Madureira Silva 0000-0002-6444-8264;Bruna Almeida Santos 0000-0002-4543-3163;Soraya Mattos Pretti 0000-0002-9835-7635;Fabrício Freire de Melo 0000-0002-5680-2753.

Gao CC

Though somewhat younger, my wife and I attend a church that caters1 to senior citizens. We like it because of the traditional service and very friendly elderly people. One in particular is a lady in her early eighties, who cheerfully greets us at the same door every Sunday morning with a smile and kind words. We look forward to seeing Betty and giving her an occasional hug.

A

Cai YX

1 Gomez Torrijos E,Gonzalez-Mendiola R,Alvarado M,Avila R,Prieto-Garcia A,Valbuena T,Borja J,Infante S,Lopez MP,Marchan E,Prieto P,Moro M,Rosado A,Saiz V,Somoza ML,Uriel O,Vazquez A,Mur P,Poza-Guedes P,Bartra J.Eosinophilic Esophagitis:Review and Update.

2018;5:247 [PMID:30364207 DOI:10.3389/fmed.2018.00247]

2 Habbal M,Scaffidi MA,Rumman A,Khan R,Ramaj M,Al-Mazroui A,Abunassar MJ,Jeyalingam T,Shetty A,Kandel GP,Streutker CJ,Grover SC.Clinical,endoscopic,and histologic characteristics of lymphocytic esophagitis:a systematic review.

2019;16:123-132 [PMID:30370453 DOI:10.1007/s10388-018-0649-1]

3 Hoversten P,Kamboj AK,Katzka DA.Infections of the esophagus:an update on risk factors,diagnosis,and management.

2018;31 [PMID:30295751 DOI:10.1093/dote/doy094]

4 Kim HP,Dellon ES.An Evolving Approach to the Diagnosis of Eosinophilic Esophagitis.

2018;14:358-366 [PMID:30166949]

5 O'Rourke A.Infective oesophagitis:epidemiology,cause,diagnosis and treatment options.

2015;23:459-463 [PMID:26371605 DOI:10.1097/MOO.0000000000000199]

6 Lundell LR,Dent J,Bennett JR,Blum AL,Armstrong D,Galmiche JP,Johnson F,Hongo M,Richter JE,Spechler SJ,Tytgat GN,Wallin L.Endoscopic assessment of oesophagitis:clinical and functional correlates and further validation of the Los Angeles classification.

1999;45:172-180 [PMID:10403727 DOI:10.1136/gut.45.2.172]

7 Beam AL,Kohane IS.Big Data and Machine Learning in Health Care.

2018;319:1317-1318 [PMID:29532063 DOI:10.1001/jama.2017.18391]

8 Gulshan V,Peng L,Coram M,Stumpe MC,Wu D,Narayanaswamy A,Venugopalan S,Widner K,Madams T,Cuadros J,Kim R,Raman R,Nelson PC,Mega JL,Webster DR.Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs.

2016;316:2402-2410 [PMID:27898976 DOI:10.1001/jama.2016.17216]

9 Stead WW.Clinical Implications and Challenges of Artificial Intelligence and Deep Learning.

2018;320:1107-1108 [PMID:30178025 DOI:10.1001/jama.2018.11029]

10 Weiss K,Khoshgoftaar TM,Wang D.A survey of transfer learning.

2016;3:9 [DOI:10.1186/s40537-016-0043-6]

11 Raghu M,Zhang C,Kleinberg J,Bengio S.Transfusion:Understanding transfer learning for medical imaging.

2019;3347-3357

12 Pogorelov K,Randel K,Griwodz C,Eskeland S,Lange T,Johansen D,Spampinato C,Dang-Nguyen D,Lux M.,Schmidt P,Riegler M,Halvorsen P.Kvasir:A Multi-Class Image Dataset for Computer Aided Gastrointestinal Disease Detection.MMSys'17 Proceedings of the 8th ACM on Multimedia Systems Conference (MMSYS);2017 June 20-23;Taipei,Taiwan.New York:Association for Computing Machinery,2017:164-169

13 Borgli H,Thambawita V,Smedsrud PH,Hicks S,Jha D,Eskeland SL,Randel KR,Pogorelov K,Lux M,Nguyen DTD,Johansen D,Griwodz C,Stensland HK,Garcia-Ceja E,Schmidt PT,Hammer HL,Riegler MA,Halvorsen P,de Lange T.HyperKvasir,a comprehensive multi-class image and video dataset for gastrointestinal endoscopy.

2020;7:283 [PMID:32859981 DOI:10.1038/s41597-020-00622-y]

14 Huang G,Liu Z,Van Der Maaten L,Weinberger KQ.Densely Connected Convolutional Networks.2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR);2017 Jul 21-26;Honolulu,HI,USA.New York:IEEE,2017:2261-2269

15 Choi J,Shin K,Jung J,Bae HJ,Kim DH,Byeon JS,Kim N.Convolutional Neural Network Technology in Endoscopic Imaging:Artificial Intelligence for Endoscopy.

2020;53:117-126 [PMID:32252504 DOI:10.5946/ce.2020.054]

16 Wimmer G,Vécsei A,Uhl A.CNN transfer learning for the automated diagnosis of celiac disease.2016 Sixth International Conference on Image Processing Theory,Tools and Applications (IPTA);2016 Dec 12-15;Oulu,Finland.New York:IEEE,2016:1-6

17 Song EM,Park B,Ha CA,Hwang SW,Park SH,Yang DH,Ye BD,Myung SJ,Yang SK,Kim N,Byeon JS.Endoscopic diagnosis and treatment planning for colorectal polyps using a deep-learning model.

2020;10:30 [PMID:31913337 DOI:10.1038/s41598-019-56697-0]

18 Namikawa K,Hirasawa T,Yoshio T,Fujisaki J,Ozawa T,Ishihara S,Aoki T,Yamada A,Koike K,Suzuki H,Tada T.Utilizing artificial intelligence in endoscopy:a clinician's guide.

2020;14:689-706 [PMID:32500760 DOI:10.1080/17474124.2020.1779058]

19 Cogan T,Cogan M,Tamil L.MAPGI:Accurate identification of anatomical landmarks and diseased tissue in gastrointestinal tract using deep learning.

2019;111:103351 [PMID:31325742 DOI:10.1016/j.compbiomed.2019.103351]

20 Majid A,Khan MA,Yasmin M,Rehman A,Yousafzai A,Tariq U.Classification of stomach infections:A paradigm of convolutional neural network along with classical features fusion and selection.

2020;83:562-576 [PMID:31984630 DOI:10.1002/jemt.23447]

21 Safarov S,Whangbo TK.A-DenseUNet:Adaptive Densely Connected UNet for Polyp Segmentation in Colonoscopy Images with Atrous Convolution.

2021;21 [PMID:33669539 DOI:10.3390/s21041441]

22 Jha D,Smedsrud PH,Johansen D,de Lange T,Johansen HD,Halvorsen P,Riegler MA.A Comprehensive Study on Colorectal Polyp Segmentation With ResUNet++,Conditional Random Field and Test-Time Augmentation.

2021;25:2029-2040 [PMID:33400658 DOI:10.1109/JBHI.2021.3049304]

23 Ker J,Wang L,Rao J,Lim T.Deep Learning Applications in Medical Image Analysis.

2018;6:9375-9389 [DOI:10.1109/ACCESS.2017.2788044]

24 Sun C,Shrivastava A,Singh S,Gupta A.Revisiting Unreasonable Effectiveness of Data in Deep Learning Era.2017 IEEE International Conference on Computer Vision (ICCV);2017 Oct 22-29;Venice,Italy.Washington:IEEE Computer Society,2017:843-852

25 Varghese TK Jr,Hofstetter WL,Rizk NP,Low DE,Darling GE,Watson TJ,Mitchell JD,Krasna MJ.The society of thoracic surgeons guidelines on the diagnosis and staging of patients with esophageal cancer.

2013;96:346-356 [PMID:23752201 DOI:10.1016/j.athoracsur.2013.02.069]

26 Short MW,Burgers KG,Fry VT.Esophageal Cancer.

2017;95:22-28 [PMID:28075104]