Deep learning in hepatocellular carcinoma: Current status and future perspectives

Joseph C Ahn, Touseef Ahmad Qureshi, Amit G Singal, Debiao Li, Ju-Dong Yang

Joseph C Ahn, Gastroenterology and Hepatology, Mayo Clinic, Rochester, MN 55904, United States

Touseef Ahmad Qureshi, Debiao Li, Biomedical Imaging Research Institute, Cedars-Sinai Medical Center, Los Angeles, CA 90048, United States

Amit G Singal, Internal Medicine, University of Texas Southwestern Medical Center, Dallas, TX 75390, United States

Ju-Dong Yang, Karsh Division of Gastroenterology and Hepatology, Cedars-Sinai Medical Center, Los Angeles, CA 90048, United States

Abstract Hepatocellular carcinoma (HCC) is among the leading causes of cancer incidence and death.Despite decades of research and development of new treatment options, the overall outcomes of patients with HCC continue to remain poor.There are areas of unmet need in risk prediction, early diagnosis, accurate prognostication, and individualized treatments for patients with HCC.Recent years have seen an explosive growth in the application of artificial intelligence (AI) technology in medical research, with the field of HCC being no exception.Among the various AI-based machine learning algorithms, deep learning algorithms are considered state-of-the-art techniques for handling and processing complex multimodal data ranging from routine clinical variables to high-resolution medical images.This article will provide a comprehensive review of the recently published studies that have applied deep learning for risk prediction, diagnosis, prognostication, and treatment planning for patients with HCC.

Key Words: Hepatocellular carcinoma; Artificial intelligence; Deep learning

INTRODUCTION

Hepatocellular carcinoma (HCC) is an aggressive primary liver cancer that develops in the setting of chronic parenchymal liver diseases, and is among the top causes of cancer incidence and mortality worldwide[1,2].While the burden of HCC has been declining with effective antiviral therapy against hepatitis B virus (HBV) and hepatitis C virus (HCV), HCC incidence related to metabolic syndrome will likely continue to rise due to the dramatic increase in the prevalence of non-alcoholic fatty liver disease (NAFLD) in the general population[3].Decades of HCC research led to the development of a screening protocol, non-invasive diagnostic modalities based on imaging, and various treatment modalities including surgical, locoregional and systemic therapies[4,5].However, the overall outcomes of patients with HCC continue to remain poor and there are areas of significant unmet need in risk prediction, early detection, accurate prognostication, and individualized treatments for patients with HCC.

Patients with HCC generate enormous amounts of health data.While promising for researchers, ensuring that such high volumes of data are turned into actionable knowledge can be a significant challenge.Artificial intelligence (AI) is thought to be capable of synthesizing and analyzing multimodal data with superhuman degrees of accuracy or reliability, and recent years have seen a rapid growth in the application of AI to many fields of medicine including hepatology[6].This “AI revolution” over the past decade has been possible due to the advent of deep learning technology.Deep learning algorithms can process a broad spectrum of medical data from structured numeric data such as vital signs and laboratory values, high dimensional data from multi-omics studies, as well as digitized high-resolution images from various radiologic and histopathologic studies.This review aims to provide an overview as well as highlight examples of the many potential applications of deep learning to improve the care of patients with HCC.

AI, MACHINE LEARNING, AND DEEP LEARNING

AI-based approaches provide a variety of methods for a range of tasks and clinical application including image classification, organ and lesion segmentation, accurate extraction of key imaging features and measurements, tumor detection, stratification of high-risk subjects, prediction of disease and treatment outcome (Figure 1).Advancements in AI in recent years, particularly in the realm of medical image processing and analysis, offer an enormous range of automated tools for extracting precise measurements of biomarkers, revealing complex features, quantifying tissue characteristics and performing radiomics for deep analysis of raw imaging data.

Figure 1 Schematic representation of the relationships between the terms artificial intelligence, machine learning, and deep learning, and how deep learning can utilize multimodal data to improve care for patients with hepatocellular carcinoma.

The term “artificial intelligence” encompasses a broad range of technology that enables machines to perform tasks typically thought to require human reasoning and problem-solving skills[7].“Machine learning” is a branch of AI in which computer algorithms train on sample data to build a mathematical model that makes predictions or decisions without being explicitly programmed to do so[8].Machine learning algorithms can be broadly divided into supervised and unsupervised learning.Supervised learning algorithms train on sample data with labeled outcome data, and their goal is to learn the relationship between the input data and the outcomes to make accurate predictions about the outcome when provided with a new set of input data[9].Examples of supervised learning algorithms include traditional techniques such as linear regression and logistic regression, as well as more sophisticated techniques including support vector machines, random forest and gradient boosting.On the other hand, unsupervised learning algorithms train on unlabeled sample data and analyze the underlying structure or distribution within the data to discover new clusters or patterns[10].Examples of unsupervised learning algorithms include K-means and principle component analysis among many others.

Among the various AI-based machine learning algorithms, artificial neural networks (ANNs) consist of layers of interconnected mathematical formulas that enable them to analyze complex non-linear relationships[11].“Deep learning (DL)” refers to highly complex AI models utilizing multiple layers of ANNs and has recently emerged as a state-of-the-art AI technique for analyzing complex, high-dimensional healthcare data.Some of the commonly used DL techniques include convolutional neural networks (CNNs) and recurrent neural networks (RNNs)[12].CNNs have connective patterns resembling those of an animal visual cortex and can detect inherent spatial features of high dimensional images.RNNs have connections forming a directed graph along a temporal sequence, and therefore can be highly useful in time series prediction.

It is crucial to recognize that any AI-based machine learning algorithms require external validation in an independent dataset as models could be overfitted and end up overestimating the performance.In this review article, the performance characteristics of the various DL models are from the validation cohorts, and not the original derivation cohorts used to train the algorithms.

HCC CLINICAL DATA

Despite multiple available risk prediction tools for HCC, none have been rigorously validated or endorsed by major liver societies.Currently, HCC surveillance is recommended for patients with cirrhosis and high risk patients with chronic HBV infection[13].Accurate prediction models utilizing more specific risk factors for HCC development at individual levels would allow health systems to implement targeted screening strategies.Ioannouet al[14] trained a RNN to predict HCC development within 3 years using 4 baseline variables and 27 longitudinal variables from 48151 patients with HCV-related cirrhosis in the national Veterans Health Administration.The RNN model significantly outperformed logistic regression and exhibited an area under the curve (AUC) of 0.759 among all samples and an AUC of 0.806 among patients with sustained virologic response.Phanet al[15] surveyed 1 million random samples from Taiwan’s National Health Insurance Research Database between 2002 to 2010 to predict liver cancer among patients with viral hepatitis.The disease history of each patient was transformed into a 108 × 998 matrix and applied to a CNN, which predicted liver cancer with an AUC of 0.886 and an accuracy of 0.980.Another study by Namet al[16] constructed a deep neural network to predict 3-year and 5-year incidence of HCC in 424 patients with HBV-related cirrhosis on entecavir therapy.When applied to an external validation cohort of 316 patients, the DL model achieved a Harrell’s C-index of 0.782 and significantly outperformed 6 previously reported models based on traditional modeling.The same group also developed another DL model called the AI-based Model of Recurrence after Liver Transplantation (MoRALAI) to predict HCC recurrence after liver transplantation using variables such as tumor diameter, age, alpha-fetoprotein (AFP), and prothrombin time[17].The MoRAL-AI showed significantly better predictive performance compared to conventional models such as the Milan, UCSF, up-to-seven, and Kyoto criteria (C-index = 0.75vs0.64, 0.62, 0.50, 0.50, respectively;P< 0.001).

HCC MULTI-OMICS

Serum AFP has been widely used as a predictive and prognostic biomarker for HCC[18], but AFP has limited sensitivity for detecting early-stage HCC and its levels do not reliably correlate with disease progression[19].Recent advances in multi-omics related to HCC are expected to address this unmet need for novel biomarkers.Multi-omics refers to an approach to biological analysis which utilizes data sets from multiple "omics", such as the genome, epigenome, transcriptome, proteome, metabolome and microbiome.Multi-omics experiments generate an enormous amount of information, and various machine learning techniques including DL that can help with the computational challenges of processing and analyzing such high dimensional data.Xieet al[20] used gene expression profiling of peripheral blood to build an ANN model that classifies HCC patients from a control group.Using a nine-gene expression system, the ANN was able to distinguish HCC patients from controls with an AUC of 0.943, 98% sensitivity, and 85% specificity, although it should be noted that the control group was healthy individuals rather than patients with cirrhosis, which could have overestimated the performance of the model.Choiet al[21] proposed a novel networkbased DL method to identify prognostic gene signaturesviaG2Vec, a modified Word2Vec model originally used for natural language processing (NLP).When applied to gene expression data for HCC from the Cancer Genome Atlas (TCGA), G2Vec showed superior prediction accuracy for patient outcomes compared to existing gene selection methods and was able to identify two distinct gene modules significantly associated with HCC prognosis.Chaudharyet al[22] used RNA se×quencing, miRNA, and methylation data of 360 HCC patients from TCGA to build an autoencoder, which is an unsupervised feed-forward neural network.Using this DL model, they were able to distinguish patients with survival differences and identify specific mutations and pathways as predictors of aggressive tumor behavior.

RADIOLOGY

HCC diagnosis and segmentation

In recent years, there have been remarkable advances in the application of AI for the interpretation of medical imaging, primarily due to the use of DL algorithms using CNN[23].CNN algorithms trained on ultrasound, computed tomography (CT), or magnetic resonance imaging (MRI) images have shown excellent performances in detection of lesions, classification of lesions, segmentation of organs or anatomic structures, and imaging reconstruction[24].

In 2012, Strebaet al[25] prospectively studied contrast-enhanced ultrasound images of 112 patients to train an ANN that classified five different types of liver tumors.The ANN showed promising performances with accuracies of 94.5% in the training set and 87.1% in the testing set.In 2017, Hassanet al[26] reported using the stacked sparse auto-encoder, an unsupervised DL technique, to segment and classify liver lesions on ultrasound images with a classification accuracy of 97.2%.Additionally, Bhartiet al[27] built a CNN using echotexture and roughness of liver surface on 754 segmented ultrasound images, which differentiated between normal liver, chronic liver disease, cirrhosis, and HCC with a classification accuracy of 96.6%.Schmauchet al[28] also created a CNN which detects and characterizes benign and malignant focal liver lesions on 2-D ultrasound images from 367 patients from various institutions.When applied to a new dataset of 177 patients, the model achieved a weighted mean AUC of 0.891.Recently, Breharet al[29] conducted a study comparing CNN’s performance for HCC detection on ultrasound images against conventional machine learning algorithms including multi-layer perceptron, support vector machines, random forest and AdaBoost.The CNN achieved an AUC of 0.95% with 91.0% accuracy, 94.4% sensitivity, and 88.4% specificity and significantly outperformed the conventional machine learning algorithms.Beyond detecting the actual presence of HCC on ultrasound images, studies have also attempted to predict the risk of future HCC development based on analyzing the ultrasound images of liver parenchyma in patients without HCC.For example, Jinet al[30] performed a DL radiomics analysis on 2-D shear wave elastography and corresponding B-mode ultrasound images of 434 chronic HBV patients, which predicted 5-year HCC development with AUC of 0.900 in the test cohort.

In addition to ultrasound images, cross-sectional imaging from CT or MRI studies serve as an extremely abundant and promising source of data for DL.In 2018, Yasakaet al[31] used CT image sets of liver masses from 460 patients to train a CNN that can classify liver lesions into five categories of: (1) HCC; (2) Other malignant tumors; (3) Indeterminate masses; (4) Hemangiomas; and (5) Cysts with a median AUC of 0.92.Shiet al[32] showed that incorporation of a CNN enabled identification of HCC using a three-phase CT imaging protocol with a diagnostic accuracy similar to that of a fourphase protocol, which would allow patients to receive lower doses of radiation.Segmentation of HCC, liver parenchyma, and other organs on CT scan is very important for determination of tumor extent and treatment planning, but manual contouring of the images is highly time-consuming and subject to inter-observer variability.The 2017 International Conference On Medical Image Computing Computer Assisted Intervention called for a Liver Tumor Segmentation Benchmark (LITS) challenge, encouraging researchers to develop automatic segmentation algorithms to segment liver lesions using 200 CT scans (training: 130; testing: 70) provided by clinical sites around the world.Several teams participating in the challenge have developed DL algorithms with promising performances for HCC segmentation using CT images[33-37].Beyond the LITS challenge, there are ongoing research efforts to improve segmentation using different architectures of DL networks[38-42].

Hammet al[43] used MRI images from 494 patients to train a CNN which can classify hepatic lesions into six different categories.When applied to random cases in the test set, the CNN outperformed expert radiologists (90% sensitivity and 98% specificityvs82.5% sensitivity and 96.5% specificity) and especially for HCC detection (90% sensitivityvs60%-70% sensitivity).The same group conducted additional studies to make their CNN interpretable by generating highlighted feature maps corresponding to liver lesions[44].Wuet al[45] built a CNN using multiphase MRI images and achieved an AUC of 0.95 for distinguishing Liver Imaging Reporting and Data System (LI-RADS) grade 3 from LI-RADS 4 and 5 lesions for HCC diagnosis.Zhenet al[46] also trained a CNN model combining unenhanced MRI images and clinical variables from 1210 patients with liver tumors, which demonstrated diagnostic performances on par with three experienced radiologists using enhanced MRI images.

HCC prognostication, treatment planning, and response to treatment

In addition to serving as accurate and efficient tools for diagnosis of HCC, DL models utilizing radiology data can also be used for prognostication, treatment planning, and assessing tumor response to therapy.Vascular invasion is a key prognostic element in patients with HCC.Recent studies developed CNN models with promising ability to detect microvascular invasion on MRI images of HCC patients undergoing surgical resection[47-49].Anet al[50] used an unsupervised CNN-based deformable image registration technique to assess the relationship between ablative margins and local tumor progression in 141 patients with single HCC who underwent microwave ablation, and demonstrated that patients with ablative margins < 5 mm were at significantly higher risk of local tumor progression.Liuet al[51] developed a DL radiomics model to predict responses to trans-arterial chemoembolization (TACE) using ultrasound images of 130 HCC patients, which accurately predicted TACE response with an AUC of 0.93.The same group also assessed their ultrasound-based DL radiomics model to predict 2-year progression-free survival among 419 HCC patients and facilitate optimized treatment selection.Penget al[52] trained a residual CNN model to predict response to TACE using CT images from 562 patients with intermediate-stage HCC undergoing TACE, which showed accuracies of 85.1% and 82.8% in two external validation cohorts.Another study developed a DL score for disease-specific survival by using CT images in a cohort of 243 patients with HCC treated with TACE, with a higher score predicting poor prognosis [hazard ratio (HR): 3.01; 95% cumulative incidence (CI): 2.02-4.50][53].Finally, Zhanget al[54] built a DLbased model predicting overall survival using CT images from 201 patients with unresectable HCC treated with TACE and sorafenib, which achieved superior predictive performance compared to the clinical nomogram (C-index of 0.730vs0.679,P= 0.023).

HCC PATHOLOGY

Automated interpretation of histopathologic images from liver biopsy is another major area of medical imaging in patients with HCC where DL can be utilized.In addition to effectively replicating the human pathologists’ jobs of diagnosing and grading HCC, DL models can help identify and analyze additional complex imaging features and patterns which are related to specific mutations and disease prognosis.Linet al[55] used images from multiphoton microscopy of 113 HCC patients to train a CNN with over 90% accuracy for determining HCC differentiation.Kianiet al[56] developed a CNN-based “Liver Cancer Assistant” which accurately differentiated hematoxylin and eosin (H&E) images of HCC and cholangiocarcinoma and helped improve the diagnostic performance of nine pathologists.Liaoet al[57] used TCGA dataset for training a CNN that distinguished HCC from adjacent normal tissues with perfect performance (AUC: 1.00) and predicted the presence of specific somatic mutations with AUCs over 0.70.Wanget al[58] trained a CNN for automated segmentation and classification of individual nuclei at single-cell levels on H&E-stained tissue sections of HCC tumors from TCGA, and performed feature extraction to identify 246 quantitative image features.Then, a clustering analysis by an unsupervised learning approach identified three distinct histologic subtypes which were independent of previously established genomic clusters and had different prognosis.Chenet al[59] trained a CNN for automatic grading of HCC tumors on histopathological H&E images, which showed 96% accuracy for benign and malignant classification and 89.6% accuracy for the degree of tumor differentiation, and predicted the presence of specific genetic mutations.

Luet al[60] applied three pre-trained CNN models to extract imaging features from HCC histopathology and performed Cox proportional hazards analysis to predict overall survival and disease-free survival, and observed significant correlations between the imaging features and established biological pathways.Saillardet al[61] used two DL algorithms based on whole-side digitized histological slides from 194 patients with HCC to predict the survival of patients treated by surgical resection.When tested on an independent validation set from TCGA, both DL models had a higher discriminatory power than a score combining all baseline variables associated with survival.Shiet al[62] built an interpretable DL framework using pathologic images from 1445 patients with HCC and developed a “tumor risk score” which showed prognostic performances independent of and superior to clinical staging systems and stratified patients into five groups of different prognosis.A recent study by Yamashitaet al[63] developed a histopathology-based DL based system which stratified patients with risk scores for postsurgical recurrence of HCC.

FUTURE DIRECTION

There are several key issues to address before DL-based AI models can be universally implemented in real world clinical practice settings.Due to their complexity, DL models are traditionally considered to be “black-box” models, meaning humans cannot understand how the DL models make their predictions.Interpretability of the DL models are crucial for physicians to accept and trust them in everyday clinical practice, and for troubleshooting and improving the models for rare cases.This is being addressed by recent developments in various “explainable AI” techniques but currently there is no clear consensus on the best methodology.Another potential limitation is the generalizability of the individual DL algorithms.Concerns have been raised that AI algorithms developed at highly specialized academic medical centers using their own patients’ data may over-represent certain groups of patients and not accurately reflect the real-world population of patients seen at local community hospitals.Finally, AI models, like other prediction models, are often not publicly available, limiting external validation.Independent validation of the proposed model and comparison to old models are as important as deriving new models.Large-scale,prospective, multi-centered studies involving diverse populations with external validation will be necessary before DL algorithms can be widely accepted.

from other FLLs Hamm et al[43] 2019 T: 434 FLL; V: 60 FLL 1 center in United States CNN MRI images FLL type CNN achieved 90% sensitivity and 98% specificity for classifying FLLs and AUC of 0.992 for HCC classification Wang et al[44] 2019 T: 434 FLL; V: 60 FLL 1 center in United States CNN MRI images FLL type Interpretable DL system achieved 76.5% PPV and 82.9% sensitivity for identifying correct radiological features Wu et al[45] 2020 89 liver tumors; (60: 20: 20)1 center in United States CNN MRI images LI-RADS grading CNN achieved AUC of 0.95, 90% accuracy, 100% sensitivity and 83.5% PPV for LI-RADS grading of liver tumors Zhen et al[46] 2020 T: 1210 liver tumors; V: 201 liver tumors 1 center in China CNN MRI images Liver tumor type CNN combined with clinical data showed AUC of 0.985 for classifying HCC with 91.9% agreement with pathology Radiology-based HCC prognostication, treatment planning, and response to treatment Zhang et al[47] 2021 T: 158 HCC; V: 79 HCC 1 center in China CNN MRI images MVI in HCC CNN achieved AUC of 0.72, 55% sensitivity, and 81% specificity for preoperative MVI in HCC patients Wang et al[48] 2020 T: 60 HCC; V: 40 HCC 1 center in China CNN MRI images MVI in HCC Fusion of deep features from MRI images yielded AUC of 0.79 for MVI prediction in HCC patients Jiang et al[49] 2021 405 HCC; (T: 80%, V: 20%)1 center in China CNN CT images MVI in HCC CNN achieved AUC of 0.906 for prediction of MVI.Mean survival was significantly better in the group without MVI An et al[50] 2020 141 single HCC resect MWA 1 center in China CNN MRI images Ablative margin Deep learning model accurately estimated ablative margins and risk of local tumor progression Liu et al[51] 2020 T: 89 HCC resect TACE; V: 41 HCC rec.TACE 1 center in China CNN Ultrasound images Response to TACE Deep learning radiomics model predicted tumor response to TACE with AUC of 0.93 Peng et al[52] 2020 T: 562 HCC resect TACE; V:227 HCC rec.TACE 3 centers in China CNN CT images Response to TACE Deep learning model had accuracies of 85.1% and 82.8% for predicting TACE response in 2 validation cohorts Liu et al[53] 2020 243 HCC resect TACE (6:1:3 split)1 center in China CNN CT images Post-TACE survival Higher DL score was an independent prognostic factor and predicted overall survival with AUCs of 0.85-0.90 Zhang et al[54] 2020 201 HCC resect TACE + sorafenib (T: 120, V: 81)3 centers in China CNN CT images OS on TACE + sorafenib Deep learning signature achieved C-index of 0.714 for predicting OS in HCC patients receiving TACE + sorafenib Histopathology-based HCC diagnosis, subtyping, and outcome predictions Lin et al[55]2019 113 HCC 1 center in China CNN Histopath images HCC differentiation CNN achieved an accuracy of 0.941 for determining HCC differentiation on multiphoton microscopy Kiani et al[56] 2020 70 WSI (35 HCC, 35 CC)TCGA CNN Histopath images HCC vs CC CNN-based “Liver Cancer Assistant” accurately differentiated HCC vs cholangiocarcinoma Liao et al[57] 2020 T: 491 HCC; V: 455 HCC TCGA; 1 center in China CNN Histopath images HCC detection, mutations CNN distinguished HCC from adjacent tissues with AUC of 1.00 and predicted specific mutations with AUC over 0.70 Unsupervised clustering identified 3 histological subtypes complementing molecular pathways and Wang et al[58] 2020 T: 99 HCC; V: 205 HCC TCGA CNN Histopath images Histological HCC subtype

ANN: Artificial neural network; AUC: Area under the curve; CC: Cholangiocarcinoma; CNN: Convolutional neural network; CV: Cross-validation; FLL: Focal liver lesion; GDC: Genomic Data Commons; HBV: Hepatitis B virus; HCC: Hepatocellular carcinoma; HCV: Hepatitis C virus; HV: Healthy volunteers; LT: Liver transplant; MVI: Microvascular invasion; MWA: Microwave ablation; OS: Overall survival; PFS: Progression-free survival; RFA: Radio-frequency ablation; RNN: Recurrent neural network; SR: Surgical resection; STS-net: Spatial transformed similarity network; SVR: Sustained virologic response; T: Training; TCGA: The Cancer Genome Atlas; V: Validation; VHA: Veterans Health Administration; WSI: Whole slide image; CT: Computed tomography; MRI: Magnetic resonance imaging; NHIRD: National Health Insurance Research Database; TNM: Tumor, Nodes, Metastasis; TACE: Trans-arterial chemoembolization; LI-RADS: Liver Imaging Reporting and Data System.

A currently under-explored, but highly promising and exciting area for the application of DL is the field of autonomous robotics.In a recent editorial, Gumbset al[64] state that while the current form of robotic surgery seems like a form of minimally invasive surgery, the true power of robotic surgery exists in its potential to create autonomous actions.Recently, a DL-based surgical instrument tracking algorithm was able to closely track the instruments during robotic surgery and evaluate the surgeons’ performance, demonstrating that DL algorithms can learn the correct steps of robotic surgery[65].With the help of DL and other AI technologies, it may be possible to imagine a future where fully autonomous robots perform resection of large, complex HCC in ways that no human surgeons can mimic.However, there are significant barriers before the idea of fully autonomous robotic surgery can become a reality, including the current technical limitations of autonomous surgical robotics, as well as the hesitation of patients and providers to fully trust autonomous robots to perform invasive operations.“Explainability” of the DL algorithms will be critical here, as humans would need to be able to understand and correct every single mistake that an autonomous robot makes during surgery.Therefore, for the foreseeable future, DL will most likely remain as a helpful, adjunctive tool to assist human surgeons.

CONCLUSION

This review has provided a comprehensive overview of various ways in which DL algorithms can be employed to assist medical providers and enhance the care of patients with HCC (Table 1).DL algorithms not only can efficiently and accurately replicate the same jobs performed by human physicians, but more importantly can help discover novel biologic pathways and disease subgroups with clinical significance by processing and analyzing complex high-dimensional data in ways impossible for the human brain.

Table 1 Studies applying deep learning for hepatocellular carcinoma

Despite some important limitations to overcome, application of state-of-the-art AI technologies such as DL for the care of patients with HCC is no longer a futuristic idea but is rapidly becoming a reality.Most of the studies covered in this review were published within the past two years, and the number of studies utilizing DL continues to increase exponentially.We anticipate that DL algorithms will soon take a major role in the diagnosis, prognostication, and treatment of patients with HCC.