Radiomics-based predictive risk score: A scoring system for preoperatively predicting risk of lymph node metastasis in patients with resectable non-small cell lung cancer

Lan He, Yanqi Huang, Lixu Yan, Junhui Zheng, Changhong Liang, Zaiyi Liu

1Department of Radiology; 2Department of Pathology, Guangdong Provincial People's Hospital, Guangdong Academy of Medical Sciences,Guangzhou 510080, China

Abstract Objective: To develop and validate a radiomics-based predictive risk score (RPRS) for preoperative prediction of lymph node (LN) metastasis in patients with resectable non-small cell lung cancer (NSCLC).Methods: We retrospectively analyzed 717 who underwent surgical resection for primary NSCLC with systematic mediastinal lymphadenectomy from October 2007 to July 2016. By using the method of radiomics analysis, 591 computed tomography (CT)-based radiomics features were extracted, and the radiomics-based classifier was constructed. Then, using multivariable logistic regression analysis, a weighted score RPRS was derived to identify LN metastasis. Apparent prediction performance of RPRS was assessed with its calibration,discrimination, and clinical usefulness.Results: The radiomics-based classifier was constructed, which consisted of 13 selected radiomics features.Multivariate models demonstrated that radiomics-based classifier, age group, tumor diameter, tumor location, and CT-based LN status were independent predictors. When we assigned the corresponding score to each variable,patients with RPRSs of 0-3, 4-5, 6, 7-8, and 9 had distinctly very low (0%-20%), low (21%-40%), intermediate(41%-60%), high (61%-80%), and very high (81%-100%) risks of LN involvement, respectively. The developed RPRS showed good discrimination and satisfactory calibration [C-index: 0.785, 95% confidence interval (95% CI):0.780-0.790]. Additionally, RPRS outperformed the clinicopathologic-based characteristics model with net reclassification index (NRI) of 0.711 (95% CI: 0.555-0.867).Conclusions: The novel clinical scoring system developed as RPRS can serve as an easy-to-use tool to facilitate the preoperatively individualized prediction of LN metastasis in patients with resectable NSCLC. This stratification of patients according to their LN status may provide a basis for individualized treatment.

Keywords: Lymph node; radiomics; risk score; CT; non-small cell lung cancer

Introduction

Lung cancer is the leading cause of cancer-related mortality worldwide, with non-small cell lung cancer (NSCLC)accounting for approximately 85% of such deaths (1,2).Precise staging is the key to appropriate prognosis and treatment strategy decision (3). For patients with resectable(stage I-IIIA) NSCLC, surgical resection remains the primary and preferred approach to the treatment with the best chance for cure (4). As recommended by National Comprehensive Cancer Network (NCCN) guidelines,patients with resectable NSCLC should receive N1 and N2 node resection and a minimum of 3 N2 stations sampled or complete lymph node (LN) dissection during pulmonary resection (5). However, patients who had negative nodes by complete mediastinal LN dissection did not have improved survival compared with systematic LN sampling (6,7), and patients with positive LN involvement have a higher risk of disease recurrence. Therefore, accurate identification of LN involvement is crucial for prognosis and treatment strategy decision in patients with resectable NSCLC.

Although LN sampling or dissection plays an important role in precise nodal staging by identifying LN, they are available only invasively and postoperatively. The accurate preoperative assessment of LN involvement in patients with resectable NSCLC has a crucial effect on therapy planning, thus aiding in pretreatment decision making(8,9). As an alternative, imaging modalities like computed tomography (CT), positron emission tomography (PET)-CT and magnetic resonance (MR) are the most widely used for preoperative work-up, and are very important in detecting LN enlargement; however, a previous study has shown that they have limitations for their predictive value to differentiate benign nodes from malignant ones (9).

Recent advances in radiomics, which extract quantitative descriptors from routinely acquired medical images noninvasively, has provided deep insights into different fields of personalized medicine in oncologic practice,including tumor detection, subtypes classification, and therapy response assessment (10-14). It is worth noting that the recent studies have indicated that radiomics predictive models have been accepted as reliable tools to quantify risk by incorporating and illustrating important factors for oncologic prognosis and prediction (15-17). Although CTbased radiomics features assessments have been applied and demonstrated to be useful for prognosis prediction in patients with NSCLC (14,16,18), to our knowledge, no published study has determined whether the individual prediction of LN metastasis in resectable NSCLC could be achieved by a radiomics clinical scoring system based on preoperative CT images.

Therefore, this study aims to develop and validate a radiomics-based predictive risk score (RPRS) that preoperatively predicts risk of LN metastasis in patients with resectable NSCLC.

Materials and methods

Patients

The Institutional Review Board of Guangdong General Hospital approved this retrospective study, with informed consent waived. We retrospectively analyzed medical records from 717 consecutive patients who underwent surgical resection for primary NSCLC with systematic mediastinal lymphadenectomy from October 2007 to July 2016. The patient recruitment process was presented in Figure 1. All patients underwent contrast-enhanced chest CT images within two weeks before surgical resection.Between October 2007 and December 2013, patients were included to form the development cohort, and 423 cases were enrolled, while between January 2014 and July 2016,other patients were evaluated to form an independent validation cohort, and 294 cases were enrolled. Of these 717 patients in total, 277 patients additionally underwent PET-CT before resection. We selected these patients for subgroup analysis.

At baseline, clinical features on primary NSCLC (age,gender, smoking history, tumor location), and the acquisition date of CT imaging were recorded.

Figure 1 Patient recruitment process. NSCLC, non-small cell lung cancer; CT, computed tomography.

LN metastasis evaluation

LNs were dissected during surgical resection of NSCLC.The International Association for the Study of Lung Cancer (IASLC) has defined adequate LN sampling as: at least 3 mediastinal LN stations, station 7 in all case, station 5/6 with left upper lobe tumors and station 9 with lower lobe tumors; the sampling of at least 3 hilar LN stations is also recommended. The tissue was examined and the number of LNs was identified by pathologist postoperatively. Note that LN status in this study was defined by case.

CT image achievement

All patients underwent contrast-enhanced (reconstruction thickness of 1.25 mm) chest CT with one of two multidetector row CT (GE Light-speed Ultra 8; GE Healthcare,Hino, Japan; 64-slice LightSpeed VCT, GE Medical systems, Milwaukee, WI, USA). The CT images acquisition parameters were as follows: 120 kV; 160 mAs;0.5- or 0.4-second rotation time; detector collimation:8×2.5 mm or 64×0.625 mm; field of view, 350 mm × 350 mm; matrix, 512×512. The contrast-enhanced CT image was performed after a 25-second delay following intravenous administration of 85 mL of iodinated contrast material (Ultravist 370, Bayer Schering Pharma, Berlin,Germany) at a rate of 3 mL/s with a pump injector (Ulrich CT Plus 150, Ulrich Medical, Ulm, Germany) after routine non-enhanced CT. We then retrieved the images in DICOM format from the picture archiving and communication system (PACS; Carestream, Canada).

Assessment of tumor diameter and determination of conventional CT-based LN status

Tumor diameter of primary NSCLC on preoperative radiologic finding was measured by the maximum diameter of the primary tumor on the trans-axial image of CT image. Then, the conventional CT-based LN status was established preoperatively according to European Society of Thoracic Surgeons (ESTS) guidelines (19). In these guidelines, an LN with a diameter smaller than 1 cm in the short axis is defined as negative; otherwise, it is defined as positive. All these measurements were done by an experienced radiologist [15 years of clinical experience in chest CT interpretation (Reader 1)] at a GE AW 4.3 workstation.

Radiomics features extraction

Figure 2 showed the flowchart of radiomics analysis. The corresponding algorithms for the region-of-interest (ROI)settings and calculation of all radiomics features were described in Supplementary Materials. In total, 591 quantitative radiomics features were extracted from each patient's contrast-enhanced chest CT images. The details of features are summarized in Supplementary Table S1.

To ensure the reproducibility and accuracy, 80 patients were randomly selected for reproducibility analysis. The intra- and inter- observer agreement of the radiomics feature extraction were initially analyzed using inter- and intra-class correlation coefficients (ICCs). For the assessment of the interobserver agreement of the radiomics feature extraction, two radiologists with 15 years (Reader 1)and 12 years (Reader 2) of experience in chest CT interpretation performed the ROI-based radiomics feature extraction procedure, in a blind fashion, respectively. Then,Reader 1 repeated the procedure with an interval of 1 week for the assessment of the interobserver agreement of the radiomics feature extraction. An ICC greater than 0.75 indicated good agreement (20).

Figure 2 Flowchart of radiomics analysis pipeline. Image segmentation is performed on contrast-enhanced computed tomography (CT)images. Region of interest (ROI) is delineated initially around the tumor outline for the largest cross-sectional area by experienced radiologists. Radiomics features are extracted from the defined ROI of tumor on CT images, including tumor intensity, texture features,Gabor features, and wavelet features. For the analysis, radiomics features are compared with clinical data.

Statistical analysis

Statistical analysis was performed by R software (Version 3.0.1; http://www.Rproject.org). Packages used in this study are listed in Supplementary materials. Two-sided P＜0.05 indicated a significant difference.

Baseline continuous variables were compared between development and validation cohort using the Mann-Whitney U test, and categorical variables were compared using the Chi-square test. The univariate analysis of the correlation between LN positive and LN negative was conducted by using the univariate logistic analysis.Nonlinear effects of continuous variables (age, tumor diameter) were evaluated using restricted cubic splines(21,22); the relationship with LN status was not statistically significantly nonlinear (Wald P=0.049 for age; Wald P＜0.001 for tumor diameter), of which age and tumor diameter could be broken as dichotomous variable (age:≤60 years old and ＞60 years old; tumor diameter: ≤3 cm and ＞3 cm).

Radiomics-based classifier construction

To make the features values have properties of a standard normal distribution, radiomics features were normalized with z-score normalization (23). Then, we filtered features based on their independence from other features as determined by Pearson's correlation coefficient among the features (cut-off value of 0.9) (24). After that, feature selection was done in the development cohort using the least absolute shrinkage and selection operator method(LASSO) logistic regression analysis (25,26). The candidate predictive features with a zero-fit weight were selected.Thereafter, radiomics signature was built through the combination of the selected features weighted by their respective coefficients. The radiomics-based classifier was constructed with optimal cutoff point on the basis of maximum Youden index value determined by a receiver operating characteristic (ROC) analysis, of which patients were classified into high-risk or low-risk group according to the score of radiomics signature.

Derivation of fitting model for RPRS

Potential predictor variables, including age group, gender,smoking history, tumor diameter, tumor location, CTbased LN status, and radiomics-based classifier, were involved in the development of the prediction model.Multivariable logistic regression analysis with a stepwise selection procedure was used to develop the prediction model with candidate predictors based on development cohort. During modeling, backward-elimination procedures were used to confirm the significant covariates with the stopping rule of Akaike information criterion(AIC) (significance level α is set to 0.157) (27,28).

On the basis of the results of the final fitting model, a weighted scoring system (i.e., RPRS) incorporating parameters for prediction of LN metastasis was devised.Each independent predictor was a score criterion and was assigned a value based on the coefficients derived from the multivariable regression coefficients, which were multiplied by two and rounded (21).

Apparent performance assessment of RPRS

The predictive ability of RPRS was assessed in terms of discrimination and calibration. Discrimination was assessed with the C-index (28,29), and calibration was assessed graphically by plotting observed outcome against the predicted probability and tested with the Hosmer-Lemeshow goodness-of-fit test (P＜0.05 implies that the model does not calibrate perfectly) (29). The RPRS system was then studied to predict LN metastasis using Chi-square test. P＜0.05 was considered to be statistically significant.

Internal and external validation of RPRS

The RPRS was first validated using 1,000 resampled bootstrapping techniques based on the development cohort. Furthermore, we also assessed the validation of the RPRS using the independent validation cohort with Cindex and calibration curve.

Incremental predictive value of radiomics-based classifier in RPRS for LN metastasis

The incremental predictive value of radiomics-based classifier in RPRS, compared with clinical model which was established only with clinicopathological risk factors for predicting LN metastasis, was assessed with the net reclassification improvement (NRI) (30,31). NRI is a novel prospective measure which quantifies the correctness of upward and downward reclassification or movement of predicted probabilities as a result of adding a new marker.A value of negative percentage indicates a net worsening in risk classification for patients with LN metastases. To demonstrate the incremental value of RPRS to the PETCT node staging, conventional CT-based LN status, and other clinicopathological risk factors for individualized assessment of LN metastasis, we selected those 277 patients with PET-CT data for comparing the predictive value.The discrimination performance was assessed with Cindex.

Results

The clinical and laboratory data of development and validation cohort are listed in Table 1. In total, 423 patients with median age of 61 (range: 21-83) years old formed the development cohort, while another 294 patients with median age of 61 (range: 19-83) years old formed independent validation cohort. Median numbers of resected nodes were 20 (range: 5-69) in the development cohort and 18 (range: 7-50) in the independent validation cohort. Positive LN metastasis was 31.0% and 32.0% in the development and validation cohorts, respectively.There were no significant differences between two cohorts in clinical characteristics (P＞0.05, Table 1) and LN prevalence (P=0.776).

In univariate analyses, factors that were significantlyassociated with LN metastasis were age group, tumor diameter, tumor location, and CT-based LN status(P＜0.05, Table 1), except for gender and smoking history(both P＞0.05, Table 1).

Table 1 Baseline demographics and clinical characteristics of study population

Inter- and intra-observer reproducibility of radiomics features extraction

There were no statistically significant differences in the radiomics features between Reader 1's twice features extractions, nor between Reader 1's first and Reader 2's extraction (P values ranged from 0.753 to 0.807). The interobserver ICCs calculated on the basis of Reader 1's and Reader 2's first-extracted features ranged from 0.751 to 0.921, and the intraobserver ICCs calculated based on Reader 1's twice feature extractions ranged from 0.773 to 0.861. Therefore, all data analysis was based on the features extracted by Reader 1.

Radiomics-based classifier construction and corresponding discrimination performance

After filtering by correlation, 92 radiomics features which showed independence from other features were used for subsequent analysis. A correlation mapping for this radiomics features was showed in Figure 3. Then 13 radiomics features with non-zero coefficients in LASSO logistic regression model were selected. The radiomics signature was constructed based on the regression analysis with a radiomics score calculated for each patient (Table 2).After radiomics signature construction, Mann-Whitney U test showed statistically significant association between the LN status and the radiomics signature both in the training and validation cohort (all P＜0.0001). For the discrimination performance, radiomics signature yielded a C-index of 0.734 [95% confidence interval (95% CI):0.729-0.739], Additionally, the C-index in the internal validation was 0.734. In the independent validation cohort,the discrimination performance of radiomics signature yielded a C-index of 0.734 (95% CI: 0.727-0.741).

Figure 3 Correlation mapping for those radiomics features which were independent from other features. All Pearson's correlation coefficients among the features were not higher than 0.9.

Table 2 Radiomics features selected in LASSO regression analysis

The optimum cutoff of radiomics signature scores was generated by ROC with value of -1.048, then the radiomics-based classifier was constructed. We included those patients with a score lower than the cutoff into the low-risk group and those with a score higher than the cutoff into the high-risk group.

Derivation of RPRS

Table 3 shows the results of multivariate analysis performed in the development cohort. By multivariable logistic analysis, radiomics-based classifier and the following four independent clinical variables were found to be significant in predicting risk of LN metastasis: age group, tumor diameter, tumor location, and CT-based LN status. These four clinical variables remained independent factors in the clinical model (all P＜0.05).

On the basis of findings in Table 3, a working score RPRS was devised. RPRS was the sum of scores for five parameters: radiomics-based classifier (low-risk =0, highrisk =3), age group (≤60 years old =1, ＞60 years old =0),tumor diameter (≤3 cm =0, ＞3 cm =1), tumor location(peripheral =0, central =3), and CT-based LN status(negative=0, positive=1) (Table 4). The summary score corresponding to a predicted probability of LN metastasis is shown in Figure 4A. The sum of all values yielded a score for each patient that ranged from 0 to 9. We applied RPRS for classifying patients into five risk categories (Table 4).RPRSs of 0-3, 4-5, 6, 7-8, and 9 were associated with very low (0-20%), low (21%-40%), intermediate (41%-60%),high (61%-80%), and very high (81% to 100%) risks of LN involvement, respectively.

Performances of RPRS

The calibration curves of RPRS and clinical model both inthe development and validation cohort were satisfactory(Figure 5), which showed good agreement between prediction and actual observation. The Hosmer-Lemeshow test yielded a nonsignificant statistic both in the development and validation cohort (RPRS: P=0.622 for development cohort and P=0.086 for validation cohort;clinical model: P=0.546 for development cohort and P=0.084 for validation cohort).

Table 3 Logistic regression prediction models developed in development cohort

Table 4 Working RPRS for predicting risk of LN involvement

For the discrimination performance, RPRS showed improved discrimination compared with clinical model [Cindex: 0.825 (95% CI: 0.821-0.829) vs. 0.785 (95% CI:0.780-0.790); P=0.005] with NRI of 0.711 (95% CI:0.555-0.867), which was confirmed in the internal validation (C-index: 0.817 vs. 0.777). As for the independent validation, performance of RPRS still outperformed the clinical model [C-index: 0.810 (95% CI:0.804-0.816) vs. 0.777 (95% CI: 0.771-0.783); P=0.001]with NRI of 0.425 (95% CI: 0.187-0.664).

Subgroup analysis for patients with PET-CT data

Compared with either the conventional CT-based LN status (C-index: 0.729; 95% CI: 0.723-0.35), clinical model(C-index: 0.802; 95% CI: 0.796-0.808), or PET-CT status(C-index: 0.745; 95% CI: 0.739-0.751), RPRS showed a better discrimination capability (C-index: 0.822; 95% CI:0.816-0.828) in cohort of 277 patients with PET-CT data(P＜0.05 for each comparison).

RPRS for application in clinical practice

For application in clinical practice, RPRS was presented as a chart in Figure 4B. The cells of the chart were classified into five groups according to ranges of the risk of LN metastasis. For each cell of the chart, the risk for a patient with values of each predictor was estimated. For example, a 56-year-old patient with a radiomics signature score of 0.136, tumor diameter of 2 cm, positive value of CT-based LN status, and central of tumor location has a 61%-80%probability of LN involvement.

Discussion

In this study, we developed and validated a simple weighted novel score RPRS composed of radiomics-based classifier and clinicopathological variables for preoperative individualized prediction of LN metastasis in patients with NSCLC. To our knowledge, this is the first study of CTbased radiomics clinical scoring system using a novel approach for evaluating risk of LN metastasis in NSCLC.Patients with RPRSs of 0-3, 4-5, 6, 7-8, and 9 had distinctly very low (0-20%), low (21%-40%), intermediate(41%-60%), high (61%-80%), and very high (81%-100%)risks of LN involvement, respectively. RPRS had successfully stratified patients into a corresponding risk of LN metastasis. Additionally, the RPRS outperformed the traditional clinicopathological variables, conventional CTbased LN status, and PET-CT status.

Figure 4 Presentation and visualization of radiomics-based predictive risk score (RPRS). (A) Probability of lymph node (LN) metastasis assessment in resectable non-small cell lung cancer (NSCLC) based on RPRS. X-axis presents the sum score; Y-axis presents the predicted probability of LN metastasis corresponded to the sum score from Table 4. After calculating a sum score for a patient, this graph can be used to determine the corresponding predicted probability with the 95% confidence interval (95% CI); (B) A graphical chart to obtain predicted individual probability for LN metastasis based on RPSR. X-axis presents the values of tumor diameter, computed tomography (CT)-based LN status, and radiomics-based classifier; Y-axis presents the values of tumor location and age group status.

For the field of radiomics, intratumoral heterogeneity has been suggested to correlate with the clinical outcomes and the construction of a radiomics signature combining a panel of individual features as a prognostic imaging marker and may be a superior choice as it has now been regarded as a potential powerful method to facilitate better clinical decision making (10,12,13,15-17,32). In this study, we analyzed 591 radiomics features to assess intratumoral heterogeneity quantitatively based on the contrastenhanced CT image with the corresponding radiomicsbased classifier constructed. The identified classifier consisted of 13 selected radiomics features, which are consisted with results of recent studies of risk stratification(10,15,16). The function of the score of the radiomics signature used in our classifier has been investigated in previous studies (15,16,32,33). The developed radiomicsbased classifier showed significant association with LN metastasis, which successfully stratified patients into lowrisk and high-risk group (P＜0.0001).

Although no published studies found age to be an independent factor for LN status of NSCLC, in our study,age group was found to be significant association with LN status in both univariate analyses and multivariate analyses.Therefore, we kept age group as a candidate factor in the process of model development. In addition, preoperative CT-based LN status which can be easily obtained, was showed to be associated with actual LN status and was identified as an independent risk factor for the prediction of LN status.

The strengths of this study are that the final RPRS system includes radiomics-based classifier and clinicopathological parameters, which stratifies patients into categories at differing risk of LN metastasis. The observed proportion of LN positivity was satisfactorily consisted with the predicted probability according to RPRS in both the development and validation cohort in clinical practices. Unlike the traditional preoperative N staging methods, radiomics approach offers a robust way to characterize the intra-tumor heterogeneity noninvasively.Our result showed that the RPRS outperformed the clinical model not only in the development cohort, but also in the validation cohort. Both physicians and patients could perform a preoperative individualized prediction of the risk of LN metastasis using this easy-to-use clinical scoring system.

The present study has its limitations, as even though the analysis examines independent validation, it was performed in a single institute in one country. Risk factors for LN metastasis may be different in other countries because of the heterogeneity in CT image acquisition and clinical data collection in different institutions. However the available development data set is sufficiently large, splitting by time and developing a model using data from one period and evaluating its performance using the data from the other period (temporal validation) is a stronger approach in our study (28,29). Temporal validation is a prospective evaluation of a model, independent of the original data and development process according to the TRIPOD statement(Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis) (29), which can be considered external in time and thus deemed as a better alternative way to assess the generalizability of the prediction model. Furthermore, the RPRS constructed in our study was remaining a satisfactory prediction performance in the independent validation cohort, which justified that the RPRS hold great potential for clinical application in prediction of LN metastasis in patients with resectable NSCLC.

Figure 5 Calibration curves of radiomics-based predictive risk score (RPRS) and clinical model in development cohort and the independent validation cohort. RPRS (A) (P=0.622) and clinical model (C) (P=0.546) in development cohort; RPRS (B) (P=0.086) and clinical model (D)(P=0.084) in independent validation cohort. Diagonal line represents a perfect prediction.

A further limitation is that whole-tumor analysis was not performed in this study. Instead, radiomics features were extracted from the largest cross-sectional area of tumor.Theoretically, whole-tumor analysis may represent more diverse component of tumor heterogeneity by avoiding sampling errors that may result from single slice selection.However, the 3D whole tumor analysis is computationally more complex and time-consuming. Previous studies have reported that there was no significant difference between texture result of 2D and 3D analysis, with the conclusion that a single slice analysis being adequate (34). Therefore,we only delineated the largest cross-sectional area for features extraction. Further investigations are warranted to exploring the potential usefulness of 3D radiomics analysis for the prediction of LN metastasis in patient with resectable NSCLC.

Conclusions

RPRS integrated radiomics-based classifier and clinicopathological factors, can serve as a novel noninvasive and easy-to-use tool to facilitate the preoperatively individualized prediction of LN metastasis in patient with resectable NSCLC.

Acknowledgements

This study was supported by the National Key Research and Development Plan of China (No. 2017YFC1309100),the National Natural Scientific Foundation of China (No.81771912, 81901910, and 81701782), and the Provincial Science and Technology Plan Project of Guangdong Province (No. 2017B020227012).

Footnote

Conflicts of Interest: The authors have no conflicts of interest to declare.