Development and validation of a simple-to-use nomogram to predict liver metastasis in patients with pancreatic neuroendocrine neoplasms: a large cohort study
BMC Gastroenterology volume 21, Article number: 101 (2021)
Liver metastasis is an important prognostic factor for pancreatic neuroendocrine neoplasms (pNENs), but the relationship between the clinical features of patients with pNEN and liver metastasis remains undetermined. The aim of this study was to establish and validate an easy-to-use nomogram to predict liver-metastasis in patients with pNEN.
We obtained the clinicopathologic data of 2960 patients with pancreatic neuroendocrine neoplasms from the Surveillance, Epidemiology and End Results (SEER) database between 2010 and 2016. Univariate and multivariate logistic regression were done to screen out independent influencing factors to establish the nomogram. The calibration plots and the area under the receiver operating characteristic curve (AUC) were used to evaluate the performance of nomogram. Decision curve analysis (DCA) was applied to compare the novel model with the conventional predictive methods.
A total of 2960 patients with pancreatic neuroendocrine neoplasms were included in the study. Among these, 1974 patients were assigned to the training group and 986 patients to the validation group. Multivariate logistic regression identified, tumor size, grade, other site metastasis, T stage and N stage as independent risk factors. The calibration plot showed good discriminative ability in the training and validation groups, with C-indexes of 0.850 for the training cohort and 0.846 for the validation cohort. The AUC values were 0.850 (95% CI 0.830–0.869) and 0.839 (95% CI 0.812–0.866), respectively. The nomogram total points (NTP) had the potential to stratify patients into low risk, medium risk and high risk (P < 0.001). Finally, comparing the nomogram with traditional prediction methods, the DCA curve showed that the nomogram had better net benefit.
Our nomogram has a good ability to predict liver metastasis of pancreatic neuroendocrine neoplasms, and it can guide clinicians to provide suitable prevention and treatment measures for patients with medium- and high-risk liver metastasis.
Pancreatic neuroendocrine neoplasms (pNENs) are relatively rare, with an estimated annual incidence of approximately 3.65/10,000 people per year [1, 2]. The natural disease progression of pancreatic neuroendocrine tumors can lead to local lymph node, liver, lung, and bone metastases. Among these, liver metastases are the most common. It is reported that more than 60% of patients with pNEN have liver metastases . Studies have found that liver metastasis is an important risk factor for prognosis . The treatment strategy and prognosis of pNEN largely depend on whether there is liver metastasis. Therefore, early diagnosis and treatment of pNEN patients with liver metastases can significantly improve the quality of life and prognosis. Due to the lack of typical clinical manifestations of nonfunctional pNEN in the early stage, 20% to 30% of pNEN patients have liver metastases when diagnosed, which seriously affects their quality of life and long-term survival [5, 6]. Therefore, it is critical that clinicians accurately identify the risk of liver metastases in patients with pNEN for optimal treatment strategies.
The routine examination for excluding liver metastasis is a computed tomography (CT), but it has low sensitivity and specificity for microscopic liver metastasis . Previous studies have shown that liver metastases from neuroendocrine tumors are correlated with a variety of clinicopathological factors, including histological type, primary site, tumor size, lymphatic invasion, and proliferative activity [8, 9]. However, the above studies are limited to some fragmentary risk factors and small sample sizes. It is essential to explore the relationship between clinicopathological factors and liver metastasis based on a large sample database and to develop a prediction model of the risk of liver metastasis in pNEN patients.
In this study, we constructed and validated a simple-to-use nomogram model. With this prediction model, clinicians can accurately identify patients with pNEN at medium and high-risk of liver metastasis patients with pNEN and provide patients with personalized prevention and treatment strategies.
Study population and data sources
The data were extracted from the Surveillance, Epidemiology, and End Results (SEER) database using SEER*Stat software Version 8.3.6. Data from patients with pNEN diagnosed in 2010–2016 who had complete information including age, sex, race, primary site, grade, marital status, T stage, N stage, tumor size, histology, and metastasis site, were included in the study. Pancreatic neuroendocrine neoplasms were selected on the basis of International Classification of Disease codes (ICD-O-3), including carcinoma (8150), malignant beta-cell tumor (8151), malignant alpha-cell tumor (8152), G-cell tumor (8153), VIPoma (8155), malignant somatostatinoma (8156), carcinoid tumor (8240), carcinoid tumor (8240) and atypical carcinoid tumor (8249). The exclusion criteria were as follows: (1) patients without definitive liver metastasis data; (2) patients with more than one primary cancer; and (3) patients without definitive grade and metastasis site information.
Construction and validation of the nomogram
We randomly assigned two-thirds of our patients to the training group and the rest of them were assigned to the validation group. The chi-square tests was used to compare the baseline characteristics of the two groups. In the training group, liver metastasis risk factors were determined through the univariate logistic regression. Variates with P values less than 0.05 were used in the multivariate logistic regression analysis. Based on the coefficients of the independent risk factors in the multivariate analysis, the prediction model was visualized in the form of the nomogram. To draw this nomogram, we needed to assign a score of 0–100 to each factor. The coefficients of the above multiple logistic regression results were transformed and are shown in the form of graphs. The nomogram's ruler for each indicator was based on the index with the most influence. The greater the influence of the risk factors, the higher the nomogram score . The whole process was done in R 3.6.2 software. The details of building the nomogram and R codes are provided in Additional file 1: Supplement Method 1.
The concordance index (C-index), the receiver operating characteristic curve (ROC), and the area under the curve (AUC) were used to evaluate the predictive accuracy and discrimination of the nomogram. The decision curve (DCA)  was used to evaluate the clinical utility of the nomogram, and compare nomogram with conventional predictive risk factors including grade, T stage, and tumor size. The details of DCA curve building and R codes were provided in Additional file 1: Supplement Method 2.
Risk group stratification and statistical analysis
According to the characteristics of each patient's risk factors, a straight line was drawn to the "point" at the top of the model to obtain each factor score. The total score was obtained by summing the scores for all the factors. To further discriminate the risk groups of liver metastasis, the patients were categorized into low-, medium- and high-risk groups based on the nomogram total points (NTP) of every pNEN patients. The optimal two cut-off values for NTP were calculated by X-tile software. The cut-off value was then validated in the validation group. The chi-square test was used to compare all risk groups.
Statistical analysis was performed using SPSS software version 23 and R version 3.6.2 software. For all analyses, P values less than 0.05 were considered statistically significant.
Baseline characteristics of the patients
There were 2960 eligible patients with pNEN who were included in this study. A total of 1974 patients were allocated to the training group and 986 cases were allocated to the validation group. The two groups had no significant difference in baseline characteristics (all P > 0.05) (Table 1). In the entire study group, the median age was 58 years. The majority of the patients were white (n = 2268, 76.6%) and married (n = 1814, 61.3%). The pancreatic tail was the most common site of pNEN tumors (n = 1058, 35.7%). The main pathological grade of neoplasms was G1 (n = 2068, 69.9%), followed by G2 (n = 577, 19.5%). During the whole follow-up, most of the patients were alive (81.9%) and only 535 (18.1%) patients died. There were 419 (21.2%) and 222 (22.5%) pNEN patients with liver metastases in the training group and validation group, respectively. Liver metastasis was found to be correlated with sex, primary site, grade, T stage, N stage, tumor size and other site metastasis in pNEN patients (Table 2).
Independent risk factors and nomogram construction
Univariate regression analysis was used to screen the risk factors for liver metastasis. The significant risk variables were included in the multivariate regression analysis. The results of multivariate logistic regression analysis showed that grade, T stage, N stage, tumor size, and other site metastasis were independent risk factors for liver metastasis (Table 3). All the above variables were used to establish the nomogram model (Fig. 1). In this model, it was found that grade, T stage and tumor size had the greatest impact on liver metastasis, followed by N stage and other site metastasis. The probability of liver metastasis in each pNEN patient can be computed by adding up the corresponding scores of all the independent risk factors.
Nomogram validation and risk classification
The calibration plot showed good agreement in the training and validation group (Fig. 2A, B). The C-index of liver metastasis prediction was 0.850 and 0.846 in the training and validation group, respectively. When the ROC curves were plotted, the training group had an AUC of 0.850 (95% CI 0.830–0.869), which was verified in the validation group (AUC = 0.839, 95% CI 0.812–0.866) (Fig. 2C, D). Decision curve analysis (DCA) was done next (Fig. 3), which is a novel method that can evaluate the clinical practicality of models. The results showed that the nomogram had satisfactory net benefits among most of the threshold probabilities in both groups. Compared with conventional predictive methods, our nomogram was more exact in predicting liver metastasis.
The training group was divided into three subgroups based on the two optimal NTP cut-off values. According to the X-tile calculation results, the optimal cut-off values were 105.5 and 156.0 respectively (Fig. 4A). The patients were divided into low-risk (NTP < 105.5, n = 1278 (64.7%)), medium-risk (105.5 ≤ NTP < 156.0, n = 368 (18.6%)) and high-risk subgroups (NTP ≥ 156.0, n = 328 (16.6%)). The same cut-off values were used for grouping in the validation group. Notably, the high-risk pNEN patients were more likely to have liver metastases in both groups (P < 0.05) (Fig. 4B, C).
Although the natural history of many pancreatic neuroendocrine tumors is characterized by slow progression and inertia, there are still patients with metastasis during the course of the disease, especially liver metastasis. For patients with resectable pNEN with liver metastases, active surgical resection of primary and liver metastases should be the preferred treatment. Previous studies have reported that surgical resection of primary and metastatic lesions could improve quality of life and prolong survival, with a 5-year survival rate of 60–80% [12,13,14,15,16]. However, due to the limited sensitivity of the current imaging modalities, early pNEN patients with liver metastasis have a high rate of missed diagnosis, which makes the patients lose their best chance of radical surgical resection when they are diagnosed. Liver biopsy has a high diagnosis rate, but it increases the risk of distant metastasis and leads to reduced survival time . Therefore, a noninvasive and simple-to-use method is required for predicting the likelihood of liver metastasis in patients with pNEN. In our study, a novel nomogram was developed for predicting the probability of liver metastasis of pNEN based on a large database. The results demonstrated that the nomogram model is significantly discriminative and thus provides an individualized prediction of the probability of liver metastasis.
Our study mainly focussed on the clinical characteristics of pNEN patients with liver metastasis, and demonstrated that grade, T stage, N stage, tumor size, and other site metastasis were independent risk factors for liver metastasis. The G1-2 group had a higher percentage of pNEN patients with liver metastases (70.5%) than the other groups. This result is similar to that of Ruzzenente (81.9%) . In addition, Spolverato  found that nonfunctional and moderate-to-poor tumors were more likely to have liver metastases. We speculate the reason that the G1-2 non-functional tumor easily neglected in the early stage due to the lack of obvious clinical symptoms, and the tumor is already in advanced stage when diagnosed. Previous studies have shown that the main cause of liver metastases is vascular invasion . During hematogenous metastasis, the liver is the first filter for tumor cell invasion. In this study, we found that the size and T stage of the primary tumor were closely related to the infiltration of neuroendocrine tumor cells into the liver. The size of the tumor is directly related to the T stage. The larger the primary tumor size, the more aggressive it is towards surrounding organs or blood vessels. This study also confirmed that the larger the tumor and the higher the T stage, the greater the probability of liver metastasis.
Apart from the route of hematogenous metastasis, pancreatic neuroendocrine tumor may also metastasize to distant sites via lymphatic pathways. In our study, LN metastasis was identified as an independent risk factor in predicting liver metastasis. Positive lymph nodes are a common sign before distant metastasis, which has been demonstrated in other tumors [21, 22]. In our study, 47.3% of patients with liver metastases had positive lymph nodes. Therefore, more attention should be paid to the presence of metastasis in the liver and other sites in patients with positive lymph nodes. Besides liver metastasis, there were also other distant site metastases (bone, lung, brain). In this study, more than 72.2% of pNEN patients with other site metastases also had liver metastases. This result reveals that there are probably other metastases when liver metastases are found. This finding is consistent with other studies [23,24,25].
The advice given to the patient and the choice made among treatment options are based on the assessment of the individual's prognosis and risk . Nomograms are graphical representations of statistical prediction models that predict the probability of an event occurring . Thus, the variables contained in the nomogram should be easy to obtain and measure. In this study, we developed a nomogram to predict live metastasis in patients with pNEN. Our nomogram model has been shown to have good discernment with high C-indexes and AUCs, in both groups. Finally, DCA curves were generated to show that the nomogram could be used to obtain a better net benefit within the derived probabilities than traditional prediction methods .
There are some limitations to this study. The major limitation of our study is the lack of important variables, such as surgical margin, Ki-67 and other molecular biomarkers. The Ki-67 index and surgical margin play an important role in the prognosis of pNEN . Unfortunately, the absence of Ki-67 and surgical margins in the SEER database made it impossible to assess its role in predicting liver metastasis of pNEN. Second, our nomogram has been verified to have excellent prediction capabilities, but further external validation based on a large multicenter data cohort is still required. Finally, since the SEER database is a retrospective database, selection bias cannot be completely avoided. Therefore, bootstrapping with 1000 resamples was performed in this study to minimize bias.
In conclusion, we successfully created and validated a simple-to-use nomogram for predicting the probability of liver metastasis in pNEN patients. This model has good predictive power and it is easy for the clinician to use. By assessing the risk of liver metastasis, clinicians could realize individualized treatment and take necessary preventive measures to reduce the risks borne by patients and improve their quality of life and prognosis.
Availability of data and materials
The datasets analyzed during the current study are available in the SEER repository (https://seer.cancer.gov/).
Pancreatic neuroendocrine neoplasms
Nomogram total points
Receiver operating characteristic
International classification of diseases for oncology, third edition
Surveillance, epidemiology, and end results
The area under the curve
The decision curve
Frilling A, Modlin IM, Kidd M, Russell C, Breitenstein S, Salem R, Kwekkeboom D, Lau WY, Klersy C, Vilgrain V, et al. Recommendations for management of patients with neuroendocrine liver metastases. Lancet Oncol. 2014;15(1):e8–21.
Sandvik OM, Soreide K, Gudlaugsson E, Kvaloy JT, Soreide JA. Epidemiology and classification of gastroenteropancreatic neuroendocrine neoplasms using current coding criteria. Br J Surg. 2016;103(3):226–32.
Bertani E, Fazio N, Radice D, Zardini C, Grana C, Bodei L, Funicelli L, Ferrari C, Spada F, Partelli S, et al. Resection of the primary tumor followed by peptide receptor radionuclide therapy as upfront strategy for the treatment of g1–g2 pancreatic neuroendocrine tumors with unresectable liver metastases. Ann Surg Oncol. 2016;23(Suppl 5):981–9.
Yao KA, Talamonti MS, Nemcek A, Angelos P, Chrisman H, Skarda J, Benson AB, Rao S, Joehl RJ. Indications and results of liver resection and hepatic chemoembolization for metastatic gastrointestinal neuroendocrine tumors. Surgery. 2001;130(4):677–85.
Ito T, Igarashi H, Nakamura K, Sasano H, Okusaka T, Takano K, Komoto I, Tanaka M, Imamura M, Jensen RT, et al. Epidemiological trends of pancreatic and gastrointestinal neuroendocrine tumors in Japan: a nationwide survey analysis. J Gastroenterol. 2015;50(1):58–64.
Wang YH, Lin Y, Xue L, Wang JH, Chen MH, Chen J. Relationship between clinical characteristics and survival of gastroenteropancreatic neuroendocrine neoplasms: a single-institution analysis (1995–2012) in South China. BMC Endocr Disord. 2012;12:30.
Rockall AG, Reznek RH. Imaging of neuroendocrine tumours (CT/MR/US). Best Pract Res Clin Endocrinol Metab. 2007;21(1):43–68.
Maithel SK, Fong Y. Hepatic ablation for neuroendocrine tumor metastases. J Surg Oncol. 2009;100(8):635–8.
Frilling A, Sotiropoulos GC, Li J, Kornasiewicz O, Plockinger U. Multimodal management of neuroendocrine liver metastases. HPB (Oxford). 2010;12(6):361–79.
Iasonos A, Schrag D, Raj GV, et al. How to build and interpret a nomogram for cancer prognosis. J Clin Oncol. 2008;26:1364–70.
Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Med Decis Making. 2006;26(6):565–74.
Chen H, Hardacre JM, Uzar A, Cameron JL, Choti MA. Isolated liver metastases from neuroendocrine tumors: does resection prolong survival? J Am Coll Surg. 1998;187(1):88–93.
Chamberlain RS, Canes D, Brown KT, Saltz L, Jarnagin W, Fong Y, Blumgart LH. Hepatic neuroendocrine metastases: does intervention alter outcomes? J Am Coll Surg. 2000;190(4):432–45.
Elias D, Lasser P, Ducreux M, Duvillard P, Ouellet JF, Dromain C, Schlumberger M, Pocard M, Boige V, Miquel C, et al. Liver resection (and associated extrahepatic resections) for metastatic well-differentiated endocrine tumors: a 15-year single center prospective study. Surgery. 2003;133(4):375–82.
Sarmiento JM, Heywood G, Rubin J, Ilstrup DM, Nagorney DM, Que FG. Surgical treatment of neuroendocrine metastases to the liver: a plea for resection to increase survival. J Am Coll Surg. 2003;197(1):29–37.
Mayo SC, de Jong MC, Bloomston M, Pulitano C, Clary BM, Reddy SK, Clark GT, Celinski SA, Kooby DA, Staley CA, et al. Surgery versus intra-arterial therapy for neuroendocrine liver metastasis: a multicenter international analysis. Ann Surg Oncol. 2011;18(13):3657–65.
Zhang L, Xiang ZL, Zeng ZC, Fan J, Tang ZY, Zhao XM. A microRNA-based prediction model for lymph node metastasis in hepatocellular carcinoma. Oncotarget. 2016;7(3):3587–98.
Ruzzenente A, Bagante F, Bertuzzo F, Aldrighetti L, Ercolani G, Giuliante F, Ferrero A, Torzilli G, Grazi GL, Ratti F, et al. A novel nomogram to predict the prognosis of patients undergoing liver resection for neuroendocrine liver metastasis: an analysis of the italian neuroendocrine liver metastasis database. J Gastrointest Surg. 2017;21(1):41–8.
Spolverato G, Bagante F, Aldrighetti L, Poultsides G, Bauer TW, Field RC, Marques HP, Weiss M, Maithel SK, Pawlik TM. Neuroendocrine liver metastasis: prognostic implications of primary tumor site on patients undergoing curative intent liver surgery. J Gastrointest Surg. 2017;21(12):2039–47.
Dong S, et al. Risk factors of liver metastasis from advanced pancreatic adenocarcinoma: a large multicenter cohort study. World J Surg Oncol. 2017;15(1):120.
Wu SG, Zhang WW, Sun JY, Li FY, Lin Q, He ZY. Patterns of distant metastasis between histological types in esophageal cancer. Front Oncol. 2018;8:302.
Gaitanidis A, Alevizakos M, Tsaroucha A, Tsalikidis C, Pitiakoudis M. Predictive nomograms for synchronous distant metastasis in rectal cancer. J Gastrointest Surg. 2018;22(7):1268–76.
Oweira H, Petrausch U, Helbling D, Schmidt J, Mannhart M, Mehrabi A, Schob O, Giryes A, Decker M, Abdel-Rahman O. Prognostic value of site-specific metastases in pancreatic adenocarcinoma: a Surveillance Epidemiology and End Results database analysis. World J Gastroenterol. 2017;23(10):1872–80.
Shiroyama T, Suzuki H, Tamiya M, Tamiya A, Tanaka A, Okamoto N, Nakahama K, Taniguchi Y, Isa SI, Inoue T, et al. Clinical characteristics of liver metastasis in nivolumab-treated patients with non-small cell lung cancer. Anticancer Res. 2018;38(8):4723–9.
He C, Zhong L, Zhang Y, Cai Z, Lin X. Development and validation of a nomogram to predict liver metastasis in patients with pancreatic ductal adenocarcinoma: a large cohort study. Cancer Manag Res. 2019;11:3981–91.
Balachandran VP, Gonen M, Smith JJ, DeMatteo RP. Nomograms in oncology: more than meets the eye. Lancet Oncol. 2015;16(4):e173–80.
Bochner BH, Kattan MW, Vora KC. Postoperative nomogram predicting risk of recurrence after radical cystectomy for bladder cancer. J Clin Oncol. 2006;24(24):3967–72.
Watzka FM, Meyer F, Staubitz JI, Fottner C, Schad A, Lang H, Musholt TJ. Prognostic assessment of non-functioning neuroendocrine pancreatic neoplasms as a basis for risk-adapted resection strategies. World J Surg. 2020;44(2):594–603.
The authors express their thanks to Xiaoting Wu at The First Affiliated Hospital of Fujian Medical University in the contribution of layout of pictures and forms.
This work was supported bythe National Natural Science Foundation of China [No. 82072074]. Startup Fund for scientific research, Fujian Medical University [No. 2019QH2017]. National Natural Science Foundation of China [No.82001895]; Joint Funds of Scientific and Technological Innovation Program of Fujian Province [No. 2018Y9039].
Ethics approval and consent to participate
As the data used was from SEER dataset (public), Ethics approval and consent to participate could be checked in SEER.
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Pan, M., Yang, Y., Teng, T. et al. Development and validation of a simple-to-use nomogram to predict liver metastasis in patients with pancreatic neuroendocrine neoplasms: a large cohort study. BMC Gastroenterol 21, 101 (2021). https://doi.org/10.1186/s12876-021-01685-w