Effect of educational lecture on the diagnostic accuracy of Japan NBI Expert Team classification for colorectal lesions

Background An educational and training program is required for generalization of Japan NBI Expert Team (JNET) classification. However, there is no detailed report on the learning curve of the diagnostic accuracy of endoscopists using JNET classification. We examined the effect of an educational lecture on beginners and less experienced endoscopists for improving their diagnostic accuracy of colorectal lesions by JNET classification. Methods Seven beginners with no endoscopy experience (NEE group), 7 less experienced endoscopists (LEE group), and 3 highly experienced endoscopists (HEE group) performed diagnosis using JNET classification for randomized NBI images of colorectal lesions from 180 cases (Type 1: 22 cases, Type 2A: 105 cases, Type 2B: 33 cases, and Type 3: 20 cases). Next, the NEE and LEE groups received a lecture on JNET classification, and all 3 groups repeated the diagnostic process. We compared the correct diagnosis rate and interobserver agreement before and after the lecture comprehensively and for each JNET type. Results In the HEE group, the correct diagnosis rate was more than 90% with good interobserver agreements (kappa value: 0.78–0.85). In the NEE and LEE groups, the correct diagnosis rate (NEE: 60.2 → 68.0%, P < 0.01; LEE: 66.4 → 86.7%, P < 0.01), high-confidence correct diagnosis rate (NEE: 19.6 → 37.2%, P < 0.01; LEE: 43.6 → 61.1%, P < 0.01), and interobserver agreement (kappa value, NEE: 0.32 → 0.43; LEE: 0.39 → 0.75) improved after the lecture. In the examination by each JNET type, the specificity and positive predictive value in the NEE and LEE groups generally improved after the lecture. Conclusion After conducting an appropriate lecture, the diagnostic ability using JNET classification was improved in beginners or endoscopists with less experience in NBI magnifying endoscopy.


Introduction
Narrow-band imaging (NBI), introduced in 2006, is one of the commonly applied endoscopic image enhancement methods. It has become a popular tool in general clinical use for the endoscopic diagnosis of colorectal lesions. In Japan, NBI magnifying colonoscopy had been used for the diagnosis of colorectal lesions under various classifications [1][2][3][4]; however, the coexistence of multiple terms for same or similar findings resulted in confusion. In 2009, the NBI International Colorectal Endoscopic (NICE) classification was proposed by the Colon Tumor NBI Interest Group [5]. The NICE classification is simple and widely used in countries where the application of magnifying colonoscopy is not extensive [6,7]. However, with NICE classification, it is difficult to distinguish between low grade dysplasia (LGD) to superficial submucosal invasive (SM-s) carcinoma. In 2014, based on the NICE classification, the Japan NBI Expert Team (JNET) proposed a unified NBI magnifying endoscopy classification, the JNET classification ( Fig. 1) [8]. The JNET classification consists of Type 1, 2A, 2B and 3 based on vessel/ surface pattern. Each type refers to the most likely histology of the colorectal lesions. Type 1 is characterized by an invisible vessel pattern and a surface pattern with dark and white spots or similar to the surrounding normal mucosa. Type 1 includes hyperplastic polyp (HP) and sessile serrated polyp (SSP). Type 2A is characterized by a regular vessel pattern, such as regular caliber or distribution, and a regular surface pattern. Type 2A includes LGD. Type 2B is characterized by an irregular vessel pattern, such as a variable caliber or irregular distribution, and an irregular to obscure surface pattern. Type 2B includes high grade dysplasia (HGD) and SM-s carcinoma. Type 3 is characterized by loose vessel areas and/or interruption of thick vessels and/or an amorphous surface pattern. Type 3 includes deep submucosal invasive (SM-d) carcinoma [8]. In the validation studies conducted thus far, Types 1, 2A, and 3 are highly reliable diagnostic indicators with high specificity for each expected histology [9][10][11][12][13][14]. Follow-up, endoscopic treatment, and surgical resection are recommended for Type 1, Type 2A, and Type 3 lesions, respectively. However, Type 2B includes a wide variety of lesions ranging from HGD to SM-d carcinoma; therefore, pit pattern diagnosis with chromoendoscopy is necessary for an accurate diagnosis. In previous papers, JNET classification was reported to be useful even for colonoscopists with little experience in NBI magnifying endoscopy [15][16][17][18]. However, it was pointed out that appropriate training is necessary to improve the diagnostic accuracy using JNET classification [16].
Although it is acknowledged that education on NBI diagnosis is useful in the differentiation of neoplastic and non-neoplastic lesions [19][20][21][22][23][24][25], there is no detailed report on the learning curve of the diagnostic accuracy using JNET classification. In this study, we examined the Microvessels are often distributed in a punctate pattern and well-ordered reticular or spiral vessels may not be observed in depressed lesions. *3. Deep submucosal invasive cancer may be included. *4. Low grade intramucosal neoplasia: low grade dysplasia. *5. High grade intramucosal neoplasia: high grade dysplasia diagnostic accuracy of JNET classification among diagnosticians with various experience levels of NBI magnifying endoscopy before and after an educational lecture on JNET classification.

Image for NBI diagnosis
The images were captured with NBI using high-definition colonoscopes (CF-H260AZI; Olympus, Tokyo, Japan) and a standard videoendoscopic system (EVIS LUCERA ELITE system; Olympus, Tokyo, Japan  [9]. Therefore, in this study, the correct answer for JNET classification was considered as the gold standard. The correct diagnosis using JNET classification was determined by an expert endoscopist who is a member of the JNET working group (S.T.) with an excellent intra-observer agreement (kappa value: 1.0). The diagnostic images before and after the lecture on JNET classification were randomly sorted. In addition, the correct answer was withheld from the evaluators until the end of the study.

Image evaluator categories
The evaluators were divided into 3 groups: 7 beginners (4 medical students and 3 initial residents) without the experience of endoscopic diagnosis (the no endoscopy experience group, NEE group), 7 endoscopists with an experience of < 5 years in endoscopic diagnosis using magnifying NBI (the less experienced endoscopist group, LEE group), and 3 endoscopy specialists with an experience of > 5 years in endoscopic diagnosis using magnifying NBI (the highly experienced endoscopist group, HEE group) (Fig. 2). LGD

Educational lecture on JNET classification
Prior to the first NBI diagnosis test, we explained the outline concept of JNET classification using Fig. 1 to the NEE/LEE groups for 5 min. Next, members of the NEE/ LEE/HEE groups took the first NBI diagnosis test. After the first test, an educational lecture was imparted to the NEE/LEE groups. The lecture was based on the definition of JNET classification with 12 concrete images representing each type of lesion and reflecting the images presented in the annotations of Fig. 1 (Type 1, Type 2A, Type 2B, and Type3: 3, 3, 4, and 2 images, respectively). In addition, we presented 6 images for distinguishing Type 1 and 2A, Type 2A and 2B, and Type 2B and 3. After the participants diagnosed the images, we presented the correct answer and explained the points of the diagnosis. The lecturer was an endoscopist familiar with NBI and JNET classification with the same experience level as that of the HEE group. The lecture was conducted using a predetermined phrase for 15 min while presenting the images on electronic slides. Five hours after the lecture, members of the NEE/LEE/HEE groups took the second NBI diagnosis test. Both tests, before and after the lecture, and the lecture were given on the same day.

Statistical analysis
The correct diagnosis rate and high-confidence correct diagnosis rate were calculated for the total tests before and after the lecture in the NEE, LEE, and HEE groups. In addition, the accuracy rate, sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) for the tests of each JNET type were calculated. The correct diagnosis rate of the total tests and the accuracy rate, sensitivity, and specificity of the tests of each JNET type were compared before and after the lecture using the McNemar test. The PPV and NPV of the tests of each JNET type were compared before and after the lecture using the Pearson's chi-squared test. P < 0.05 was considered statistically significant. The statistical analyses were performed using JMP version 14 (SAS Institute Inc., Cary, NC, USA). The interobserver agreement in each group was assessed using kappa values as follows: < 0.4: poor agreement; 0.41-0.60: fair agreement; 0.61-0.80: good agreement; > 0.8: excellent agreement. Kappa statistics were calculated using the Fleiss method of R version 3.6.0 (R Core Team (2019). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL https :// www.R-proje ct.org/).
The sensitivity, specificity, PPV, NPV, and accuracy rate in the HEE group were all 100%.

Diagnostic performance of JNET type 2A
The sensitivity, specificity, PPV, NPV, and accuracy rate in the NEE group were 57.4%, 91.6%, 90.6%, 60.6%, and 71.7% before the lecture and 63.9%, 94.5%, 94.2%, 65.2%, and 76.7% after the lecture, respectively ( Table 4). The sensitivity, PPV, and accuracy rate significantly increased after the lecture. The interobserver agreements in the NEE group were poor before and after the lecture (0.126 → 0.112).  The sensitivity, specificity, PPV, NPV, and accuracy rate in the LEE group were 63.7%, 89.3%, 89.3%, 63.7%, and 74.4% before the lecture and 86.0%, 98.1%, 98.4%, 83.3%, and 91.0% after the lecture, respectively. All parameters in the LEE group significantly increased after the lecture. The interobserver agreements in the LEE group were poor before and after the lecture (0.13 → 0.357).

Diagnostic performance of JNET type 2B
The sensitivity, specificity, PPV, NPV, and accuracy rate in the NEE group were 59.3%, 68.5%, 29.7%, 88.2%, and 66.8% before the lecture and 63.6%, 73.8%, 35.3%, 90.0%, and 71.9% after the lecture, respectively ( Table 5). The specificity and accuracy rate significantly increased after the lecture. The interobserver agreements in the NEE group were poor before and after the lecture (0.0589 → 0.0705).
The sensitivity, specificity, PPV, NPV, and accuracy rate in the LEE group were 68.8%, 69.5%, 33.6%, 90.9%, and 69.4% before the lecture and 86.6%, 87.7%, 61.2%, 96.7%, and 87.5% after the lecture, respectively. All parameters in the LEE group significantly increased after the lecture. The interobserver agreements in the LEE group were poor before and after the lecture (0.00736 → 0.144).

Discussion
The clinical usefulness of JNET classification has been examined, and the description of Types 1, 2A, and 3 is considered useful for not only experts but also inexperienced evaluators in diagnosing the histological type or invasion depth of a colorectal tumor [9][10][11][12][13][14][15][16][17][18]. However, the educational effects and learning curves remain unclear. In this study, the pre-lecture diagnostic performance for JNET classification in the NEE and LEE groups was similar to that of previous studies [15,16,18]. Even in the NEE group, it was possible to make correct diagnosis of typical cases to some extent before the lecture. Therefore, JNET classification is useful for beginners. In addition, a short educational lecture on JNET classification resulted in an improvement in diagnostic ability and interobserver agreement in the NEE and LEE groups. In the HEE group, high sensitivity and specificity were obtained for all types. In addition to having a large number of NBI diagnoses, members in the HEE group have discussed cases using the JNET classification during a daily endoscopic procedure or in case conferences at our institution for at least 3 years. This practice may result in a good diagnostic ability using the JNET classification.
With regard to Type 1 cases, the specificity in the NEE and LEE groups was as good as 95% or more before and after the lecture. Therefore, it is considered there is little risk that cases requiring resection will not be treated. On the other hand, the sensitivity before the lecture was low in the NEE and LEE groups. Comparing images that were misdiagnosed by more than half of each group with other images (Additional file 1: Fig. 3), most of the misdiagnosed images presented visible vessel patterns. It was considered that the misdiagnosis was caused by the misunderstanding of the following wordings in the JNET classification chart: if the vessel pattern is visible, the honeycomb pattern and caliber in the lesion is similar to that of the surrounding normal mucosa.
The specificity and PPV were good for Type 2A cases in the NEE and LEE groups. On the other hand, the sensitivity before the lecture was low in both groups. By examining the misdiagnosed cases, it was found that most misdiagnoses were Type 2B. As shown in the misdiagnosis example in Additional file 1: Fig. 4, although the surface pattern is regular, the cases wherein vessels in the intervening part have various appearances or the vessel pattern is partially obscured because of the angle of incidence were likely to be misdiagnosed as Type 2B.
A majority of cases misdiagnosed as Type 2B were Type 3, whereas a majority of cases misdiagnosed as Type 3 were Type 2B. A lesion was diagnosed as Type 3 if at least one of the criteria terms in the Type 3 column in the chart was met; however, judgment may be difficult in terms of the boundary analysis between Type 2B and 3 as in the example shown in Additional file 1: Fig. 5. On the other hand, the specificity in type 3 was 90% or more in both the NEE and LEE groups before and after the lecture, which was considered to contribute to the prevention of unnecessary surgery. We suggest that development of widely available educational tools that take care of the pitfalls clarified in this study will contribute to the improvement of diagnostic ability of endoscopists using JNET classification in facilities without specialists. Recently, endocytoscopy (Olympus, Tokyo, Japan) was developed and its high diagnostic ability has been reported [26,27]. However, the special equipment, the knowledge and experience of endoscopists are required for the procedures and diagnoses for colorectal lesions. In addition, there have been several studies on computer-aided diagnoses that predict pathological diagnosis of colorectal lesions by NBI magnifying endoscopy [28][29][30][31][32]; however, sufficient accuracy of differentiation for the depth of tumor invasion has not been achieved and further research is needed. Therefore, it is considered that improvement in the diagnostic ability of endoscopists by imparting educational lectures for generalized interpretation of JNET classification will play an important role for the time being.
This study has some limitations. First, it is a singlecenter study, and the members of the HEE group may have been involved in obtaining and diagnosing images that were used for the NBI diagnosis tests, which may have affected the diagnostic ability. Second, as the cases were consecutive, there were differences in the number of images of each type. However, it can be considered that the data reflect actual clinical practice. Third, since all experiments were performed in one day, we have not evaluated the diagnostic ability days later after the lecture. However, we did not give the correct answer for the first test to all diagnosticians, and there was a fivehour interval between the lecture and the second NBI diagnosis test.
In conclusion, a short lecture was found to be useful for improving the diagnostic accuracy of endoscopists using JNET classification even in inexperienced or less experienced diagnosticians. Further improvement of the lecture tool is expected and may contribute to the improvement of diagnostic ability and generalization of JNET classification all over the world.