Joon-Il Choi, MD, Seung Eun Jung, MD, Pyo Nyun Kim, MD, Sang Hoon Cha, MD, Jae Kwan Jun, MD, Hoo-Yeon Lee, MD, Eun-Cheol Park, MD

Transcription

1 ORIGINAL RESEARCH Quality Assurance in Ultrasound Screening for Hepatocellular Carcinoma Using a Standardized Phantom and Standard Clinical Images A 3-Year National Investigation in Korea Joon-Il Choi, MD, Seung Eun Jung, MD, Pyo Nyun Kim, MD, Sang Hoon Cha, MD, Jae Kwan Jun, MD, Hoo-Yeon Lee, MD, Eun-Cheol Park, MD Article includes CME test Received July 9, 2013, from the Department of Radiology, Seoul St Mary s Hospital, School of Medicine, Catholic University of Korea, Seoul, Korea (J.-I.C., S.E.J.); Department of Radiology and Research Institute of Radiology, University of Ulsan College of Medicine, Seoul, Korea (P.N.K.); Department of Radiology, Ansan Hospital, College of Medicine, Korea University, Ansan-si, Korea (S.H.C.); National Cancer Control Institute, National Cancer Center, Goyang-si, Korea (J.K.J.); Department of Social Medicine, College of Medicine, Dankook University, Cheonan-si, Korea (H.-Y.L.); and Department of Preventive Medicine and Institute of Health Services Research, Yonsei University College of Medicine, Seoul, Korea (E.-C.P.). Revision requested August 21, Revised manuscript accepted for publication October 16, We thank Seung Whan Lee for assistance with manuscript preparation. This study was supported by a grant from the National Cancer Control Institute of the National Cancer Center of Korea and by the Ministry of Health, Welfare, and Family Affairs of Korea. This study was performed with the help of the Korean Society of Radiology and the Korean Institute for Accreditation of Medical Imaging. Address correspondence to Seung Eun Jung, MD, Department of Radiology, Seoul St Mary s Hospital, College of Medicine, Catholic University of Korea, 222 Banpo-daero, Seocho-gu, Seoul , Korea. sejung@catholic.ac.kr Abbreviations QA, quality assurance; US, ultrasound doi: /ultra Objectives The purpose of this study was to investigate the quality of ultrasound (US) imaging for hepatocellular carcinoma screening. Methods The investigation was performed at all medical institutes participating in the National Cancer Screening Program in Korea. For assessment of personnel, we inquired who was performing the US screenings. For phantom image evaluation, the dead zone, vertical and horizontal measurements, axial and lateral resolution, sensitivity, and gray scale/dynamic range were evaluated. For clinical image evaluation, US images of patients were evaluated in terms of the standard images, technical information, overall image quality, appropriateness of depth, foci, annotations, and the presence of any artifacts. Results Failure rates for phantom and clinical image evaluations at general hospitals, smaller hospitals, and private clinics were 20.9%, 24.5%, 24.1% and 5.5%, and 14.8% and 9.5%, respectively. No statistically significant difference was observed in the failure rates for the phantom images among groups of different years of manufacture. For the clinical image evaluation, the results of radiologists were significantly better than those of other professional groups (P =.0001 and.0004 versus nonradiology physicians and nonphysicians, respectively). The failure rate was also higher when the storage format was analog versus digital (P <.001). Conclusions Approximately 20% of US scanners failed the phantom image evaluation. The year of scanner manufacture was not significantly associated with the results of the phantom image evaluation. The quality of the clinical images obtained by radiologists was the best. Key Words hepatocellular carcinoma; phantom; public policy; quality assurance; screening; ultrasound A s the role of imaging in the diagnosis and treatment of patients grows, the importance of quality assurance (QA) in diagnostic imaging receives more emphasis. In the United States, the Mammography Quality Standards Act was introduced for QA in mammography, and accreditation standards for computed tomography and magnetic resonance imaging programs are broadly accepted. Ultrasound (US) imaging is currently one of the most frequently used imaging tools alongside computed tomography 2014 by the American Institute of Ultrasound in Medicine J Ultrasound Med 2014; 33:

2 and magnetic resonance imaging, and QA standards for it are being recommended by many major international scientific bodies, such as the American Association of Physicists in Medicine, American Institute of Ultrasound in Medicine, and American College of Radiology. 1 3 However, a standardized set of criteria for QA of US has not yet been formally established. One of the major reasons is the fact that US examinations are performed in diverse medical settings for varying purposes. Also, there is no legal regulation for US devices because they do not generate ionizing radiation. Therefore, most scientific bodies suggest only vague, nonspecific standards for QA of US. 3 Therefore, for QA of US, concentrating on testing for a specific purpose will produce more meaningful and reasonable data rather than attempting to generate a set of all-encompassing standards. We decided to investigate the quality of screening liver US imaging, which is the most broadly used method in cancer surveillance. The goals of the Korean government also influenced this decision. In Korea, US screenings of the liver for those at risk of hepatocellular carcinoma (ie, carriers of hepatitis B and hepatitis C viruses and patients with liver cirrhosis of any cause) are included in the National Cancer Screening Program. This program is run by the National Cancer Control Institute and is funded through taxation. The government wanted to evaluate the quality of these screenings; thus, we designed a 3-year investigation conducted from 2008 to The purpose of this study was to evaluate the quality of US screening for hepatocellular carcinoma and to analyze the factors affecting it. Furthermore, we hoped to establish QA standards for US examinations based on these results. Materials and Methods Participating Medical Institutes and Data Acquisition This study was a national survey sponsored by the Korean government and was approved by the Institutional Review Board of the National Cancer Center of Korea. Written informed consent was waived because of the retrospective nature of the study. The investigation was performed at all of the medical institutes participating in the National Cancer Screening Program for hepatocellular carcinoma in Korea from 2008 to General hospitals were investigated in 2008, smaller hospitals other than general hospitals in 2009, and private clinics in In total, 271 general hospitals, 467 smaller hospitals, and 1184 private clinics were evaluated. The imaging QA evaluation was divided into 3 categories: personnel evaluation, phantom image evaluation, and clinical image evaluation. We evaluated personnel by reviewing inquires returned by the medical institutes as to who performed US scanning. For the phantom image evaluation, our research assistants, who were researchers at the Korean Institute for Accreditation of Medical Imaging and learned how to scan the standard phantom, visited every participating institute and acquired the phantom images. For the clinical image evaluation, we analyzed recently scanned US images of patients, which were submitted by the medical institutes themselves. All phantom and clinical US images were reviewed by 12 experienced abdominal radiologists with more than 5 years of experience in US examinations. Before their assessment, all reviewers were trained for 2 hours on the phantom and clinical image evaluations for QA of US. They were blinded to hospital groups, machine ages, and personnel who performed US scans. All phantom and clinical images were reviewed by a single reviewer. However, in cases of failure, confirmation by a second or third reviewer was mandatory. Evaluation of Personnel Performing US Examinations For the personnel evaluation, we investigated who (eg, radiologists, physicians other than radiologists, or nonphysicians) performed the screening US examinations. This evaluation was done by reviewing surveys submitted by the medical institutes. We analyzed the results according to facility groups (ie, general hospitals, smaller hospitals, and private clinics). Phantom Image Evaluation The phantom image evaluation was done to assess the performance and quality of US hardware and software. The criteria evaluated consisted of conditions and parameters of contrast, spatial resolution, accuracy of measurements, and penetration of the US beam. For our study, we used an ATS-539 multipurpose phantom (ATS Laboratories, Inc, Bridgeport, CT), 4 which is the standard for US phantom images recommended by the Korean Society of Radiology and the Korean Society of Ultrasound in Medicine since The phantom is a rubber-based tissue mimic that matches the acoustic and anatomic properties of human tissue. Six research assistants transported the standard phantoms to the medical institutes and obtained phantom images from each US machine with a MHz curved array probe and software settings for abdominal US imaging. Image settings were optimized by the physician on site and our research assistants, and images were acquired by research assistants because they had better knowledge of phantom images. Soft copy images were collected when medical institutes could support and submit them; otherwise, hard copy images were acquired. 986 J Ultrasound Med 2014; 33:

3 The test items for the phantom image evaluation included the dead zone, vertical and horizontal measurements, axial and lateral resolution, sensitivity, and gray scale/dynamic range. Failure on any of the 6 test items rendered the imaging as failed. The dead zone is the distance from the front face of the transducer to the first identifiable echo at the phantom or patient interface, and no clinical data can be collected in the dead zone. The vertical and horizontal distance measurements were obtained both parallel and perpendicular to the axis of the sound beam. We measured 10.0 cm along the axis of the sound beam for vertical measurement and 8.0 cm perpendicular to the sound beam for horizontal measurement, and the resulting measurements were compared to the actual distance. Axial and lateral resolution is the test for spatial resolution, and 11 line targets were present in the phantom. We counted the number of line targets that were identifiable separately. Sensitivity refers to the penetration depth of the US beam and was recorded as the deepest target structure that was displayed on the US image. The gray scale/dynamic range assessed the contrast of the images. Six cylindrical targets with varying degrees of brightness were visible on the US images. We counted the number of targets that appeared as discrete round structures. The images for each phantom image evaluation are displayed in Figure 1. The cutoff values for each test item are summarized in Table 1. Detailed explanations of the test items and the cutoff values of the items in the phantom image evaluation have been given in previous studies. 5,6 We analyzed the results of the phantom image evaluation according to the facility groups and identified the common causes of failure. Also, considering that the phantom image evaluation is the test of the hardware and software of US machines, we categorized US machines into 3 groups by the year of manufacture: within the last 5 years, 5 to 10 years ago, and more than 10 years ago. The results of each group were compared. Clinical Image Evaluation The clinical image evaluation assessed the diagnostic quality of the patient images. This evaluation tests the appropriateness of the US protocol and the knowledge and skill of the personnel performing the scan. Because this study was a survey, we asked medical institutes to submit their best clinical images instead of designating clinical images of a specific patient. The clinical images could be submitted as either hard or soft copies. Items for the clinical image evaluation included the number of good images (16 points), identification (6 points), information from equipment (30 points), standard images (40 points), and artifacts (8 points). 7 The score for the number of good images was perfect when there were 8 good images, as we recommended 8 standard images for US scanning. The importance of standard images was emphasized because fulfillment of all 8 images can guarantee scanning of the whole liver, limiting the chances of missing an area. The standard images were from the protocol established by the Korean Society of Radiology and the Korean Society of Ultrasound in Medicine. The 8 selected standard images were as follows: (1) transverse and (2) longitudinal scans of the left hemiliver; (3) subcostal and (4) intercostal scans of the right hemiliver; (5) transverse plane of the right and left portal veins; (6) hepatic veins at the right hepatic dome; (7) longitudinal scan of the gallbladder; and (8) longitudinal scan of the extrahepatic duct. 7 Also, technical information, including the name, sex, and age of the patient, name of the medical center, name of the examiner who performed the scan, date of the examination, identification number of the patient, overall image quality, appropriateness of depth, location of foci, annotations, the presence of any artifacts were also collected. A perfect score for the clinical image evaluation was 100 points, and the test was passed when the score was higher than 60 points. Test items and scores are summarized in Table 2. We analyzed the results of the clinical image evaluation according to facility groups. Also, we analyzed the results of the clinical image evaluation on the basis of the data storage format (digital or analog) and personnel who performed the US examinations. Statistical Analyses To compare the failure rates among different facility groups, different years of machine manufacture, and different personnel groups, the Fisher exact test was performed. To compare the average scores and numbers of good images in the different storage formats in the clinical image evaluation, an unpaired t test was performed. All statistical analyses were performed with MedCalc version 9.2 software for Windows (MedCalc Software, Mariakerke, Belgium), and all the charts were created with Microsoft Excel 2010 software (Microsoft Corporation, Redmond, WA). Differences were considered significant at P <.05. Results Evaluation of Personnel Performing US Examinations In all 271 general hospitals, US examinations (scanning and interpretation) were performed only by radiologists. J Ultrasound Med 2014; 33:

4 Figure 1. Images from the phantom image evaluation. The ATS-539 multipurpose phantom can be used for testing the performance of US scanners. A, Dead zone. Nine line targets (arrowheads) are located 2 to 10 mm below the scan surface. The distance between line targets is 1 mm. B, Vertical measurement. A 10-cm distance along the US beam axis is measured. In this US scanner, the vertical measurement is cm. C, Horizontal measurement. An 8-cm distance perpendicular to the US beam axis is measured. In this US scanner, the horizontal measurement is 8.53 cm. D, Axial and lateral resolution. Eleven line targets with a curved array are visible separately. The distance between line targets range from 1 mm in the central area to 5 mm in the peripheral area. The central part of the image is zoomed in the right inset. (opposite page) E, Sensitivity, which is a test for US beam penetration. Eight 8-mm anechoic round structures are noted in the vertical direction (arrows). The distance between the structures is 2 cm; ie, if all 8 structures are visible, the minimum penetration of the US beam is greater than 16 cm. F, Gray scale/dynamic range. There are 6 cylindrical structures with different contrast levels relative to the background (+15, +6, +3, 3, 6, and 15 db). Four or more structures should be visible clearly over 180 to pass the test. A C B D 988 J Ultrasound Med 2014; 33:

5 E In smaller hospitals, however, radiologists constituted 33.5% of US examination performers. In private clinics, only 17.1% of US examinations were performed by radiologists. However, there were many physicians whose specialty was unknown in the survey; therefore, the true proportion of radiologists is not clear. In private clinics, 4.2% of US scanning was performed by nonphysicians, even though US scanning by nonphysicians is not permitted for the patients of the National Cancer Screening in Korea. The results are summarized in Figure 2. Phantom Image Evaluation The phantom image evaluation was performed for 1140 US machines. Table 3 summarizes the results of the phantom and clinical image evaluations by facility groups. There was no significant difference in the failure rates among the different groups of facilities for the phantom image evaluation. The failure rate for each test item in the phantom image evaluation is summarized in Table 4; the most common cause of failure was gray scale/dynamic range, consisting of 42.6% (Figure 3). The number of machines and failure rates for phantom images across 3 different periods of manufacture (last 5 years, 5 10 years ago, and >10 years ago) were 232, 144, and 69 and 18.8%, 23.0%, and 17.9%, respectively. There was no significant difference among the groups based on the year of manufacture (Figure 4). Figure 5 shows images from failed cases. F Clinical Image Evaluation The clinical image evaluation was performed for 2055 US machines. When comparing the results of the clinical image evaluation according to facility groups, general hospitals had the lowest failure rate, at 5.5% (Table 3). For all items, average scores of failed cases were poorer than those of passed cases. The differences in scores for the number of good images and standard images were 8.5 and 19.4 points, respectively, and consisted of 27.4% and 62.6% of the total differences in scores and the 2 most common causes of poorer scores (Table 5). When analyzing the results for standard images, the rate of successful acquisition of each standard image was significantly higher for passed cases in all 8 standard images (Figure 6). For analog and digital formats, the average numbers of good images were 13.8 and 15.3, respectively (P <.0001), and the failure rates were 14.1% and 3.3% (P <.0001). The failure rates for the clinical image evaluation of radiologists, nonradiology physicians, and nonphysicians were 6.4%, 20.8%, and 21.4%, respectively. The results for the radiologists were significantly better than those for the other professional groups (P =.0001 and.0004 versus J Ultrasound Med 2014; 33:

6 Table 1. Cutoff Values for Test Items in the Phantom Image Evaluation Test Item Dead zone Vertical measurement Horizontal measurement Axial and lateral resolution Sensitivity Gray scale/dynamic range Cutoff Value <2 mm for the dead zone Within 5% discrepancy (10.0 ± 0.5 cm) Within 7.5% discrepancy (8.0 ± 0.6 cm) All 11 identifiable line targets >14 cm >4 cylindrical structures identifiable Table 2. Scores for Test Items in the Clinical Image Evaluation Test Item Subitem Score 1. No. of good images 1. No. of qualified images 2 or 0/image (total 16) 2. Identification 1. Patient s name 2. Age/sex 3. Registration number 4. Examiner s name 5. Date 6. Name of medical institution 1 or 0/item (total 6) 3. Information from equipment 1. Similar and appropriate brightness 2. Proper position of focal zone 3. Control of depth 4. Display of scaler 5. Display of direction or body mark 6. Kind of transducer or frequency 5, 3, or 0/item (total 30) 4. Standard images 1. Longitudinal scan of left hemiliver 2. Transverse scan of left hemiliver 3. Transverse plane of right and left portal veins 4. Hepatic veins at hepatic dome 5. Subcostal scan of right hemiliver 6. Intercostal scan of right hemiliver 7. Longitudinal scan of gallbladder 8. Long-axis scan of extrahepatic duct 5, 3, or 0/item (total 40) 5. Artifacts 1. Motion artifact 2. Mechanical artifact from damage of elements 4, 2, or 0/item (total 8) Points were assigned as follows: 1. No. of good images: A. Number of images easy on the eyes among examined images; B. 2 points per good image up to 16 points (8 images). 2. Identification: A. 1 point per item. 3. Information from equipment: A. 5 points in cases with adjustment of more than half of images; B. 3 points in cases with adjustment of less than half of images. 4. Standard images: A. 5 points in cases with complete visualization of each anatomic structure; B. 3 points in cases with partial visualization of each anatomic structure. 5. Artifacts: A. Motion artifact: a. 4 points in cases with no artifact; b. 2 points in cases with depiction in less than half of images; c. No points in cases with depiction in more than half of images. B. Damage of elements: a. 4 points in cases with no artifact; b. 2 points when an artifact was at the periphery of the transducer; c. No points in cases with a central artifact. 990 J Ultrasound Med 2014; 33:

7 nonradiology physicians and nonphysicians, respectively). The failure rate for nonradiology physicians was not significantly different from that for nonphysicians (P >.99; Figure 7). Figure 8 shows a failed case from the clinical image evaluation. Discussion Ultrasound examinations are commonly recommended in the United States, Europe, Japan, and Korea as screening tests for patients at risk of hepatocellular carcinoma The American Association for the Study of Liver Disease has emphasized the need for QA of US examinations for hepatocellular carcinoma screening. 8 In Korea, regulations for QA of imaging modalities were established in However, only computed tomography, magnetic resonance imaging, and mammography had been assessed. To the best of our knowledge, this study is the first of its kind. We hope that its meaningful results will contribute to the establishment of a QA system for US. In the phantom image evaluation, there were no significant differences in failure rates among the different facility groups. More than 20% of the US scanners constantly failed regardless of the facility group. This finding reflects a need for improvement in the quality of US machines themselves. We also evaluated the effect of the year of manufacture on the failure rate in the phantom image evaluation. This evaluation tests the hardware and software, and hardware performance varies across time due to inevitable deterioration with use, especially if machines are being used without proper maintenance. Also, improved image quality Figure 2. Proportion of personnel who performed US examinations (scanning and interpretation). A, Smaller hospitals other than general hospitals. B, Private clinics. A B Table 3. Results of Phantom and Clinical Image Evaluations According to Facility Group Smaller Hospitals Evaluation Parameter General Hospitals Other Than General Hospitals Private Clinics Phantom image All machines, n Failed machines, n Failure rate, % P a P b.9335 Clinical image All machines, n Failed machines, n Failure rate (%) P a P b.0024 a Comparison with general hospitals. b Comparison with smaller hospitals other than general hospitals. J Ultrasound Med 2014; 33:

8 is expected in newer machines due to updated software. Contrary to our expectations, however, there was no significant difference in failure rates among the groups based on the time of manufacture. This finding might have been due to a selection bias. Older machines with poor performance may have already been discarded, and remaining older machines might have been ones in with good condition with regular maintenance. The small number of machines that were older than 10 years somewhat supports this notion. Also, the year of manufacture of 695 of 1140 Table 4. Failed Cases and Failure Rates for Test Items in the Phantom Image Evaluation Test Item Failed Cases, n Failure Rate, % Dead zone Vertical measurement Horizontal measurement Axial and lateral resolution Sensitivity Gray scale/dynamic range US machines was not known, which could have been another cause of a selection bias. However, regular maintenance of equipment can extend the longevity of hardware. The major causes of failure in the clinical image evaluations were the number of good images and standard images. Rates for successful acquisition of standard images were significantly poorer for all 8 standard images in failed cases. Also, results of the clinical image evaluations submitted via the analog route were poorer than those provided in digital format, and these results might have been caused by the small number of images submitted. If the number of images is small, not only are penalty points taken, but it also becomes difficult to gain enough points for standard image items, thus resulting in failure. Also, digital images can be further manipulated by the reviewer, which might have been another cause of the difference. Since acquisition of all standard images is a prerequisite of whole-liver scanning, it is crucial to train examiners to acquire adequate standard images. Especially, the long-axis scan of the extrahepatic duct was acquired less than 50% of the time even in passed cases, indicating that this technique clearly needs further training. Figure 3. Proportion of causes of failure for the phantom image evaluation. Figure 4. Comparison of failure rates for the phantom image evaluation by the different years of machine manufacture. Table 5. Average Scores of Test Items for Passed and Failed Cases in the Clinical Image Evaluation Difference Between Test Item Total Passed Failed Passed/Failed P a No. of good images <.01 Identification <.01 Information from equipment <.01 Standard images <.01 Artifacts <.01 Overall <.01 a Comparison of passed and failed cases. 992 J Ultrasound Med 2014; 33:

9 For the clinical imaging evaluation, the best results were achieved when radiologists performed the scanning. This finding was likely due to the superior knowledge and skills of radiologists. The American Association for the Study of Liver Disease also stresses the importance of having welltrained personnel perform screening US examinations. Although difficult to apply to every country, it is essential that radiologists, experienced physicians, or at least welltrained nonphysicians perform US examinations for hepatocellular carcinoma screening. This study had several limitations. First, the standards for qualification were set arbitrarily. There are no known cutoff values for the phantom image evaluation, from the phantom manufacturer or otherwise. Therefore, we arbitrarily set the standard cutoff values for the phantom image evaluation based on a few previous studies. 5 7 Also, scoring of the clinical image evaluation was set arbitrarily. There are no specific cutoff values recommended by major scientific bodies; therefore, we had to set the values on the basis of our experience and opinions of some experts. However, this study was a survey and not a legal test, and these data will be helpful to establish a basis for standards and cutoff values for the clinical image evaluation. Second, we adopted subjective visual methods for the phantom image evaluation. Both subjective visual methods and objective computer-based approaches can be used to Figure 5. Failed cases from the phantom image evaluation. A, Axial and lateral resolution failure. Central line targets (arrow) show conglomeration, and no discrete separation is visible. B, Gray scale/dynamic range failure. Only 3 cylindrical structures are shown as round structures (arrowheads). A Figure 6. Comparison of the rates of adequately acquired images for the standard images in the clinical image evaluation for passed (black bars) and failed (gray bars) examinations. Figure 7. Comparison of failure rates in the clinical image evaluation according to the personnel who performed US scanning. B J Ultrasound Med 2014; 33:

10 perform a phantom image evaluation. In general, objective computer-based methods are known to be more accurate than subjective methods However, considering the large number of US machines included in this study and Figure 8. Failed case in the clinical image evaluation. Only 3 images were acquired from this patient. Thermal paper was submitted for this case. A, Longitudinal scan of the left hemiliver. The position of the focal zone (arrow) is too deep for the liver. B, Scan of a kidney. This scan is not included in the standard images. C, Scan of hepatic veins at the right hepatic dome. The brightness is not appropriate, and the image is too dark. Overall, this case scored 28 points and failed the clinical image evaluation. A B the variety of image storage formats, including thermal paper, computer-based analyses could not be done. Third, we asked medical institutes to submit their best images for the clinical image evaluation; therefore, the results of this study are not representative of everyday practice, and this factor might have been the cause of the lower failure rates for the clinical image evaluation than the phantom image evaluation. Fourth, there is insufficient evidence that the results of this QA test are indeed indicative of the performance of hepatocellular carcinoma screening by US imaging. Comparing the hepatocellular carcinoma detection rate for passed machines with that for failed ones may make the results more meaningful and reliable. However, that kind of the study could not be performed because we did not have a standardized patient. However, if the phantom images are poor, the resolution and contrast of the patient images are also expected to be poor. Also, a sufficient evaluation of the liver could not be done if the scanning protocol was violated. Nevertheless, the results of these QA tests can be correlated with the quality of US examinations for hepatocellular carcinoma screening. In conclusion, we have performed a nationwide investigation of the imaging quality of US screening for patients at risk of hepatocellular carcinoma. Failure rates for the phantom image evaluation were similar in all facility categories; however, those for the clinical image evaluation were highest in the smaller hospitals. There was no significant difference in the failure rates among machines of different years of manufacture. Radiologists had the best quality in the clinical image evaluation for screening US examinations of hepatocellular carcinoma compared to nonradiology physicians and nonphysicians; thus, adequate education for other US staff may be mandatory. References C 1. Goodsitt MM, Carson PL, Witt S, Hykes DL, Kofler JM Jr. Real-time B- mode ultrasound quality control test procedures: report of AAPM Ultrasound Task Group No. 1. Med Phys 1998; 25: American Institute of Ultrasound in Medicine. Quality Assurance Manual for Gray-Scale Ultrasound Scanners: Stage 2. Laurel, MD: American Institute of Ultrasound in Medicine; American College of Radiology. ACR technical standard for diagnostic medical physics performance monitoring of real time ultrasound equipment. American College of Radiology website. ACR/Documents/PGTS/standards/MonitorUSEquipment.pdf. Accessed October 26, ATS Laboratories Inc. Clinical Quality Assurance Phantoms: Multipurpose Phantom Model 539. Bridgeport, CT: ATS Laboratories Inc; J Ultrasound Med 2014; 33:

11 5. Choi JI, Kim PN, Jeong WK, et al. Establishing cutoff values for a quality assurance test using an ultrasound phantom in screening ultrasound examinations for hepatocellular carcinoma: an initial report of a nationwide survey in Korea. J Ultrasound Med 2011; 30: Kim PN, Lim JW, Kim HC, et al. Quality assessment of ultrasonographic equipment using an ATS-539 multipurpose phantom. J Korean Radiol Soc 2008; 58: Kim PN, Kim KW, Byun JH. Quality assessment of hepatic ultrasound images examined after a medical check-up. J Korean Soc Ultrasound Med 209; 28: Bruix J, Sherman M. Management of hepatocellular carcinoma. Hepatology 2005; 42: Bruix J, Sherman M. Management of hepatocellular carcinoma: an update. Hepatology 2011; 53: European Association for the Study of the Liver; European Organisation for Research and Treatment of Cancer. EASL-EORTC clinical practice guidelines: management of hepatocellular carcinoma. J Hepatol 2012; 56: Kudo M, Izumi N, Kokudo N, et al. Management of hepatocellular carcinoma in Japan: consensus-based clinical practice guidelines proposed by the Japan Society of Hepatology (JSH) 2010 updated version. Dig Dis 2011; 29: Park JW. Practice guideline for diagnosis and treatment of hepatocellular carcinoma. Korean J Hepatol 2004; 10: Kim YS, Jung SE, Choi BG, et al. Image quality improvement after implementation of a CT accreditation program. Korean J Radiol 2010; 11: Browne JE, Watson AJ, Gibson NM, Dudley NJ, Elliott AT. Objective measurements of image quality. Ultrasound Med Biol 2004; 30: Thijssen JM, Weijers G, de Korte CL. Objective performance testing and quality assurance of medical ultrasound equipment. Ultrasound Med Biol 2007; 33: Sipilä O, Mannila V, Vartiainen E. Quality assurance in diagnostic ultrasound. Eur J Radiol 2010; 80: Dudley NJ, Griffith K, Houldsworth G, Holloway M, Dunn MA. A review of two alternative ultrasound quality assurance programmes. Eur J Ultrasound 2001; 12: Gibson NM, Dudley NJ, Griffith K. A computerized quality control testing system for B-mode ultrasound. Ultrasound Med Biol 2001; 27: J Ultrasound Med 2014; 33: