A study of x-ray image perception for pneumoconiosis detection

1. A study of x-ray image perception for pneumoconiosis detection MS Thesis Presentation Varun Jampani (200502027) varunjampani@research.iiit.ac.in Adviser: Prof. Jayanthi Sivaswamy Center for Visual Information Technology IIIT-Hyderabad

2. introduction

3. Medical Imaging Technologies • Forms one of the most effective diagnostic tools in medicine • Used for planning treatment and surgery • Several imaging technologies: PET, MRI, X-ray, Nuclear medicine, Ultrasound etc… • X-ray is still ubiquitous in clinical practice and will likely remain so for quite some time

4. Perception in medical imaging • Information in medical images itself not sufficient • Information has to be interpreted in accurate and timely manner • Sever factors affect reading medical images • Observer independent factors such as image quality and viewing settings • Observer dependent perceptual and cognitive factors The present work deals with understanding of some perceptual and cognitive factors involved in the diagnostic assessment of Pneumoconiosis.

5. Pneumoconiosis • Inflammation of lungs • Caused by prolonged inhalation of industrial dust [Mason and Broaddus 2005] • Formation of scar tissue making lungs less flexible and porous • Symptoms • Shortness of breath, cough, restless sleep, chest discomfort. • Mainly diagnosed through chest x-rays • Effective way to prevent progress of this disease is to get regular check ups • Deemed to be the most common and serious occupational lung disease in developing countries like China and India [Wang and Christiani 2003]

6. Diagnosis of Pneumoconiosis • Complex process and requires certain level of expertize [Morgan et al. 1973] • International labor organization (ILO) classification scheme [IRPAINIR Committee 1980] • Hierarchy of readers • Each lung is divided into 3 zones – Total 6 zones • Profusion level (concentration of small opacities) is assessed for each zone

7. Importance of Perception Research • Radiologist’s interpretation of medical images is highly subjective • Inter and Intra observer variations [Krupinski 2000] • At least half of errors made in clinical practice are perceptual in nature [Krupinski et al. 1998] • The ultimate aim of all perception research is to improve diagnostic accuracy by reducing errors due to perceptual and cognitive factors • Understanding perceptual factors helps in development of • Better image acquisition and viewing systems • Computer aided diagnostic (CAD) tools • Better training regimens for resident radiologists

8. Objectives of present work • We are interested in answering the following research questions • What is the role of expertize and contralateral symmetry information present in chest x-rays on the diagnostic error, time and eye movements of the observer? • Does the distribution of eye fixations change with observer error and observer assessment of pneumoconiosis? • What is the inter observer and intra observer variability of eye fixations? • What is the role of anatomical features in attracting the gaze of the observers? • What is the role of bottom up image features in attracting the gaze of the observers? • How do the visual strategies of the observers of different expertize levels change with time?

9. methodology

10. Methodology • Eye tracking experiment is chosen method of study • Experiments were conducted in a room dedicated to eye tracking experiments • Experimentally manipulated variables • Expertize • Contralateral symmetry • Disease level • Recorded variables • Profusion categorization for each lung zone • Time of analysis • Eye fixations

11. Experimental Images • Good quality PA chest x-ray images of pneumoconiosis • Single and double lung images to study the role of contralateral symmetry • Different disease stages

12. Participant (observer) details • Expertize varying from novices to staff radiologists • Total number of observers: 23

13. Eye tracker settings • Remote head free eye tracker • Model: SR Research Eyelink 1000 • Mean spatial accuracy of eye tracker is 0.50 visual angle and sampling rate was 500 Hz. • 17 inch LCD monitor • Approximate distance between the observer and the screen was around 60 cm

14. Experimental Procedure • With-in subject design • Steps in experiment • Consent form • Training session • Cover story • 9-point camera calibration • All 33 experimental images were shown one after other to each observer • Unlimited viewing time • After viewing is done, observer has to note down profusion level for each lung zone in the report form given to him/her • Eye-movement data, response times and profusion levels were recorded for each observer and for each image

15. Eye movement terminology

16. Sample saccade maps

17. Role of expertize and contralateral symmetry

18. Research Questions 1. What is the role of expertize and contralateral symmetry information on the diagnostic error, time and eye movements of the observer? 2. Does the distribution of eye fixations change with observer error and observer assessment of pneumoconiosis?

19. Some existing results on pneumoconiosis diagnosis • Image quality plays serious and significant role [Pearson et al. 1965, Reger et al. 1972] • Marked tendency to award higher readings to under-penetrated films while the opposite is true for over-penetrated films • Experienced radiologists are more able to adjust for unsatisfactory film quality • Substantial inter-reader and some intra-reader variation in assessment of pneumoconiosis [Reger et al. 1972, Kruger et al. 1974] • Experienced radiologists have more consistency • No eye tracking experiments on pneumoconiosis

20. Contralateral symmetry in chest x-rays • Detection of symmetry is one of the characteristics of human visual perception • Important role in face perception [Chen et al. 2007] and attractiveness [Grammer et al. 1994] • Contralateral subtraction technique • Proved to be useful for highlighting tumor regions in chest x-rays [Tsukuda et al. 2002] • Also used in computer aided analysis of tumors in chest x-rays [Li et al. 2000] • No empirical studies on the role of contralateral symmetry (CS) in diagnosing lung diseases. Source: Tsukuda et al. 2002

21. Analysis of observer performance • Sum of absolute differences • Observer error for each observer is obtained by averaging over all images

22. Average sum of absolute differences

23. Analysis • Observer error for double lung images is seen to vary significantly with expertise, which was confirmed by Kruskal-wallis test (χ2(6) = 13.38, p = .038) • Thus, there is decrease in error with increase in expertize • The observer error for single lung images (Mdn = 0.813) is significantly higher than that of double lung images (Mdn = 0.620). (wilcoxon signed rank test: Z = 3.13, p < .001) • Significant difference between doctors (Mdn = 0.38) and non-doctors (Mdn = 0.18) when considering the difference of observer error between single and double lung images. (Mann-Whitney test: U = 28, p = .038) Doctors: residents and staff Non-Doctors: Other groups

24. Penalize Over and Penalize Under • Penalize Over: Number of times an observer has given a profusion rating higher than that of ground truth. • Penalize Under: Number of times an observer has given a profusion rating lower than that of ground truth. • Over-estimation and Under-estimation

25. Penalize over • More penalize over in double lung images (Mdn = 0.31) than in single lung images (Mdn = 0.25) (Wilcoxon signed rank test: Z = 2.13, p = .033)

26. Penalize Under • More penalize under in single lung images (Mdn = 0.34) than in double lung images (Mdn = 0.28) (Z = 3.89, p < .001) • Thus CS information is helping in not under-estimating profusion values

27. Inferences • CS plays a significant role in the diagnosis of pneumoconiosis and its role is more important in doctors than in the case of non-doctors. • A previous study [Rockoff and Schwartz 1988] on underestimation of asbestosis (a variant of pneumoconiosis) • Thus, CS information helps by reducing the tendency of giving less profusion ratings • More experiments required to study at what level (image/zonal/local) this CS information is being used

28. Time Analysis • Doctors (Mdn = 16.69s) took less time than non-doctors (Mdn = 33.24s) (Mann-Whitney test: U = 21, p = .011) • Time taken for double lung images (Mdn = 30.84s) is less than double the time taken for single lung images (Mdn = 38.38s) (Wilcoxon signed rank test: Z = 4.19, p < .001)

29. Eye fixation analysis • Average saccade velocity of doctors is significantly higher than that of non-doctors (Mann-Whitney test: U = 20, p = .01) • Average saccade amplitude is also significantly higher for doctors than that of non-doctors (U = 24, p = .022) • Doctors seem to be moving eyes more quickly and over more distances compared to non-doctors.

30. Fixations vs. observer ratings • Average percentage fixation time in lung zones with observer ratings of 1 and 2 is significantly higher than in the lung zones with observer ratings of 0 and 3, in both single and double lung images (Mann-Whitney test: p < .001). • Zones considered by the observer as definite normal (profusion rating – 0) and definite abnormal (3), are less viewed when compared to that of other zones

31. Fixations vs. observer error • Significantly less time is spent on those zones with absolute error of 3 when compared to that of zones with absolute errors of 0,1 and 2. • This shows the importance of careful analysis of each lung zone

32. Inferences • Doctors are quick and efficient where as non-doctors are slow and inefficient • For good diagnostic results, all zones should be looked carefully. • X-rays should not be speed read • Some of these results may not be applicable to x-rays of localized lung diseases such as lung cancer etc.

33. What attracts observer’s eyes while reading chest x-rays of pneumoconiosis?

34. Objectives • To get some insights into the factors guiding the attention of the observers with different expertize levels • We mainly concentrate on the study of the role of anatomical features and bottom-up saliency in guiding the fixations of the observers • Long term goal: Develop a system which predicts fixations of observers on a given chest x-ray • Can be done by analyzing the image features underlying the fixation points of radiologists.

35. Visual attention • Visual Attention: Process of selectively attending to an area of visual field while ignoring the surrounding visual areas • Mostly done by actively moving eyes over the visual scene • The eye movement control is mostly unconscious • In general, where radiologists attend to in medical images differs from what they think they have attended to

36. Top down and bottom up influences • Bottom up influences • Dependent on the features of visual stimulus • Independent of the observer • Stimulus driven or exogenous attention • Top down influences • Image independent factors such as given task or goal and knowledge of the observer • Goal-driven or endogenous attention Eye movement recordings of an observer over a picture while performing different tasks (source: Yarbus 1967)

37. Computational models of visual attention • Provides computational details of the process of visual attention • Many existing models are biologically motivated • Output of any computational visual attention system is saliency map • Many computational models have been proposed • Most are bottom-up models

38. Itti-Koch Model Source: Itti and Koch 2000

39. Eye tracking research on chest x-rays • Most work done on chest x-rays of localized lung diseases like tumors • Large areas in chest x-rays are not sampled by fixated [Kundel 2000] • Radiologists move eyes in a pattern that is neither random nor the same as that of a layman [Kundel and Wright 1969] • Evolution of fixation pattern from that of an untrained person to that of a radiologist [Kundel and La Follete 1972]

40. Global focal model of visual search • 3 main components • Overall pattern recognition • Focal attention to image detail • Decision making • Initial global response involving entire retina followed by a series of checking fixations Source: Nodine et al. 1987

41. Data Analysis in present study • Only first 80 fixations are considered for each observer • Non-parametric statistical tests are used as most data didn’t pass the Normality test • P-values less than 0.05 are considered significant • Two tailed p-values are considered whenever two groups are compared Total Mean : 79.77

42. ROC analysis for comparing saliency maps with fixations • An ROC metric is used to evaluate the performance of saliency maps to predict eye fixations • Saliency map from the fixation locations of one observer is treated as a binary classifier on every pixel in the image • Saliency maps are thresholded such that given percentage of image pixels are classified as fixated and the rest are classified as not fixated. • The fixations from remaining observers are treated as ground truth • Standard approach used in eye tracking studies

43. 40% Thresholded saliency maps 50% 60% Top: A saliency map Right: Corresponding saliency map thresholded to different percentage of pixels 10% salient 70% 80% 20% 30% 90%

44. ROC analysis • ROC curve is drawn by varying the threshold • Area under ROC curve indicates how well the saliency map of one observer can predict the ground truth fixations (fixations of remaining observers) • The more the ROC area, the better is the predicting capability of the observer saliency map • For perfect classifier: ROC area – 100 • For random classifier: ROC area - 50

45. An example ROC curve • 33% fixations in top 10% salient regions • 58% fixations in top 20% salient regions • 74% fixations in top 30% salient regions • 85% fixations in top 40% salient regions • 92% fixations in top 50% salient regions • 96% fixations in top 60% salient regions • 98% fixations in top 70% salient regions • 100% fixations in top 80% salient regions Average AUC = 79.28 (fair accuracy)

46. Inter observer fixation consistency • One of the aims is to automatically detect the areas of interest for the radiologists • A basic assumption behind this is that all the observers would look at similar locations in a given image • This assumption should be validated Does different observers fixate at same locations, in a given chest x-ray?

47. Previous research • It has been shown [Kundel 2008] that, while detecting lung nodules, even though different observers have different scan paths, the distribution of their eye fixations is similar • [Judd et al. 2009] found the good consistency between the eye fixations of different observers while free viewing the natural images Source: Kundel 2008

48. Human Saliency Maps • Fixation maps are convolved with Gaussian to get human saliency maps • Fixation points with more duration are more emphasized Human saliency map of an observer: Fixation map convolved with Gaussians; and saliency map overlayed on the original chest x-ray

49. Results: Inter-observer consistency • Median AUC for all observers: 79 (reasonably good accuracy) • More agreement in fixations among the observers of lower expertize groups than that of higher expertize groups • Thus more common factors guiding the fixations of lower expertize groups

50. Intra-observer fixation consistency • Since all images are PA chest x-rays, we can expect some intra observer fixation consistency also • What is the consistency of eye fixations of an observer while diagnosing pneumoconiosis? Does an observer fixate at same locations, in different x-ray images?