Accessing Tacit Knowledge and Linking it to the Peer-Reviewed Literature

1. Accessing Tacit Knowledge and Linking it to the Peer-Reviewed Literature Michael Shepherd Web Information Filtering Lab Faculty of Computer Science Dalhousie University

2. Research Team • Students • Qiufen Qiu (MD and MCS) • Zhixin Chen (MHI and BSc) • Computer Science Faculty • Michael Shepherd • Qigang Gao • Syed Sibte Raza Abidi • Anaesthesia & Psychology • G. Allen Finley

3. Overview • Introduction • Research Program • Results to Date • Summary

4. Pediatric Pain Discussion List • Clinical discussion on pediatric pain • Informal email-based discussion among professionals • Initiated in 1993 • Over 700 subscribers world-wide • More than 10,000 messages

5. Date: Wed, 04 Jan 1995 16:54:48 -0500 (EST) From: poster Subject: opioids and meningitis X is a 13 month (9.8kg) old boy suffering from acute meningitis (pneumocoque) treated with IV cefotaxime; at day three, I have been called as pediatric pain consultant to assess X; I have discovered an extreme painfull state: one could not handle or touch him without producing screaming. The child was unable to move spontaneously he looked paralysed by pain and hypertonia ; he also presented a neurological complication : ptosis at the right side.The pain treatment was IV acetaminophen. The first day I have prescribed IV Nalbuphine (weak opioid u antagonist and agonist) 11mg/24h after a loading dose of 1.4 mg; Pain at rest has been succesfully relieved but not the mobilisation pain; the dose has been increased at 14 mg/day wihout relieving the pain associated with moving; he has moved spontaneously limbs 2 days later; nalbuphine has been stopped 4 days later. Neurological examination and CT scan have been still normal (except ptosis) during this period. No opioid's side effects have been observed. What do you think of this case ?Have you any experience with opioids and acute meningitis ? Dr Poster, Pediatric pain unit, Poster Hospital

6. Date: Wed, 04 Jan 1995 17:27:25 -0500 (EST) From: first reply Subject: re: opioids and meningitis Is there any periosteal involvement? If so an NSAID (ibuprofen or naproxen) may be much more effective than even opioid. -------------------------

7. Date: Wed, 04 Jan 1995 19:06:32 -0400 From: second reply Subject: Re: opioids and meningitis Poster writes: > X is a 13 month (9.8kg) old boy suffering from acute meningitis... > extreme painfull state: one could not handle or touch him without > producing screaming.... > The first day I have prescribed IV Nalbuphine ... > succesfully relieved but not the mobilisation pain;... > has moved spontaneously limbs 2 days later; nalbuphine has been stopped 4 > days later. Neurological examination and CT scan have been still normal... I haveused IV morphine for similar severe meningitis pain, with success. I wouldn't hesitate to use a pure opioid agonist (in conjunction with acetaminophen, NSAID, and/or tricyclics). However, it sounds like you have the situation under control. Second Reply, Associate Professor, Dept and University -------------------------

8. Date: Thu, 05 Jan 1995 18:58:32 -0800 (PST) From: Third Reply Subject: Re: opioids and meningitis I wonder if the problem is not due to severe arachnoiditis that is secondary to the inflammation. I would suggest a trial of steroids in this patient, perhaps in combination with a benzodiazepine to reduce the spasm. Narcotics may reduce the pain but I would not like to keep X on them for too long. Good luck Third Reply -------------------------  

9. Tacit and Explicit Knowledge • Tacit knowledge is what the knower knows and is derived from experience • Explicit knowledge is represented by some artifact such as a document or journal article

10. Externalization Tacit Knowledge Explicit Knowledge Combination Socialization Internalization Knowledge Transformation Processes

11. Knowledge Transformation Processes

12. Research Questions • Externalization How can we capture the tacit knowledge in such a discussion list and transform it into explicit knowledge? • Combination How can we organize this explicit knowledge? • Internalization How do we provide access to this explicit knowledge so that users can internalize this knowledge? • Linking Tacit Knowledge to Best Evidence How do we map this transformed tacit knowledge to the appropriate best evidence literature?

13. Access Thread Clusters PPML Data Cleaning Thread Creation PubMed Articles Mapping Tacit Knowledge to Explicit Knowledge in Medical Literature Mesh Terminology Map Linking Externalization Internalization Combination

14. Data Cleaning • Remove duplicate messages (subject & time stamp) • Remove responses that were generated automatically by “vacation” mail programs • Remove other “junk” e-mails • Removing unnecessary content of the messages themselves. This unnecessary content included non-textual material such as images that would not be used in the clustering process and included original messages that were more than ten lines long as these would skew the clustering process. • The initial stage of this cleaning was done manually until patterns were recognized and then programs were written to clean the data based on these patterns.

15. Externalization: Creating Threads • Messages were threaded based on time stamps and subject headings. • Those messages that had a blank subject field were processed based on the included original messages to which they had replied.

16. Thread Representation • Each thread is treated as though it were a contiguous document • The original messages that are embedded in the reply messages are removed. • Stop words are removed • If not on the stop list, they are matched against a synonym dictionary manually created by a pediatric pain specialist. • The remaining terms are stemmed • The stemmed terms are assigned tf.idf weights

17. Data Set • An archived sample of 6939 messages from 1993-1999 • After cleaning 4033 messages • After threading 1289 threads • Each thread is represented by a vector of 4111 term weights

18. Thread-Term Matrix term1 term2 term3 . . . term4111 thread1w1,1w1,2 . . . w1,4111 thread2w2,1w2,2 . . . w2,4111 . . . thread1289w1289,1w1289,2 . . . w1289,4111

19. Combination: Organizing the Threads • Text clustering • unsupervised learning process • groups documents into clusters so that the documents within a cluster have high similarity with one another, but are very dissimilar to the documents in the other clusters • Text classification or categorization • supervised learning process • Assigns documents to pre-defined classes or categories

20. k-means clustering with k=2 3 1 5 7 6 2 4

21. k-means clustering with k=2 7 1 3 5 6 4 2

22. k-means clustering with k=2 4 6 3 2 7 1 5

25. Evaluation of Clustering • Performed a study in which 100 randomly selected threads were presented to two experts for clustering and to our clustering algorithm • Results of clustering between the experts measured • Results of clustering between the experts and the system measured

26. Clusters and labels created by expert 1 – a psychologist

27. Clusters and labels created by expert 2 – a medical doctor

28. Inter-Rater Reliability The Redundancy(X, Y) is the proportion of uncertainty about X that is removed by knowing Y In this instance, X and Y represent the two sets of clusters generated by the experts. The measure is asymmetrical and the calculated redundancy measures are: R(Expert-1, Expert-2) = 0.51 R(Expert-2, Expert-1) = 0.44

29. Evaluation of the Automatically Generated Clustering • Assume each manually created cluster is correct • Compare the manually created cluster against an automatically created cluster • Recall – the proportion of those items in the manually created cluster that appear together in the same automatically generated cluster • Precision – the proportion of those items in an automatically created cluster that appear together in the same manually created cluster • F–measure = 2PR / (P+R)

30. E-1 E-2 E-n Hierarchy – k=2 C-1,1 C-2,1 C-2,2 C-3,1 C-3,3 C-3,4 C-3,2 C-4,1 C-4,2

31. F-measure for a classification The overall F-measure is used to reflect the quality of the whole hierarchy. The overall F-measure is the average weighted F-measure for all the clusters in a humanly generated clustering and is defined to be: Overall F-measure = ∑ ( |T| * F(T)) / ∑ |T|

32. E-2 E-1 0.48 0.47 k-means Evaluation of Clustering • Each expert’s set of clusters was compared to the automatically generated hierarchical clustering. The hierarchy was generated ten times using different seed centroids for each run. • The results of the paired-samples t tests (p=0.05) show that there was no significant difference between the two sets of manually generated clusters when used to evaluate the automatically generated clustering (k = 6).

33. Evaluation of k-means Clustering • We now have 3 different clusterings with inter-rater reliability of < .50 • k-means generated a large number of term representatives for each cluster with no elegant way of mapping the terms into MeSH. • Therefore, the k-means clustering algorithm was replaced with a SOM in the expectation that the clustering results would be better and that a smaller set of term representatives for each cluster might be identified.

34. SOM – Self Organizing Maps • Invented by Teuvo Kohonen • Provide a way of representing multidimensional data in much lower dimensional spaces - usually one or two dimensions. • Create a network that stores information in such a way that any topological relationships within the training set are maintained

35. Example 2-D Lattice of Nodes

36. Red = 240 Green = 89 Blue = 48 R G B 240 89 48 37 202 219

37. Mapping 3 Dimensional Colour Vectors Into 2 Dimensions Notice that in addition to clustering the colours into distinct regions, regions of similar properties are usually found adjacent to each other.

38. SOM Neighbourhood Decreases

39. Mapping 3 Dimensional Colour Vectors Into 2 Dimensions Notice that in addition to clustering the colours into distinct regions, regions of similar properties are usually found adjacent to each other.

40. Thread-Term Matrix term1 term2 term3 . . . term4111 thread1w1,1w1,2 . . . w1,4111 thread2w2,1w2,2 . . . w2,4111 . . . thread1289w1289,1w1289,2 . . . w1289,4111

41. Principal Component Analysis for Feature Length Reduction Eigen Values PCA Vectors

42. SOM – Vector Length 150

43. Growing Hierarchical SOM

44. SOM Results

45. Thread Clusters PPML Data Cleaning Thread Creation PubMed Articles Problems Mesh Terminology Map Linking Externalization Internalization Combination

46. Access Threads PPML Data Cleaning Thread Creation PubMed Articles Mapping Tacit Knowledge to Explicit Knowledge in Medical Literature Mesh and UMLS Internalization Linking Externalization Combination

47. Combination: Organizing the Threads • Text clustering • unsupervised learning process • groups documents into clusters so that the documents within a cluster have high similarity with one another, but are very dissimilar to the documents in the other clusters • Text classification or categorization • supervised learning process • Assigns documents to pre-defined classes or categories

48. MetaMap Transfer (MMTx) • Discovers UMLS Metathesaurus concepts in text • Text is parsed into components including sentences, paragraphs, phrases, lexical elements and tokens. Produces a shallow syntacitc analysis with part-of-speech tagging. • Variants are generated from the resulting phrases. Includes acronyms, abbreviations and synonyms. • Candidate concepts from the UMLS Metathesaurus are retrieved and evaluated against the phrases. • The best of the candidates are organized into a final mapping in such a way as to best cover the text.

49. Metathesaurus Candidates The word "discharge" returns Semantic Group: Anatomy Discharge, Body Substance (C0012621) - [Body Substance] Discharge, Body Substance, Sample (C0600083) - [Body Substance]Semantic Group: Procedures Patient Discharge (C0030685) - [Health Care Activity]from the UMLS Knowledge Server

50. MMTx Scores MeSH Concept Number MeSH Concept Term UMLS Semantic Type Metathesaurus Candidates "He is to be discharged home..."Phrase: "discharged"Meta Candidates (3) 966 C0030685:Discharge <1> (Patient Discharge) [Health Care Activity] 966 C0600083:Discharge <3> (Discharge, Body Substance, Sample) [Body Substance] 966 C0012621:Discharge, NOS (Discharge, Body Substance) [Body Substance]Phrase: "home"Meta Candidates (3) 1000 C0442517:Home [Manufactured Object] 928 C0237154:homeless <1> (Homelessness) [Finding] 928 C0019863:homeless <2> (Homeless persons) [Population Group]