Appraisal Assistant workshop

Appraisal Assistant workshop Evidence collection (ISO15504-5 assessments)

Overview • Evidence collection overview • Evidence related screens • Getting evidences into the database • Importing a list of documents • Using spreadsheet to collect initial evidences • Using evidence registry to collect evidences • Manage evidences of process outcomes/attribute achievements • Transfer evidence data between assessment projects Workshop exercise setup: project : ISO15504, with ACQ.1, ACQ.2 on target CL5, with instantiation: Org.Process

Evidence collection- Overview • Evidence collection is guided by the assessment indicators • Map the evidence to the indicator when • (Part of) the evidence exhibits the same characteristic as a work product indicator • (Part of ) the evidence is considered the input/output of implementing a practice • (Part of) the evidence implies the non/existence of the required infrastructure/resource. • Evidence indicates the strength/weakness of the implementation • Do NOT need to find evidences for all assessment indicators • Ratings are based on the satisfaction of the process outcomes / process attribute achievements, not the fulfilment of indicators • Evidences are automatic mapped to process outcomes / attribute achievements • Evidences can NOT be mapped to outcomes / achievements without first being mapped to related assessment indicators

Evidence related screens – Evidence registry panel • Map evidence to assessment indicators • Evidences is automatic mapped to generic work product when it is mapped to specific work product • Comment how/why evidence is mapped to the indicator • Evidence detail information • Evidence coverage of process outcomes / attribute achievements

Evidence related screen – Process Attribute rating tab • Form the bases for rating process attributes • Evidences mapped to process outcomes / achievements • Can NOT add new evidences • Can remove irrelevant evidence mappings • Characterize the nature of the evidence-outcome mapping • Comment on how/why the evidence indicates the implementation of process outcomes / attribute achievements

Importing a list of documents - 1 • Documents are NOT mapped to assessment indicators (unlike the PII spreadsheets) • Can be extracted from the OU’s document database • Instantiation name must be the same as defined in assessment scope • The row will be ignored if the Appraisal Assistant can not find a matched instantiation in the assessment scope • Use the included DocumentList.XLS to ensure correct file format • Save the Evidence list tab as TAB delimited Text before importing

Importing a list of documents - 2 • 3 documents • Belongs to 2 instantiations • Not mapped to any practice • Check error log for rows that are ignored

Importing a list of documents - 3 Result of importing the previous document list

Using spreadsheet to collect initial evidences - 1 • One row per evidence, insert rows for more evidences • Leave the row EMPTY if there is NO evidence • Each evidence can be mapped to multiple assessment indicators • Each evidence can NOT be mapped to the same assessment indicator more than once • Do NOT need to find evidence for ALL indicators • Consists of all indicators of the entire assessment scope • User ensures data accuracy and consistency • XML spreadsheets can be opened by Microsoft Excel 2002 or newer • Save as ‘Tab Delimited Text’ file before being imported • Program merges evidences only if they have the same name AND the same document link strings

Using spreadsheet to collect initial evidences -2 • 2 evidences here, mapped to 4 indicators • Process Instantiation name in the top most cell • Detail descriptions of process / process attribute / assessment indicator in cell comments • ACQ.1/18-08 has 2 evidences • ACQ.1/BP1 has no evidence

Using spreadsheet to collect initial evidences - Imported • 2 evidences imported from the previous spreadsheet • All 2 evidences belong to the same instantiation • Evidences mapped to specific work products are automatic mapped to the generic products of the same process

Using spreadsheet to collect initial evidences - 4 • 4 evidences mapped to 4 specific work products • The 2 ‘Test evidence A’ have different document links • The 2 ‘Test evidence B’ do not have document link

Using spreadsheet to collect initial evidences - 5 • 4 evidences are imported from the previous spreadsheet • Evidences with duplicated name strings will cause confusions when validating data • Use unique name string for evidence • Import routine can merge evidences if they have the same name AND the same document link

Using spreadsheet to collect initial evidences – Exercise

Using evidence registry – Collect evidence - 1 • Organization provides evidences but unsure how they are related to the assessment model • Assessors familiar with the ISO15504-5 model • Deleting an evidence record will also remove it from related process outcomes, achievements, and indicators

Using evidence registry – Collect evidence - 2 • Click “Add” button to add a new evidence record • Enter the Name of the evidence • Specify the instantiation (optional) • Map the evidence to assessment indicator(s) • If the evidence is mapped to a specific work product, the program will also map the evidence to its generic work product • Comment on why/how the evidence is mapped to the assessment indicator(s)

Using evidence registry – Collect evidence exercise

Using evidence registry - Collect evidence by assessment indicators - 1 • Organization provides evidences but unsure how they are related to the assessment model • Assessors unfamiliar with the ISO15504-5 model • Use indicator description in the tool to help identify what evidences to look for • No need to find evidence for ALL indicators • Deleting an evidence record will also remove it from related process outcomes, achievements, and indicators

Using evidence registry – Collect evidence by assessment indicators - 2 • Select the assessment indicator of interest, use the indicator description as a guide, look for evidence • Click “Add” button to add a new evidence • The evidence is automatic mapped to the selected indicator • Name the new evidence • Specify the instantiation of the evidence (optional) • Comment on why/how the evidence is mapped to the indicator

Using evidence registry – Collect evidence by indicators - Exercise

Using evidence registry – Evidence detail information • Only evidence name is mandatory • Evidence status, document link can also be changed through context menu of the evidence list • Evidence name and instantiation can be changed on the evidence list • Label: a short string assigned by the assessment team to identify the documents. • Source/Affirmation: the source/owner of the evidence

Using evidence registry – Evidence coverage report • Show evidence count for process outcomes /attribute achievements • Red cells highlight the model elements with 0 evidence • User needs to ensure the data gathered is sufficient to cover the assessment scope (ISO15504-2, 4.2.2 b/c) • Select the process outcome /attribute achievement to display all related indicators

Process attribute tab – Remove irrelevant evidences mappings - 1 • Evidences are automatic mapped to process outcomes/attribute achievements • The mapping is done by looking up the mapping table defined in ISO15504-5 • No guaranty all mapped evidences are meaningful for the process outcome/attribute achievement • Same evidence can be mapped to the same process outcome more than once through different indicators

Process attribute tab – Remove irrelevant evidences mappings - 2 • Clear the check box to remove an evidence mapping. • Click ‘Delete Unchecked’ will deleted it from the list • Click ‘Reset All evidence’ to reset the list back to the initial state

Process attribute tab – characterize and comment evidences mappings • An evidence can be characterized as strength or weakness • Comment on how the evidence contribute to the rating of process outcome/attribute achievement • Not the same as the evidence-indicator comment • Evidence-indicator comments are shown at the bottom as read only

Transfer evidences between assessment projects • Evidences that are out of destination scope are NOT imported • Import routine does NOT check for duplication with the existing evidences • Import the database once and ONLY once. • Import the database before generating any mini teams

Workshop revision history

Appraisal Assistant workshop