Profiling postgraduate workplace-based assessment implementation in Ireland: a retrospective cohort study
Identifieur interne : 000107 ( Pmc/Corpus ); précédent : 000106; suivant : 000108Profiling postgraduate workplace-based assessment implementation in Ireland: a retrospective cohort study
Auteurs : Aileen Barrett ; Rose Galvin ; Yvonne Steinert ; Albert Scherpbier ; Ann O Haughnessy ; Gillian Walsh ; Mary HorganSource :
- SpringerPlus [ 2193-1801 ] ; 2016.
Abstract
In 2010, workplace-based assessment (WBA) was formally integrated as a method of formative trainee assessment into 29 basic and higher specialist medical training (BST/HST) programmes in six postgraduate training bodies in Ireland. The aim of this study is to explore how WBA is being implemented and to examine if WBA is being used formatively as originally intended. A retrospective cohort study was conducted and approved by the institution’s Research Ethics Committee. A profile of WBA requirements was obtained from 29 training programme curricula. A data extraction tool was developed to extract anonymous data, including written feedback and timing of assessments, from Year 1 and 2 trainee ePortfolios in 2012–2013. Data were independently quality assessed and compared to the reference standard number of assessments mandated annually where relevant. All 29 training programmes mandated the inclusion of at least one case-based discussion (max = 5; range 1–5). All except two non-clinical programmes (93 %) required at least two mini-Clinical Evaluation Exercise assessments per year and Direct Observation of Procedural Skills assessments were mandated in 27 training programmes over the course of the programme. WBA data were extracted from 50 % of randomly selected BST ePortfolios in four programmes (n = 142) and 70 % of HST ePortfolios (n = 115) in 21 programmes registered for 2012–2013. Four programmes did not have an eligible trainee for that academic year. In total, 1142 WBAs were analysed. A total of 164 trainees (63.8 %) had completed at least one WBA. The average number of WBAs completed by HST trainees was 7.75 (SD 5.8; 95 % CI 6.5–8.9; range 1–34). BST trainees completed an average of 6.1 assessments (SD 9.3; 95 % CI 4.01–8.19; range 1–76). Feedback—of varied length and quality—was provided on 44.9 % of assessments. The majority of WBAs were completed in the second half of the year. There is significant heterogeneity with respect to the frequency and quality of feedback provided during WBAs. The completion of WBAs later in the year may limit available time for feedback, performance improvement and re-evaluation. This study sets the scene for further work to explore the value of formative assessment in postgraduate medical education.
Url:
DOI: 10.1186/s40064-016-1748-x
PubMed: 26933632
PubMed Central: 4761346
Links to Exploration step
PMC:4761346Le document en format XML
<record><TEI><teiHeader><fileDesc><titleStmt><title xml:lang="en">Profiling postgraduate workplace-based assessment implementation in Ireland: a retrospective cohort study</title><author><name sortKey="Barrett, Aileen" sort="Barrett, Aileen" uniqKey="Barrett A" first="Aileen" last="Barrett">Aileen Barrett</name><affiliation><nlm:aff id="Aff1">Education and Professional Development Unit, Royal College of Physicians of Ireland, Frederick House, 19 South Frederick St, Dublin 2, Ireland</nlm:aff></affiliation><affiliation><nlm:aff id="Aff2">School of Medicine, College of Medicine and Health Sciences, Brookfield Health Sciences Complex, University College Cork, Cork, Ireland</nlm:aff></affiliation></author><author><name sortKey="Galvin, Rose" sort="Galvin, Rose" uniqKey="Galvin R" first="Rose" last="Galvin">Rose Galvin</name><affiliation><nlm:aff id="Aff3">Discipline of Physiotherapy, Department of Clinical Therapies, Faculty of Education and Health Sciences, University of Limerick, Limerick, Ireland</nlm:aff></affiliation></author><author><name sortKey="Steinert, Yvonne" sort="Steinert, Yvonne" uniqKey="Steinert Y" first="Yvonne" last="Steinert">Yvonne Steinert</name><affiliation><nlm:aff id="Aff4">Centre for Medical Education, Faculty of Medicine, McGill University, Lady Meredith House, 1110 Pine Avenue West, Montreal, QC H3A 1A3 Canada</nlm:aff></affiliation></author><author><name sortKey="Scherpbier, Albert" sort="Scherpbier, Albert" uniqKey="Scherpbier A" first="Albert" last="Scherpbier">Albert Scherpbier</name><affiliation><nlm:aff id="Aff5">Faculty of Health, Medicine and Life Sciences, University of Maastricht, Universiteitssingel 60, 6229 ER Maastricht, The Netherlands</nlm:aff></affiliation></author><author><name sortKey="O Haughnessy, Ann" sort="O Haughnessy, Ann" uniqKey="O Haughnessy A" first="Ann" last="O Haughnessy">Ann O Haughnessy</name><affiliation><nlm:aff id="Aff1">Education and Professional Development Unit, Royal College of Physicians of Ireland, Frederick House, 19 South Frederick St, Dublin 2, Ireland</nlm:aff></affiliation></author><author><name sortKey="Walsh, Gillian" sort="Walsh, Gillian" uniqKey="Walsh G" first="Gillian" last="Walsh">Gillian Walsh</name><affiliation><nlm:aff id="Aff1">Education and Professional Development Unit, Royal College of Physicians of Ireland, Frederick House, 19 South Frederick St, Dublin 2, Ireland</nlm:aff></affiliation></author><author><name sortKey="Horgan, Mary" sort="Horgan, Mary" uniqKey="Horgan M" first="Mary" last="Horgan">Mary Horgan</name><affiliation><nlm:aff id="Aff2">School of Medicine, College of Medicine and Health Sciences, Brookfield Health Sciences Complex, University College Cork, Cork, Ireland</nlm:aff></affiliation></author></titleStmt><publicationStmt><idno type="wicri:source">PMC</idno><idno type="pmid">26933632</idno><idno type="pmc">4761346</idno><idno type="url">http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4761346</idno><idno type="RBID">PMC:4761346</idno><idno type="doi">10.1186/s40064-016-1748-x</idno><date when="2016">2016</date><idno type="wicri:Area/Pmc/Corpus">000107</idno><idno type="wicri:explorRef" wicri:stream="Pmc" wicri:step="Corpus" wicri:corpus="PMC">000107</idno></publicationStmt><sourceDesc><biblStruct><analytic><title xml:lang="en" level="a" type="main">Profiling postgraduate workplace-based assessment implementation in Ireland: a retrospective cohort study</title><author><name sortKey="Barrett, Aileen" sort="Barrett, Aileen" uniqKey="Barrett A" first="Aileen" last="Barrett">Aileen Barrett</name><affiliation><nlm:aff id="Aff1">Education and Professional Development Unit, Royal College of Physicians of Ireland, Frederick House, 19 South Frederick St, Dublin 2, Ireland</nlm:aff></affiliation><affiliation><nlm:aff id="Aff2">School of Medicine, College of Medicine and Health Sciences, Brookfield Health Sciences Complex, University College Cork, Cork, Ireland</nlm:aff></affiliation></author><author><name sortKey="Galvin, Rose" sort="Galvin, Rose" uniqKey="Galvin R" first="Rose" last="Galvin">Rose Galvin</name><affiliation><nlm:aff id="Aff3">Discipline of Physiotherapy, Department of Clinical Therapies, Faculty of Education and Health Sciences, University of Limerick, Limerick, Ireland</nlm:aff></affiliation></author><author><name sortKey="Steinert, Yvonne" sort="Steinert, Yvonne" uniqKey="Steinert Y" first="Yvonne" last="Steinert">Yvonne Steinert</name><affiliation><nlm:aff id="Aff4">Centre for Medical Education, Faculty of Medicine, McGill University, Lady Meredith House, 1110 Pine Avenue West, Montreal, QC H3A 1A3 Canada</nlm:aff></affiliation></author><author><name sortKey="Scherpbier, Albert" sort="Scherpbier, Albert" uniqKey="Scherpbier A" first="Albert" last="Scherpbier">Albert Scherpbier</name><affiliation><nlm:aff id="Aff5">Faculty of Health, Medicine and Life Sciences, University of Maastricht, Universiteitssingel 60, 6229 ER Maastricht, The Netherlands</nlm:aff></affiliation></author><author><name sortKey="O Haughnessy, Ann" sort="O Haughnessy, Ann" uniqKey="O Haughnessy A" first="Ann" last="O Haughnessy">Ann O Haughnessy</name><affiliation><nlm:aff id="Aff1">Education and Professional Development Unit, Royal College of Physicians of Ireland, Frederick House, 19 South Frederick St, Dublin 2, Ireland</nlm:aff></affiliation></author><author><name sortKey="Walsh, Gillian" sort="Walsh, Gillian" uniqKey="Walsh G" first="Gillian" last="Walsh">Gillian Walsh</name><affiliation><nlm:aff id="Aff1">Education and Professional Development Unit, Royal College of Physicians of Ireland, Frederick House, 19 South Frederick St, Dublin 2, Ireland</nlm:aff></affiliation></author><author><name sortKey="Horgan, Mary" sort="Horgan, Mary" uniqKey="Horgan M" first="Mary" last="Horgan">Mary Horgan</name><affiliation><nlm:aff id="Aff2">School of Medicine, College of Medicine and Health Sciences, Brookfield Health Sciences Complex, University College Cork, Cork, Ireland</nlm:aff></affiliation></author></analytic><series><title level="j">SpringerPlus</title><idno type="eISSN">2193-1801</idno><imprint><date when="2016">2016</date></imprint></series></biblStruct></sourceDesc></fileDesc><profileDesc><textClass></textClass></profileDesc></teiHeader><front><div type="abstract" xml:lang="en"><p>In 2010, workplace-based assessment (WBA) was formally integrated as a method of formative trainee assessment into 29 basic and higher specialist medical training (BST/HST) programmes in six postgraduate training bodies in Ireland. The aim of this study is to explore how WBA is being implemented and to examine if WBA is being used formatively as originally intended. A retrospective cohort study was conducted and approved by the institution’s Research Ethics Committee. A profile of WBA requirements was obtained from 29 training programme curricula. A data extraction tool was developed to extract anonymous data, including written feedback and timing of assessments, from Year 1 and 2 trainee ePortfolios in 2012–2013. Data were independently quality assessed and compared to the reference standard number of assessments mandated annually where relevant. All 29 training programmes mandated the inclusion of at least one case-based discussion (max = 5; range 1–5). All except two non-clinical programmes (93 %) required at least two mini-Clinical Evaluation Exercise assessments per year and Direct Observation of Procedural Skills assessments were mandated in 27 training programmes over the course of the programme. WBA data were extracted from 50 % of randomly selected BST ePortfolios in four programmes (n = 142) and 70 % of HST ePortfolios (n = 115) in 21 programmes registered for 2012–2013. Four programmes did not have an eligible trainee for that academic year. In total, 1142 WBAs were analysed. A total of 164 trainees (63.8 %) had completed at least one WBA. The average number of WBAs completed by HST trainees was 7.75 (SD 5.8; 95 % CI 6.5–8.9; range 1–34). BST trainees completed an average of 6.1 assessments (SD 9.3; 95 % CI 4.01–8.19; range 1–76). Feedback—of varied length and quality—was provided on 44.9 % of assessments. The majority of WBAs were completed in the second half of the year. There is significant heterogeneity with respect to the frequency and quality of feedback provided during WBAs. The completion of WBAs later in the year may limit available time for feedback, performance improvement and re-evaluation. This study sets the scene for further work to explore the value of formative assessment in postgraduate medical education.</p></div></front><back><div1 type="bibliography"><listBibl><biblStruct><analytic><author><name sortKey="Ali, J" uniqKey="Ali J">J Ali</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Bennett, Re" uniqKey="Bennett R">RE Bennett</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Bindal, T" uniqKey="Bindal T">T Bindal</name></author><author><name sortKey="Wall, D" uniqKey="Wall D">D Wall</name></author><author><name sortKey="Goodyear, Hm" uniqKey="Goodyear H">HM Goodyear</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Black, D" uniqKey="Black D">D Black</name></author><author><name sortKey="Welch, J" uniqKey="Welch J">J Welch</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Bok, H" uniqKey="Bok H">H Bok</name></author><author><name sortKey="Teunissen, P" uniqKey="Teunissen P">P Teunissen</name></author><author><name sortKey="Favier, R" uniqKey="Favier R">R Favier</name></author><author><name sortKey="Rietbroek, N" uniqKey="Rietbroek N">N Rietbroek</name></author><author><name sortKey="Theyse, L" uniqKey="Theyse L">L Theyse</name></author><author><name sortKey="Brommer, H" uniqKey="Brommer H">H Brommer</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Bok, H" uniqKey="Bok H">H Bok</name></author><author><name sortKey="Teunissen, P" uniqKey="Teunissen P">P Teunissen</name></author><author><name sortKey="Spruijt, A" uniqKey="Spruijt A">A Spruijt</name></author><author><name sortKey="Fokkema, J" uniqKey="Fokkema J">J Fokkema</name></author><author><name sortKey="Van Beukelen, P" uniqKey="Van Beukelen P">P Van Beukelen</name></author><author><name sortKey="Jaarsma, D" uniqKey="Jaarsma D">D Jaarsma</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Bok, Hgj" uniqKey="Bok H">HGJ Bok</name></author><author><name sortKey="Jaarsma, Dadc" uniqKey="Jaarsma D">DADC Jaarsma</name></author><author><name sortKey="Spruijt, A" uniqKey="Spruijt A">A Spruijt</name></author><author><name sortKey="Van Beukelen, P" uniqKey="Van Beukelen P">P Van Beukelen</name></author><author><name sortKey="Van Der Vleuten, Cpm" uniqKey="Van Der Vleuten C">CPM Van Der Vleuten</name></author><author><name sortKey="Teunissen, Pw" uniqKey="Teunissen P">PW Teunissen</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Clark, I" uniqKey="Clark I">I Clark</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Cohen, S" uniqKey="Cohen S">S Cohen</name></author><author><name sortKey="Farrant, P" uniqKey="Farrant P">P Farrant</name></author><author><name sortKey="Taibjee, S" uniqKey="Taibjee S">S Taibjee</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Cook, D" uniqKey="Cook D">D Cook</name></author><author><name sortKey="Zendejas, B" uniqKey="Zendejas B">B Zendejas</name></author><author><name sortKey="Hamstra, S" uniqKey="Hamstra S">S Hamstra</name></author><author><name sortKey="Hatala, R" uniqKey="Hatala R">R Hatala</name></author><author><name sortKey="Brydges, R" uniqKey="Brydges R">R Brydges</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Cook, Da" uniqKey="Cook D">DA Cook</name></author><author><name sortKey="Brydges, R" uniqKey="Brydges R">R Brydges</name></author><author><name sortKey="Ginsburg, S" uniqKey="Ginsburg S">S Ginsburg</name></author><author><name sortKey="Hatala, R" uniqKey="Hatala R">R Hatala</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Driessen, E" uniqKey="Driessen E">E Driessen</name></author><author><name sortKey="Scheele, F" uniqKey="Scheele F">F Scheele</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Fokkema, J" uniqKey="Fokkema J">J Fokkema</name></author><author><name sortKey="Teunissen, Pw" uniqKey="Teunissen P">PW Teunissen</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Fokkema, Jpi" uniqKey="Fokkema J">JPI Fokkema</name></author><author><name sortKey="Teunissen, Pw" uniqKey="Teunissen P">PW Teunissen</name></author><author><name sortKey="Westerman, M" uniqKey="Westerman M">M Westerman</name></author><author><name sortKey="Van Der Lee, N" uniqKey="Van Der Lee N">N van der Lee</name></author><author><name sortKey="Van Der Vleuten, Cpm" uniqKey="Van Der Vleuten C">CPM van der Vleuten</name></author><author><name sortKey="Scherpbier, Ajja" uniqKey="Scherpbier A">AJJA Scherpbier</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Hassell, A" uniqKey="Hassell A">A Hassell</name></author><author><name sortKey="Bullock, A" uniqKey="Bullock A">A Bullock</name></author><author><name sortKey="Whitehouse, A" uniqKey="Whitehouse A">A Whitehouse</name></author><author><name sortKey="Wood, L" uniqKey="Wood L">L Wood</name></author><author><name sortKey="Jones, P" uniqKey="Jones P">P Jones</name></author><author><name sortKey="Wall, D" uniqKey="Wall D">D Wall</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Hatala, R" uniqKey="Hatala R">R Hatala</name></author><author><name sortKey="Cook, D" uniqKey="Cook D">D Cook</name></author><author><name sortKey="Brydges, R" uniqKey="Brydges R">R Brydges</name></author><author><name sortKey="Hawkins, R" uniqKey="Hawkins R">R Hawkins</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Kogan, Jr" uniqKey="Kogan J">JR Kogan</name></author><author><name sortKey="Holmboe, Es" uniqKey="Holmboe E">ES Holmboe</name></author><author><name sortKey="Hauer, Ke" uniqKey="Hauer K">KE Hauer</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Kogan, Jr" uniqKey="Kogan J">JR Kogan</name></author><author><name sortKey="Conforti, Ln" uniqKey="Conforti L">LN Conforti</name></author><author><name sortKey="Bernabeo, Ec" uniqKey="Bernabeo E">EC Bernabeo</name></author><author><name sortKey="Durning, Sj" uniqKey="Durning S">SJ Durning</name></author><author><name sortKey="Hauer, Ke" uniqKey="Hauer K">KE Hauer</name></author><author><name sortKey="Holmboe, Es" uniqKey="Holmboe E">ES Holmboe</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Menon, S" uniqKey="Menon S">S Menon</name></author><author><name sortKey="Winston, M" uniqKey="Winston M">M Winston</name></author><author><name sortKey="Sullivan, G" uniqKey="Sullivan G">G Sullivan</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Menon, S" uniqKey="Menon S">S Menon</name></author><author><name sortKey="Winston, M" uniqKey="Winston M">M Winston</name></author><author><name sortKey="Sullivan, G" uniqKey="Sullivan G">G Sullivan</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Nicol, Dj" uniqKey="Nicol D">DJ Nicol</name></author><author><name sortKey="Macfarlane Dick, D" uniqKey="Macfarlane Dick D">D Macfarlane-Dick</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Norcini, Jj" uniqKey="Norcini J">JJ Norcini</name></author><author><name sortKey="Blank, Ll" uniqKey="Blank L">LL Blank</name></author><author><name sortKey="Arnold, Gk" uniqKey="Arnold G">GK Arnold</name></author><author><name sortKey="Kimball, Hr" uniqKey="Kimball H">HR Kimball</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="St Onge, C" uniqKey="St Onge C">C St-Onge</name></author><author><name sortKey="Young, M" uniqKey="Young M">M Young</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Swayamprakasam, Ap" uniqKey="Swayamprakasam A">AP Swayamprakasam</name></author><author><name sortKey="Segaran, A" uniqKey="Segaran A">A Segaran</name></author><author><name sortKey="Allery, L" uniqKey="Allery L">L Allery</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Vandenbroucke, Jp" uniqKey="Vandenbroucke J">JP Vandenbroucke</name></author><author><name sortKey="Ev, Elm" uniqKey="Ev E">Elm Ev</name></author><author><name sortKey="Altman, Dg" uniqKey="Altman D">DG Altman</name></author><author><name sortKey="G Tzsche, Pc" uniqKey="G Tzsche P">PC Gøtzsche</name></author><author><name sortKey="Mulrow, Cd" uniqKey="Mulrow C">CD Mulrow</name></author><author><name sortKey="Pocock, Sj" uniqKey="Pocock S">SJ Pocock</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Von Elm, E" uniqKey="Von Elm E">E von Elm</name></author><author><name sortKey="Altman, Dg" uniqKey="Altman D">DG Altman</name></author><author><name sortKey="Egger, M" uniqKey="Egger M">M Egger</name></author><author><name sortKey="Pocock, Sj" uniqKey="Pocock S">SJ Pocock</name></author><author><name sortKey="G Tzsche, Pc" uniqKey="G Tzsche P">PC Gøtzsche</name></author><author><name sortKey="Vandenbroucke, Jp" uniqKey="Vandenbroucke J">JP Vandenbroucke</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Watling, C" uniqKey="Watling C">C Watling</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Watling, C" uniqKey="Watling C">C Watling</name></author><author><name sortKey="Driessen, E" uniqKey="Driessen E">E Driessen</name></author><author><name sortKey="Van Der Vleuten, Cpm" uniqKey="Van Der Vleuten C">CPM van der Vleuten</name></author><author><name sortKey="Vanstone, M" uniqKey="Vanstone M">M Vanstone</name></author><author><name sortKey="Lingard, L" uniqKey="Lingard L">L Lingard</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Watling, C" uniqKey="Watling C">C Watling</name></author><author><name sortKey="Driessen, E" uniqKey="Driessen E">E Driessen</name></author><author><name sortKey="Van Der Vleuten, Cpm" uniqKey="Van Der Vleuten C">CPM van der Vleuten</name></author><author><name sortKey="Vanstone, M" uniqKey="Vanstone M">M Vanstone</name></author><author><name sortKey="Lingard, L" uniqKey="Lingard L">L Lingard</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Watling, C" uniqKey="Watling C">C Watling</name></author><author><name sortKey="Driessen, E" uniqKey="Driessen E">E Driessen</name></author><author><name sortKey="Van Der Vleuten, Cpm" uniqKey="Van Der Vleuten C">CPM van der Vleuten</name></author><author><name sortKey="Vanstone, M" uniqKey="Vanstone M">M Vanstone</name></author><author><name sortKey="Lingard, L" uniqKey="Lingard L">L Lingard</name></author></analytic></biblStruct></listBibl></div1></back></TEI><pmc article-type="research-article"><pmc-dir>properties open_access</pmc-dir>
<front><journal-meta><journal-id journal-id-type="nlm-ta">Springerplus</journal-id><journal-id journal-id-type="iso-abbrev">Springerplus</journal-id><journal-title-group><journal-title>SpringerPlus</journal-title></journal-title-group><issn pub-type="epub">2193-1801</issn><publisher><publisher-name>Springer International Publishing</publisher-name><publisher-loc>Cham</publisher-loc></publisher></journal-meta><article-meta><article-id pub-id-type="pmid">26933632</article-id><article-id pub-id-type="pmc">4761346</article-id><article-id pub-id-type="publisher-id">1748</article-id><article-id pub-id-type="doi">10.1186/s40064-016-1748-x</article-id><article-categories><subj-group subj-group-type="heading"><subject>Research</subject></subj-group></article-categories><title-group><article-title>Profiling postgraduate workplace-based assessment implementation in Ireland: a retrospective cohort study</article-title></title-group><contrib-group><contrib contrib-type="author" corresp="yes"><name><surname>Barrett</surname><given-names>Aileen</given-names></name><address><phone>+353 1 863 9700</phone><email>aileenbarrett@rcpi.ie</email><email>aileenbarrett@ymail.com</email></address><xref ref-type="aff" rid="Aff1"></xref><xref ref-type="aff" rid="Aff2"></xref></contrib><contrib contrib-type="author"><name><surname>Galvin</surname><given-names>Rose</given-names></name><address><email>Rose.Galvin@ul.ie</email></address><xref ref-type="aff" rid="Aff3"></xref></contrib><contrib contrib-type="author"><name><surname>Steinert</surname><given-names>Yvonne</given-names></name><address><email>yvonne.steinert@mcgill.ca</email></address><xref ref-type="aff" rid="Aff4"></xref></contrib><contrib contrib-type="author"><name><surname>Scherpbier</surname><given-names>Albert</given-names></name><address><email>a.scherpbier@maastrichtuniversity.nl</email></address><xref ref-type="aff" rid="Aff5"></xref></contrib><contrib contrib-type="author"><name><surname>O’Shaughnessy</surname><given-names>Ann</given-names></name><address><email>annoshaughnessy@rcpi.ie</email></address><xref ref-type="aff" rid="Aff1"></xref></contrib><contrib contrib-type="author"><name><surname>Walsh</surname><given-names>Gillian</given-names></name><address><email>gillianwalsh@rcpi.ie</email></address><xref ref-type="aff" rid="Aff1"></xref></contrib><contrib contrib-type="author"><name><surname>Horgan</surname><given-names>Mary</given-names></name><address><email>m.horgan@ucc.ie</email></address><xref ref-type="aff" rid="Aff2"></xref></contrib><aff id="Aff1"><label></label>Education and Professional Development Unit, Royal College of Physicians of Ireland, Frederick House, 19 South Frederick St, Dublin 2, Ireland</aff><aff id="Aff2"><label></label>School of Medicine, College of Medicine and Health Sciences, Brookfield Health Sciences Complex, University College Cork, Cork, Ireland</aff><aff id="Aff3"><label></label>Discipline of Physiotherapy, Department of Clinical Therapies, Faculty of Education and Health Sciences, University of Limerick, Limerick, Ireland</aff><aff id="Aff4"><label></label>Centre for Medical Education, Faculty of Medicine, McGill University, Lady Meredith House, 1110 Pine Avenue West, Montreal, QC H3A 1A3 Canada</aff><aff id="Aff5"><label></label>Faculty of Health, Medicine and Life Sciences, University of Maastricht, Universiteitssingel 60, 6229 ER Maastricht, The Netherlands</aff></contrib-group><pub-date pub-type="epub"><day>20</day><month>2</month><year>2016</year></pub-date><pub-date pub-type="pmc-release"><day>20</day><month>2</month><year>2016</year></pub-date><pub-date pub-type="collection"><year>2016</year></pub-date><volume>5</volume><elocation-id>133</elocation-id><history><date date-type="received"><day>6</day><month>10</month><year>2015</year></date><date date-type="accepted"><day>9</day><month>2</month><year>2016</year></date></history><permissions><copyright-statement>© Barrett et al. 2016</copyright-statement><license license-type="OpenAccess"><license-p><bold>Open Access</bold>This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (<ext-link ext-link-type="uri" xlink:href="http://creativecommons.org/licenses/by/4.0/">http://creativecommons.org/licenses/by/4.0/</ext-link>), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.</license-p></license></permissions><abstract id="Abs1"><p>In 2010, workplace-based assessment (WBA) was formally integrated as a method of formative trainee assessment into 29 basic and higher specialist medical training (BST/HST) programmes in six postgraduate training bodies in Ireland. The aim of this study is to explore how WBA is being implemented and to examine if WBA is being used formatively as originally intended. A retrospective cohort study was conducted and approved by the institution’s Research Ethics Committee. A profile of WBA requirements was obtained from 29 training programme curricula. A data extraction tool was developed to extract anonymous data, including written feedback and timing of assessments, from Year 1 and 2 trainee ePortfolios in 2012–2013. Data were independently quality assessed and compared to the reference standard number of assessments mandated annually where relevant. All 29 training programmes mandated the inclusion of at least one case-based discussion (max = 5; range 1–5). All except two non-clinical programmes (93 %) required at least two mini-Clinical Evaluation Exercise assessments per year and Direct Observation of Procedural Skills assessments were mandated in 27 training programmes over the course of the programme. WBA data were extracted from 50 % of randomly selected BST ePortfolios in four programmes (n = 142) and 70 % of HST ePortfolios (n = 115) in 21 programmes registered for 2012–2013. Four programmes did not have an eligible trainee for that academic year. In total, 1142 WBAs were analysed. A total of 164 trainees (63.8 %) had completed at least one WBA. The average number of WBAs completed by HST trainees was 7.75 (SD 5.8; 95 % CI 6.5–8.9; range 1–34). BST trainees completed an average of 6.1 assessments (SD 9.3; 95 % CI 4.01–8.19; range 1–76). Feedback—of varied length and quality—was provided on 44.9 % of assessments. The majority of WBAs were completed in the second half of the year. There is significant heterogeneity with respect to the frequency and quality of feedback provided during WBAs. The completion of WBAs later in the year may limit available time for feedback, performance improvement and re-evaluation. This study sets the scene for further work to explore the value of formative assessment in postgraduate medical education.</p></abstract><kwd-group xml:lang="en"><title>Keywords</title><kwd>Workplace-based assessment</kwd><kwd>Postgraduate medical education</kwd><kwd>Retrospective cohort study</kwd></kwd-group><custom-meta-group><custom-meta><meta-name>issue-copyright-statement</meta-name><meta-value>© The Author(s) 2016</meta-value></custom-meta></custom-meta-group></article-meta></front><body><sec id="Sec1"><title>Background</title><p>Workplace-based assessment (WBA) was originally mooted as a formative—or ‘assessment-<italic>for</italic>-learning’— practice with a primary aim of impacting trainee learning and development and to assist in focusing the trainee’s learning plans (Norcini et al. <xref ref-type="bibr" rid="CR22">1995</xref>). The format of the assessment takes place in real time, with the supervisor observing the trainee in a specific aspect of clinical practice. Since its introduction many tools have been developed (Kogan et al. <xref ref-type="bibr" rid="CR17">2009</xref>) to structure feedback on specific aspects of a trainee’s performance.</p><p>Over time, the use of WBA has expanded to include a quality assurance role (Black and Welch <xref ref-type="bibr" rid="CR4">2009</xref>) and has been mooted as a method of early identification of poor performance (Cohen et al. <xref ref-type="bibr" rid="CR9">2009</xref>). Implementation of WBA internationally has met with varied levels of success and acceptability (Fokkema et al. <xref ref-type="bibr" rid="CR14">2013</xref>) with many ongoing reservations regarding the practical feasibility of performing multiple assessments in order to comply with recommendations for good reliability while attempting to maintain the formative function of these assessments (Bok et al. <xref ref-type="bibr" rid="CR5">2013</xref>). The introduction of what is viewed as an additional demand on trainer and trainee time, in an increasingly busy and unstructured environment has also impacted on the acceptability of these learning ‘innovations’ (Fokkema et al. <xref ref-type="bibr" rid="CR14">2013</xref>; Fokkema and Teunissen <xref ref-type="bibr" rid="CR13">2013</xref>).</p><p>One of the main criticisms of the implementation of WBA has emerged where the assessments are not mapped to training programme outcomes or aligned with a defined programme of assessment throughout training (Driessen and Scheele <xref ref-type="bibr" rid="CR12">2013</xref>). Poor communication of the formative purpose of WBA has also emerged as a critical barrier to successful implementation of these tools (Bok et al. <xref ref-type="bibr" rid="CR5">2013</xref>). Attempts to communicate the formative nature of the assessments in the UK by changing the name to ‘supervised learning events’ have also been met with mixed opinions (Ali <xref ref-type="bibr" rid="CR1">2014</xref>).</p><p>The focus of workplace-based assessment research has, however, begun to take a new direction. While acknowledging the limitations of workplace-based assessment as individual summative judgments of performance, the place of these tools within a programme of assessment hinges more on their validity as formative assessments, than their reliability as summative assessments (Cook et al. <xref ref-type="bibr" rid="CR10">2014</xref>; Hatala et al. <xref ref-type="bibr" rid="CR16">2015</xref>; Cook et al. <xref ref-type="bibr" rid="CR11">2015</xref>; St-Onge and Young <xref ref-type="bibr" rid="CR23">2015</xref>). The role of narrative feedback in this conceptualisation of validity becomes therefore increasingly important.</p><p>In the Irish context, WBA was introduced as mandatory component of postgraduate medical training across six training bodies in 2010. The mini-clinical evaluation exercise (Mini-CEX) and case-based discussion (CbD) were included across all disciplines while the Direct Observation of Procedural Skills (DOPS) assessment was included for disciplines with procedural skill requirements. The Objective Structured Assessment of Technical Skills (OSATS)—with procedure-specific adaptations—was implemented in both basic and higher specialist training programmes in Obstetrics and Gynaecology. Procedure-specific DOPS forms were also developed and implemented for higher specialist training in gastroenterology.</p><sec id="Sec2"><title>Research aim</title><p>The research question posed by this study is ‘how have workplace-based assessments been integrated into higher specialist training programmes in medicine in Ireland?’</p><p>The study comprises three key objectives:<list list-type="order"><list-item><p>to describe the level of implementation of WBA in postgraduate Basic Specialist Training (BST) and Higher Specialist Training (HST) programmes in one postgraduate medical training institution in Ireland.</p></list-item><list-item><p>to compare the findings with those published from other training jurisdictions.</p></list-item><list-item><p>to explore the quality of written feedback provided in these assessments.</p></list-item></list></p></sec><sec id="Sec3"><title>Conceptual framework</title><p>This study was guided by work in two key areas of educational research, formative assessment theory (Clark <xref ref-type="bibr" rid="CR8">2012</xref>; Bennett <xref ref-type="bibr" rid="CR2">2011</xref>) and guidelines for good practice in effective feedback (Nicol and Macfarlane-Dick <xref ref-type="bibr" rid="CR21">2006</xref>; Watling <xref ref-type="bibr" rid="CR27">2014</xref>). Contemporary formative assessment theory proposes that all assessment should guide learning and development (Bok et al. <xref ref-type="bibr" rid="CR6">2013</xref>, <xref ref-type="bibr" rid="CR7">2015</xref>). Guidelines for good practice suggest that in order to be effective, feedback must be, among other factors, specific, timely and result in a further plan for development (Nicol and Macfarlane-Dick <xref ref-type="bibr" rid="CR21">2006</xref>). The mechanisms by which feedback can be deemed to be successful in this purpose remain challenging to elucidate and the learner’s response to that feedback—and therefore its ultimate use—is less predictable (Watling et al. <xref ref-type="bibr" rid="CR28">2012</xref>, <xref ref-type="bibr" rid="CR29">2013a</xref>, <xref ref-type="bibr" rid="CR30">b</xref>). This study therefore only addressed evidence of feedback provided on written assessments and did not attempt to link this directly to evidence of learning.</p></sec></sec><sec id="Sec4"><title>Methods</title><sec id="Sec5"><title>Study design</title><p>This study was conducted using a retrospective cohort design. The STROBE standardised reporting guidelines were followed to ensure the standardised conduct and reporting of the research (Vandenbroucke et al. <xref ref-type="bibr" rid="CR25">2007</xref>; von Elm et al. <xref ref-type="bibr" rid="CR26">2007</xref>). Ethical approval was obtained from the institution’s Research Ethics Committee.</p></sec><sec id="Sec6"><title>Setting and study size</title><p>The study was conducted over a 3-month period from September to December 2013. Data were extracted anonymously from trainee ePortfolios for the academic year 2012–2013 (July–July). In 2011 a new ePortfolio replaced an existing paper-based recording system for trainees commencing programmes in that year. Therefore only data for Year I and Year II trainees (BST and HST) were available to access for this study. In order to obtain a truly representative picture of the level of implementation of WBAs, and considering the small total population size, 50 % of registered BST ePortfolios and 70 % of HST ePortfolios were included in the study.</p></sec><sec id="Sec7"><title>Data extraction</title><p>A data extraction tool was developed to extract anonymous data from trainee ePortfolios prior to the study commencement. This tool (Fig. <xref rid="Fig1" ref-type="fig">1</xref>) was designed to extract data on key ‘quality indicators’ of effective feedback, adapted from a number of sources including Nicol and MacFarlane-Dick’s ‘seven principles of good feedback practice’ (Nicol and Macfarlane-Dick <xref ref-type="bibr" rid="CR21">2006</xref>) and the WBA form content in use on these assessments. These indicators were assessed as binary outcomes (present/absent) and included the presence of learner-centred feedback specific to the assessment, learning goals and further follow-up where any competence was deemed to be ‘borderline’ or ‘below expectation’. The tool was piloted using data from five sample ePortfolios with one minor change to the use of ‘weeks’ instead of months in ascertaining the timing of the assessment completion. The timing of WBAs was therefore measured in weeks from the start of the academic year (9th July 2012).<fig id="Fig1"><label>Fig. 1</label><caption><p>Data extraction tool</p></caption><graphic xlink:href="40064_2016_1748_Fig1_HTML" id="MO1"></graphic></fig></p></sec><sec id="Sec8"><title>Quality check</title><p>Data were extracted by the principal investigator (AB) and a quality check of 10 % of the data extraction sheets was conducted by a second author (RG) prior to analysis. No extraction errors were identified; however it was agreed by the two authors to exclude three trainees’ data from the final analysis due to completion errors identified in those ePortfolios.</p></sec><sec id="Sec9"><title>Data analysis</title><p>The profile of WBA requirements was analysed descriptively from an Excel spreadsheet as were data extracted from ePortfolios. Binary data is presented as proportions where the denominator represents the total number of assessments completed in the programme. Summary means and standard deviations (SDs) are reported for continuous data, with corresponding 95 % CIs. Ranges are reported to illustrate the spread in the data. Data were compared to the reference standard number of assessments mandated annually, where relevant.</p></sec></sec><sec id="Sec10"><title>Results</title><p>Data were extracted from a random selection of 50 % of BST ePortfolios in four programmes (n = 142) and 70 % of HST ePortfolios (n = 115) in 21 programmes registered for 2012–2013. Four programmes did not have an eligible trainee for that academic year. A total of 1142 individual assessments were analysed across both training programmes.</p><sec id="Sec11"><title>WBA programme integration profile</title><p>All 29 programme curricula mandated at least one CbD annually (range 1–5). Annual mini-CEX assessments were required in all but two non-clinical specialties (range 1–4). DOPS requirements varied from 0 to 37 and most were required over the course of the training programme to allow for variations in opportunities to develop procedural skills in individual rotations. Two ‘non-procedural’ programmes did not have any DOPS requirement.</p><p>In HST, General Internal Medicine (GIM) training is completed alongside one of eight subspecialties. Trainees in these programmes complete at least 1 year of ‘high intensity GIM’ in which they must complete GIM curriculum requirements only and a ‘non-GIM’ year in which they complete their specialty requirements. For all other years, trainees complete requirements for both their GIM and specialty curriculum.</p></sec><sec id="Sec12"><title>WBA completion profile</title><p>The majority of trainee ePortfolios (164; 63.8 %) contained at least one completed WBA (76.5 % HST; 53.5 % BST). The average number of WBAs completed by individual HST trainees was 7.75 (SD 5.8; 95 % CI 6.5–8.9; range 1–34). BST trainees completed an average of 6.1 assessments (SD 9.3; 95 % CI 4.01–8.19; range 1–76).</p><p>The ‘quality indicators’ for each WBA are detailed in Tables <xref rid="Tab1" ref-type="table">1</xref> and <xref rid="Tab2" ref-type="table">2</xref>.<table-wrap id="Tab1"><label>Table 1</label><caption><p>Basic specialist training results</p></caption><table frame="hsides" rules="groups"><thead><tr><th align="left"></th><th align="left">DOPS/OSATS</th><th align="left">Mini-CEX</th><th align="left">CbD</th></tr></thead><tbody><tr><td align="left">Total no completed</td><td align="left">281</td><td align="left">88</td><td align="left">94</td></tr><tr><td align="left">Average no. weeks in post before WBA completed</td><td align="left">31.7<break></break>(95 % CI 30.1–33.3; SD 13.6; range 3–52)</td><td align="left">35.2<break></break>(95 % CI 32.4–38.1; SD 13.5; range 7–52)</td><td align="left">34.4<break></break>(95 % CI 31.4–37.3; SD 14.5; range 5–52)</td></tr><tr><td align="left">Entries demonstrating defined goals</td><td align="left">0</td><td align="left">1<break></break>1.13 %</td><td align="left">0</td></tr><tr><td align="left">Entries with evidence of feedback</td><td align="left">174<break></break>61.9 %</td><td align="left">54<break></break>(61.3 %)</td><td align="left">33<break></break>(35.1 %)</td></tr><tr><td align="left">Evidence of any competence ‘borderline’ or ‘below expectation’</td><td align="left">38<break></break>(13.5 %)</td><td align="left">1<break></break>1.13 %</td><td align="left">1<break></break>1.06 %</td></tr><tr><td align="left">Evidence of follow-up</td><td align="left">17<break></break>(44.7 %)</td><td align="left">0</td><td align="left">0</td></tr></tbody></table></table-wrap><table-wrap id="Tab2"><label>Table 2</label><caption><p>Higher specialist training results</p></caption><table frame="hsides" rules="groups"><thead><tr><th align="left"></th><th align="left">DOPS/OSATS</th><th align="left">Mini-CEX</th><th align="left">CbD</th></tr></thead><tbody><tr><td align="left">Total no completed</td><td align="left">359</td><td align="left">153</td><td align="left">167</td></tr><tr><td align="left">Average no. weeks in post before WBA completed</td><td align="left">30.1<break></break>(95 % CI 28.6–31.5; SD 14.1; range 1–52)</td><td align="left">33.6<break></break>(95 % CI: 31.6–35.6′ SD 13.7; range 3–52)</td><td align="left">32.6<break></break>(95 % CI: 31.7–35.5; SD 12.4; range 2–52)</td></tr><tr><td align="left">Entries demonstrating defined goals</td><td align="left">1<break></break>0.27 %</td><td align="left">1<break></break>0.65 %</td><td align="left">1<break></break>0.59 %</td></tr><tr><td align="left">Entries with evidence of feedback</td><td align="left">104<break></break>(28.9 %)</td><td align="left">102 (66.6 %)</td><td align="left">46<break></break>(27.5 %)</td></tr><tr><td align="left">Evidence of any competence ‘borderline’ or ‘below expectation’</td><td align="left">10<break></break>(2.78 %)</td><td align="left">0</td><td align="left">2<break></break>(1.19 %)</td></tr><tr><td align="left">Evidence of follow-up</td><td align="left">0</td><td align="left">n/a</td><td align="left">0</td></tr></tbody></table></table-wrap></p><p>Assessments were mostly completed in the second half of the training year, after week 30.</p><p>Trainees were more likely to complete DOPS/OSATS than Mini-CEX or CbD assessments (ratio 3:1); 76 BST trainees completed 281 DOPS/OSATS, 88 Mini-CEX and 94 CbD assessments. A similar pattern emerged at HST where 88 trainees completed 359 DOPS/OSATS, 153 Mini-CEX and 167 CbD assessments. There were many errors in ePortfolio completion among ‘dual’ specialty trainees with WBAs entered into the incorrect logbook or use of the same WBA in both.</p><p>Feedback was provided on 44.9 % of assessments however the content of this feedback varied from one word (e.g. excellent) to complete sentences about the assessment episode. Trainer comments that pertained to the case (e.g. ‘complex case’) were not included as feedback in the analysis.</p><p>A total of 40 BST WBAs (8.63 %) and 12 HST WBAs (1.76 %) extracted contained a competence or component that was ‘borderline’ or ‘below expectation’. Of the 38 BST DOPS/OSATS assessments with a component deemed to be ‘borderline’ or ‘below expectation’, all were from within one speciality and 17 (44.7 %) were followed up with a second WBA in the same procedure. The 10 HST DOPS identified as ‘borderline’ or ‘below expectation’ were also from the same specialty; however none of these ePortfolios demonstrated evidence of follow-up.</p></sec></sec><sec id="Sec13"><title>Discussion</title><p>The aim of this study was to determine the patterns of workplace-based assessment integration throughout postgraduate medical training curricula in six training bodies. Our main findings demonstrate that while the level of implementation has been varied, the majority of trainees have experienced at least one WBA during the academic year.</p><p>The picture that has emerged in this observational study compares in many ways with the issues identified internationally; particularly those related to ineffective feedback and limited formative impact. We identified that the <italic>documentation</italic> of effective written feedback was limited; however, as these assessments take place in real-time with the trainer and trainee present, <italic>verbal</italic> feedback, which is not then transferred to the assessment forms, may also take place. A number of international institutions have implemented WBA smart-phone and tablet ‘apps’ which allow for real-time completion and uploading of the assessment feedback.</p><p>Another barrier to the provision of feedback in our study may have been the lack of an explicitly-titled free-text ‘<italic>feedback</italic>’ section; on these assessments the free text section was titled ‘<italic>comments</italic>’ and therefore was interpreted by some trainers as comments on the case, not on the trainee performance.</p><p>In our study, both at BST and HST level, trainees were more likely to complete DOPS assessments than the mini-CEX or CbD. This finding is in keeping with a UK study of dermatology trainees where the authors reported that 138 trainees completed 251 DOPS compared with 142 mini-CEX assessments (Cohen et al. <xref ref-type="bibr" rid="CR9">2009</xref>). In this study respondents reported that the Mini-CEX and Multisource Feedback (MSF) tended to feel more ‘artificial’ than DOPS; they also reported dissatisfaction with the quality of feedback provided on all assessments, despite an overall positivity about the benefits of WBAs. While there is limited empirical research exploring trainer and trainee preferences regarding assessment, it may be that trainers and trainees perceive DOPS as a more objective measure of performance as opposed to the more subjectively-perceived assessments of, for example, communication and professionalism. However, it is interesting to note that in a 2009 study of psychiatry trainees—for whom procedure-based WBAs are not usually required—Menon et al. (<xref ref-type="bibr" rid="CR19">2009</xref>) also reported that trainees were ‘unimpressed’ with the introduction of these assessments, querying their reliability, validity and impact on the quality of training.</p><p>Our study found that the majority of WBAs took place in the second half of the year. This pattern, along with the limited provision of written feedback and follow-up assessments, appears to point towards a limited use of these assessments to inform learning and development. During the implementation of WBA in the UK, one 2011 study of paediatric trainees (Bindal et al. <xref ref-type="bibr" rid="CR3">2011</xref>) reported that WBAs were still viewed as a ‘tick-box’ exercise. Menon et al. (<xref ref-type="bibr" rid="CR20">2012</xref>) reported that psychiatry trainers and trainees (Menon et al. <xref ref-type="bibr" rid="CR19">2009</xref>) understood that the introduction of WBA was both driven by a desire to improve training but that it was also ‘politically driven’; comments from these trainees also referenced the ‘tick-box exercise’ designed purely to fulfil end-of-year assessment requirements. In a recent review of the issues underlying the problems encountered in WBA implementation Swayamprakasam et al. (<xref ref-type="bibr" rid="CR24">2014</xref>) also pointed towards the need for widespread communication strategies to inform—or re-inform—the understanding of the purpose of WBA.</p><p>The potential ‘floor’ and ‘ceiling’ effect of WBA also warrants further investigation. In this study, the low number of assessments documenting a competence that was ‘borderline’ or ‘below expectation’ raises a number of issues around ‘failure to fail’. The reluctance and anxiety of trainers around the delivery of negative feedback is well documented (Kogan et al. <xref ref-type="bibr" rid="CR18">2012</xref>) as are issues with the rating systems used to structure this feedback (Hassell et al. <xref ref-type="bibr" rid="CR15">1035</xref>). In our assessments, the use of an ‘expectations’ rating system (i.e. ‘above expectation’, ‘meets expectations’) in Mini-CEX and CbD assessments, without explicit reference to curriculum outcomes or competencies, may also have been perceived as overly-subjective and less conducive to learning.</p><p>This is the first large-scale study of WBA implementation in Ireland. The methodology employed to conduct the study was rigorous and quality checks were implemented to ensure the quality and accuracy of the data. The study provides and overview of the varied integration of the assessments since the introduction of the tools and has highlighted similar issues to those identified internationally. The study was designed to provide a thorough background in developing an extensive programme of research on WBA in the Irish postgraduate medical education context and will form the basis of a large in-depth qualitative study to explore the value of WBA to both trainers and trainees. The findings have also highlighted a number of areas for further development of the assessment, particularly regarding the implementation and assessment of same. One of the main limitations of the study lies in the evaluation of the quality of feedback; only written feedback was extracted which may not accurately or fully reflect the quality or richness of verbal feedback provided at the end of the workplace-based assessments.</p></sec><sec id="Sec14"><title>Conclusion</title><p>This study was developed as a ‘scene-setting’ exploration of what has happened within our medical training programmes at our institution since the introduction of workplace-based assessment in 2010; however it reflects and adds to the international body of work on workplace-based assessment implementation. As is the case internationally, issues persist in the successful implementation of formative assessment in postgraduate medical education. Recommendations based on this study and a subsequent larger qualitative study, are currently in motion with the aim of further contributing to the international discussion on the value of formative assessment in trainee development.</p></sec></body><back><ack><title>Authors’ contributions</title><p>AB developed the study concept. AB and RG and carried out data extraction and analysis. AB and RG drafted manuscript. RG, MH, AO’S, GW, YS and AS provided guidance throughout the study and provided write-up assistance. All authors read and approved the final manuscript.</p><sec id="FPar1"><title>Acknowledgements</title><p>Dr. Ian Callanan, St. Vincent’s University Hospital provided theoretical and practical guidance on data collection.</p></sec><sec id="FPar2"><title>Competing interests</title><p>The authors declare that they have no competing interests.</p></sec></ack><ref-list id="Bib1"><title>References</title><ref id="CR1"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Ali</surname><given-names>J</given-names></name></person-group><article-title>Workplace-based assessments: Lost in translation?</article-title><source>Clin Teach</source><year>2014</year><volume>11</volume><issue>1</issue><fpage>68</fpage><lpage>69</lpage><pub-id pub-id-type="doi">10.1111/tct.12085</pub-id><pub-id pub-id-type="pmid">24405927</pub-id></element-citation></ref><ref id="CR2"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Bennett</surname><given-names>RE</given-names></name></person-group><article-title>Formative assessment: a critical review</article-title><source>Assess Educ Princ Policy Pract</source><year>2011</year><volume>18</volume><issue>1</issue><fpage>5</fpage><lpage>25</lpage></element-citation></ref><ref id="CR3"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Bindal</surname><given-names>T</given-names></name><name><surname>Wall</surname><given-names>D</given-names></name><name><surname>Goodyear</surname><given-names>HM</given-names></name></person-group><article-title>Trainee doctors’ views on workplace-based assessments: Are they just a tick box exercise?</article-title><source>Med Teach</source><year>2011</year><volume>33</volume><issue>11</issue><fpage>919</fpage><lpage>927</lpage><pub-id pub-id-type="doi">10.3109/0142159X.2011.558140</pub-id><pub-id pub-id-type="pmid">22022902</pub-id></element-citation></ref><ref id="CR4"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Black</surname><given-names>D</given-names></name><name><surname>Welch</surname><given-names>J</given-names></name></person-group><article-title>The under-performing trainee—concerns and challenges for medical educators</article-title><source>Clin Teach</source><year>2009</year><volume>6</volume><issue>2</issue><fpage>79</fpage><lpage>82</lpage><pub-id pub-id-type="doi">10.1111/j.1743-498X.2009.00273.x</pub-id></element-citation></ref><ref id="CR5"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Bok</surname><given-names>H</given-names></name><name><surname>Teunissen</surname><given-names>P</given-names></name><name><surname>Favier</surname><given-names>R</given-names></name><name><surname>Rietbroek</surname><given-names>N</given-names></name><name><surname>Theyse</surname><given-names>L</given-names></name><name><surname>Brommer</surname><given-names>H</given-names></name><etal></etal></person-group><article-title>Programmatic assessment of competency-based workplace learning: when theory meets practice</article-title><source>BMC Med Educ</source><year>2013</year><volume>13</volume><issue>1</issue><fpage>123</fpage><pub-id pub-id-type="doi">10.1186/1472-6920-13-123</pub-id><pub-id pub-id-type="pmid">24020944</pub-id></element-citation></ref><ref id="CR6"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Bok</surname><given-names>H</given-names></name><name><surname>Teunissen</surname><given-names>P</given-names></name><name><surname>Spruijt</surname><given-names>A</given-names></name><name><surname>Fokkema</surname><given-names>J</given-names></name><name><surname>Van Beukelen</surname><given-names>P</given-names></name><name><surname>Jaarsma</surname><given-names>D</given-names></name><etal></etal></person-group><article-title>Clarifying students’ feedback-seeking behaviour in clinical clerkships</article-title><source>Med Educ</source><year>2013</year><volume>47</volume><fpage>282</fpage><lpage>291</lpage><pub-id pub-id-type="doi">10.1111/medu.12054</pub-id><pub-id pub-id-type="pmid">23398014</pub-id></element-citation></ref><ref id="CR7"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Bok</surname><given-names>HGJ</given-names></name><name><surname>Jaarsma</surname><given-names>DADC</given-names></name><name><surname>Spruijt</surname><given-names>A</given-names></name><name><surname>Van Beukelen</surname><given-names>P</given-names></name><name><surname>Van Der Vleuten</surname><given-names>CPM</given-names></name><name><surname>Teunissen</surname><given-names>PW</given-names></name></person-group><article-title>Feedback-giving behaviour in performance evaluations during clinical clerkships</article-title><source>Med Teach</source><year>2015</year></element-citation></ref><ref id="CR8"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Clark</surname><given-names>I</given-names></name></person-group><article-title>Formative assessment: assessment is for self-regulated learning</article-title><source>Educ Psychol Rev</source><year>2012</year><volume>24</volume><issue>2</issue><fpage>205</fpage><lpage>249</lpage><pub-id pub-id-type="doi">10.1007/s10648-011-9191-6</pub-id></element-citation></ref><ref id="CR9"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Cohen</surname><given-names>S</given-names></name><name><surname>Farrant</surname><given-names>P</given-names></name><name><surname>Taibjee</surname><given-names>S</given-names></name></person-group><article-title>Assessing the assessments: U.K. dermatology trainees’ views of the workplace assessment tools</article-title><source>Br J Dermatol</source><year>2009</year><volume>16</volume><issue>1</issue><fpage>34</fpage><lpage>39</lpage><pub-id pub-id-type="doi">10.1111/j.1365-2133.2009.09097.x</pub-id><pub-id pub-id-type="pmid">19416267</pub-id></element-citation></ref><ref id="CR10"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Cook</surname><given-names>D</given-names></name><name><surname>Zendejas</surname><given-names>B</given-names></name><name><surname>Hamstra</surname><given-names>S</given-names></name><name><surname>Hatala</surname><given-names>R</given-names></name><name><surname>Brydges</surname><given-names>R</given-names></name></person-group><article-title>What counts as validity evidence? Examples and prevalence in a systematic review of simulation-based assessment</article-title><source>Adv Health Sci Educ</source><year>2014</year><volume>19</volume><issue>2</issue><fpage>233</fpage><lpage>250</lpage><pub-id pub-id-type="doi">10.1007/s10459-013-9458-4</pub-id></element-citation></ref><ref id="CR11"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Cook</surname><given-names>DA</given-names></name><name><surname>Brydges</surname><given-names>R</given-names></name><name><surname>Ginsburg</surname><given-names>S</given-names></name><name><surname>Hatala</surname><given-names>R</given-names></name></person-group><article-title>A contemporary approach to validity arguments: a practical guide to Kane’s framework</article-title><source>Med Educ</source><year>2015</year><volume>49</volume><issue>6</issue><fpage>560</fpage><lpage>575</lpage><pub-id pub-id-type="doi">10.1111/medu.12678</pub-id><pub-id pub-id-type="pmid">25989405</pub-id></element-citation></ref><ref id="CR12"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Driessen</surname><given-names>E</given-names></name><name><surname>Scheele</surname><given-names>F</given-names></name></person-group><article-title>What is wrong with assessment in postgraduate training? Lessons from clinical practice and educational research</article-title><source>Med Teach</source><year>2013</year><volume>35</volume><issue>7</issue><fpage>569</fpage><lpage>574</lpage><pub-id pub-id-type="doi">10.3109/0142159X.2013.798403</pub-id><pub-id pub-id-type="pmid">23701250</pub-id></element-citation></ref><ref id="CR13"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Fokkema</surname><given-names>J</given-names></name><name><surname>Teunissen</surname><given-names>PW</given-names></name></person-group><article-title>Assessing the assessment of interventions: we’re not there yet</article-title><source>Med Educ</source><year>2013</year><volume>47</volume><issue>10</issue><fpage>954</fpage><lpage>956</lpage><pub-id pub-id-type="doi">10.1111/medu.12273</pub-id><pub-id pub-id-type="pmid">24016162</pub-id></element-citation></ref><ref id="CR14"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Fokkema</surname><given-names>JPI</given-names></name><name><surname>Teunissen</surname><given-names>PW</given-names></name><name><surname>Westerman</surname><given-names>M</given-names></name><name><surname>van der Lee</surname><given-names>N</given-names></name><name><surname>van der Vleuten</surname><given-names>CPM</given-names></name><name><surname>Scherpbier</surname><given-names>AJJA</given-names></name><etal></etal></person-group><article-title>Exploration of perceived effects of innovations in postgraduate medical education</article-title><source>Med Educ</source><year>2013</year><volume>47</volume><issue>3</issue><fpage>271</fpage><lpage>281</lpage><pub-id pub-id-type="doi">10.1111/medu.12081</pub-id><pub-id pub-id-type="pmid">23398013</pub-id></element-citation></ref><ref id="CR15"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Hassell</surname><given-names>A</given-names></name><name><surname>Bullock</surname><given-names>A</given-names></name><name><surname>Whitehouse</surname><given-names>A</given-names></name><name><surname>Wood</surname><given-names>L</given-names></name><name><surname>Jones</surname><given-names>P</given-names></name><name><surname>Wall</surname><given-names>D</given-names></name></person-group><article-title>Effect of rating scales on scores given to junior doctors in multi-source feedback</article-title><source>Postgrad Med J</source><year>1035</year><volume>2012</volume><issue>88</issue><fpage>10</fpage><lpage>14</lpage></element-citation></ref><ref id="CR16"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Hatala</surname><given-names>R</given-names></name><name><surname>Cook</surname><given-names>D</given-names></name><name><surname>Brydges</surname><given-names>R</given-names></name><name><surname>Hawkins</surname><given-names>R</given-names></name></person-group><article-title>Constructing a validity argument for the objective structured assessment of technical skills (OSATS): a systematic review of validity evidence</article-title><source>Adv Health Sci Educ</source><year>2015</year></element-citation></ref><ref id="CR17"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Kogan</surname><given-names>JR</given-names></name><name><surname>Holmboe</surname><given-names>ES</given-names></name><name><surname>Hauer</surname><given-names>KE</given-names></name></person-group><article-title>Tools for direct observation and assessment of clinical skills of medical trainees</article-title><source>JAMA</source><year>2009</year><volume>302</volume><issue>12</issue><fpage>1316</fpage><lpage>1326</lpage><pub-id pub-id-type="doi">10.1001/jama.2009.1365</pub-id><pub-id pub-id-type="pmid">19773567</pub-id></element-citation></ref><ref id="CR18"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Kogan</surname><given-names>JR</given-names></name><name><surname>Conforti</surname><given-names>LN</given-names></name><name><surname>Bernabeo</surname><given-names>EC</given-names></name><name><surname>Durning</surname><given-names>SJ</given-names></name><name><surname>Hauer</surname><given-names>KE</given-names></name><name><surname>Holmboe</surname><given-names>ES</given-names></name></person-group><article-title>Faculty staff perceptions of feedback to residents after direct observation of clinical skills</article-title><source>Med Educ</source><year>2012</year><volume>46</volume><issue>2</issue><fpage>201</fpage><lpage>215</lpage><pub-id pub-id-type="doi">10.1111/j.1365-2923.2011.04137.x</pub-id><pub-id pub-id-type="pmid">22239334</pub-id></element-citation></ref><ref id="CR19"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Menon</surname><given-names>S</given-names></name><name><surname>Winston</surname><given-names>M</given-names></name><name><surname>Sullivan</surname><given-names>G</given-names></name></person-group><article-title>Workplace-based assessment: survey of psychiatric trainees in Wales</article-title><source>Psychiatr Bull</source><year>2009</year><volume>33</volume><fpage>468</fpage><lpage>474</lpage><pub-id pub-id-type="doi">10.1192/pb.bp.108.020503</pub-id></element-citation></ref><ref id="CR20"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Menon</surname><given-names>S</given-names></name><name><surname>Winston</surname><given-names>M</given-names></name><name><surname>Sullivan</surname><given-names>G</given-names></name></person-group><article-title>Workplace based assessment: attitudes and perceptions among consultant trainers and comparison with those of trainees</article-title><source>Psychiatrist</source><year>2012</year><volume>36</volume><fpage>16</fpage><lpage>24</lpage><pub-id pub-id-type="doi">10.1192/pb.bp.110.032110</pub-id></element-citation></ref><ref id="CR21"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Nicol</surname><given-names>DJ</given-names></name><name><surname>Macfarlane-Dick</surname><given-names>D</given-names></name></person-group><article-title>Formative assessment and self-regulated learning: a model and seven principles of good feedback practice</article-title><source>Stud High Educ</source><year>2006</year><volume>31</volume><issue>2</issue><fpage>199</fpage><lpage>218</lpage><pub-id pub-id-type="doi">10.1080/03075070600572090</pub-id></element-citation></ref><ref id="CR22"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Norcini</surname><given-names>JJ</given-names></name><name><surname>Blank</surname><given-names>LL</given-names></name><name><surname>Arnold</surname><given-names>GK</given-names></name><name><surname>Kimball</surname><given-names>HR</given-names></name></person-group><article-title>The Mini-CEX (clinical evaluation exercise): a preliminary investigation</article-title><source>Ann Intern Med</source><year>1995</year><volume>123</volume><issue>10</issue><fpage>795</fpage><lpage>799</lpage><pub-id pub-id-type="doi">10.7326/0003-4819-123-10-199511150-00008</pub-id><pub-id pub-id-type="pmid">7574198</pub-id></element-citation></ref><ref id="CR23"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>St-Onge</surname><given-names>C</given-names></name><name><surname>Young</surname><given-names>M</given-names></name></person-group><article-title>Evolving conceptualisations of validity: impact on the process and outcome of assessment</article-title><source>Med Educ</source><year>2015</year><volume>49</volume><issue>6</issue><fpage>548</fpage><lpage>550</lpage><pub-id pub-id-type="doi">10.1111/medu.12734</pub-id><pub-id pub-id-type="pmid">25989399</pub-id></element-citation></ref><ref id="CR24"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Swayamprakasam</surname><given-names>AP</given-names></name><name><surname>Segaran</surname><given-names>A</given-names></name><name><surname>Allery</surname><given-names>L</given-names></name></person-group><article-title>Work-based assessments: making the transition from participation to engagement</article-title><source>JRSM Open</source><year>2014</year></element-citation></ref><ref id="CR25"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Vandenbroucke</surname><given-names>JP</given-names></name><name><surname>Ev</surname><given-names>Elm</given-names></name><name><surname>Altman</surname><given-names>DG</given-names></name><name><surname>Gøtzsche</surname><given-names>PC</given-names></name><name><surname>Mulrow</surname><given-names>CD</given-names></name><name><surname>Pocock</surname><given-names>SJ</given-names></name><etal></etal></person-group><article-title>Strengthening the reporting of observational studies in epidemiology (STROBE): explanation and elaboration</article-title><source>Ann Intern Med</source><year>2007</year><volume>147</volume><issue>8</issue><fpage>W-163</fpage><pub-id pub-id-type="doi">10.7326/0003-4819-147-8-200710160-00010-w1</pub-id><pub-id pub-id-type="pmid">17938389</pub-id></element-citation></ref><ref id="CR26"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>von Elm</surname><given-names>E</given-names></name><name><surname>Altman</surname><given-names>DG</given-names></name><name><surname>Egger</surname><given-names>M</given-names></name><name><surname>Pocock</surname><given-names>SJ</given-names></name><name><surname>Gøtzsche</surname><given-names>PC</given-names></name><name><surname>Vandenbroucke</surname><given-names>JP</given-names></name></person-group><article-title>The strengthening the reporting of observational studies in epidemiology (STROBE) statement: guidelines for reporting observational studies</article-title><source>Prev Med</source><year>2007</year><volume>45</volume><issue>4</issue><fpage>247</fpage><lpage>251</lpage><pub-id pub-id-type="doi">10.1016/j.ypmed.2007.08.012</pub-id><pub-id pub-id-type="pmid">17950122</pub-id></element-citation></ref><ref id="CR27"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Watling</surname><given-names>C</given-names></name></person-group><article-title>Cognition, culture, and credibility: deconstructing feedback in medical education</article-title><source>Perspect Med Educ</source><year>2014</year><volume>3</volume><issue>2</issue><fpage>124</fpage><lpage>128</lpage><pub-id pub-id-type="doi">10.1007/s40037-014-0115-2</pub-id><pub-id pub-id-type="pmid">24604762</pub-id></element-citation></ref><ref id="CR28"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Watling</surname><given-names>C</given-names></name><name><surname>Driessen</surname><given-names>E</given-names></name><name><surname>van der Vleuten</surname><given-names>CPM</given-names></name><name><surname>Vanstone</surname><given-names>M</given-names></name><name><surname>Lingard</surname><given-names>L</given-names></name></person-group><article-title>Understanding responses to feedback: the potential and limitations of regulatory focus theory</article-title><source>Med Educ</source><year>2012</year><volume>46</volume><issue>6</issue><fpage>593</fpage><lpage>603</lpage><pub-id pub-id-type="doi">10.1111/j.1365-2923.2012.04209.x</pub-id><pub-id pub-id-type="pmid">22626051</pub-id></element-citation></ref><ref id="CR29"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Watling</surname><given-names>C</given-names></name><name><surname>Driessen</surname><given-names>E</given-names></name><name><surname>van der Vleuten</surname><given-names>CPM</given-names></name><name><surname>Vanstone</surname><given-names>M</given-names></name><name><surname>Lingard</surname><given-names>L</given-names></name></person-group><article-title>Beyond individualism: professional culture and its influence on feedback</article-title><source>Med Educ</source><year>2013</year><volume>47</volume><issue>6</issue><fpage>585</fpage><lpage>594</lpage><pub-id pub-id-type="doi">10.1111/medu.12150</pub-id><pub-id pub-id-type="pmid">23662876</pub-id></element-citation></ref><ref id="CR30"><element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Watling</surname><given-names>C</given-names></name><name><surname>Driessen</surname><given-names>E</given-names></name><name><surname>van der Vleuten</surname><given-names>CPM</given-names></name><name><surname>Vanstone</surname><given-names>M</given-names></name><name><surname>Lingard</surname><given-names>L</given-names></name></person-group><article-title>Music lessons: revealing medicine’s learning culture through a comparison with that of music</article-title><source>Med Educ</source><year>2013</year><volume>47</volume><issue>8</issue><fpage>842</fpage><lpage>850</lpage><pub-id pub-id-type="doi">10.1111/medu.12235</pub-id><pub-id pub-id-type="pmid">23837431</pub-id></element-citation></ref></ref-list></back></pmc></record>Pour manipuler ce document sous Unix (Dilib)
EXPLOR_STEP=$WICRI_ROOT/Wicri/Musique/explor/MusiqueCeltiqueV1/Data/Pmc/Corpus
HfdSelect -h $EXPLOR_STEP/biblio.hfd -nk 000107 | SxmlIndent | more
Ou
HfdSelect -h $EXPLOR_AREA/Data/Pmc/Corpus/biblio.hfd -nk 000107 | SxmlIndent | more
Pour mettre un lien sur cette page dans le réseau Wicri
{{Explor lien
|wiki= Wicri/Musique
|area= MusiqueCeltiqueV1
|flux= Pmc
|étape= Corpus
|type= RBID
|clé= PMC:4761346
|texte= Profiling postgraduate workplace-based assessment implementation in Ireland: a retrospective cohort study
}}
Pour générer des pages wiki
HfdIndexSelect -h $EXPLOR_AREA/Data/Pmc/Corpus/RBID.i -Sk "pubmed:26933632" \
| HfdSelect -Kh $EXPLOR_AREA/Data/Pmc/Corpus/biblio.hfd \
| NlmPubMed2Wicri -a MusiqueCeltiqueV1
| This area was generated with Dilib version V0.6.38. |  |