Links to Exploration step
Le document en format XML
<record><TEI><teiHeader><fileDesc><titleStmt><title xml:lang="en">Which factors may determine the necessary and feasible type of effectiveness evidence? A mixed methods approach to develop an instrument to help coverage decision-makers</title><author><name sortKey="De Groot, Saskia" sort="De Groot, Saskia" uniqKey="De Groot S" first="Saskia" last="De Groot">Saskia De Groot</name><affiliation><nlm:aff id="af1"><addr-line>Department of Health Policy and Management</addr-line>,<institution>Institute for Medical Technology Assessment, Erasmus University Rotterdam</institution>,<addr-line>Rotterdam</addr-line>,<country>The Netherlands</country></nlm:aff></affiliation></author><author><name sortKey="Rijnsburger, Adriana J" sort="Rijnsburger, Adriana J" uniqKey="Rijnsburger A" first="Adriana J" last="Rijnsburger">Adriana J. Rijnsburger</name><affiliation><nlm:aff id="af1"><addr-line>Department of Health Policy and Management</addr-line>,<institution>Institute for Medical Technology Assessment, Erasmus University Rotterdam</institution>,<addr-line>Rotterdam</addr-line>,<country>The Netherlands</country></nlm:aff></affiliation></author><author><name sortKey="Versteegh, Matthijs M" sort="Versteegh, Matthijs M" uniqKey="Versteegh M" first="Matthijs M" last="Versteegh">Matthijs M. Versteegh</name><affiliation><nlm:aff id="af1"><addr-line>Department of Health Policy and Management</addr-line>,<institution>Institute for Medical Technology Assessment, Erasmus University Rotterdam</institution>,<addr-line>Rotterdam</addr-line>,<country>The Netherlands</country></nlm:aff></affiliation></author><author><name sortKey="Heymans, Juanita M" sort="Heymans, Juanita M" uniqKey="Heymans J" first="Juanita M" last="Heymans">Juanita M. Heymans</name><affiliation><nlm:aff id="af2"><institution>Dutch National Health Care Institute (ZIN) (formerly named CVZ)</institution>,<addr-line>Diemen</addr-line>,<country>The Netherlands</country></nlm:aff></affiliation></author><author><name sortKey="Kleijnen, Sarah" sort="Kleijnen, Sarah" uniqKey="Kleijnen S" first="Sarah" last="Kleijnen">Sarah Kleijnen</name><affiliation><nlm:aff id="af2"><institution>Dutch National Health Care Institute (ZIN) (formerly named CVZ)</institution>,<addr-line>Diemen</addr-line>,<country>The Netherlands</country></nlm:aff></affiliation></author><author><name sortKey="Redekop, W Ken" sort="Redekop, W Ken" uniqKey="Redekop W" first="W Ken" last="Redekop">W Ken Redekop</name><affiliation><nlm:aff id="af1"><addr-line>Department of Health Policy and Management</addr-line>,<institution>Institute for Medical Technology Assessment, Erasmus University Rotterdam</institution>,<addr-line>Rotterdam</addr-line>,<country>The Netherlands</country></nlm:aff></affiliation></author><author><name sortKey="Verstijnen, Ilse M" sort="Verstijnen, Ilse M" uniqKey="Verstijnen I" first="Ilse M" last="Verstijnen">Ilse M. Verstijnen</name><affiliation><nlm:aff id="af2"><institution>Dutch National Health Care Institute (ZIN) (formerly named CVZ)</institution>,<addr-line>Diemen</addr-line>,<country>The Netherlands</country></nlm:aff></affiliation></author></titleStmt><publicationStmt><idno type="wicri:source">PMC</idno><idno type="pmid">26220869</idno><idno type="pmc">4521513</idno><idno type="url">http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4521513</idno><idno type="RBID">PMC:4521513</idno><idno type="doi">10.1136/bmjopen-2014-007241</idno><date when="2015">2015</date><idno type="wicri:Area/Pmc/Corpus">000256</idno><idno type="wicri:explorRef" wicri:stream="Pmc" wicri:step="Corpus" wicri:corpus="PMC">000256</idno></publicationStmt><sourceDesc><biblStruct><analytic><title xml:lang="en" level="a" type="main">Which factors may determine the necessary and feasible type of effectiveness evidence? A mixed methods approach to develop an instrument to help coverage decision-makers</title><author><name sortKey="De Groot, Saskia" sort="De Groot, Saskia" uniqKey="De Groot S" first="Saskia" last="De Groot">Saskia De Groot</name><affiliation><nlm:aff id="af1"><addr-line>Department of Health Policy and Management</addr-line>,<institution>Institute for Medical Technology Assessment, Erasmus University Rotterdam</institution>,<addr-line>Rotterdam</addr-line>,<country>The Netherlands</country></nlm:aff></affiliation></author><author><name sortKey="Rijnsburger, Adriana J" sort="Rijnsburger, Adriana J" uniqKey="Rijnsburger A" first="Adriana J" last="Rijnsburger">Adriana J. Rijnsburger</name><affiliation><nlm:aff id="af1"><addr-line>Department of Health Policy and Management</addr-line>,<institution>Institute for Medical Technology Assessment, Erasmus University Rotterdam</institution>,<addr-line>Rotterdam</addr-line>,<country>The Netherlands</country></nlm:aff></affiliation></author><author><name sortKey="Versteegh, Matthijs M" sort="Versteegh, Matthijs M" uniqKey="Versteegh M" first="Matthijs M" last="Versteegh">Matthijs M. Versteegh</name><affiliation><nlm:aff id="af1"><addr-line>Department of Health Policy and Management</addr-line>,<institution>Institute for Medical Technology Assessment, Erasmus University Rotterdam</institution>,<addr-line>Rotterdam</addr-line>,<country>The Netherlands</country></nlm:aff></affiliation></author><author><name sortKey="Heymans, Juanita M" sort="Heymans, Juanita M" uniqKey="Heymans J" first="Juanita M" last="Heymans">Juanita M. Heymans</name><affiliation><nlm:aff id="af2"><institution>Dutch National Health Care Institute (ZIN) (formerly named CVZ)</institution>,<addr-line>Diemen</addr-line>,<country>The Netherlands</country></nlm:aff></affiliation></author><author><name sortKey="Kleijnen, Sarah" sort="Kleijnen, Sarah" uniqKey="Kleijnen S" first="Sarah" last="Kleijnen">Sarah Kleijnen</name><affiliation><nlm:aff id="af2"><institution>Dutch National Health Care Institute (ZIN) (formerly named CVZ)</institution>,<addr-line>Diemen</addr-line>,<country>The Netherlands</country></nlm:aff></affiliation></author><author><name sortKey="Redekop, W Ken" sort="Redekop, W Ken" uniqKey="Redekop W" first="W Ken" last="Redekop">W Ken Redekop</name><affiliation><nlm:aff id="af1"><addr-line>Department of Health Policy and Management</addr-line>,<institution>Institute for Medical Technology Assessment, Erasmus University Rotterdam</institution>,<addr-line>Rotterdam</addr-line>,<country>The Netherlands</country></nlm:aff></affiliation></author><author><name sortKey="Verstijnen, Ilse M" sort="Verstijnen, Ilse M" uniqKey="Verstijnen I" first="Ilse M" last="Verstijnen">Ilse M. Verstijnen</name><affiliation><nlm:aff id="af2"><institution>Dutch National Health Care Institute (ZIN) (formerly named CVZ)</institution>,<addr-line>Diemen</addr-line>,<country>The Netherlands</country></nlm:aff></affiliation></author></analytic><series><title level="j">BMJ Open</title><idno type="eISSN">2044-6055</idno><imprint><date when="2015">2015</date></imprint></series></biblStruct></sourceDesc></fileDesc><profileDesc><textClass></textClass></profileDesc></teiHeader><front><div type="abstract" xml:lang="en"><sec><title>Objectives</title><p>Reimbursement decisions require evidence of effectiveness and, in general, a blinded randomised controlled trial (RCT) is the preferred study design to provide it. However, there are situations where a cohort study, or even patient series, can be deemed acceptable. The aim of this study was to develop an instrument that first examines which study characteristics of a blinded RCT are necessary, and then, if particular characteristics are considered necessary, examines whether these characteristics are feasible.</p></sec><sec><title>Design</title><p>We retrospectively studied 22 interventions from 20 reimbursement reports concerning medical specialist care made by the Dutch National Health Care Institute (ZIN) to identify any factors that influenced the necessity and feasibility of blinded RCTs, and their constituent study characteristics, that is, blinding, randomisation and a control group. A literature review was performed to identify additional factors. Additional expertise was included by interviewing eight experts in epidemiology, medicine and ethics. The resulting instrument was called the FIT instrument (Feasible Information Trajectory), and was prospectively validated using three consecutive reimbursement reports.</p></sec><sec><title>Results</title><p>(Blinded) RCT evidence was lacking in 5 of 11 positive reimbursement decisions and 3 of 11 negative decisions. In the reimbursement reports, we found no empirical evidence supporting situations where a blinded RCT is unnecessary. The literature also revealed few arguments against the necessity of a blinded RCT. In contrast, many factors influencing the feasibility of randomisation, a control group and blinding, were found in the reimbursement reports and the literature; for example, when a patient population is too small or when an intervention is common practice, randomisation will be hindered.</p></sec><sec><title>Conclusions</title><p>Policy regarding the necessity and feasibility of different types of evidence of effectiveness would benefit from systematic guidance. The FIT instrument has the potential to support transparent, reproducible and well-founded decisions on appropriate evidence of effectiveness in medical specialist care.</p></sec></div></front><back><div1 type="bibliography"><listBibl><biblStruct></biblStruct><biblStruct><analytic><author><name sortKey="Sackett, Dl" uniqKey="Sackett D">DL Sackett</name></author><author><name sortKey="Rosenberg, Wm" uniqKey="Rosenberg W">WM Rosenberg</name></author><author><name sortKey="Gray, Ja" uniqKey="Gray J">JA Gray</name></author></analytic></biblStruct><biblStruct></biblStruct><biblStruct><analytic><author><name sortKey="Fischer, Ke" uniqKey="Fischer K">KE Fischer</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Black, N" uniqKey="Black N">N Black</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Rawlins, M" uniqKey="Rawlins M">M Rawlins</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Guyatt, Gh" uniqKey="Guyatt G">GH Guyatt</name></author><author><name sortKey="Oxman, Ad" uniqKey="Oxman A">AD Oxman</name></author><author><name sortKey="Vist, Ge" uniqKey="Vist G">GE Vist</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Heymans, Jm" uniqKey="Heymans J">JM Heymans</name></author><author><name sortKey="Kleijnen, S" uniqKey="Kleijnen S">S Kleijnen</name></author><author><name sortKey="Verstijnen, Im" uniqKey="Verstijnen I">IM Verstijnen</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Bowling, A" uniqKey="Bowling A">A Bowling</name></author></analytic></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct></biblStruct><biblStruct><analytic><author><name sortKey="Manchikanti, L" uniqKey="Manchikanti L">L Manchikanti</name></author><author><name sortKey="Hirsch, Ja" uniqKey="Hirsch J">JA Hirsch</name></author><author><name sortKey="Smith, Hs" uniqKey="Smith H">HS Smith</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Bekkering, Ge" uniqKey="Bekkering G">GE Bekkering</name></author><author><name sortKey="Kleijnen, J" uniqKey="Kleijnen J">J Kleijnen</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Kunz, R" uniqKey="Kunz R">R Kunz</name></author><author><name sortKey="Oxman, Ad" uniqKey="Oxman A">AD Oxman</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Devereaux, Pj" uniqKey="Devereaux P">PJ Devereaux</name></author><author><name sortKey="Yusuf, S" uniqKey="Yusuf S">S Yusuf</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Kirtane, Aj" uniqKey="Kirtane A">AJ Kirtane</name></author><author><name sortKey="Gupta, A" uniqKey="Gupta A">A Gupta</name></author><author><name sortKey="Iyengar, S" uniqKey="Iyengar S">S Iyengar</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Buchbinder, R" uniqKey="Buchbinder R">R Buchbinder</name></author><author><name sortKey="Osborne, Rh" uniqKey="Osborne R">RH Osborne</name></author><author><name sortKey="Ebeling, Pr" uniqKey="Ebeling P">PR Ebeling</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Higgins, Jpt" uniqKey="Higgins J">JPT Higgins</name></author><author><name sortKey="Altman, Dg" uniqKey="Altman D">DG Altman</name></author><author><name sortKey="Sterne, Jac" uniqKey="Sterne J">JAC Sterne</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Hill, Ab" uniqKey="Hill A">AB Hill</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Mcculloch, P" uniqKey="Mcculloch P">P McCulloch</name></author><author><name sortKey="Taylor, I" uniqKey="Taylor I">I Taylor</name></author><author><name sortKey="Sasako, M" uniqKey="Sasako M">M Sasako</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Freedman, B" uniqKey="Freedman B">B Freedman</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Sege, Rd" uniqKey="Sege R">RD Sege</name></author><author><name sortKey="De Vos, E" uniqKey="De Vos E">E De Vos</name></author></analytic></biblStruct><biblStruct></biblStruct><biblStruct><analytic><author><name sortKey="Khoury, Mj" uniqKey="Khoury M">MJ Khoury</name></author><author><name sortKey="Coates, Rj" uniqKey="Coates R">RJ Coates</name></author><author><name sortKey="Evans, Jp" uniqKey="Evans J">JP Evans</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Ergina, Pl" uniqKey="Ergina P">PL Ergina</name></author><author><name sortKey="Cook, Ja" uniqKey="Cook J">JA Cook</name></author><author><name sortKey="Blazeby, Jm" uniqKey="Blazeby J">JM Blazeby</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Craig, P" uniqKey="Craig P">P Craig</name></author><author><name sortKey="Dieppe, P" uniqKey="Dieppe P">P Dieppe</name></author><author><name sortKey="Macintyre, S" uniqKey="Macintyre S">S Macintyre</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Campbell, M" uniqKey="Campbell M">M Campbell</name></author><author><name sortKey="Fitzpatrick, R" uniqKey="Fitzpatrick R">R Fitzpatrick</name></author><author><name sortKey="Haines, A" uniqKey="Haines A">A Haines</name></author></analytic></biblStruct><biblStruct></biblStruct><biblStruct><analytic><author><name sortKey="Guyatt, Gh" uniqKey="Guyatt G">GH Guyatt</name></author><author><name sortKey="Oxman, Ad" uniqKey="Oxman A">AD Oxman</name></author><author><name sortKey="Kunz, R" uniqKey="Kunz R">R Kunz</name></author></analytic></biblStruct><biblStruct></biblStruct><biblStruct><analytic><author><name sortKey="Van Loon, J" uniqKey="Van Loon J">J van Loon</name></author><author><name sortKey="Grutters, J" uniqKey="Grutters J">J Grutters</name></author><author><name sortKey="Macbeth, F" uniqKey="Macbeth F">F Macbeth</name></author></analytic></biblStruct><biblStruct><analytic><author><name sortKey="Clement, Fm" uniqKey="Clement F">FM Clement</name></author><author><name sortKey="Harris, A" uniqKey="Harris A">A Harris</name></author><author><name sortKey="Li, Jj" uniqKey="Li J">JJ Li</name></author></analytic></biblStruct></listBibl></div1></back></TEI><pmc article-type="research-article"><pmc-dir>properties open_access</pmc-dir>
<front><journal-meta><journal-id journal-id-type="nlm-ta">BMJ Open</journal-id><journal-id journal-id-type="iso-abbrev">BMJ Open</journal-id><journal-id journal-id-type="hwp">bmjopen</journal-id><journal-id journal-id-type="publisher-id">bmjopen</journal-id><journal-title-group><journal-title>BMJ Open</journal-title></journal-title-group><issn pub-type="epub">2044-6055</issn><publisher><publisher-name>BMJ Publishing Group</publisher-name><publisher-loc>BMA House, Tavistock Square, London, WC1H 9JR</publisher-loc></publisher></journal-meta><article-meta><article-id pub-id-type="pmid">26220869</article-id><article-id pub-id-type="pmc">4521513</article-id><article-id pub-id-type="publisher-id">bmjopen-2014-007241</article-id><article-id pub-id-type="doi">10.1136/bmjopen-2014-007241</article-id><article-categories><subj-group subj-group-type="heading"><subject>Health Policy</subject><subj-group><subject>Research</subject></subj-group></subj-group><subj-group subj-group-type="hwp-journal-coll"><subject>1506</subject><subject>1703</subject><subject>1704</subject></subj-group></article-categories><title-group><article-title>Which factors may determine the necessary and feasible type of effectiveness evidence? A mixed methods approach to develop an instrument to help coverage decision-makers</article-title></title-group><contrib-group><contrib contrib-type="author"><name><surname>de Groot</surname><given-names>Saskia</given-names></name><xref ref-type="aff" rid="af1">1</xref></contrib><contrib contrib-type="author"><name><surname>Rijnsburger</surname><given-names>Adriana J</given-names></name><xref ref-type="aff" rid="af1">1</xref></contrib><contrib contrib-type="author"><name><surname>Versteegh</surname><given-names>Matthijs M</given-names></name><xref ref-type="aff" rid="af1">1</xref></contrib><contrib contrib-type="author"><name><surname>Heymans</surname><given-names>Juanita M</given-names></name><xref ref-type="aff" rid="af2">2</xref></contrib><contrib contrib-type="author"><name><surname>Kleijnen</surname><given-names>Sarah</given-names></name><xref ref-type="aff" rid="af2">2</xref></contrib><contrib contrib-type="author"><name><surname>Redekop</surname><given-names>W Ken</given-names></name><xref ref-type="aff" rid="af1">1</xref></contrib><contrib contrib-type="author"><name><surname>Verstijnen</surname><given-names>Ilse M</given-names></name><xref ref-type="aff" rid="af2">2</xref></contrib></contrib-group><aff id="af1"><label>1</label><addr-line>Department of Health Policy and Management</addr-line>,<institution>Institute for Medical Technology Assessment, Erasmus University Rotterdam</institution>,<addr-line>Rotterdam</addr-line>,<country>The Netherlands</country></aff><aff id="af2"><label>2</label><institution>Dutch National Health Care Institute (ZIN) (formerly named CVZ)</institution>,<addr-line>Diemen</addr-line>,<country>The Netherlands</country></aff><author-notes><corresp><label>Correspondence to</label> Saskia de Groot; <email>s.degroot@bmg.eur.nl</email></corresp></author-notes><pub-date pub-type="collection"><year>2015</year></pub-date><pub-date pub-type="epub"><day>28</day><month>7</month><year>2015</year></pub-date><volume>5</volume><issue>7</issue><elocation-id>e007241</elocation-id><history><date date-type="received"><day>18</day><month>11</month><year>2014</year></date><date date-type="rev-recd"><day>2</day><month>4</month><year>2015</year></date><date date-type="accepted"><day>12</day><month>4</month><year>2015</year></date></history><permissions><copyright-statement>Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://group.bmj.com/group/rights-licensing/permissions</copyright-statement><copyright-year>2015</copyright-year><license license-type="open-access"><license-p>This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: <ext-link ext-link-type="uri" xlink:href="http://creativecommons.org/licenses/by-nc/4.0/">http://creativecommons.org/licenses/by-nc/4.0/</ext-link></license-p></license></permissions><self-uri xlink:title="pdf" xlink:href="bmjopen-2014-007241.pdf"></self-uri><abstract><sec><title>Objectives</title><p>Reimbursement decisions require evidence of effectiveness and, in general, a blinded randomised controlled trial (RCT) is the preferred study design to provide it. However, there are situations where a cohort study, or even patient series, can be deemed acceptable. The aim of this study was to develop an instrument that first examines which study characteristics of a blinded RCT are necessary, and then, if particular characteristics are considered necessary, examines whether these characteristics are feasible.</p></sec><sec><title>Design</title><p>We retrospectively studied 22 interventions from 20 reimbursement reports concerning medical specialist care made by the Dutch National Health Care Institute (ZIN) to identify any factors that influenced the necessity and feasibility of blinded RCTs, and their constituent study characteristics, that is, blinding, randomisation and a control group. A literature review was performed to identify additional factors. Additional expertise was included by interviewing eight experts in epidemiology, medicine and ethics. The resulting instrument was called the FIT instrument (Feasible Information Trajectory), and was prospectively validated using three consecutive reimbursement reports.</p></sec><sec><title>Results</title><p>(Blinded) RCT evidence was lacking in 5 of 11 positive reimbursement decisions and 3 of 11 negative decisions. In the reimbursement reports, we found no empirical evidence supporting situations where a blinded RCT is unnecessary. The literature also revealed few arguments against the necessity of a blinded RCT. In contrast, many factors influencing the feasibility of randomisation, a control group and blinding, were found in the reimbursement reports and the literature; for example, when a patient population is too small or when an intervention is common practice, randomisation will be hindered.</p></sec><sec><title>Conclusions</title><p>Policy regarding the necessity and feasibility of different types of evidence of effectiveness would benefit from systematic guidance. The FIT instrument has the potential to support transparent, reproducible and well-founded decisions on appropriate evidence of effectiveness in medical specialist care.</p></sec></abstract><kwd-group><kwd>Health policy < HEALTH SERVICES ADMINISTRATION & MANAGEMENT</kwd><kwd>Decision making</kwd><kwd>Reimbursement</kwd><kwd>Evidence based medicine</kwd><kwd>Randomised controlled trial</kwd></kwd-group></article-meta></front><body><boxed-text position="float" orientation="portrait"><caption><title>Strengths and limitations of this study</title></caption><list list-type="bullet"><list-item><p>In this study, multiple sources were used, including 20 reimbursement reports made by the Dutch National Health Care Institute, along with literature and expert opinion.</p></list-item><list-item><p>Since most items used to examine which study characteristics of a blinded randomised controlled trial remain necessary and feasible are based on general epidemiological principles, results of this study might also be useful for reimbursement agencies in countries other than the Netherlands.</p></list-item><list-item><p>Not all possible study characteristics are taken into account; the study was limited to the necessity and feasibility of randomisation, a control group and blinding.</p></list-item><list-item><p>The instrument's completeness requires further testing, as does its impact on the decision-making process in terms of efficiency, reliability (such as inter/intraobserver reliability) and additional forms of validity.</p></list-item></list></boxed-text><sec sec-type="intro" id="s1"><title>Introduction</title><p>Reimbursement decisions require evidence of effectiveness. To demonstrate effectiveness, a blinded randomised controlled trial (RCT) is, in general, the preferred study design. The central reimbursement authority in the Netherlands, the Dutch National Health Care Institute (ZIN), formerly named Health Care Insurance Board (CVZ), applies the principles of evidence-based medicine (EBM) to determine whether care is effective.<xref rid="R1" ref-type="bibr">1</xref> EBM is “the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients”.<xref rid="R2" ref-type="bibr">2</xref> It integrates the best available external evidence with individual clinical expertise and patient preferences. In 2015, ZIN will update its framework for decision-making, and formally integrate the ‘Grading of Recommendations Assessment, Development and Evaluation’ (GRADE) system to assess and grade available evidence.<xref rid="R3" ref-type="bibr">3</xref></p><p>Although blinded RCTs are often preferred, they are not always available, and reimbursement decisions have to be taken despite suboptimal evidence. A systematic review by Fischer<xref rid="R4" ref-type="bibr">4</xref> showed that, indeed, the presence of suboptimal evidence plays an important role in coverage decision-making. Whereas evidence from RCTs is absolutely necessary to overcome problems such as confounding by indication, it has been stressed that, in some situations, an RCT may be unnecessary, inappropriate, impossible or inadequate.<xref rid="R5" ref-type="bibr">5</xref> For example, in situations where successful interventions for otherwise fatal conditions become available, an RCT is not required.<xref rid="R2" ref-type="bibr">2</xref> Observational studies may provide valuable information, in particular on rare or long-term harms,<xref rid="R6" ref-type="bibr">6</xref> but also on the benefits of an intervention.</p><p>The assessment of the quality of available external evidence has received much scientific attention resulting in elaborate rating methods such as the GRADE system.<xref rid="R7" ref-type="bibr">7</xref> An assessment of the necessity and feasibility of attaining certain evidence requirements has, by contrast, received less attention. The aim of this study was to develop an instrument that first examines which study characteristics of a blinded RCT are necessary, and then, if particular characteristics are considered necessary, examines whether these characteristics are feasible.<xref rid="R8" ref-type="bibr">8</xref> Together, the necessary and feasible characteristics define the optimal study design. A blinded RCT is, by consequence, necessary and feasible when all three characteristics (blinding, randomisation and a control group) are considered necessary and feasible. If they are not, a different study design may be considered optimal.</p><p>We focused on evidence of effectiveness for non-pharmaceutical, therapeutic medical specialist care as a starting point. Although the instrument is based on this type of care, it may be applicable to other types of interventions.</p></sec><sec sec-type="methods" id="s2"><title>Methods</title><p>Three information sources were used to identify factors influencing the necessity and feasibility of different types of evidence of effectiveness. These sources are specified below.</p><sec id="s2a"><title>Review of reimbursement reports</title><p>All medical specialist care reports published by ZIN between 1 January 2007 and 31 December 2010 were collected. Reports were available on all reimbursement decisions. 1 January 2007 was chosen as a starting point, since from this date onwards, ZIN officially applied the principles of EBM to determine whether care is effective. As data extraction was performed in 2011, 2010 was the last year for which complete reports were available. A stratified sample was implemented in order to prevent ending up with a sample only including reimbursement reports for which the necessity and feasibility of blinded RCTs was evident. The reimbursement reports were therefore classified into three groups based on their level of complexity, namely, simple (few necessity or feasibility issues), intermediate (moderate necessity or feasibility issues) or complex (complex necessity or feasibility issues), by the three researchers from ZIN (JH, SK, IV). Subsequently, a random sample of each group was drawn (simple N=5, intermediate N=7 and complex N=8). The resulting sample concerned 22 medical specialist care interventions (two reports evaluated two interventions). Then, data from these reports were extracted in a uniform matter on the following items: the intervention, the patient group, the reimbursement decision (negative or positive), the available evidence of effectiveness, and any arguments that influenced the necessity and feasibility of different types of evidence of effectiveness. A negative or positive reimbursement decision implies whether the intervention was considered suitable for reimbursement from the basic health insurance package in the Netherlands. These decisions did not always rely on the assessment of effectiveness only, but may take additional reimbursement criteria into account.</p></sec><sec id="s2b"><title>Literature review</title><p>A literature review in Medline (1996 to March 2011) was performed to identify publications that discussed factors influencing the necessity and feasibility of different types of evidence. The Ovid search interface was used. The search strategy included the following keyword combinations: level? or degree? or criteri$ or hierarch$ or require$ or assess$ or standard$ and evidence-based practice. The resulting publications were screened systematically. Selection on relevance was carried out by two researchers (SdG, AR). First, titles and abstracts were screened; publications that might discuss any factors influencing the necessity and feasibility of different types of evidence of effectiveness were included. The full texts of the selected publications were then examined; publications that did not discuss any factors influencing the necessity and feasibility of different types of evidence of effectiveness were excluded, as were publications not in English and publications not available as full text. The reference lists of relevant publications were screened by hand to identify any additional publications for inclusion. Special attention was given to those factors affecting the constituent study characteristics of blinded RCTs; namely, randomisation, a control group and blinding, because there are factors that may influence one of the characteristics but not the other. Each of these characteristics by itself is meant to reduce bias, hence the more these characteristics are preserved (up to a full-blown blinded RCT) the less biased the research outcomes will be. Obviously, less biased outcomes form better grounds for decision-making.</p></sec><sec id="s2c"><title>Interviews</title><p>A draft instrument was drawn based on the factors influencing the necessity and feasibility of different types of evidence of effectiveness found in the reimbursement reports and in the literature. Subsequently, additional expertise was included by interviewing eight experts in epidemiology, medicine and ethics. All participants were informed about the purpose and content of the interview and agreed to participate by email. Interviews were semistructured, that is, questions were predefined, but interviewees were allowed to raise other relevant issues not covered by the interview schedule.<xref rid="R9" ref-type="bibr">9</xref> Seven of eight interviews were performed face-to-face, and one was by telephone. The interviews were intended to elicit views and opinions from respondents on the factors included in the draft instrument, and to identify additional factors.</p><p><xref ref-type="fig" rid="BMJOPEN2014007241F1">Figure 1</xref> shows the steps of the research process, including its aims.</p><fig id="BMJOPEN2014007241F1" orientation="portrait" position="float"><label>Figure 1</label><caption><p>Steps of the research process including its aims.</p></caption><graphic xlink:href="bmjopen2014007241f01"></graphic></fig></sec><sec id="s2d"><title>Definitions of necessity and feasibility</title><p>The resulting instrument was called the FIT instrument (Feasible Information Trajectory). The factors included were categorised in two groups: The first group deals with the necessity of randomisation, a control group and blinding, relating to situations in which one or more of these three characteristic are not required. The second group deals with the feasibility of randomisation, a control group and blinding, and was subdivided into two groups (2A and 2B). Group 2A refers to factors that, stand alone, are sufficiently strong to deviate from randomisation, a control group and/or blinding. Group 2B refers to factors that, by themselves, are insufficiently strong, but may jointly provide a compelling case to do so.</p></sec><sec id="s2e"><title>Prospective validation</title><p>As a last step, the FIT instrument was prospectively validated by four ZIN decision-makers who were not involved in the study. They each applied the instrument on one of three consecutive reimbursement reports regarding medical specialist care which included immunotherapy for high-risk neuroblastoma, zygomatic implants for the atrophic edentulous maxilla and endovascular treatment of complex aortic aneurysms. The instrument's face validity was discussed in a joint meeting with the project team. The main question in this respect was whether the instrument contained all facets to identify the necessity and feasibility of the constituent study characteristics of blinded RCTs. Therefore, the relevance and clarity of the factors included in the instrument, as well as the completeness and user-friendliness of the instrument, were discussed. Instructions about the validation were sent to the decision-makers beforehand. These instructions explained the aim of the validation, and pointed out particular questions to the decision-makers, such as ‘Do all questions apply?’, ‘Do you miss any questions?’ and ‘Are the questions clearly stated?’. Suggestions for improvement of the instrument were applied if everyone (the decision-makers and the project team) agreed.</p><p>The final FIT instrument is programmed in Excel and requires the user to answer questions regarding the factors that influence the necessity and feasibility of randomisation, a control group and blinding.</p><p>Since this article does not contain any personal medical information about an identifiable living individual, ethics approval was not required.</p></sec></sec><sec sec-type="results" id="s3"><title>Results</title><p>Eleven of 22 reimbursement decisions were negative, that is, the intervention was not considered suitable for reimbursement from the basic health insurance package in the Netherlands, and the remaining 11 decisions were positive. In 5 of 11 positive reimbursement decisions, evidence from (blinded) RCTs was lacking. Arguments that contributed to the acceptance of the available evidence in these cases are presented in <xref ref-type="table" rid="BMJOPEN2014007241TB1">table 1</xref>. The arguments listed in the table are the only arguments explicitly mentioned in the reimbursement reports.</p><table-wrap id="BMJOPEN2014007241TB1" orientation="portrait" position="float"><label>Table 1</label><caption><p>Positive reimbursement decisions in reports with suboptimal evidence of effectiveness; arguments that advocated for the acceptance of the available suboptimal evidence</p></caption><table frame="hsides" rules="groups"><colgroup span="1"><col align="left" span="1"></col><col align="left" span="1"></col></colgroup><thead valign="bottom"><tr><th align="left" rowspan="1" colspan="1">Intervention</th><th align="left" rowspan="1" colspan="1">Arguments that advocated for the acceptance of the available suboptimal evidence‡</th></tr></thead><tbody><tr><td rowspan="1" colspan="1">I. Metal on metal (MoM) resurfacing arthroplasty of the hip for patients with primary or secondary osteoarthritis*<xref rid="R10" ref-type="bibr">10</xref></td><td rowspan="1" colspan="1"><list list-type="bullet"><list-item><p><bold>Technique has already been used</bold> for approximately 10 years</p></list-item><list-item><p><bold>Patients will probably not participate in a randomised study</bold> given the available data</p></list-item><list-item><p><bold>Availability of lower level evidence of effectiveness</bold> (comparative studies, short to medium term)</p></list-item></list></td></tr><tr><td rowspan="1" colspan="1">II. Preimplantation genetic diagnosis (PGD) to predict β-thalassaemia in second child with human leucocyte antigen (HLA) typing during PGD for stem cell transplantation<xref rid="R11" ref-type="bibr">11</xref></td><td rowspan="1" colspan="1"><list list-type="bullet"><list-item><p><bold>PGD as a technique has been proven effective</bold> in predicting other diseases</p></list-item><list-item><p><bold>It is unfeasible to perform comparative studies</bold> on the use of PGD to predict β-thalassaemia</p></list-item><list-item><p><bold>It is unfeasible to perform comparative studies</bold> of HLA typing</p></list-item></list></td></tr><tr><td rowspan="1" colspan="1">III. Potassium-titanyl-phosphate (KTP) laser treatment for patients with lower urinary tract symptoms (LUTS) caused by benign prostatic hyperplasia†<xref rid="R12" ref-type="bibr">12</xref></td><td rowspan="1" colspan="1"><list list-type="bullet"><list-item><p><bold>(Double)-blinding is unfeasible</bold> in comparing KTP laser treatment with transurethral resection of the prostate (TURP) for patients with a mildly enlarged prostate (surgical intervention)</p></list-item><list-item><p><bold>Technique is already frequently used</bold></p></list-item><list-item><p><bold>Patients will probably not participate in a randomised study</bold></p></list-item><list-item><p><bold>No alternative treatment</bold> if drugs provide no adequate relief (for high-risk patients or patients being treated with anticoagulants)</p></list-item><list-item><p><bold>Availability of lower level evidence of effectiveness and safety</bold> (only non-comparative studies for high-risk patients)</p></list-item></list></td></tr><tr><td rowspan="1" colspan="1">IV. Transcatheter pulmonary valve implantation (TPVI) for patients with an abnormal pulmonary valve due to a congenital heart defect<xref rid="R13" ref-type="bibr">13</xref></td><td rowspan="1" colspan="1"><list list-type="bullet"><list-item><p><bold>(Double)-blinding is unfeasible</bold> in comparing TPVI with surgical pulmonary valve replacement</p></list-item><list-item><p><bold>Rare condition;</bold> only 30 patients are eligible for percutaneous pulmonary valve implantation in the Netherlands per year</p></list-item><list-item><p><bold>Safety concerns seem limited</bold> regarding percutaneous pulmonary valve implantation, particularly in comparison to open cardiac surgery</p></list-item><list-item><p><bold>Availability of lower level evidence of effectiveness and safety,</bold> case series showed good short-term success rates</p></list-item></list></td></tr><tr><td rowspan="1" colspan="1">V. Proton therapy for patients with intraocular tumours, chordomas and chondrosarcomas, and paediatric tumours<xref rid="R14" ref-type="bibr">14</xref></td><td rowspan="1" colspan="1"><list list-type="bullet"><list-item><p><bold>Side effects are rare and mostly happen (many) years later,</bold> and the purpose of the therapy, in particular, is to reduce or prevent late side effects (instead of to prove effectiveness)</p></list-item><list-item><p><bold>Rare conditions</bold></p></list-item><list-item><p><bold>Technique is already frequently used</bold> and there is international consensus between radiation therapists and oncologists</p></list-item><list-item><p><bold>Availability of lower level evidence of effectiveness</bold> (mainly case series)</p></list-item></list></td></tr></tbody></table><table-wrap-foot><fn><p>*This intervention complied with medical science and medical practice, according to the Dutch National Health Care Institute (ZIN), for patients with primary or secondary osteoarthritis who failed on conservative treatment, or patients younger than 65 years of age with a sufficient level of activity. However, in January 2012, this advice was changed in line with the advice of the Netherlands Orthopaedic Association, based on new national and international published experiences, to not place large-head MoM hip implants and MoM resurfacing implants.</p></fn><fn><p>†This intervention complied with medical science and medical practice, according to ZIN, for patients with a mildly enlarged prostate, high-risk patients or patients being treated with anticoagulants.</p></fn><fn><p>‡Essential arguments are shown in bold.</p></fn></table-wrap-foot></table-wrap><p>Of the 11 negative reimbursement decisions, evidence from (blinded) RCTs was lacking in three of them. Besides suboptimal evidence of effectiveness, these three decisions revealed arguments against the acceptance of the available evidence, such as poor quality of the available evidence and limited evidence on long-term safety (<xref ref-type="table" rid="BMJOPEN2014007241TB2">table 2</xref>). The arguments listed in the table are the only arguments explicitly mentioned in the reimbursement reports. Apparently, there were no decisive arguments to accept suboptimal evidence.</p><table-wrap id="BMJOPEN2014007241TB2" orientation="portrait" position="float"><label>Table 2</label><caption><p>Negative reimbursement decisions in reports with suboptimal evidence of effectiveness; arguments that advocated against the acceptance of the available evidence</p></caption><table frame="hsides" rules="groups"><colgroup span="1"><col align="left" span="1"></col><col align="left" span="1"></col></colgroup><thead valign="bottom"><tr><th align="left" rowspan="1" colspan="1">Intervention</th><th align="left" rowspan="1" colspan="1">Arguments that advocated against the acceptance of the available evidence†</th></tr></thead><tbody><tr><td rowspan="1" colspan="1">VI. Transcatheter aortic valve implantation (TAVI) for patients with aortic valve stenosis*<xref rid="R13" ref-type="bibr">13</xref></td><td rowspan="1" colspan="1"><list list-type="bullet"><list-item><p><bold>Substantial intervention-related mortality</bold>, probably (partly) related to a priori risk profile of study population</p></list-item><list-item><p><bold>No comparative studies available</bold> and the quality of the available evidence is poor (ie, limited duration and limited size)</p></list-item><list-item><p><bold>Randomised controlled trial</bold> (<bold>RCT) is on-going</bold></p></list-item></list></td></tr><tr><td rowspan="1" colspan="1">VII. Psychoanalysis<xref rid="R15" ref-type="bibr">15</xref></td><td rowspan="1" colspan="1"><list list-type="bullet"><list-item><p><bold>Insufficient studies (qualitatively adequate) on the effectiveness available</bold></p></list-item><list-item><p>Existence of several comparative effectiveness studies on long-term psychoanalytic psychotherapy makes it likely that such studies are also possible for psychoanalysis</p></list-item></list></td></tr><tr><td rowspan="1" colspan="1">VIII. Breast augmentation with autologous lipofilling<xref rid="R16" ref-type="bibr">16</xref></td><td rowspan="1" colspan="1"><list list-type="bullet"><list-item><p><bold>Insufficient studies on the effectiveness available</bold> (majority are case reports and non-comparative studies)</p></list-item><list-item><p><bold>No consensus</bold> on breast augmentation with autologous lipofilling (unclear whether possible microcalcifications influence the assessment of mammograms)</p></list-item></list></td></tr></tbody></table><table-wrap-foot><fn><p>*In October 2011, this advice was changed. In the opinion of the Dutch National Health Care Institute (ZIN), TAVI is care that complies with established medical science and medical practice for insured persons with severe stenosis of the aortic valve and for whom the surgical risks are unacceptably high. TAVI belongs in the insured basic package for these insured persons.</p></fn><fn><p>†Essential arguments are shown in bold.</p></fn></table-wrap-foot></table-wrap><p>In some reimbursement decisions, additional reimbursement criteria besides effectiveness were taken into account. These criteria could have influenced the reimbursement decision, but are not included in <xref ref-type="table" rid="BMJOPEN2014007241TB2">table 2</xref>.</p><p>Evidence from (blinded) RCTs was available in 14 of 22 reimbursement decisions (six positive and eight negative). Negative decisions were mostly motivated by the insufficient quality of the RCT, a lack of long-term evidence, and absence of consensus within the literature or the profession on the effectiveness and safety of the intervention. Again, besides effectiveness, other reimbursement criteria could have played a role.</p><p>The literature search in Medline identified 98 publications. Seventy-eight publications were discarded based on their title or abstract. Additionally, three publications were excluded, because no full text was available. Another three publications were excluded, because they were not written in English. The full texts of the remaining 14 publications were examined. Six publications were excluded since they did not provide any factors influencing the necessity and feasibility of different types of evidence of effectiveness. The resulting eight publications revealed overlapping, but also showed factors in addition to those found in the reimbursement reports.</p><p><xref ref-type="table" rid="BMJOPEN2014007241TB3">Table 3</xref> comprises the final FIT instrument, and summarises factors influencing the necessity and feasibility of the constituent study characteristics of blinded RCTs, derived from the reimbursement reports, the literature review and the interviews.</p><table-wrap id="BMJOPEN2014007241TB3" orientation="portrait" position="float"><label>Table 3</label><caption><p>Final FIT instrument summarising factors influencing the necessity and feasibility of randomisation, a control group and blinding</p></caption><table frame="hsides" rules="groups"><colgroup span="1"><col align="left" span="1"></col><col align="left" span="1"></col><col align="left" span="1"></col><col align="left" span="1"></col><col align="left" span="1"></col><col align="left" span="1"></col><col align="left" span="1"></col></colgroup><thead valign="bottom"><tr><th rowspan="1" colspan="1"></th><th align="left" rowspan="1" colspan="1">Randomisation?</th><th align="left" rowspan="1" colspan="1">Control group?</th><th align="left" rowspan="1" colspan="1">Blinding of outcome assessors?</th><th align="left" rowspan="1" colspan="1">Blinding of patients and physicians?</th><th align="left" rowspan="1" colspan="1">Reference to reimbursement reports</th><th align="left" rowspan="1" colspan="1">Reference to the literature</th></tr></thead><tbody><tr><td colspan="7" rowspan="1"><bold>1. When is randomisation, a control group or blinding necessary?</bold></td></tr><tr><td rowspan="1" colspan="1"> ▸ Confounding by indication</td><td rowspan="1" colspan="1">+</td><td rowspan="1" colspan="1">+</td><td rowspan="1" colspan="1"></td><td rowspan="1" colspan="1"></td><td rowspan="1" colspan="1"></td><td rowspan="1" colspan="1"><xref rid="R17" ref-type="bibr">17–20</xref></td></tr><tr><td rowspan="1" colspan="1"> ▸ Natural decrease of symptoms over time</td><td rowspan="1" colspan="1"></td><td rowspan="1" colspan="1">+</td><td rowspan="1" colspan="1"></td><td rowspan="1" colspan="1"></td><td rowspan="1" colspan="1"><xref rid="R35" ref-type="bibr">35</xref></td><td rowspan="1" colspan="1"><xref rid="R22" ref-type="bibr">22</xref></td></tr><tr><td rowspan="1" colspan="1"> ▸ Subjective outcome measures</td><td rowspan="1" colspan="1"></td><td rowspan="1" colspan="1">+</td><td rowspan="1" colspan="1">+</td><td rowspan="1" colspan="1"></td><td rowspan="1" colspan="1"></td><td rowspan="1" colspan="1"><xref rid="R17" ref-type="bibr">17</xref><xref rid="R20" ref-type="bibr">20</xref><xref rid="R23" ref-type="bibr">23</xref></td></tr><tr><td rowspan="1" colspan="1"> ▸ Differential behaviours across intervention groups</td><td rowspan="1" colspan="1"></td><td rowspan="1" colspan="1"></td><td rowspan="1" colspan="1"></td><td rowspan="1" colspan="1">+</td><td rowspan="1" colspan="1"></td><td rowspan="1" colspan="1"><xref rid="R17" ref-type="bibr">17</xref><xref rid="R20" ref-type="bibr">20</xref><xref rid="R23" ref-type="bibr">23</xref></td></tr><tr><td rowspan="1" colspan="1"> ▸ Dramatic/immediate effects</td><td rowspan="1" colspan="1">−</td><td rowspan="1" colspan="1">−</td><td rowspan="1" colspan="1"></td><td rowspan="1" colspan="1"></td><td rowspan="1" colspan="1"></td><td rowspan="1" colspan="1"><xref rid="R17" ref-type="bibr">17</xref><xref rid="R19" ref-type="bibr">19</xref><xref rid="R20" ref-type="bibr">20</xref></td></tr><tr><td colspan="7" rowspan="1"><bold>2A. When is randomisation, a control group or blinding hard to achieve?</bold></td></tr><tr><td rowspan="1" colspan="1"> ▸ Lack of equipoise, that is, consensus about the preferred treatment</td><td rowspan="1" colspan="1">−</td><td rowspan="1" colspan="1"></td><td rowspan="1" colspan="1"></td><td rowspan="1" colspan="1"></td><td rowspan="1" colspan="1">Report I, III<xref rid="R10" ref-type="bibr">10</xref><xref rid="R12" ref-type="bibr">12</xref></td><td rowspan="1" colspan="1"><xref rid="R25" ref-type="bibr">25</xref><xref rid="R26" ref-type="bibr">26</xref></td></tr><tr><td rowspan="1" colspan="1"> ▸ Outcomes occur in the distant future</td><td rowspan="1" colspan="1">−</td><td rowspan="1" colspan="1"></td><td rowspan="1" colspan="1"></td><td rowspan="1" colspan="1">−</td><td rowspan="1" colspan="1"></td><td rowspan="1" colspan="1"><xref rid="R17" ref-type="bibr">17</xref><xref rid="R18" ref-type="bibr">18</xref><xref rid="R22" ref-type="bibr">22</xref><xref rid="R27" ref-type="bibr">27</xref></td></tr><tr><td rowspan="1" colspan="1"> ▸ Small adaptation of an intervention that has already been proven effective and reimbursed</td><td rowspan="1" colspan="1">−</td><td rowspan="1" colspan="1">−</td><td rowspan="1" colspan="1"></td><td rowspan="1" colspan="1"></td><td rowspan="1" colspan="1"></td><td rowspan="1" colspan="1"><xref rid="R25" ref-type="bibr">25</xref></td></tr><tr><td rowspan="1" colspan="1"> ▸ Extension of the indication area of a procedure that has already been proven effective and reimbursed</td><td rowspan="1" colspan="1">−</td><td rowspan="1" colspan="1">−</td><td rowspan="1" colspan="1"></td><td rowspan="1" colspan="1"></td><td rowspan="1" colspan="1">Report II<xref rid="R11" ref-type="bibr">11</xref></td><td rowspan="1" colspan="1"></td></tr><tr><td colspan="7" rowspan="1"><bold>2B. Which factors hinder the feasibility of randomisation, a control group or blinding?</bold></td></tr><tr><td rowspan="1" colspan="1"> ▸ Small patient population</td><td rowspan="1" colspan="1">−</td><td rowspan="1" colspan="1"></td><td rowspan="1" colspan="1"></td><td rowspan="1" colspan="1"></td><td rowspan="1" colspan="1">Report IV, V<xref rid="R13" ref-type="bibr">13</xref><xref rid="R14" ref-type="bibr">14</xref></td><td rowspan="1" colspan="1"><xref rid="R17" ref-type="bibr">17</xref><xref rid="R25" ref-type="bibr">25</xref></td></tr><tr><td rowspan="1" colspan="1"> ▸ Poor prognosis/no alternative treatment</td><td rowspan="1" colspan="1">−</td><td rowspan="1" colspan="1">−</td><td rowspan="1" colspan="1"></td><td rowspan="1" colspan="1"></td><td rowspan="1" colspan="1">Report III<xref rid="R12" ref-type="bibr">12</xref></td><td rowspan="1" colspan="1"><xref rid="R29" ref-type="bibr">29</xref></td></tr><tr><td rowspan="1" colspan="1"> ▸ Intervention is common practice</td><td rowspan="1" colspan="1">−</td><td rowspan="1" colspan="1"></td><td rowspan="1" colspan="1"></td><td rowspan="1" colspan="1"></td><td rowspan="1" colspan="1">Report I, III, V<xref rid="R10" ref-type="bibr">10</xref><xref rid="R12" ref-type="bibr">12</xref><xref rid="R14" ref-type="bibr">14</xref></td><td rowspan="1" colspan="1"><xref rid="R25" ref-type="bibr">25</xref><xref rid="R29" ref-type="bibr">29</xref></td></tr><tr><td rowspan="1" colspan="1"> ▸ Urgency of the intervention</td><td rowspan="1" colspan="1">−</td><td rowspan="1" colspan="1"></td><td rowspan="1" colspan="1"></td><td rowspan="1" colspan="1">−</td><td rowspan="1" colspan="1"></td><td rowspan="1" colspan="1"><xref rid="R25" ref-type="bibr">25</xref></td></tr><tr><td rowspan="1" colspan="1"> ▸ Complexity of the intervention</td><td rowspan="1" colspan="1">−</td><td rowspan="1" colspan="1"></td><td rowspan="1" colspan="1"></td><td rowspan="1" colspan="1">−</td><td rowspan="1" colspan="1"><xref rid="R33" ref-type="bibr">33</xref></td><td rowspan="1" colspan="1"><xref rid="R27" ref-type="bibr">27</xref></td></tr><tr><td rowspan="1" colspan="1"> ▸ Availability of good quality low level evidence of effectiveness</td><td rowspan="1" colspan="1">−</td><td rowspan="1" colspan="1">−</td><td rowspan="1" colspan="1">−</td><td rowspan="1" colspan="1">−</td><td rowspan="1" colspan="1">Report I, III, IV, V<xref rid="R10" ref-type="bibr">10</xref><xref rid="R12" ref-type="bibr">12–14</xref></td><td rowspan="1" colspan="1"></td></tr></tbody></table><table-wrap-foot><fn><p>A plus sign indicates that randomisation, a control group or blinding is necessary. A minus sign indicates that randomisation, a control group or blinding is unnecessary, is hard to achieve or is hindered.</p></fn><fn><p>FIT, Feasible Information Trajectory.</p></fn></table-wrap-foot></table-wrap><sec id="s3a"><title>When is randomisation, a control group or blinding necessary?</title><p>In RCTs, patients are allocated randomly to one of the interventions being studied, to minimise imbalances in confounding variables between experimental and control groups.<xref rid="R17" ref-type="bibr">17–20</xref> The merits of a random allocation procedure are particularly important if, for example, confounding by indication is likely. This happens when the indication for an intervention is related to the health outcome and, as a result, the effect estimate is distorted because it is mixed with the effect of a confounding variable. A systematic review and meta-analysis of drug-eluting stents versus bare metal stents showed that the rates of death and myocardial infarction were found to be significantly reduced with the use of drug-eluting stents in the observational studies with an attenuated effect in the RCTs.<xref rid="R21" ref-type="bibr">21</xref> One of the explanations for this reduction, as mentioned by the authors, was the non-randomised choice of either drug-eluting stents or bare metal stents. The decision to use drug-eluting stents may be based on unmeasured patient characteristics and may importantly affect subsequent treatment decisions, including medication use.</p><p>A control group for the intervention of interest can involve placebo, active treatment, no treatment, a different dose or regimen of the study treatment or external or historical controls.<xref rid="R17" ref-type="bibr">17</xref> A control group is particularly needed if the disease has a favourable natural history, for example, when complaints naturally diminish over time. Uncontrolled or poorly controlled studies tend to overestimate the treatment effect, because they do not take into account this favourable natural history of the disease.<xref rid="R22" ref-type="bibr">22</xref></p><p>Blinding of study participants, health providers and investigators reduces the risk that knowledge of which intervention was received, rather than the intervention itself, affects outcomes.<xref rid="R23" ref-type="bibr">23</xref> Blinding of outcome assessors combined with a control group is particularly needed if the primary outcome measure is subjective, such as quality of life, instead of objective, such as recurrence from disease or even death.<xref rid="R17" ref-type="bibr">17</xref><xref rid="R20" ref-type="bibr">20</xref><xref rid="R23" ref-type="bibr">23</xref> Blinding of patients and physicians prevents them, for example, from differential drop-out or differential administration of cointerventions, thereby eliminating possible effects of differential behaviours across intervention groups.<xref rid="R17" ref-type="bibr">17</xref><xref rid="R20" ref-type="bibr">20</xref><xref rid="R23" ref-type="bibr">23</xref></p><p>There may, however, be conditions where randomisation and a control group are not specifically needed, for example, in case of a ‘dramatic’ or ‘immediate’ effect, since it is less likely that the effect will be explained by (unknown) confounders only.<xref rid="R17" ref-type="bibr">17</xref><xref rid="R19" ref-type="bibr">19</xref><xref rid="R20" ref-type="bibr">20</xref> Insulin supplementation in diabetic patients is an example in which the effect rapidly follows the intervention. A ‘dramatic’ or ‘immediate’ effect in combination with a plausible relationship between the pathophysiology of the condition and the mechanism of the intervention may legitimise suboptimal evidence. Besides plausibility, additional factors contribute to our belief that the relationship between an intervention and its effect is indeed causal, such as consistency (across research settings), temporality (effect follows treatment) and biological gradient (dose–response relation).<xref rid="R24" ref-type="bibr">24</xref></p></sec><sec id="s3b"><title>When is randomisation, a control group or blinding hard to achieve?</title><p>First, randomisation may be hard to achieve if equipoise is lacking,<xref rid="R25" ref-type="bibr">25</xref> that is, if there is consensus about the preferred treatment. In this case, patients and clinicians would both be unwilling to participate in a randomised study. Furthermore, ethicists claim that it would be unethical to randomise patients.<xref rid="R26" ref-type="bibr">26</xref></p><p>Second, if outcomes occur in the distant future and intermediate outcome measures are absent, an extended follow-up is needed, which complicates randomisation.<xref rid="R17" ref-type="bibr">17</xref><xref rid="R18" ref-type="bibr">18</xref><xref rid="R22" ref-type="bibr">22</xref><xref rid="R27" ref-type="bibr">27</xref> Also, blinding of patients and physicians will be problematic, since eventually, for example, physicians may need to know what treatment the patient has received in deciding on future treatment.</p><p>In addition, in case of a small adaptation of an intervention that has already been proven effective and reimbursed, randomisation and a control group may also be hard to achieve. Surgical techniques generally progress through a number of successive, small modifications.<xref rid="R25" ref-type="bibr">25</xref> Strictly speaking, evidence from RCTs is still required to demonstrate effectiveness unless there is a minimal chance that the treatment effect will be poorer than the effect previously demonstrated. If evidence from RCTs is required, there are reasons why randomisation and a control group may be hard to achieve. First, if there is an imbalance between the amount of effort needed to provide this evidence and the degree of reduction in uncertainty about the effectiveness of the intervention, conducting an RCT will not be very rational. Second, if RCTs were needed to evaluate each small modification, progress would be slowed.<xref rid="R25" ref-type="bibr">25</xref> Third, excluding half of the population from the new intervention in order to reduce a small degree of uncertainty about the treatment effect might not be ethical. If the reimbursement assessment involves an extension of the indication area of a procedure that has already been proven effective and reimbursed for a smaller indication, this may also legitimise the use of suboptimal evidence.<xref rid="R11" ref-type="bibr">11</xref></p><p>Lastly, characteristics of the intervention could make blinding of the outcome assessors as well as blinding of patients and physicians impossible. For example, with most surgical interventions it is impossible to blind the surgeon. Also, blinding of patients may be impossible, as was seen in the reimbursement report regarding sacral neurostimulation in patients with faecal incontinence. Since this device sends mild electrical pulses to the sacral nerves that control the bowel and rectum, patients will feel the treatment they receive, making blinding impossible.<xref rid="R28" ref-type="bibr">28</xref> When blinding of patients and physicians is impossible, it still might be possible to blind the outcome assessors, especially if the intervention is not visible.</p></sec><sec id="s3c"><title>Which factors hinder the feasibility of randomisation, a control group or blinding?</title><p>Besides factors that make randomisation, a control group or blinding unnecessary or hard to achieve, there may be several characteristics of the patient population and the intervention that hinder the feasibility of randomisation, a control group or blinding. Factors that hinder the feasibility of randomisation, a control group or blinding may by themselves be insufficiently strong, but may jointly argue for deviating from randomisation, a control group or blinding.</p><p>First, a small patient population limits the feasibility of an RCT with sufficient power to detect a real effect as statistically significant.<xref rid="R17" ref-type="bibr">17</xref><xref rid="R25" ref-type="bibr">25</xref> This was suggested in the reimbursement report on the transcatheter pulmonary valve implantation and in the reimbursement report on proton therapy for patients with intraocular tumours, chordomas and chondrosarcomas and paediatric tumours.<xref rid="R13" ref-type="bibr">13</xref><xref rid="R14" ref-type="bibr">14</xref> While international trials are performed to obtain sufficient power in orphan drug research, performing international trials for non-pharmaceutical medical specialist care may be complicated, especially when treatments, such as surgical procedures, vary across borders.</p><p>Second, a poor prognosis in combination with withholding the only possibly effective treatment could hinder the feasibility of randomisation and a control group, and could therefore legitimatise the acceptance of suboptimal evidence.<xref rid="R29" ref-type="bibr">29</xref> If treatment is urgent and patients experience a significant burden of the disease while the new intervention provides the only chance on improvement, one could argue that it would be unethical to randomise patients to either doing nothing or the intervention. This was proposed in appraising the effectiveness of KTP laser treatment for high-risk patients with benign prostatic hyperplasia, since drugs do not always provide adequate relief for these patients.<xref rid="R12" ref-type="bibr">12</xref></p><p>Third, an intervention may be common practice without the availability of evidence from blinded RCTs.<xref rid="R25" ref-type="bibr">25</xref><xref rid="R29" ref-type="bibr">29</xref> These interventions are frequently applied in clinical practice and sometimes even included in clinical guidelines. Performing an RCT would be considered as ‘outdated’, and randomisation would therefore be hindered. Heart, liver, kidney and lung transplantations are examples of interventions that are not validated with RCTs.<xref rid="R30" ref-type="bibr">30</xref> This argument was also mentioned in the reimbursement reports on metal on metal resurfacing arthroplasty, treatment with KTP laser and proton therapy.<xref rid="R10" ref-type="bibr">10</xref><xref rid="R12" ref-type="bibr">12</xref><xref rid="R14" ref-type="bibr">14</xref></p><p>Fourth, the urgency of an intervention, that is, time urgency, could hinder randomisation and obtaining informed consent.<xref rid="R25" ref-type="bibr">25</xref> Fifth, the feasibility of an RCT may be limited because of the complexity of interventions.<xref rid="R27" ref-type="bibr">27</xref> Complex interventions are generally described as interventions that contain several interacting components, but according to Craig <italic>et al</italic>,<xref rid="R31" ref-type="bibr">31</xref> complex interventions have some additional characteristics including the “number and difficulty of behaviours required by those delivering or receiving the intervention, the number of groups or organisational levels targeted by the intervention, the number and variability of outcomes and the degree of flexibility or tailoring of the intervention permitted”. The evaluation of a specialist stroke unit is an example, since such an evaluation would have to consider the expertise of various health professionals as well as investigations, drugs, treatment guidelines, and arrangements for discharge and follow-up. Furthermore, the organisation, management and skill mix of stroke units differ.<xref rid="R32" ref-type="bibr">32</xref> In the reimbursement report on rheopheresis, complexity was put forward to argue that blinding became unfeasible.<xref rid="R33" ref-type="bibr">33</xref></p><p>Lastly, the availability of good quality low level evidence may limit the feasibility of randomisation, a control group and blinding, since this influences both patient and clinical equipoise. Moreover, if this evidence is methodologically strong, and shows large and consistent results, we may be confident about the results,<xref rid="R34" ref-type="bibr">34</xref> and the necessity of randomisation, a control group and blinding may also be limited.</p></sec><sec id="s3d"><title>Prospective validation</title><p>During the prospective validation, four ZIN decision-makers applied the FIT instrument to assess three new interventions. In assessing immunotherapy for high-risk neuroblastoma, the FIT instrument revealed that blinding of outcome assessors, patients and physicians would be hard to achieve as a result of the characteristics of the intervention, including its intensity. Although some factors that hinder randomisation or a control group were identified, such as a small patient population and a poor prognosis, randomisation and a control group were still considered necessary and feasible. Blinding of outcome assessors, patients and physicians also appeared hard to achieve as a result of the characteristics of the second intervention (zygomatic implants for the atrophic edentulous maxilla). Although the FIT instrument revealed some factors that hinder randomisation and/or a control group, such as a small patient population and complexity of the intervention, randomisation and a control group were still considered necessary and feasible. In assessing endovascular treatment of complex aortic aneurysms, a small patient population and a poor prognosis, combined with the absence of an alternative treatment for some of the patients, were identified by the FIT instrument as factors that hinder randomisation and a control group.</p><p>In general, the ZIN decision-makers recognised the factors included in the instrument and they all found the instrument useful in deciding what can be considered the appropriate evidence of effectiveness. Based on discussions with the decision-makers, two factors hindering the feasibility of randomisation, a control group or blinding were excluded: lack of funding and lack of a good research infrastructure (sufficient expertise, facilities and materials). The reason for these exclusions was that these factors have an organisational nature, whereas all other factors relate to characteristics of the patient population or the intervention. None of the decision-makers contributed additional factors, the instrument therefore appeared complete.</p></sec></sec><sec sec-type="discussion" id="s4"><title>Discussion</title><p>As reimbursement agencies are under pressure to make rapid and well-founded decisions on reimbursement, there is a need for systematic guidance on what can be considered the appropriate evidence of effectiveness. We therefore developed an instrument that first examines which study characteristics of a blinded RCT are necessary, and then, if particular characteristics are considered necessary, examines whether these characteristics are feasible. As shown in <xref ref-type="table" rid="BMJOPEN2014007241TB3">table 3</xref>, in the reimbursement reports we found no empirical evidence supporting situations where a blinded RCT is unnecessary. The literature revealed only one situation in which a blinded RCT was not required, where, in case of a ‘dramatic’ or ‘immediate’ effect, randomisation and a control group become unnecessary. In contrast, many factors influencing the feasibility of randomisation, a control group and blinding were identified in both the literature and the reimbursement reports. One could therefore argue that a blinded RCT is almost always necessary, but not always feasible.</p><p>Recently, Van Loon <italic>et al</italic><xref rid="R36" ref-type="bibr">36</xref> considered five arguments that limit the feasibility of RCTs in the evaluation of novel radiotherapy technologies, namely, rare indications, narrow inclusion criteria, end points that require data for late toxic effects or second malignant disease, limited funding and a strong belief in effectiveness of the novel technique. Additionally, the authors proposed guidelines for prospective comparative cohort studies when RCTs are not feasible. The FIT instrument reveals which study characteristics of a blinded RCT are necessary and feasible resulting in a richer research palette of study designs that go beyond prospective comparative cohort studies. For example, differentiation between controlled observational studies, such as cohort studies, and observational studies without a control group, is possible, since the instrument distinguishes between factors affecting the necessity and feasibility of a control group, and those affecting the necessity and feasibility of randomisation and blinding. Also, research with a historic control group may still be possible when, for some reason, randomisation proved unfeasible.</p><p>This study has some limitations. First, in the context of decision-making, the fundamental question is ‘what to decide’ given the particular context of an intervention including the existing evidence. The FIT instrument does not address this question, but addresses a vital question that needs to be answered beforehand, that is, the FIT instrument examines which study characteristics of a blinded RCT are necessary, and then, whether these characteristics are feasible. Thereafter, the appropriate evidence of effectiveness can be compared with the actual available evidence. If the available evidence lacks one or more of the study characteristics that were deemed necessary, the decision-maker may advise conditional reimbursement, the condition being that further evidence has to be assembled after reimbursement. However, the decision-maker may decide <italic>not</italic> to advise conditional reimbursement if the explanations for the existing lack of necessary study characteristics provided by the FIT instrument are considered legitimate. In this latter situation, the decision-maker will assume that no better-fitting evidence will probably appear in the future, and that a decision based on the available evidence will have to be made. Next to reasons for a suboptimal evidence base, additional factors are considered important in reimbursement decisions, such as disease severity and budget impact. These latter criteria concern equity in a given society as well as overall capacity for reimbursement, and fell beyond the scope of our research.</p><p>The second limitation concerns the instrument's completeness. Various disciplines have been active in this research area, making a complete overview complicated. Where clinical equipoise originates from ethics, the size of the patient population is an epidemiological argument to consider suboptimal evidence. Clinical explanations, such as the urgency of an intervention, are also included in the instrument. Furthermore, there are practical reasons to deviate from randomisation, a control group or blinding. Therefore, we do not claim to have developed a definitive list of factors to assess the necessity and feasibility of randomisation, a control group and blinding. Nevertheless, we have tested the instrument on three consecutive reimbursement reports and found, besides its usefulness, that these reports did not identify any additional arguments. However, we recommend testing the instrument in future reimbursement decisions and adding any new argument wherever necessary.</p><p>Third, not only does the instrument's completeness require further testing, but so does its impact on the decision-making process in terms of efficiency, reliability (such as inter/intraobserver reliability) and additional forms of validity. The current validation phase was too short to properly address these issues. However, regardless of its impact on decisions, the instrument provides a degree of transparency in defining the appropriate evidence for an intervention, and this is in itself an important improvement over the status quo. The FIT instrument is frequently used by ZIN decision-makers since it was made available, which will lead to further modifications. ZIN intends to examine the FIT instrument's validity using the results of its application, and also intends to compare the results of its application to the available evidence for the interventions they have assessed using the FIT instrument.</p><p>Fourth, the FIT instrument focuses on the study characteristics of a blinded RCT, while an RCT has its disadvantages, such as limited generalisability and limited follow-up. Whereas efficacy might be more important for market authorisation to address causality, effectiveness might be more important from a decision-making perspective, which is the focus of this study. The optimal study design depends on the outcome of interest. If the outcome of interest appears to be a safety outcome, a blinded RCT may no longer be the optimal study design; the FIT instrument will show that randomisation is hard to achieve since ‘outcomes occur in the distant future’. Therefore, the FIT instrument should be used for all outcomes of interest. Moreover, study design is not the only criterion to assess the quality of available evidence. This was pointed out by Guyatt <italic>et al</italic>,<xref rid="R7" ref-type="bibr">7</xref> who developed the GRADE system, which first qualifies evidence derived from RCTs as high quality evidence while observational studies are qualified as low quality evidence. Confidence in the estimate of effect is subsequently graded downwards and upwards based on criteria such as study limitations and inconsistency of results. For example, observational studies can be graded upwards if there is evidence of a dose–response relationship. While the GRADE approach focuses on assessing the quality of available evidence, we identified factors that outline which types of evidence can, in principle, be available, thereby providing arguments for reimbursement decisions when the available evidence does not match the evidence that is considered necessary and feasible.</p><p>Fifth, findings of this study were partly based on Dutch reimbursement reports, whereas reimbursement decisions vary across countries, since they are influenced by context, agency process, ability to engage in price negotiation and, perhaps, differences in social values.<xref rid="R37" ref-type="bibr">37</xref> Since most items in the FIT instrument are based on general epidemiological principles, the FIT instrument might be useful for reimbursement agencies in other countries as well.</p><p>Decision-makers strive to attain the most optimal evidence in the assessment of effectiveness, which mostly is a blinded RCT. There are, however, situations where an RCT is not specifically needed, for example, in case of a ‘dramatic’ or ‘immediate’ effect. Furthermore, if an RCT is needed, as a consequence of ethical as well as real world practical barriers, combined with the desire to efficiently allocate research funds, the most optimal evidence might still not be an RCT. Policy regarding the necessity and feasibility of different types of evidence of effectiveness would benefit from systematic guidance. Although the instrument needs further refinement, and although a critical appraisal of the factors influencing the necessity and feasibility of blinded RCTs and its constituent study characteristics remains essential, this instrument has the potential to support transparent, reproducible and well-founded decisions on appropriate evidence of effectiveness in medical specialist care.</p></sec></body><back><ack><p>This work was financially supported by the Dutch National Health Care Institute. The authors would like to thank Bart Boksteijn, Joke Derksen, Angèl Link and Paula Staal for their cooperation in developing the FIT instrument, as well as to thank the four staff members who participated in the prospective validation of the instrument. Furthermore, the authors would like to thank the interviewees who gave their thoughts on the complexities of suboptimal evidence in reimbursement decisions. The authors are also grateful to Bert Boer for his critical comments on preliminary versions of this manuscript.</p></ack><fn-group><fn><p><bold>Contributors:</bold> SdG collected the data (reimbursement reports, literature review and interviews), analysed the data, and drafted and revised the paper. She is guarantor. AR, JH and KR monitored data collection. All the authors helped in interpreting the findings, and writing and revising the draft paper.</p></fn><fn><p><bold>Funding:</bold> Dutch National Health Care Institute.</p></fn><fn><p><bold>Competing interests:</bold> SdG, AR, MV and KR received financial support from the Dutch National Health Care Institute for the submitted work.</p></fn><fn><p><bold>Provenance and peer review:</bold> Not commissioned; externally peer reviewed.</p></fn><fn><p><bold>Data sharing statement:</bold> No additional data are available.</p></fn></fn-group><ref-list><title>References</title><ref id="R1"><label>1</label><mixed-citation publication-type="book"><collab>Dutch National Health Care Institute (ZIN) (formerly named CVZ)</collab>. <source>Assessment current medical science and practice [in Dutch: Beoordeling stand van de wetenschap en praktijk]</source>. <publisher-loc>Diemen</publisher-loc>, <year>2007</year><comment>Report No: 27071300</comment>.</mixed-citation></ref><ref id="R2"><label>2</label><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Sackett</surname><given-names>DL</given-names></name>, <name><surname>Rosenberg</surname><given-names>WM</given-names></name>, <name><surname>Gray</surname><given-names>JA</given-names></name><etal></etal></person-group><article-title>Evidence based medicine: what it is and what it isn't</article-title>. <source>BMJ</source><year>1996</year>;<volume>312</volume>:<fpage>71</fpage>–<lpage>2</lpage>. <ext-link ext-link-type="uri" xlink:href="http://dx.doi.org/10.1136/bmj.312.7023.71">doi:10.1136/bmj.312.7023.71</ext-link><pub-id pub-id-type="pmid">8555924</pub-id></mixed-citation></ref><ref id="R3"><label>3</label><mixed-citation publication-type="book"><collab>Dutch National Health Care Institute (ZIN)</collab>. <source>Assessment current medical science and practice (updated version 2015) [in Dutch: Beoordeling stand van de wetenschap en praktijk (geactualiseerde versie 2015)]</source>. <publisher-loc>Diemen</publisher-loc>, <year>2015</year><comment>Report No: 2014116583</comment>.</mixed-citation></ref><ref id="R4"><label>4</label><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Fischer</surname><given-names>KE</given-names></name></person-group><article-title>A systematic review of coverage decision-making on health technologies-evidence from the real world</article-title>. <source>Health Policy</source><year>2012</year>;<volume>107</volume>:<fpage>218</fpage>–<lpage>30</lpage>. <ext-link ext-link-type="uri" xlink:href="http://dx.doi.org/10.1016/j.healthpol.2012.07.005">doi:10.1016/j.healthpol.2012.07.005</ext-link><pub-id pub-id-type="pmid">22867939</pub-id></mixed-citation></ref><ref id="R5"><label>5</label><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Black</surname><given-names>N</given-names></name></person-group><article-title>Why we need observational studies to evaluate the effectiveness of health care</article-title>. <source>BMJ</source><year>1996</year>;<volume>312</volume>:<fpage>1215</fpage>–<lpage>18</lpage>. <ext-link ext-link-type="uri" xlink:href="http://dx.doi.org/10.1136/bmj.312.7040.1215">doi:10.1136/bmj.312.7040.1215</ext-link><pub-id pub-id-type="pmid">8634569</pub-id></mixed-citation></ref><ref id="R6"><label>6</label><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Rawlins</surname><given-names>M</given-names></name></person-group><article-title>De testimonio: on the evidence for decisions about the use of therapeutic interventions</article-title>. <source>Lancet</source><year>2008</year>;<volume>372</volume>:<fpage>2152</fpage>–<lpage>61</lpage>. <ext-link ext-link-type="uri" xlink:href="http://dx.doi.org/10.1016/S0140-6736(08)61930-3">doi:10.1016/S0140-6736(08)61930-3</ext-link><pub-id pub-id-type="pmid">19101391</pub-id></mixed-citation></ref><ref id="R7"><label>7</label><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Guyatt</surname><given-names>GH</given-names></name>, <name><surname>Oxman</surname><given-names>AD</given-names></name>, <name><surname>Vist</surname><given-names>GE</given-names></name><etal></etal></person-group><article-title>GRADE: an emerging consensus on rating quality of evidence and strength of recommendations</article-title>. <source>BMJ</source><year>2008</year>;<volume>336</volume>:<fpage>924</fpage>–<lpage>6</lpage>. <ext-link ext-link-type="uri" xlink:href="http://dx.doi.org/10.1136/bmj.39489.470347.AD">doi:10.1136/bmj.39489.470347.AD</ext-link><pub-id pub-id-type="pmid">18436948</pub-id></mixed-citation></ref><ref id="R8"><label>8</label><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Heymans</surname><given-names>JM</given-names></name>, <name><surname>Kleijnen</surname><given-names>S</given-names></name>, <name><surname>Verstijnen</surname><given-names>IM</given-names></name></person-group><article-title>‘Fitting’ evidence preferable when evaluating effectiveness of interventions</article-title>. <source>Ned Tijdschr Geneeskd</source><year>2013</year>;<volume>157</volume>:<fpage>A5479</fpage>.<pub-id pub-id-type="pmid">23575289</pub-id></mixed-citation></ref><ref id="R9"><label>9</label><mixed-citation publication-type="book"><person-group person-group-type="author"><name><surname>Bowling</surname><given-names>A</given-names></name></person-group><article-title>Data collection methods in quantitative research: questionnaires, interviews and their response rates</article-title>. In: <person-group person-group-type="editor"><name><surname>Bowling</surname><given-names>A</given-names></name></person-group>, ed. <source>Research methods in health, investigating health and health services</source>. <publisher-loc>Maidenhead, UK</publisher-loc>: <publisher-name>Open University Press</publisher-name>, <year>2002</year>:<fpage>257</fpage>–<lpage>72</lpage>.</mixed-citation></ref><ref id="R10"><label>10</label><mixed-citation publication-type="book"><collab>Dutch National Health Care Institute (ZIN) (formerly named CVZ)</collab>. <source>Report on Metal on metal (MoM) resurfacing arthroplasty of the hip [in Dutch: MoM-heupprothese]</source>. <publisher-loc>Diemen</publisher-loc>, <year>2007</year><comment>Report No: 27024808 (27052545)</comment>.</mixed-citation></ref><ref id="R11"><label>11</label><mixed-citation publication-type="book"><collab>Dutch National Health Care Institute (ZIN) (formerly named CVZ)</collab>. <source>Report on preimplantation genetic diagnosis (PGD) with HLA typing for stem cell transplantation [in Dutch: Preïmplantatie genetische diagnostiek (PGD) in combinatie met HLA-typering van IVF-embryo's ten behoeve van eventuele stamceltransplantatie]</source>. <publisher-loc>Diemen</publisher-loc>, <year>2007</year><comment>Report No: 27028502 (27070951)</comment>.</mixed-citation></ref><ref id="R12"><label>12</label><mixed-citation publication-type="book"><collab>Dutch National Health Care Institute (ZIN) (formerly named CVZ)</collab>. <source>Assessment Diagnosis Treatment Combination ‘Treatment for benign prostatic hyperplasia with KTP laser’ [in Dutch: Beoordeling DBC ‘de behandeling van benigne prostaat hyperplasie met KTP laser’]</source>. <publisher-loc>Diemen</publisher-loc>, <year>2009</year><comment>Report No: 28106169</comment>.</mixed-citation></ref><ref id="R13"><label>13</label><mixed-citation publication-type="book"><collab>Dutch National Health Care Institute (ZIN) (formerly named CVZ)</collab>. <source>Assessment Diagnosis Treatment Combination ‘Transcatheter aortic valve implantation and Transcatheter pulmonary valve implantation’ [in Dutch: Beoordeling DBC ‘transkatheter aortaklep- en pulmonalisklepimplantatie’]</source>. <publisher-loc>Diemen</publisher-loc>, <year>2009</year><comment>Report No: 29022345</comment>.</mixed-citation></ref><ref id="R14"><label>14</label><mixed-citation publication-type="book"><collab>Dutch National Health Care Institute (ZIN) (formerly named CVZ)</collab>. <source>Report on indications for proton therapy (part 1): Intraocular tumours, chordomas/chondrosarcomas, paediatric tumours [in Dutch: Indicaties voor protonentherapie (deel 1): Intra-oculaire tumoren, chordomen/ chondrosarcomen, pediatrische tumoren]</source>. <publisher-loc>Diemen</publisher-loc>, <year>2010</year><comment>Report No: 2010001046</comment>.</mixed-citation></ref><ref id="R15"><label>15</label><mixed-citation publication-type="book"><collab>Dutch National Health Care Institute (ZIN) (formerly named CVZ)</collab>. <source>Report on psychoanalysis and long-term psychoanalytic psychotherapy [in Dutch: Psychoanalyse en langdurige psychoanalytische psychotherapie]</source>. <publisher-loc>Diemen</publisher-loc>, <year>2010</year><comment>Report No: 2010036278</comment>.</mixed-citation></ref><ref id="R16"><label>16</label><mixed-citation publication-type="book"><collab>Dutch National Health Care Institute (ZIN) (formerly named CVZ)</collab>. <source>Background report on the assessment current medical science and practice: Breast augmentation with autologous lipofilling [in Dutch: Achtergrondrapportage beoordeling stand van de wetenschap en praktijk: Mamma-augmentatie met autologe lipofilling]</source>. <publisher-loc>Diemen</publisher-loc>, <year>2008</year><comment>Report No: 28048276</comment>.</mixed-citation></ref><ref id="R17"><label>17</label><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Manchikanti</surname><given-names>L</given-names></name>, <name><surname>Hirsch</surname><given-names>JA</given-names></name>, <name><surname>Smith</surname><given-names>HS</given-names></name></person-group><article-title>Evidence-based medicine, systematic reviews, and guidelines in interventional pain management: Part 2: randomized controlled trials</article-title>. <source>Pain Physician</source><year>2008</year>;<volume>11</volume>:<fpage>717</fpage>–<lpage>73</lpage>.<pub-id pub-id-type="pmid">19057624</pub-id></mixed-citation></ref><ref id="R18"><label>18</label><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Bekkering</surname><given-names>GE</given-names></name>, <name><surname>Kleijnen</surname><given-names>J</given-names></name></person-group><article-title>Procedures and methods of benefit assessments for medicines in Germany</article-title>. <source>Eur J Health Econ</source><year>2008</year>;<volume>9</volume><issue>(Suppl 1)</issue>:<fpage>5</fpage>–<lpage>29</lpage>. <ext-link ext-link-type="uri" xlink:href="http://dx.doi.org/10.1007/s10198-008-0122-5">doi:10.1007/s10198-008-0122-5</ext-link><pub-id pub-id-type="pmid">18987905</pub-id></mixed-citation></ref><ref id="R19"><label>19</label><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Kunz</surname><given-names>R</given-names></name>, <name><surname>Oxman</surname><given-names>AD</given-names></name></person-group><article-title>The unpredictability paradox: review of empirical comparisons of randomised and non-randomised clinical trials</article-title>. <source>BMJ</source><year>1998</year>;<volume>317</volume>:<fpage>1185</fpage>–<lpage>90</lpage>. <ext-link ext-link-type="uri" xlink:href="http://dx.doi.org/10.1136/bmj.317.7167.1185">doi:10.1136/bmj.317.7167.1185</ext-link><pub-id pub-id-type="pmid">9794851</pub-id></mixed-citation></ref><ref id="R20"><label>20</label><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Devereaux</surname><given-names>PJ</given-names></name>, <name><surname>Yusuf</surname><given-names>S</given-names></name></person-group><article-title>The evolution of the randomized controlled trial and its role in evidence-based decision making</article-title>. <source>J Intern Med</source><year>2003</year>;<volume>254</volume>:<fpage>105</fpage>–<lpage>13</lpage>. <ext-link ext-link-type="uri" xlink:href="http://dx.doi.org/10.1046/j.1365-2796.2003.01201.x">doi:10.1046/j.1365-2796.2003.01201.x</ext-link><pub-id pub-id-type="pmid">12859691</pub-id></mixed-citation></ref><ref id="R21"><label>21</label><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Kirtane</surname><given-names>AJ</given-names></name>, <name><surname>Gupta</surname><given-names>A</given-names></name>, <name><surname>Iyengar</surname><given-names>S</given-names></name><etal></etal></person-group><article-title>Safety and efficacy of drug-eluting and bare metal stents: comprehensive meta-analysis of randomized trials and observational studies</article-title>. <source>Circulation</source><year>2009</year>;<volume>119</volume>:<fpage>3198</fpage>–<lpage>206</lpage>. <ext-link ext-link-type="uri" xlink:href="http://dx.doi.org/10.1161/CIRCULATIONAHA.108.826479">doi:10.1161/CIRCULATIONAHA.108.826479</ext-link><pub-id pub-id-type="pmid">19528338</pub-id></mixed-citation></ref><ref id="R22"><label>22</label><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Buchbinder</surname><given-names>R</given-names></name>, <name><surname>Osborne</surname><given-names>RH</given-names></name>, <name><surname>Ebeling</surname><given-names>PR</given-names></name><etal></etal></person-group><article-title>A randomized trial of vertebroplasty for painful osteoporotic vertebral fractures</article-title>. <source>N Engl J Med</source><year>2009</year>;<volume>361</volume>:<fpage>557</fpage>–<lpage>68</lpage>. <ext-link ext-link-type="uri" xlink:href="http://dx.doi.org/10.1056/NEJMoa0900429">doi:10.1056/NEJMoa0900429</ext-link><pub-id pub-id-type="pmid">19657121</pub-id></mixed-citation></ref><ref id="R23"><label>23</label><mixed-citation publication-type="book"><person-group person-group-type="author"><name><surname>Higgins</surname><given-names>JPT</given-names></name>, <name><surname>Altman</surname><given-names>DG</given-names></name>, <name><surname>Sterne</surname><given-names>JAC</given-names></name></person-group> (eds). <comment>Chapter 8: Assessing risk of bias in included studies. In: Higgins JPT, Green S (editors). Cochrane Handbook for Systematic Reviews of Interventions. Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. <ext-link ext-link-type="uri" xlink:href="www.cochrane-handbook.org">www.cochrane-handbook.org</ext-link></comment></mixed-citation></ref><ref id="R24"><label>24</label><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Hill</surname><given-names>AB</given-names></name></person-group><article-title>The environment and disease: association or causation?</article-title><source>Proc R Soc Med</source><year>1965</year>;<volume>58</volume>:<fpage>295</fpage>–<lpage>300</lpage>.<pub-id pub-id-type="pmid">14283879</pub-id></mixed-citation></ref><ref id="R25"><label>25</label><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>McCulloch</surname><given-names>P</given-names></name>, <name><surname>Taylor</surname><given-names>I</given-names></name>, <name><surname>Sasako</surname><given-names>M</given-names></name><etal></etal></person-group><article-title>Randomised trials in surgery: problems and possible solutions</article-title>. <source>BMJ</source><year>2002</year>;<volume>324</volume>:<fpage>1448</fpage>–<lpage>51</lpage>. <ext-link ext-link-type="uri" xlink:href="http://dx.doi.org/10.1136/bmj.324.7351.1448">doi:10.1136/bmj.324.7351.1448</ext-link><pub-id pub-id-type="pmid">12065273</pub-id></mixed-citation></ref><ref id="R26"><label>26</label><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Freedman</surname><given-names>B</given-names></name></person-group><article-title>Equipoise and the ethics of clinical research</article-title>. <source>N Engl J Med</source><year>1987</year>;<volume>317</volume>:<fpage>141</fpage>–<lpage>5</lpage>. <ext-link ext-link-type="uri" xlink:href="http://dx.doi.org/10.1056/NEJM198707163170304">doi:10.1056/NEJM198707163170304</ext-link><pub-id pub-id-type="pmid">3600702</pub-id></mixed-citation></ref><ref id="R27"><label>27</label><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Sege</surname><given-names>RD</given-names></name>, <name><surname>De Vos</surname><given-names>E</given-names></name></person-group><article-title>Evidence-based health care for children: what are we missing?</article-title><source>Issue Brief (Commonw Fund)</source><year>2010</year>;<volume>85</volume>:<fpage>1</fpage>–<lpage>14</lpage>.<pub-id pub-id-type="pmid">20455296</pub-id></mixed-citation></ref><ref id="R28"><label>28</label><mixed-citation publication-type="book"><collab>Dutch National Health Care Institute (ZIN) (formerly named CVZ)</collab>. <source>Background report on the assessment current medical science and practice: Sacral neurostimulation/ neuromodulation for fecal incontinence [in Dutch: Achtergrondrapportage beoordeling stand van de wetenschap en praktijk: Sacrale neurostimulatie/ neuromodulatie bij faecale incontinentie]</source>. <publisher-loc>Diemen</publisher-loc>, <year>2008</year><comment>Report No: 28085299</comment>.</mixed-citation></ref><ref id="R29"><label>29</label><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Khoury</surname><given-names>MJ</given-names></name>, <name><surname>Coates</surname><given-names>RJ</given-names></name>, <name><surname>Evans</surname><given-names>JP</given-names></name></person-group><article-title>Evidence-based classification of recommendations on use of genomic tests in clinical practice: dealing with insufficient evidence</article-title>. <source>Genet Med</source><year>2010</year>;<volume>12</volume>:<fpage>680</fpage>–<lpage>3</lpage>. <ext-link ext-link-type="uri" xlink:href="http://dx.doi.org/10.1097/GIM.0b013e3181f9ad55">doi:10.1097/GIM.0b013e3181f9ad55</ext-link><pub-id pub-id-type="pmid">20975567</pub-id></mixed-citation></ref><ref id="R30"><label>30</label><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Ergina</surname><given-names>PL</given-names></name>, <name><surname>Cook</surname><given-names>JA</given-names></name>, <name><surname>Blazeby</surname><given-names>JM</given-names></name><etal></etal></person-group><article-title>Challenges in evaluating surgical innovation</article-title>. <source>Lancet</source><year>2009</year>;<volume>374</volume>:<fpage>1097</fpage>–<lpage>104</lpage>. <ext-link ext-link-type="uri" xlink:href="http://dx.doi.org/10.1016/S0140-6736(09)61086-2">doi:10.1016/S0140-6736(09)61086-2</ext-link><pub-id pub-id-type="pmid">19782875</pub-id></mixed-citation></ref><ref id="R31"><label>31</label><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Craig</surname><given-names>P</given-names></name>, <name><surname>Dieppe</surname><given-names>P</given-names></name>, <name><surname>Macintyre</surname><given-names>S</given-names></name><etal></etal></person-group><article-title>Developing and evaluating complex interventions: the new Medical Research Council guidance</article-title>. <source>BMJ</source><year>2008</year>;<volume>337</volume>:<fpage>a1655</fpage><ext-link ext-link-type="uri" xlink:href="http://dx.doi.org/10.1136/bmj.a1655">doi:10.1136/bmj.a1655</ext-link><pub-id pub-id-type="pmid">18824488</pub-id></mixed-citation></ref><ref id="R32"><label>32</label><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Campbell</surname><given-names>M</given-names></name>, <name><surname>Fitzpatrick</surname><given-names>R</given-names></name>, <name><surname>Haines</surname><given-names>A</given-names></name><etal></etal></person-group><article-title>Framework for design and evaluation of complex interventions to improve health</article-title>. <source>BMJ</source><year>2000</year>;<volume>321</volume>:<fpage>694</fpage>–<lpage>6</lpage>. <ext-link ext-link-type="uri" xlink:href="http://dx.doi.org/10.1136/bmj.321.7262.694">doi:10.1136/bmj.321.7262.694</ext-link><pub-id pub-id-type="pmid">10987780</pub-id></mixed-citation></ref><ref id="R33"><label>33</label><mixed-citation publication-type="book"><collab>Dutch National Health Care Institute (ZIN) (formerly named CVZ)</collab>. <source>Report on rheopheresis in the treatment of dry macular degeneration [in Dutch: Rheopherese-behandeling bij droge maculadegeneratie]</source>. <publisher-loc>Diemen</publisher-loc>, <year>2010</year><comment>Report No.: 2010081059 (2010122609)</comment>.</mixed-citation></ref><ref id="R34"><label>34</label><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Guyatt</surname><given-names>GH</given-names></name>, <name><surname>Oxman</surname><given-names>AD</given-names></name>, <name><surname>Kunz</surname><given-names>R</given-names></name><etal></etal></person-group><article-title>What is “quality of evidence” and why is it important to clinicians?</article-title><source>BMJ</source><year>2008</year>;<volume>336</volume>:<fpage>995</fpage>–<lpage>8</lpage>. <ext-link ext-link-type="uri" xlink:href="http://dx.doi.org/10.1136/bmj.39490.551019.BE">doi:10.1136/bmj.39490.551019.BE</ext-link><pub-id pub-id-type="pmid">18456631</pub-id></mixed-citation></ref><ref id="R35"><label>35</label><mixed-citation publication-type="book"><collab>Dutch National Health Care Institute (ZIN) (formerly named CVZ)</collab>. <source>Report on vertebroplasty en ballon kyphoplasty [in Dutch: Vertebroplastiek en ballonkyphoplastiek]</source>. <publisher-loc>Diemen</publisher-loc>, <year>2010</year><comment>Report No.: 29060581 (2010106817)</comment>.</mixed-citation></ref><ref id="R36"><label>36</label><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>van Loon</surname><given-names>J</given-names></name>, <name><surname>Grutters</surname><given-names>J</given-names></name>, <name><surname>Macbeth</surname><given-names>F</given-names></name></person-group><article-title>Evaluation of novel radiotherapy technologies: what evidence is needed to assess their clinical and cost effectiveness, and how should we get it?</article-title><source>Lancet Oncol</source><year>2012</year>;<volume>13</volume>:<fpage>e169</fpage>–<lpage>77</lpage>. <ext-link ext-link-type="uri" xlink:href="http://dx.doi.org/10.1016/S1470-2045(11)70379-5">doi:10.1016/S1470-2045(11)70379-5</ext-link><pub-id pub-id-type="pmid">22469127</pub-id></mixed-citation></ref><ref id="R37"><label>37</label><mixed-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Clement</surname><given-names>FM</given-names></name>, <name><surname>Harris</surname><given-names>A</given-names></name>, <name><surname>Li</surname><given-names>JJ</given-names></name><etal></etal></person-group><article-title>Using effectiveness and cost-effectiveness to make drug coverage decisions: a comparison of Britain, Australia, and Canada</article-title>. <source>JAMA</source><year>2009</year>;<volume>302</volume>:<fpage>1437</fpage>–<lpage>43</lpage>. <ext-link ext-link-type="uri" xlink:href="http://dx.doi.org/10.1001/jama.2009.1409">doi:10.1001/jama.2009.1409</ext-link><pub-id pub-id-type="pmid">19809025</pub-id></mixed-citation></ref></ref-list></back></pmc></record>Pour manipuler ce document sous Unix (Dilib)
EXPLOR_STEP=$WICRI_ROOT/Wicri/Santé/explor/EdenteV2/Data/Pmc/Corpus
HfdSelect -h $EXPLOR_STEP/biblio.hfd -nk 000256 | SxmlIndent | more
Ou
HfdSelect -h $EXPLOR_AREA/Data/Pmc/Corpus/biblio.hfd -nk 000256 | SxmlIndent | more
Pour mettre un lien sur cette page dans le réseau Wicri
{{Explor lien
|wiki= Wicri/Santé
|area= EdenteV2
|flux= Pmc
|étape= Corpus
|type= RBID
|clé=
|texte=
}}
| This area was generated with Dilib version V0.6.32. |  |