User Interaction in Semi-Automatic Segmentation of Organs at Risk: a Case Study in Radiotherapy
Identifieur interne : 003D46 ( Ncbi/Merge ); précédent : 003D45; suivant : 003D47User Interaction in Semi-Automatic Segmentation of Organs at Risk: a Case Study in Radiotherapy
Auteurs : Anjana Ramkumar [Pays-Bas] ; Jose Dolz [France] ; Hortense A. Kirisli [France] ; Sonja Adebahr [Allemagne] ; Tanja Schimek-Jasch [Allemagne] ; Ursula Nestle [Allemagne] ; Laurent Massoptier [France] ; Edit Varga [Pays-Bas] ; Pieter Jan Stappers [Pays-Bas] ; Wiro J. Niessen [Pays-Bas] ; Yu Song [Pays-Bas]Source :
- Journal of Digital Imaging [ 0897-1889 ] ; 2015.
Abstract
Accurate segmentation of organs at risk is an important step in radiotherapy planning. Manual segmentation being a tedious procedure and prone to inter- and intra-observer variability, there is a growing interest in automated segmentation methods. However, automatic methods frequently fail to provide satisfactory result, and post-processing corrections are often needed. Semi-automatic segmentation methods are designed to overcome these problems by combining physicians’ expertise and computers’ potential. This study evaluates two semi-automatic segmentation methods with different types of user interactions, named the “strokes” and the “contour”, to provide insights into the role and impact of human-computer interaction. Two physicians participated in the experiment. In total, 42 case studies were carried out on five different types of organs at risk. For each case study, both the human-computer interaction process and quality of the segmentation results were measured subjectively and objectively. Furthermore, different measures of the process and the results were correlated. A total of 36 quantifiable and ten non-quantifiable correlations were identified for each type of interaction. Among those pairs of measures, 20 of the contour method and 22 of the strokes method were strongly or moderately correlated, either directly or inversely. Based on those correlated measures, it is concluded that: (1) in the design of semi-automatic segmentation methods, user interactions need to be less cognitively challenging; (2) based on the observed workflows and preferences of physicians, there is a need for flexibility in the interface design; (3) the correlated measures provide insights that can be used in improving user interaction design.
Url:
DOI: 10.1007/s10278-015-9839-8
PubMed: 26553109
PubMed Central: 4788616
Links toward previous steps (curation, corpus...)
- to stream Pmc, to step Corpus: 000827
- to stream Pmc, to step Curation: 000827
- to stream Pmc, to step Checkpoint: 000221
Links to Exploration step
PMC:4788616Le document en format XML
<record><TEI><teiHeader><fileDesc><titleStmt><title xml:lang="en">User Interaction in Semi-Automatic Segmentation of Organs at Risk: a Case Study in Radiotherapy</title>
<author><name sortKey="Ramkumar, Anjana" sort="Ramkumar, Anjana" uniqKey="Ramkumar A" first="Anjana" last="Ramkumar">Anjana Ramkumar</name>
<affiliation wicri:level="1"><nlm:aff id="Aff1">Faculty of Industrial Design Engineering, Delft University of Technology, Landbergstraat 15, 2628CE Delft, The Netherlands</nlm:aff>
<country xml:lang="fr">Pays-Bas</country>
<wicri:regionArea>Faculty of Industrial Design Engineering, Delft University of Technology, Landbergstraat 15, 2628CE Delft</wicri:regionArea>
<wicri:noRegion>2628CE Delft</wicri:noRegion>
</affiliation>
</author>
<author><name sortKey="Dolz, Jose" sort="Dolz, Jose" uniqKey="Dolz J" first="Jose" last="Dolz">Jose Dolz</name>
<affiliation wicri:level="1"><nlm:aff id="Aff2">Aquilab, Loos-les-Lille, Lille, France</nlm:aff>
<country xml:lang="fr">France</country>
<wicri:regionArea>Aquilab, Loos-les-Lille, Lille</wicri:regionArea>
<placeName><settlement type="city">Lille</settlement>
</placeName>
</affiliation>
</author>
<author><name sortKey="Kirisli, Hortense A" sort="Kirisli, Hortense A" uniqKey="Kirisli H" first="Hortense A." last="Kirisli">Hortense A. Kirisli</name>
<affiliation wicri:level="1"><nlm:aff id="Aff2">Aquilab, Loos-les-Lille, Lille, France</nlm:aff>
<country xml:lang="fr">France</country>
<wicri:regionArea>Aquilab, Loos-les-Lille, Lille</wicri:regionArea>
<placeName><settlement type="city">Lille</settlement>
</placeName>
</affiliation>
</author>
<author><name sortKey="Adebahr, Sonja" sort="Adebahr, Sonja" uniqKey="Adebahr S" first="Sonja" last="Adebahr">Sonja Adebahr</name>
<affiliation wicri:level="3"><nlm:aff id="Aff3">Department of Radiation Oncology, University Medical Center Freiburg, Freiburg, Germany</nlm:aff>
<country xml:lang="fr">Allemagne</country>
<wicri:regionArea>Department of Radiation Oncology, University Medical Center Freiburg, Freiburg</wicri:regionArea>
<placeName><region type="land" nuts="1">Bade-Wurtemberg</region>
<region type="district" nuts="2">District de Fribourg-en-Brisgau</region>
<settlement type="city">Fribourg-en-Brisgau</settlement>
</placeName>
</affiliation>
</author>
<author><name sortKey="Schimek Jasch, Tanja" sort="Schimek Jasch, Tanja" uniqKey="Schimek Jasch T" first="Tanja" last="Schimek-Jasch">Tanja Schimek-Jasch</name>
<affiliation wicri:level="3"><nlm:aff id="Aff3">Department of Radiation Oncology, University Medical Center Freiburg, Freiburg, Germany</nlm:aff>
<country xml:lang="fr">Allemagne</country>
<wicri:regionArea>Department of Radiation Oncology, University Medical Center Freiburg, Freiburg</wicri:regionArea>
<placeName><region type="land" nuts="1">Bade-Wurtemberg</region>
<region type="district" nuts="2">District de Fribourg-en-Brisgau</region>
<settlement type="city">Fribourg-en-Brisgau</settlement>
</placeName>
</affiliation>
</author>
<author><name sortKey="Nestle, Ursula" sort="Nestle, Ursula" uniqKey="Nestle U" first="Ursula" last="Nestle">Ursula Nestle</name>
<affiliation wicri:level="3"><nlm:aff id="Aff3">Department of Radiation Oncology, University Medical Center Freiburg, Freiburg, Germany</nlm:aff>
<country xml:lang="fr">Allemagne</country>
<wicri:regionArea>Department of Radiation Oncology, University Medical Center Freiburg, Freiburg</wicri:regionArea>
<placeName><region type="land" nuts="1">Bade-Wurtemberg</region>
<region type="district" nuts="2">District de Fribourg-en-Brisgau</region>
<settlement type="city">Fribourg-en-Brisgau</settlement>
</placeName>
</affiliation>
</author>
<author><name sortKey="Massoptier, Laurent" sort="Massoptier, Laurent" uniqKey="Massoptier L" first="Laurent" last="Massoptier">Laurent Massoptier</name>
<affiliation wicri:level="1"><nlm:aff id="Aff2">Aquilab, Loos-les-Lille, Lille, France</nlm:aff>
<country xml:lang="fr">France</country>
<wicri:regionArea>Aquilab, Loos-les-Lille, Lille</wicri:regionArea>
<placeName><settlement type="city">Lille</settlement>
</placeName>
</affiliation>
</author>
<author><name sortKey="Varga, Edit" sort="Varga, Edit" uniqKey="Varga E" first="Edit" last="Varga">Edit Varga</name>
<affiliation wicri:level="3"><nlm:aff id="Aff4">Department of Medical Informatics, Erasmus MC, Rotterdam, The Netherlands</nlm:aff>
<country xml:lang="fr">Pays-Bas</country>
<wicri:regionArea>Department of Medical Informatics, Erasmus MC, Rotterdam</wicri:regionArea>
<placeName><settlement type="city">Rotterdam</settlement>
<region nuts="2" type="province">Hollande-Méridionale</region>
</placeName>
</affiliation>
</author>
<author><name sortKey="Stappers, Pieter Jan" sort="Stappers, Pieter Jan" uniqKey="Stappers P" first="Pieter Jan" last="Stappers">Pieter Jan Stappers</name>
<affiliation wicri:level="1"><nlm:aff id="Aff1">Faculty of Industrial Design Engineering, Delft University of Technology, Landbergstraat 15, 2628CE Delft, The Netherlands</nlm:aff>
<country xml:lang="fr">Pays-Bas</country>
<wicri:regionArea>Faculty of Industrial Design Engineering, Delft University of Technology, Landbergstraat 15, 2628CE Delft</wicri:regionArea>
<wicri:noRegion>2628CE Delft</wicri:noRegion>
</affiliation>
</author>
<author><name sortKey="Niessen, Wiro J" sort="Niessen, Wiro J" uniqKey="Niessen W" first="Wiro J." last="Niessen">Wiro J. Niessen</name>
<affiliation wicri:level="3"><nlm:aff id="Aff4">Department of Medical Informatics, Erasmus MC, Rotterdam, The Netherlands</nlm:aff>
<country xml:lang="fr">Pays-Bas</country>
<wicri:regionArea>Department of Medical Informatics, Erasmus MC, Rotterdam</wicri:regionArea>
<placeName><settlement type="city">Rotterdam</settlement>
<region nuts="2" type="province">Hollande-Méridionale</region>
</placeName>
</affiliation>
<affiliation wicri:level="1"><nlm:aff id="Aff5">Faculty of Applied Science, Delft University of Technology, Delft, The Netherlands</nlm:aff>
<country xml:lang="fr">Pays-Bas</country>
<wicri:regionArea>Faculty of Applied Science, Delft University of Technology, Delft</wicri:regionArea>
<wicri:noRegion>Delft</wicri:noRegion>
</affiliation>
</author>
<author><name sortKey="Song, Yu" sort="Song, Yu" uniqKey="Song Y" first="Yu" last="Song">Yu Song</name>
<affiliation wicri:level="1"><nlm:aff id="Aff1">Faculty of Industrial Design Engineering, Delft University of Technology, Landbergstraat 15, 2628CE Delft, The Netherlands</nlm:aff>
<country xml:lang="fr">Pays-Bas</country>
<wicri:regionArea>Faculty of Industrial Design Engineering, Delft University of Technology, Landbergstraat 15, 2628CE Delft</wicri:regionArea>
<wicri:noRegion>2628CE Delft</wicri:noRegion>
</affiliation>
</author>
</titleStmt>
<publicationStmt><idno type="wicri:source">PMC</idno>
<idno type="pmid">26553109</idno>
<idno type="pmc">4788616</idno>
<idno type="url">http://www.ncbi.nlm.nih.gov/pmc/articles/PMC4788616</idno>
<idno type="RBID">PMC:4788616</idno>
<idno type="doi">10.1007/s10278-015-9839-8</idno>
<date when="2015">2015</date>
<idno type="wicri:Area/Pmc/Corpus">000827</idno>
<idno type="wicri:Area/Pmc/Curation">000827</idno>
<idno type="wicri:Area/Pmc/Checkpoint">000221</idno>
<idno type="wicri:Area/Ncbi/Merge">003D46</idno>
</publicationStmt>
<sourceDesc><biblStruct><analytic><title xml:lang="en" level="a" type="main">User Interaction in Semi-Automatic Segmentation of Organs at Risk: a Case Study in Radiotherapy</title>
<author><name sortKey="Ramkumar, Anjana" sort="Ramkumar, Anjana" uniqKey="Ramkumar A" first="Anjana" last="Ramkumar">Anjana Ramkumar</name>
<affiliation wicri:level="1"><nlm:aff id="Aff1">Faculty of Industrial Design Engineering, Delft University of Technology, Landbergstraat 15, 2628CE Delft, The Netherlands</nlm:aff>
<country xml:lang="fr">Pays-Bas</country>
<wicri:regionArea>Faculty of Industrial Design Engineering, Delft University of Technology, Landbergstraat 15, 2628CE Delft</wicri:regionArea>
<wicri:noRegion>2628CE Delft</wicri:noRegion>
</affiliation>
</author>
<author><name sortKey="Dolz, Jose" sort="Dolz, Jose" uniqKey="Dolz J" first="Jose" last="Dolz">Jose Dolz</name>
<affiliation wicri:level="1"><nlm:aff id="Aff2">Aquilab, Loos-les-Lille, Lille, France</nlm:aff>
<country xml:lang="fr">France</country>
<wicri:regionArea>Aquilab, Loos-les-Lille, Lille</wicri:regionArea>
<placeName><settlement type="city">Lille</settlement>
</placeName>
</affiliation>
</author>
<author><name sortKey="Kirisli, Hortense A" sort="Kirisli, Hortense A" uniqKey="Kirisli H" first="Hortense A." last="Kirisli">Hortense A. Kirisli</name>
<affiliation wicri:level="1"><nlm:aff id="Aff2">Aquilab, Loos-les-Lille, Lille, France</nlm:aff>
<country xml:lang="fr">France</country>
<wicri:regionArea>Aquilab, Loos-les-Lille, Lille</wicri:regionArea>
<placeName><settlement type="city">Lille</settlement>
</placeName>
</affiliation>
</author>
<author><name sortKey="Adebahr, Sonja" sort="Adebahr, Sonja" uniqKey="Adebahr S" first="Sonja" last="Adebahr">Sonja Adebahr</name>
<affiliation wicri:level="3"><nlm:aff id="Aff3">Department of Radiation Oncology, University Medical Center Freiburg, Freiburg, Germany</nlm:aff>
<country xml:lang="fr">Allemagne</country>
<wicri:regionArea>Department of Radiation Oncology, University Medical Center Freiburg, Freiburg</wicri:regionArea>
<placeName><region type="land" nuts="1">Bade-Wurtemberg</region>
<region type="district" nuts="2">District de Fribourg-en-Brisgau</region>
<settlement type="city">Fribourg-en-Brisgau</settlement>
</placeName>
</affiliation>
</author>
<author><name sortKey="Schimek Jasch, Tanja" sort="Schimek Jasch, Tanja" uniqKey="Schimek Jasch T" first="Tanja" last="Schimek-Jasch">Tanja Schimek-Jasch</name>
<affiliation wicri:level="3"><nlm:aff id="Aff3">Department of Radiation Oncology, University Medical Center Freiburg, Freiburg, Germany</nlm:aff>
<country xml:lang="fr">Allemagne</country>
<wicri:regionArea>Department of Radiation Oncology, University Medical Center Freiburg, Freiburg</wicri:regionArea>
<placeName><region type="land" nuts="1">Bade-Wurtemberg</region>
<region type="district" nuts="2">District de Fribourg-en-Brisgau</region>
<settlement type="city">Fribourg-en-Brisgau</settlement>
</placeName>
</affiliation>
</author>
<author><name sortKey="Nestle, Ursula" sort="Nestle, Ursula" uniqKey="Nestle U" first="Ursula" last="Nestle">Ursula Nestle</name>
<affiliation wicri:level="3"><nlm:aff id="Aff3">Department of Radiation Oncology, University Medical Center Freiburg, Freiburg, Germany</nlm:aff>
<country xml:lang="fr">Allemagne</country>
<wicri:regionArea>Department of Radiation Oncology, University Medical Center Freiburg, Freiburg</wicri:regionArea>
<placeName><region type="land" nuts="1">Bade-Wurtemberg</region>
<region type="district" nuts="2">District de Fribourg-en-Brisgau</region>
<settlement type="city">Fribourg-en-Brisgau</settlement>
</placeName>
</affiliation>
</author>
<author><name sortKey="Massoptier, Laurent" sort="Massoptier, Laurent" uniqKey="Massoptier L" first="Laurent" last="Massoptier">Laurent Massoptier</name>
<affiliation wicri:level="1"><nlm:aff id="Aff2">Aquilab, Loos-les-Lille, Lille, France</nlm:aff>
<country xml:lang="fr">France</country>
<wicri:regionArea>Aquilab, Loos-les-Lille, Lille</wicri:regionArea>
<placeName><settlement type="city">Lille</settlement>
</placeName>
</affiliation>
</author>
<author><name sortKey="Varga, Edit" sort="Varga, Edit" uniqKey="Varga E" first="Edit" last="Varga">Edit Varga</name>
<affiliation wicri:level="3"><nlm:aff id="Aff4">Department of Medical Informatics, Erasmus MC, Rotterdam, The Netherlands</nlm:aff>
<country xml:lang="fr">Pays-Bas</country>
<wicri:regionArea>Department of Medical Informatics, Erasmus MC, Rotterdam</wicri:regionArea>
<placeName><settlement type="city">Rotterdam</settlement>
<region nuts="2" type="province">Hollande-Méridionale</region>
</placeName>
</affiliation>
</author>
<author><name sortKey="Stappers, Pieter Jan" sort="Stappers, Pieter Jan" uniqKey="Stappers P" first="Pieter Jan" last="Stappers">Pieter Jan Stappers</name>
<affiliation wicri:level="1"><nlm:aff id="Aff1">Faculty of Industrial Design Engineering, Delft University of Technology, Landbergstraat 15, 2628CE Delft, The Netherlands</nlm:aff>
<country xml:lang="fr">Pays-Bas</country>
<wicri:regionArea>Faculty of Industrial Design Engineering, Delft University of Technology, Landbergstraat 15, 2628CE Delft</wicri:regionArea>
<wicri:noRegion>2628CE Delft</wicri:noRegion>
</affiliation>
</author>
<author><name sortKey="Niessen, Wiro J" sort="Niessen, Wiro J" uniqKey="Niessen W" first="Wiro J." last="Niessen">Wiro J. Niessen</name>
<affiliation wicri:level="3"><nlm:aff id="Aff4">Department of Medical Informatics, Erasmus MC, Rotterdam, The Netherlands</nlm:aff>
<country xml:lang="fr">Pays-Bas</country>
<wicri:regionArea>Department of Medical Informatics, Erasmus MC, Rotterdam</wicri:regionArea>
<placeName><settlement type="city">Rotterdam</settlement>
<region nuts="2" type="province">Hollande-Méridionale</region>
</placeName>
</affiliation>
<affiliation wicri:level="1"><nlm:aff id="Aff5">Faculty of Applied Science, Delft University of Technology, Delft, The Netherlands</nlm:aff>
<country xml:lang="fr">Pays-Bas</country>
<wicri:regionArea>Faculty of Applied Science, Delft University of Technology, Delft</wicri:regionArea>
<wicri:noRegion>Delft</wicri:noRegion>
</affiliation>
</author>
<author><name sortKey="Song, Yu" sort="Song, Yu" uniqKey="Song Y" first="Yu" last="Song">Yu Song</name>
<affiliation wicri:level="1"><nlm:aff id="Aff1">Faculty of Industrial Design Engineering, Delft University of Technology, Landbergstraat 15, 2628CE Delft, The Netherlands</nlm:aff>
<country xml:lang="fr">Pays-Bas</country>
<wicri:regionArea>Faculty of Industrial Design Engineering, Delft University of Technology, Landbergstraat 15, 2628CE Delft</wicri:regionArea>
<wicri:noRegion>2628CE Delft</wicri:noRegion>
</affiliation>
</author>
</analytic>
<series><title level="j">Journal of Digital Imaging</title>
<idno type="ISSN">0897-1889</idno>
<idno type="eISSN">1618-727X</idno>
<imprint><date when="2015">2015</date>
</imprint>
</series>
</biblStruct>
</sourceDesc>
</fileDesc>
<profileDesc><textClass></textClass>
</profileDesc>
</teiHeader>
<front><div type="abstract" xml:lang="en"><p>Accurate segmentation of organs at risk is an important step in radiotherapy planning. Manual segmentation being a tedious procedure and prone to inter- and intra-observer variability, there is a growing interest in automated segmentation methods. However, automatic methods frequently fail to provide satisfactory result, and post-processing corrections are often needed. Semi-automatic segmentation methods are designed to overcome these problems by combining physicians’ expertise and computers’ potential. This study evaluates two semi-automatic segmentation methods with different types of user interactions, named the “strokes” and the “contour”, to provide insights into the role and impact of human-computer interaction. Two physicians participated in the experiment. In total, 42 case studies were carried out on five different types of organs at risk. For each case study, both the human-computer interaction process and quality of the segmentation results were measured subjectively and objectively. Furthermore, different measures of the process and the results were correlated. A total of 36 quantifiable and ten non-quantifiable correlations were identified for each type of interaction. Among those pairs of measures, 20 of the contour method and 22 of the strokes method were strongly or moderately correlated, either directly or inversely. Based on those correlated measures, it is concluded that: (1) in the design of semi-automatic segmentation methods, user interactions need to be less cognitively challenging; (2) based on the observed workflows and preferences of physicians, there is a need for flexibility in the interface design; (3) the correlated measures provide insights that can be used in improving user interaction design.</p>
</div>
</front>
<back><div1 type="bibliography"><listBibl><biblStruct><analytic><author><name sortKey="Burnet, Ng" uniqKey="Burnet N">NG Burnet</name>
</author>
<author><name sortKey="Thomas, Sj" uniqKey="Thomas S">SJ Thomas</name>
</author>
<author><name sortKey="Burton, Ke" uniqKey="Burton K">KE Burton</name>
</author>
<author><name sortKey="Jefferies, Sj" uniqKey="Jefferies S">SJ Jefferies</name>
</author>
</analytic>
</biblStruct>
<biblStruct></biblStruct>
<biblStruct><analytic><author><name sortKey="Whitfield, Ga" uniqKey="Whitfield G">GA Whitfield</name>
</author>
<author><name sortKey="Price, P" uniqKey="Price P">P Price</name>
</author>
<author><name sortKey="Price, Gj" uniqKey="Price G">GJ Price</name>
</author>
<author><name sortKey="Moore, Cj" uniqKey="Moore C">CJ Moore</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Heckel, F" uniqKey="Heckel F">F Heckel</name>
</author>
<author><name sortKey="Moltz, Jh" uniqKey="Moltz J">JH Moltz</name>
</author>
<author><name sortKey="Tietjen, C" uniqKey="Tietjen C">C Tietjen</name>
</author>
<author><name sortKey="Hahn, Hk" uniqKey="Hahn H">HK Hahn</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Isambert, A" uniqKey="Isambert A">A Isambert</name>
</author>
<author><name sortKey="Dhermain, F" uniqKey="Dhermain F">F Dhermain</name>
</author>
<author><name sortKey="Bidault, F" uniqKey="Bidault F">F Bidault</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Wu, K" uniqKey="Wu K">K Wu</name>
</author>
<author><name sortKey="Ung, Yc" uniqKey="Ung Y">YC Ung</name>
</author>
<author><name sortKey="Hwang, D" uniqKey="Hwang D">D Hwang</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Smith, Cm" uniqKey="Smith C">CM Smith</name>
</author>
<author><name sortKey="Smith, J" uniqKey="Smith J">J Smith</name>
</author>
<author><name sortKey="Williams, Sk" uniqKey="Williams S">SK Williams</name>
</author>
<author><name sortKey="Rodriguez, Jj" uniqKey="Rodriguez J">JJ Rodriguez</name>
</author>
<author><name sortKey="Hoying, Jb" uniqKey="Hoying J">JB Hoying</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Grau, V" uniqKey="Grau V">V Grau</name>
</author>
<author><name sortKey="Mewes, Auj" uniqKey="Mewes A">AUJ Mewes</name>
</author>
<author><name sortKey="Alcaniz, M" uniqKey="Alcaniz M">M Alcaniz</name>
</author>
<author><name sortKey="Kikinis, R" uniqKey="Kikinis R">R Kikinis</name>
</author>
<author><name sortKey="Warfield, Sk" uniqKey="Warfield S">SK Warfield</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Mondal, T" uniqKey="Mondal T">T Mondal</name>
</author>
<author><name sortKey="Jain, A" uniqKey="Jain A">A Jain</name>
</author>
<author><name sortKey="Sardana, Hk" uniqKey="Sardana H">HK Sardana</name>
</author>
</analytic>
</biblStruct>
<biblStruct></biblStruct>
<biblStruct><analytic><author><name sortKey="Sims, R" uniqKey="Sims R">R Sims</name>
</author>
<author><name sortKey="Isambert, A" uniqKey="Isambert A">A Isambert</name>
</author>
<author><name sortKey="Gregoire, V" uniqKey="Gregoire V">V Grégoire</name>
</author>
</analytic>
</biblStruct>
<biblStruct></biblStruct>
<biblStruct><analytic><author><name sortKey="Yeo, Sy" uniqKey="Yeo S">SY Yeo</name>
</author>
<author><name sortKey="Xie, X" uniqKey="Xie X">X Xie</name>
</author>
<author><name sortKey="Sazonov, I" uniqKey="Sazonov I">I Sazonov</name>
</author>
<author><name sortKey="Nithiarasu, P" uniqKey="Nithiarasu P">P Nithiarasu</name>
</author>
</analytic>
</biblStruct>
<biblStruct></biblStruct>
<biblStruct></biblStruct>
<biblStruct><analytic><author><name sortKey="Olabarriaga, S" uniqKey="Olabarriaga S">S Olabarriaga</name>
</author>
<author><name sortKey="Smeulders, A" uniqKey="Smeulders A">A Smeulders</name>
</author>
</analytic>
</biblStruct>
<biblStruct></biblStruct>
<biblStruct></biblStruct>
<biblStruct></biblStruct>
<biblStruct></biblStruct>
<biblStruct><analytic><author><name sortKey="Heckel, F" uniqKey="Heckel F">F Heckel</name>
</author>
<author><name sortKey="Moltz, Jh" uniqKey="Moltz J">JH Moltz</name>
</author>
<author><name sortKey="Tietjen, C" uniqKey="Tietjen C">C Tietjen</name>
</author>
<author><name sortKey="Hahn, Hk" uniqKey="Hahn H">HK Hahn</name>
</author>
</analytic>
</biblStruct>
<biblStruct></biblStruct>
<biblStruct><analytic><author><name sortKey="Sherbondy, Aj" uniqKey="Sherbondy A">AJ Sherbondy</name>
</author>
<author><name sortKey="Holmlund, D" uniqKey="Holmlund D">D Holmlund</name>
</author>
<author><name sortKey="Rubin, Gd" uniqKey="Rubin G">GD Rubin</name>
</author>
<author><name sortKey="Schraedley, Pk" uniqKey="Schraedley P">PK Schraedley</name>
</author>
<author><name sortKey="Winograd, T" uniqKey="Winograd T">T Winograd</name>
</author>
<author><name sortKey="Napel, S" uniqKey="Napel S">S Napel</name>
</author>
</analytic>
</biblStruct>
<biblStruct></biblStruct>
<biblStruct></biblStruct>
<biblStruct><analytic><author><name sortKey="Li, Y" uniqKey="Li Y">Y Li</name>
</author>
<author><name sortKey="Sun, J" uniqKey="Sun J">J Sun</name>
</author>
<author><name sortKey="Tang, Ck" uniqKey="Tang C">CK Tang</name>
</author>
<author><name sortKey="Shum, Hy" uniqKey="Shum H">HY Shum</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Yang, W" uniqKey="Yang W">W Yang</name>
</author>
<author><name sortKey="Cai, J" uniqKey="Cai J">J Cai</name>
</author>
</analytic>
</biblStruct>
<biblStruct></biblStruct>
<biblStruct><analytic><author><name sortKey="Dalah, Ez" uniqKey="Dalah E">EZ Dalah</name>
</author>
<author><name sortKey="Nisbet, A" uniqKey="Nisbet A">A Nisbet</name>
</author>
<author><name sortKey="Bradley, D" uniqKey="Bradley D">D Bradley</name>
</author>
</analytic>
</biblStruct>
<biblStruct></biblStruct>
<biblStruct></biblStruct>
<biblStruct><analytic><author><name sortKey="Ju, W" uniqKey="Ju W">W Ju</name>
</author>
<author><name sortKey="Leifer, L" uniqKey="Leifer L">L Leifer</name>
</author>
</analytic>
</biblStruct>
<biblStruct></biblStruct>
<biblStruct></biblStruct>
<biblStruct></biblStruct>
<biblStruct><analytic><author><name sortKey="Clark, Re" uniqKey="Clark R">RE Clark</name>
</author>
<author><name sortKey="Pugh, Cm" uniqKey="Pugh C">CM Pugh</name>
</author>
<author><name sortKey="Yates, Ka" uniqKey="Yates K">KA Yates</name>
</author>
<author><name sortKey="Inaba, K" uniqKey="Inaba K">K Inaba</name>
</author>
<author><name sortKey="Green, Dj" uniqKey="Green D">DJ Green</name>
</author>
<author><name sortKey="Sullivan, Me" uniqKey="Sullivan M">ME Sullivan</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Jaspers, Mw" uniqKey="Jaspers M">MW Jaspers</name>
</author>
<author><name sortKey="Steen, T" uniqKey="Steen T">T Steen</name>
</author>
<author><name sortKey="Van Den Bos, C" uniqKey="Van Den Bos C">C van den Bos</name>
</author>
<author><name sortKey="Geenen, M" uniqKey="Geenen M">M Geenen</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Lee, S" uniqKey="Lee S">S Lee</name>
</author>
<author><name sortKey="Koubek, Rj" uniqKey="Koubek R">RJ Koubek</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Coen, Rn" uniqKey="Coen R">RN Coen</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Kotani, K" uniqKey="Kotani K">K Kotani</name>
</author>
<author><name sortKey="Horii, K" uniqKey="Horii K">K Horii</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Mcguinness, K" uniqKey="Mcguinness K">K McGuinness</name>
</author>
<author><name sortKey="O Onnor, Ne" uniqKey="O Onnor N">NE O’Connor</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Annett, J" uniqKey="Annett J">J Annett</name>
</author>
</analytic>
</biblStruct>
<biblStruct></biblStruct>
<biblStruct><analytic><author><name sortKey="Hornb K, K" uniqKey="Hornb K K">K Hornbæk</name>
</author>
</analytic>
</biblStruct>
<biblStruct></biblStruct>
<biblStruct></biblStruct>
<biblStruct></biblStruct>
<biblStruct></biblStruct>
<biblStruct></biblStruct>
<biblStruct></biblStruct>
<biblStruct><analytic><author><name sortKey="Dice, Lr" uniqKey="Dice L">LR Dice</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Dancey, C" uniqKey="Dancey C">C Dancey</name>
</author>
<author><name sortKey="Reidy, J" uniqKey="Reidy J">J Reidy</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Freudenthal, A" uniqKey="Freudenthal A">A Freudenthal</name>
</author>
<author><name sortKey="Studeli, T" uniqKey="Studeli T">T Stüdeli</name>
</author>
<author><name sortKey="Lamata, P" uniqKey="Lamata P">P Lamata</name>
</author>
<author><name sortKey="Samset, E" uniqKey="Samset E">E Samset</name>
</author>
</analytic>
</biblStruct>
<biblStruct><analytic><author><name sortKey="Yurko, Yy" uniqKey="Yurko Y">YY Yurko</name>
</author>
<author><name sortKey="Scerbo, Mw" uniqKey="Scerbo M">MW Scerbo</name>
</author>
<author><name sortKey="Prabhu, As" uniqKey="Prabhu A">AS Prabhu</name>
</author>
<author><name sortKey="Acker, Ce" uniqKey="Acker C">CE Acker</name>
</author>
<author><name sortKey="Stefanidis, D" uniqKey="Stefanidis D">D Stefanidis</name>
</author>
</analytic>
</biblStruct>
</listBibl>
</div1>
</back>
</TEI>
<pmc article-type="research-article"><pmc-dir>properties open_access</pmc-dir>
<front><journal-meta><journal-id journal-id-type="nlm-ta">J Digit Imaging</journal-id>
<journal-id journal-id-type="iso-abbrev">J Digit Imaging</journal-id>
<journal-title-group><journal-title>Journal of Digital Imaging</journal-title>
</journal-title-group>
<issn pub-type="ppub">0897-1889</issn>
<issn pub-type="epub">1618-727X</issn>
<publisher><publisher-name>Springer International Publishing</publisher-name>
<publisher-loc>Cham</publisher-loc>
</publisher>
</journal-meta>
<article-meta><article-id pub-id-type="pmid">26553109</article-id>
<article-id pub-id-type="pmc">4788616</article-id>
<article-id pub-id-type="publisher-id">9839</article-id>
<article-id pub-id-type="doi">10.1007/s10278-015-9839-8</article-id>
<article-categories><subj-group subj-group-type="heading"><subject>Article</subject>
</subj-group>
</article-categories>
<title-group><article-title>User Interaction in Semi-Automatic Segmentation of Organs at Risk: a Case Study in Radiotherapy</article-title>
</title-group>
<contrib-group><contrib contrib-type="author"><name><surname>Ramkumar</surname>
<given-names>Anjana</given-names>
</name>
<xref ref-type="aff" rid="Aff1"></xref>
</contrib>
<contrib contrib-type="author"><name><surname>Dolz</surname>
<given-names>Jose</given-names>
</name>
<xref ref-type="aff" rid="Aff2"></xref>
</contrib>
<contrib contrib-type="author"><name><surname>Kirisli</surname>
<given-names>Hortense A.</given-names>
</name>
<xref ref-type="aff" rid="Aff2"></xref>
</contrib>
<contrib contrib-type="author"><name><surname>Adebahr</surname>
<given-names>Sonja</given-names>
</name>
<xref ref-type="aff" rid="Aff3"></xref>
</contrib>
<contrib contrib-type="author"><name><surname>Schimek-Jasch</surname>
<given-names>Tanja</given-names>
</name>
<xref ref-type="aff" rid="Aff3"></xref>
</contrib>
<contrib contrib-type="author"><name><surname>Nestle</surname>
<given-names>Ursula</given-names>
</name>
<xref ref-type="aff" rid="Aff3"></xref>
</contrib>
<contrib contrib-type="author"><name><surname>Massoptier</surname>
<given-names>Laurent</given-names>
</name>
<xref ref-type="aff" rid="Aff2"></xref>
</contrib>
<contrib contrib-type="author"><name><surname>Varga</surname>
<given-names>Edit</given-names>
</name>
<xref ref-type="aff" rid="Aff4"></xref>
</contrib>
<contrib contrib-type="author"><name><surname>Stappers</surname>
<given-names>Pieter Jan</given-names>
</name>
<xref ref-type="aff" rid="Aff1"></xref>
</contrib>
<contrib contrib-type="author"><name><surname>Niessen</surname>
<given-names>Wiro J.</given-names>
</name>
<xref ref-type="aff" rid="Aff4"></xref>
<xref ref-type="aff" rid="Aff5"></xref>
</contrib>
<contrib contrib-type="author" corresp="yes"><name><surname>Song</surname>
<given-names>Yu</given-names>
</name>
<address><phone>+31 152788286</phone>
<email>Y.Song@tudelft.nl</email>
</address>
<xref ref-type="aff" rid="Aff1"></xref>
</contrib>
<aff id="Aff1"><label></label>
Faculty of Industrial Design Engineering, Delft University of Technology, Landbergstraat 15, 2628CE Delft, The Netherlands</aff>
<aff id="Aff2"><label></label>
Aquilab, Loos-les-Lille, Lille, France</aff>
<aff id="Aff3"><label></label>
Department of Radiation Oncology, University Medical Center Freiburg, Freiburg, Germany</aff>
<aff id="Aff4"><label></label>
Department of Medical Informatics, Erasmus MC, Rotterdam, The Netherlands</aff>
<aff id="Aff5"><label></label>
Faculty of Applied Science, Delft University of Technology, Delft, The Netherlands</aff>
</contrib-group>
<pub-date pub-type="epub"><day>9</day>
<month>11</month>
<year>2015</year>
</pub-date>
<pub-date pub-type="pmc-release"><day>9</day>
<month>11</month>
<year>2015</year>
</pub-date>
<pub-date pub-type="ppub"><month>4</month>
<year>2016</year>
</pub-date>
<volume>29</volume>
<issue>2</issue>
<fpage>264</fpage>
<lpage>277</lpage>
<permissions><copyright-statement>© The Author(s) 2015</copyright-statement>
<license license-type="OpenAccess"><license-p><bold>Open Access</bold>
This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.</license-p>
</license>
</permissions>
<abstract id="Abs1"><p>Accurate segmentation of organs at risk is an important step in radiotherapy planning. Manual segmentation being a tedious procedure and prone to inter- and intra-observer variability, there is a growing interest in automated segmentation methods. However, automatic methods frequently fail to provide satisfactory result, and post-processing corrections are often needed. Semi-automatic segmentation methods are designed to overcome these problems by combining physicians’ expertise and computers’ potential. This study evaluates two semi-automatic segmentation methods with different types of user interactions, named the “strokes” and the “contour”, to provide insights into the role and impact of human-computer interaction. Two physicians participated in the experiment. In total, 42 case studies were carried out on five different types of organs at risk. For each case study, both the human-computer interaction process and quality of the segmentation results were measured subjectively and objectively. Furthermore, different measures of the process and the results were correlated. A total of 36 quantifiable and ten non-quantifiable correlations were identified for each type of interaction. Among those pairs of measures, 20 of the contour method and 22 of the strokes method were strongly or moderately correlated, either directly or inversely. Based on those correlated measures, it is concluded that: (1) in the design of semi-automatic segmentation methods, user interactions need to be less cognitively challenging; (2) based on the observed workflows and preferences of physicians, there is a need for flexibility in the interface design; (3) the correlated measures provide insights that can be used in improving user interaction design.</p>
</abstract>
<kwd-group xml:lang="en"><title>Keywords</title>
<kwd>Radiotherapy</kwd>
<kwd>Organs at risk</kwd>
<kwd>Semi-automatic segmentation</kwd>
<kwd>Human-computer interaction</kwd>
<kwd>Evaluation</kwd>
<kwd>Correlations</kwd>
</kwd-group>
<funding-group><award-group><funding-source><institution>False</institution>
</funding-source>
</award-group>
</funding-group>
<custom-meta-group><custom-meta><meta-name>issue-copyright-statement</meta-name>
<meta-value>© Society for Imaging Informatics in Medicine 2016</meta-value>
</custom-meta>
</custom-meta-group>
</article-meta>
</front>
<body><sec id="Sec1" sec-type="introduction"><title>Introduction</title>
<p>In radiotherapy planning, three fundamental axioms are often applied [<xref ref-type="bibr" rid="CR1">1</xref>
]: (1) an increased dose to the tumor normally improves the local control; (2) improving local control of tumors improves overall cure rate, as it prevents metastatic spread from local recurrence; and (3) sparing normal tissues decreases the side effects of radiotherapy. Thus, to maximize the delivery of radiation dose to the tumor while sparing the normal tissues, accurate segmentation of tumor and organs at risk on medical images is a prerequisite.</p>
<p>Manual segmentation performed by experts is often used as the reference standard in radiotherapy planning [<xref ref-type="bibr" rid="CR2">2</xref>
]. Using manual segmentation methods, physicians segment the organs by drawing contours on medical images slice by slice based on their clinical knowledge. The process is generally time consuming, demands high workload due to intensive human-computer interactions (HCI) and lacks reproducibility [<xref ref-type="bibr" rid="CR3">3</xref>
, <xref ref-type="bibr" rid="CR4">4</xref>
].</p>
<p>To overcome the limitations of manual segmentation, automated segmentation methods have been introduced. These methods have shown to be an effective solution for various applications [<xref ref-type="bibr" rid="CR5">5</xref>
, <xref ref-type="bibr" rid="CR6">6</xref>
] as they are usually faster than manual segmentation methods, and require no or few user interactions during the segmentation process [<xref ref-type="bibr" rid="CR7">7</xref>
–<xref ref-type="bibr" rid="CR9">9</xref>
]. However, the outcomes are sensitive to image quality, which highly depends on the acquisition protocols [<xref ref-type="bibr" rid="CR10">10</xref>
]. In many cases, automatic segmentation methods can only be applied successfully within pre-defined conditions and extensive post-processing is often needed. For instance, Wu et al. [<xref ref-type="bibr" rid="CR6">6</xref>
] identified that their automatic segmentation method performed well for large organs, while manual corrections were often required for smaller organs. Sims et al. [<xref ref-type="bibr" rid="CR11">11</xref>
] also concluded that careful review and manual editing were required for most segmentation results obtained by automatic methods.</p>
<p>By engaging physicians in between the computational algorithm, semi-automatic segmentation (SAS) methods were developed. SAS are partially supervised automatic methods and they provide solutions by combining physicians’ expertise and the computing power of the computer [<xref ref-type="bibr" rid="CR12">12</xref>
–<xref ref-type="bibr" rid="CR14">14</xref>
]. Figure <xref rid="Fig1" ref-type="fig">1</xref>
presents a typical information flow of the SAS method [<xref ref-type="bibr" rid="CR15">15</xref>
, <xref ref-type="bibr" rid="CR16">16</xref>
]. The flow starts from a physician, who first perceives the information on the dataset to get familiarized with the case. After acquiring the information from the dataset, the physician decides on the next step in the segmentation process and performs the required <italic>action</italic>
. Here the term <italic>action</italic>
refers to the physical activities performed by the physician such as moving their hand to choose the input device, scrolling the mouse button to select the desired plane/tool, pressing the zoom-in/out button, initializing the segmentation by drawing contours, and positioning their hand in case of gesture interaction. <italic>Actions</italic>
performed by the physician are interpreted by software via the graphical user interface. Once confirmed, the medical images are processed by a computational algorithm(s) utilizing the input(s), and the output data is displayed on the user interface. This process iterates until a satisfied result is achieved.<fig id="Fig1"><label>Fig. 1</label>
<caption><p>Information flow of human-computer interaction in a SAS method</p>
</caption>
<graphic xlink:href="10278_2015_9839_Fig1_HTML" id="MO1"></graphic>
</fig>
</p>
<p>Effectiveness and efficiency of a SAS method depend on the proper combination of physicians’ expertise and the capability of the computational method [<xref ref-type="bibr" rid="CR17">17</xref>
]. Though physicians play a crucial role in the segmentation process, research on the development of SAS methods has mainly focused on the computational part [<xref ref-type="bibr" rid="CR18">18</xref>
]. The cognitive aspects of physicians and designed human-computer interaction in the segmentation process have only been addressed in few works [<xref ref-type="bibr" rid="CR16">16</xref>
, <xref ref-type="bibr" rid="CR19">19</xref>
, <xref ref-type="bibr" rid="CR20">20</xref>
].</p>
<p>In this paper, we investigate the effects of user interaction in SAS methods regarding the segmentation of organs at risk for radiotherapy planning in order to propose suggestions for further improvements. To achieve this, two SAS methods with the same workflow but two different types of user interaction were developed. A case study was conducted where physicians were asked to segment five organs using the two SAS methods. In the investigation, both human-computer interaction process and the quality of the segmentation results were measured subjectively and objectively. To aid in the interpretation of the results, we identified correlations between the measurements obtained. In this way, we were able to distinguish the effectiveness and efficiency of user interactions in various steps of the SAS methods. Finally, suggestions regarding the design of user interactions in SAS methods are proposed based on these findings.</p>
<p>The remainder of this paper is organized as follows: In section 2, research regarding the workflow of SAS methods, HCI in SAS methods and evaluation methods are reviewed. The two SAS methods used in this research are introduced in section 3 with the focus on the workflow and the design of HCIs. Section 4 lists the setup and protocol of the experiment. Experimental results are analyzed and presented in section 5. The findings in those results are discussed in section 6 where suggestions for the design of user interactions are presented as well. Finally, conclusions are drawn in section 7.</p>
</sec>
<sec id="Sec2"><title>Literature Review</title>
<p>In a SAS method, the workflow is often designed based on the characteristic of computational algorithms and available HCI devices. A review of the literature indicates that three different types of workflow are often implemented [<xref ref-type="bibr" rid="CR16">16</xref>
]. In the first type, a physician initializes the segmentation algorithm and depending on the outcome, manual editing may be performed until a satisfactory result is achieved [<xref ref-type="bibr" rid="CR21">21</xref>
]. The second one is similar to the first: a physician first initializes the segmentation algorithm and if the result is not satisfactory, instead of editing the result manually, he/she may re-initialize the segmentation algorithm [<xref ref-type="bibr" rid="CR19">19</xref>
]. In the third type, the physician modifies the obtained result in a local region such that only the area where the segmentation is not satisfactory is indicated by the physician and is corrected automatically using various algorithm [<xref ref-type="bibr" rid="CR22">22</xref>
]. In the workflow of a SAS method, mouse, keyboard, and screen are the most often used human-computer interaction devices. However, there are many other devices which may facilitate this process. For instance, Harders et al. [<xref ref-type="bibr" rid="CR20">20</xref>
] evaluated the value of haptic feedback in a multimodal setting and found that the used approach is only applicable to linear structures. Sherbondy et al. [<xref ref-type="bibr" rid="CR23">23</xref>
] evaluated user input devices such as trackball, pen-tablet, jog-shuttle wheel, and mouse. They found that the pen-tablet in two distinct configurations performed faster than the mouse and trackball in a simulated angiography localization task. Besides those devices, a different approach to interactive segmentation was introduced by Sadeghi et al. [<xref ref-type="bibr" rid="CR24">24</xref>
], who used eye gaze to guide the segmentation. However, accurate placement of strokes might be strenuous on the eyes for complicated medical images.</p>
<p>Using HCI devices, physicians may select different HCI tools to perform interaction. Olabarriaga et al. [<xref ref-type="bibr" rid="CR16">16</xref>
] investigated HCI issues in 2D segmentation and one main focus was on the segmentation tools used, such as deform, edit boundary, and rectangle. Aselmaa et al. [<xref ref-type="bibr" rid="CR25">25</xref>
] concluded that in manual segmentation tasks, brush tool, 3D pencil, smart brush, and nudging were often used. Using these tools, physicians may perform different types of interactions such as fine tuning parameters, drawing lines, marking points, and drawing bounding boxes [<xref ref-type="bibr" rid="CR26">26</xref>
], to provide inputs to computational algorithms. Using HCI tools, various types of user input can be designed. Yang et al. [<xref ref-type="bibr" rid="CR27">27</xref>
] concluded from their study that the type of user input is an important factor that has to be taken into account as it also affects the outcome of the segmentation result. Hebbalaguppe et al. [<xref ref-type="bibr" rid="CR28">28</xref>
] compared three different types of user input for semi-automatic segmentation and identified the relations between them and the segmentation result. The Lazy Snapping work [<xref ref-type="bibr" rid="CR26">26</xref>
] integrated intuitive user interfaces, such as foreground/background strokes and boundary polygon editing, to emphasize the desirability of a limited amount of user inputs.</p>
<p>Another relevant aspect is the HCI patterns observed during the process of SAS methods. HCI patterns are a series of repetitive physical actions that are executed during segmentation, for instance, continuous zooming in and out, constant scrolling through a set of images, and constant alteration of window levels. These patterns are developed based on physicians’ clinical knowledge and personal preferences, and the outcome of segmentation is influenced by these patterns. A study conducted by Dalah et al. [<xref ref-type="bibr" rid="CR29">29</xref>
] proved that changes in window level settings during segmentation produced about 2 mm discrepancies in the outcomes. Other studies [<xref ref-type="bibr" rid="CR30">30</xref>
, <xref ref-type="bibr" rid="CR31">31</xref>
] on HCI patterns also revealed the influence while performing certain tasks. Ju and Leifer [<xref ref-type="bibr" rid="CR32">32</xref>
] discussed that identifying the HCI patterns can be useful for designers to overcome the interaction design problems and help them leverage existing linguistic, sociological, or ethnographic techniques for designing better human-computer interaction.</p>
<p>In order to improve the usability of the input devices, tools, and types of user input, a proper usability evaluation of current designs is required. ISO 9241 part 11 [<xref ref-type="bibr" rid="CR33">33</xref>
] defines usability as “<italic>the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use</italic>
”. Here effectiveness refers to the degree of completeness and accuracy with which the work/goal is achieved. Efficiency refers to how much effort and how much time physicians spent to finish a task. Satisfaction denotes to what extent physicians are satisfied with efficiency and effectiveness of the task. Thus, in the usability evaluation of a SAS method, both the result and the process should be assessed to measure effectiveness, efficiency, and satisfaction.</p>
<p>A variety of usability evaluation methods have been used to detect the usability problems related to technology. They are the following: heuristic evaluation [<xref ref-type="bibr" rid="CR34">34</xref>
], cognitive walkthrough [<xref ref-type="bibr" rid="CR35">35</xref>
], cognitive task analysis [<xref ref-type="bibr" rid="CR36">36</xref>
, <xref ref-type="bibr" rid="CR37">37</xref>
], think-aloud protocol [<xref ref-type="bibr" rid="CR38">38</xref>
], usability surveys [<xref ref-type="bibr" rid="CR39">39</xref>
], etc. Most of the usability studies include subjective and objective measures, some quantifying the HCI process, others quantifying the result. Among those studies, objective measures of the HCI process have gradually gained attention. For instance, Coen [<xref ref-type="bibr" rid="CR40">40</xref>
] evaluated the HCI input devices by measuring the number of mouse clicks, mouse movement, zooming, panning, scrolling, corrections, and related the interaction patterns regarding the segmentation result. A similar study in radiotherapy conducted by Kotani and Horii [<xref ref-type="bibr" rid="CR41">41</xref>
] compared interactions between the pen-tablet and the mouse. In their study, the error rate was a measure of the result and electromyography was a measure of the process. Hebbalaguppe et al. [<xref ref-type="bibr" rid="CR29">29</xref>
] assessed the cognitive workload by means of electroencephalogram signals. In their study, electroencephalogram signals were found to be correlated to attention, emotion, and decision-making of the users. Olabarriaga and Smeulders [<xref ref-type="bibr" rid="CR16">16</xref>
] evaluated the effectiveness of HCI by measuring the accuracy and reproducibility of the system. McGuinness and O’Connor [<xref ref-type="bibr" rid="CR42">42</xref>
] compared four interactive segmentation techniques by comparing users’ perception and the measurement result. Though considerable effort has been devoted to this area, the use of the subjective or/and objective measures in usability evaluation is still a challenging question [<xref ref-type="bibr" rid="CR43">43</xref>
, <xref ref-type="bibr" rid="CR44">44</xref>
]. For instance, Hornbæk [<xref ref-type="bibr" rid="CR45">45</xref>
] concluded that identifying relations between the HCI process and the outcomes from the measurement are relevant direction for future research.</p>
</sec>
<sec id="Sec3" sec-type="materials|methods"><title>Two Types of User Interaction in the Proposed SAS Methods</title>
<p>In the proposed research, two SAS methods with the same workflow but different interactions were developed. The first SAS method, which is referred as the “contour” method, requires the physician to draw contours in a limited number of slices as shown in Fig. <xref rid="Fig2" ref-type="fig">2a</xref>
and the algorithm then computes the segmented volume in 3D. Physicians are often familiar with the contour method, as it is used for segmentation in their clinical routine. Using this method, physicians were instructed to trace the boundary of the organ accurately on the slice they select. It is assumed that the interaction can be physically and mentally demanding for the physician. In this context physical demand refers to the laborious and time-consuming contouring. Mental demand refers to the task which involves considerable thinking and scrolling, in which the physician needs to be more focused.<fig id="Fig2"><label>Fig. 2</label>
<caption><p>User interfaces of the proposed two SAS methods. <bold>a</bold>
User Interface of the contour method <bold>b</bold>
User Interface of the strokes method</p>
</caption>
<graphic xlink:href="10278_2015_9839_Fig2_HTML" id="MO2"></graphic>
</fig>
</p>
<p>The second SAS method is the “strokes” method which is designed to reduce the physical and mental demands of physicians. The physician draws strokes to indicate the foreground (as the two red strokes in Fig. <xref rid="Fig2" ref-type="fig">2b</xref>
) that represents the region the physician wants to include as an organ and the background strokes (as the four blue strokes in Fig. <xref rid="Fig2" ref-type="fig">2b</xref>
) that distinguishes the areas which should not be included in the organ contour. The algorithm then computes the segmentation volume. With strokes interaction, physicians may indicate the region of interest by drawing a line or placing some dots, and it is expected that the physical and mental demands are lower than using the contour method. However, compared to contour method, strokes method is not widely used in radiotherapy.</p>
<p>In order to make a valid comparison of the effects of user interactions in using SAS methods, the second type of workflow presented in section 2 was adopted in both methods as Fig. <xref rid="Fig3" ref-type="fig">3</xref>
. The reason for using this workflow is to maximally preserve the combined effects of HCI and the algorithm. If manual modifications were allowed, then the quality of the outcome would be hard to judge, as it would be unclear whether it was produced by the SAS method or manual modifications. In the workflow, after the physician loads a new dataset, he/she can choose either the contour or the strokes method to segment the organ. Physicians can perform actions on axial, sagittal, or coronal planes with the help of HCI tools. The physician may scroll through all the slices, provide certain input on the desired slices and modify until a satisfied input for the algorithm is achieved. Then the physician runs the algorithm with the provided input and evaluates the outcome. If the outcome is not satisfactory, the physician may re-define the inputs of the algorithm and re-run the segmentation process. Maximally five iterations for each organ were given to the physician and if the result is not satisfactory after the fifth iteration, the segmentation is considered to be unsuccessful.<fig id="Fig3"><label>Fig. 3</label>
<caption><p>Workflow of the proposed SAS methods</p>
</caption>
<graphic xlink:href="10278_2015_9839_Fig3_HTML" id="MO3"></graphic>
</fig>
</p>
<p>A prototype of both SAS methods was developed as a plug-in on the medical imaging and interaction toolkit (MITK) platform, version 2013.09.0 [<xref ref-type="bibr" rid="CR45">45</xref>
]. For both SAS methods, a combination of graph-cut and watershed-based algorithms was developed by Dolz et al. [<xref ref-type="bibr" rid="CR46">46</xref>
, <xref ref-type="bibr" rid="CR47">47</xref>
], and was implemented as the computational part in the prototype. Figure <xref rid="Fig2" ref-type="fig">2</xref>
shows screenshots of two methods in the prototype. The left window of the display contains the data manager, which allows the physician to select and view the dataset. The main rendering window is presented at the center with four quadrants, three of them displaying different orthogonal views. The bottom right quadrant shows the segmentation result as a 3D rendering. 2D HCI inputs can be performed in the axial, the coronal, and the sagittal view with a mouse. <italic>Tools</italic>
which can be used for drawing and modifications are on the right side of the interface. In the contour method, a “free hand” tool can be selected by clicking the “add” button on the interface. Besides, physicians can also use a “paint” (paintbrush) tool, with adjustable brush size. In the strokes method, the accuracy requirement of the interaction is not high, thus the “paint” was the only tool that was provided.</p>
<p>Similar to the prototype developed by Heckel et al. [<xref ref-type="bibr" rid="CR21">21</xref>
], the prototype used in this research is designed in such a way that physicians can give their inputs in any orthogonal planes. Currently in clinical practice, physicians often use only axial view to give their inputs and the other views are often used to check if the segmentation result is satisfactory. By giving the freedom to draw in any orthogonal planes, physicians may choose the plane which requires few HCI. For instance, when segmenting the spinal cord, physicians can segment in the sagittal or coronal planes. It is expected that this design may reduce the number of user inputs, as well as the time taken for drawing the contours/strokes due to fewer slices.</p>
</sec>
<sec id="Sec4"><title>User Testing Setup and Protocol</title>
<p>For a better preparation of user testing, a series of evaluations were performed as shown in Fig. <xref rid="Fig4" ref-type="fig">4</xref>
. The evaluation started with functional testing. Functional testing refers to the test of computational algorithms to evaluate their stability and accuracy. Only after a satisfactory functional testing, usability inspection was performed. Problems identified in the usability inspection were also reported to the developers. Once the issues were fixed, a pilot study [<xref ref-type="bibr" rid="CR48">48</xref>
] was conducted to: (a) verify the experimental setup and protocols; (b) overcome the learning curve of physicians, especially for using the strokes method and giving input in different orthogonal planes. After testing the protocols, the case studies were performed and measurements regarding the process and result were collected.<fig id="Fig4"><label>Fig. 4</label>
<caption><p>The evaluation methods applied in this research</p>
</caption>
<graphic xlink:href="10278_2015_9839_Fig4_HTML" id="MO4"></graphic>
</fig>
</p>
<sec id="Sec5"><title>Materials</title>
<p>The pilot study was conducted at the Department of Radiation Oncology, University Medical Center Freiburg, Freiburg, Germany in February 2014. Table <xref rid="Tab1" ref-type="table">1</xref>
presents the materials used for pilot and follow-up case studies. Utilization of the datasets for this study was approved by the Ethics Committee of the University Medical Center, Freiburg. Before the test, a senior physician was asked to manually segment the organs in each dataset and the outcomes were used as the reference standards.<table-wrap id="Tab1"><label>Table 1</label>
<caption><p>Materials used in the pilot testing and case studies</p>
</caption>
<table frame="hsides" rules="groups"><thead><tr><th></th>
<th>Pilot testing</th>
<th>Case studies</th>
<th>Details</th>
</tr>
</thead>
<tbody><tr><td>Time</td>
<td>February 2014</td>
<td>May 2014 and August 2014</td>
<td></td>
</tr>
<tr><td>Datasets</td>
<td>7 datasets (lung region) who underwent planning CT</td>
<td>5 datasets (lung region) who underwent planning CT</td>
<td>All the five datasets were acquired on a Philips® Gemini TF Big Bore PET/CT. Every scan was taken based on the lung protocol followed in the University Medical Center Freiburg, Germany.</td>
</tr>
<tr><td>Participants</td>
<td>2 physicians</td>
<td>2 physicians (P1, P2)</td>
<td>Clinicians with 7.5 years and 5 years of experience respectively, both from University Medical Center Freiburg, Germany.</td>
</tr>
<tr><td>Types of SAS methods</td>
<td>Strokes only</td>
<td>Strokes and contour</td>
<td></td>
</tr>
<tr><td>Number of organs to be segmented</td>
<td>Spinal cord, lung, heart, trachea and proximal bronchial tree (5 organs)</td>
<td>Spinal cord, lung, heart, trachea and oesophagus (5 organs)</td>
<td>Each physician contoured 42 (21 + 21) case studies using both methods. Due to time constrains the lung and oesophagus were segmented only in 3 datasets and rest of the organs were segmented in 5 datasets</td>
</tr>
</tbody>
</table>
</table-wrap>
</p>
</sec>
<sec id="Sec6"><title>Test Setup and Protocol</title>
<p>The case studies were also conducted at the Department of Radiation Oncology, University Medical Center Freiburg, Freiburg, Germany in May 2014 and August 2014. The same study was conducted twice to assess the reproducibility of the findings. Figure <xref rid="Fig5" ref-type="fig">5</xref>
shows the experimental setup. In the experiment, the prototype was installed on a laptop. The laptop display (Screen 1) was mirrored on a 22-inch monitor (Screen 2), which is the screen size that physicians are familiar with. A camera was setup in front of the laptop screen to record the complete interaction process.<fig id="Fig5"><label>Fig. 5</label>
<caption><p>Setup of the user test</p>
</caption>
<graphic xlink:href="10278_2015_9839_Fig5_HTML" id="MO5"></graphic>
</fig>
</p>
<p>Prior to the study, both physicians signed an informed consent form. Subsequently, physicians were informed that this prototype has two SAS methods, and the designed user interactions in the prototype were explained. During the user testing, physicians were given 10 min to get familiarized with the prototype. The sequence of the segmentation task was performed based on the types of organ, i.e., physicians were asked to segment one organ for all the cases using both SAS methods and afterwards, physicians moved to the next type of organ. In case of uncertainty regarding the anatomical extension of the organs, a Radiation Therapy Oncology Group (RTOG) [<xref ref-type="bibr" rid="CR49">49</xref>
] atlas was provided. As the user interface was new for the physicians compared to their daily work, a flow chart of the workflow was provided as well.</p>
</sec>
<sec id="Sec7"><title>Evaluation Method and Measures</title>
<p>As the main objectives of this study are to identify the relations of the HCI process and the quality of the result, the presented evaluation of the SAS methods focused on two aspects: (1) measurement of process (HCI actions) and (2) measurement of the result. For both of them, subjective and objective measures were deployed.<list list-type="alpha-lower"><list-item><p>Objective measure of the process</p>
<p>In the evaluation of the HCI process, efficiency was measured from two different aspects: (1) the time taken for performing interaction and (2) the thinking/scrolling time. The time necessary for interaction is related to the physical workload, while thinking or scrolling is related to the cognitive workload of physicians during segmentation. These two measures were identified from the video analysis. From the video analysis, we also acquired data regarding other objective measures, such as interaction patterns. The interaction patterns in this experiment refer to the order of selection of the slice for segmenting, tools selection, and selection of different orthogonal planes. This provided insights whether there were any variations in the interaction patterns and if that variation was associated with the segmentation result.</p>
</list-item>
<list-item><p>Subjective measure of the process</p>
<p>In the experiment, the NASA-TLX questionnaire was used in each case study to determine the physical demand, mental demand, temporal demand, performance, effort, and frustration of the physician from a subjective point of view. The NASA-TLX [<xref ref-type="bibr" rid="CR50">50</xref>
] is a self-reported subjective technique for assessing mental workload and was developed by NASA.</p>
</list-item>
<list-item><p>Objective measure of the result</p>
<p>For each type of user interaction, the Dice similarity coefficient (DSC) [<xref ref-type="bibr" rid="CR51">51</xref>
] between the outcome and the reference standard was computed to measure the accuracy of the segmentation result. Dice similarity coefficient is denoted as <italic>S</italic>
= 2<italic>c</italic>
/(<italic>a</italic>
+ <italic>b</italic>
), where <italic>a</italic>
is the volume of segmentation result, <italic>b</italic>
is the volume of the reference standard and <inline-formula id="IEq1"><alternatives><tex-math id="M1">\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$ c $$\end{document}</tex-math>
<mml:math id="M2"><mml:mi>c</mml:mi>
</mml:math>
<inline-graphic xlink:href="10278_2015_9839_Article_IEq1.gif"></inline-graphic>
</alternatives>
</inline-formula>
is the intersection of <italic>a</italic>
and <italic>b</italic>
. Besides, the Wilcoxon-signed rank test was also used to find out if there are any statistically significant differences in the result.</p>
</list-item>
<list-item><p>Subjective measure of the result</p>
<p>A semi-structured interview was conducted at the end of the testing to find out if physicians were satisfied with the result and also to find out about the preference of the two SAS methods.</p>
</list-item>
<list-item><p>Correlations of subjective and objective measures regarding the process and the result</p>
<p>Table <xref rid="Tab2" ref-type="table">2</xref>
lists the subjective and objective measures that were used in the presented research. To gain additional insights, correlations between the process and the result measures were computed using the Pearson product–moment correlation coefficient. These correlations could be (1) correlated; (2) inversely correlated; or (3) not correlated. This study considered 0.7–0.99 as strongly correlated, 0.4–0.69 as moderately correlated, and 0.1–0.39 as weakly correlated [<xref ref-type="bibr" rid="CR52">52</xref>
].<table-wrap id="Tab2"><label>Table 2</label>
<caption><p>Subjective and objective measures of the process and the result</p>
</caption>
<table frame="hsides" rules="groups"><thead><tr><th></th>
<th>Objective</th>
<th>Subjective</th>
</tr>
</thead>
<tbody><tr><td>Process</td>
<td>Drawing time<break></break>
Scrolling/thinking time<break></break>
Use of tools<break></break>
Drawing</td>
<td>NASA-TLX questionnaire (mental demand, physical demand, temporal demand, performance, effort, frustration)</td>
</tr>
<tr><td>Results</td>
<td>Dice coefficient</td>
<td>Subjective preference</td>
</tr>
</tbody>
</table>
</table-wrap>
</p>
</list-item>
</list>
</p>
</sec>
</sec>
<sec id="Sec8" sec-type="results"><title>Results</title>
<p>In this section, the subjective and objective measures of both the process and results are presented. In total, 42 segmentation results from physicians were compared. Out of 42 cases, 18 segmentations were rejected by physicians because of unsatisfactory outcomes. In the rejected segmentations, 14 were using the contour interaction method, while the rest, four were using the strokes interaction method.</p>
<sec id="Sec9"><title>Drawing and Scrolling time of the Strokes and the Contour Methods</title>
<p>The drawing time of both physicians is shown in Table <xref rid="Tab3" ref-type="table">3</xref>
. When the two methods were compared against each other for both physicians, lung segmentation showed significant difference in drawing time (<italic>p</italic>
= 0.0007, <italic>p</italic>
= 0.0001) using the Wilcoxon two sampled test, where the strokes method was much faster than the contour method. Even though there was difference in the mean oesophagus segmentation time for physician 2, there was no statistical difference (<italic>p</italic>
= 0.7, <italic>p</italic>
= 0.5). It was found that the mean drawing time for physician 2 was always higher than for physician 1 in all the cases with both methods. In addition the contour method took longer time than strokes method in almost all cases.<table-wrap id="Tab3"><label>Table 3</label>
<caption><p>The drawing and scrolling time (in seconds) of physicians’ using the strokes and the contour methods</p>
</caption>
<table frame="hsides" rules="groups"><thead><tr><th></th>
<th></th>
<th colspan="2">Physician 1</th>
<th colspan="2">Physician 2</th>
</tr>
<tr><th>Organs</th>
<th></th>
<th>Strokes (s)</th>
<th>Contour (s)</th>
<th>Strokes (s)</th>
<th>Contour (s)</th>
</tr>
</thead>
<tbody><tr><td>SC</td>
<td>Drawing time</td>
<td>71 ± 10</td>
<td>135 ± 20</td>
<td>135 ± 15</td>
<td>157 ± 40</td>
</tr>
<tr><td></td>
<td>Scrolling time</td>
<td>91 ± 30</td>
<td>342 ± 21</td>
<td>151 ± 26</td>
<td>191 ± 51</td>
</tr>
<tr><td>Lungs</td>
<td>Drawing time</td>
<td>91 ± 8</td>
<td>554 ± 98</td>
<td>95 ± 12</td>
<td>1256 ± 176</td>
</tr>
<tr><td></td>
<td>Scrolling time</td>
<td>106 ± 14</td>
<td>116 ± 13</td>
<td>143 ± 10</td>
<td>790 ± 241</td>
</tr>
<tr><td>Heart</td>
<td>Drawing time</td>
<td>136 ± 15</td>
<td>196 ± 32</td>
<td>209 ± 30</td>
<td>216 ± 31</td>
</tr>
<tr><td></td>
<td>Scrolling time</td>
<td>155 ± 19</td>
<td>244 ± 32</td>
<td>143 ± 48</td>
<td>222 ± 33</td>
</tr>
<tr><td>Trachea</td>
<td>Drawing time</td>
<td>127 ± 21</td>
<td>153 ± 7</td>
<td>184 ± 36</td>
<td>192 ± 43</td>
</tr>
<tr><td></td>
<td>Scrolling time</td>
<td>72 ± 34</td>
<td>149 ± 15</td>
<td>162 ± 28</td>
<td>149 ± 49</td>
</tr>
<tr><td>Oesophagus</td>
<td>Drawing time</td>
<td>258 ± 89</td>
<td>225 ± 56</td>
<td>400 ± 127</td>
<td>300 ± 36</td>
</tr>
<tr><td></td>
<td>Scrolling time</td>
<td>193 ± 74</td>
<td>473 ± 29</td>
<td>320 ± 183</td>
<td>434 ± 62</td>
</tr>
</tbody>
</table>
</table-wrap>
</p>
<p>Table <xref rid="Tab3" ref-type="table">3</xref>
shows the average scrolling time of strokes and the contour methods for both physicians, respectively. The scrolling time for segmenting the spinal cord with the strokes method was statistically significant different between physician 1 and 2 (<italic>p</italic>
= 0.0002). For the rest of the organs, there was no statistically significant difference in using both methods. When the two methods were compared against each other, the time spent in segmenting the trachea has statistically significant difference for physician 1 (<italic>p</italic>
= 0.04) and the time spent in segmenting the spinal cord and segmenting the lung showed statistically significant differences for physician 2 (<italic>p</italic>
= 0.03, <italic>p</italic>
= 0.008). Even though there was difference in the mean segmentation time for other organs, it is not statistically significant.</p>
</sec>
<sec id="Sec10"><title>Interaction Pattern</title>
<p>The interaction pattern of contour and the strokes method were analyzed during the first initialization step for both physicians. A consistent drawing pattern was observed for physician 2 in using both methods. For instance, in the use of the strokes method, physician 2 often drew on the first slice, the middle, and the last slices of the dataset. Therefore, the physician only interacted with three or maximum four slices during the first initialization. In using the contour method for heart segmentation, physician 2 contoured in every third or fourth slice, which is similar to the pattern observed in using the strokes method. In the case of segmenting the trachea using the contour method, physician 2 performed interactions in every other fifth slice. Even though physician 1 often started with the center slice, no real pattern could be observed in segmenting the spinal cord, lung, trachea, and oesophagus. In the process of segmenting the heart, physician 1 placed five strokes in almost all the cases. The five strokes were drawn in such a way that the physician started the segmentation on the middle slice in all the cases, moving afterwards to one of the extremes. Figure <xref rid="Fig6" ref-type="fig">6</xref>
shows the interaction pattern observed for both physicians of using the contour method in segmenting the spinal cord. In the figure, the horizontal axis represents the intervals between the consecutive slices in which physicians performed interactions, the vertical axis indicates the frequency. The blue bar represents physician 1 while the green represents physician 2. It can be seen in Fig. <xref rid="Fig6" ref-type="fig">6</xref>
that almost for every 12 slice intervals, physician 2 performed interactions by drawing the succeeding contours. It was also found that physician 1 did not have a clear drawing pattern and the interactions were performed in random slices.<fig id="Fig6"><label>Fig. 6</label>
<caption><p>Interaction pattern of the contour method to segment the spinal cord during the initialization</p>
</caption>
<graphic xlink:href="10278_2015_9839_Fig6_HTML" id="MO6"></graphic>
</fig>
</p>
</sec>
<sec id="Sec11"><title>Usage of the HCI Tools</title>
<p>The strokes method had only one tool for drawing. Most of the time spent by physicians was related to a consistent usage of this tool, i.e., physicians placed the foreground strokes first and subsequently the background. When advancing to the next slice they started with the background, followed by the foreground to save time. For the contour method, both the “free hand” tool and the “paint” tool are used. The time spent in using the “paint” tool was 3–4 s less than the “free hand” tool in each slice for both physicians. It is also worth mentioning that both physicians needed a certain amount of time to get acquainted with the tools. For instance, physician 1 used the “free hand” tool to correct the segmented boundary where as the “paint” tool was more efficient for this action.</p>
</sec>
<sec id="Sec12"><title>NASA Task Load Questionnaire</title>
<p>The NASA task load (NASA-TLX) indices of workload associated with each case study for each physician are shown in Fig. <xref rid="Fig7" ref-type="fig">7</xref>
. In the figure, the horizontal axis indicates the different types of workload and the vertical axis shows the demand levels ranging from 0–100. For physician 1, the overall workload for oesophagus segmentation was found to be the highest with both methods. For physician 2, the overall workload for segmenting the spinal cord was highest for the strokes method and for the contour method, though the workload for segmenting the lung was highest; there was only a marginal difference over the workload of segmenting the oesophagus.<fig id="Fig7"><label>Fig. 7</label>
<caption><p>NASA task load of physician 1 (<bold>a</bold>
) and physician 2(<bold>b</bold>
). Contour method is indicated using <italic>light color</italic>
and strokes method using <italic>dark color</italic>
for various segmented organs (<italic>blue</italic>
spinal cord, <italic>pink</italic>
lungs, <italic>yellow</italic>
heart, <italic>brown</italic>
trachea, and <italic>green</italic>
oesophagus)</p>
</caption>
<graphic xlink:href="10278_2015_9839_Fig7_HTML" id="MO7"></graphic>
</fig>
</p>
</sec>
<sec id="Sec13"><title>Physicians’ Subjective Preference</title>
<p>Table <xref rid="Tab4" ref-type="table">4</xref>
shows the subjective preferences of the physicians for each method in experiment 1. The only difference between experiment 1 and 2 is that physician 2 also preferred the strokes method for the spinal cord in experiment 2.<table-wrap id="Tab4"><label>Table 4</label>
<caption><p>Physicians’ subjective preference</p>
</caption>
<table frame="hsides" rules="groups"><thead><tr><th>Organs</th>
<th>Physician 1</th>
<th>Physician 2</th>
</tr>
</thead>
<tbody><tr><td>Spinal cord</td>
<td>Strokes</td>
<td>Contour</td>
</tr>
<tr><td>Lung</td>
<td>Strokes</td>
<td>Strokes</td>
</tr>
<tr><td>Heart</td>
<td>Strokes or contour</td>
<td>Strokes or contour</td>
</tr>
<tr><td>Trachea</td>
<td>Contour</td>
<td>Contour</td>
</tr>
<tr><td>Oesophagus</td>
<td>Contour</td>
<td>Contour</td>
</tr>
</tbody>
</table>
</table-wrap>
</p>
</sec>
<sec id="Sec14"><title>Dice Similarity Coefficients of the Result</title>
<p>Using the reference standards of each organ, the Dice similarity coefficients of all the organs segmented in experiment 1 are computed as shown in Table <xref rid="Tab5" ref-type="table">5</xref>
. P1S indicates physician 1 using the strokes method and P1C refers to physician 1 using the contour method. The Dice similarity coefficients of the spinal cord, the lung, and the heart are higher than 0.8 in almost all cases.<table-wrap id="Tab5"><label>Table 5</label>
<caption><p>Dice similarity coefficient of experiment 1</p>
</caption>
<table frame="hsides" rules="groups"><thead><tr><th></th>
<th colspan="4">Spinal cord</th>
<th colspan="4">Lung</th>
<th colspan="4">Heart</th>
<th colspan="4">Trachea</th>
<th colspan="4">Oesophagus</th>
</tr>
<tr><th>Dataset</th>
<th>P1S</th>
<th>P2S</th>
<th>P1C</th>
<th>P2C</th>
<th>P1S</th>
<th>P2S</th>
<th>P1C</th>
<th>P2C</th>
<th>P1S</th>
<th>P2S</th>
<th>P1C</th>
<th>P2C</th>
<th>P1S</th>
<th>P2S</th>
<th>P1C</th>
<th>P2C</th>
<th>P1S</th>
<th>P2S</th>
<th>P1C</th>
<th>P2C</th>
</tr>
</thead>
<tbody><tr><td>Pt 01</td>
<td>0.89</td>
<td>0.87</td>
<td>0.88</td>
<td>0.87</td>
<td>0.97</td>
<td>0.97</td>
<td>0.72</td>
<td>0.97</td>
<td>0.93</td>
<td>0.93</td>
<td>0.93</td>
<td>0.94</td>
<td>0.61</td>
<td>0.62</td>
<td>0.68</td>
<td>0.62</td>
<td>0.75</td>
<td>0.64</td>
<td>0.44</td>
<td>0.29</td>
</tr>
<tr><td>Pt 02</td>
<td>0.87</td>
<td>0.86</td>
<td>0.87</td>
<td>0.86</td>
<td>0.95</td>
<td>0.95</td>
<td>0.94</td>
<td>0.94</td>
<td>0.90</td>
<td>0.90</td>
<td>0.90</td>
<td>0.91</td>
<td>0.61</td>
<td>0.63</td>
<td>0.68</td>
<td>0.60</td>
<td>0.66</td>
<td>0.68</td>
<td>0.22</td>
<td>0.47</td>
</tr>
<tr><td>Pt 03</td>
<td>0.84</td>
<td>0.85</td>
<td>0.84</td>
<td>0.26</td>
<td>0.95</td>
<td>0.96</td>
<td>0.96</td>
<td>0.39</td>
<td>0.93</td>
<td>0.93</td>
<td>0.93</td>
<td>0.94</td>
<td>0.57</td>
<td>0.57</td>
<td>0.69</td>
<td>0.33</td>
<td>0.75</td>
<td>0.69</td>
<td>0.49</td>
<td>0.33</td>
</tr>
<tr><td>Pt 04</td>
<td>0.88</td>
<td>0.88</td>
<td>0.88</td>
<td>0.88</td>
<td>0.98</td>
<td>0.98</td>
<td></td>
<td></td>
<td>0.93</td>
<td>0.93</td>
<td>0.94</td>
<td>0.90</td>
<td>0.71</td>
<td>0.62</td>
<td>0.48</td>
<td>0.54</td>
<td></td>
<td></td>
<td></td>
<td></td>
</tr>
<tr><td>Pt 05</td>
<td>0.90</td>
<td>0.88</td>
<td>0.72</td>
<td>0.89</td>
<td>0.98</td>
<td>0.97</td>
<td></td>
<td></td>
<td>0.95</td>
<td>0.92</td>
<td>0.94</td>
<td>0.58</td>
<td>0.63</td>
<td>0.69</td>
<td>0.73</td>
<td>0.66</td>
<td></td>
<td></td>
<td></td>
<td></td>
</tr>
</tbody>
</table>
</table-wrap>
</p>
</sec>
<sec id="Sec15"><title>Correlations</title>
<p>Table <xref rid="Tab6" ref-type="table">6a–c</xref>
shows the correlations between measures of the HCI process and performance criteria of the segmentation. For the nine quantitative measures used in the HCI process evaluation, we paired each measure to the others for both types of interaction. A total of 36 pairs were identified for each method. The Pearson correlation coefficient of those pairs is presented in Table <xref rid="Tab6" ref-type="table">6a, b</xref>
, regarding the contour method and the strokes methods, respectively. Among those pairs of measures, 20 of contour method and 22 of strokes methods were strongly or moderately correlated, either directly or inversely.<table-wrap id="Tab6"><label>Table 6</label>
<caption><p>Correlations among different measures in using the contour and the strokes methods</p>
</caption>
<graphic xlink:href="10278_2015_9839_Tab6_HTML" id="MO8"></graphic>
<table-wrap-foot><p>(a) The correlations of using the contour method. Green: strongly correlated, light green: inversely strongly correlated, orange: moderately correlated and light orange: inversely moderately correlated</p>
<p>(b) The correlations of using the strokes method. Green strongly correlated, light green inversely strongly correlated, orange moderately correlated and light orange inversely moderately correlated</p>
<p>(c) List of correlated measures</p>
</table-wrap-foot>
</table-wrap>
</p>
<p>Table <xref rid="Tab6" ref-type="table">6c</xref>
shows the correlations of non-quantifiable pairs. A total of ten non-quantifiable pairs were identified for both interactions. The first three pairs are subjective and objective measures in the process and the remaining seven are paired between measures in the process and the result.</p>
</sec>
</sec>
<sec id="Sec16" sec-type="discussion"><title>Discussion</title>
<p>In this study, we identified the impact of user interactions on the segmentation result using two interactive segmentation methods. The user interactions were evaluated subjectively and objectively.</p>
<sec id="Sec17"><title>The Use of Correlated Measures</title>
<p>Table <xref rid="Tab2" ref-type="table">2</xref>
provides both objective and subjective measures for evaluating the interactive segmentation procedure. We correlated those measures and identified the strong, moderate, and weakly correlated pairs. With the paired combinations, it is possible to identify how much effect the designed user interaction has on the HCI process and result. Also the correlated measures provide insights that can be used in improving user interaction design. For example, based on the correlated measures, it was clear that mental demand, physical demand. and temporal demand are correlated to the efforts in both types of interactions and efforts have a direct correlation with frustration. In the use of the contour method, it was observed that frustration and the Dice similarity coefficient are inversely correlated. Hence, efforts and frustration of the users affect the segmentation outcome, as the Dice coefficients represent the quality of the outcome. Thus in future design, the demands of physicians regarding these two aspects should be as low as possible in order to achieve a satisfactory segmentation procedure.</p>
</sec>
<sec id="Sec18"><title>The Workflow</title>
<p>In the study design, it was assumed that the mental demand is related to the scrolling or thinking time. However, this cannot be verified in this study using the correlations. The physicians indicated that with this prototype they scrolled more than they did in the clinical practice. This could be due to the workflow design. In each iteration of the proposed methods, physicians need to scroll through the entire dataset to evaluate the result and if the result was not satisfactory, they had to scroll again to give the inputs. This should be taken into consideration in the future design of the workflow of SAS methods, as increased mental demands will lead to increased efforts. One way of avoiding this is by showing the result on the current slice and by predicting the result in the next slice. In this way physicians can correct the outcome while scrolling through slices. Another way could be to use the third type of SAS workflows as mentioned in section 2. Using this workflow, the algorithm will re-compute the result only in a small region, rather than re-computing for the whole dataset. It may reduce the processing time of the algorithm and physicians only need to visualize the result in limited regions. However, it will take physicians extra efforts in specifying those “problem” regions.</p>
</sec>
<sec id="Sec19"><title>The Tools</title>
<p>In the experiment, it was identified that the choice of the interaction tools has some effects on the drawing time, e.g., using the “paint” tool the segmentation time was less than using the “free hand” tool. However, this study did not identify any correlation between the use of a certain tool and the physical demand or efforts. It was also identified that physicians used combinations of different tools while segmenting, for instance, one tool for drawing and another one to adjust the boundary. However, frequent shifting between tools is considered cognitively demanding. Thus, the usage of combined tools may lead to undesired effects, such as an increase in the drawing time and higher mental demand. Hence in the future design, providing a suitable tool for continuous usage is necessary.</p>
</sec>
<sec id="Sec20"><title>Drawing Pattern</title>
<p>The drawing pattern was another measure which was evaluated in this study. However, only for one physician we identified some systematic drawing patterns. As there was no statistically significant difference on the Dice similarity coefficient we could not conclude that the drawing pattern affects segmentation result.</p>
</sec>
<sec id="Sec21"><title>Subjective Preference</title>
<p>The subjective preferences of both physicians were the same for all the cases except in segmenting the spinal cord. In segmenting the spinal cord and the lung, the subjective preferences of the physicians were directly correlated with the NASA-TLX indices. For instance, in segmenting spinal cord, physician 2 mentioned that it was easier to draw only contours rather than drawing both the foreground and background strokes. In the case of segmenting the heart, physicians did not give a concrete preference. They felt that with the strokes method they need to increase the size of their paint brush and should contour the whole region in order to get the perfect shape of the organ. Another finding is that when segmenting organs such as the trachea and the oesophagus, physicians always included the cartilage or the organ wall in their clinical routine. In the developed two SAS methods, this was only possible with the contour method. The strokes method was only able to detect the empty volume insides the trachea and the oesophagus. This made the physicians prefer using the contour method for the trachea and the oesophagus, which is different from other OARs. This finding confirms that in the development of the SAS method, physicians should be engaged in the early stage of the development as indicated by Freudenthal et al. [<xref ref-type="bibr" rid="CR53">53</xref>
].</p>
</sec>
<sec id="Sec22"><title>Differences in Using the Strokes and the Contour Methods</title>
<p>Most of the correlations were nearly the same for using either the contour or the strokes method. However, there are exceptions. One major difference is that drawing time and the subjective performance measure from the questionnaire are strongly correlated for using the strokes method, but not for contour. Also we noticed that the drawing time and efforts are strongly correlated in the use of the strokes method. From Table <xref rid="Tab3" ref-type="table">3</xref>
it can be seen that the drawing time is less for the strokes interaction in almost all the cases except for segmenting the oesophagus. This concludes that the strokes method was more efficient and effective than contour method. However it was mentioned by the physicians during the experiment that the cognitive demand of drawing background strokes are higher than drawing foreground strokes. In some case, this higher cognitive demand shifted their preference from using the strokes method to the contour method.</p>
<p>Different from the study conducted by Yurko et al. [<xref ref-type="bibr" rid="CR54">54</xref>
], our study did not show a strong correlation between mental demand and performance. From Fig. <xref rid="Fig7" ref-type="fig">7</xref>
, it is clear that the frustration level of the contour methods is always higher than the strokes method. Also the frustration level and the Dice similarity coefficient were inversely correlated in using the contour method. With the inverse correlation and from Table <xref rid="Tab5" ref-type="table">5</xref>
, it can be seen that outcomes from the contour method are not as good as the strokes method for all the cases and the mental demand, performance and effort were low in using the strokes method. Hence, strokes can be considered as a preferred interaction in future prototypes.</p>
</sec>
<sec id="Sec23"><title>Limitations</title>
<p>First, only two participants were included in the study, which limits the study regarding inter-observer variation. Second, due to the novelty of the strokes method, we only introduced mouse as the HCI device. If new input devices were introduced, it would have been difficult to identify the cause of changes in the process and the result. Coen [<xref ref-type="bibr" rid="CR39">39</xref>
] discussed that HCI input devices may also influence the segmentation. Thus, different types of input devices should be considered after physicians are familiar with this method.</p>
</sec>
</sec>
<sec id="Sec24" sec-type="conclusion"><title>Conclusion</title>
<p>In the proposed research, we investigated the role of user interaction in SAS methods for segmenting the organs at risk in radiotherapy planning. In total, 42 case studies were conducted on five organs with two different SAS methods. Thirty-six quantifiable and ten non-quantifiable correlations were identified for each interaction. Among those pairs of measures, 20 of the contour method and 22 of the strokes methods were strongly or moderately correlated, either directly or inversely. Those correlated measures helped us to confirm that besides the performance of the algorithm, the quality of the segmentation also depends on the physician and the HCI process. Furthermore, the direct and inverse correlated measures provide useful insights for future user interaction design in interactive segmentation. Among the two developed SAS methods, it is clear that the strokes method is more efficient, less cognitively demanding, and requires less effort than the contour method. However it is hard to replace physicians’ subjective preference since cognitively, drawing a contour at the boundary ensures correct segmentation of organs and drawing background strokes was more cognitively demanding. Besides, it is also identified that random and regular drawing pattern did not influence the quality of the result and the duration of the process. These findings suggest that in the future HCI design of SAS methods, user interactions need to be less cognitively challenging and there is a need for flexibility in the interface design.</p>
<p>Current research is directed towards further development of the HCI designs of SAS methods. More HCI devices, for instance, pen-tablet and touch screen, will be introduced to the study. New measures, such as eye tracking, will be introduced as well. The results from the current study will be used to design novel HCI tools in the future studies to improve the effectiveness and efficiency of user interaction.</p>
</sec>
</body>
<back><ack><p>The presented research is part of Software for the Use of Multi-Modality images in External Radiotherapy (SUMMER) project which is funded by European Commission (FP7-PEOPLE-2011-ITN) under grant agreement PITN-GA-2011-290148. The authors would like to express their appreciations to other members of the SUMMER consortium for their valuable advices regarding the proposed research.</p>
</ack>
<notes notes-type="conflict-interest"><title>Compliance with Ethical Standards</title>
<sec id="FPar1"><title>Ethical Approval</title>
<p>Utilization of the datasets for this study was approved by the Ethics Committee of the University Medical Center, Freiburg.</p>
</sec>
<sec id="FPar2"><title>Consent to Participate</title>
<p>Prior to the study, both physicians signed an informed consent form.</p>
</sec>
</notes>
<ref-list id="Bib1"><title>References</title>
<ref id="CR1"><label>1.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Burnet</surname>
<given-names>NG</given-names>
</name>
<name><surname>Thomas</surname>
<given-names>SJ</given-names>
</name>
<name><surname>Burton</surname>
<given-names>KE</given-names>
</name>
<name><surname>Jefferies</surname>
<given-names>SJ</given-names>
</name>
</person-group>
<article-title>Defining the tumour and target volumes for radiotherapy</article-title>
<source>Cancer Imaging: Off Publ Int Cancer Imaging Soc</source>
<year>2004</year>
<volume>4</volume>
<issue>2</issue>
<fpage>153</fpage>
<lpage>61</lpage>
<pub-id pub-id-type="doi">10.1102/1470-7330.2004.0054</pub-id>
</element-citation>
</ref>
<ref id="CR2"><label>2.</label>
<mixed-citation publication-type="other">Moltz JH, Braunewell S, Ruhaak J, et al.: Analysis of variability in manual liver tumor delineation in CT scans, IEEE International Symposium on Biomedical Imaging: From Nano to Macro, 1974–1977, 2011</mixed-citation>
</ref>
<ref id="CR3"><label>3.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Whitfield</surname>
<given-names>GA</given-names>
</name>
<name><surname>Price</surname>
<given-names>P</given-names>
</name>
<name><surname>Price</surname>
<given-names>GJ</given-names>
</name>
<name><surname>Moore</surname>
<given-names>CJ</given-names>
</name>
</person-group>
<article-title>Automated delineation of radiotherapy volumes: are we going in the right direction?</article-title>
<source>Br J Radiol</source>
<year>2013</year>
<volume>86</volume>
<issue>1021</issue>
<fpage>20110718</fpage>
<pub-id pub-id-type="doi">10.1259/bjr.20110718</pub-id>
<pub-id pub-id-type="pmid">23239689</pub-id>
</element-citation>
</ref>
<ref id="CR4"><label>4.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Heckel</surname>
<given-names>F</given-names>
</name>
<name><surname>Moltz</surname>
<given-names>JH</given-names>
</name>
<name><surname>Tietjen</surname>
<given-names>C</given-names>
</name>
<name><surname>Hahn</surname>
<given-names>HK</given-names>
</name>
</person-group>
<article-title>Sketch-Based Editing Tools for Tumour Segmentation in 3D Medical Images</article-title>
<source>Comput Graphics Forum</source>
<year>2013</year>
<volume>32</volume>
<issue>8</issue>
<fpage>144</fpage>
<lpage>157</lpage>
<pub-id pub-id-type="doi">10.1111/cgf.12193</pub-id>
</element-citation>
</ref>
<ref id="CR5"><label>5.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Isambert</surname>
<given-names>A</given-names>
</name>
<name><surname>Dhermain</surname>
<given-names>F</given-names>
</name>
<name><surname>Bidault</surname>
<given-names>F</given-names>
</name>
<etal></etal>
</person-group>
<article-title>Evaluation of an atlas-based automatic segmentation software for the delineation of brain organs at risk in a radiation therapy clinical context</article-title>
<source>Radiother Oncol</source>
<year>2008</year>
<volume>87</volume>
<issue>1</issue>
<fpage>93</fpage>
<lpage>99</lpage>
<pub-id pub-id-type="doi">10.1016/j.radonc.2007.11.030</pub-id>
<pub-id pub-id-type="pmid">18155791</pub-id>
</element-citation>
</ref>
<ref id="CR6"><label>6.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Wu</surname>
<given-names>K</given-names>
</name>
<name><surname>Ung</surname>
<given-names>YC</given-names>
</name>
<name><surname>Hwang</surname>
<given-names>D</given-names>
</name>
<etal></etal>
</person-group>
<article-title>Auto-contouring and manual contouring: which is the better method for target delineation using 18F-FDG PET/CT in non-small cell lung cancer?</article-title>
<source>J Nucl Med</source>
<year>2010</year>
<volume>51</volume>
<issue>10</issue>
<fpage>1517</fpage>
<lpage>23</lpage>
<pub-id pub-id-type="doi">10.2967/jnumed.110.077974</pub-id>
<pub-id pub-id-type="pmid">20847172</pub-id>
</element-citation>
</ref>
<ref id="CR7"><label>7.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Smith</surname>
<given-names>CM</given-names>
</name>
<name><surname>Smith</surname>
<given-names>J</given-names>
</name>
<name><surname>Williams</surname>
<given-names>SK</given-names>
</name>
<name><surname>Rodriguez</surname>
<given-names>JJ</given-names>
</name>
<name><surname>Hoying</surname>
<given-names>JB</given-names>
</name>
</person-group>
<article-title>Automatic thresholding of three-dimensional microvascular structures from confocal microscopy images</article-title>
<source>J Microsc</source>
<year>2007</year>
<volume>225</volume>
<issue>3</issue>
<fpage>244</fpage>
<lpage>257</lpage>
<pub-id pub-id-type="doi">10.1111/j.1365-2818.2007.01739.x</pub-id>
<pub-id pub-id-type="pmid">17371447</pub-id>
</element-citation>
</ref>
<ref id="CR8"><label>8.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Grau</surname>
<given-names>V</given-names>
</name>
<name><surname>Mewes</surname>
<given-names>AUJ</given-names>
</name>
<name><surname>Alcaniz</surname>
<given-names>M</given-names>
</name>
<name><surname>Kikinis</surname>
<given-names>R</given-names>
</name>
<name><surname>Warfield</surname>
<given-names>SK</given-names>
</name>
</person-group>
<article-title>Improved watershed transform for medical image segmentation using prior information</article-title>
<source>IEEE Trans Med Imaging</source>
<year>2004</year>
<volume>23</volume>
<issue>4</issue>
<fpage>447</fpage>
<lpage>458</lpage>
<pub-id pub-id-type="doi">10.1109/TMI.2004.824224</pub-id>
<pub-id pub-id-type="pmid">15084070</pub-id>
</element-citation>
</ref>
<ref id="CR9"><label>9.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Mondal</surname>
<given-names>T</given-names>
</name>
<name><surname>Jain</surname>
<given-names>A</given-names>
</name>
<name><surname>Sardana</surname>
<given-names>HK</given-names>
</name>
</person-group>
<article-title>Automatic craniofacial structure detection on cephalometric images</article-title>
<source>IEEE Trans Image Process</source>
<year>2011</year>
<volume>20</volume>
<issue>9</issue>
<fpage>2606</fpage>
<lpage>2614</lpage>
<pub-id pub-id-type="doi">10.1109/TIP.2011.2131662</pub-id>
<pub-id pub-id-type="pmid">21435982</pub-id>
</element-citation>
</ref>
<ref id="CR10"><label>10.</label>
<mixed-citation publication-type="other">Jin Y, Angelini E, Laine A: Wavelets in medical image processing: denoising, segmentation, and registration. Handbook 832 of biomedical image analysis, Kluwer Academic/Plenum Publishers, New York, Vol.1, Segmentation Models, Chapter 6:305–358, 2005</mixed-citation>
</ref>
<ref id="CR11"><label>11.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Sims</surname>
<given-names>R</given-names>
</name>
<name><surname>Isambert</surname>
<given-names>A</given-names>
</name>
<name><surname>Grégoire</surname>
<given-names>V</given-names>
</name>
<etal></etal>
</person-group>
<article-title>A pre-clinical assessment of an atlas-based automatic segmentation tool for the head and neck</article-title>
<source>Radiother Oncol</source>
<year>2009</year>
<volume>93</volume>
<issue>3</issue>
<fpage>474</fpage>
<lpage>478</lpage>
<pub-id pub-id-type="doi">10.1016/j.radonc.2009.08.013</pub-id>
<pub-id pub-id-type="pmid">19758720</pub-id>
</element-citation>
</ref>
<ref id="CR12"><label>12.</label>
<mixed-citation publication-type="other">Boykov Y, Jolly MP: Interactive graph cuts for optimal boundary and region segmentation of objects in N-D images. Proc. IEEE Int. Conf. Comput. Vis., 105–112, 2001</mixed-citation>
</ref>
<ref id="CR13"><label>13.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Yeo</surname>
<given-names>SY</given-names>
</name>
<name><surname>Xie</surname>
<given-names>X</given-names>
</name>
<name><surname>Sazonov</surname>
<given-names>I</given-names>
</name>
<name><surname>Nithiarasu</surname>
<given-names>P</given-names>
</name>
</person-group>
<article-title>Geometrically induced force interaction for three-dimensional deformable models</article-title>
<source>IEEE Trans Image Process</source>
<year>2011</year>
<volume>20</volume>
<issue>5</issue>
<fpage>1373</fpage>
<lpage>1387</lpage>
<pub-id pub-id-type="doi">10.1109/TIP.2010.2092434</pub-id>
<pub-id pub-id-type="pmid">21078578</pub-id>
</element-citation>
</ref>
<ref id="CR14"><label>14.</label>
<mixed-citation publication-type="other">Lee N, Smith RT, Laine AF: Interactive segmentation for geographic atrophy in retinal fundus images, Proc. 42nd Asilomar Conf. Signals, Systems and Computers, 655–658, 2008</mixed-citation>
</ref>
<ref id="CR15"><label>15.</label>
<mixed-citation publication-type="other">Story MF: FDA Human Factors Draft Guidance Document, <ext-link ext-link-type="uri" xlink:href="http://www.fda.gov/downloads/MedicalDevices/">www.fda.gov/downloads/MedicalDevices/</ext-link>
Device Regulation and Guidance/ Human Factors/UCM320901.pdf, 2011, accessed on Sep. 1, 2015</mixed-citation>
</ref>
<ref id="CR16"><label>16.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Olabarriaga</surname>
<given-names>S</given-names>
</name>
<name><surname>Smeulders</surname>
<given-names>A</given-names>
</name>
</person-group>
<article-title>Interaction in the segmentation of medical images: A survey</article-title>
<source>Med Image Anal</source>
<year>2001</year>
<volume>5</volume>
<issue>2</issue>
<fpage>127</fpage>
<lpage>142</lpage>
<pub-id pub-id-type="doi">10.1016/S1361-8415(00)00041-4</pub-id>
<pub-id pub-id-type="pmid">11516707</pub-id>
</element-citation>
</ref>
<ref id="CR17"><label>17.</label>
<mixed-citation publication-type="other">Karray F, Alemzadeh M, Saleh JA, Arab MN: Human-Computer Interaction: Overview on State of the Art, 1(1):137–159,2008</mixed-citation>
</ref>
<ref id="CR18"><label>18.</label>
<mixed-citation publication-type="other">Blake A, Rother C, Brown M, Perez P, and Torr P: Interactive image segmentation using an adaptive gmmrf model, European Conference on Computer Vision (ECCV), 428–441, 2004</mixed-citation>
</ref>
<ref id="CR19"><label>19.</label>
<mixed-citation publication-type="other">Yang W, Cai J, Member S, Zheng J, Luo J: User-friendly Interactive Image Segmentation through Unified Combinatorial User Inputs(c), 1–10, 2010</mixed-citation>
</ref>
<ref id="CR20"><label>20.</label>
<mixed-citation publication-type="other">Harders M, Member S, Székely G, Member A: Enhancing Human–Computer Interaction in Medical Segmentation, Proceedings of the IEEE, 91(9), 2003</mixed-citation>
</ref>
<ref id="CR21"><label>21.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Heckel</surname>
<given-names>F</given-names>
</name>
<name><surname>Moltz</surname>
<given-names>JH</given-names>
</name>
<name><surname>Tietjen</surname>
<given-names>C</given-names>
</name>
<name><surname>Hahn</surname>
<given-names>HK</given-names>
</name>
</person-group>
<article-title>Sketch-Based Editing Tools for Tumour Segmentation in 3D Medical Images</article-title>
<source>Comput Graphics Forum</source>
<year>2013</year>
<volume>32</volume>
<issue>8</issue>
<fpage>144</fpage>
<lpage>157</lpage>
<pub-id pub-id-type="doi">10.1111/cgf.12193</pub-id>
</element-citation>
</ref>
<ref id="CR22"><label>22.</label>
<mixed-citation publication-type="other">Barret WA, Mortensen EN: Fast, accurate, and reproducible live wire boundary extraction. Vis. Biomedical Computing, 183–192,1996</mixed-citation>
</ref>
<ref id="CR23"><label>23.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Sherbondy</surname>
<given-names>AJ</given-names>
</name>
<name><surname>Holmlund</surname>
<given-names>D</given-names>
</name>
<name><surname>Rubin</surname>
<given-names>GD</given-names>
</name>
<name><surname>Schraedley</surname>
<given-names>PK</given-names>
</name>
<name><surname>Winograd</surname>
<given-names>T</given-names>
</name>
<name><surname>Napel</surname>
<given-names>S</given-names>
</name>
</person-group>
<article-title>Alternative input devices for efficient navigation of large CT angiography data sets</article-title>
<source>Radiology</source>
<year>2005</year>
<volume>234</volume>
<fpage>391</fpage>
<lpage>398</lpage>
<pub-id pub-id-type="doi">10.1148/radiol.2342032017</pub-id>
<pub-id pub-id-type="pmid">15670996</pub-id>
</element-citation>
</ref>
<ref id="CR24"><label>24.</label>
<mixed-citation publication-type="other">Sadeghi GHHM, Tien G and Atkins MS: Hands-free interactive image segmentation using eyegaze. SPIE Medical Imaging conference, 2009</mixed-citation>
</ref>
<ref id="CR25"><label>25.</label>
<mixed-citation publication-type="other">Aselmaa A, Goossens RH, Laprie A, Ken S, Fechter T, Ramkumar A, Freudenthal A: Workflow Analysis Report, <ext-link ext-link-type="uri" xlink:href="http://summer-project.eu/work/deliverables/">http://summer-project.eu/work/deliverables/</ext-link>
, 2013, accessed 1 Sep 2015</mixed-citation>
</ref>
<ref id="CR26"><label>26.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Li</surname>
<given-names>Y</given-names>
</name>
<name><surname>Sun</surname>
<given-names>J</given-names>
</name>
<name><surname>Tang</surname>
<given-names>CK</given-names>
</name>
<name><surname>Shum</surname>
<given-names>HY</given-names>
</name>
</person-group>
<article-title>Lazy snapping</article-title>
<source>ACM Trans Graph</source>
<year>2004</year>
<volume>23</volume>
<issue>3</issue>
<fpage>303</fpage>
<lpage>308</lpage>
<pub-id pub-id-type="doi">10.1145/1015706.1015719</pub-id>
</element-citation>
</ref>
<ref id="CR27"><label>27.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Yang</surname>
<given-names>W</given-names>
</name>
<name><surname>Cai</surname>
<given-names>J</given-names>
</name>
</person-group>
<article-title>User-friendly Interactive Image Segmentation through Unified Combinatorial User Inputs</article-title>
<source>IEEE Trans Image Process</source>
<year>2010</year>
<volume>19</volume>
<issue>9</issue>
<fpage>2470</fpage>
<lpage>2479</lpage>
<pub-id pub-id-type="doi">10.1109/TIP.2010.2048611</pub-id>
<pub-id pub-id-type="pmid">20716497</pub-id>
</element-citation>
</ref>
<ref id="CR28"><label>28.</label>
<mixed-citation publication-type="other">Hebbalaguppe R, McGuinness K, Kuklyte J, Healy G, O’Connor N and Smeaton A: User-Centered Computer Vision (UCCV), 1st IEEE Workshop on, 19–24, 2013</mixed-citation>
</ref>
<ref id="CR29"><label>29.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Dalah</surname>
<given-names>EZ</given-names>
</name>
<name><surname>Nisbet</surname>
<given-names>A</given-names>
</name>
<name><surname>Bradley</surname>
<given-names>D</given-names>
</name>
</person-group>
<article-title>Effect of window level on target volume delineation in treatment planning</article-title>
<source>Appl Radiat Isot</source>
<year>2010</year>
<volume>68</volume>
<fpage>602</fpage>
<lpage>604</lpage>
<pub-id pub-id-type="doi">10.1016/j.apradiso.2009.09.007</pub-id>
<pub-id pub-id-type="pmid">19828323</pub-id>
</element-citation>
</ref>
<ref id="CR30"><label>30.</label>
<mixed-citation publication-type="other">Chun H, Kwak H, Eom YH, Ahn YY, Moon S, and Jeong H: Comparison of online social relations in volume vs interaction: a case study of cyworld, Proceeding of the 8th Internet Measurement Conference (IMC’08), 2008</mixed-citation>
</ref>
<ref id="CR31"><label>31.</label>
<mixed-citation publication-type="other">Viswanath B, Mislove A, Cha M, Gummadi KP: On the evolution of user interaction in Facebook, The Second ACM SIGCOMM Workshop on Online Social Networks (WOSN’09), 2009</mixed-citation>
</ref>
<ref id="CR32"><label>32.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Ju</surname>
<given-names>W</given-names>
</name>
<name><surname>Leifer</surname>
<given-names>L</given-names>
</name>
</person-group>
<article-title>The Design of Implicit Interactions: Making Interactive Systems Less Obnoxious</article-title>
<source>Des Issues</source>
<year>2008</year>
<volume>24</volume>
<issue>3</issue>
<fpage>72</fpage>
<lpage>84</lpage>
<pub-id pub-id-type="doi">10.1162/desi.2008.24.3.72</pub-id>
</element-citation>
</ref>
<ref id="CR33"><label>33.</label>
<mixed-citation publication-type="other">ISO/IEC, 9241–11: Ergonomic requirements for office work with visual display terminals (VDT)s - Part 11 Guidance on usability, ISO/IEC 9241–11, 1998</mixed-citation>
</ref>
<ref id="CR34"><label>34.</label>
<mixed-citation publication-type="other">Nielsen J, Molich R: Heuristic evaluation of user interfaces, Proc. ACM CHI'90 Conference, 249–256, 1990</mixed-citation>
</ref>
<ref id="CR35"><label>35.</label>
<mixed-citation publication-type="other">Liu Y, Osvalder AL, Dahlman S: Exploring user background settings in cognitive walkthrough evaluation of medical prototype interfaces: a case study, International Journal of Industrial Ergonomics, 2005</mixed-citation>
</ref>
<ref id="CR36"><label>36.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Clark</surname>
<given-names>RE</given-names>
</name>
<name><surname>Pugh</surname>
<given-names>CM</given-names>
</name>
<name><surname>Yates</surname>
<given-names>KA</given-names>
</name>
<name><surname>Inaba</surname>
<given-names>K</given-names>
</name>
<name><surname>Green</surname>
<given-names>DJ</given-names>
</name>
<name><surname>Sullivan</surname>
<given-names>ME</given-names>
</name>
</person-group>
<article-title>The Use of Cognitive Task Analysis to Improve Instructional Descriptions of Procedures</article-title>
<source>J Surg Res</source>
<year>2012</year>
<volume>173</volume>
<issue>1</issue>
<fpage>e37</fpage>
<lpage>e42</lpage>
<pub-id pub-id-type="doi">10.1016/j.jss.2011.09.003</pub-id>
<pub-id pub-id-type="pmid">22099596</pub-id>
</element-citation>
</ref>
<ref id="CR37"><label>37.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Jaspers</surname>
<given-names>MW</given-names>
</name>
<name><surname>Steen</surname>
<given-names>T</given-names>
</name>
<name><surname>van den Bos</surname>
<given-names>C</given-names>
</name>
<name><surname>Geenen</surname>
<given-names>M</given-names>
</name>
</person-group>
<article-title>The think aloud method: a guide to user interface design</article-title>
<source>Int J Med Inform</source>
<year>2004</year>
<volume>73</volume>
<issue>11–12</issue>
<fpage>781</fpage>
<lpage>795</lpage>
<pub-id pub-id-type="doi">10.1016/j.ijmedinf.2004.08.003</pub-id>
<pub-id pub-id-type="pmid">15491929</pub-id>
</element-citation>
</ref>
<ref id="CR38"><label>38.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Lee</surname>
<given-names>S</given-names>
</name>
<name><surname>Koubek</surname>
<given-names>RJ</given-names>
</name>
</person-group>
<article-title>The effects of usability and web design attributes on user preference for e-commerce web sites</article-title>
<source>J Comput Ind</source>
<year>2010</year>
<volume>61</volume>
<issue>4</issue>
<fpage>329</fpage>
<lpage>341</lpage>
<pub-id pub-id-type="doi">10.1016/j.compind.2009.12.004</pub-id>
</element-citation>
</ref>
<ref id="CR39"><label>39.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Coen</surname>
<given-names>RN</given-names>
</name>
</person-group>
<article-title>Human-computer interaction in radiotherapy target volume delineation: a prospective, multi-institutional comparison of user input devices</article-title>
<source>J Digit Imaging</source>
<year>2011</year>
<volume>24</volume>
<issue>5</issue>
<fpage>794</fpage>
<lpage>803</lpage>
<pub-id pub-id-type="doi">10.1007/s10278-010-9341-2</pub-id>
<pub-id pub-id-type="pmid">20978922</pub-id>
</element-citation>
</ref>
<ref id="CR40"><label>40.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Kotani</surname>
<given-names>K</given-names>
</name>
<name><surname>Horii</surname>
<given-names>K</given-names>
</name>
</person-group>
<article-title>An analysis of muscular load and performance in using a pen–tablet system</article-title>
<source>J Physiol Anthropol</source>
<year>2003</year>
<volume>22</volume>
<fpage>89</fpage>
<lpage>95</lpage>
<pub-id pub-id-type="doi">10.2114/jpa.22.89</pub-id>
</element-citation>
</ref>
<ref id="CR41"><label>41.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>McGuinness</surname>
<given-names>K</given-names>
</name>
<name><surname>O’Connor</surname>
<given-names>NE</given-names>
</name>
</person-group>
<article-title>A comparative evaluation of interactive segmentation algorithms</article-title>
<source>Pattern Recogn</source>
<year>2010</year>
<volume>43</volume>
<issue>2</issue>
<fpage>434</fpage>
<lpage>444</lpage>
<pub-id pub-id-type="doi">10.1016/j.patcog.2009.03.008</pub-id>
</element-citation>
</ref>
<ref id="CR42"><label>42.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Annett</surname>
<given-names>J</given-names>
</name>
</person-group>
<article-title>Subjective rating scales science or art?</article-title>
<source>Ergonomics</source>
<year>2002</year>
<volume>45</volume>
<issue>14</issue>
<fpage>966</fpage>
<lpage>987</lpage>
<pub-id pub-id-type="doi">10.1080/00140130210166951</pub-id>
<pub-id pub-id-type="pmid">12569049</pub-id>
</element-citation>
</ref>
<ref id="CR43"><label>43.</label>
<mixed-citation publication-type="other">Balakrishnan R, Baudel T, Kurtenbach G, and Fitzmaurice GW: “The rockin’mouse: Integral 3D manipulation on a plane,” Proc. Conf. Human Factors Computing Systems, 311–318, 1997</mixed-citation>
</ref>
<ref id="CR44"><label>44.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Hornbæk</surname>
<given-names>K</given-names>
</name>
</person-group>
<article-title>Current practice in measuring usability: Challenges to usability studies and research</article-title>
<source>Int J Hum-Comput Stud</source>
<year>2006</year>
<volume>64</volume>
<issue>2</issue>
<fpage>79</fpage>
<lpage>102</lpage>
<pub-id pub-id-type="doi">10.1016/j.ijhcs.2005.06.002</pub-id>
</element-citation>
</ref>
<ref id="CR45"><label>45.</label>
<mixed-citation publication-type="other">The Medical Imaging Interaction Toolkit (MITK), <ext-link ext-link-type="uri" xlink:href="http://www.mitk.org/">http://www.mitk.org/</ext-link>
accessed on 1 Sep 2015</mixed-citation>
</ref>
<ref id="CR46"><label>46.</label>
<mixed-citation publication-type="other">Dolz J, Kirisli HA, Viard R, Massoptier L: Combining watershed and graph cuts methods to segment organs at risk in radiotherapy, Proc. SPIE 9034, Medical Imaging: Image Processing, 2014</mixed-citation>
</ref>
<ref id="CR47"><label>47.</label>
<mixed-citation publication-type="other">Dolz J, Kirisli HA, Viard R, Massoptier L: Interactive approach to segment organs at risk in radiotherapy treatment planning, Proc. SPIE 9034, Medical Imaging: Image Processing, 2014</mixed-citation>
</ref>
<ref id="CR48"><label>48.</label>
<mixed-citation publication-type="other">Ramkumar A, Dolz J, Kirisli HA, et al.: Human-computer interaction in segmenting organs at risk for radiotherapy: a pilot study. Multimodal imaging towards individualized radiotherapy treatments, 69–79, ISBN: 978-94-6186-309-6, 2014</mixed-citation>
</ref>
<ref id="CR49"><label>49.</label>
<mixed-citation publication-type="other">Guidelines for organs at risk delineation in thoracic radiation therapy. Based on RTOG guidelines. <ext-link ext-link-type="uri" xlink:href="http://www.rtog.org/CoreLab/ContouringAtlases/LungAtlas.aspx">http://www.rtog.org/CoreLab/ContouringAtlases/LungAtlas.aspx</ext-link>
, accessed on 1 Dec 2014</mixed-citation>
</ref>
<ref id="CR50"><label>50.</label>
<mixed-citation publication-type="other">Hart SG, Staveland LE: Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research, in: Hancock, P.A. and Meshkati, N. (Eds.) Human Mental Workload, North-Holland, 1998</mixed-citation>
</ref>
<ref id="CR51"><label>51.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Dice</surname>
<given-names>LR</given-names>
</name>
</person-group>
<article-title>Measures of the amount of ecologic association between species</article-title>
<source>Ecology</source>
<year>1945</year>
<volume>26</volume>
<issue>3</issue>
<fpage>297</fpage>
<lpage>302</lpage>
<pub-id pub-id-type="doi">10.2307/1932409</pub-id>
</element-citation>
</ref>
<ref id="CR52"><label>52.</label>
<element-citation publication-type="book"><person-group person-group-type="author"><name><surname>Dancey</surname>
<given-names>C</given-names>
</name>
<name><surname>Reidy</surname>
<given-names>J</given-names>
</name>
</person-group>
<source>Statistics without Maths for Psychology: using SPSS for Windows</source>
<year>2004</year>
<publisher-loc>London</publisher-loc>
<publisher-name>Prentice Hall</publisher-name>
</element-citation>
</ref>
<ref id="CR53"><label>53.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Freudenthal</surname>
<given-names>A</given-names>
</name>
<name><surname>Stüdeli</surname>
<given-names>T</given-names>
</name>
<name><surname>Lamata</surname>
<given-names>P</given-names>
</name>
<name><surname>Samset</surname>
<given-names>E</given-names>
</name>
</person-group>
<article-title>Collaborative co-design of emerging multi-technologies for surgery</article-title>
<source>J Biomed Inform</source>
<year>2011</year>
<volume>44</volume>
<issue>2</issue>
<fpage>198</fpage>
<lpage>215</lpage>
<pub-id pub-id-type="doi">10.1016/j.jbi.2010.11.006</pub-id>
<pub-id pub-id-type="pmid">21118727</pub-id>
</element-citation>
</ref>
<ref id="CR54"><label>54.</label>
<element-citation publication-type="journal"><person-group person-group-type="author"><name><surname>Yurko</surname>
<given-names>YY</given-names>
</name>
<name><surname>Scerbo</surname>
<given-names>MW</given-names>
</name>
<name><surname>Prabhu</surname>
<given-names>AS</given-names>
</name>
<name><surname>Acker</surname>
<given-names>CE</given-names>
</name>
<name><surname>Stefanidis</surname>
<given-names>D</given-names>
</name>
</person-group>
<article-title>Higher mental workload is associated with poorer laparoscopic performance as measured by the NASA-TLX tool. Simulation in Healthcare </article-title>
<source>J Soc Simul Healthcare</source>
<year>2010</year>
<volume>5</volume>
<issue>5</issue>
<fpage>267</fpage>
<lpage>71</lpage>
<pub-id pub-id-type="doi">10.1097/SIH.0b013e3181e3f329</pub-id>
</element-citation>
</ref>
</ref-list>
</back>
</pmc>
<affiliations><list><country><li>Allemagne</li>
<li>France</li>
<li>Pays-Bas</li>
</country>
<region><li>Bade-Wurtemberg</li>
<li>District de Fribourg-en-Brisgau</li>
<li>Hollande-Méridionale</li>
</region>
<settlement><li>Fribourg-en-Brisgau</li>
<li>Lille</li>
<li>Rotterdam</li>
</settlement>
</list>
<tree><country name="Pays-Bas"><noRegion><name sortKey="Ramkumar, Anjana" sort="Ramkumar, Anjana" uniqKey="Ramkumar A" first="Anjana" last="Ramkumar">Anjana Ramkumar</name>
</noRegion>
<name sortKey="Niessen, Wiro J" sort="Niessen, Wiro J" uniqKey="Niessen W" first="Wiro J." last="Niessen">Wiro J. Niessen</name>
<name sortKey="Niessen, Wiro J" sort="Niessen, Wiro J" uniqKey="Niessen W" first="Wiro J." last="Niessen">Wiro J. Niessen</name>
<name sortKey="Song, Yu" sort="Song, Yu" uniqKey="Song Y" first="Yu" last="Song">Yu Song</name>
<name sortKey="Stappers, Pieter Jan" sort="Stappers, Pieter Jan" uniqKey="Stappers P" first="Pieter Jan" last="Stappers">Pieter Jan Stappers</name>
<name sortKey="Varga, Edit" sort="Varga, Edit" uniqKey="Varga E" first="Edit" last="Varga">Edit Varga</name>
</country>
<country name="France"><noRegion><name sortKey="Dolz, Jose" sort="Dolz, Jose" uniqKey="Dolz J" first="Jose" last="Dolz">Jose Dolz</name>
</noRegion>
<name sortKey="Kirisli, Hortense A" sort="Kirisli, Hortense A" uniqKey="Kirisli H" first="Hortense A." last="Kirisli">Hortense A. Kirisli</name>
<name sortKey="Massoptier, Laurent" sort="Massoptier, Laurent" uniqKey="Massoptier L" first="Laurent" last="Massoptier">Laurent Massoptier</name>
</country>
<country name="Allemagne"><region name="Bade-Wurtemberg"><name sortKey="Adebahr, Sonja" sort="Adebahr, Sonja" uniqKey="Adebahr S" first="Sonja" last="Adebahr">Sonja Adebahr</name>
</region>
<name sortKey="Nestle, Ursula" sort="Nestle, Ursula" uniqKey="Nestle U" first="Ursula" last="Nestle">Ursula Nestle</name>
<name sortKey="Schimek Jasch, Tanja" sort="Schimek Jasch, Tanja" uniqKey="Schimek Jasch T" first="Tanja" last="Schimek-Jasch">Tanja Schimek-Jasch</name>
</country>
</tree>
</affiliations>
</record>
Pour manipuler ce document sous Unix (Dilib)
EXPLOR_STEP=$WICRI_ROOT/Ticri/CIDE/explor/HapticV1/Data/Ncbi/Merge
HfdSelect -h $EXPLOR_STEP/biblio.hfd -nk 003D46 | SxmlIndent | more
Ou
HfdSelect -h $EXPLOR_AREA/Data/Ncbi/Merge/biblio.hfd -nk 003D46 | SxmlIndent | more
Pour mettre un lien sur cette page dans le réseau Wicri
{{Explor lien |wiki= Ticri/CIDE |area= HapticV1 |flux= Ncbi |étape= Merge |type= RBID |clé= PMC:4788616 |texte= User Interaction in Semi-Automatic Segmentation of Organs at Risk: a Case Study in Radiotherapy }}
Pour générer des pages wiki
HfdIndexSelect -h $EXPLOR_AREA/Data/Ncbi/Merge/RBID.i -Sk "pubmed:26553109" \ | HfdSelect -Kh $EXPLOR_AREA/Data/Ncbi/Merge/biblio.hfd \ | NlmPubMed2Wicri -a HapticV1
![]() | This area was generated with Dilib version V0.6.23. | ![]() |