Much Ado About Time: Exhaustive Annotation of Temporal Data
Identifieur interne : 000389 ( Hal/Curation ); précédent : 000388; suivant : 000390Much Ado About Time: Exhaustive Annotation of Temporal Data
Auteurs : Gunnar A. Sigurdsson [États-Unis] ; Olga Russakovsky [États-Unis] ; Ali Farhadi [États-Unis] ; Ivan Laptev [France] ; Abhinav Gupta [États-Unis]Source :
Abstract
Large-scale annotated datasets allow AI systems to learn from and build upon the knowledge of the crowd. Many crowdsourcing techniques have been developed for collecting image annotations. These techniques often implicitly rely on the fact that a new input image takes a negligible amount of time to perceive. In contrast, we investigate and determine the most cost-effective way of obtaining high-quality multi-label annotations for temporal data such as videos. Watching even a short 30-second video clip requires a significant time investment from a crowd worker; thus, requesting multiple annotations following a single viewing is an important cost-saving strategy. But how many questions should we ask per video? We conclude that the optimal strategy is to ask as many questions as possible in a HIT (up to 52 binary questions after watching a 30-second video clip in our experiments). We demonstrate that while workers may not correctly answer all questions, the cost-benefit analysis nevertheless favors consensus from multiple such cheap-yet-imperfect iterations over more complex alternatives. When compared with a one-question-per-video baseline, our method is able to achieve a 10% improvement in recall 76.7% ours versus 66.7% baseline) at comparable precision (83.8% ours versus 83.0% baseline) in about half the annotation time (3.8 minutes ours compared to 7.1 minutes baseline). We demonstrate the effectiveness of our method by collecting multi-label annotations of 157 human activities on 1,815 videos.
Url:
Links toward previous steps (curation, corpus...)
- to stream Hal, to step Corpus: Pour aller vers cette notice dans l'étape Curation :000390
Links to Exploration step
Hal:hal-01431527Le document en format XML
<record><TEI><teiHeader><fileDesc><titleStmt><title xml:lang="en">Much Ado About Time: Exhaustive Annotation of Temporal Data</title>
<author><name sortKey="Sigurdsson, Gunnar A" sort="Sigurdsson, Gunnar A" uniqKey="Sigurdsson G" first="Gunnar A." last="Sigurdsson">Gunnar A. Sigurdsson</name>
<affiliation wicri:level="1"><hal:affiliation type="laboratory" xml:id="struct-87723" status="VALID"> <orgName>Computer Science Department - Carnegie Mellon University</orgName>
<desc> <address> <addrLine>Computer Science Department Carnegie Mellon University Pittsburgh, PA</addrLine>
<country key="US"></country>
</address>
<ref type="url">http://www.cs.cmu.edu/</ref>
</desc>
<listRelation> <relation active="#struct-378064" type="direct"></relation>
</listRelation>
<tutelles><tutelle active="#struct-378064" type="direct"><org type="institution" xml:id="struct-378064" status="INCOMING"> <orgName>University of Pittsburgh</orgName>
<desc> <address> <country key="FR"></country>
</address>
</desc>
</org>
</tutelle>
</tutelles>
</hal:affiliation>
<country>États-Unis</country>
<placeName><settlement type="city">Pittsburgh</settlement>
<region type="state">Pennsylvanie</region>
</placeName>
<orgName type="university">Université de Pittsburgh</orgName>
</affiliation>
</author>
<author><name sortKey="Russakovsky, Olga" sort="Russakovsky, Olga" uniqKey="Russakovsky O" first="Olga" last="Russakovsky">Olga Russakovsky</name>
<affiliation wicri:level="1"><hal:affiliation type="laboratory" xml:id="struct-87723" status="VALID"> <orgName>Computer Science Department - Carnegie Mellon University</orgName>
<desc> <address> <addrLine>Computer Science Department Carnegie Mellon University Pittsburgh, PA</addrLine>
<country key="US"></country>
</address>
<ref type="url">http://www.cs.cmu.edu/</ref>
</desc>
<listRelation> <relation active="#struct-378064" type="direct"></relation>
</listRelation>
<tutelles><tutelle active="#struct-378064" type="direct"><org type="institution" xml:id="struct-378064" status="INCOMING"> <orgName>University of Pittsburgh</orgName>
<desc> <address> <country key="FR"></country>
</address>
</desc>
</org>
</tutelle>
</tutelles>
</hal:affiliation>
<country>États-Unis</country>
<placeName><settlement type="city">Pittsburgh</settlement>
<region type="state">Pennsylvanie</region>
</placeName>
<orgName type="university">Université de Pittsburgh</orgName>
</affiliation>
</author>
<author><name sortKey="Farhadi, Ali" sort="Farhadi, Ali" uniqKey="Farhadi A" first="Ali" last="Farhadi">Ali Farhadi</name>
<affiliation wicri:level="1"><hal:affiliation type="institution" xml:id="struct-300433" status="VALID"> <orgName>University of Washington [Seattle]</orgName>
<desc> <address> <addrLine>Seattle, Washington 98105</addrLine>
<country key="US"></country>
</address>
<ref type="url">http://www.washington.edu/</ref>
</desc>
</hal:affiliation>
<country>États-Unis</country>
</affiliation>
</author>
<author><name sortKey="Laptev, Ivan" sort="Laptev, Ivan" uniqKey="Laptev I" first="Ivan" last="Laptev">Ivan Laptev</name>
<affiliation wicri:level="1"><hal:affiliation type="institution" xml:id="struct-300009" status="VALID"><orgName>Institut National de Recherche en Informatique et en Automatique</orgName>
<orgName type="acronym">Inria</orgName>
<desc><address><addrLine>Domaine de VoluceauRocquencourt - BP 10578153 Le Chesnay Cedex</addrLine>
<country key="FR"></country>
</address>
<ref type="url">http://www.inria.fr/en/</ref>
</desc>
</hal:affiliation>
<country>France</country>
</affiliation>
</author>
<author><name sortKey="Gupta, Abhinav" sort="Gupta, Abhinav" uniqKey="Gupta A" first="Abhinav" last="Gupta">Abhinav Gupta</name>
<affiliation wicri:level="1"><hal:affiliation type="laboratory" xml:id="struct-87723" status="VALID"> <orgName>Computer Science Department - Carnegie Mellon University</orgName>
<desc> <address> <addrLine>Computer Science Department Carnegie Mellon University Pittsburgh, PA</addrLine>
<country key="US"></country>
</address>
<ref type="url">http://www.cs.cmu.edu/</ref>
</desc>
<listRelation> <relation active="#struct-378064" type="direct"></relation>
</listRelation>
<tutelles><tutelle active="#struct-378064" type="direct"><org type="institution" xml:id="struct-378064" status="INCOMING"> <orgName>University of Pittsburgh</orgName>
<desc> <address> <country key="FR"></country>
</address>
</desc>
</org>
</tutelle>
</tutelles>
</hal:affiliation>
<country>États-Unis</country>
<placeName><settlement type="city">Pittsburgh</settlement>
<region type="state">Pennsylvanie</region>
</placeName>
<orgName type="university">Université de Pittsburgh</orgName>
</affiliation>
</author>
</titleStmt>
<publicationStmt><idno type="wicri:source">HAL</idno>
<idno type="RBID">Hal:hal-01431527</idno>
<idno type="halId">hal-01431527</idno>
<idno type="halUri">https://hal.inria.fr/hal-01431527</idno>
<idno type="url">https://hal.inria.fr/hal-01431527</idno>
<date when="2016">2016</date>
<idno type="wicri:Area/Hal/Corpus">000390</idno>
<idno type="wicri:Area/Hal/Curation">000390</idno>
</publicationStmt>
<sourceDesc><biblStruct><analytic><title xml:lang="en">Much Ado About Time: Exhaustive Annotation of Temporal Data</title>
<author><name sortKey="Sigurdsson, Gunnar A" sort="Sigurdsson, Gunnar A" uniqKey="Sigurdsson G" first="Gunnar A." last="Sigurdsson">Gunnar A. Sigurdsson</name>
<affiliation wicri:level="1"><hal:affiliation type="laboratory" xml:id="struct-87723" status="VALID"> <orgName>Computer Science Department - Carnegie Mellon University</orgName>
<desc> <address> <addrLine>Computer Science Department Carnegie Mellon University Pittsburgh, PA</addrLine>
<country key="US"></country>
</address>
<ref type="url">http://www.cs.cmu.edu/</ref>
</desc>
<listRelation> <relation active="#struct-378064" type="direct"></relation>
</listRelation>
<tutelles><tutelle active="#struct-378064" type="direct"><org type="institution" xml:id="struct-378064" status="INCOMING"> <orgName>University of Pittsburgh</orgName>
<desc> <address> <country key="FR"></country>
</address>
</desc>
</org>
</tutelle>
</tutelles>
</hal:affiliation>
<country>États-Unis</country>
<placeName><settlement type="city">Pittsburgh</settlement>
<region type="state">Pennsylvanie</region>
</placeName>
<orgName type="university">Université de Pittsburgh</orgName>
</affiliation>
</author>
<author><name sortKey="Russakovsky, Olga" sort="Russakovsky, Olga" uniqKey="Russakovsky O" first="Olga" last="Russakovsky">Olga Russakovsky</name>
<affiliation wicri:level="1"><hal:affiliation type="laboratory" xml:id="struct-87723" status="VALID"> <orgName>Computer Science Department - Carnegie Mellon University</orgName>
<desc> <address> <addrLine>Computer Science Department Carnegie Mellon University Pittsburgh, PA</addrLine>
<country key="US"></country>
</address>
<ref type="url">http://www.cs.cmu.edu/</ref>
</desc>
<listRelation> <relation active="#struct-378064" type="direct"></relation>
</listRelation>
<tutelles><tutelle active="#struct-378064" type="direct"><org type="institution" xml:id="struct-378064" status="INCOMING"> <orgName>University of Pittsburgh</orgName>
<desc> <address> <country key="FR"></country>
</address>
</desc>
</org>
</tutelle>
</tutelles>
</hal:affiliation>
<country>États-Unis</country>
<placeName><settlement type="city">Pittsburgh</settlement>
<region type="state">Pennsylvanie</region>
</placeName>
<orgName type="university">Université de Pittsburgh</orgName>
</affiliation>
</author>
<author><name sortKey="Farhadi, Ali" sort="Farhadi, Ali" uniqKey="Farhadi A" first="Ali" last="Farhadi">Ali Farhadi</name>
<affiliation wicri:level="1"><hal:affiliation type="institution" xml:id="struct-300433" status="VALID"> <orgName>University of Washington [Seattle]</orgName>
<desc> <address> <addrLine>Seattle, Washington 98105</addrLine>
<country key="US"></country>
</address>
<ref type="url">http://www.washington.edu/</ref>
</desc>
</hal:affiliation>
<country>États-Unis</country>
</affiliation>
</author>
<author><name sortKey="Laptev, Ivan" sort="Laptev, Ivan" uniqKey="Laptev I" first="Ivan" last="Laptev">Ivan Laptev</name>
<affiliation wicri:level="1"><hal:affiliation type="institution" xml:id="struct-300009" status="VALID"><orgName>Institut National de Recherche en Informatique et en Automatique</orgName>
<orgName type="acronym">Inria</orgName>
<desc><address><addrLine>Domaine de VoluceauRocquencourt - BP 10578153 Le Chesnay Cedex</addrLine>
<country key="FR"></country>
</address>
<ref type="url">http://www.inria.fr/en/</ref>
</desc>
</hal:affiliation>
<country>France</country>
</affiliation>
</author>
<author><name sortKey="Gupta, Abhinav" sort="Gupta, Abhinav" uniqKey="Gupta A" first="Abhinav" last="Gupta">Abhinav Gupta</name>
<affiliation wicri:level="1"><hal:affiliation type="laboratory" xml:id="struct-87723" status="VALID"> <orgName>Computer Science Department - Carnegie Mellon University</orgName>
<desc> <address> <addrLine>Computer Science Department Carnegie Mellon University Pittsburgh, PA</addrLine>
<country key="US"></country>
</address>
<ref type="url">http://www.cs.cmu.edu/</ref>
</desc>
<listRelation> <relation active="#struct-378064" type="direct"></relation>
</listRelation>
<tutelles><tutelle active="#struct-378064" type="direct"><org type="institution" xml:id="struct-378064" status="INCOMING"> <orgName>University of Pittsburgh</orgName>
<desc> <address> <country key="FR"></country>
</address>
</desc>
</org>
</tutelle>
</tutelles>
</hal:affiliation>
<country>États-Unis</country>
<placeName><settlement type="city">Pittsburgh</settlement>
<region type="state">Pennsylvanie</region>
</placeName>
<orgName type="university">Université de Pittsburgh</orgName>
</affiliation>
</author>
</analytic>
</biblStruct>
</sourceDesc>
</fileDesc>
<profileDesc><textClass></textClass>
</profileDesc>
</teiHeader>
<front><div type="abstract" xml:lang="en">Large-scale annotated datasets allow AI systems to learn from and build upon the knowledge of the crowd. Many crowdsourcing techniques have been developed for collecting image annotations. These techniques often implicitly rely on the fact that a new input image takes a negligible amount of time to perceive. In contrast, we investigate and determine the most cost-effective way of obtaining high-quality multi-label annotations for temporal data such as videos. Watching even a short 30-second video clip requires a significant time investment from a crowd worker; thus, requesting multiple annotations following a single viewing is an important cost-saving strategy. But how many questions should we ask per video? We conclude that the optimal strategy is to ask as many questions as possible in a HIT (up to 52 binary questions after watching a 30-second video clip in our experiments). We demonstrate that while workers may not correctly answer all questions, the cost-benefit analysis nevertheless favors consensus from multiple such cheap-yet-imperfect iterations over more complex alternatives. When compared with a one-question-per-video baseline, our method is able to achieve a 10% improvement in recall 76.7% ours versus 66.7% baseline) at comparable precision (83.8% ours versus 83.0% baseline) in about half the annotation time (3.8 minutes ours compared to 7.1 minutes baseline). We demonstrate the effectiveness of our method by collecting multi-label annotations of 157 human activities on 1,815 videos.</div>
</front>
</TEI>
<hal api="V3"><titleStmt><title xml:lang="en">Much Ado About Time: Exhaustive Annotation of Temporal Data</title>
<author role="aut"><persName><forename type="first">Gunnar A.</forename>
<surname>Sigurdsson</surname>
</persName>
<idno type="halauthorid">1450357</idno>
<affiliation ref="#struct-87723"></affiliation>
</author>
<author role="aut"><persName><forename type="first">Olga</forename>
<surname>Russakovsky</surname>
</persName>
<idno type="halauthorid">1450358</idno>
<affiliation ref="#struct-87723"></affiliation>
</author>
<author role="aut"><persName><forename type="first">Ali</forename>
<surname>Farhadi</surname>
</persName>
<idno type="halauthorid">1436482</idno>
<affiliation ref="#struct-300433"></affiliation>
<affiliation ref="#struct-476519"></affiliation>
</author>
<author role="aut"><persName><forename type="first">Ivan</forename>
<surname>Laptev</surname>
</persName>
<idno type="halauthorid">1167151</idno>
<affiliation ref="#struct-300009"></affiliation>
</author>
<author role="aut"><persName><forename type="first">Abhinav</forename>
<surname>Gupta</surname>
</persName>
<idno type="halauthorid">1265373</idno>
<affiliation ref="#struct-87723"></affiliation>
<affiliation ref="#struct-476519"></affiliation>
</author>
<editor role="depositor"><persName><forename>Guilhem</forename>
<surname>Chéron</surname>
</persName>
<email type="md5">cd67ad3c2d635c934b82f2746a255736</email>
<email type="domain">inria.fr</email>
</editor>
</titleStmt>
<editionStmt><edition n="v1" type="current"><date type="whenSubmitted">2017-01-11 09:09:34</date>
<date type="whenWritten">2016</date>
<date type="whenModified">2017-05-09 15:58:38</date>
<date type="whenReleased">2017-01-11 09:09:34</date>
<date type="whenProduced">2016</date>
</edition>
<respStmt><resp>contributor</resp>
<name key="320411"><persName><forename>Guilhem</forename>
<surname>Chéron</surname>
</persName>
<email type="md5">cd67ad3c2d635c934b82f2746a255736</email>
<email type="domain">inria.fr</email>
</name>
</respStmt>
</editionStmt>
<publicationStmt><distributor>CCSD</distributor>
<idno type="halId">hal-01431527</idno>
<idno type="halUri">https://hal.inria.fr/hal-01431527</idno>
<idno type="halBibtex">sigurdsson:hal-01431527</idno>
<idno type="halRefHtml">HCOMP 2016 Camera Ready. 2016</idno>
<idno type="halRef">HCOMP 2016 Camera Ready. 2016</idno>
</publicationStmt>
<seriesStmt><idno type="stamp" n="INRIA">INRIA - Institut National de Recherche en Informatique et en Automatique</idno>
</seriesStmt>
<notesStmt><note type="commentary">HCOMP 2016 Camera Ready</note>
</notesStmt>
<sourceDesc><biblStruct><analytic><title xml:lang="en">Much Ado About Time: Exhaustive Annotation of Temporal Data</title>
<author role="aut"><persName><forename type="first">Gunnar A.</forename>
<surname>Sigurdsson</surname>
</persName>
<idno type="halauthorid">1450357</idno>
<affiliation ref="#struct-87723"></affiliation>
</author>
<author role="aut"><persName><forename type="first">Olga</forename>
<surname>Russakovsky</surname>
</persName>
<idno type="halauthorid">1450358</idno>
<affiliation ref="#struct-87723"></affiliation>
</author>
<author role="aut"><persName><forename type="first">Ali</forename>
<surname>Farhadi</surname>
</persName>
<idno type="halauthorid">1436482</idno>
<affiliation ref="#struct-300433"></affiliation>
<affiliation ref="#struct-476519"></affiliation>
</author>
<author role="aut"><persName><forename type="first">Ivan</forename>
<surname>Laptev</surname>
</persName>
<idno type="halauthorid">1167151</idno>
<affiliation ref="#struct-300009"></affiliation>
</author>
<author role="aut"><persName><forename type="first">Abhinav</forename>
<surname>Gupta</surname>
</persName>
<idno type="halauthorid">1265373</idno>
<affiliation ref="#struct-87723"></affiliation>
<affiliation ref="#struct-476519"></affiliation>
</author>
</analytic>
<monogr><imprint></imprint>
</monogr>
<idno type="arxiv">1607.07429</idno>
</biblStruct>
</sourceDesc>
<profileDesc><langUsage><language ident="en">English</language>
</langUsage>
<textClass><classCode scheme="halDomain" n="info.info-hc">Computer Science [cs]/Human-Computer Interaction [cs.HC]</classCode>
<classCode scheme="halDomain" n="info.info-cv">Computer Science [cs]/Computer Vision and Pattern Recognition [cs.CV]</classCode>
<classCode scheme="halTypology" n="UNDEFINED">Preprints, Working Papers, ...</classCode>
</textClass>
<abstract xml:lang="en">Large-scale annotated datasets allow AI systems to learn from and build upon the knowledge of the crowd. Many crowdsourcing techniques have been developed for collecting image annotations. These techniques often implicitly rely on the fact that a new input image takes a negligible amount of time to perceive. In contrast, we investigate and determine the most cost-effective way of obtaining high-quality multi-label annotations for temporal data such as videos. Watching even a short 30-second video clip requires a significant time investment from a crowd worker; thus, requesting multiple annotations following a single viewing is an important cost-saving strategy. But how many questions should we ask per video? We conclude that the optimal strategy is to ask as many questions as possible in a HIT (up to 52 binary questions after watching a 30-second video clip in our experiments). We demonstrate that while workers may not correctly answer all questions, the cost-benefit analysis nevertheless favors consensus from multiple such cheap-yet-imperfect iterations over more complex alternatives. When compared with a one-question-per-video baseline, our method is able to achieve a 10% improvement in recall 76.7% ours versus 66.7% baseline) at comparable precision (83.8% ours versus 83.0% baseline) in about half the annotation time (3.8 minutes ours compared to 7.1 minutes baseline). We demonstrate the effectiveness of our method by collecting multi-label annotations of 157 human activities on 1,815 videos.</abstract>
</profileDesc>
</hal>
</record>
Pour manipuler ce document sous Unix (Dilib)
EXPLOR_STEP=$WICRI_ROOT/Wicri/Amérique/explor/PittsburghV1/Data/Hal/Curation
HfdSelect -h $EXPLOR_STEP/biblio.hfd -nk 000389 | SxmlIndent | more
Ou
HfdSelect -h $EXPLOR_AREA/Data/Hal/Curation/biblio.hfd -nk 000389 | SxmlIndent | more
Pour mettre un lien sur cette page dans le réseau Wicri
{{Explor lien |wiki= Wicri/Amérique |area= PittsburghV1 |flux= Hal |étape= Curation |type= RBID |clé= Hal:hal-01431527 |texte= Much Ado About Time: Exhaustive Annotation of Temporal Data }}
This area was generated with Dilib version V0.6.38. |