3D Visual Speech Animation from Image Sequences
Identifieur interne : 000835 ( Main/Exploration ); précédent : 000834; suivant : 0008363D Visual Speech Animation from Image Sequences
Auteurs : Utpala Musti [Finlande] ; Slim Ouni [France] ; Zhou Ziheng [Finlande]Source :
English descriptors
- mix :
Abstract
In this paper we describe an early version of our system which synthesizes 3D visual speech including tongue and teeth from frontal facial image sequences. This system is developed for 3D Visual Speech Animation (VSA) using images generated by an existing state-of-the-art image-based VSA system. In fact, the prime motivation for this system is to have a 3D VSA system from limited amount of training data when compared to that required for developing a conventional corpus based 3D VSA system. It consists of two modules. The rst module iteratively estimates the 3D shape of the external facial surface for each image in the input sequence. The second module complements the external face with 3D tongue and teeth to complete the perceptually crucial visual speech information. This has the added advantages of a 3D visual speech, which are render ability of the face in dierent poses and illumination conditions and, enhanced visual information of tongue and teeth. The first module for 3D shape estimation is based on the detectionof facial landmarks in images. It uses a prior 3D Morphable Models (3D-MM) trained using 3D facial data. For the time being it is developed for a person-specic domain, i.e., the 3D-MM and the 2D facial landmark detector are trained using the data of a single person and tested with the same person-specic data. The estimated 3D shape sequences are provided as input to the second module along with the phonetic segmentation. For any particular 3D shape, tongue and teeth information is generated by rotating the lower jaw based on few skin points on the jaw and animating a rigid 3D tongue through keyframe interpolation.
Url:
Affiliations:
- Finlande, France
- Grand Est, Lorraine (région)
- Nancy
- Institut national polytechnique de Lorraine, Université Nancy 2, Université de Lorraine
Links toward previous steps (curation, corpus...)
- to stream Hal, to step Corpus: 000063
- to stream Hal, to step Curation: 000063
- to stream Hal, to step Checkpoint: 000769
- to stream Main, to step Merge: 000836
- to stream Main, to step Curation: 000835
Le document en format XML
<record><TEI><teiHeader><fileDesc><titleStmt><title xml:lang="en">3D Visual Speech Animation from Image Sequences</title><author><name sortKey="Musti, Utpala" sort="Musti, Utpala" uniqKey="Musti U" first="Utpala" last="Musti">Utpala Musti</name><affiliation wicri:level="1"><hal:affiliation type="institution" xml:id="struct-300847" status="VALID"><orgName>University of Oulu [Oulu]</orgName><desc><address><addrLine>Pentti Kaiteran katu 1, 90014 Oulu</addrLine><country key="FI"></country></address><ref type="url">http://www.oulu.fi/english/</ref></desc></hal:affiliation><country>Finlande</country></affiliation></author><author><name sortKey="Ouni, Slim" sort="Ouni, Slim" uniqKey="Ouni S" first="Slim" last="Ouni">Slim Ouni</name><affiliation wicri:level="1"><hal:affiliation type="researchteam" xml:id="struct-2359" status="OLD"><idno type="RNSR">200118295L</idno><orgName>Analysis, perception and recognition of speech</orgName><orgName type="acronym">PAROLE</orgName><desc><address><country key="FR"></country></address><ref type="url">http://www.inria.fr/equipes/parole</ref></desc><listRelation><relation active="#struct-160" type="direct"></relation><relation name="UMR7503" active="#struct-441569" type="indirect"></relation><relation active="#struct-300009" type="indirect"></relation><relation active="#struct-300291" type="indirect"></relation><relation active="#struct-300292" type="indirect"></relation><relation active="#struct-300293" type="indirect"></relation><relation active="#struct-2496" type="direct"></relation></listRelation><tutelles><tutelle active="#struct-160" type="direct"><org type="laboratory" xml:id="struct-160" status="OLD"><orgName>Laboratoire Lorrain de Recherche en Informatique et ses Applications</orgName><orgName type="acronym">LORIA</orgName><desc><address><addrLine>Campus Scientifique BP 239 54506 Vandoeuvre-lès-Nancy Cedex</addrLine><country key="FR"></country></address><ref type="url">http://www.loria.fr</ref></desc><listRelation><relation name="UMR7503" active="#struct-441569" type="direct"></relation><relation active="#struct-300009" type="direct"></relation><relation active="#struct-300291" type="direct"></relation><relation active="#struct-300292" type="direct"></relation><relation active="#struct-300293" type="direct"></relation></listRelation></org></tutelle><tutelle name="UMR7503" active="#struct-441569" type="indirect"><org type="institution" xml:id="struct-441569" status="VALID"><idno type="ISNI">0000000122597504</idno><idno type="IdRef">02636817X</idno><orgName>Centre National de la Recherche Scientifique</orgName><orgName type="acronym">CNRS</orgName><date type="start">1939-10-19</date><desc><address><country key="FR"></country></address><ref type="url">http://www.cnrs.fr/</ref></desc></org></tutelle><tutelle active="#struct-300009" type="indirect"><org type="institution" xml:id="struct-300009" status="VALID"><orgName>Institut National de Recherche en Informatique et en Automatique</orgName><orgName type="acronym">Inria</orgName><desc><address><addrLine>Domaine de VoluceauRocquencourt - BP 10578153 Le Chesnay Cedex</addrLine><country key="FR"></country></address><ref type="url">http://www.inria.fr/en/</ref></desc></org></tutelle><tutelle active="#struct-300291" type="indirect"><org type="institution" xml:id="struct-300291" status="OLD"><orgName>Université Henri Poincaré - Nancy 1</orgName><orgName type="acronym">UHP</orgName><date type="end">2011-12-31</date><desc><address><addrLine>24-30 rue Lionnois, BP 60120, 54 003 NANCY cedex, France</addrLine><country key="FR"></country></address></desc></org></tutelle><tutelle active="#struct-300292" type="indirect"><org type="institution" xml:id="struct-300292" status="OLD"><orgName>Université Nancy 2</orgName><date type="end">2011-12-31</date><desc><address><addrLine>91 avenue de la Libération, BP 454, 54001 Nancy cedex</addrLine><country key="FR"></country></address></desc></org></tutelle><tutelle active="#struct-300293" type="indirect"><org type="institution" xml:id="struct-300293" status="OLD"><orgName>Institut National Polytechnique de Lorraine</orgName><orgName type="acronym">INPL</orgName><date type="end">2011-12-31</date><desc><address><country key="FR"></country></address></desc></org></tutelle><tutelle active="#struct-2496" type="direct"><org type="laboratory" xml:id="struct-2496" status="OLD"><orgName>INRIA Lorraine</orgName><desc><address><addrLine>615 rue du Jardin Botanique 54600 Villers-lès-Nancy</addrLine><country key="FR"></country></address><ref type="url">http://www.inria.fr/centre-de-recherche-inria/nancy-grand-est</ref></desc><listRelation><relation active="#struct-300009" type="direct"></relation></listRelation></org></tutelle></tutelles></hal:affiliation><country>France</country><placeName><settlement type="city">Nancy</settlement><region type="region" nuts="2">Grand Est</region><region type="old region" nuts="2">Lorraine (région)</region></placeName><orgName type="university">Université Nancy 2</orgName><orgName type="institution" wicri:auto="newGroup">Université de Lorraine</orgName><placeName><settlement type="city">Nancy</settlement><region type="region" nuts="2">Grand Est</region><region type="old region" nuts="2">Lorraine (région)</region></placeName><orgName type="university">Institut national polytechnique de Lorraine</orgName><orgName type="institution" wicri:auto="newGroup">Université de Lorraine</orgName></affiliation></author><author><name sortKey="Ziheng, Zhou" sort="Ziheng, Zhou" uniqKey="Ziheng Z" first="Zhou" last="Ziheng">Zhou Ziheng</name><affiliation wicri:level="1"><hal:affiliation type="institution" xml:id="struct-300847" status="VALID"><orgName>University of Oulu [Oulu]</orgName><desc><address><addrLine>Pentti Kaiteran katu 1, 90014 Oulu</addrLine><country key="FI"></country></address><ref type="url">http://www.oulu.fi/english/</ref></desc></hal:affiliation><country>Finlande</country></affiliation></author></titleStmt><publicationStmt><idno type="wicri:source">HAL</idno><idno type="RBID">Hal:hal-01086073</idno><idno type="halId">hal-01086073</idno><idno type="halUri">https://hal.archives-ouvertes.fr/hal-01086073</idno><idno type="url">https://hal.archives-ouvertes.fr/hal-01086073</idno><date when="2014-12-14">2014-12-14</date><idno type="wicri:Area/Hal/Corpus">000063</idno><idno type="wicri:Area/Hal/Curation">000063</idno><idno type="wicri:Area/Hal/Checkpoint">000769</idno><idno type="wicri:explorRef" wicri:stream="Hal" wicri:step="Checkpoint">000769</idno><idno type="wicri:Area/Main/Merge">000836</idno><idno type="wicri:Area/Main/Curation">000835</idno><idno type="wicri:Area/Main/Exploration">000835</idno></publicationStmt><sourceDesc><biblStruct><analytic><title xml:lang="en">3D Visual Speech Animation from Image Sequences</title><author><name sortKey="Musti, Utpala" sort="Musti, Utpala" uniqKey="Musti U" first="Utpala" last="Musti">Utpala Musti</name><affiliation wicri:level="1"><hal:affiliation type="institution" xml:id="struct-300847" status="VALID"><orgName>University of Oulu [Oulu]</orgName><desc><address><addrLine>Pentti Kaiteran katu 1, 90014 Oulu</addrLine><country key="FI"></country></address><ref type="url">http://www.oulu.fi/english/</ref></desc></hal:affiliation><country>Finlande</country></affiliation></author><author><name sortKey="Ouni, Slim" sort="Ouni, Slim" uniqKey="Ouni S" first="Slim" last="Ouni">Slim Ouni</name><affiliation wicri:level="1"><hal:affiliation type="researchteam" xml:id="struct-2359" status="OLD"><idno type="RNSR">200118295L</idno><orgName>Analysis, perception and recognition of speech</orgName><orgName type="acronym">PAROLE</orgName><desc><address><country key="FR"></country></address><ref type="url">http://www.inria.fr/equipes/parole</ref></desc><listRelation><relation active="#struct-160" type="direct"></relation><relation name="UMR7503" active="#struct-441569" type="indirect"></relation><relation active="#struct-300009" type="indirect"></relation><relation active="#struct-300291" type="indirect"></relation><relation active="#struct-300292" type="indirect"></relation><relation active="#struct-300293" type="indirect"></relation><relation active="#struct-2496" type="direct"></relation></listRelation><tutelles><tutelle active="#struct-160" type="direct"><org type="laboratory" xml:id="struct-160" status="OLD"><orgName>Laboratoire Lorrain de Recherche en Informatique et ses Applications</orgName><orgName type="acronym">LORIA</orgName><desc><address><addrLine>Campus Scientifique BP 239 54506 Vandoeuvre-lès-Nancy Cedex</addrLine><country key="FR"></country></address><ref type="url">http://www.loria.fr</ref></desc><listRelation><relation name="UMR7503" active="#struct-441569" type="direct"></relation><relation active="#struct-300009" type="direct"></relation><relation active="#struct-300291" type="direct"></relation><relation active="#struct-300292" type="direct"></relation><relation active="#struct-300293" type="direct"></relation></listRelation></org></tutelle><tutelle name="UMR7503" active="#struct-441569" type="indirect"><org type="institution" xml:id="struct-441569" status="VALID"><idno type="ISNI">0000000122597504</idno><idno type="IdRef">02636817X</idno><orgName>Centre National de la Recherche Scientifique</orgName><orgName type="acronym">CNRS</orgName><date type="start">1939-10-19</date><desc><address><country key="FR"></country></address><ref type="url">http://www.cnrs.fr/</ref></desc></org></tutelle><tutelle active="#struct-300009" type="indirect"><org type="institution" xml:id="struct-300009" status="VALID"><orgName>Institut National de Recherche en Informatique et en Automatique</orgName><orgName type="acronym">Inria</orgName><desc><address><addrLine>Domaine de VoluceauRocquencourt - BP 10578153 Le Chesnay Cedex</addrLine><country key="FR"></country></address><ref type="url">http://www.inria.fr/en/</ref></desc></org></tutelle><tutelle active="#struct-300291" type="indirect"><org type="institution" xml:id="struct-300291" status="OLD"><orgName>Université Henri Poincaré - Nancy 1</orgName><orgName type="acronym">UHP</orgName><date type="end">2011-12-31</date><desc><address><addrLine>24-30 rue Lionnois, BP 60120, 54 003 NANCY cedex, France</addrLine><country key="FR"></country></address></desc></org></tutelle><tutelle active="#struct-300292" type="indirect"><org type="institution" xml:id="struct-300292" status="OLD"><orgName>Université Nancy 2</orgName><date type="end">2011-12-31</date><desc><address><addrLine>91 avenue de la Libération, BP 454, 54001 Nancy cedex</addrLine><country key="FR"></country></address></desc></org></tutelle><tutelle active="#struct-300293" type="indirect"><org type="institution" xml:id="struct-300293" status="OLD"><orgName>Institut National Polytechnique de Lorraine</orgName><orgName type="acronym">INPL</orgName><date type="end">2011-12-31</date><desc><address><country key="FR"></country></address></desc></org></tutelle><tutelle active="#struct-2496" type="direct"><org type="laboratory" xml:id="struct-2496" status="OLD"><orgName>INRIA Lorraine</orgName><desc><address><addrLine>615 rue du Jardin Botanique 54600 Villers-lès-Nancy</addrLine><country key="FR"></country></address><ref type="url">http://www.inria.fr/centre-de-recherche-inria/nancy-grand-est</ref></desc><listRelation><relation active="#struct-300009" type="direct"></relation></listRelation></org></tutelle></tutelles></hal:affiliation><country>France</country><placeName><settlement type="city">Nancy</settlement><region type="region" nuts="2">Grand Est</region><region type="old region" nuts="2">Lorraine (région)</region></placeName><orgName type="university">Université Nancy 2</orgName><orgName type="institution" wicri:auto="newGroup">Université de Lorraine</orgName><placeName><settlement type="city">Nancy</settlement><region type="region" nuts="2">Grand Est</region><region type="old region" nuts="2">Lorraine (région)</region></placeName><orgName type="university">Institut national polytechnique de Lorraine</orgName><orgName type="institution" wicri:auto="newGroup">Université de Lorraine</orgName></affiliation></author><author><name sortKey="Ziheng, Zhou" sort="Ziheng, Zhou" uniqKey="Ziheng Z" first="Zhou" last="Ziheng">Zhou Ziheng</name><affiliation wicri:level="1"><hal:affiliation type="institution" xml:id="struct-300847" status="VALID"><orgName>University of Oulu [Oulu]</orgName><desc><address><addrLine>Pentti Kaiteran katu 1, 90014 Oulu</addrLine><country key="FI"></country></address><ref type="url">http://www.oulu.fi/english/</ref></desc></hal:affiliation><country>Finlande</country></affiliation></author></analytic></biblStruct></sourceDesc></fileDesc><profileDesc><textClass><keywords scheme="mix" xml:lang="en"><term>3D facial shape estimation from images</term><term>3D visual speech</term><term>facial landmark detection</term><term>visual speech animation</term></keywords></textClass></profileDesc></teiHeader><front><div type="abstract" xml:lang="en">In this paper we describe an early version of our system which synthesizes 3D visual speech including tongue and teeth from frontal facial image sequences. This system is developed for 3D Visual Speech Animation (VSA) using images generated by an existing state-of-the-art image-based VSA system. In fact, the prime motivation for this system is to have a 3D VSA system from limited amount of training data when compared to that required for developing a conventional corpus based 3D VSA system. It consists of two modules. The rst module iteratively estimates the 3D shape of the external facial surface for each image in the input sequence. The second module complements the external face with 3D tongue and teeth to complete the perceptually crucial visual speech information. This has the added advantages of a 3D visual speech, which are render ability of the face in dierent poses and illumination conditions and, enhanced visual information of tongue and teeth. The first module for 3D shape estimation is based on the detectionof facial landmarks in images. It uses a prior 3D Morphable Models (3D-MM) trained using 3D facial data. For the time being it is developed for a person-specic domain, i.e., the 3D-MM and the 2D facial landmark detector are trained using the data of a single person and tested with the same person-specic data. The estimated 3D shape sequences are provided as input to the second module along with the phonetic segmentation. For any particular 3D shape, tongue and teeth information is generated by rotating the lower jaw based on few skin points on the jaw and animating a rigid 3D tongue through keyframe interpolation.</div></front></TEI><affiliations><list><country><li>Finlande</li><li>France</li></country><region><li>Grand Est</li><li>Lorraine (région)</li></region><settlement><li>Nancy</li></settlement><orgName><li>Institut national polytechnique de Lorraine</li><li>Université Nancy 2</li><li>Université de Lorraine</li></orgName></list><tree><country name="Finlande"><noRegion><name sortKey="Musti, Utpala" sort="Musti, Utpala" uniqKey="Musti U" first="Utpala" last="Musti">Utpala Musti</name></noRegion><name sortKey="Ziheng, Zhou" sort="Ziheng, Zhou" uniqKey="Ziheng Z" first="Zhou" last="Ziheng">Zhou Ziheng</name></country><country name="France"><region name="Grand Est"><name sortKey="Ouni, Slim" sort="Ouni, Slim" uniqKey="Ouni S" first="Slim" last="Ouni">Slim Ouni</name></region></country></tree></affiliations></record>Pour manipuler ce document sous Unix (Dilib)
EXPLOR_STEP=$WICRI_ROOT/Wicri/Lorraine/explor/InforLorV4/Data/Main/Exploration
HfdSelect -h $EXPLOR_STEP/biblio.hfd -nk 000835 | SxmlIndent | more
Ou
HfdSelect -h $EXPLOR_AREA/Data/Main/Exploration/biblio.hfd -nk 000835 | SxmlIndent | more
Pour mettre un lien sur cette page dans le réseau Wicri
{{Explor lien
|wiki= Wicri/Lorraine
|area= InforLorV4
|flux= Main
|étape= Exploration
|type= RBID
|clé= Hal:hal-01086073
|texte= 3D Visual Speech Animation from Image Sequences
}}
| This area was generated with Dilib version V0.6.33. |  |