An alternative scheme for perplexity estimation and its assessment for the evaluation of language models
Identifieur interne : 009A03 ( Main/Merge ); précédent : 009A02; suivant : 009A04An alternative scheme for perplexity estimation and its assessment for the evaluation of language models
Auteurs : Frédérie Bimbot [France] ; Marc El-Beze ; Stéphane Igounet ; Michèle Jardino ; Kamel Smaili ; Imed ZitouniSource :
- Computer speech & language [ 0885-2308 ] ; 2001.
Descripteurs français
- Pascal (Inist)
- Wicri :
- topic : Statistique.
English descriptors
- KwdEn :
Abstract
Language models are usually evaluated on test texts using the perplexity derived from the model likelihood function computed on these texts (test set perplexity). In order to use this measure in the framework of a comparative evaluation campaign, we have developed an alternative scheme for estimating the test set perplexity. The method is derived from the Shannon game and based on a gambling approach on the next word to come in a truncated sentence. We also study the entropy bounds proposed by Shannon and based on the rank of the correct answer, in order to estimate a perplexity interval for non-probabilistic language models. The relevance of the approach is validated on an example. We then report the results of a preliminary comparative evaluation using the proposed scheme.
Links toward previous steps (curation, corpus...)
- to stream PascalFrancis, to step Corpus: 000931
- to stream PascalFrancis, to step Curation: 000A12
- to stream PascalFrancis, to step Checkpoint: 000920
Links to Exploration step
Francis:524-02-10502Le document en format XML
<record><TEI><teiHeader><fileDesc><titleStmt><title xml:lang="en" level="a">An alternative scheme for perplexity estimation and its assessment for the evaluation of language models</title><author><name sortKey="Bimbot, Frederie" sort="Bimbot, Frederie" uniqKey="Bimbot F" first="Frédérie" last="Bimbot">Frédérie Bimbot</name><affiliation wicri:level="3"><inist:fA14 i1="01"><s1>TENST/CNRS, 46 rue Barrault</s1><s2>75634 Paris</s2><s3>FRA</s3></inist:fA14><country>France</country><placeName><region type="region" nuts="2">Île-de-France</region><settlement type="city">Paris</settlement></placeName></affiliation><affiliation wicri:level="3"><inist:fA14 i1="02"><s1>IRISA(CNRS & INRIA), Campus Universitaire de Beaulieu</s1><s2>35042 Rennes LIA</s2> <s3>FRA</s3></inist:fA14><country>France</country><placeName><region type="region" nuts="2">Région Bretagne</region><settlement type="city">Rennes LIA</settlement> </placeName></affiliation><affiliation wicri:level="4"><inist:fA14 i1="03"><s1>Université d'Avignon, BP 1228, Agroparc</s1><s2>84911 Avignon</s2><s3>FRA</s3></inist:fA14><country>France</country><placeName><region type="region" nuts="2">Provence-Alpes-Côte d'Azur</region><settlement type="city">Avignon</settlement></placeName><orgName type="university">Université d'Avignon</orgName></affiliation><affiliation wicri:level="4"><inist:fA14 i1="04"><s1>LIMSI/CNRS, BP 133, Université Paris-Sud</s1><s2>91403 Orsay</s2><s3>FRA</s3></inist:fA14><country>France</country><placeName><region type="region" nuts="2">Île-de-France</region><settlement type="city">Orsay</settlement></placeName><orgName type="university">Université Paris-Sud</orgName></affiliation><affiliation wicri:level="3"><inist:fA14 i1="05"><s1>LORIA(INRIA & CNRS), BP 239</s1><s2>54506 Vandœuvre-lès-Nancy cedex</s2><s3>FRA</s3></inist:fA14><country>France</country><placeName><region type="region" nuts="2">Grand Est</region><region type="old region" nuts="2">Lorraine (région)</region><settlement type="city">Vandœuvre-lès-Nancy</settlement></placeName></affiliation></author><author><name sortKey="El Beze, Marc" sort="El Beze, Marc" uniqKey="El Beze M" first="Marc" last="El-Beze">Marc El-Beze</name></author><author><name sortKey="Igounet, Stephane" sort="Igounet, Stephane" uniqKey="Igounet S" first="Stéphane" last="Igounet">Stéphane Igounet</name></author><author><name sortKey="Jardino, Michele" sort="Jardino, Michele" uniqKey="Jardino M" first="Michèle" last="Jardino">Michèle Jardino</name></author><author><name sortKey="Smaili, Kamel" sort="Smaili, Kamel" uniqKey="Smaili K" first="Kamel" last="Smaili">Kamel Smaili</name></author><author><name sortKey="Zitouni, Imed" sort="Zitouni, Imed" uniqKey="Zitouni I" first="Imed" last="Zitouni">Imed Zitouni</name></author></titleStmt><publicationStmt><idno type="wicri:source">INIST</idno><idno type="inist">524-02-10502</idno><date when="2001">2001</date><idno type="stanalyst">FRANCIS 524-02-10502 INIST</idno><idno type="RBID">Francis:524-02-10502</idno><idno type="wicri:Area/PascalFrancis/Corpus">000931</idno><idno type="wicri:Area/PascalFrancis/Curation">000A12</idno><idno type="wicri:Area/PascalFrancis/Checkpoint">000920</idno><idno type="wicri:explorRef" wicri:stream="PascalFrancis" wicri:step="Checkpoint">000920</idno><idno type="wicri:doubleKey">0885-2308:2001:Bimbot F:an:alternative:scheme</idno><idno type="wicri:Area/Main/Merge">009A03</idno></publicationStmt><sourceDesc><biblStruct><analytic><title xml:lang="en" level="a">An alternative scheme for perplexity estimation and its assessment for the evaluation of language models</title><author><name sortKey="Bimbot, Frederie" sort="Bimbot, Frederie" uniqKey="Bimbot F" first="Frédérie" last="Bimbot">Frédérie Bimbot</name><affiliation wicri:level="3"><inist:fA14 i1="01"><s1>TENST/CNRS, 46 rue Barrault</s1><s2>75634 Paris</s2><s3>FRA</s3></inist:fA14><country>France</country><placeName><region type="region" nuts="2">Île-de-France</region><settlement type="city">Paris</settlement></placeName></affiliation><affiliation wicri:level="3"><inist:fA14 i1="02"><s1>IRISA(CNRS & INRIA), Campus Universitaire de Beaulieu</s1><s2>35042 Rennes LIA</s2> <s3>FRA</s3></inist:fA14><country>France</country><placeName><region type="region" nuts="2">Région Bretagne</region><settlement type="city">Rennes LIA</settlement> </placeName></affiliation><affiliation wicri:level="4"><inist:fA14 i1="03"><s1>Université d'Avignon, BP 1228, Agroparc</s1><s2>84911 Avignon</s2><s3>FRA</s3></inist:fA14><country>France</country><placeName><region type="region" nuts="2">Provence-Alpes-Côte d'Azur</region><settlement type="city">Avignon</settlement></placeName><orgName type="university">Université d'Avignon</orgName></affiliation><affiliation wicri:level="4"><inist:fA14 i1="04"><s1>LIMSI/CNRS, BP 133, Université Paris-Sud</s1><s2>91403 Orsay</s2><s3>FRA</s3></inist:fA14><country>France</country><placeName><region type="region" nuts="2">Île-de-France</region><settlement type="city">Orsay</settlement></placeName><orgName type="university">Université Paris-Sud</orgName></affiliation><affiliation wicri:level="3"><inist:fA14 i1="05"><s1>LORIA(INRIA & CNRS), BP 239</s1><s2>54506 Vandœuvre-lès-Nancy cedex</s2><s3>FRA</s3></inist:fA14><country>France</country><placeName><region type="region" nuts="2">Grand Est</region><region type="old region" nuts="2">Lorraine (région)</region><settlement type="city">Vandœuvre-lès-Nancy</settlement></placeName></affiliation></author><author><name sortKey="El Beze, Marc" sort="El Beze, Marc" uniqKey="El Beze M" first="Marc" last="El-Beze">Marc El-Beze</name></author><author><name sortKey="Igounet, Stephane" sort="Igounet, Stephane" uniqKey="Igounet S" first="Stéphane" last="Igounet">Stéphane Igounet</name></author><author><name sortKey="Jardino, Michele" sort="Jardino, Michele" uniqKey="Jardino M" first="Michèle" last="Jardino">Michèle Jardino</name></author><author><name sortKey="Smaili, Kamel" sort="Smaili, Kamel" uniqKey="Smaili K" first="Kamel" last="Smaili">Kamel Smaili</name></author><author><name sortKey="Zitouni, Imed" sort="Zitouni, Imed" uniqKey="Zitouni I" first="Imed" last="Zitouni">Imed Zitouni</name></author></analytic><series><title level="j" type="main">Computer speech & language</title><title level="j" type="abbreviated">Comput. speech lang.</title><idno type="ISSN">0885-2308</idno><imprint><date when="2001">2001</date></imprint></series></biblStruct></sourceDesc><seriesStmt><title level="j" type="main">Computer speech & language</title><title level="j" type="abbreviated">Comput. speech lang.</title><idno type="ISSN">0885-2308</idno></seriesStmt></fileDesc><profileDesc><textClass><keywords scheme="KwdEn" xml:lang="en"><term>Assessment</term><term>Formalization</term><term>Language model</term><term>Mathematical linguistics</term><term>Methodology</term><term>Probabilistic model</term><term>Speech processing</term><term>Speech recognition</term><term>Statistics</term><term>Test</term></keywords><keywords scheme="Pascal" xml:lang="fr"><term>Linguistique mathématique</term><term>Traitement automatique de la parole</term><term>Reconnaissance de la parole</term><term>Formalisation</term><term>Modèle probabiliste</term><term>Evaluation</term><term>Méthodologie</term><term>Test</term><term>Statistique</term><term>Estimation de la perplexité</term><term>Modèle de langage</term></keywords><keywords scheme="Wicri" type="topic" xml:lang="fr"><term>Statistique</term></keywords></textClass></profileDesc></teiHeader><front><div type="abstract" xml:lang="en">Language models are usually evaluated on test texts using the perplexity derived from the model likelihood function computed on these texts (test set perplexity). In order to use this measure in the framework of a comparative evaluation campaign, we have developed an alternative scheme for estimating the test set perplexity. The method is derived from the Shannon game and based on a gambling approach on the next word to come in a truncated sentence. We also study the entropy bounds proposed by Shannon and based on the rank of the correct answer, in order to estimate a perplexity interval for non-probabilistic language models. The relevance of the approach is validated on an example. We then report the results of a preliminary comparative evaluation using the proposed scheme.</div></front></TEI><affiliations><list><country><li>France</li></country><region><li>Grand Est</li><li>Lorraine (région)</li><li>Provence-Alpes-Côte d'Azur</li><li>Région Bretagne</li><li>Île-de-France</li></region><settlement><li>Avignon</li><li>Orsay</li><li>Paris</li><li>Rennes LIA</li> <li>Vandœuvre-lès-Nancy</li></settlement><orgName><li>Université Paris-Sud</li><li>Université d'Avignon</li></orgName></list><tree><noCountry><name sortKey="El Beze, Marc" sort="El Beze, Marc" uniqKey="El Beze M" first="Marc" last="El-Beze">Marc El-Beze</name><name sortKey="Igounet, Stephane" sort="Igounet, Stephane" uniqKey="Igounet S" first="Stéphane" last="Igounet">Stéphane Igounet</name><name sortKey="Jardino, Michele" sort="Jardino, Michele" uniqKey="Jardino M" first="Michèle" last="Jardino">Michèle Jardino</name><name sortKey="Smaili, Kamel" sort="Smaili, Kamel" uniqKey="Smaili K" first="Kamel" last="Smaili">Kamel Smaili</name><name sortKey="Zitouni, Imed" sort="Zitouni, Imed" uniqKey="Zitouni I" first="Imed" last="Zitouni">Imed Zitouni</name></noCountry><country name="France"><region name="Île-de-France"><name sortKey="Bimbot, Frederie" sort="Bimbot, Frederie" uniqKey="Bimbot F" first="Frédérie" last="Bimbot">Frédérie Bimbot</name></region><name sortKey="Bimbot, Frederie" sort="Bimbot, Frederie" uniqKey="Bimbot F" first="Frédérie" last="Bimbot">Frédérie Bimbot</name><name sortKey="Bimbot, Frederie" sort="Bimbot, Frederie" uniqKey="Bimbot F" first="Frédérie" last="Bimbot">Frédérie Bimbot</name><name sortKey="Bimbot, Frederie" sort="Bimbot, Frederie" uniqKey="Bimbot F" first="Frédérie" last="Bimbot">Frédérie Bimbot</name><name sortKey="Bimbot, Frederie" sort="Bimbot, Frederie" uniqKey="Bimbot F" first="Frédérie" last="Bimbot">Frédérie Bimbot</name></country></tree></affiliations></record>Pour manipuler ce document sous Unix (Dilib)
EXPLOR_STEP=$WICRI_ROOT/Wicri/Lorraine/explor/InforLorV4/Data/Main/Merge
HfdSelect -h $EXPLOR_STEP/biblio.hfd -nk 009A03 | SxmlIndent | more
Ou
HfdSelect -h $EXPLOR_AREA/Data/Main/Merge/biblio.hfd -nk 009A03 | SxmlIndent | more
Pour mettre un lien sur cette page dans le réseau Wicri
{{Explor lien
|wiki= Wicri/Lorraine
|area= InforLorV4
|flux= Main
|étape= Merge
|type= RBID
|clé= Francis:524-02-10502
|texte= An alternative scheme for perplexity estimation and its assessment for the evaluation of language models
}}
| This area was generated with Dilib version V0.6.33. |  |