Adaptive workflow processing and execution in Pegasus
Identifieur interne : 000517 ( Istex/Curation ); précédent : 000516; suivant : 000518Adaptive workflow processing and execution in Pegasus
Auteurs : Kevin Lee [Royaume-Uni] ; Norman W. Paton [Royaume-Uni] ; Rizos Sakellariou [Royaume-Uni] ; Ewa Deelman [États-Unis] ; Alvaro A. A. Fernandes [Royaume-Uni] ; Gaurang Mehta [États-Unis]Source :
- Concurrency and Computation: Practice and Experience [ 1532-0626 ] ; 2009-11.
English descriptors
- KwdEn :
Abstract
Workflows are widely used in applications that require coordinated use of computational resources. Workflow definition languages typically abstract over some aspects of the way in which a workflow is to be executed, such as the level of parallelism to be used or the physical resources to be deployed. As a result, a workflow management system has the responsibility of establishing how best to execute a workflow given the available resources. The Pegasus workflow management system compiles abstract workflows into concrete execution plans, and has been widely used in large‐scale e‐Science applications. This paper describes an extension to Pegasus whereby resource allocation decisions are revised during workflow evaluation, in the light of feedback on the performance of jobs at runtime. The contributions of this paper include: (i) a description of how adaptive processing has been retrofitted to an existing workflow management system; (ii) a scheduling algorithm that allocates resources based on runtime performance; and (iii) an experimental evaluation of the resulting infrastructure using grid middleware over clusters. Copyright © 2009 John Wiley & Sons, Ltd.
Url:
DOI: 10.1002/cpe.1446
Links toward previous steps (curation, corpus...)
- to stream Istex, to step Corpus: Pour aller vers cette notice dans l'étape Curation :000517
Links to Exploration step
ISTEX:E9865BF338161F2DE0A77B1AF1E62FC63910A1FELe document en format XML
<record><TEI wicri:istexFullTextTei="biblStruct"><teiHeader><fileDesc><titleStmt><title xml:lang="en">Adaptive workflow processing and execution in Pegasus</title>
<author><name sortKey="Lee, Kevin" sort="Lee, Kevin" uniqKey="Lee K" first="Kevin" last="Lee">Kevin Lee</name>
<affiliation wicri:level="1"><mods:affiliation>School of Computer Science, University of Manchester, Oxford Road, Manchester M13 9PL, U.K.</mods:affiliation>
<country xml:lang="fr">Royaume-Uni</country>
<wicri:regionArea>School of Computer Science, University of Manchester, Oxford Road, Manchester M13 9PL</wicri:regionArea>
</affiliation>
</author>
<author><name sortKey="Paton, Norman W" sort="Paton, Norman W" uniqKey="Paton N" first="Norman W." last="Paton">Norman W. Paton</name>
<affiliation wicri:level="1"><mods:affiliation>School of Computer Science, University of Manchester, Oxford Road, Manchester M13 9PL, U.K.</mods:affiliation>
<country xml:lang="fr">Royaume-Uni</country>
<wicri:regionArea>School of Computer Science, University of Manchester, Oxford Road, Manchester M13 9PL</wicri:regionArea>
</affiliation>
</author>
<author><name sortKey="Sakellariou, Rizos" sort="Sakellariou, Rizos" uniqKey="Sakellariou R" first="Rizos" last="Sakellariou">Rizos Sakellariou</name>
<affiliation wicri:level="1"><mods:affiliation>School of Computer Science, University of Manchester, Oxford Road, Manchester M13 9PL, U.K.</mods:affiliation>
<country xml:lang="fr">Royaume-Uni</country>
<wicri:regionArea>School of Computer Science, University of Manchester, Oxford Road, Manchester M13 9PL</wicri:regionArea>
</affiliation>
</author>
<author><name sortKey="Deelman, Ewa" sort="Deelman, Ewa" uniqKey="Deelman E" first="Ewa" last="Deelman">Ewa Deelman</name>
<affiliation wicri:level="1"><mods:affiliation>University of Southern California, Information Sciences Institute, 4676 Admiralty Way, Marina Del Ray, CA 90292, U.S.A.</mods:affiliation>
<country xml:lang="fr">États-Unis</country>
<wicri:regionArea>University of Southern California, Information Sciences Institute, 4676 Admiralty Way, Marina Del Ray, CA 90292</wicri:regionArea>
</affiliation>
</author>
<author><name sortKey="Fernandes, Alvaro A A" sort="Fernandes, Alvaro A A" uniqKey="Fernandes A" first="Alvaro A. A." last="Fernandes">Alvaro A. A. Fernandes</name>
<affiliation wicri:level="1"><mods:affiliation>School of Computer Science, University of Manchester, Oxford Road, Manchester M13 9PL, U.K.</mods:affiliation>
<country xml:lang="fr">Royaume-Uni</country>
<wicri:regionArea>School of Computer Science, University of Manchester, Oxford Road, Manchester M13 9PL</wicri:regionArea>
</affiliation>
</author>
<author><name sortKey="Mehta, Gaurang" sort="Mehta, Gaurang" uniqKey="Mehta G" first="Gaurang" last="Mehta">Gaurang Mehta</name>
<affiliation wicri:level="1"><mods:affiliation>University of Southern California, Information Sciences Institute, 4676 Admiralty Way, Marina Del Ray, CA 90292, U.S.A.</mods:affiliation>
<country xml:lang="fr">États-Unis</country>
<wicri:regionArea>University of Southern California, Information Sciences Institute, 4676 Admiralty Way, Marina Del Ray, CA 90292</wicri:regionArea>
</affiliation>
</author>
</titleStmt>
<publicationStmt><idno type="wicri:source">ISTEX</idno>
<idno type="RBID">ISTEX:E9865BF338161F2DE0A77B1AF1E62FC63910A1FE</idno>
<date when="2009" year="2009">2009</date>
<idno type="doi">10.1002/cpe.1446</idno>
<idno type="url">https://api.istex.fr/document/E9865BF338161F2DE0A77B1AF1E62FC63910A1FE/fulltext/pdf</idno>
<idno type="wicri:Area/Istex/Corpus">000517</idno>
<idno type="wicri:Area/Istex/Curation">000517</idno>
</publicationStmt>
<sourceDesc><biblStruct><analytic><title level="a" type="main" xml:lang="en">Adaptive workflow processing and execution in Pegasus</title>
<author><name sortKey="Lee, Kevin" sort="Lee, Kevin" uniqKey="Lee K" first="Kevin" last="Lee">Kevin Lee</name>
<affiliation wicri:level="1"><mods:affiliation>School of Computer Science, University of Manchester, Oxford Road, Manchester M13 9PL, U.K.</mods:affiliation>
<country xml:lang="fr">Royaume-Uni</country>
<wicri:regionArea>School of Computer Science, University of Manchester, Oxford Road, Manchester M13 9PL</wicri:regionArea>
</affiliation>
</author>
<author><name sortKey="Paton, Norman W" sort="Paton, Norman W" uniqKey="Paton N" first="Norman W." last="Paton">Norman W. Paton</name>
<affiliation wicri:level="1"><mods:affiliation>School of Computer Science, University of Manchester, Oxford Road, Manchester M13 9PL, U.K.</mods:affiliation>
<country xml:lang="fr">Royaume-Uni</country>
<wicri:regionArea>School of Computer Science, University of Manchester, Oxford Road, Manchester M13 9PL</wicri:regionArea>
</affiliation>
</author>
<author><name sortKey="Sakellariou, Rizos" sort="Sakellariou, Rizos" uniqKey="Sakellariou R" first="Rizos" last="Sakellariou">Rizos Sakellariou</name>
<affiliation wicri:level="1"><mods:affiliation>School of Computer Science, University of Manchester, Oxford Road, Manchester M13 9PL, U.K.</mods:affiliation>
<country xml:lang="fr">Royaume-Uni</country>
<wicri:regionArea>School of Computer Science, University of Manchester, Oxford Road, Manchester M13 9PL</wicri:regionArea>
</affiliation>
</author>
<author><name sortKey="Deelman, Ewa" sort="Deelman, Ewa" uniqKey="Deelman E" first="Ewa" last="Deelman">Ewa Deelman</name>
<affiliation wicri:level="1"><mods:affiliation>University of Southern California, Information Sciences Institute, 4676 Admiralty Way, Marina Del Ray, CA 90292, U.S.A.</mods:affiliation>
<country xml:lang="fr">États-Unis</country>
<wicri:regionArea>University of Southern California, Information Sciences Institute, 4676 Admiralty Way, Marina Del Ray, CA 90292</wicri:regionArea>
</affiliation>
</author>
<author><name sortKey="Fernandes, Alvaro A A" sort="Fernandes, Alvaro A A" uniqKey="Fernandes A" first="Alvaro A. A." last="Fernandes">Alvaro A. A. Fernandes</name>
<affiliation wicri:level="1"><mods:affiliation>School of Computer Science, University of Manchester, Oxford Road, Manchester M13 9PL, U.K.</mods:affiliation>
<country xml:lang="fr">Royaume-Uni</country>
<wicri:regionArea>School of Computer Science, University of Manchester, Oxford Road, Manchester M13 9PL</wicri:regionArea>
</affiliation>
</author>
<author><name sortKey="Mehta, Gaurang" sort="Mehta, Gaurang" uniqKey="Mehta G" first="Gaurang" last="Mehta">Gaurang Mehta</name>
<affiliation wicri:level="1"><mods:affiliation>University of Southern California, Information Sciences Institute, 4676 Admiralty Way, Marina Del Ray, CA 90292, U.S.A.</mods:affiliation>
<country xml:lang="fr">États-Unis</country>
<wicri:regionArea>University of Southern California, Information Sciences Institute, 4676 Admiralty Way, Marina Del Ray, CA 90292</wicri:regionArea>
</affiliation>
</author>
</analytic>
<monogr></monogr>
<series><title level="j">Concurrency and Computation: Practice and Experience</title>
<title level="j" type="abbrev">Concurrency Computat.: Pract. Exper.</title>
<idno type="ISSN">1532-0626</idno>
<idno type="eISSN">1532-0634</idno>
<imprint><publisher>John Wiley & Sons, Ltd.</publisher>
<pubPlace>Chichester, UK</pubPlace>
<date type="published" when="2009-11">2009-11</date>
<biblScope unit="volume">21</biblScope>
<biblScope unit="issue">16</biblScope>
<biblScope unit="page" from="1965">1965</biblScope>
<biblScope unit="page" to="1981">1981</biblScope>
</imprint>
<idno type="ISSN">1532-0626</idno>
</series>
<idno type="istex">E9865BF338161F2DE0A77B1AF1E62FC63910A1FE</idno>
<idno type="DOI">10.1002/cpe.1446</idno>
<idno type="ArticleID">CPE1446</idno>
</biblStruct>
</sourceDesc>
<seriesStmt><idno type="ISSN">1532-0626</idno>
</seriesStmt>
</fileDesc>
<profileDesc><textClass><keywords scheme="KwdEn" xml:lang="en"><term>adaptive</term>
<term>rescheduling</term>
<term>workflows</term>
</keywords>
</textClass>
<langUsage><language ident="en">en</language>
</langUsage>
</profileDesc>
</teiHeader>
<front><div type="abstract" xml:lang="en">Workflows are widely used in applications that require coordinated use of computational resources. Workflow definition languages typically abstract over some aspects of the way in which a workflow is to be executed, such as the level of parallelism to be used or the physical resources to be deployed. As a result, a workflow management system has the responsibility of establishing how best to execute a workflow given the available resources. The Pegasus workflow management system compiles abstract workflows into concrete execution plans, and has been widely used in large‐scale e‐Science applications. This paper describes an extension to Pegasus whereby resource allocation decisions are revised during workflow evaluation, in the light of feedback on the performance of jobs at runtime. The contributions of this paper include: (i) a description of how adaptive processing has been retrofitted to an existing workflow management system; (ii) a scheduling algorithm that allocates resources based on runtime performance; and (iii) an experimental evaluation of the resulting infrastructure using grid middleware over clusters. Copyright © 2009 John Wiley & Sons, Ltd.</div>
</front>
</TEI>
</record>
Pour manipuler ce document sous Unix (Dilib)
EXPLOR_STEP=$WICRI_ROOT/Ticri/CIDE/explor/CyberinfraV1/Data/Istex/Curation
HfdSelect -h $EXPLOR_STEP/biblio.hfd -nk 000517 | SxmlIndent | more
Ou
HfdSelect -h $EXPLOR_AREA/Data/Istex/Curation/biblio.hfd -nk 000517 | SxmlIndent | more
Pour mettre un lien sur cette page dans le réseau Wicri
{{Explor lien |wiki= Ticri/CIDE |area= CyberinfraV1 |flux= Istex |étape= Curation |type= RBID |clé= ISTEX:E9865BF338161F2DE0A77B1AF1E62FC63910A1FE |texte= Adaptive workflow processing and execution in Pegasus }}
![]() | This area was generated with Dilib version V0.6.25. | ![]() |