Traditional behavior analysis usually operates at the implementation level of a malicious behavior. Yet, it is mostly concerned with the identification of a given behavior, independently of its technical implementation, and is therefore more naturally defined at a functional level. In this thesis, we define a form of program behavior analysis which operates on the function realized by a program rather than on its elementary interactions with the system. This function is extracted from program traces, a process we call abstraction. We define in a simple, intuitive and formal way the basic functionalities to abstract and the behaviors to detect, then we propose an abstraction mechanism applicable both to a static or to a dynamic analysis setting, with practical algorithms of reasonable complexity, finally we describe a behavior analysis technique integrating this abstraction mechanism. Our method is particularly suited to the analysis of programs written in high level languages or with a known source code, for which static analysis is facilitated: programs intended for virtual machines like Java or .NET, Web scripts, browser addons, off-the-shelf components. The formalism we propose for behavior analysis by abstraction relies on the theory of string and terms rewriting, word and tree languages and model checking. It allows an efficient identification of functionalities in traces and thus the construction of a representation of traces at a functional level; it defines functionalities and behaviors in a natural way, using temporal logic formulas, which assure their simplicity and their flexibility and enables the use of model checking techniques for behavior detection; it operates on an unrestricted set of execution traces; it handles the data flow in execution traces; and it allows the consideration of uncertainty in the identification of functionalities, with no complexity overhead. We validate our results on a set of experiments, which we conducted on existing malicious codes, whose traces are obtained either by dynamic binary instrumentation or by static analysis.

EXPLOR_STEP=$WICRI_ROOT/Wicri/Lorraine/explor/InforLorV4/Data/Hal/Checkpoint

HfdSelect -h $EXPLOR_STEP/biblio.hfd -nk 001A54 | SxmlIndent | more

HfdSelect -h $EXPLOR_AREA/Data/Hal/Checkpoint/biblio.hfd -nk 001A54 | SxmlIndent | more

{{Explor lien
   |wiki=    Wicri/Lorraine
   |area=    InforLorV4
   |flux=    Hal
   |étape=   Checkpoint
   |type=    RBID
   |clé=     Hal:tel-00646395
   |texte=   Analysis of Malware by Behavior Abstraction
}}