Quality Evaluation¶

Introduction¶

The best way to measure the quality of overall decompilation is still an open problem in decompilation research. However, there have been a variety of methods for evaluating the quality of individual fundamental components in decompilation.

Control Flow Structuring Metrics¶

Early evaluations in control flow structuring utilized metrics from software engineering. As such, these metrics were not compared to their original source, but instead used standalone. This was useful because these metrics could be computed in real-world scenarios of decompilation use and reduced in runtime. The following early metrics were used:

• Gotos¹²³⁴⁵
• Full-function Recompilability²
• Lines of Code (LoC)⁴
• McCabe Cyclomatic Complexity (MCC)⁵

In all but the case of function recompilability, the decompiler would want to optimize for reducing all of these metrics. However, recent work has argued against the pure reduction of these metrics⁵. The SAILR paper⁵, argues that these metrics should be measured relative to its source (like machine translation). The following metrics were used as compared to source:

• Gotos
• Boolean Expression
• Function Calls
• Graph Edit Distance (GED)

In each metric, the score closest to the source is the best structured decompilation. Since these metrics can only be used with knowledge of source, they can only be used when developing a decompiler (not optimized in runtime).

Variable Typing Metrics¶

Unlike control flow structuring metrics, most evaluations for variable typing have focused on comparing choices to the source⁶⁷⁸⁹. Generally, these evaluations would flow like the following:

1. Extract the ground-truth types from the source (usually with DWARF info)
2. Guess variables and types from decompilation
3. See how often (or how close) each chosen variable location and type was to source

Some evaluations have also measured how conservative they were at approximating types⁷.

Human Evaluation¶

Some work has utilized human evaluations to understand the quality of decompilation¹⁰¹¹. These works have focused on qualitative metrics, such as users' perception of the code's complexity¹⁰. Additionally, the Decomperson¹² paper studied how humans might make decompilation themselves when given assembly.

Since how humans rate decompilation is related to how they reverse engineer, some work has explored how humans reverse and exploit binaries. These studies have mainly focused on how hackers use tooling and techniques for reverse engineering¹⁵¹⁶.

Other General Metrics¶

Some work has constructed a taxonomy of decompiler-to-source errors¹³. That work categorized and identified many errors that occur when decompiling a binary.

Other work has identified general methods by which we may test the correctness of decompilation¹⁴. Relatedly, some structuring work has attempted to measure this through recompiliability².

Dataset Generation¶

The question of "what dataset?" to use has also been an issue in decompilation. Additionally, compiled binaries can sometimes be hard to access. Some related work has explored ways to generate bigger binary datasets for the evaluation of binary tools like decompilers¹⁷.