[2022.04.06] Reinforcement Learning-based Hierarchical Seed Scheduling for Greybox Fuzzing

Abstract

Coverage metrics play an essential role in greybox
fuzzing. Recent work has shown that fine-grained coverage
metrics could allow a fuzzer to detect bugs that cannot be covered
by traditional edge coverage. However, fine-grained coverage
metrics will also select more seeds, which cannot be efficiently
scheduled by existing algorithms. This work addresses this problem
by introducing a new concept of multi-level coverage metric
and the corresponding reinforcement-learning-based hierarchical
scheduler. Evaluation of our prototype on DARPA CGC showed
that our approach outperforms AFL and AFLFAST significantly: it
can detect 20% more bugs, achieve higher coverage on 83 out of
180 challenges, and achieve the same coverage on 60 challenges.
More importantly, it can detect the same number of bugs and
achieve the same coverage faster. On FuzzBench, our approach
achieves higher coverage than AFL++ (Qemu) on 10 out of 20
projects.