Cross-post originally posted on the Ada Logics blog by David Korczynski, CEO of Ada Logics.
This fall we partnered with Ada Logics on securing the Fluent Bit log processor project by way of fuzzing integration, vulnerability analysis and bug fixing of Fluent Bit. This service is a bit different from a regular security audit in that it takes a holistic approach to security analysis of a project and the goal of the project was exactly to increase the overall security posture of Fluent Bit, both in the short term by fixing the bugs found as well as the long term by integrating continuous vulnerability analysis into Fluent Bit. The project yielded great results with more than 30 bugs uncovered of which 16 are fixed already. Additionally, the fuzzers continue to run on a daily basis in order to further explore the code and discover when regressions happen. In this post we will cover the project from a high-level perspective and details on the project can be found in the published report (link).
Fluent Bit is an open source log processor and forwarder which allows you to collect any data like metrics and logs from different sources. It is designed with performance in mind and is, therefore, written in the C programming language. This makes Fluent Bit susceptible to memory corruption vulnerabilities and bugs, and it is imperative to eliminate these from both security and reliability perspectives. In recent years there has been great progress in tooling support for assisting the process of finding these bugs and vulnerabilities, and fuzzing is one of the techniques where tool support has significantly improved.
From a high-level perspective, fuzzing is a technique that analyses code by executing it with an endless stream of random inputs. In essence, a fuzzer simply executes a piece of code with a random input over and over again, and the goal is to stress test the target code. This in itself is useful to catch certain classes of bugs in the code, however, fuzzing works exceptionally well when used in combination with runtime sanitizers. Sanitizers are techniques that compile into a target program with the purpose of adding additional bug-analysis during runtime. This additional analysis enables sanitizers to catch bugs that may otherwise not crash an application, but that are still important from security and reliability perspectives, for example, Address Sanitizer can detect buffer-overflows of various sorts.
The entry barrier for integrating fuzzing into open source projects has also reduced a lot in recent years and writing a fuzzer for a given project is comparable to writing a unit test (see this presentation link). In addition to this, Google offers OSS-Fuzz, which is a free service for open source projects that will run the fuzzers continuously and report back to developers when bugs are found. This proves incredibly useful as the infrastructure for actually running is completely outsourced, while control of the fuzzers still resides with the developers.
The goal of this project was to integrate the above fuzzing techniques into the Fluent Bit project. Specifically, the goal was to integrate fuzzers that analyse the Fluent Bit source code and then integrate the project into OSS-Fuzz in order to ensure continuous analysis. Finally, we also performed a mixture of root-cause analysis and bug fixing on the bugs that our newly developed fuzzers found.
We implemented a diverse set of fuzzers for Fluent Bit. The top priority was to cover all of the parsers in the project with fuzzers, including parsers for json, logfmt and ltsv. We also created fuzzers for many of the utility routines in Fluent Bit, such as string processing functions, routines for packing data, http routines and more. The entire set up was integrated into the build system of Fluent Bit, which makes it straight-forward to build and run any of the fuzzers developed.
The fuzzers found a large variety of bugs, including bugs of various types as well as severity. The following diagram shows the division of bugs that were fixed during the engagement.
The majority of the fixed bugs were heap-overflows and NULL dereferences. The heap-overflows range between read-based and write-based overflows as well as the size of the overflow. In addition to this we also uncovered and fixed a stack-based buffer overflow as well as several memory leaks. In total, we fixed sixteen bugs during the engagement. In addition to the bugs fixed during the engagement the fuzzers found several more that are in the pipeline of being fixed.
The following list summaries the results of the engagement:
- 16 fuzzers developed for Fluent Bit;
- More than 30 bugs found in the code;
- 16 vulnerabilities and bugs fixed; and
- Integrated fuzzing to be continuous by way of OSS-Fuzz
In addition to this, we also found that two third party libraries that are embedded into Fluent Bit had not been updated for a while. These libraries were updated shortly following the engagement per recommendations by Ada Logics.
The details and specifics on how we wrote the fuzzers and the specific execution of the project is available in the report that exists in the Fluent Bit source code repository and can be downloaded here. The report shows details of a specific fuzzer as well as an example of a bug that is non-trivial to catch, but was found by the fuzzers. To get the full exposure, the source code developed during the engagement has all been pushed to the main branch of Fluent Bit and is accessible here.
The engagement proved fruitful in analysing the Fluent Bit engine for memory corruption issues. This has drastically increased the security posture of the engine, however, there is room for slight improvement in terms of increasing the coverage of the Fluent Bit project as well as fixing remaining bugs found by the existing fuzzers. We leave this for future work.
If you are interested in Ada Logics’s fuzzing integration, vulnerability analysis and bug fixing you can see details on there service using this link