JavaBlack Box Testing for Regulation Compliance and Software Certification

Black Box Testing for Regulation Compliance and Software Certification content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.

Black box testing is a testing approach that tries to find software flaws (usually security weaknesses) by examining the inputs and outputs of the program, and without using any knowledge on the application’s internal. The application is therefore a “black box” that can be interacted with, but not probed internally.

Black box testing is by no means a new technique, but as far as developers are concerned, until very recently it was not a very efficient one. This means security testing was typically done along with QA, using static analysis or other source code analysis technologies.

However, although black box testing was largely neglected by developers, it was heavily utilized on the “other side of the fence.” Hackers were using black box testing tools to find software vulnerabilities in products. These tools used the technique known as fuzzing, a simplistic brute-force approach that performed various input variations to try and cause it to fail and expose vulnerabilities.

Much to the dismay of many product vendors, fuzzing is constantly used by hackers to find and publicize security holes in various software components. These holes are released publicly, putting clients at risk and introducing vendors to unnecessary press, development, and potential legal costs.

As an example, this past July, security researcher H.D. Moore wrote a blog-a-day called “browserfun,” that released a new browser security vulnerability every day of the month. He called it the “month of browser bugs.”

As an emphasis to the effect this has had, two major vulnerabilities were disclosed the following month. In Black Hat 2006, a presentation showed how fuzzing was used to find a vulnerability in the Macintosh wireless drivers; the vulnerability allowed for a complete compromise of the computer.

There were two main reasons hackers could use black box testing and software companies could not. First and foremost was the time it took the fuzzing tool to find vulnerabilities. Although hackers seem to have all the time in the world, software companies are always pressed for time because time is money and efficiency is key.

The second reason developers missed out on fuzzing tools was the level of expertise these tools required from their operator. Black box testing tools often would be amateurish and built by the hacker him- (or her-) self, without much thought given to a usable user interface or proper documentation. Making use of these tools thus required a high level of experience as well as a lot of time.

In the past decade, some large software vendors, such as Cisco and Microsoft, started making use of home-grown black box testing tools, nicknamed “fuzzers.” These tools were very similar in nature to their amateurish brothers available publicly in that they were very specialized and not ready for mass consumption.

In most cases, they would be used by internal security groups, performing research more than they did QA. They were called “fuzzers” because the testing was done by altering, or “fuzzing,” a valid input to the application and looking at the response to see whether a flaw was found. These large software vendors quickly realized that using the same tools hackers use to test their own products would enable them to locate flaws in products during development, and spare the embarrassment of a public flaw disclosure.

Automated QA Security

In the past few months, several security vendors announced fuzzing-based black box testing products. In this new world of commercial fuzzers, software vendors now have a variety of QA security tools to automate vulnerability tackling on the QA level and in other stages of the development cycle.

Automated commercial fuzzers close the gap between the amateurish early fuzzers and an automated tool that can be used in proper product development lifecycle. In addition, testing automation reduces the time it takes to check the application for a security flaw into time spans that are acceptable for testing.

Similarly to the way the secret ingredient in Coca-Cola transformed foul-tasting medicine into a popular soft drink, commercial fuzzing tools have made black box testing by fuzzing available to common software testers, and have made fuzzing popular, much to the chagrin of hackers who wanted to keep the secret to themselves.

Regulation Compliance and Software Certification

An interesting usage for fuzzing comes in meeting regulation requirements and supplying certified secure code. One reason for the use of fuzzing in certification is the lack of false positives. When a black box testing tool, such as a fuzzer, finds a security vulnerability, you know the flaw is there because it was found by actually trying to trigger the problem. After all, if it walks like a duck and quacks like a duck, it must be a duck.

In addition, automated fuzzers go through a very deterministic set of tests (the count is usually hundreds of millions of tests); this means that the testing is the same for different products. The “black box” approach also helps certification because it does not require the product vendor to submit potentially sensitive design information or source code.

Large corporations such as banks, telcos, and others have been testing third-party vendor software before deployment in their networks. In the past, establishing a certain product’s reliability and security was limited. Stress testing and a study of the vulnerability history of the product and vendor was the only metric, and the accuracy was low.

Those organizations are now using fuzzing technology to automate the testing process and benchmark vendors’ products. Several network equipment manufacturers have received a black eye when their large telco customer found several security vulnerabilities in their latest version of the networking product.

By using automated fuzzing tools, these organizations also can perform comprehensive testing on the software they consider purchasing without the availability of the source code, much like hackers do. They can stress the resilience of the product against malicious input and discover how easy or how difficult it is to crash it and potentially execute code.

Not every organization can afford to keep testing personnel on staff, and this is where security software certification comes in. So far, this has been limited to standard tools such as anti viruses, firewalls, and some others that compose a very specific set of security tools that are certified today. However, the introduction of advanced fuzzing technology makes the process of testing software for security issues a lot easier; this enables certification organizations to test a wide range of products for security flaws practically and provide certification to those who pass the test.

New certifications are bound to appear and, as time progresses, these certifications will be required with every shipped product in a similar way to how some organizations today only initiate partnerships and alliances with others who are BS9977 (ISO19977) compliant.

Another type of certification that is already manifesting itself in the market is “required by client.” Vendors in the networking world, for example, who work to certify themselves as very large clients, demand that they be certified as tested. “You need to be tested by XYZ to work with us.” Fuzzing provides a way to test those products efficiently and objectively.


Commercial fuzzing and the advances in black box testing technology that Second Generation Fuzzing introduced elevate fuzzing from the hackers’ playground into the corporate world.

Software security is a complex field, and more than just one bullet, silver or otherwise, is required if a high level of security is to be achieved. Still, the technique for black box testing introduces the ability for product vendors, customers, and third parties to test software and find faults.

Implementing these tools as standard processes is a step most organizations should consider. After all, the alternative is to bury your head in the sand. The bad guys are already using these tools to find zero-day vulnerabilities and sell them to the highest bidder.

About the Author

Gadi Evron works for the McLean, VA based vulnerability assessment solution vendor Beyond Security as Security Evangelist and is the chief editor of the security portal SecuriTeam. He is a known leader in the world of Internet security operations, and especially in the realm of botnets and phishing as well as is the operations manager for the Zeroday Emergency Response Team (ZERT). He is a known expert on corporate security and espionage threats. Previously Gadi was the Israeli Government Internet Security Operations Manager (CISO) and the Israeli Government CERT Manager which he founded.

Get the Free Newsletter!

Subscribe to Developer Insider for top news, trends & analysis

Latest Posts

Related Stories