Can Bug Bounty Programmes Replace Pen Testing?
Security vulnerabilities are a real problem for software developers everywhere, and the organisations that employ them. Many issues are annoying but harmless, but the worst vulnerabilities in a piece of software could compromise security, and cause it to be vulnerable to a breach, which if realised, could cause untold damage to the organisation. The traditional approach to checking for vulnerabilities is penetration testing (or pen testing), but bug bounty programmes are increasingly being used by organisations to discover issues in their live products.
Bug bounty programmes are gaining traction, increasingly used to uncover vulnerabilities by some of the biggest names in technology, including Microsoft, Google, and even the US Government with their Hack the Pentagon program. These organisations pay out millions of dollars a year to bug bounty hunters, but can bug bounty programmes really replace penetration tests? Or can they serve a different, but complimentary, purpose?
Bug bounty programmes are used by organisations of all shapes and sizes to challenge independent security professionals to discover new vulnerabilities in their applications, software, websites, APIs, and more. Any security professional who finds a bug that has never been reported before will receive a bounty, or reward. Bug bounty programs provide a way for organisations to use their cyber security budget more effectively and only pay for results.
Bug bounty programs are often hosted on bug bounty platforms such as HackerOne, BugCrowd, or Synack. The organisation defines the goals and scope of the program, and sets bounties for different levels of bug, The program is then hosted on the bug bounty platforms open to all security professionals who are registered on them.
The reasons why an organisation will create a bug bounty program are varied, but they include: Getting continuous insight into the security of key systems; Utilising the skills of people outside the organisation, and getting access to very different skills that they might otherwise not access; Exposing the systems to people from outside the organisation who are not invested in it, and can therefore provide a realistic picture of the security situation; and ultimately to find bugs and resolve them before hackers do, and they become victims of a zero day attack.
Like bug bounty programs penetration testing uses third party ethical hackers to “attack” applications testing them for flaws and weaknesses. However, unlike bug bounty hunters, the penetration testers are usually accredited, and work within a cyber security company.
Penetration testing takes place within a defined scope and time frame, during which the tester is expected to uncover as many flaws as they can find, and provide a detailed security assessment of the application, website, or system being tested, including a list of flaws, and recommended mitigations to fix them. The penetration tester or cyber security company may also continue to work with the organisation, supporting the development team on an ongoing basis.
Both bug bounty programs and penetration tests are forms of ethical hacking used by organisations to improve the security of their products and systems. While the two testing models have similar end goals, below is a summary of some of the main differences between them.
Bug bounty programmes |
Penetration tests |
Pay for success – testers are only paid if they find proven bugs before anyone else |
Pay for time – testers are paid for a set of hours or days or by project |
Bug hunters are freelancers or contractors, registered on bug bounty platforms |
Pen testers work in cyber security companies |
Bug bounty hunters choose the projects they work on – the company has no control over who does the testing |
Organisations sign a contract with a specific company or tester to carry out the penetration tests |
Usually carried out on publicly accessible, published, or live products |
Can be used earlier in the process, before a product goes live |
Less defined or rigid scope for testing |
Conducted based on the specific terms of the client |
No specific deadlines for a programme enabling continuous testing |
Carried out as a snapshot in time, usually 2 or 3 weeks |
Focused on discovering vulnerabilities with little to no follow up |
Testers provide feedback, mitigation recommendations, and even ongoing support |
Table 1: Bug bounty programmes vs penetration testing. Which one is right for you?
While bug bounty programs several key benefits for organisations looking to improve the security of their products, there are some limitations to consider.
Loss of control over what is tested or reported
The organisations commissioning the bug bounty program simply define the program and then withdraw, leaving whoever is out there to do their thing. Therefore they do not have any say which bugs are being hunted. On some programs, hackers may be focused on more lucrative bugs, not identifying or reporting the bugs which attract smaller bounties, which could remain undetected until they are used to carry out an attack. In other programs, the bug bounty hunters may lack the skills to test more difficult systems choosing instead to focus on websites which are easier to check rather than operating systems, which are more complex.
Security concerns limit testing to published systems only
Bug bounty programmes are open to any independent security professional to test, and the system being checked is vulnerable to them. As a result, most organisations limit the systems being tested to ones that are already open to the public. Penetration testing by contrast can be carried out on applications before they are published, when it is more cost effective to fix vulnerabilities, and more secure overall.
No support for fixing vulnerabilities
When it comes to improving the security of a system or application, bug bounty programmes only carry out half the task. The bug bounty hunter will submit the bug, including a proof of concept of how they discovered it, but unlike a penetration testing team, they are not required to provide remediation advice, nor are they available to support the development team in recreating the vulnerability or remediating it. This requires the organisation to still have a level of skill and cyber security competence in order to fix the identified vulnerabilities.
They do not replace secure software development practices
The best way to secure a product is to build it securely from the start. Bug bounty programmes are a patch delivered on final products rather than coming into the process at an earlier stage before the vulnerability is released into the wild. As such, bug bounty programmes are not a replacement for a vulnerability management program which takes every element of the product, from development to ongoing monitoring into consideration.
Bug bounty programmes are not widely accepted by compliance frameworks
While bug bounty programs do demonstrate a commitment to security, and can help improve security overall, they sit outside the usual compliance framework. Many auditors, regulators, and even customers will not recognise bug bounty programs as an effective vulnerability management tool, and may require the organisation to carry out penetration tests by a certified tester in addition to the program, making the organisation pay twice for the same bugs.
Bug bounty programmes and penetration testing both work towards the same goal – improving the security posture of systems and applications. They both have a place in any organisation’s vulnerability management, but while an organisation may choose to just use penetration tests in their security management cycle, bug bounty programmes alone are not enough.
Where bug bounty programmes are a good way to get regular feedback on different areas of an organisation’s infrastructure, penetration tests carried out by a trusted professional with whom there is an ongoing relationship, and who is on hand to support mitigation efforts will provide more lasting benefits.
Stay safe