Security Through Effective Penetration Testing 

Download Article Article in Digital Form

In the spring and summer of 2010, a team of eight people was diligently pursuing one common goal: hacking into the network of the State of Colorado, USA. The team had spent countless days, nights and weekends trying to breach the state’s systems, modems, wireless network devices and 67,000 Internet Protocol addresses to gain access to personally identifiable, sensitive and/or confidential information.

On one particular afternoon, the team achieved a significant victory. A state agency’s public-facing web site had been misconfigured, allowing the team to view the entire site’s directory of files, including one called “upload. html.” Within minutes of identifying the file, the team had uploaded software that allowed them to gain administrator access to the agency’s internal network inside the firewall. From there, it was a matter of how quickly they could access other connected systems and obtain the information they sought before moving on to a different target at a different agency. Figure 1 illustrates how the agency’s firewall was bypassed to attack the agency’s internal network.

Figure 1

The good news for the State of Colorado was that the team was not a group of professional hackers, but rather three IT auditors employed by the Colorado Office of the State Auditor (OSA) and five additional computer security professionals contracting with the OSA. The OSA team’s assignment was to conduct a large-scale, covert penetration test that would assess the state’s risk of being compromised by a malicious attacker and to recommend steps for preventing such attacks.

Over a six-month testing period, the team compromised several state government networks and systems and gained unauthorized access to thousands of individuals’ records, including records containing confidential data such as US Social Security numbers, income levels, birth dates and contact information. The team also gained access to usernames and passwords belonging to state employees and other individuals. Based on national averages, a data breach of this magnitude by a malicious individual would have cost the state between US $7 and $15 million to remediate.1 This estimate does not include the cost to individual citizens whose data would have been stolen. The audit concluded that the state was at a high risk of a system compromise and/or data breach by malicious individuals, including individuals both internal and external to the state.

The OSA published the general results of its penetration test in a November 2010 report that was released publicly.2 In addition, the OSA drafted confidential reports containing specific details of the test results and provided those to individual state agencies whose systems and protected information had been breached. It was one of the first times that an entire state government had undergone such a large-scale assessment to identify security vulnerabilities across various critical resources and to evaluate how well IT staff at different agencies responded to simulated attacks. No target was considered too minor; hacking into individual e-mail accounts was as much a goal as trying to pilfer protected taxpayer data from agency databases. Successful penetration of any critical resource could identify vulnerabilities that could become targets for real hackers.

Benefits of Penetration Testing

Large-scale, covert penetration tests can be an effective tool for governments, private companies, and other national and international organizations to assess the security of their critical resources. According to the US National Institute of Standards and Technology, the purpose of covert security testing is to “examine the damage or impact an adversary can cause,” rather than to identify specific vulnerabilities.3 For instance, covert penetration tests do not “test every security control, identify each vulnerability or assess all systems within an organization.”4 Given their unique nature, these tests can provide various benefits, which are discussed in the following sections.

The Element of Surprise
Covert penetration tests provide a valuable element of surprise because the IT staff who must respond to a system or data breach are not aware that the breach is a test. Small-scale, overt penetration tests typically focus on specific systems and are conducted under the watchful eye of an organization’s IT staff, whereas large-scale, covert penetration tests can be unrestricted in scope and occur without the knowledge of those responsible for addressing system or data compromises. In Colorado, only the state’s chief information security officer (CISO) and other executive-level staff were aware of and authorized the OSA to simulate real attacks against the state’s critical resources.

“Covert penetration testing allowed the State of Colorado to measure the effectiveness of application, host and network monitoring and detection of anomalous and malicious activity and identified gaps with our statewide and departmental incident response capabilities,” says Travis Schack, the State of Colorado’s CISO. “Noncovert testing would not have provided objective and accurate evidence for areas of improvement in our technical capabilities, processes, and user education and awareness of their security role and responsibilities.”

It is important to remember that professional hackers will not provide advance warning of an attack. A common US military training doctrine is that troops will fight like they train; this same philosophy should be applied to IT security staff. It is best for training to be as realistic as possible.

Motivation for Decision Makers to Address IT Security Vulnerabilities
Overt penetration testers often stop their efforts once they break into a system, usually proving their success by taking screenshots of systems they have breached or leaving behind a text file to announce their presence. A drawback to that approach is that it does not fully demonstrate the real-life impact an organization would sustain if it were to fall victim to a hacker. As a result, high-ranking decision makers may not fully appreciate the potential business implications of ignoring IT security vulnerabilities. The ISACA publication An Introduction to the Business Model for Information Security discusses this phenomenon, noting that information security managers “are speaking in terms of specific threats, risks, controls and technologies while business managers are talking about cost, productivity and return on investment.”5

The OSA team decided to maximize the covert nature of its test and actually demonstrate what could happen if real hackers were to penetrate the state’s systems. The team members did not stop when they accessed a network or system. They also pillaged as much personally identifiable, sensitive and/or confidential information as they could to quantify what costs the state would have incurred if the breach had been real. Informing state policy makers that taxpayer information had been “stolen” and that addressing an actual data breach would have cost the state millions was much more compelling than providing them with a technical analysis of the state’s IT security vulnerabilities.

“When a person loses their wallet, they understand the impact it has to them and usually know when it happens,” says Schack. “Most people have a hard time understanding cyber and information security and the impacts an incident may have to their business. Usually, this is the result of security professionals not being successful in demonstrating the impact and translating security risks to business risks.”

Broader Understanding of IT Security Vulnerabilities
A large-scale penetration test allows an organization to assess all three pillars of its IT security posture—people, processes and technology—thereby providing a more comprehensive view of the organization’s IT security environment. Some industry experts believe that strategically integrating these three pillars into information security policies will help address the challenge of achieving effective cybersecurity.6 In Colorado, the OSA team attacked all three pillars of the state’s IT security posture (figure 2).

Figure 2

Through a phishing attack, for example, the OSA team obtained valid usernames and passwords of employees, which the team used to conduct simulated malicious activity. The OSA team tested security processes by attempting to bypass security barriers or controls at 18 physical sites or state buildings containing computer hardware or documents containing confidential information. The team was successful enough that it rated the state’s physical security as a high-risk area.

When testing technology, the OSA team treated low- and high-risk applications as equally important because hackers who gain access to unsecured low-risk applications could find the entry they need to breach higher-risk, better secured applications (figure 3).

Figure 3

Process testing included reviewing information about IT staff member roles, responsibilities and authorization levels, as well as security plans that Colorado law requires state agencies to develop. The OSA team found that 60 percent of agencies required to develop an information security plan had not done so by the statutory deadline. In addition, the plans were often incomplete, inaccurate, and lacking in detailed and meaningful information.

Evaluating all aspects of an organization’s IT security environment can provide a more thorough evaluation of its risks, information that is useful to IT professionals and decision makers.

To be successful, a large-scale, covert penetration test should be conducted as a partnership between the testers and a representative of the IT staff who will be responsible for remediating any vulnerabilities. Constant communication between the testers and that individual is critical. Before testing begins, both parties should determine how and when to communicate when certain events happen, such as a successful breach, the detection of a breach, problems caused by the testing or situations that could potentially involve law enforcement.

Even after testing is completed, an ongoing dialog should be maintained between the penetration testers and the IT staff responsible for securing the organization’s critical assets. When an organization faces financial or time limitations, this dialog becomes especially important to help the organization prioritize its remediation efforts. For a large-scale, covert penetration test to be a truly valuable exercise, the test results should be continually addressed and referenced long after the final report has been printed.

Preparing for Successful Penetration Testing
Commercial and government organizations conducting penetration tests should consider the following actions to help ensure a successful testing process:

  • Obtain unwavering executive management or managerial-level support for the project. Start early because obtaining buy-in can take a long time and is often one of the most challenging aspects of the entire process.
  • Develop a detailed scope that outlines goals and objectives, systems to be tested, systems not to be tested, and the project calendar. The scope should also incorporate rules of engagement that address areas including:
    • How worms and/or viruses will be used and how compromised machines will be repaired
    • Whether the penetration testing will be overt or covert
    • How potential legal ramifications will be handled if the testing team gains unauthorized access to systems or physical property
    • How denial of service can be prevented if an organization’s services or processes are interrupted by successful system breaches
  • Establish a plan for communicating to the individuals involved with and knowledgeable about the penetration test. The plan should specify how and when communication will occur if the testing team successfully compromises a system, IT security staff detect a breach and/or the penetration testing activity causes unintended problems for the organization.
  • Build a testing team of internal staff and/or contractors based on the organization’s IT environment, the type of penetration testing that will be performed, the knowledge and abilities of internal staff, and the desire to provide internal staff with opportunities to gain experience from working with specialized contractors. If using contractors, the organization should establish an effective contract monitoring process so contracted testers are held accountable to clearly defined expectations.
  • Prepare the testing team for success before it achieves it. Perhaps most important is obtaining a contract that states the legality of the engagement. This document should serve as a “get out of jail free” card if the team encounters law enforcement that believes a breach is the real thing. The team should also be equipped with tools and technological devices, such as wireless transmitters and receivers, switches, cables, and software (such as scanners, exploits, password crackers and proxies).

Overall Findings and Recommendations

In Public CIO, Travis Schack recommends that information security practitioners focus on consistently doing the basics of information security.7

From the OSA penetration test results, it was determined that most of the vulnerabilities that were exploited resulted from the lack of consistent application of information security best practices by IT and information security staff. The majority of the vulnerabilities exploited could be categorized as follows:

  • Weak and/or unchanged default passwords
  • Default system and application configurations
  • Failure to patch known vulnerabilities and use secure configurations enterprisewide
  • Failure to consistently apply the least-privilege access control model
  • Failure to use secure coding
  • Lack of security awareness by system users

The OSA team did not identify any new zero-day exploits and rarely required the use of sophisticated programming skills, but instead, exploited well-known vulnerabilities and process weaknesses that would have not existed if security best practices had been followed. Of particular concern to those performing the penetration test was that the majority of the systems exploited, including those that led to access of confidential records, could be performed by people with limited skills and knowledge and with tools and techniques readily available on the Internet. Additionally, the penetration test revealed serious weaknesses and vulnerabilities that would not have been identified and remediated through routine audits and/or security evaluations (e.g., exploitation of two-way trusts, the ability to pivot through the organization to attack well-secured systems and the likelihood of using social engineering to gain access to an organization’s most precious data).

In addition to routine IT audits and vulnerability assessments, commercial and government organizations should periodically (every three to five years) perform covert penetration tests of their enterprises. These tests should follow industry best practices and include methodologies to attack or test each of the three pillars of information security.8 The tests should also be conducted keeping in mind the key steps for successful penetration testing provided in this article.

Additionally, as part of the covert penetration test, the organization should assess the ability of its staff and systems to identify and respond to an ongoing attack. At the conclusion of the test, the testers should be thoroughly debriefed by the organization’s information security staff and should work in cooperation with security staff to identify key weaknesses, based on risk, and develop a detailed mitigation and remediation plan.


1 General industry information was obtained from Ponemon Institute LLC, “Fourth Annual US Cost of Data Breach Study,” USA, January 2009,, and applied to an analysis that the State of Colorado (USA) Office of the State Auditor (OSA) conducted during the audit, which resulted in the US $7 to $15 million cost estimate.
2 State of Colorado OSA, “Office of Cyber Security, Governor’s Office of Information Technology, Performance Audit November 2010,” USA, 2010
3 Scarfone, Karen; Murugiah Souppaya; Amanda Cody; Angela Orebaugh; US National Institute of Standards and Technology (NIST) Special Publication 800-115 “Technical Guide to Information Security Testing and Assessment,” NIST, USA, 2008,
4 Ibid.
5 ISACA, An Introduction to the Business Model for Information Security, USA, 2009,
6 Narus Inc., “Three Components of Cyber Security: People, Process and Technology,” 18 October 2010,
7 Schack, Travis; “Back to Security Basics,” Public CIO, October/November 2010,
8 ISACA, IS Auditing Procedure P8 Security Assessment—Penetration Testing and Vulnerability Analysis, USA, 2004,

Jonathan Trull, CISA, CFE, OSCP, is the deputy state auditor at the Colorado Office of the State Auditor (OSA), primarily responsible for overseeing complex, value-added IT audits in the State of Colorado, USA. He is also responsible for overseeing select performance audits, the OSA’s internal IT plan and infrastructure, and the OSA’s internal report writing and external communications functions. With more than 13 years of experience in governmental auditing and management, Trull has established himself as an innovative leader of high-quality, objective audits and reviews.

Enjoying this article? To read the most current ISACA Journal articles, become a member or subscribe to the Journal.

The ISACA Journal is published by ISACA. Membership in the association, a voluntary organization serving IT governance professionals, entitles one to receive an annual subscription to the ISACA Journal.

Opinions expressed in the ISACA Journal represent the views of the authors and advertisers. They may differ from policies and official statements of ISACA and/or the IT Governance Institute and their committees, and from opinions endorsed by authors’ employers, or the editors of this Journal. ISACA Journal does not attest to the originality of authors’ content.

© 2012 ISACA. All rights reserved.

Instructors are permitted to photocopy isolated articles for noncommercial classroom use without fee. For other copying, reprint or republication, permission must be obtained in writing from the association. Where necessary, permission is granted by the copyright owners for those registered with the Copyright Clearance Center (CCC), 27 Congress St., Salem, MA 01970, to photocopy articles owned by ISACA, for a flat fee of US $2.50 per article plus 25¢ per page. Send payment to the CCC stating the ISSN (1526-7407), date, volume, and first and last page number of each article. Copying for other than personal use or internal reference, or of articles or columns not owned by the association without express permission of the association or the copyright owner is expressly prohibited.