Traditional security metrics are haphazard at best; at worst they give a false impression of security that leads to inefficient or unsafe implementation of security measures. This paper presents an approach whereby maturity and quality are combined to provide a more complete and orderly picture of an organization’s security posture. The approach will be referred to as the Security Program Maturity Model.
Security metrics—the measurement of the effectiveness of the organization’s security efforts over time—have always been difficult to evaluate. How can an organization determine whether it is secure? The measure of the quality of the security program can be truly tested only when the organization is stressed by a crisis. Yet, this situation is exactly what the security effort is designed to prevent.
Management needs some measure of how secure the organization is. Organizations need to ask themselves:
How many resources does it take to be “safe”?
How can the cost of new security measures be justified?
Is the organization getting its money’s worth?
When does the organization know it is “safe”?
How does the organization compare its posture with others in the industry and with best practice standards?
The traditional answer to these questions relates to risk assessment and the residual risk the organization is willing to take based on business needs and budget limits. Risk management may beg the question, not necessarily leading to a stronger security stance.
Imagine, for instance, a risk assessment that lists a threat matrix and the cost to mitigate the risks. Some items on the list would be of insignificant cost. Other items will be very expensive (figure 1). Often, management may decide to mitigate the most items for the least amount of money, possibly bypassing the most expensive items. The assumption is that aggregating risk reduction controls is a better buy. Thus, there is a tendency to buy large quantities of security tools and avoid the more expensive, less glamorous controls. The more difficult controls tend to be organizational in nature requiring cultural change (such as a disaster recovery plan) rather than specific turnkey solutions [such as firewalls and intrusion detection systems (IDSs)]. Management thinks it is buying more security for less money.
However, who is to say that more security is purchased? How can the organization measure the relative protection gained by each purchase? Is the organization purchasing the security safeguards in the right order? Is the organization being exposed to more risk because of the unsystematic approach toward implementation?
Building security programs from the ground up allows for the development of new approaches toward these traditional security metrics problems. A fresh look at these problems enables the development of a comprehensive solution for any industry.
This newer, more systematic approach toward security metrics will:
Generate reproducible and justifiable measurements
Measure something of value to the organization
Determine real progress in security posture
Apply to a broad range of organizations while producing similar results
Determine the order in which security controls should be applied
Determine the resources needed to apply to the security program
Traditional Security Metrics—What to Count?
A measurement, by itself, is not a metric. Time has to be brought into the picture, and a metric alone is not the answer to all the organization’s problems. People have to think through and analyze the time meaning of the metrics. The trick is to develop metrics that are simple and provide useful management information matching security-related, goal-setting objectives. The metrics have to enlighten the organization by showing some type of progress.
Obviously, the task of security metrics is to count or measure something. But what should be counted? How can security be measured? Figure 2 shows samples of traditionally used security metrics. Many organizations count incidents handled, e.g., cyberviruses caught or logged events. How does this provide a measure of the quality of the security program? How does this show progress?
Figure 2—Traditional Security Metrics
Metric
Presumed Measurement
Pitfalls
Number of computer viruses/malicious code caught
Effectiveness of automated antivirus controls
Why are so many viruses getting through in the firstplace? How many got through that never got caught?
Number of security incidents and investigations
Activity of monitoring security events
What threshold causes an incident or investigation to get triggered? Are incidents triggered because of organizational process flaws?
Cost of security breaches
True business loss related to security failures
What residual risks did the business choose to take? Is it a measure of a crisis or disaster response, but not necessarily a function of reasonable safeguards in place?
Time and materials assigned to security functions
True business cost of running a security program
Are the tools, assignments or procedures inefficient, causing people to waste time?
Compliance with security rules
Level of compliance matching security program goals
How does compliance relate to effectiveness? What is the order of compliance? Once compliance is achieved, is the security program“finished”?
Incident totals are unreliable measures for this reason: imagine a small town with one police officer. He does no other police work other than patrolling the highway with a radar gun, pulling over hundreds of speeders. Now imagine a large town with many police officers. They do not use radar guns and have caught very few speeders but have a large defensive driving program and an active anti-drunk-driving program. Is the small town safer than the large town? The count of speeders is only as good as the sensing mechanism, but that number has no depth to it. What about the small town with nonspeeders who are drunk—are they not potentially more dangerous?
Now compare this to an antivirus tool in an information systems environment. The fact that it reports a large number of virus infections may make the security team feel good that its antivirus tool is working, but what does it really say about security? Why are so many viruses getting through in the first place? How many get through and remain undetected? How does that measure the quality of the security program? It should be considered a great success if the antivirus tool never finds any viruses because none ever gets into the environment!
Another traditional security metric is time spent on a task—how long people spend doing security-related functions. In some cases, from a project management point of view, this may be a valuable metric because the only two real resources people bring to an organization are their brainpower and the time they spend using it. But from a security standpoint, people’s time may not be a valuable metric. For instance, when measuring time spent on security investigations, if more time is spent, is that necessarily an indication of improved security posture? It could be that time is inefficiently used investigating security incidents because the organization’s procedures are weak—triggering more incidents to be handled by the security group when they could be prevented in other ways, such as better training.
Finally, another classic metric is cost to the business from damage by a security incident. This assumes something bad has happened in the first place. While it may measure the effectiveness of disaster response, it is not necessarily a good measure of the quality of the security program. Some incidents are a function of the residual risk the business is willing to take, combined with unlucky circumstances. Alternatively, other incidents may be the result of poor security practices opening the door for disaster. How can these be differentiated? Or, maybe one of these two events occurred and the crisis management was so good, it caused minimum business impact. Safeguards provide only a certain degree of security; there will always be some risks.
The key to security metrics is obtaining measurements that have the following ideal characteristics:
They should measure organizationally meaningful things.
They should be reproducible.
They should be objective and unbiased.
Over time, they should be able to measure some type of progression toward a goal.
In practice, almost all published security metrics are missing one or more of these characteristics. Traditional security metrics were a “take what you can get” affair, i.e., whatever metrics were available were grabbed and reported. This way of thinking should change.
A more systematic approach is needed for the development of metrics that directly fit the previously mentioned characteristics.
Security Program Maturity
One piece of the security metrics puzzle is to measure progress of the security program against a maturity model. This approach directly targets at least two of the four characteristics listed above: it measures organizationally meaningful things and progression toward a goal.
The few published security maturity models are summarized in figure 3. For some reason, each has only five levels of maturity. Each model seems to suffer from its own biases about the definition of maturity.
It is herewith proposed that a new standard for maturity be used. Maturity should be a measure of only the program’s progress over time, not necessarily the quality of the elements of the program. This definition of maturity has several important characteristics:
It provides the blueprint for a complete security program.
It tells management the order in which to implement security elements.
It leads toward the use of best practice standards (e.g., ISO 17799).1
As long as a standard is used, it provides a way to compare one organization’s security program to another.
Figure 3—Published Security Maturity Models
Model
Description
Comments
NIST CSEAT IT Security Maturity maturity: Model 2
Five levels of progressive
1. Policy
2. Procedure
3. Implementation
4. Testing
5. Integration
Focused toward levels of documentation
Citigroup’s Information Security Evaluation Model (CITI-ISEM)3
Five levels of progressive maturity:
1. Complacency
2. Acknowledgment
3. Integration
4. Common practice
5. Continuousimprovement
Focused toward organizational awareness and adoption
COBIT® Maturity Model 4
Five levels of progressive maturity:
1. Initial/ad hoc
2. Repeatable but intuitive
3. Defined process
4. Managed and measurable
5. Optimized
Focused toward auditing specific procedures
SSE-CMMModel 5
Five levels of progressive maturity:
1. Performed informally
2. Planned and tracked
3. Well-defined
4. Quantitatively controlled
5. Continuously improving
Focused toward security engineering and software design
CERT/CSO Security Capability Assessment 6
Five levels of progressive maturity:
1. Exists
2. Repeatable
3. Designated person
4. Documented
5. Reviewed and updated
Measures using four levels:
1. Initial
2. Evolving
3. Established
4. Managed
Focused toward measurement of quality relative to levels of documentation
Following this maturity standard, previous models suffer from three key deficiencies:
They confuse quality with existence. One must learn to walk before one can run. The quality of how well one walks is not necessarily an indication of one’s running ability.
The existing models need to be customized specifically for the organization. Therefore, it is difficult to directly compare results from one organization to another.
The current models tend to come from engineering or project management perspectives. Thus, they focus on program elements meeting certain engineering-style specifications. They do not necessarily drive the program toward a particular organizational goal and therefore work poorly for a security program. The incorporation of total quality management (TQM) and six Sigma philosophies are examples of this.
With security, the result should be more like an insurance policy. It is not like manufacturing a product. Instead, the corporation is socializing infrastructure and culture.
This new approach toward a detailed security maturity model (called the Security Program Maturity Model) takes a management systems approach. Therefore, it follows the ISO 17799 standards for developing a complete security program. It involves the existence or nonexistence of a large number of elements (figure 4).
Figure 4—General Outline of the Security Maturity Model
ISO 17799 Categories
No. of Elements Measured
Items Covered by Elements
1. Overall security management
11
Business need, strategy, commitment, roles and responsibilities, policies and procedures
2. Asset classification and control
5
Valuation, risk assessment, business ownership, labeling and handling, inventory
3. Personnel security
8
Hiring and termination, roles and responsibilities, screening, training, reporting, review
Perimeters, risk assessment, access controls, authentication, need to know,user responsibility, access updating, monitoring, mobile computing, incident handling
6. System development and maintenance
9
Standards, life cycle model, review, gap analysis, requirements planning, testing integrity and certification, code repository, release management, retirement
7. Communications and operations management
16
Standards, all methods of e-communications, operations procedures, monitoring, backups, exception handling, updates and patches, help desk, change management, cryptographic systems, media handling, malicious code, system acceptance, documentation library, capacity planning
8. Organizational security
11
Security function, monitoring, advisory, auditing, forum, awareness training, segregation of duties, penetration and vulnerability testing, incident handling, cooperation
9. Business continuity management
7
Risk assessment, prioritization, backups, business continuity/disaster recovery planning, testing, updates
10. Compliance
10
Regulatory, contractual, intellectual property, labeling and handling, record retention, auditing, sanctions
Since very little judgment is involved, the results are reproducible and objective. Quality or effectiveness of the element implementation is not measured, though certain elements (such as auditing programs), if executed, can lead toward other quality controls. This is similar to health department inspections of restaurants. Inspection scores can tell which restaurants to avoid, but they tell nothing about whether one will get a good meal at those that pass inspection.
Maturity level is an important measure when comparing one organization to another. If an organization’s maturity score is 75 percent, would it want to network into another organization whose score is only 25 percent?
Maturity level leads an organization to better understand its security program relative to its peers. It provides a yardstick with which to assess the degree of trust that can be placed with interconnected computer systems between different organizations.
This security maturity model is also a blueprint for the order that program elements should be implemented. Figure 5 provides an example of the step-by-step approach toward the implementation of elements. In a maturing program, the elements are executed based on the outcome of previous implementation steps. Consequently, it directly tells management when to “buy” security. It answers the question posed in figure 1.
Figure 5—Sample Elements From Security Maturity Model
Order
Performed
Program Maturity Elements—
2. Asset Classification and Control
1
2.1 Valuation is performed to identify and understandinformation assets to protect.
2
2.2 Risk assessment is performed to identify and quantify threats to information assets.
3
2.3 Information assets have defined system custodians and business owners.
4
2.4 Information assets classification labeling and handling procedures are developed.
5
2.5 An asset management inventory program is installed to handle assets on an ongoing basis.
It also avoids the pitfall of implementing security measures in the wrong order, introducing security risks precisely because safeguards are unsystematically implemented. For example, in figure 5 the organization could actually be harmed if an active inventory management system (element 2.5) is implemented before valuing the assets and performing a risk assessment (elements 2.1 and 2.2). If it is completed in the wrong order, it could take years of wasteful redesign work to the inventory system to properly categorize and ultimately protect the assets.
Since this model is essentially a detailed compliance tool, management can possibly misinterpret program maturity. A high level of maturity may give the false impression of project management closure. It might indicate to management that the organization is now “safe” and there is no more need to support the security effort, while it may instead be an indication of a safe posture only if the elements have been executed at a high-quality level. But, on the other hand, just because an organization has completed a particular security element, it does not necessarily mean it is doing a good job of it. This is why a separate measurement tool must be employed to measure quality or effectiveness of the actual implementation.
Security Posture
An improvement to the maturity model is the security posture, which essentially modifies the maturity model based on the quality of the implementation of each element. An advantage of adding a quality measure is that, unlike the maturity score, security posture is not a static achievement level. It is dynamic and can change based on the quality of the continued execution of the program elements. It requires active management of the security program to maintain a certain security posture.
Quality is a subjective measure. But, since it is separate from maturity, the two values cannot be confused, as in other models. A three-tiered factor (high, medium and low), as shown in figure 6, is suggested. With detailed threshold descriptions, it is possible to be objective and produce results. This scheme is similar to the US National Security Agency (NSA) Infosec Assessment Methodology (IAM) criticality model, where items also have three tiers of quality.7 But, unlike the NSA IAM model, which is only focused on data and systems, a richer picture of the entire organization’s security program is provided.
Figure 6—Sample of a Quality Measure From Security Maturity Model
Program Maturity Element
If maturity element is implemented, then…
Low-quality Threshold
Medium-quality Threshold
High-quality Threshold
2.4 Information Assets classification labeling and handling procedures developed
Procedures developed but not implemented
Assets partially classified
Pervasive classification throughout entire organization
An ideal security quality metric could be used as a dashboard display for management. It could give a near real-time view of the organization’s security posture (see figures 7 and 8). It should be measured on a weekly basis; ownership of individual program maturity elements should be assigned to specific departments. Thus, in sorting the elements by departments, management can gain a security understanding specifically customized by the way the organization is structured.
Figure 9 provides a simulated example of one such fictional organization. At a glance, it can be seen how each department rates in program maturity and quality. For example, department 2 is more mature, yet its quality is lower than the other two departments, and while departments 1 and 3 are at nearly the same level of maturity, the quality of department 1’s implementation is higher.
These appraisals need active management on an ongoing basis. By using security metrics in this manner, the organization incorporates security deeply into its structure. Security metrics then become a meaningful gauge of organizational performance, because they were designed to meet the initial objectives for metrics. An organization can easily demonstrate security posture improvements over time. Moreover, as security elements become adopted in a more systematic way, management can begin to understand the costs and benefits of an organized, mature and high-quality security program.
They are sorted by ISO 17799 categories with the numbers in parentheses representing the actual number of program elements used for the maturity score.
All the quality measures for existing program elements are aggregated at two different times.
Endnotes
1 ISO/IEC 17799:2000(E), Information Technology—Code of Practice for Information Security Management, December, 2000 2 National Institute of Standards and Technology (NIST) Computer Security Expert Assist Team (CSEAT) IT Model, csrc.nist.gov/cseat/CSEAT_IT_Security_ _Levels.htm 3 Dunbar, Thomas M.; “Information Metrics @ Citigroup,” 14 June 2000, Computer System Security and Privacy Advisory Board (CSSPAB) workshop “Approaches to Measuring Security,” http://csrc.nist.gov/ispab/june13-15/Citigroup.pdf 4 IT Governance Institute (ITGI), Control Objectives for Information and related Technology (COBIT), USA, 2000, www.itgi.org. Also see www.auckland.ac.nz/security/ InfomationSecurityMaturityAssessment.htm, draft version 0.1, 2003. 5 Systems Security Engineering Capability Maturity Model, (SSE-CMM), Carnegie Melon University, 1999, www.sse-cmm.org 6 “CERT Security Capability Assessment Tool,” CSO Online, CXO Media Inc. and Carnegie Melon University, 2003, www.csoonline.com/surveys/securitycapability.html 7 G. Miles, et al., Security Assessment: Case Studies for Implementing the NSA IAM, Sygress Publishing Inc., 2004
David Chapin, CISA, CISM, CISSP, IAM is a private security consultant. His current interests include security infrastructure design, security management, information assessment and organizational security. He holds three patents in business processes. He can be contacted at dchapin@earthlink.net.
Steven Akridge, JD, CISM, CM, CISSP, IAM is a private security consultant. His current interests include security management, organizational security, assessment methodology, forensics, cryptography and security intelligence. He has served on numerous security industry committees, including the Federal Public Key Infrastructure Steering Committee, the Partnership for Critical Infrastructure Security and the National Association of State CIOs. He may be contacted at steveakridge@cissp.com.
Information Systems Control Journal, formerly the IS Audit & Control Journal, is published by ISACA®, Inc.. Membership in the association, a voluntary organization of persons interested in information systems (IS) auditing, control and security, entitles one to receive an annual subscription to the Information Systems Control Journal.
Opinions expressed in the Information Systems Control Journal represent the views of the authors and advertisers. They may differ from policies and official statements of ISACA® and/or the IT Governance Institute® and their committees, and from opinions endorsed by authors' employers, or the editors of this Journal. Information Systems Control Journal does not attest to the originality of authors' content.
Instructors are permitted to photocopy isolated articles for noncommercial classroom use without fee. For other copying, reprint or republication, permission must be obtained in writing from the association. Where necessary, permission is granted by the copyright owners for those registered with the Copyright Clearance Center (CCC), 27 Congress St., Salem, Mass. 01970, to photocopy articles owned by ISACA® Inc., for a flat fee of US $2.50 per article plus 25¢ per page. Send payment to the CCC stating the ISSN (1526-7407), date, volume, and first and last page number of each article. Copying for other than personal use or internal reference, or of articles or columns not owned by the association without express permission of the association or the copyright owner is expressly prohibited.
