Several years ago all risks not complying with the definition of market or credit risks were included among operational risks. This vague and, strictly speaking, negative definition was unable to completely, correctly and efficiently manage the operational risks. The basis of a risk measurement is understanding the risk’s nature, potential sources and possible consequences, and this is something that cannot be done without a clear and precise definition.
Basel II (International Convergence of Capital Measurement and Capital Standards) defines operational risk as “a risk of loss resulting from inadequate or failed internal processes, people and systems or from external events.” This risk management standard is intended for banks and other financial institutions, where the term “operational risk” is well known and already settled.
In other (nonbanking) organizations, operational risk is often linked to information systems and, thus, operational risk is usually understood as a risk primarily related to system operation. Therefore, its measurement is related closely to assessing risks of the information systems and to the quality and level of information systems security.
To understand the issue of operational risk measurement, the above definitions and terms are of great help; however, as is the case with other similar definitions, they leave too much space for discussion on the topic of what risk measurement really is about. These discussions should be aimed toward the fundamental goals and objectives of measurement, rather than on the precise definition. “What should measurement bring to the organization?” and “How can measurement contribute to the increase of security?” are the essential questions to be asked. Answers to these questions will lead eventually to the definition of risk measurement relevant to the specific environment of the firm or organization. To understand the essence of risks, measurement should quantify relevant data regardless of the definition of risk measurement used, or whether these risks are called “operational” or something else. Therefore, operational risk measurement can be divided into two fundamental processes:
- Measuring operational risk value
- Measuring the level of operational risk treatment
The objective of both measurements is to quantify a predetermined value, which represents the risk level, the extent of risk reduction or the number/amount (or percentage) of accepted risk. In an ideal risk management environment, the benefits of operational risk measurement should also be included in the process. The possibility to quantify the effectiveness of implemented security controls and/or the increase of security level should be in the portfolio of all risk managers; however, measuring the benefits of operational risk treatment is not an easy task. Furthermore, current tools and methods used are restricted to simple Excel tables.
According to the Slovakian Internal Control Act No. 12/2004, every Slovakian bank shall perform periodic reviews of operational risks related to information technologies. It is similar to the Czech National Bank Internal Control Act No. 2 of 3 February 2004, on the internal control system of a bank, as amended by subsequent provisions, and partly of the Regulation No. 123/2007 of the Act No. 256/2004, which is basically an implementation of Basel II.
While risk analysis comprised an important part of the project, the main objective was to establish and implement a process of operational risk measurement in the Slovak bank. The risk management process had to be in compliance with Act no. 12/2004, mentioned previously. The executed risk analysis was the first step that helped to estimate the current risk levels. The process also included adaptation of a proven methodology and establishment of repeatable steps to manage operational risks. Implementation of comprehensible and transparent metrics to measure operational risk values and the level of their treatment was a self-evident part of the job. There were several important parameters these metrics had to accomplish, based mainly on experiences collected by experts in the US.1 Each metric used had to be:
- Measured in a consistent way, without any subjective criteria, so that even if more than one person uses the metrics identical results will be obtained
- Easily fulfilled, optimally in an automatic way
- Expressed in cardinal numbers or percentages, and not in the qualitative scale of low, medium and high
- Expressed in the same unit, e.g., hours, incidents, financial losses
Global metrics to measure the level of risks and the level of risk treatment were prepared in the first phase. Subsequently, other, more detailed metrics that focused on specific operational risks and vulnerabilities (e.g., password strength in individual systems, security incidents) and compliance with the ISO/IEC 27001 standard were implemented. All metrics were developed in a way that enables easy and fast collection of data and are as automatic as possible.
The secondary objective for 2008 is to prepare an interface for the Risk Management Association (RMA) and RiskBusiness KRI Framework Study methodology, used by the bank as a primary approach to risk management to reach compliance with Basel II requirements.
Measuring Operational Risk Value
Value, or measure of risk, as it is also called, was the first result from the performed risk analysis using qualitative methodology. Possible impacts of security violations on the bank’s activities and on reaching their business goals were estimated in the first part of the analysis. The probability that security violations would occur was estimated by assessing levels of threats and information and communication technology (ICT) vulnerabilities.
The metric used to express measure of risk was on a scale from one to seven; altogether, 27,893 values of risk were calculated. This number, however immense it may seem, includes all possible combinations of identified impacts, probabilities and bank assets. From the operational-risk perspective, only 21 risks were assessed; these risks were presented to management at the end of the project.
Figure 1 shows the risk value and its characteristics, i.e., the ratio of an individual risk’s component (the first being impact and the following two being probability components, of which the first is the level of threat and the second is the vulnerability, each expressed in different colors) and how they contribute to overall risk.
Measuring the Level of Operational Risk Treatment
Upon finishing the analysis part, all risks were evaluated against preestablished criteria, and it was then decided whether they were to be accepted, treated or avoided.
For those risks that were not acceptable or could not be avoided, countermeasures that treat (reducing their consequence or the likelihood of the risk) specific operational risks were proposed. Existing countermeasures were considered while proposing new sets of controls. Assessing the current status of controls was an important factor later used to make decisions on the priorities of the risks and the sequence in which they would be treated. The methodology used contains an extensive list of countermeasures offering numerous recommendations on how to deal with individual risks (treatment options).
After evaluation, the recommended countermeasures were divided into two sets, and based on these groups, metrics were developed. Countermeasures recommended to reduce the identified risks, and that were found to be already in place, formed the first, green group. Countermeasures not in place (needing to be implemented) formed the second, red group. By comparing the total number of countermeasures to the numbers of existing “green” and to-be-implemented “red” countermeasures, the level of risk treatment was identified.
The relationship between risk and countermeasures was m:n, meaning that the countermeasure recommended to cover specific risks applied to other risks as well. The foundation of the metric was a percentage of individual risks and, thus, every countermeasure was unique. Therefore, although the countermeasure was calculated in the metric several times, the metric itself was not adversely affected.
The results of the recommended security controls evaluation is shown in figure 2 (again in the form of a management summary).
The big picture about the state of security in the bank is stressed by placing the graphs next to each other. When presenting the results of the analysis, both metrics were presented simultaneously to give a uniform view, on both the operational risk and the level of its treatment.
The very first risk presented reached the measure of five (on the scale from one to seven) and is covered at 93 percent. Thus, this approach, in a transparent way, fulfills two basic processes of measurement: the value of the operational risk and the level of its treatment. With respect to its transparency, the results were presented in this simple way; no miscommunication occurred and the results were easily and clearly presented to management.
Measuring the Level of Operational Risk Treatment—Detailed Metrics
The security controls were not only assessed as described (i.e., binary green-red); the risk analysis methodology enabled the reviewer to select from more statuses than just those that the recommended countermeasures were in at the time.
Some of the countermeasures that were recommended initially to cover specific operational risks were later rated as at an “acceptable” level of risk. It was decided that, however contributive the countermeasure could be, it would not be implemented for various reasons. In such cases, the risk remained partially untreated and it was necessary to determine whether the remaining risk was higher than the preset acceptable level of 8 percent (in reality, the level of 4.11 percent was acceptable).
Selected controls were also marked as “not applicable” in cases where the proposed actions to raise the security level were not viable technologically or were against the core business goals of the bank. Controls that were not in place, but were in progress, were marked as “implementing recommendation.”
Using other states of security controls, more detailed metrics were prepared to provide new views on the information security within the bank. The process of operational risk treatment was established correctly; however, some risk existed that still needed to be treated (red column). Figure 3 shows the expected progress in the implementation of new controls, which were implemented during the execution of the analysis (altogether, the analysis lasted five months).
Figure 4 depicts individual security areas and the ratio of implemented, not applicable, recommended and other security controls.
Each of the metrics described in figure 4 is based on thousands of values collected (due to the level of their analytical detail, these values could not be presented). The objective of these simplified figures was to show the results of the analysis and the level of operational-risk treatment in an easy and comprehensive way. Each figure selected at the beginning of the project was constructed based on the four parameters of metrics, so they can be updated with minimal effort in the future. The procedures for the continuous data collection were developed to enable the risk treatment level to be assessed in time.
Figure 5 displays how the number of implemented controls has increased over time, i.e., the risk treatment process’s improvement. Other states of countermeasures decreased and the overall number of controls also changed. The process also identified new risks as well as changes in the information system. As part of the change management process, security impacts were assessed, and an optimal set of countermeasures to cover newly identified operational risks was recommended.
Measuring Contribution of Operational Risk Treatment
The final implementation of recommended countermeasures is never an easy task. The best approach has proven to be development of a set of implementation projects to be realized by the bank’s staff. For example, it is more difficult to increase the level of security of the security access system (SAS) authentication system than to execute a project for the implementation of smart cards in the SAS, although, in reality, these are identical projects. The problem is usually in the allocation and approval of financial resources for the project. To find a sponsor for the implementation of security countermeasures is, in financially oriented institutions such as banks, almost an impossible task. Using the other description, finding a project sponsor is still intricate and tedious, but not impossible.
Individual controls or groups of recommended countermeasures were formed into implementation projects; each of these was focused on covering specific security areas. In the specific situation of this bank, 13 implementation projects were created, five of which were:
- Log management—Acquisition and implementation of a solution for continuous assessment of specific applications and system logs
- Two-factor authentication for the virtual private network (VPN)—Acquisition and implementation of two-factor authentication objects for VPN connection
- Intrusion protection system (IPS) and intrusion detection system (IDS)—Increasing the effectiveness of IPS and IDS for the automatic detection of security intrusion and raising alarms
- Disaster recovery plans—Preparation of disaster recovery and business continuity plans for all critical systems within the bank
- Handling the media—Establishment and implementation of procedures for manipulation of media (e.g., tapes, CDs, DVDs, flash memory cards)
The goal was to implement a specific set of countermeasures, which in reality meant changing the status of the safeguards from red to green. Measuring the contribution of countermeasures’ implementation was an easy task, as the described metrics work with the countermeasures’ states. For the purpose of measuring implementation projects’ contribution, the presentation of metrics to the bank’s management was based on security areas (e.g., IT, communication, personnel, physical, administrative).
The executed project resulted in changes to the status of countermeasures in specific security areas and, in this way, was a contributing factor in those specific areas. The security level of the areas was increased, resulting from the fact that relevant risks were treated sufficiently. The increase of the risk treatment level was measured by the number of implemented countermeasures.
The first graph in figure 6 shows the status in the physical security area before the implementation of security controls, as these were implemented during one project. The second graph displays the state after the project was finished. Countermeasures recommended for implementation (red) have changed to “implemented” (green).
Based on the number of countermeasures and their current states, it was possible to clearly predict the future levels of risk treatment after the implementation projects were completed. As the current state was known and the future state of countermeasures was predictable, it was possible to determine the current and future levels of risk treatment, as well as the future level of security within the bank or in specific security areas within the bank.
The cornerstone of operational risks’ measurement is to implement a consistent methodology or to use a sufficiently transparent interface among different methodologies and tools. It is necessary to develop metrics that will be comprehended easily and will enable fast and easy determination of risk levels and the levels of their treatment over time, thus enabling tracking of risk trends or performance of periodic benchmarking.
Operational-risk measurements, as well as other measurements of security, are a newer area that is becoming more and more pronounced. Now, many IT-driven companies have implemented information security management systems and learned how to measure the level of risk.
Tracking the contribution of security changes in risk treatment levels or security progress at this time is an ability that, so far, only a few security managers hold. One of the main reasons for this is ignorance of the measurement processes and little knowledge about security metrics, their preparation, usage and effective presentation. The highly anticipated international standard from the International Organization for Standardization and International Electrotechnical Commission, ISO/IEC 27004 Information Security Management Measurements, should shed more light on the area of security metrics and the overall process of measurement. Several other standards and publications are also available.2
1 Jaquith, Andrew; Security Metrics: Replacing Fear, Uncertainty, and Doubt, Addison-Wesley, 2007
2 Australia/New Zealand standard, AZ/NZS 4360:2004 Risk Management; ISO/IEC 27001 and 27002 Implementation Guidance and Metrics; BSI Group’s BIP 0074 Measuring the Effectiveness of Your ISMS Implementation; National Institute of Standards and Technology (NIST) Special Publication (SP) 800-55 Security Metrics Guide for Information Technology Systems; NIST SP 800-80 Guide for Developing Performance Metrics for Information Security
Jan Mikulecky, CISM
has been working as a senior consultant at Risk Analysis Consultants since 1999. His principal specialization is risk analysis of information systems and implementation of information security management systems and business continuity management in large organizations. He also gives lectures in the Czech Republic and other European countries on methodologies and standards in the area of information security. He can be reached at email@example.com.
Information Systems Control Journal, formerly the IS Audit & Control Journal, is published by the ISACA. Membership in the association, a voluntary organization of persons interested in information systems (IS) auditing, control and security, entitles one to receive an annual subscription to the Information Systems Control Journal.
Opinions expressed in the Information Systems Control Journal represent the views of the authors and advertisers. They may differ from policies and official statements of the Information Systems Audit and Control Association and/or the IT Governance Institute® and their committees, and from opinions endorsed by authors' employers, or the editors of this Journal. Information Systems Control Journal does not attest to the originality of authors' content.
Instructors are permitted to photocopy isolated articles for noncommercial classroom use without fee. For other copying, reprint or republication, permission must be obtained in writing from the association. Where necessary, permission is granted by the copyright owners for those registered with the Copyright Clearance Center (CCC), 27 Congress St., Salem, Mass. 01970, to photocopy articles owned by the Information Systems Audit and Control Association Inc., for a flat fee of US $2.50 per article plus 25¢ per page. Send payment to the CCC stating the ISSN (1526-7407), date, volume, and first and last page number of each article. Copying for other than personal use or internal reference, or of articles or columns not owned by the association without express permission of the association or the copyright owner is expressly prohibited.