Tips for Developing or Improving Metrics
Everywhere we turn, vast amounts of facts, figures, numbers, records and files are being processed, interpreted, organized, structured and presented in a way that turns those data bits and bytes into meaningful information. Putting the raw data into context is what makes information useful for business decisions and underlies many dashboards being developed across the enterprise. Data and information are important components for measurement and, if put into a suitable context, may also become meaningful metrics.
Let us begin with a few definitions and examples:
- Data—Raw, unorganized facts, records, numbers, etc. An example is the number 2 or the letters “e, g, s.” By themselves, it is hard to know what exactly is meant by their use.
- Information—Data that are structured, organized or presented in context to make them useful. An example is “I had 2 eggs for breakfast.”
- Measure (or measurement)—Is the value of a specific characteristic of data. An example is “the number of staff that completed information security awareness training.” Without more context, it is hard to know what value is derived from the statement.
- Metric—The aggregation of one or more measures to create a piece of business intelligence, in context. An example is “percentage of staff trained vs. expected (planned vs. actual numbers)” or “percentage of new users (internal and external) who have satisfactorily completed information security awareness training before being granted network access.” These statements give context for whether or not the information provided is meeting the intended objective. If I have 10 staff members and 9 of them have completed the relevant training, then my percentage of satisfactory completion is 90%. If I have 10,000 staff members and only 900 of them have completed the relevant training, then I know I still have more work to do, especially if the untrained staff have been granted access to the network.
Consistent, timely and accurate metrics are an important feedback mechanism for managing any activity. When seeking to develop or improve metrics, here are some considerations to keep in mind:
- Establish objectives—What questions are intended to be answered with the metric? Who is the audience for the metric? Which information needs will be satisfied with the metric? Who collects the measurement data? What techniques for analysis and reporting will be used?
- Prioritize objectives—Data collection and analysis are costly and time consuming. It is important to consider the purpose and intended use of the metrics. What actions or decisions would the metric inform? If no action, decision or behavior change occurs as a result of the metric, then why are you spending resources to collect and analyze the data?
- Identify candidate metrics—Candidate metrics should be based on documented measurement objectives. Identify existing metrics that may already address the objective. Metrics may already exist to satisfy 1 purpose and may also be used for additional purposes or to answer additional questions.
- Specify data collection and storage procedures—Procedures should be based on the objective to be satisfied and the capability of the organization for collecting, storing, managing and disposing of data. Remember, data by themselves may not be sensitive or personally identifiable, but when aggregated, there may need to be explicit procedures for protecting and sustaining the information and subsequently developed metrics. Being explicit about data collection and storage may also help with overall data management, maintaining data integrity and governance. Other considerations in this category are frequency of collection and where the source data are created, stored, used, transported, etc. Data flow diagrams are useful for better understanding the data’s unique characteristics and attributes.
- Update objectives as needed—Do not be afraid to retire a metric if it is not driving decisions, behavior or actions. The most important consideration here is to ask yourself, “What is the value of this metric in comparison to another metric?” If the metric is not meeting the intended objective, then it is no longer useful to collect and maintain. You may need to iterate several times before getting to a small set of meaningful metrics that drive better decisions, actions and behaviors. Often, the best metrics are conveyed by reporting trends over time versus a single point-in-time metric.
Make sure your questions are the ones most important to your target audience (management, operations, strategic) and your assumptions are stated. If there are estimates used in the metric calculations (because you do not have a piece of data or have just started collecting and have no trends in the data), make sure to state that somewhere in your visualization. Good metrics are those that are used often, answer important business questions, cost little to collect in relation to their value, are easily collected and do not require extensive manual intervention or manipulation. There is a difference between metrics and metrics that matter.
Lisa Young, CISA, CISM, is the past president of the ISACA West Florida (Tampa, Florida, USA) Chapter and a frequent speaker at information security conferences worldwide.
Learn to Implement Compliance by Design for Your Enterprise in This Webinar
Source: Paul Taylor
Next-generation governance focuses on compliance and speed. With businesses needing to move faster, traditional audit methodologies must be updated. Auditors need to find ways to verify the compliance of a complex, dynamic enterprise deployment—ways just short of automating the audit process. Compliance by design may just be one way to accomplish this.
To help increase your understanding of compliance by design, ISACA presents the “Compliance by Design” webinar. This webinar takes place on 25 January at 11AM CST (UTC -6 hours). ISACA members can earn 1 continuing professional education (CPE) hour by attending this webinar and completing a related survey.
Pamela Nigro, CISA, CRISC, CGEIT, CRMA, will lead the webinar as a subject matter expert in IT controls, and as senior director of information security at Heath Care Service Corporation (HCSC). She is responsible for IT risk and compliance testing for the 5 Blue Cross Blue Shield Plans comprising HCSC and will use her experience with healthcare compliance to help aid your implementation of compliance by design for your enterprise.
To learn more about this webinar or to register for it, visit the Compliance by Design page of the ISACA website.
Ensure Cybersecurity for All With This Audit Program
According to the Center for Internet Security (CIS), cybersecurity can be improved by changing organizational thoughts from “what should our enterprise do” to “what we should all be doing.” Every organization and industry are subject to cyberrisk, and every organization and industry can benefit from a more strategic cyberdefense policy and procedure. CIS describes many critical cyberdefense controls in its publication Critical Security Controls for Effective Cyber Defense, which have been used to craft this audit program. The publication includes 149 individual control activities that assess design and operating effectiveness. Implementing these controls may prevent sensitive data and intellectual property leakage, protect networks from malware, maintain device integrity, and uphold enterprise reputation.
The ISACA and CIS CIS Controls Audit/Assurance Program provides a thorough, comprehensive and outcome-based assessment of an enterprise’s cybersecurity environment and can help yield meaningful and tangible recommendations to strengthen organizational cyberdefense. This audit program also:
- Provides management with an assessment of critical CIS cybersecurity controls and helps to evaluate their operational effectiveness
- Identifies enterprise internal control and regulatory deficiencies
- Identifies security control concerns that could affect the reliability, accuracy and security of enterprise data
Conducting a formal assessment of an organization’s cybersecurity controls allows an enterprise to avoid reputational risk, financial risk and data loss. The ISACA and CIS CIS Controls Audit/Assurance Program provides IT auditors with a tool to evaluate all their systems and recommend proactive security measures. This audit program can be downloaded by visiting the CIS Controls Audit/Assurance Program page of the ISACA website.
Explore GDPR in New ISACA Podcast
The ISACA Podcast provides listeners a way to obtain industry knowledge on the go. The podcast features interviews with ISACA Journal columnists and authors. These industry leaders cover a range of topics in the podcasts, including distributed denial-of-service attacks, the cybersecurity skills gap and COBIT 5. Because the amount of technical content in these podcasts varies, the podcasts are relevant to professionals at any career stage. Most podcasts correlate with ISACA Journal articles or ISACA white papers.
ISACA’s most recent podcast, “Governance, Risk, Compliance and a Big Data Case Study,” discusses a big data project at a major bank and how the project would have differed had it taken place with current governance, risk and compliance (GRC) imperatives, including the European Union General Data Protection Regulation (GDPR). This podcast is an interview with ISACA Journal author Guy Pearce and contains his insights on big data and GRC.
To ensure you never miss a podcast, subscribe to the ISACA Podcast on iTunes, Google Play or SoundCloud for automatic notifications of new podcasts.
Tech Brief: Smart Contract Technology Evolution
The popularity of Bitcoin has put the spotlight on blockchain technology. However, blockchain helps innovate much more than just cryptocurrency. For instance, smart contracts use blockchain to simplify execution, lower costs, save parties time and enforce contractual arrangements. To help increase the understanding of how smart contracts differ from traditional contracts, ISACA has released the ISACA Tech Brief: Understanding Smart Contracts.
This complimentary tech brief details to the lay person how smart contract technology is evolving in the marketplace. Decentralized ledger technology and smart contract software allow individuals and enterprises to automate contractual functions by employing rules for if/then actions—if this action occurs, then that action results.
To learn more and download this tech brief, visit the ISACA Tech Brief: Understanding Smart Contracts page of the ISACA website.