• Bookmark

Benchmarking of COBIT 5 PAM Assessments Performed in Brazilian Public Sector Banking Organizations

By Joao Souza Neto, Ph.D., CGEIT, CRISC, PMP, Geraldo Loureiro, CRISC and Diana Santos, PMP

COBIT Focus | 24 August 2015

This article presents the process capability assessments of the governance domain of COBIT 5 for 3 Brazilian public sector banking organizations. The goal was to put the new assessment ruler (Process Assessment Model [PAM], based on the ISO/IEC 15504 approach) into practice and verify how the organizations would perform while employing criteria that are different from the ones used in the COBIT 4.1 Capability Maturity Model (CMM) approach.


Methods

In the Brazilian public sector, the Court of Audit implements a biannual survey on IT governance and management. In order to have an impression of the governance capabilities of some organizations in the survey, this research began by using the new assessment “ruler” on the processes of the COBIT 5 Evaluate, Direct and Monitor (EDM) domain. The capability assessments for level 1 of the governance domain for all 3 organizations were assessed. PAM defines 3 assessment classes, distinct in their purpose, assessor requirements and evidential requirements. A class 3 assessment was chosen because it generates capability assessment results that may indicate critical opportunities for improvement or identify key issues that would support a later class 1 or 2 assessment for the processes in question. This class of assessment considers only 1 instance of each process, evidences and execution by an independent certified assessor.1 Prior to the assessment, each organization received a nondisclosure agreement signed by the assessors and also answered a prequestionnaire consisting of:

  • The unit to be assessed
  • The sponsor of the assessment
  • The local assessment coordinator
  • Managers who will participate in the assessment
  • Assessors
  • Scope of the assessment (capability level 1 of the EDM processes)
  • Target rating levels
  • Current governance problems related to the EDM processe
  • Class of the PAM assessment (class 3)
  • COBIT 4.1 and COBIT 5 knowledge of the team
  • Awareness of what is involved in a PAM assessment

Some information provided by the organizations in the prequestionnaires is presented in figure 1.


Figure 1—Information Included in the Prequestionnaire

Organizations

The Unit to be Assessed

The Role of the Sponsor

The Roles of the Managers

Current Governance Problems

A IT governance planning, building and support (GPBS) IT director Executive manager of GPBS, executive manager of IT infrastructure, IT manager, IT director advisor Communication with the stakeholders is irregular and occurs mainly on demand. IT investments management does not address the whole life cycle, jeopardizing cost management efforts.
B IT governance department IT governance general manager IT executive manager, information systems manager, IT consultant, senior IT executive assistant IT risk is not adequately appropriated by the operational risk manager, which increases the organization´s risk exposure. Asset management has a limited scope, hindering a global total cost of ownership (TCO) analysis.
C IT department Head of the IT department Deputy head of the IT department, head of IT infrastructure division, deputy head of IT infrastructure division, IT coordinator, IT analyst IT value delivery is not managed properly, so it is always difficult to clearly show all the benefits that IT brings to the business. The organization’s risk appetite is not taken into account in the design of IT-enabled business solutions.

Source: J. Souza Neto, G. Loureiro and D. Santos. Reprinted with permission.


Regarding the property of the rating records, it was agreed that the assessment records would be owned by the assessors and the organizations and, in case of publication of the results of the assessment, neither the name of the organization nor information that may enable its identification would be disclosed.


The first part of the assessment was a meeting with the stakeholders where the sponsor explained the objective and the importance of the assessment. The assessors and the local assessment coordinator were presented, COBIT Process Assessment Model (PAM): Using COBIT 5 was explained, the EDM processes goals, practices, activities and outcomes were detailed, and the prequestionnaire was discussed. In this meeting, the stakeholders were asked to choose the target level for their actual processes that mapped to the EDM processes. Regarding target levels, 2 organizations wanted to target governance processes as fully achieving PA1.1 (fully performing processes), whereas the third organization favored targeting processes that performed largely as described in COBIT 5.


These levels are defined in PAM according to a standard rating scale defined in the ISO/IEC 15504 standard. These ratings consist of:

  • N—Not achieved. There is little or no evidence of achievement of the defined attribute in the assessed process.
  • P—Partially achieved. There is some evidence of an approach to, and some achievement of, the defined attribute in the assessed process. Some aspects of achievement of the attribute may be unpredictable.
  • L—Largely achieved. There is evidence of a systematic approach to, and significant achievement of, the defined attribute in the assessed process. Some weaknesses related to this attribute may exist in the assessed process.
  • F—Fully achieved. There is evidence of a complete and systematic approach to, and full achievement of, the defined attribute in the assessed process. No significant weaknesses related to this attribute exist in the assessed process.

Before starting the assessment, the rules for attribute classification were presented. For the PA1.1 assessment of EDM processes, there are exactly 3 outcomes to be verified in each process. Here is an extract of the rules proposed:

  • If the outcomes are F, F and L, the process is considered L.
  • If the outcomes are F, F and P, the process is considered P.
  • If the outcomes are F, F and N, the process is considered P.
  • If the outcomes are F, L and L, the process is considered L.
  • If the outcomes are F, L and P, the process is considered P.
  • If the outcomes are F, L and N, the process is considered P.
  • If the outcomes are F, L and L, the process is considered L.

In the second part, the capacity of each EDM process was evaluated at level 1 (PA1.1). The assessment consisted of discussing the purpose and the outcomes of each process, and the statements made had to be confirmed with evidence. In all organizations, evidence was presented electronically, on a large-screen monitor that displayed the contents of the intranet of the organizations.


Four days after the assessment, all organizations received a detailed report with a list of the evidences provided for each outcome, the results obtained and suggestions on how to improve the capacity of each process with a gap. Figure 2 presents some suggestions for organization B.


Figure 2—Suggestions for Gap Reduction

Outcome

Evidence

Rating

Gap

Actions to Reduce the Gap

EDM02.01: The enterprise is securing optimal value from its portfolio of approved IT-enabled initiatives, services and assets.
None
N
F-N
Make evident how IT adds value to the organization and how the value added is optimized.
EDM02.02: Optimal value is derived from IT investment through effective value management practices in the enterprise.
None
N
F-N
Implement IT value management practices in the organization.

Source: J. Souza Neto, G. Loureiro and D. Santos. Reprinted with permission.


One valuable lesson learned is that it is clearer for organizations to start by assessing the outcomes, not the purpose of the process. By doing so, the assessment of the purpose becomes straightforward.


Results

Though the sponsor and participants were briefed before the assessments about the differences in the COBIT 5 and COBIT 4.1 approaches, the results were disappointing for the organizations because they are used to very high maturity levels in the COBIT 4.1 model. Figures 3, 4 and 5 show the combined attribute classification, the corresponding capacity level, and the gap to reach classification F or L for each organization. The gap was classified as “absent” when the attribute had already reached the target level; “small” when the attribute was P for a target level L, or L for a target level F; and “significant” when the attribute was N for a target level F or L, or P for a target level F.


Figure 3—Assessment for COBIT 5 Governance Processes of Organization A

Governance Domain

Attribute Classification

Capability Level

Gap for Classification F

EDM01—Ensure governance framework setting and maintenance 
F
1
Absent
EDM02—Ensure benefits delivery
F
1
Absent
EDM03—Ensure risk optimization
F
1
Absent
EDM04—Ensure resource optimization
P
0
Significant
EDM05—Ensure stakeholder transparency
P
0
Significant

Source: J. Souza Neto, G. Loureiro and D. Santos. Reprinted with permission.


Figure 4—Assessment for COBIT 5 Governance Processes of Organization B
 

Governance Domain

Attribute Classification

Capability Level

Gap for Classification F

EDM01—Ensure governance framework setting and maintenance
F
1
Absent
EDM02—Ensure benefits delivery
N
0
Significant
EDM03—Ensure risk optimization
N
0
Significant
EDM04—Ensure resource optimization
N
0
Significant
EDM05—Ensure stakeholder transparency
P
0
Significant

Source: J. Souza Neto, G. Loureiro and D. Santos. Reprinted with permission.


Figure 5—Assessment for COBIT 5 Governance Processes of Organization C
 

Governance Domain

Attribute Classification

Capability Level

Gap for Classification L

EDM01—Ensure governance framework setting and maintenance
F
1
Absent
EDM02—Ensure benefits delivery
N
0
Significant
EDM03—Ensure risk optimization
P
0
Small
EDM04—Ensure resource optimization
L
1
Absent
EDM05—Ensure stakeholder transparency
P
0
Small

Source: J. Souza Neto, G. Loureiro and D. Santos. Reprinted with permission.


Discussion

The assessments highlighted that the capability level of process EDM05 Ensure stakeholder transparency is 0 in all 3 organizations, as seen in figure 6. It was verified that if information was not demanded, or “pulled,” it would not be easily accessible.


Figure 6—Benchmarking of the Organizations for Level 1 of EDM Processes

 

Organization A

Organization B

Organization C

EDM01—Ensure governance framework setting and maintenance
1
1
1
EDM02—Ensure benefits delivery
1
0
0
EDM03—Ensure risk optimization
1
0
0
EDM04—Ensure resource optimization
0
0
1
EDM05—Ensure stakeholder transparency
0
0
0

Source: J. Souza Neto, G. Loureiro and D. Santos. Reprinted with permission.


In some organizations, there were doubts about what could be considered evidence of value management and resource optimization.


Conclusion

The main benefit perceived in the COBIT 5 process capability assessment was the detailed analysis of the execution of the process being evaluated, assessing whether the process achieves its goals and produces the required outcomes.


Another important benefit was the introduction of the COBIT 5 PAM for these organizations, which had never tried an ISO/IEC 15504 type of assessment for enterprise IT governance.


This research is an ongoing activity of the ISACA Brasilia (Brazil) Chapter.


Joao Souza Neto, Ph.D., CGEIT, CRISC, PMP

Has more than 8 years of experience in IT governance, applying COBIT within Brazil Post. He is also responsible for the IT governance research area in the Universidade Catolica de Brasilia (Brazil). He is founder and educational director of the ISACA Brasilia Chapter.


Geraldo Loureiro, CRISC

Has more than 8 years of experience in IT auditing, applying COBIT, and more than 8 years of experience as a chief information officer in public organizations in Brazil. He is founder and president of the ISACA Brasilia Chapter.


Diana Santos, PMP

Is chief IT advisor at a Brazilian federal government agency related to the judiciary. She has collaborated on IT governance and the implementation of COBIT processes. She is marketing and communication director of the ISACA Brasilia Chapter.


Endnotes

1 ISACA, Assessor Guide: Using COBIT 5, USA, 2013

THIS WEBSITE USES INFORMATION GATHERING TOOLS INCLUDING COOKIES, AND OTHER SIMILAR TECHNOLOGY.
BY USING THIS WEBSITE, YOU CONSENT TO USE OF THESE TOOLS. IF YOU DO NOT CONSENT, DO NOT USE THIS WEBSITE. USE OF THIS WEBSITE IS NOT REQUIRED BY ISACA. OUR PRIVACY POLICY IS LOCATED HERE.