Download (214K)

1.  BACKGROUND

1.1 Linkage to Standards

1.1.1 Standard S6 Evidence states, “During the course of the audit, the IS auditor should obtain sufficient, reliable and relevant evidence to achieve the audit objectives. The audit findings and conclusions are to be supported by appropriate analysis and interpretation of this evidence."
1.1.2 Standard S8 Follow-up states, “After the reporting of findings and recommendations, the IS auditor should request and evaluate relevant information to conclude whether appropriate action has been taken by management in a timely manner.”

1.2 Linkage to COBIT

1.2.1 High-level control objective M4, Provide for Independent Audit, states, “Control over the IT process of providing for independent audit that satisfies the business requirement to increase confidence levels and benefit from best practice advice is enabled by independent audits carried out at regular intervals and takes into consideration:

•  Audit independence
• Proactive audit involvement
• Performance of audits by qualified personnel
• Clearance of findings and recommendations
• Follow-up activities
•  Impact assessments of audit recommendations (costs, benefits and risks)

1.2.2 Detailed control objective M4.6, Performance of Audit Work, states, “Audits should be appropriately supervised to provide assurance that audit objectives are achieved and applicable professional auditing standards are met. Auditors should ensure that they obtain sufficient, reliable, relevant and useful evidence to achieve the audit objectives effectively. The audit findings and conclusions should be supported by appropriate analysis and interpretation of the evidence.”

1.3 Reference to COBIT

1.3.1 The COBIT references offer the specific objectives or processes of COBIT to consider when reviewing the area addressed by this guidance. Selection of the most relevant material in COBIT applicable to the scope of the particular audit is based on the choice of specific COBIT IT processes and consideration of COBIT’s information criteria.
1.3.2 In a post-implementation review, the first review after the implementation of an IT solution, the following processes are more relevant:

• PO2—Define the Information Architecture
• PO4—Define the IT organisation and relationship
• PO5—Manage the IT investment
• PO8—Ensure Compliance with External Requirements
• PO9—Assess risks
• PO10—Manage projects
• PO11—Manage quality
• AI1—Identify automated solutions
• AI2—Acquire and maintain application software
• AI3—Acquire and maintain technology infrastructure
• AI5—Install and accredit systems
• AI6—Manage changes
• DS7—Educate and Train Users
• DS11—Manage Data 
• M1—Monitor the processes
• M2—Assess Internal Control Adequacy
• M3—Obtain Independent Assurance
• M4—Provide for Independent Audit

1.3.3 The information criteria most relevant to the post-implementation review are:

• Primary—Effectiveness and efficiency 
• Secondary—Availability, compliance, confidentiality, reliability and integrity

1.3.4 International Federation of Accountants (IFAC) Information Technology Committee (ITC) Guidelines include:

• Implementation of Information Technology Solutions
• Managing Information Technology Planning For Business Impact

1.4 Purpose of the Guideline

1.4.1 The purpose of this guideline is to describe the recommended practices in carrying out the post-implementation review of information technology solutions, so that the relevant standards for information systems auditing are complied with during the course of the review.
1.4.2 Organisations implement various IT solutions to meet their business requirements. Once the solutions are implemented, post-implementation reviews are generally carried out by IS auditors to assess the effectiveness and efficiency of the IT solutions and their implementation, initiate actions to improve the solution (where necessary) and serve as a learning tool for the future.
1.4.3 Certain practices recommended in this guideline may also be appropriate for reviews of projects where implementations are unsuccessful or aborted prior to implementation.
1.4.4 This guideline provides guidance in applying IS Auditing Standards S6 Performance of Audit Work and S8 Follow-up Activities while conducting a post-implementation review. The IS auditor should consider it in determining how to achieve implementation of these standards, use professional judgment in its application and be prepared to justify any departure.

1.5 Guideline Application

1.5.1 When applying this guideline, the IS auditor should consider its guidance in relation to other relevant ISACA guidelines

1.6 Definition and General Coverage

1.6.1 Post-implementation review, for the purpose of this guideline, means the first or subsequent review of an IT solution and/or the process of its implementation, performed after its implementation, to assess any or all of the following:

• Whether the intended objectives of the solution are realised 
• Whether actual costs and benefits are compared against budget
• The effectiveness and appropriateness of the implementation process
• Causes of time and/or cost overruns, and quality and/or performance issues, if any
• Productivity and performance improvements resulting from the solution
• Whether business process and internal controls are implemented
• Whether user access controls are implemented in accordance with organisational policy
• Whether users have been appropriately trained
• Whether the system is maintainable and can be further developed effectively and efficiently
• Whether available features and procedures, as relevant, have been implemented
• Compliance with relevant statutory requirements and organisational policies
• Compliance with COBIT Control Objectives and COBIT Management Guidelines, as relevant
• Opportunities for further improvement in either the solution or implementation process

1.6.2   The objectives of a post-implementation review might include:

•  Ensure that the intended objectives of implementing the IT solution are met and aligned to meet the business objectives of the organisation
• Evaluate the adequacy of procedures and controls over input, processing and output to ensure that information captured is complete and accurate, information processing complies with required business rules, and information generated is accurate, reliable and timely
• Evaluate the adequacy of procedures and controls over the maintenance and monitoring of the management trail produced by the IT solution
• Verify the accuracy of financial and management reports generated by the IT solution
• Ensure the adequacy of application-level access control enforced by the IT solution
• Verify the adequacy of availability features inherent in the IT solution to recover from any unexpected shutdowns and maintain data integrity
• Ensure that the IT solution can be supported and maintained efficiently and effectively in the absence of the specific personnel responsible for its development and implementation
• Identify potential risks and weakness in controls, as well as provide solutions to mitigate risks and strengthen controls

1.6.3  The post-implementation review essentially seeks to determine whether the investment in the IT solution was worthwhile (as determined and measurable by the organisation) and whether the delivered IT solution can be adequately managed and controlled. These investment returns can be covered as a unique, separate review often called a benefits realisation review (section 8.1). The scope of a post-implementation review should consider:

• The nature of the IT solution
• The intended usage of the IT solution (for what purpose, by whom, when and where)
• The criticality of the IT solution in achieving business objectives
• The scope of the review agreed with the auditee (organisation) management
• Whether the IT solution was subject to audit review during the initiation, development and testing stage
• Where there has been any non-audit involvement of IS auditors during the project implementation

2. AUDIT CHARTER

2.1 Mandate

2.1.1 Before commencing a post-implementation review, the IS auditor should have the requisite mandate to carry out the review. Where the review is initiated by a third party, the IS auditor should obtain reasonable assurance that the third party has the appropriate authority to commission the review.

3. INDEPENDENCE

3.1 Professional Objectivity

3.1.1 Before accepting the assignment, the IS auditor should provide reasonable assurance that his/her interest, if any, in the IT solution that is the subject of the post-implementation review will not impair the objectivity of the review. Any possible conflict of interest should be communicated explicitly to the organisation, and if possible, a written statement of the organisation’s awareness of the conflict should be obtained before accepting the assignment.
 3.1.2 Where the IS auditor had any non-audit roles in the implementation of the IT solution being reviewed, the IS auditor should consider guideline G17, Effect of Non-audit Roles on the IS Auditor’s Independence which provides guidance.

4. PROFESSIONAL ETHICS AND STANDARDS

4.1 Pre- and Post-implementation Reviews

4.1.1 As compared to a pre-implementation review, post-implementation review is ordinarily performed when the IT solution has been in operation for a reasonable period (ordinarily a number of months or process cycles) and user procedures, as well as application-level securities, have been implemented.
4.1.2 Pre-implementation reviews examine the conceptual design of controls and management trails, or how they operate in test environments. Post-implementation reviews examine how controls and management trails are operating once the IT solution is installed, configured and operating in the production environment. Where a pre-implementation review has been performed satisfactorily, the IS auditor should use his/her discretion whether to limit the post-implementation review to examine actual operation of the system in production.
4.1.3 It is preferable to perform both pre-implementation and post-implementation reviews, if resources are available,
4.1.4  When carrying out a post-implementation review the IS auditor should provide reasonable assurance that the project owner responsible for implementing the IT solution and the project team is involved in the review process. Team members consulted as part of the review should typically include:

• People connected with the design, development and deployment of the IT solution
• People with working knowledge of the area under review, and current and proposed business processes
• People with relevant technical knowledge
• People with knowledge of the organisation’s business strategy and the proposed contribution of the IT solution to the achievement of the strategy
•  People involved in the measurement and monitoring of the benefits realisation process

5. COMPETENCE

5.1 Skills and Knowledge

5.1.1 The IS auditor also should provide reasonable assurance that he/she possess the relevant skills and knowledge to carry out the post-implementation review of the IT solution. Where expert input is necessary, appropriate input should be obtained 

6. PLANNING

6.1 Scope and Objectives of the Review

6.1.1 The IS auditor, in consultation with the organisation as appropriate, should clearly define the scope and objectives of the Post Implementation Review. The aspects to be covered by the review should be stated explicitly as part of the scope.
6.1.2 For the purpose of the review, the stakeholders in the implementation should be identified.
6.1.3 The findings and conclusions of any prior reviews of the IT solution or implementation process—pre-implementation or concurrent reviews—should be considered in determining the scope and in audit planning.

6.2 Sign-off for the Terms of Reference

6.2.1 Depending on the organisational practices, the IS auditor should obtain the concurrence of the relevant parties in the organisation for the terms of reference and the approach. If the review is being initiated by a third party, they should also agree to the terms of reference.

6.3 Approach

6.3.1 The IS auditor should formulate the approach to provide reasonable assurance that the scope and objectives of the review can be fulfilled in an objective and professional manner. The approach should be appropriately documented. The use of expert input should be specified as part of the approach. Post-implementation reviews are not limited to the first review after implementation of the IT solution. Multiple reviews may be performed to identify improvements in the implemented solution.

7. PERFORMANCE OF AUDIT WORK

7.1 Execution of Post-Implementation Review

7.1.1 A post-implementation review should be scheduled at a reasonable time after the IT solution has been implemented. Typical periods can range from four weeks to six months, depending upon the type of solution and its environment.
7.1.2     A post-implementation review is intended to be an assessment and review of the final working IT solution. Ideally, there should have been at least one full implementation and reporting cycle completed to perform a proper review. The review should not be performed while still dealing with initial issues and teething troubles, or while still training, and educating users. However, where possible, the review should be performed while the opportunity remains to incorporate final improvements to derive optimum benefit from the IT solution.
7.1.3 Review procedures should include the study of available documentation (such as business case, business requirements including business controls, feasibility study, system, operational and user documentation, progress reports, minutes of meetings, cost/benefit reports, testing and training plans), discussions with stakeholders, hands-on experimentation and familiarisation with the IT solution, observation and inquiry of business and project personnel, and examination of operational and control documentation. 
7.1.4 Appropriate resources to carry out the post-implementation review should be identified and allocated, and the performance of the review should be planned in conjunction with relevant auditee personnel.
7.1.5 Agreement should be reached regarding the format, content, audience and timing, where possible, of reporting the results of the post-implementation review.
7.1.6 The stated objectives of the IT solution, costs and benefits should be studied in detail. The extent of achievement of the objectives and actual costs and benefits should be evaluated together with the processes and systems used to capture, monitor and report performance, costs and benefits. As part of this exercise, the productivity/performance improvements delivered by the IT solution should also be studied. Suitable measurement criteria should be used in this context. The cost and/or time overrun, if any, should be analysed by reference to their causes and their effects. Controllable and uncontrollable causes should be identified separately.
7.1.7 The process followed for defining and implementing the IT solution should be evaluated with reference to its appropriateness, as well as its effectiveness. 
7.1.8   The adequacy and effectiveness of education and training provided to users and staff supporting the IT solution should be reviewed.
7.1.9 The reports of any prior reviews performed either internally or by external reviewers on a pre-implementation basis or concurrently with the implementation process should be studied, and the status of recommendations and actions taken verified.
7.1.10   Since the post-implementation review is examining an IT solution, in general, the IT solution should satisfy appropriate COBIT control objectives. The extent of compliance with relevant control objectives and the effect of noncompliance should be analysed and reported. Further, critical success factors, key goal indicators, key performance indicators and maturity model benchmarks from COBIT Management Guidelines should be adapted as appropriate for the IT solution and implementation process being reviewed.
7.1.11 Appropriate management trails should be maintained for the data gathered, analysis made, inferences arrived at as well as corrective actions recommended.
7.1.12  The extent of compliance with statutory and regulatory requirements and organisational policies and standards of the IT solution and implementation process should be reviewed.
7.1.13 Where appropriate, automated testing tools and CAATs may be used to test relevant aspects of the IT solution.
7.1.14 The review should highlight risks and issues for necessary corrective action, together with opportunities for improvement in controls or increased effectiveness of the implementation process.
7.1.15 Reported findings, conclusions and recommendations should be based on an objective analysis and interpretation of the information and evidence obtained during the post-implementation review.

8. BENEFITS REALISATION REVIEWS

8.1 Benefits Realisation Review

8.1.1 All IT projects are actually business projects and should have a business rationale from the outset. Their success or failure should be measured either in financial terms or as a contribution to achievement of the strategic business plan. Benefits realisation reviews should focus not only on what has been achieved but what remains to be done. Organisations that undertake benefits realisation reviews to fine tune best practices and learn lessons reap benefits when their next project is undertaken.

8.2 Benefits Realisation Review Objectives

8.2.1 The objectives of a benefits realisation review are to evaluate the operational success of the new IT solution, and to assess the actual costs, benefits and savings in comparison with budgeted amounts. The review may also examine the effectiveness of the process used to deliver and implement the IT solution. A key consideration is whether or not the original system objectives and schedules have been achieved. This requires a detailed understanding of as-is and to-be processes, to assess the extent to which the objectives of the to-be processes have been achieved.
8.2.2 The benefits realisation component of a post-implementation review report should address:

• Actual costs compared to budgeted costs
• Actual benefits compared to budgeted benefits
• Return on investment
• Actual savings compared to budgeted savings
• Actual project completion dates compared to planned completion dates
• Original objectives compared to accomplished objectives
• Assessment of the adequacy and quality of documentation and controls, including management trails
• Actual IT solution performance compared to anticipated performance
• Overall user satisfaction and understanding of the new IT solution system
• Performance improvement suggestions for future IT solution implementation projects

9. OUTSOURCING

9.1 Outsourcing of IT

9.1.1 Where the organisation has partially or fully delegated some or all of its IT solution implementation to an external provider of such services (the service provider), the IS auditor should assess the effect of such arrangements and review the adequacy of, and conformance/compliance with, contracts, agreements and regulations with the service provider.
9.1.2  The IS auditor should obtain an understanding of the nature, timing and extent of the outsourced services. Also, the IS auditor should establish what controls the service user has put in place to address the business requirements and controls required by the organisation (refer to guideline G4 Outsourcing of IS Activities to Other Organisations).

10. REPORTING

10.1     Report Content

10.1.1 The report on the post-implementation review should address the following aspects depending on the objectives and scope of the review:

• The scope, objectives, methodology followed and assumptions made
• Assessment of whether the intended objectives of implementing the IT solution has been met, and whether the IT Solutions are aligned to meet the business objectives.
• An overall assessment of the implementation process in terms of key strengths and weaknesses as well as the likely effects of the weaknesses 
• Recommendations to overcome the significant weaknesses and to improve the implementation process
• Potential risks and means to mitigate such risks
• The extent of compliance with COBIT’s information criteria
• Recommendations to improve future IT solutions and implementation processes
• Training of the users on the IT solution implemented
• Acceptance and adaptability of the IT solutions across the organisations

10.1.2  The observations and recommendations should be validated with the stakeholders and organisation (and service provider if applicable), as appropriate, and responses obtained before finalising the report.

10.2 Weaknesses

10.2.1  Weaknesses identified during the post-implementation review, either due to lack of controls, poor implementation processes or nonmitigation of associated risks to acceptable levels, should be brought to the attention of the business process owner and to IS management responsible for the implementation of the IT solution. Where weaknesses identified during the post-implementation review are considered to be significant or material, the appropriate level of management should be advised immediately to allow early corrective action.
10.2.2 Since effective controls over IT solutions are dependent on general IT controls, any weaknesses in these areas should also be reported. In the event that general IT controls are not examined, this fact should be included in the report.
10.2.3  The IS auditor should include appropriate recommendations in the report to strengthen controls to mitigate associated risks.

11. FOLLOW-UP ACTIVITIES

11.1    Timeliness

11.1.1 The effects of any weaknesses identified by the post-implementation review are likely to be wide-ranging and high-risk. Therefore, the IS auditor should carry out, where appropriate, sufficient, timely follow-up work to verify that management action is taken to address weaknesses and manage risk effectively.

12.    EFFECTIVE DATE

 12.1 This guideline is effective for all information systems audits beginning 1 January 2005. A full glossary of terms can be found on the ISACA web site at www.isaca.org/glossary.

APPENDIX

Reference
 IS Auditing Guideline G23, System Development Life Cycle (SDLC) Review