Buck Kulkarni, CISA, CGEIT, PgMP, ACP
Many organizations have a well-established practice of conducting postimplementation reviews and/or audits of major projects to verify the business value delivered as well as performance against key parameters. The current reviews are largely aligned to the project process and, thus, have a built-in bias toward sequential processes, e.g., integration testing begins only after unit testing is signed-off, defects flow down the waterfall, money and time flow with the waterfall. The so-called “iron triangle” of time, cost and scope largely provides the framework for review.
Agile is becoming a preferred way of developing software and it brings some new paradigms for the auditor. The new drivers of scope are the product vision, product road map, release plan and iteration plans, and not the traditional, relatively easy-to-understand “frozen requirements.” The concept of feature prioritization is largely driven by the business, though IT may provide inputs and comments. There are somewhat ambiguous (at least initially) concepts, such as user stories, story points, done-done, velocity and deployment frequency as a metric for success. The project (and its outcome) is not as discrete as it used to be, but is just a point on the continuum that is expected to deliver consistent, measurable business value over a longer time horizon, say 3 to 5 years.
Consider the financial performance measurement. Traditionally, we tied scope and money and they had to change in tandem. As the agile practitioners like to say (and is music to the ears of business folks), “We welcome change.” So, to what do you allocate money? The team will tell you that they are focused on achieving their story points and velocity goals. The auditor will have to understand how these relate to money. Similarly, time is now a box of a set number of resources working for a set amount of time (such as a sprint) and what they will be developing in their next sprint will not be known until the sprint begins.
Consider the additional paradigms of lean and just-in-time. Was the project truly lean? Was waste minimized? Did the project release working software as frequently as expected? Is technical debt at a manageable level? These are old questions in manufacturing, but somewhat new to IT projects and auditors.
IT auditors will have to invest time and effort into learning these new paradigms to perform effective postimplementation reviews and audits.