Value Assessment Tool for ICT Projects at the European Commission 

 
Download Article Article in Digital Form

The Directorate-General for Informatics (DIGIT) enables the European Commission to make effective and efficient use of information and communication technologies to achieve its organisational and political objectives. More than 10 years ago, many European Commission departments, led by the Internal Audit Service and Directorate-General for Agriculture, selected COBIT as a framework for the assessment and improvement of IT processes.

IT governance processes at DIGIT involve strategy and portfolio management, project and development methodology, and enterprise architecture. Major challenges faced by DIGIT today involve improving the integration of business and IT planning cycles as well as optimising investments of scarce resources to maximise the business value of IT.

The Value Assessment Tool (VAST) research is one of many IT governance implementation initiatives led by Francisco Garcia Moran, director general of DIGIT, and Declan Deasy, director of information systems and interoperability solutions. This research takes advantage of frameworks such as ISACA’s Val IT and categorises non-financial benefits of projects to highlight and compare their expected value.

When speaking to DIGIT officials about the governance of IT within the Commission, their alignment with ISACA principles and their focus on the five areas of governance that are supported by COBIT are evident.

Georges Ataya, CISA, CISM, CGEIT, CISSP
Academic Director of IT Management Education, Solvay Brussels School of Economics and Management


Electronic government (e-government) is now mainstream for transforming the public sector so that it achieves its political objectives in an effective, efficient and transparent manner. Today, practically all policy initiatives result in related information and communication technologies (ICT) projects,1 and ICT has become a key enabler for policy impact, transparency and compliance to norms and standards.

At the same time, public organisations have increasingly limited resources, so new investments have to be made carefully. This trend has been reinforced due to the current economic crisis. Additionally, the complex characteristics of the public sector2 may further influence new initiatives, so projects could often go beyond their initial scope and budget, and require more time than had been envisaged. They, however, may still be considered successful.

While concepts such as cost-effectiveness and return on investment (ROI) can be easily used to define the success of a project in the private sector, within the public sector the created ‘public value’ has the biggest weighting.3 Costly and risky projects must be undertaken to comply with legislative requirements; extended scope is often accepted to satisfy all stakeholders; deadlines are extended to cover all business needs put forward. Therefore, the ‘public value’ created by an ICT project will determine its success and it should be differentiated from the conventional concepts of project benefits.

Based on similar reflections within the European Commission, the recently set up Corporate Project Office (CPO) had to look for a way to evaluate and prioritise promising ICT projects. Such evaluation needed to distinguish between the public value created by the project (qualitative value) and the cost effectiveness of the project (quantitative value), and at the same time take into account the environment in which this new ICT project would be developed, implemented and operated.

A number of well-established methodologies used in private and public organisations were evaluated for their potential to be reused in the European Commission’s context. The analyses concluded that none of the evaluated solutions fit with the specific organisational setup. Therefore, building on these methodologies, the aim was to define a custom-made, easy-to-use and automated solution, allowing the Commission’s services to assess the expected value of envisaged projects. The result of the work is the Value Assessment Tool (VAST) of the European Commission and is the subject of this article.

ICT Context at the European Commission

The European Commission is a complex, decentralised organisation composed of 41 services with a great level of autonomy, each under the leadership of a director-general. This organisational setup is reflected in the ICT aspects of the institution:

  • At the business process level, services are fully autonomous and harmonisation is done on an ad hoc and voluntary basis.
  • At the information systems level, services are also autonomous, but corporate systems are mainly developed by the Directorate-General for Informatics (DIGIT), which is also responsible for defining the development and operating the underlying infrastructure for which certain layers are managed centrally.
  • For proximity support services, a consolidation exercise is underway.
  • At the infrastructure level, the network, for example, is managed centrally.

DIGIT is also responsible for e-government: internally with the eCommission initiative and with Member States through the Interoperability Solutions for European Public Administrations (ISA) programme.

Like other public administrations, the European Commission is subject to constraints: constant or diminishing resources, in the face of increasing demand. Therefore, priorities have to be carefully established, and the launch of new ICT projects should be based on their expected value. Within the Commission, this aspect is reinforced by the nature of the organisation, since duplications might easily occur when ICT is managed at several, sometimes independent, levels.

To alleviate such difficulties, it is essential to assess the value promised by a given project at an early stage of its inception and, most important, to distinguish between its potential qualitative and quantitative value. It is also important to benefit from a fair comparison element between projects coming from different services. These elements triggered the need for a value assessment methodology.

However, due to the organisational context of the European Commission, such a methodology could not serve its purpose if it was used only at corporate level by the CPO. A potential value assessment methodology needs to be widely adopted by the decentralised structures responsible for ICT. Only suitability to the whole ICT community would reveal the full potential of the selected method.

Therefore, for such a methodology to be accepted, a first requirement is its ease of use and ‘self-training’. Qualitative value assessments should take 30 minutes, given the fact that people conducting the assessment have familiarised themselves with the new ICT project through the standard project documentation for the Commission (e.g., business case and vision) or they are part of the project team. The latter case of usage could be defined as self-assessment.

Furthermore, this methodology has to be usable by both ICT and non-ICT professionals so that they can complement each other’s views (business and technological) during the project assessment process. Moreover, when decisions have to be taken based on the assessment, the chosen method should provide meaningful, but at the same time, concise, output so that it could be used as a communication means with top management.

Lastly, going beyond the organisational setup and environment, a specific and unique requirement to the Commission is the need to estimate the value of a project at the level of the European Union.

Value Assessment Methodologies Review

Taking into account the ICT organisational context of the Commission and the specific requirements that it imposed, well-established and practically oriented frameworks/ methodologies from both the private and the public sector were selected and examined.

Val IT Framework
Val IT is a governance framework initiated to address the lack of IT investment and management guidelines. Its goal is to ensure the delivery of optimal value from IT investment at adequate costs and levels of risk. The Val IT framework provides extensive guidelines and describes processes to be set up and followed in three main domains: Value Governance, Portfolio Management and Investment Management.4

On the positive side, it was considered that Val IT gives a holistic, high-level overview of the mechanisms that can be used to manage the value derived from IT. However, it was not possible to be applied at the European Commission due to the highly decentralised organisational setup when managing ICT. Bearing in mind this constraint, Val IT was used only as an insight for the endeavour.

Demand and Value Assessment Methodology for Better Government Services
The Demand and Value Assessment Methodology for better government services is an initiative of the Australian government that assesses the:

  • Demand of e-government services from the viewpoint of end users
  • Value of such services, based on the more traditional costs and benefits, but taking into account social and governance implications

It is supported by a spreadsheet-based tool5.

The major advantages of the Demand and Value Assessment tool are that it covers both financial and non-financial value and is assisted by a semi-automated tool giving graphical representation of the results. These also closely matched the projects requirements. However, the assessment criteria of the methodology were chosen for the assessment of service provision at the national or local government level and, therefore, differ greatly from those considered at the European level. Furthermore, as it is rather detailed and provides an ‘open’ structure (criteria and objectives need to be defined by the evaluator), the use of the methodology entailed training, and this was not in line with the aim of easy use and quick results.

Economic Efficiency Assessment Methodology (WiBe) 4.0
WiBe has been used since 1992 by the German federal administration to ensure the economic efficiency of its ICT projects.6 It is based on two main steps:7

  1. Identifying parameters that may have an impact on the economic efficiency of the project (a general catalogue of criteria is provided)
  2. Determining the economic efficiency of the project with the support of detailed guidelines

The core of WiBe is an exhaustive list of criteria, of which some can be quantified in monetary terms, and some in non-monetary terms. In order to evaluate a project, one should pick the applicable criteria from this catalogue. The strong point of this approach is that it may entail accurate cost/benefit analysis in both monetary and non-monetary terms. However, due to the differences between the chosen criteria, the cross-comparison between projects could be questioned. Again, a disadvantage for the use of this methodology is that it may require up to one day or even more of training if it is used for the first time.8

MAREVA
MAREVA was launched in 2005 by the French eGovernment Agency (ADAE) and is widely used by the French governmental organisation, and more recently in Quebec.9 This methodology bases its assessment on the following axes:10

  • Profitability
  • Risk control
  • Values both qualitative and quantitative for the whole public sector
  • Values both qualitative and quantitative generated outside of the public sector (citizens and enterprises)
  • The project’s necessity

MAREVA is composed of two spreadsheet-based files. One addresses the first axis, and the other targets the other four axes. It has been positively evaluated that this approach is structured, and the tool comes with a training package. However, the method focuses on the financial aspects, is greatly detailed, and requires training to understand the various concepts and the calculations. On the contrary, the approach of the European Commission was to focus on the qualitative value.

Value-measuring Methodology
The Value-measuring Methodology (VMM) was developed between 2001 and 2002 under the co-ordination of the Federal Chief Information Officer Council and has the main objective of sound ICT investment management. VMM encompasses four steps:11

  1. Develop a decision framework.
  2. Perform an alternative analysis.
  3. Pull the information together.
  4. Communicate and document.

This approach is closely linked to the establishment of a business case for new projects and could assist portfolio management practitioners.

The major advantage of this methodology is that it aims to assess both qualitative and quantitative values and builds clear processes to be followed. However, VMM does not propose the set of criteria to be used; it suggests only how to select these criteria and prioritise them to fit the assessed investment closely. While this may entail a close match and precision, the criteria selection process may require time. Further, the cross-comparison between initiatives may be weak if the criteria differ from project to project.

The scope used to evaluate these methodologies has been confirmed by Gartner’s report on Public-Value-of-IT Frameworks,12 in which worldwide examples were selected and reviewed, underlining both their strengths and their weaknesses. Four out of the five assessed methodologies form part of this report.

Project Approach

The analyses of the existing methodologies provided a good overview and important insights on the subject of value assessment in the private and public sectors. They have also shown that for such a methodology to serve its purpose, it should be carefully tailored to the environment in which it will be used. However, considering that none of the methodologies fully satisfied the requirements posed by the specific ICT context of the European Commission, a decision was taken to develop a customised solution.

As already explained, the assessment needed to go beyond the traditional financial benefits. Therefore, both qualitative and quantitative criteria had to be used, and the qualitative criteria had to evaluate explicitly the value for the European Union promised by the new project. Furthermore, all ICT projects’ assessments should use the same set of criteria to allow cross-comparison and prioritisation. Although keeping its focus on ICT, it should use, as much as possible, business-oriented terms to assure suitability for both business and IT services. Finally, in order to assure effortless adoption, ease of use, and concise and quick presentation of results, a spreadsheet-based tool was selected. Thus, a certain level of automation and enabling flexibility was achieved while the overall approach was still in its adoption phase.

To assure the success of the tool, several iterations of development, tests and feedback sessions with a subset of the European Commission services were completed until a stable version was produced, supported by guidelines for a methodological reference for the use of the tool and for the analyses of the results. The next section of this article looks at the custom-made value assessment tool in more detail.

Presentation of the Value Assessment Tool

VAST is a spreadsheet-based tool that consists of an Index page, five Value perspectives and a graphical Results page (depicted in figure1 ). The tool is also supported by guidelines that serve as a methodological reference and help its use. Each of these parts is presented in the following sub-sections.

Figure 1

Project Identification
The Index page collects general information about the project: project name, contacts, business owner of the project and date of assessment. This Index page also serves as a central point of the tool, and in doing so, it provides shortcuts to the other parts of the tool.

Value Perspectives
VAST consists of five value perspectives: four estimate the qualitative value of the ICT projects (Value for European Union [EU], Value for European Commission [EC], Risks and Necessity) and one estimates the quantitative value (Financial Costs and Benefits). The four qualitative perspectives consist of sets of criteria that are grouped into a number of sections and sub-sections. The quantitative perspective requires financial information on the project, and some exact figures need to be provided. The five perspectives, with their objectives, are outlined as follows.

The Value for EU perspective looks at the assessment of the external value of an ICT project. Any benefits delivered outside the Commission itself (value to the European society or to European citizens) are considered external value. If the project does not have external users and is used for purely administrative purposes, this value perspective can be omitted.

The Value for EC perspective encompasses criteria that assess the internal value of an ICT project. All factors that can contribute to the improvement of the Commission performance are considered to deliver an internal value, including:

  • Political value—Whether the IT solution contributes to achieving the Commission’s strategic objectives
  • Administrative value—Whether the project will contribute to the work efficiency and effectiveness
  • IT governance value—Whether the project will contribute to the rationalisation of the Commission’s information systems portfolio
  • Internal users’ value—The value for the Commission’s employees

The Risks perspective indicates risks related to the need for adequate project management to deliver the ICT project. It also assesses technical, security, business, legal and acceptance-related risks.

The Necessity perspective assesses the need for supporting or developing the project by looking at four subject areas: external demand, internal demand, business needs and technical needs. This perspective tries to answer questions such as ‘Do we really need to undertake this project?’ and ‘Why do we need to support it?’

The Financial Costs and Benefits perspective aims to quantify, in monetary terms, the costs and benefits of the ICT project. The approach consists of identifying every cost for development, maintenance, support, training and infrastructure and the benefits from saved time, reduction in direct operation costs and reduction in IT costs.

Results of the Value Assessment
Each qualitative criterion has a pre-assigned weight (from 1 to 3), which is based on its importance. Furthermore, each criterion has four possible assessments for which it receives between 0 and 3 points multiplied by the criterion weight. This approach promotes both a consistent way of evaluation and fine-tuning and precision of the achieved results. The quantitative criteria are expressed in numbers.

The defined formulas are calculated by the spreadsheet application and are consolidated in the Results page of VAST. The Results page is literally one page (printed) that graphically presents the four qualitative perspectives. The quantitative perspective follows a similar approach and is represented in a different graph.

The assessment is based on the already-provided data. However, to have a complete overview of the project, some supplementary information needs to be introduced into the Results page. This supplementary information is comprised of the project executive summary, the addressed business domain, the main stakeholders and the time frame. Adding this information to the value assessment allows the Results page to be used independently from the tool.

Guidelines
VAST is supported by guidelines that explain each criterion addressed in the five value perspectives. The document lists the criteria in the exact same way as the tool so that a user can easily navigate through it. The guidelines also provide general information on how the tool should be used and how to analyse the achieved results and use the tool as a methodological reference. The guidelines are, thus, entirely sufficient for self-training on VAST.

Practical Applications of the Value Assessment Tool

For an evaluation and validation of the tool, three iterations of development were undertaken. The chosen approach and the set of criteria were presented to the Commission IT community, and the tool was provided for free use by the Commission services. Interested parties were requested to provide detailed feedback from tests undertaken. Building upon these remarks, an adjusted and stable version of the tool was produced. Throughout these iterations, the achievement of the requirements put forward has been confirmed. Some limitations of the tool were also revealed.

The IT professionals at the Commission agreed that it is rather difficult to express in a concise manner the benefits of ICT projects, and they agreed that the VAST tool, which allows qualitative and quantitative value to be distinguished, helps to do this. The VAST tool also sheds light on otherwise overlooked areas of projects. For example, if a system aims at serving needs of the European citizens (external user needs), the spotlight would usually focus on the satisfaction of these needs. However, the project might be using innovative technology or be producing reusable modules. These possibilities additionally increase its value, and VAST can demonstrate this.

Utilising the exact same criteria for each assessment allows a valid cross-comparison of similar projects. For example, VAST can be very useful when the business requests a higher number of projects than what the IT entity can feasibly deliver. The use of the tool allows justified prioritisation, and the projects with higher value can be put forward. However, when comparison needs to be done between projects coming from different services, it is considered best that the evaluation be performed via a mediator (e.g., by the CPO).

The objective for the tool to be suitable for both business and the IT professionals was confirmed by the Health and Consumer Protection policies service. It established a practice to perform value assessment with VAST by representatives from the business and IT at the beginning of a project. Going together through each criterion, discussions identified the weak areas of the project. In this way, many concepts are clarified for the two parties, and a true partnership between business and IT can emerge from this practice.

The results of VAST comprise one self-explanatory page, which gives an overview of not only the qualitative and the quantitative value, but also the risks and necessity of the ICT project. Therefore, the tool can be used for communication purposes to engage senior management and stakeholders of the project. The Results page can be attached to other documentation or can be the subject of a specific meeting. Going even further, as found by the service for Trade Policies, VAST can be considered a helpful management tool.

Overall, it has been confirmed by the Commission services that the tool is easy to use and adopt. A project assessment lasts between approximately 30 minutes and one hour (at first use), assuming that the individuals conducting the assessment have familiarised themselves with the standard project documentation required at the Commission or that they are part of the project team on the business or technology side. It was also confirmed that there is no need for specific training, and where difficulties appear, the guidelines provide a sufficient level of clarification. However, the financial costs and benefits perspective was often noted as challenging to fill in, as concepts like saved time and workload are difficult to express in monetary terms.

Other European institutions (beyond the Commission) also showed interest in adopting VAST. However, as the tool is highly tailored to the Commission context, the adoption requires certain customisation, which has been the case with the European Chemical Agency.

Although this method of internal validation of the tool served its purposes, in order to assess VAST’s wider applicability and benefits, a systematic comparison of VAST with similar value frameworks should be conducted.

Summary and Conclusions

Public organisations are increasingly managed with limited resources, and decisions for new investments have to be taken cautiously. At the same time, risky and costly projects have to be undertaken due to the circumstances in which such organisations work: compliance with legal regulations, various stakeholders’ needs, etc. Thus, costly projects, from a financial perspective, can still bring benefits to public organisations and their stakeholders. This is especially true for the traditionally costly and complex ICT projects.

The public sector requires clear differentiation between the benefits of an ICT project (qualitative value) and its financial cost effectiveness (quantitative value). This work addresses precisely this issue. Building upon the well-established value assessment methodologies, VAST, the Value Assessment Tool of the European Commission, was delivered. To validate the work, three iterations of development were undertaken with the Commission’s services, which confirmed the achievement of the tool’s requirements and, thus, revealed its benefits: demonstrated value and benefits of ICT projects, cross-comparison and prioritisation, enhanced communication between project stakeholders, suitability for both the business and IT communities, ease of use, and adoptability.

Despite the benefits of VAST, the process of validation also revealed its weak points. It was observed that the financial costs and benefits perspective is challenging to fill in as it is sometimes hard to express in financial terms some of the benefits of a project (e.g., saved time and workload). Further, comparison between projects emanating from different services may be difficult if the assessment is not performed by a mediator, for example the CPO. The tool is highly tailored to the Commission’s context, and to be adopted by other organisations, it may require customisation. Finally, the tool has been tested only within the Commission, and to prove its general applicability, further tests should be undertaken. However, as it has been developed in the public domain, the tool package is freely available upon request.

Going beyond the scope of this work, the approach of using a tool for the assessment of important, but intangible areas of ICT management was positively accepted within the European Commission. Using the same approach, a tool for evaluation of the IT governance maturity of the organisation is in its pilot phase.

This article was previously published in Electronic Government and Electronic Participation:  Joint Proceedings of Ongoing Research and Projects of IFIP EGOV and ePart 2010 Conferences (Trauner, Austria, 2010), edited by Jean-Loup Chappelet, Olivier Glassey, Marijn Janssen, Ann Machintosh, Jochen Scholl, Eftimios Tambouris and Maria A. Wimmer. Permission to republish was granted by the editors of the IFIP EGOV Conference.

The latest version of VAST and its guidelines can be downloaded at http://ec.europa.eu/dgs/informatics/vast.

Endnotes

1 Schäuble, Wolfgang; 2007, www.wolfgang-schaeuble.de/ fileadmin/user_upload/PDF/070301egovernment.pdf
2 Rainey, Hal; Robert Backoff; Charles Levine; ‘Comparing public and private organizations’, Public Administration Review, 36/2/1976, p. 233–244
3 Halachmi, Arie; Tony Bovaird; ‘Process reengineering in the public sector: learning some private sector lessons’, Technovation, 5/17/1997, p. 227–235
4 IT Governance Institute, Enterprise Value:  Governance of IT Investments, The Val IT Framework 2.0, USA, 2006
5 Australian Government, Information Management Office, Demand and Value Assessment Methodology for Better Government Services, Canberra, Australia, 2004
6 ePractice, 2004, www.epractice.eu/en/library/281229
7 Federal Ministry of the Interior, Department IT 2 (KBSt), WiBe 4.0—Recommendations on Economic Efficiency Assessments in the German Federal Administration in Particular with Regard to the Use of Information Technology, Berlin, 2004
8 Ibid.
9 ePractice, 2007, www.epractice.eu/en/cases/mareva
10 Le portail du ministère du Budget, des Comptes publics, de la Fonction publique et de la Réforme de l’État, 2007, https://mioga.minefi.gouv.fr/DB/public/controlegestion/web/pages/CHAP_4_8_Methode_MAREVA.html
11 Chief Information Officer Council, 2007, www.cio.gov/documents/ValueMeasuring_Methodology_HowToGuide_Oct_2002.pdf
12 Gartner, Industry Research, ‘Worldwide Examples of Public-Value-of-IT Frameworks’ (ID Number:  G00146056), 2007

Stefka Dzhumalieva
a member of the strategy and portfolio management team at the European Commission since 2008, works on enforcing IT governance within the European Commission and steering the development of the e-Commission program. She can be reached at stefka. dzhumalieva@ec.europa.eu.

Franck Noël
was deputy head of the unit and led the strategy and portfolio management team at the European Commission. He is currently head of the IT office at the European Court of Auditors and is responsible for IT governance in the institution. He can be reached at franck.noel@eca.europa.eu.

Sébastien Baudu
a member of the strategy and portfolio management team at the European Commission since 2006, works on enforcing IT governance within the European Commission and steering the development of the e-Commission program. He can be reached at sebastien.baudu@ec.europa.eu.


Enjoying this article? To read the most current ISACA® Journal articles, become a member or subscribe to the Journal.

The ISACA Journal is published by ISACA. Membership in the association, a voluntary organization serving IT governance professionals, entitles one to receive an annual subscription to the ISACA Journal.

Opinions expressed in the ISACA Journal represent the views of the authors and advertisers. They may differ from policies and official statements of ISACA and/or the IT Governance Institute® and their committees, and from opinions endorsed by authors’ employers, or the editors of this Journal. ISACA Journal does not attest to the originality of authors’ content.

© 2011 ISACA. All rights reserved.

Instructors are permitted to photocopy isolated articles for noncommercial classroom use without fee. For other copying, reprint or republication, permission must be obtained in writing from the association. Where necessary, permission is granted by the copyright owners for those registered with the Copyright Clearance Center (CCC), 27 Congress St., Salem, MA 01970, to photocopy articles owned by ISACA, for a flat fee of US $2.50 per article plus 25¢ per page. Send payment to the CCC stating the ISSN (1526-7407), date, volume, and first and last page number of each article. Copying for other than personal use or internal reference, or of articles or columns not owned by the association without express permission of the association or the copyright owner is expressly prohibited.