Planning and monitoring costs and benefits of information system procurement

Conclusions of the National Audit Office

The purpose of the audit was to investigate whether the costs and benefits of central government information system investments are being anticipated and monitored with appropriate procedures. The identified basic starting point was that the development of information systems is usually connected to a more comprehensive development effort. The investments are usually realised as projects, and the projects may consist of several consecutive or parallel procurement processes.

The audited entities included the Ministry of Justice, the Ministry of Defence, the Ministry of the Interior and the Ministry of Finance. Within the administrative sector of these ministries, practices used by the Defence Forces, the Emergency Response Centre Administration, the Finnish Immigration Service, the State Treasury, the Tax Administration and the occupational safety and health department of the Ministry of Social Affairs and Health in information system development, as well as some example projects, were investigated.

The audit revealed some areas that require development and some good practices that should be shared with all agencies involved in such development. Furthermore, a need to improve cooperation between the agencies and a need to more actively spread good practices were detected.

Original information system cost estimates are inaccurate, but monitoring and management of costs are usually working well

The original cost estimate for an information system investment is often exceeded. This is not a problem inherent to central government or the public sector, however: it is an observation that applies to the development of information systems in general. Based on the audit observations, the key reasons why the cost estimates are inaccurate are firstly the fact that the idea about the entity being developed is defective or unclear at the preparation stage and secondly the fact that problems encountered during development can lead to an increase of the anticipated development costs.

Financial motions on investment decisions and the justification therein should clearly indicate whether the cost estimate is indicative only or whether it is based on, for instance, a more specific determination of the functional scope so that the decision‐makers would have the prerequisites needed to assess reliability of the cost estimate and how likely it is that it will be exceeded. During the implementation of development projects, the monitoring of progress and risk management practices must allow for reacting to any situations that may increase the costs or cause other problems in due time before the risks are realised.

The audit revealed that the actual development costs are monitored and expenditure forecasts are regularly prepared. No major problems in the management of costs during the use and maintenance of information systems were detected.

Assessments regarding benefits that are prepared at investment preparation stage are defective and actual benefits are only occasionally monitored

Based on the audit observations, the benefits achieved with information system investments are only occasionally monitored. The foundation for the monitoring of benefits is laid at the investment preparation stage, but descriptions of the benefits that are prepared at the early stages are often defective and do not support monitoring during the latter stages of the investment project. The desired benefits should be emphasised throughout the project period and related estimates should be update as necessary, particularly in the case of any major changes. In addition to an improvement of internal efficiency and the profitability of administration, the projects aim at benefits and effects that can be seen By external customers.

The fact that the benefits are not properly monitored may be due to the fact that the agencies do not have any special incentive or obligation to do so. For example, reports on achieved benefits have not been required from projects that received “productivity funding”. Monitoring of the achieved benefits is important, because the investments were originally justified with the benefits. Appropriateness should be taken into account when assessing and measuring the benefits, however, as the measuring and verification of benefits is not an unambiguous process. All in all, the management of development projects should enable and support the achievement of the benefits for which the investment aims.

Applied financial planning processes do not support methods used in development of information systems

Development of operations and information systems is usually planned based on the calendar and the budget. The process is well‐established and creates a framework for the planning of investments, but it is based on the traditional way of thinking that costs can be fairly accurately anticipated before the implementation of a project – often several years in advance. This way of thinking does not apply to the agile operating methods that are more and more often used in development, which emphasise constant prioritisation and control instead of detailed plans made in advance in order to make the process more flexible.

The current budgeting process is not an actual impediment to agile development, as the agencies have been able to successfully utilise agile methods, but the process does not encourage agility. The new development methods do not eliminate the need to realise the Project preplanning and preparation stages, but these stages are often focused on different issues than in the case of a more traditional project model. That is why financial planning processes should better take into account new development practices. The ability of the organisations to engage in the new kind of development should also be ensured. This may require, for instance, closer control and more decision‐making by the management than the traditional operating models.

Recommendations of the National Audit Office

1. Benefits achieved by information system investments should be emphasised more. At the first stage, the Ministry of Finance should create benefit monitoring practices covering at least all projects that have received “productivity funding”.

2. Ministries should ensure that the materials used for the preparation of information system investments clearly indicate the basis of the given cost estimates and how accurate the cost estimates are.

3. The Ministry of Finance should assess, together with other ministries, the financial planning and monitoring processes related to the funding and budgeting of investments to create factors or operating models that support the methods used in the development of the operations and in the development of information systems.

4. The Ministry of Finance should promote a framework and shared operating methods to encourage the agencies to share the good practices related to their operations and the development of information systems. The experiences may involve cost‐benefit estimates, the implementation, control or cooperation methods used for investment projects, the application of procurement procedures, or risk management practices, for example.