Download INTRODUCTION TO COMPUTING PRINCIPLES

Performance Measures Indicating IT Project
Effectiveness and Investment Success
INTOSAI Working Group on IT Audit
The 6th Performance Auditing Seminar
Beijing, China
April 2010
Prepared By
State Audit Bureau of Kuwait
Information Technology Audit
Technical Support Department
2010
Performance Measures Indicating IT Project Effectiveness and Investment Success / 1
Benefits of Performance
Measurement
1. Benchmark the contribution and alignment of an
IT investment with strategies and business goals.
2. Provides an accurate and a quantifiable
assessment of improving and degrading
measured areas.
3. Assist in assigning focus to new areas of
improvement to help in directing funds in the
appropriate direction and magnitude.
2010
Performance Measures Indicating IT Project Effectiveness and Investment Success / 2
IT Investment Measurement
Approach
• The term “IT investment” is broad. It can be a
product, service or program.
• Areas of measurement can include financial
management, IT service/infrastructure availability,
quality, delivery time, cost, customer satisfaction,
resource utilization…etc
2010
Performance Measures Indicating IT Project Effectiveness and Investment Success / 3
IT Investment Measurement
Approach
• Having a set of measurements to cover multiple
IT areas helps in establishing a metric.
• Gradually building up with the investments goals
in mind will yield more specific measurements.
• Linking of measurements to Organizational and
IT goals gives a strong insight on areas with
potential development.
2010
Performance Measures Indicating IT Project Effectiveness and Investment Success / 4
Measurements
• Focus based: Financial, Customer, Internal,
Learning and Growth.
• Measurement unit: Percentage, number, Ratio
…etc
• Measurement direction and Target
2010
Performance Measures Indicating IT Project Effectiveness and Investment Success / 5
IT Investment Case
• Initial audit conducted revealed findings that were
reported.
• Measurements can be applied to in order to track
performance areas of the findings.
2010
Performance Measures Indicating IT Project Effectiveness and Investment Success / 6
IT System Description
1. Our case required the audit of a Rationing system
developed by the IT department of a Government agency
to automate and optimize the delivery of essential
subsidized commodities to eligible beneficiaries. The
Ration Department is in charge of setting the rationing
regulations and laws by Setting eligibility criteria and
Determining quotas and prices.
2. The IT department of the ministry is in charge of the
Rationing System to:
2010
•
Issue, renew, amend the ration card data for
beneficiaries.
•
Distribute the goods to consumers using the system in
branches of Co-operative wholesale societies.
Performance Measures Indicating IT Project Effectiveness and Investment Success / 7
Expected System Benefits
As reported by the Ration Department, the system
is expected to deliver the following benefits:
1.Provide the information on beneficiaries, ration
cards and transactional data in an electronic format
for easier accessibility and accuracy.
2.Non-issuance of duplicate ration cards per
beneficiary.
3.Avoid the current manual issuance and indexing of
ration cards.
2010
Performance Measures Indicating IT Project Effectiveness and Investment Success / 8
Expected System Benefits
4. Link the branches to the main system at the
ministry to automate change.
5. Issue/print the new ration cards.
6. Better management of inventory and governing
of transactions.
7. Provide accurate statistical reports in a timely
fashion with ease.
2010
Performance Measures Indicating IT Project Effectiveness and Investment Success / 9
Audit Goal
The goal of the audit is to:
1. Verify the efficiency and effectiveness of the
system of issuing ration cards and the system of
distribution/sales.
2. Verification of the systems availability.
3. Alignment to benefits and requirements.
4. Measure the security and safety of the systems.
2010
Performance Measures Indicating IT Project Effectiveness and Investment Success / 10
Initial Observation
There was an absence of organizational and IT
goals and objectives.
Impact: Limits the possibility of relating the outcome
of performance measures to the Organizational and
IT goals as IT investment processes, resources and
activities cannot be aligned/linked to a higher level
business objectives. Thus, the measures will be
focused on IT processes, resources and activities
surrounding the Rationing system.
2010
Performance Measures Indicating IT Project Effectiveness and Investment Success / 11
About the Findings
Next, we will discuss some of the findings of our
audit. Since there were no pre-established
measures, these audit findings will be used to
suggest measures to the concerned agency.
2010
Performance Measures Indicating IT Project Effectiveness and Investment Success / 12
Audit Findings Where Performance
Measures are Applicable
Finding (1): Absence of documented and approved
procedures and processes for project
management.
Potential performance measures:
1. Percent of meeting the predefined business
benefit
2. Percent of budget deviation value compared to
the assigned budget
3. Percent of project progress (on time and on
budget)
2010
Performance Measures Indicating IT Project Effectiveness and Investment Success / 13
Audit Findings Where Performance
Measures are Applicable
4. Percent of following project management
standards and practices
2010
Performance Measures Indicating IT Project Effectiveness and Investment Success / 14
Audit Findings Where Performance
Measures are Applicable
Finding (2): Absence of standardized procedures
and processes for IT resource procurement.
Potential performance measures:
1. Amount of reduction of the purchasing cost
2. Percent of key stakeholders satisfied with
suppliers
3. Percent of initial requirements addressed by the
selected solution
4. Percent of compliance with standing procurement
policies and procedures
2010
Performance Measures Indicating IT Project Effectiveness and Investment Success / 15
Audit Findings Where Performance
Measures are Applicable
5. Percent of variance amongst budget, forecast
and actual cost
6. Percent of overall cost that is allocated according
to the agreed-upon cost models
7. Percent of disputed costs by business
2010
Performance Measures Indicating IT Project Effectiveness and Investment Success / 16
Audit Findings Where Performance
Measures are Applicable
Finding (3): Absence of technological direction.
Potential performance measures:
1. Number and type of deviations from technology
infrastructure plan
2. Percent of compliance with defined IT
architecture and technology standards
3. Number of emergency changes related to the
infrastructure components
2010
Performance Measures Indicating IT Project Effectiveness and Investment Success / 17
Audit Findings Where Performance
Measures are Applicable
Finding (4): Absence of IT investment feasibility
study.
Potential performance measures:
1. Percent of stated benefits that were not achieved
due to incorrect feasibility assumptions
2. Percent of stakeholders satisfied with the
accuracy of the feasibility study
2010
Performance Measures Indicating IT Project Effectiveness and Investment Success / 18
Audit Findings Where Performance
Measures are Applicable
Finding (5): Absence of disaster recovery and
business continuity plan and incident logs.
Potential performance measures:
1. Percentage of downtime due to unplanned
outages
2. Percent of availability SLAs met
3. Frequency of service interruption of critical
functions
2010
Performance Measures Indicating IT Project Effectiveness and Investment Success / 19
Audit Findings Where Performance
Measures are Applicable
Finding (6): Operation and user manuals are not
updated.
Potential performance measures:
1. Percent of business owners satisfied with
application training and support materials
2. Number of incidents caused by deficient user and
operational documentation and training
3. Satisfaction scores for training and
documentation related to user and operational
procedures
2010
Performance Measures Indicating IT Project Effectiveness and Investment Success / 20
Audit Findings Where Performance
Measures are Applicable
Finding (7):
1) The Ration card system does not restrict the
issuance of cards to ineligible individuals.
2) The Ration card system does not implement any
restrictions regarding the creation of non-existing
beneficiaries.
3) The Ration card system allows the creation of
beneficiaries with out names or contact
information.
2010
Performance Measures Indicating IT Project Effectiveness and Investment Success / 21
Audit Findings Where Performance
Measures are Applicable
Potential performance measures:
1. Percentage of ineligible beneficiaries
2. Percentage non-existing beneficiaries
3. Percentage of beneficiaries with missing or
incomplete information
4. Percent of data integrity
5. Frequency of internal control incidents
6. Cost of non-compliance, including settlements
and fines
2010
Performance Measures Indicating IT Project Effectiveness and Investment Success / 22
Audit Findings Where Performance
Measures are Applicable
Finding (8):
1) The Ration distribution/sales system
transactional database contains sales of
commodities that are unidentifiable on the
system.
2) The Ration distribution/sales system
transactional database contains sales with
miscalculated commodity prices.
2010
Performance Measures Indicating IT Project Effectiveness and Investment Success / 23
Audit Findings Where Performance
Measures are Applicable
Potential performance measures:
1. Percentage of transactions with unidentifiable
commodity.
2. Percentage of transactions with miscalculations.
3. Percent of data integrity.
2010
Performance Measures Indicating IT Project Effectiveness and Investment Success / 24
Audit Findings Where Performance
Measures are Applicable
Finding (9): The distribution centers run a client
software that saves the transactions on an
unprotected local database that is reconciled with
the main collective database of transactions at
the end of the day due to design limitations.
Some databases were reported to be found
emptied at the time of reconciliation which
suggests also the possibility of any other
database being tampered with.
2010
Performance Measures Indicating IT Project Effectiveness and Investment Success / 25
Audit Findings Where Performance
Measures are Applicable
Potential performance measures:
1. Number of security incidents with business
impact
2. Number and type of suspected and actual access
violations
3. Number of occurrences of an inability to recover
data critical to business process
4. Revenue lost due to loss of data
2010
Performance Measures Indicating IT Project Effectiveness and Investment Success / 26
Summary
• The audits findings we communicated to the
agency in order to:
1. Resolves issues permanently where possible.
2. Apply appropriate controls and adopt
suggested measures to follow up with actual
performance measurement.
The audit concluded for the Ration system
provided the agency with the means to
establish areas of improvement, measurement
baselines and targets to achieve.
2010
Performance Measures Indicating IT Project Effectiveness and Investment Success / 27
Discussion
Thank you…
2010
Performance Measures Indicating IT Project Effectiveness and Investment Success / 28