Download The Evaluation of Information Systems Success: A new Perspective

6th Global Conference on Business & Economics
ISBN : 0-9742114-6-X
The Evaluation of Information Systems
Success: a New Perspective
Hafid Agourram
King Fahd University of Petroleum & Minerals, Saudi Arabia
Email: [email protected]
(966) 3 860 2678
Amine Nehari Talet
King Fahd University of Petroleum & Minerals, Saudi Arabia.
Email: [email protected]
Mailing address:
P O. Box 805
King Fahd University of Petroleum & Minerals,
Dhahran 31261
Saudi Arabia.
Key words: Information System Success (IS), Alignment, Fit, National culture, Business
Strategy, Information System Strategy
Acknowledgement:
We would like to thank King Fahd University of Petroleum and
Minerals and specifically the College of Industrial Management for their generous supports.
OCTOBER 15-17, 2006
GUTMAN CONFERENCE CENTER, USA
1
6th Global Conference on Business & Economics
ISBN : 0-9742114-6-X
Abstract
Information system success has been a challenging concept for researchers for many
years. Research on the definition and the evaluation of information system (IS)
success was growing yet confusing. The social nature of IS success concept is a
major factor that may explain this confusion. From the global and multicultural
perspectives, a recent study showed that the meaning of IS success is not
homogenous across different national cultures. Organizations are more and more
attracted by doing business in different cultures and become global. This attraction is
also yielded by the successful technical capabilities of the new information
technologies that support the globalization strategy. The existing models of IS
success require a large amount of modifications and adaptations before they can be
applied in local and international organizations. This paper proposes a
comprehensive framework that is easy to use, integrates existing knowledge on the
alignment and IS success domains and can be applied to any local and national
context. The power of the framework is in its practicality and flexibility.
Introduction
The use of information technology and information systems in today’s organization
is growing in a phenomenon way. The advanced sophistication of the new
information technologies combined with their yielding benefits are major factors
that justify their massive use in almost every type of organization. Morton (1991)
reported that the rate at which new technology is introduced is increasing 20 to 30
percent annually and the numbers are promising to grow in the future. However, IS
OCTOBER 15-17, 2006
GUTMAN CONFERENCE CENTER, USA
2
6th Global Conference on Business & Economics
ISBN : 0-9742114-6-X
investment decision makers are confused and frustrated because they are unable and
find it very difficult to justify their IT and IS investments by using existing tools of
IS performance and success. Researchers have developed many IS success models to
help managers measure the performance of their IS (DeLone and McLean, 1992,
2002; Seddon, 1997; Robey and Boudreau, 200.). The major problems with these
models are: First, they cannot be applied to all contexts and to all types of
information systems. Second, these models show how to evaluate IS success
independently from other organizational variables such as structure, strategy, and
processes. Lastly, the values of these measurements may sometimes lead to
confusion because they are not compared to a set of predefined expected values that
define success to the organization. For example, it may be that an organization
evaluated the success of its information system by using the impact of the system on
the decision making process dimension and found that this impact is average. Is that
a success or not? The answer is we don’t know because we luck a predefined level
of the impact of the IS on its users to which we can compare the actual value. It has
been largely documented that the organization and its information systems must fit
each other; that is, one cannot be separated from the other. In this paper, we use this
parallel philosophy between the organization and its IS to develop a very
comprehensive process framework that uses existing knowledge on the fit theory.
The framework can be used in any national and organizational culture and is very
practical for managers who wish to evaluate IS success. The following sections will
discuss existing IS success theory, the fit theory, our proposed framework, and the
benefits of the framework.
OCTOBER 15-17, 2006
GUTMAN CONFERENCE CENTER, USA
3
6th Global Conference on Business & Economics
ISBN : 0-9742114-6-X
Information Systems success
Information system (IS) success and its determinants have long been considered
critical to the field of information systems. Despite many attempts to model success
(Delone and McLean, 1992, 2003), IS success definition and measure is still
problematic for many factors. The first factor is the mixture of the technical and
social aspects of an IS. IS success is then a perspective that emerges from the social
and technical interplay within organizations (Kanellis, Lycett and Paul, 1998).
Second, Alter (2000) argues that information technology and work practices are now
so intertwined that it is difficult to identify their respective contribution to success.
Other researchers link the difficulty of defining IS success to the methodological
aspects involved in measuring IS success «Specifying a dependent variable is
difficult because of the many theoretical and methodological issues involved in
measuring IS success» (Garrity and Sanders; 1998, p. 14). Seddon, Staples,
Patnayakuni, and Bowetell, (1999) argue that IS success is a fuzzy concept
contingent upon different stakeholders and different types of IT. In the practice
community, Markus and Tanis (2000) claimed that there is a fundamental gap in
both practical and academic thinking about information systems lack of consensus
and clarity concerning the meaning of success where information systems are
concerned.
The problem of IS definition and measurement becomes more difficult and more
complicated if we add the international dimension (Ishman, 1998; Garrity and
Sanders,1998, Pauleen et al., 2006). The international dimension includes cultural
OCTOBER 15-17, 2006
GUTMAN CONFERENCE CENTER, USA
4
6th Global Conference on Business & Economics
ISBN : 0-9742114-6-X
terms such as values and assumptions which may be at the heart of the differing
perceptions and interpretations of IS success. Shing-Kao (1997) argues that
«Research has shown that people notice, interpret and retain information based on
their values, assumptions and expectations. Different assumptions and values lead to
different ways of looking at the same thing» (p. 13).
This international differing perception of the meaning of a phenomenon is an
important topic in International Management or Cross-cultural Management
disciplines. Do theories and concepts born in a specific culture apply or have the
same meaning in other cultures? Hofstede (1993), for example, after a large survey
on work-related values in sixty countries, concluded that management theories and
findings are not automatically transferable from one context to another. Shing-Kao
(1997), Kedia and Bhagat (1988), and Robichaux and Cooper (1998) add that the
majority of theories of management have a Western orientation.
Rosenzweig (1994) argues that a central concern in scientific research is external
validity. That is, the extent to which a theorized or observed relationship among
variables can be generalized to other settings. Rosenzweig (1994) claims that the
main question should not be, «Are scientific management theories that interest us
valid elsewhere? But how can we best understand management, as it exists around
the world? » (p.37).
Researchers agreed that the measurement of IS success is not an easy task. The
major problem with existing IS success model is that, first, they are very abstract
and don’t rend service to managers who are always searching for more practical
tools and techniques. Second, they are used independently from the organizational
OCTOBER 15-17, 2006
GUTMAN CONFERENCE CENTER, USA
5
6th Global Conference on Business & Economics
ISBN : 0-9742114-6-X
and national contexts. The organizational context has been largely documented by
strategic alignment researchers such as Henderson, Venkatraman and Luftman and
others who built a large body of knowledge that stressed the necessity to align and to
fit IS and organizational strategies. On the other hand, the national context was
always ignored by existing IS success models. Agourram (2004) conducted a cross
cultural research that involves France, Germany, and Canada and found that IS
success is perceived and defined differently in these cultures. Therefore, concludes
the author, existing IS success models cannot be applied as they are in different
national contexts. The following section discusses the alignment concept.
The alignment theory
Alignment of business and information strategies referred to the extent to which
business strategies were enabled, supported, and stimulated by information
strategies. Evidence for alignment was sought in the use of information or
information technology, or both, which provided a comparative advantage to an
organization over its competitors. The importance of aligning the objectives and
strategies of an organization's information systems (IS) group with those of the
broader organization has been recognized for some time (Lederer and Mendelow,
1986; Henderson, 1990; Chan and Huff, 1993; Henderson and Venkatraman, 1996;
Luftman, 1996; Kearns and Lederer, 2000; Reich and Benbasat, 2000; Chan, 2002),
and is usually referred to as strategic alignment. The early research into strategic
alignment tended to be theoretical in nature (Henderson and Venkatraman, 1990)
providing the platform on which later work was developed. Based on these early
OCTOBER 15-17, 2006
GUTMAN CONFERENCE CENTER, USA
6
6th Global Conference on Business & Economics
ISBN : 0-9742114-6-X
developments, a number of dimensions of strategic alignment have emerged. The
social dimension of strategic alignment has been defined as "the state in which
business and IT executives within an organizational unit understand and are
committed to the business and IT mission, objectives, and plans" (Reich and
Benbasat, 2000, p. 82). The intellectual dimension of strategic alignment is defined
as "the state in which a high-quality set of interrelated IT and business plans exist"
(Reich and Benbasat, 2000, p. 82). Put simply, research into the social dimension
tends to "... focus on the people involved in the creation of alignment" whilst the
intellectual dimension of alignment tends to "... concentrate on the content of plans
and planning methodologies" (Reich and Benbasat, 1996). This concentration has
naturally led to research at the CIO/CEO level, at which planning methodologies are
chosen and strategies developed. Furthermore the little research that has been
conducted on the social dimension of alignment has also been focused at this level
(Reich and Benbasat, 1996, 2000; Chan, 2002). There is evidence to suggest that
strategies developed at senior management level are often modified or even
sabotaged at lower levels of the organization during implementation (Davies, 1993;
Alien and Wilson, 2003; Nordstrom and Soderstrom, 2003). Chan and Huff (1992)
have also made the observation that strategy is developed at executive level but
essentially implemented at lower levels of an organization, with the consequence
that greater attention needs to be paid to the dynamics of alignment at lower levels
of the organization.
In studying strategic information systems (IS) alignment, it is important to examine
the associated processes over time rather than viewing alignment as an isolated
OCTOBER 15-17, 2006
GUTMAN CONFERENCE CENTER, USA
7
6th Global Conference on Business & Economics
ISBN : 0-9742114-6-X
event. Organizations experience a series of interdependent changes in business
strategy and Information System strategy which impose to increase the alignment
between them. Alignment is a dynamic and emergent concept because of the
ongoing change in business and information strategies. As business strategy
changes, IS strategy must change in parallel. However, moving from one pattern of
alignment between business and IS strategies turns out to be a rather difficult
proposition. Consequently, organizations struggle to bring IS and business strategies
into alignment and sometimes go through potentially problematic trajectories, where
the business strategy changes but one (or more) of the IS strategy components fails
to change appropriately.
The strategic use of information systems has been a fundamental concern for every
organization. IS are used today to help organization implement and apply their
strategies. They are used as major tools that help organizations gain competitive
advantages over their competitors. Researchers such as Henderson and Venkatraman
(1993) and Luftman (1993, 1998) have always stressed the fact that the realization
of these competitive advantages is conditioned by the strategic use of IS. The
alignment of organizational strategy, objectives and goals with those of IS, enhance
the degree of the benefits these organizations can gain. Venkatraman (1993)
developed the alignment model which is defined in terms of four constructs:
business strategy, information technology strategy, organizational infrastructure and
processes, and information technology infrastructure and processes. The model
explains two fundamental characteristics of strategic management: strategic fit and
function integration.
OCTOBER 15-17, 2006
GUTMAN CONFERENCE CENTER, USA
8
6th Global Conference on Business & Economics
ISBN : 0-9742114-6-X
Greater alignment or "fit" between an organization's business strategy and IS
strategy implies that the information systems are targeted on areas that are critical to
successful business performance. "Alignment between business and IS strategies
heightens managers' awareness and use of information systems, and it enables a firm
to better use IS to help realize its goals and objectives or obtain a competitive
advantage. Weak fit is problematic in two ways. First, it may create competing
motivations for different people in different parts of the organization as with the
tension between strategy and structure. Second, in the absence of a unifying theme
or logic arising from tight fit, organizational players may be confused as to the
appropriate decisions and behaviors in respect to achieving existing operational
performance targets." (Rudy Hirschheim, Rajiv Sabherwal 2001).
. The following section discusses our framework that uses Venkatraman theory.
The new IS success framework
Figure 1 shows our proposed framework. The framework is a process and not a
variant framework. The framework uses existing concepts from the alignment
theory. The framework is a solution to many IS success evaluation problems. Its
benefits will be discussed in the last section.
OCTOBER 15-17, 2006
GUTMAN CONFERENCE CENTER, USA
9
6th Global Conference on Business & Economics
OCTOBER 15-17, 2006
GUTMAN CONFERENCE CENTER, USA
10
ISBN : 0-9742114-6-X
6th Global Conference on Business & Economics
ISBN : 0-9742114-6-X
The framework shows that IS success or IS performance is a process which is made of the
following phases:
1. In phase 1, the strategic orientation of the organization is established. This document
includes the directions and the plans as well as the goals of the organization. Parallel
to the strategic orientation of the organization, IS people establish the strategic
orientation of the information systems. These two processes are done at the same time
and the two must fit each other. (Jerry Luftman, 1993) has extensively reported
research work that discusses this first process.
2. Phase 2 includes two steps that are also executed in parallel. In the first step and after
having a clear idea on the direction of the organization, managers need to document
the expected outcomes of this direction; that is the expected performance of the
organization. At the same time, IS professionals would document the expected
performance of their information systems; that is IS success. IS expected performance
must be aligned with the expected organizational performance. Moreover, the
organizational expected performance determines the IS expected performance. If for
example management consider that the business processes of the organization must be
fluid and employ less people as one expected organizational performance element, this
would push IS professionals to consider maximum automation of business processes
as an element of the expected IS performance.
OCTOBER 15-17, 2006
GUTMAN CONFERENCE CENTER, USA
11
6th Global Conference on Business & Economics
ISBN : 0-9742114-6-X
3. In phase 3, business people implement organizational structures and procedures and
design the new processes to help realize the expected organizational performance. At
the same time, IS people implement the information systems that would help realize
the expected IS performance. The implementation components and logic must fit the
organizational infrastructures and procedures. In this step, and in case information
systems are acquired and not developed in house, the acquired system built-in
assumptions are taken into consideration in the IS implementation step.
4. Phase 4 does not execute right away. It would take few years before we start actions
in phase 4. In this phase, we compare the actual organizational performance with the
expected organizational performance, and we compare the actual IS performance with
the expected IS performance. Comparative research techniques can be used in this
phase. In this phase, we don’t need to search for organizational and IS performance
models and see it we can apply them to our organization. In this phase, the actual
organizational performance is the DEGREE to which the actual organizational
performance matches the expected organizational performance, and the actual IS
performance is the DEGREE to which the actual IS performance matches the expected
IS performance. If case the gaps between the expected and the actual performances is
large, control is sent back to the first phase.
OCTOBER 15-17, 2006
GUTMAN CONFERENCE CENTER, USA
12
6th Global Conference on Business & Economics
ISBN : 0-9742114-6-X
Conclusions
Our framework yields to the following benefits:

It pushes people to think about the outcomes and the results before the actual
implementation of the systems. This is a major exercise that helps convince decision
makers to invest in information systems,

The model requires a continuous fit and alignment between the organization and the
information system which has been proved by research to be an excellent exercise, and

When it comes to evaluating the performance, we don’t need to borrow from very
theoretical model and see if we can apply them to the actual context. The variety of
information systems requires a dynamic IS evaluation system. Moreover, this phase
which used to be the hardest and the most researchable becomes the easiest. We just
compare what we have with what we expected. If the gap is large, actions can be easily
implemented to reduce it and hence ameliorate the actual IS success.
OCTOBER 15-17, 2006
GUTMAN CONFERENCE CENTER, USA
13
6th Global Conference on Business & Economics
ISBN : 0-9742114-6-X
References
1. Alter, S. (2000). The Siamese Twin Problem: A Central Issue Ignored by
Dimensions of Information Effectiveness. Communication of AIS. 2(20), 1-55.
2. Agourram, H, :The Impact of National Culture on The Meaning of Information
Systems Success, D.B.A Dissertation, University of Sherbrooke, May 2004.
3. Alien, D. and Wilson, T., "Vertical trust/mistrust during information strategy
formation", International Journal of Information Management, Vol. 23, pp.
223-37. 2003
4. Chan, Y.E., "Why haven't we mastered alignment? The importance of the
informal organization structure", MIS Quarterly Executive, Vol. 1 No. 2, pp.
97-112. 2002
5. Chan, Y.E. and Huff, S.L., "Investigating information systems strategic
alignment", Proceedings of the 14th International Conference on Information
Systems, Orlando, Florida. 1993.
6. Davies, R., "Making strategy happen: common patterns of strategic success",
European Management Journal, Vol. 11 No. 2, pp. 201-13. 1993.
7. DeLone, W.H and McLean, E.R. (1992). Information Systems Success: The
Quest for The Dependent Variable. Information Systems Research, 3(1), 60-95.
8. DeLone, W.H and McLean, E.R (2003). The Delone and McLean model of
Information systems success: A ten-year update. Journal of Information
Systems Management, 19(4), 9-26.
9. Henderson, J.C and Venkatraman, N, Strategic alignment: Leveraging
Information technology for transforming organizations, IBM Systems Journal,
v32, no 1, 1993
10. Garrity, E.J. and Sanders, G.L. (1998). Introduction to Information Systems
Success Measurement. In E.J. Garrity and G.L, Sanders (dir), Information
Systems Success Measurement (p. 1-12). Idea Group Publishing.
11. Henderson, J.C. , :"Plugging into strategic partnerships: the critical IS
connection", Skan Management Review, Vol. 3, pp. 7-18. 1990.
12. Henderson, J. C., & Venkatraman, N. (1994). Strategic alignment: a model for
organizational transformation via information technology. In T. J. Allen & M.
S. Scott Morton (Eds.), Information technology and the corporation of the
1990s (pp. 202–220). New York: Oxford University Press.
13. Henderson, J.C. and Venkatraman, N.,: "Aligning business and IT strategies",
in Luftman, J.F. (Ed.), Competing in the Information Age: Strategic Alignment
in Practice, Oxford University Press, New York, NY, pp. 21-42. 1996.
14. Hofstede, G. (1993). Cultures and organizations: software of the mind.
London, McGraw-Hill.
15. Ishman, M. (1998). Measuring Information Success at the individual level in
cross-cultural Environments. In E.J. Garrity and G.L, Sanders (dir),
Information Systems Success Measurement. Idea Group Publishing.
OCTOBER 15-17, 2006
GUTMAN CONFERENCE CENTER, USA
14
6th Global Conference on Business & Economics
ISBN : 0-9742114-6-X
16. Jerry N. Luftman, Paul R. Lewis, Scott H. Oldach: Transforming the
Enterprise: The Alignment of Business and Information Technology Strategies.
IBM Systems Journal 32(1): 198-221 (1993)
17. Kanellis, P., Lycett, M. and Paul, R.J. (1998). An Interpretive Approach to the
Measurement of Information System Success: From Concept to Practical
Application In E.J. Garrity and G.L, Sanders (dir), Information Systems
Success Measurement (p. 133-151). Idea Group Publishing.
18. Kearns, G.S. and Lederer, A.L., :"The effect of strategic alignment on the use
of IS-based resources for competitive advantage", Strategic Information
Systems, Vol. 9, pp. 265-93. 2000
19. Kedia, B.L. and Bhagat, R.S. (1988). Cultural Constraints on Transfer of
Technology Across Nations: Implications for Research in International and
Comparative Management. Academy of Management Review, 13(4), 559-571.
20. Lederer, A.L. and Mendelow, A.L.: "Co-ordination of information systems
plans with business plans", Journal of Management Information Systems, Vol.
6 No. 2, pp. 5-19. 1986
21. Luftman, J, Lewis, P.R, Oldach, S.H, Transforming the enterprise: the
alignment of business and information strategies. IBM Systems Journal, v32,
n1, 1993
22. Luftman, J., :"Applying the strategic alignment model", in Luftman, J. (Ed.),
Competing in the Information Age: Strategic Alignment in Practice, Oxford
University Press, New York, NY, pp. 43-69. 1996.
23. Luftman, J. Assessing IT/business alignment, Information Systems in
Management. v20, i4, 2003.
24. Lyftman, J, Enablers and Inhibitors--- Despite Working on Different Levels,
IT and Business Executives Must Communicate Better In Order For
Companies to Experience the Benefits of Technology, Information week, Sept
14, n700, 1998.
25. Markus, M.L and Tanis, C. (2000). The Enterprise System Experience – From
Adoption to Success. In R. Zmud (dir.), Framing the Domain of IT
Management (p. 173-207). Pinnaflex.
26. Nordstrom, T. and Soderstrom, M., "Study of implementing an IT-impregnated
corporate strategy", paper presented at the llth European Conference on
Information Systems, Naples, 16-21 June. 2003
27. Pauleen, D., Evaristo, R., Davison, R., Ang, S., Alanis, M., and Klein (2006),
"Cultural Bias in Information Systems Research and Practice: Are You Coming
from the same place I am?" Communications of the AIS, 17, 354-372.
28. Reich, B.H. and Benbasat, I. , "Measuring the linkage between business and
information technology objectives", MIS Quarterly, Vol. 20 No. 1, pp. 55-81.
1996
29. Reich, B.H. and Benbasat, I., "Factors that influence the social dimension of
alignment between business and information technology objectives", MIS
Quarterly, Vol. 24 No. 1, pp. 81-113. 2000.
30. Robichaux, B.P. and Cooper, R.B. (1998). GSS Participation: A Cultural
Examination. Information and Management, 33, 287-300.
OCTOBER 15-17, 2006
GUTMAN CONFERENCE CENTER, USA
15
6th Global Conference on Business & Economics
ISBN : 0-9742114-6-X
31. Rosenzweig, P.M. (1994). When Can Management Science Research Be
Generalized Internationally? Management Science, 40 (1), 28-39.
32. Rudy Hirschheim, Rajiv Sabherwal : Detours in the path toward strategic
information systems alignment, California Management Review. Berkeley:
Vol.44, Iss. 1; pg. 87, 23 pgs Fall 2001.
33. Seddon, P.B., Staples, S., Patnayakuni, R. and Bowetell, M. (1999).
Dimensions of Information Success. Communication of the ACM. 2(20), 1-40.
34. Shing-Kao, L. (1997). A study of National Culture versus Corporate Culture in
International Management. Dissertation abstract international, Nova Southeastern University.
35. Venkatraman, N., Henderson, J. C., & Oldach, S. : ‘Continuous Strategic
Alignment: Exploiting Information Technology Capabilities for Competitive
Success’, European Management Journal, vol. 11, no. 2, pp. 139 – 149. 1993
OCTOBER 15-17, 2006
GUTMAN CONFERENCE CENTER, USA
16