Download Distributed Computing Environment: Approaches and Applications

Distributed Computing Environment: Approaches and Applications
Li Zeng, Lida Xu, Zhongzhi Shi, Maoguang Wang, Wenjuan Wu
Abstract—In information systems in particular Business
Intelligence systems, for data produced at physically distributed
locations most traditional data mining approaches require data
to be transmitted to a single location for centralized processing
and mining. However, the continual transmission of a large
number of data to a central location must be impractical and
expensive. Thus, distributed and parallel data mining
algorithms and applications were rapidly developed. The paper
surveys the-state-of-the art in approaches and applications of
distributed computing environment. The goal is to summarize a
brief introduction to this field with pointers for further
exploration.
Keywords—Distributed Computing; Distributed and Parallel;
Data Mining
I. INTRODUCTION
istributed computing environment is not far from
reality since its inception. With the rapid development of
Internet, intranets, and wireless sensor networks, et al.
analyzing and mining geographically distributed data sources
require data mining algorithms to design for distributed or
parallel applications, which have attracted tremendous
interest among researcher and practitioners. This paper is
intended as a short introduction to the study of distributed and
parallel data mining, emphasis on the approaches, categories,
and applications.
D
II. THE STATE OF THE ART
Traditional data mining algorithms often make the
assumption that the data is centralized, memory-resident, and
static. However, this assumption is no longer feasible
anywhere. More and more huge amount of data are producing
and storing in local databases every day. In these distributed
circumstances, centralized data mining algorithms must
This work was supported by the Outstanding Overseas Chinese Scholars
Fund of Chinese Academy of Sciences, the National Science Foundation of
China (No. 60435010, 90604017, 60675010), 863 National High-Tech
Program (No.2006AA01Z128), National Basic Research Priorities
Programme (No. 2003CB317004) and the Nature Science Foundation of
Beijing (No. 4052025).
L. Zeng, L. D. Xu, Z.Z. Shi, and M. G. Wang are with the Key Laboratory
of Intelligent Information Processing, Institute of Computing Technology,
Chinese Academy of Sciences, and Graduate University of Chinese
Academy of Sciences (e-mail: [email protected]).
W. J. Wu is with the School of Information, Renmin University of China.
1-4244-0991-8/07/$25.00/©2007 IEEE
waste lots of I/O resources, and excessive communication
overhead to transmit huge amount of data. Another difficulty
is to update and mine incrementally immense amounts of data
for centralized data mining algorithms when the data is
dynamic or immediate-change. Several inherent fields where
large amount of data are stored in decentralized structure
include WWW, multinational Corp, sensor network,
scientific field, manufacturing industries troubleshooting [1],
and health care where daily diagnostic data and medical
information stored by hospital management information
system.
All these have prompted the need for distributed and
parallel data mining algorithms. Either distributed or parallel
data mining is still a particular process involving the
application of specific algorithms, that is, distributed or
parallel algorithms, for extracting pattern from distributed
and large-scale data. In the meantime, the additional steps in
the data mining process can not be ignored, such as data
preparation, data reduction, data cleaning, and data
transformation.
III. APPROACHES AND APPLICATIONS
A. Distributed mining systems
A. Most distributed data mining projects are interested in
discovering hidden patterns in a complete data set that is
essentially partitioned and physically distributed at different
data sources. That is, the distributed algorithms perform
further analysis based on local results to achieve global
results. Different strategies are possible depending upon the
data, its distribution, and the resources availability. The most
representative example of DDM is distributed association
rule mining [16] and distributed decision tree [17]. Kargupta
et al. [21, 22] first proposed collective data mining which
works on vertically partitioned data and combines immediate
results from local data sources. Several systems have been
developed for distributed data mining. Perhaps the most
outstanding systems include JAM system developed by
Stolfo et al. [26], Kensington system developed by Guo et al.
[30], and Papyrus developed by Bailey et al. [31] JAM
employs meta-learning, while Kensington employs
3240
knowledge probing. Papyrus is designed to support different
data, task and model strategies. In contrast to JAM, Papyrus
can not only move models from node to node, but can also
move data from node to node, when that strategy is desired.
Also, Papyrus is a specialized system which is designed for
clusters, meta-clusters, and super-clusters, while most
systems are designed only for mining data distributed over the
Internet.
Distributed data mining can choose to move data, to move
intermediate results, to move predictive models or to move
the final results of a data mining algorithm. Distributed data
mining systems which employ local learning build models at
each site and move the models to a central location. Systems
which employ centralized learning move the data to a central
location for model building. Systems can also employ hybrid
learning, that is, strategies which combine local and
centralized learning. Most distributed data mining systems all
employ local learning. [31]
In respect of task strategy, distributed data mining systems
can choose to coordinate a data mining algorithm over several
sites or to apply data mining algorithms independently at each
site. With independent learning, data mining algorithms are
applied to each site independently. With coordinated learning,
one or more sites coordinate the tasks within a data mining
algorithm across several sites.
Several different methods have been employed for
combining predictive models built at different sites. The
simplest method is to use voting and combine the outputs of
the various methods with a majority vote. Meta-learning
combines several models by building a separate meta-model
whose inputs are the outputs of the various models and whose
output is the desired outcome [26, 30]. Multiple models, or
what are often called ensembles or committees of models,
have been used for quite a while in centralized data mining.
Methods for combining models in an ensemble include
Bayesian model averaging and model selection [32], partition
learning [33].
B. Peer-to-peer based
Communication bandwidth is often a scarce resource for
data mining in decentralized circumstances. Approaches
collecting all local data at one given node (or centre node)
will absolutely result in communication bottlenecks or
messages implosion at somewhere. Remarkably, peer-to-peer
(P2P) network provide a natural and well-suited platform for
such distributed data mining, as well as information
dissemination, file sharing, Voice over IP, and e-commerce.
As mentioned, DDM has introduced distribution version of
many traditional data mining algorithms, such as association
rule mining, clustering, and classification. However, most
DDM efforts assume a stable network and data, and they can
not be applied directly to dynamic topology such as node
failure, link failure, and immediate join-leave behavior. In
distributed data mining over P2P network, most recent
researches focused on developing local algorithms for
primitive operations such as average, sum, max, random
sampling, quantize, and other aggregate functions.
Wojtek et al. [8] used the newscast model to estimate the
mean value of distributed data, and provided the theoretical
and experimental evidence for its feasibility to distributed
data mining over P2P networks. They used empirical
probabilities instead of guaranteed correctness. Another
approach conducted by David [9] relied on uniform
gossip-based randomized algorithms for computing
aggregate information.
Mortada et al. [7] proposed a class of asynchronous
distributed algorithms, which is a Laplacian-based approach
to averaging the local inputs of nodes on a P2P network.
Remarkably, their algorithm did not rely on synchronization,
coordinated parameter values, and the apriori knowledge of
global topology. Souptik et al. [2, 5] presents an overview of
efforts to develop DDM application and algorithms in P2P
networks, and describe both exact and approximate local P2P
data mining algorithms that work in a decentralized manner.
Ran et al. [3, 4] introduced a local algorithm for computing
the majority vote, and proposed an algorithm for monitoring
K-means clustering in P2P networks. Similar work by Brian
et al. [6] assumed a centralized coordinator site and a
hierarchical topology for global conflict resolution, and
proved that distributed communication is only necessary on
occasion.
C. Internet and gird computing based
Internet and Grid computing seek to change the way we
tackle complex problems. Internet provides not only huge
volume of distributed data, but also plenty distributed
computing resources to construct a supercomputer than ever
before. Complex and data-intensive computing such as
bioinformatics data mining, climate prediction models or
airplane computer-aided design systems used to run only on
supercomputer. Recently, however, rapid improvements in
Internet, distributed algorithms and the increase in speed and
memory of the PC machine, lead to consider more
decentralized approaches to the problem of computing power.
There are over 400 million PCs around the world, and most
are idle much of the time. Internet computing exploits these
3241
idle workstations and PCs to create powerful distributed
computing systems with global reach. It has been possible to
undertake many significant projects required supercomputer
capabilities before on normal and cheap equipments. Some
examples of popular internet computing projects include:
z Models@Home[14]
Distributed computing in bioinformatics;
z SETI@home [12,15] setiathome.ssl.berkeley.edu
z FightAIDSatHome [13] www.fightaidsathome.org
z ClimatePrediction [11] www.climateprediction.net
z Casino 21: www.climate-dynamics.rl.ac.uk
Projects Models@Home [14] which is distributed
computing in bioinformatics using a screensaver based
approach, as done by SETI@Home, [12, 15] the world's
largest distributed computing project to detect intelligent life
outside Earth, have demonstrated the principles and
techniques of distributed computing which provides a
mechanism for engineer or researchers to tap into the unused
resource of millions of distributed PCs. Other similar projects
are FightAIDSatHome [13] which is the first biomedical
distributed computing project ever launched. It is run by the
Olson Laboratory at The Scripps Research Institute in
California. FightAIDSatHome uses scattered computer's idle
resources to assist fundamental research in discovering new
drugs, building on growing knowledge of the structural
biology of AIDS. These projects above require the
participant’s machines to download client, data and models
that executed on local machine, and then upload a result of
specified form to server. The ClimatePrediction project
developed by Stainforth et al. [11] takes the distributed
computing paradigm a step further by inviting participants to
download a full-scale climate model and run it locally to
simulate 100 years of the Earth’s climate. Each participant
independently carries out a subset of result which is then
combined with other participant’s results to ensemble a
global climate prediction result. Obviously, low
communication overhead was generated during the running
as each participant has the complete model.
A grid is a geographically distributed computation
infrastructure composed of a set of heterogeneous machines
that users can access via a single interface. Grids provide
common resource-access technology and operational services
across widely distributed boundary. Grid computing therefore
is described as seamless, pervasive access to resources and
services. It plays a significant role in providing an effective
computational support for application of distributed data
mining. Grid computing has been proposed as a novel
computational model, distinguished from conventional
distributed computing by its focus on large-scale resource
sharing, innovative applications, and, in some cases,
high-performance orientation. Today grids can be used as
effective infrastructures for distributed high-performance
computing and data processing [24, 25]. Cannataro et al. [24]
discussed how to design and implement data mining
applications by using the knowledge grid tools starting from
searching grid resources, composing software and data
components, and executing the data mining process on a grid.
In addition, Luo [27] proposed a two-phase scheduling
framework including external scheduling and internal
scheduling in two-level grid environment, upon which a
system named DMGCE (Data Mining Grid Computing
Environment) has been developed and implemented.
D. Meta-learning, agent based and Service-oriented
Meta-learning, loosely defined as learning from learned
knowledge, is another technique recently developed that
deals with the problem of computing a “global” model from
large and inherently distributed databases. The goal of
meta-learning is to compute a number of independent model
(classifiers) by applying learning programs in parallel, that is,
without transferring or directly accessing the data sites. Stolfo
et al. [26] developed such a system so-called JAM (Java
Agent for Meat-learning) mentioned above to achieved the
goal. JAM is a distributed, scalable and portable agent-based
data mining system that provides a set of meta-learning
agents for combining multiple models that were learned at
different sites. As long as a machine learning program is
defined and encapsulated as an object conforming to the
interface requirements, it can be imported and used directly in
JAM. [26] This plug and play characteristic makes JAM truly
powerful and extensible data mining facility, although using a
centralized approach to maintain the global configuration
may be an obvious sequential bottleneck. JAM is so far the
most representative system to mine distributed databases by
means of meta-learning and intelligent agents.
Service-oriented architecture (SOA) provides seamless
integration of self-contained computational services that can
communicate and coordinate with each other to perform
goal-directed computation. The concept has blossomed in the
past few years due to the development of Web service related
standards and technologies, including WSDL, Universal
Description, Discovery, and Integration (UDDI), and SOAP.
WEKA4WS [27], an open-source framework derived from
the Weka toolkit for supporting distributed data mining on
Grid environments, exposes all the data mining algorithms
3242
provided by as WSRF-compliant services (WSRF: Web
Services Resource Framework), which enable important
benefits, such as dynamic service discovery and composition,
standard support for authorization and cryptography, and so
on. Grid-enable Weka [28] is another proposed system found
on Weka, which is a widely used toolkit for machine learning
and data mining written in Java. In Grid-enabled Weka,
execution of these tasks can be distributed across computers
in an ad hoc Grid environment. Tasks that can be executed
using Grid-enable Weka include building a classifier on a
remote machine, testing a classifier on a dataset, and
cross-validation. Grid-enable Weka modifies the Weka
toolkit to enable the use of multiple computational resources
when performing data analysis. WekaG [29] is another
system derived from the Weka. It’s based on client/server
architecture. The server side provides a set of services that
implement the functionalities and the process of the different
data mining algorithms. The client side is responsible for
communicating with server side and offering user interface.
FAEHIM
(Federated
Analysis
Environment
for
Heterogeneous Intelligent Mining) is a Web Services-based
toolkit for supporting distributed data mining. This toolkit
consists of a set of data mining services, a set of tools to
interact with these services, and a workflow system used to
assemble these services and tools.
IV. CONCLUSIONS
Both distributed computing environment and parallel
computing environment can speed up the data mining
progress, however, they are still distinct in several unaware
ways. Distributed data mining often applies the same or
different mining algorithms to tackles local data, to exchange
or communicate among multiple process units, and then
combining the local pattern discovery by local data mining
algorithms from local databases into a global knowledge. In
this case the discovered knowledge is often different from the
knowledge discovered by sequentially applying the data
mining algorithms to the entire datasets. The accuracy or
efficiency of distributed data mining is somewhat difficult to
predict. Because it depends on data partitioning, task
scheduling, and global synthesizing. In contrast to distributed
data mining, in principle a parallel data mining algorithm
discovers the knowledge as same as the knowledge found by
its sequential algorithm due to applying the global parallel
algorithm on the entire data sets. Its accuracy may be more
guaranteed than the distributed data mining in some case.
Distributed
computing
environment
enables
geographically distributed science and engineering teams to
collaborate in new ways. Despite recent advances in
Peer-to-Peer and service-oriented, grid computing,
high-performance computing, issues on distributed and
parallel computing such as limitation on data transmission
bandwidth, skew distribution, very-large data size, privacy
preserve still need serious and immediate attention. For
example, the execution environments of geographically
distributed applications are far less deterministic than those of
locally centralized. In addition, distributed applications
become highly complex when large-scale and skewed data
distributes in unpredictable locations in which identifying and
correcting performance bottlenecks exposed may not be
useful repeatedly, because the network bandwidths and
computing resources often vary from one place to another.
The work here is only a part of related research piece.
Although we are convinced that distributed data mining or
parallel data mining is the way to go, it will still be quite a
journey.
REFERENCES
[1]
R. Heider, Troubleshooting CFM 56-3 Engines for the Boeing
737—Using CBR and Data-Mining, Spinger-Verlag, New York, vol.
1168, pp. 512–523, 1996. Lecture Notes in Computer Science.
[2] S. Datta, K. Bhaduri, Chris Giannella and Hillol Kargupta, IEEE
Internet Computing, IEEE Computer Society, July 2006, pp.18-27
[3] R. Wolff and A. Schuster, “Association Rule Mining in Peer-to-Peer
Systems,” IEEE Trans. Systems, Man, and Cybernetics, Part B, vol. 34,
no. 6, 2004, pp.2426–2438.
[4] R. Wolff, K. Bhaduri, and H. Kargupta,“Local l2-Thresholding Based
Data Mining in Peer-to-Peer Systems,” Proc. 2006 SIAM Conf. Data
Mining (SDM06), SIAM Press, 2006, pp. 430-441.
[5] S. Datta, C. Giannella, and H. Kargupta,“K-Means Clustering over
Large, Dynamic Networks,” Proc. 2006 SIAM Conf. Data Mining
(SDM 06), SIAM Press, 2006, pp. 153–164.
[6] B. Babcock and C. Olston, “Distributed Top-K Monitoring,” Proc.
ACM SIGMOD 2003 Int’l Conf. Management of Data, ACM Press,
2003, pp. 28–39.
[7] M. Mehyar, D. Spanos, J. Pongsajapan, et al., “Distributed Averaging
on a Peer-to-Peer Network,” Proc. IEEE Conf. Decision and Control,
IEEE CS Press, 2005.
[8] W. Kowalczyk, M. Jelasity, and A. Eiben,“Towards Data Mining in
Large and Fully Distributed Peerto-Peer Overlay Networks,” Proc. 15th
Belgian-Dutch Conf. Artificial Intelligence (BNAIC 03), Univ.of
Nijmegen Press, 2003, pp. 203–210.
[9] D. Kempe, A. Dobra, J. Gehrke, “Gossip-based computation of
aggregate information”, Proc. 44th Annual IEEE Symposium on.
Foundation of Computer Science, IEEE CS Press, 2003, pp.1-10
[10] I. Flockhart, N. Radcliffe, “GA-MINER: Parallel Data Mining with
Hierarchical Genetic Algorithms”, Final Report. University of
Edinburgh, 1995
[11] D. Stainforth, J. Kettleborough, M. Allen, et al. “Distributed
Computing for Public-interest Climate Modeling Research”, IEEE
Computing in Science & Engineering, 2002, pp.82-90
3243
[12] E. Korpela et al., “SETI@home: Massively Distributed Computing for
SETI,” Computing in Science & Eng., vol. 3, no. 1, Jan./Feb. 2001, pp.
78–83.
[13] Fight AIDS at Home Project, http://fightaidsathome.scripps.edu/
[14] E. Krieger, G. Vriend. “Distributed Computing in Bioinformatics using
a screensaver based approach”, Oxford Journals, Oxford University
Press, 2007
[15] D. Anderson, J. Cobb, E. Korpela, et al. “SETI@home: an experiment
in public-resource computing”, Communications of the ACM,
2002,45(11):56-61.
[16] M.J. Zaki, “Parallel and Distributed Association Mining: A Survey,”
IEEE Concurrency, vol. 7, no. 4, 1999, pp. 14–25.
[17] K. Kubota, A. Nakase, H. Sakai, et al. “Parallelization of Decision Tree
Algorithm and Its Performance Evaluation”, High Performance
Computing in the Asia-Pacific Region'00, The 4th International
Conference, Volume 2, 2000
[18] The Predictive Model Markup Language Version 0.9, The Data Mining
Group, 1998, http://www.dmg.org.
[19] R. Grossman, S. Bailey, A. Ramu, et al. The Management and Mining
of Multiple Predictive Models Using the Predictive Modeling Markup
Language (PMML), Information and System Technology, Volume 41,
pages 589-595, 1999
[20] A. L. Prodromidis, P. K. Chan, “Meta-learning in distributed data
mining systems: Issues and Approaches”. Book on Advances of
Distributed Data Mining, editors Hillol Kargupta and Philip Chan,
AAAI press, 2000
[21] H. Karagupta, E. Johnson, E. R. Sanseverino, et al. Scalable Data
Mining from Vertically Partitioned Feature Space Using Collective
Mining and Gene Expression Based Genetic Algorithms, KDD-98
Workshop on Distributed Data Mining, 1999.
[22] H. Kargupta et al., “Collective Data Mining: A New Perspective
towards Distributed Data Mining,” Advances in Distributed and
Parallel Knowledge Discovery, H. Kargupta and P. Chan, eds.,
MIT/AAAI Press, 2000, pp. 133–184.
[23] M. Cannataro, C. Comito, “A Data Mining Ontology for Grid
Programming”, In Proc 1st Workshop Semantics in Peer-to-Peer and
Grid Computing, 2003, pp 113–134.
[24] M. Cannataro, A. Congiusta, A. Pugliese, et al. “Distributed Data
Mining on Grids: Services, Tools, and Applications”, IEEE Trans.
Systems, Man, and Cybernetics, Part B, vol. 34, no. 6, 2004,
pp.2451–2466.
[25] I. Foster and C. Kesselman, “The anatomy of the grid: Enabling
scalable virtual organizations. S. Tuecke,” Int. J. Supercomput.
Applicat., vol. 15, no. 3, 2001.
[26] S. Stolfo, A. L. Prodromidis, and P. K. Chan, JAM: Java Agents for
Meta-Learning over Distributed Databases, Proceedings of the Third
International Conference on Knowledge Discovery and Data Mining,
AAAI Press, Menlo Park, California, 1997.
[27] D. Talia, P. Trunfio, and O. Verta, “Weka4WS: A WSRF-Enabled
Weka Toolkit for Distributed Data Mining on Grids”, PKDD 2005,
pp.309-320, 2005
[28] R. Khoussainov, X. Zuo and N. Kushmerick “Grid-enabled Weka: A
Toolkit for Machine Learning on the Grid”, ERCIM News No.59, Oct
2004.
[29] M. Perez, A. Sanchez, P. Herrero, et al. “Adapting the Weka Data
Mining Toolkit to a Grid based environment”, 3rd Atlantic Web
Intelligence Conf. 2005.
[30] Y. Guo, S. M. Rueger, J. Sutiwaraphun, et al., “Meta-Learning for
Parallel Data Mining”, in Proceedings of the Seventh Parallel
Computing Workshop, pp1-2, 1997
[31] R. Grossman, S. Bailey, A. Ramu, et al., “The Preliminary Design of
Papyrus: A System for High Performance, Distributed Data Mining
over Clusters”, in Advances in Distributed and Parallel Knowledge
Discovery, H. Kargupta and P. Chan, editors, AAAI Press/The MIT
Press, MenloPark, California, 2000, pages 259-275.
[32] A. Raftery, D. Madigan, and J. Hoeting, “Bayesian Model Averaging
for Linear Regression Models”, Journal of the American Statistical
Association, Volume92, pages 179–191, 1996.
[33] R. Grossman, H. Bodek, D. Northcutt, et al., Data Mining and
Tree-based Optimization, in the Proceedings of the Second
International Conference on Knowledge Discovery and Data Mining,
AAAI Press, MenloPark, California, page 323-326, 1996.
3244