Download Proposal of knowledge discovery platform for big data

Mathematics and Computers in Sciences and Industry
Proposal of knowledge discovery platform for big
data processing in manufacturing
Lukas Spendla, Lukas Hrcka, Pavol Tanuska
Faculty of Materials Science and Technology
Slovak University of Technology
Trnava, Slovakia
[email protected], [email protected], [email protected]
Abstract—In the proposed paper, we described the approach
to building Data Lake based knowledge discovery platform. The
proposal is focused on integrating Data Lake based storage, built
on Hadoop framework and NoSQL systems, into traditional data
warehouse discovery platform, preserving the well proven and
robust data warehouse decision support and analytic tools. The
proposed knowledge discovery platform processes data from all
hierarchical control levels in manufacturing and can be used to
address the main manufacturing issues in the knowledge
discovery domain
II. HIERARCHICAL CONTROL MODEL
Current information and control systems primarily employ
hierarchical (pyramid) architecture integrated as a whole with
elements of physical and logical distribution thus providing
open and scalable solutions. Many of hierarchical control
systems are built as multiprocessor control systems enabling
both horizontal and vertical communication. Intelligent
features arising from deploying sensors and actuators have
been intensely utilised recently with direct hierarchical
relations being transformed into network relations. Emerging
tendencies such as connecting previously independent systems
leading to new behaviour attributes are strongly reflected in
current systems.[1] [2]
Keywords—knowledge discovery; data warehouse; data lake;
hadoop; manufacturing; hierarchical control
I. INTRODUCTION
The current trend in manufacturing is marked by the large
increase in amount of data, originating from the field level of
hierarchical control. This increase is mainly due to
implementation of new automation technologies and machines
based on internet of things concept, a part of Industry 4.0,
enabling direct communication with upper control levels.
Each parameter of manufacturing process is represented by
a large amount of production data applicable in information or
control systems at various levels. Despite the fact that most of
manufacturing companies gather these data, they are not
further used as information or knowledge in decision support
process.
This was one of the reasons resulting in the urgent need for
storing and processing large quantities of data and yet, it will
be possible to work with them flexibly. These needs are
reflected by current big data technologies based on NoSQL
systems and Hadoop framework. However, integrating these
new technologies into a company structure disrupts the wellestablished architecture based on data warehouses. This
structure represents proven and robust solution from the
company decision support point of view. Therefore, these new
technologies must be integrated into manufacturing companies
in a way allowing users to preserve the currently used
solutions based on the data warehouse concept, while
exploiting the advantages of the deployed NoSQL or Hadoop
solution.
ISBN: 978-1-61804-327-6
Fig. 1. Hierarchy of the industrial control system
Therefore, also the process control is nowadays being
implemented deploying control systems with a hierarchical
structure. The model of complex control process, so-called
pyramid model, is shown in Fig. 1. [1] [2]
At all levels of the production process control model, large
amount of data are produced, collected and stored often
resulting in data redundancy. Still, the fact that different levels
produce different types of data needs to be respected.
150
Mathematics and Computers in Sciences and Industry
III. PROBLEMS INDENTIFICATION IN MANUFACTURING
A. Control Level
Technology (control) level is the lowest layer of the
pyramid model of hierarchical process control and constitutes
a basic interface with production. It consists of production
lines, machines and equipment, which include integrated
sensors and actuators, communicating using technology
network with control computers, mainly with PLC
(Programmable Logic Controller).[3] At this level, collecting
and primary processing of technological parameters is carried
out. Data are collected in real-time, with different sampling
times, which results in collection of large amounts of data to
be saved or archived for further implementation. The cyclic
data collection traditionally used to collect data without
transmitting the signal differences leads to redundancy.
Collected data represents also a part of the noisy industrial
environment and contains errors stemming technological
information processing. Collected data are often noisy from
manufacturing environment and contain errors primary
processing of technological information. Removing these
adverse conditions, filtering the required data and its
subsequent validation are the tasks necessary to be carried out.
From the application area point of view, manufacturing
process does not focus only on production itself, but extends
and integrates data from all hierarchical control levels. For
effective process control and management, not only
production data are required, but also data including
customers, resources, and suppliers’ information from the
upper hierarchy control levels are inevitable.
Manufacturing can therefore generate big amounts of data
suitable for application of data mining and / or knowledge
discovery process and might provide suitable means to deal
with the problems arising in the field of production systems.
Existing approaches in manufacturing utilising data mining
techniques can be divided into five main application areas [4]:
 Quality analysis of products to correlate output quality
and system parameters, esp. machine settings, in order
to identify causes for deteriorating product quality
 Failure analysis of production resources, esp. machines,
to analyse causes of errors and prevent break downs in
the future
B. Production (Supervisory )Level
Supervisory level of pyramid model is a higher
(intermediate) level of the hierarchical process control, which
is alternatively called as SCADA / HMI (Supervisory Control
and Data Acquisition / Human Machine Interface) level. It is
used for primary collecting and integrating process data, for
monitoring, visualization, evaluation and direct interference in
the process. [3] At this level of control, system data are mostly
stored in SCADA systems having an apparent purpose:
processing alarm, monitoring mixing ratios, batch processes
and data history, as well as archiving the operational variables.
All other data stored at higher levels of pyramid model of
hierarchical process control.
 Maintenance analysis to enhance the availability of
production resources, e. g., by optimized maintenance
planning
 Production planning and scheduling analysis to
improve planning quality, e. g., by a higher capacity
utilisation of production resources
 Strategically planning and scheduling analysis to
improve customer relationship and increase sales, e.g.
by identification of customer behaviour
Each of these application areas covers multiple applied
techniques and also different approaches form hierarchical
model point of view. Therefore, it is impossible to identify a
specific technique and an approach that needs to be applied to
optimize or solve issues in selected application areas. It should
be noted that all application areas span across multiple
hierarchical control levels and therefore, it is impossible to
assign them to a one specific level. Due to this fact, we
identified the dependencies based on various research applied
in the manufacturing area utilising data mining techniques.
MES systems, typical representatives of information
systems at the production level, are responsible for obtaining
and collecting data from manufacturing. Obtained data are
processed and in real-time stored in an aggregated form into
data storage (mostly transactional SQL database). The data
saved in the database in a structured form containing the
current value of the variable, validity and timestamp (VTQ =
Value / Time / Quality).
Generally, the data mining approaches used in main
application areas in manufacturing mostly utilise manual
processing of specific data collection to analyse specific
manufacturing process aspects in various manufacturing
specific cases, e.g. machines, equipment, products, quality,
etc. Most of the approaches can be integrated into real time
support systems; however they mainly focus only on the
approaches and methods themselves. [5][6][7]
C. Management Level
Management level of pyramid model covers the previous
levels. It consists of database resources for higher levels of
control, management information system and tools for internet
visualization. It is the level of planning and management. At
this level, the data are archived and processed and long-term
decisions for production are accepted. [3] At the management
level of the pyramid model, data are not directly collected
from manufacturing process, but are transferred in the
transaction mode from information system of a real-time
interface using the ERP Gateway. As the ERP does not
operate continuously, continuous data transfer is carried out
utilising ERP Gateway. Therefore, huge volumes of
predominantly structured data arise in ERP systems.
ISBN: 978-1-61804-327-6
IV. CURRENT STATE OF KNOWLEDGE DISCOVERY IN
MANUFACTURING AREA
At present, the application of data mining and knowledge
discovery is very broad. However, according to recent studies
[8][9], data mining is mostly employed in the fields like
marketing, consumer analytics, finance, telecommunication,
151
Mathematics and Computers in Sciences and Industry
The main advantage of this approach is robustness and
stability, due to the widespread deployment and long term real
world experience in various enterprise areas. This factor is
important due to the fact that the company KPIs, affecting the
company management and business bottomline, are based on
data from the data warehouse provided by the data marts.
insurance, health care etc. The usage of data mining in
manufacturing is usually between 9 and 10 percent. Major part
of this share is created by large international industries.
The weakness of the current form of manufacturing is
often in the subjective perception of global production aims
(profitability, production efficiency, plant productivity and
product quality), frequent and often unforeseen variations in
both manufacturing parameters and variables, the subjective
decision making, and also in the vast amount of unstructured
data provided by various information systems. [10]
The data mining and knowledge discovery process is
usually based on data warehouse integrating all data required
in this process. This concept of analytic environment is
captures in Fig. 2.
Multiple systems operate at various hierarchical control
levels, each using its own databases mostly independent from
each other. [11] Very often there is no defined relationship
between data in each system, e.g. manufactured product
identifier has different numbering schema and order across
control level data, SCADA, MES and ERP data. Therefore, it
is necessary to integrate these data together to perform
analytic reporting and knowledge discovery process.
In most large manufacturing companies, data warehouse is
used to store the data from various company systems. Data
integrated in the data warehouse serve as the basis for decision
support, through the corresponding data mart or decision
support tools [12]. Therefore, the ETL process transforming
data into data warehouse for further use in business
intelligence and analytic tools is extremely important.
Fig. 2. Current state of knowledge discovery platform in manufacturing
The knowledge discovery process in company analytic
environment is usually performed according to company
methodology. This methodology can be specific for each
individual company. However in recent years, more and more
companies are starting to adopt the CRISP-DM methodology.
Most companies however, don’t adopt this methodology
strictly. Due to this fact, the methodology is usually modified
to suit the company needs. Since the knowledge discovery and
data mining methodology is part of company know-how and
not publicly accessible, it is impossible to generalise it as a
whole.
The obtained data are accessed through data marts, created
through ETL process from the data warehouse, providing
organised view on the data from various business perspectives.
[12] Data marts for various company specific aspects, like
management, manufacturing, quality, etc. provide basis for
decision making process.
It should be noted, that the data in data marts are not
always integrated in the data warehouse itself. In other words,
the data warehouse and data mart data can be separated. In
order to obtain the complex view on company data for
reporting and knowledge discovery, the data stored in data
marts must supplement the company data stored in data
warehouse.
The discovery platform must be set over all company data
stored in data marts. If data warehouse does not integrate all
data, discovery platform must be able to obtain and process
them. In companies the discovery platform is mostly used for
KPI based reporting and quality assurance [13]. The main
advantages of data mining and knowledge discovery have still
not been fully exploited.
A variety of knowledge discovery and analytic tools used
in discovery platform is available. All major software tools
provide connectors for relational databases and data
warehouses. However most of the data operations must be
handled by tools themselves, whether it is a standalone
workstation or a client-server solution.
Fig. 3. Relationship between different phases of CRISP-DM [15]
ISBN: 978-1-61804-327-6
152
Mathematics and Computers in Sciences and Industry
However, most of the companies preserve the continuity of
the main phases of the CRISP-DM model, as shown in Fig. 3.
Hence, the CRISP-DM methodology is, with a certain degree
of abstraction, applicable in any manufacturing industry. [14]
V. KNOWLEDGE DISCOVERY APPROACH PROPOSAL IN
MANUFACTURING AREA
The common analytics environment at most big
manufacturing companies includes a data warehouse, or
collection of federated data marts, which house and integrate
the data for knowledge discovery process. This includes
various ranges of analysis function and business intelligence
and analytics tools enabling decision support utilising ad hoc
queries, dashboards and data mining.
Data mining is now used in many different areas in
manufacturing engineering to extract knowledge for the use in
predictive maintenance, fault detection, design, production,
quality assurance, scheduling, and decision support systems.
Data can be analysed to identify hidden patterns in the
parameters controlling manufacturing processes or to
determine and improve the quality of products. It clearly
indicates data mining can be used in many different
application areas of manufacturing. [16]
Large manufacturing companies with large investments in
their data warehouses have neither the resources, nor the will
to replace the existing environment that works well and do
what it was designed to. The majority of large companies
utilise a coexistence strategy combining the best of data
warehouse and analytics environment, with the new trends in
big data solutions.
However, the manufacturing process brings a huge amount
of data stored in databases containing enormous number of
records. Every record has attributes needed to be explored to
discover useful information and knowledge. All of this factors
clearly demonstrates, that the choosing the right methods is
crucial to successful discovery of knowledge. [17] Nowadays,
there are a lot of types of methods, techniques and algorithms
used for data mining process. Kdnuggets in [18] carried out a
survey asking companies utilising data mining algorithms in
their company. According to this research, the most used
algorithms are: decision trees (rules), regression, clustering,
statistics, visualization and time series.
Many companies want to continue to rely on data
warehouses for standard BI and analytics reporting, including
sales reports, customer dashboards, risk history , etc. The
coexistence strategy allows the companies to use data
warehouse with its standard workload and storing historical
data to establish robust traditional business intelligence and
analytics tools. [19]
Despite the robustness of traditional business intelligence
and analytics tools, semi-structured and unstructured data
from the data collection process do not fit well into traditional
data warehouses. Furthermore, data warehouses may not be
able to handle the processing of frequently or even continually
updated big data sets. As a result, organisations are looking for
possibilities to collect, store and analyse big sets of data.
Newer class of technologies including the Hadoop
framework and NoSQL systems are often deployed for this
task. [20] In some cases, these technologies are being used as
staging areas for data before they are transformed into a data
warehouse, often in summarised form that is more suitable for
relational structures. Big data solution vendors are
increasingly pushing the concept of Hadoop Data Lake that is
used as central repository for raw data streams present in the
company. [21]
This coexistence approach, incorporating Data Lake as the
central repository serves as a baseline for our knowledge
discovery approach in manufacturing area, captured in Fig. 5.
The proposed knowledge discovery analytic environment
is based on common data warehouse approach. The data
warehouse integrates various data from heterogeneous systems
across various hierarchical control levels. These
heterogeneous data are extracted, transformed and integrated
into a data warehouse using ETL process. This approach is
mostly suitable for discontinuous and non-real-time data from
higher hierarchical control levels.
Fig. 4. Survey of methods and algorithms usage in data analysis [18]
Many of the methods are exploitable in several areas, but it
is very important to perform detailed analysis of the tasks to
be solved, because methods are not universally applicable, but
depend on the problem to be solved.
ISBN: 978-1-61804-327-6
The data marts, created from the data warehouse data,
provide organised view on data from business unit perspective
(like management, manufacturing, quality, etc.) and provide
basis for decision making process in selected area. The data
loaded into data marts needs to be extracted and transformed,
to create the data structure suitable for further use.
153
Mathematics and Computers in Sciences and Industry
Fig. 5. Knowledge discovery analytic environment proposal
The ETL process transforming data from a Data Lake into
a data warehouse is performed only for data not transferred
into data warehouse directly. The main use of this particular
ETL process is loading the manufacturing data from the field
level stored in a Data Lake into a data warehouse. [20]
The Data Lake, based on Hadoop framework, provides
central data storage for raw manufacturing data. The Data
Lake extracts and loads data from heterogeneous database
systems and stores them in a raw (original) form. Therefore
the data does not have to be transformed to be stored in the
Data Lake. In the Hadoop Data Lake cluster, subsets of the
data can be analysed using batch query tools, stream
processing software and SQL on Hadoop technologies that run
interactively or using ad hoc queries in SQL.
One of the biggest issues in obtaining manufacturing data
is the way of collecting and processing data from the field
level of hierarchical control. All these data serves as a basis
for decision support at higher hierarchical control levels, the
used field level data are usually aggregated into data more
suitable for particular decision support task. Therefore the data
suitability for business intelligence or analytic tools is very
limited.
The discovery platform in this environment is built on the
data integrated in the Data Lake. Due to the use of Hadoop
cluster, this environment provides higher performance when
working in big data sets than the traditional data warehouse.
Big advantage is also the availability of raw data from the
manufacturing that cannot be easily stored in the data
warehouse.
However with the increasing number of sensors connected
to network in production chain, it is easier to collect the
production chain data. This feature is provided by Field Level
Bus.
The offer of tools for discovery platform over Hadoop
cluster is not very wide. Most of the standard knowledge
discovery tools cannot connect to Hadoop cluster using SQLon-Hadoop solutions. However, this way most of the data
manipulation operations must be performed by the tool itself
and not by the Hadoop cluster. In order to enable utilising the
full potential of data manipulation performance of Hadoop,
software manufacturers offer add-ons or software solutions
able to perform selected sets of operations and algorithms
directly in a Hadoop cluster. This approach is preferred, since
the discovery platform must be able to process the big sets of
collected data.
ISBN: 978-1-61804-327-6
The Field Level Bus collects data from various industrial
control systems, and loads them into the Data Lake storage.
Due to the big amounts of periodic or continuous data
collected at this level, Data Lake builds on the Hadoop cluster
technology which is the most suitable solution to store the raw
field level data.
Main task of Field Level Bus is preparing the data which
is a fundamental step for the further use of field level data, as
the data can be collected from various, sensors, PLC, devices,
systems, etc.
154
Mathematics and Computers in Sciences and Industry
Data collected at the field level can also be inconsistent.
Therefore, transforming the collected data into cleaned forms
storable in Data Lake storage is necessary. This Field Level
Bus addresses the need of data analysis aimed at cleaning the
raw data. [22]
[4]
[5]
VI. CONCLUSION
[6]
Knowledge discovery analytic platform proposed in this
paper incorporates novel trends and methods used in the
knowledge discovery in manufacturing area. The traditional
data warehouse approach for knowledge discovery platform is
supplemented with Hadoop cluster, to store big data collected
at the field level of hierarchical control.
[7]
The proposed analytic platform preserves the robustness
and well-proven technology for traditional business
intelligence and analytic tools, and creates space for
knowledge discovery in frequently and continually updated
manufacturing data in a raw form. Therefore it represents an
ideal compromise between existing traditional tools and the
need for strong business intelligence, reporting and analytic
platform.
[8]
[9]
[10]
The main disadvantage is the necessity of integrating all
data in a Data Lake, which makes it difficult to ensure the
integrity and security of company data. In traditional relational
databases and data warehouses various approaches, methods
and tools for maintaining integrity and security of company
data are available. In Data Lake represented by Hadoop
cluster, all data needs to be integrated altogether, and the
discovery platform must have access to all the data. This is
one of the main issues addressed when implementing Data
Lake.
[11]
[12]
[13]
[14]
The proposed approach focuses on all hierarchical control
levels in manufacturing. Therefore, manufacturing area as a
whole represents the main application area of this approach.
With a certain degree of abstraction, the approach can be
applied also in other industrial fields, where lots of data needs
to be collected frequently or continuously.
[15]
[16]
[17]
ACKNOWLEDGMENT
This publication is the result of implementation of the
project VEGA 1/0673/15: “Knowledge discovery for
hierarchical control of technological and production processes”
supported by the VEGA.
[18]
[19]
REFERENCES
[1]
[2]
[3]
[20]
J. Jadlovský, S. Laciňák, M. Čopík and J. Ilkovič, “Technological level
of flexible manufacturing system control,” Acta Electrotechnica
Informatica, vol.11, No.1, pp. 20-24, 2011.
P. Tanuška, P. Važan, M. Kebísek and D. Jurovatá, “Knowledge
discovery from production databases for hierarchical process control,”
International Journal of Mechanical, Aerospace, Industrial, Mechatronic
and Manufacturing Engineering vol.7, No:11, 2013.
J. Jadlovský, J. Laciňák, J. Chovaňák and J. Ilkovič. “Návrh
distribuovaného systému riadenia pružnej výrobnej linky,”
In:
ISBN: 978-1-61804-327-6
[21]
[22]
155
International Conference – Cybernetics and informatics. Vyšná Boca.
2010.
C. Gröger, F. Niedermann and B. Mitschang, “Data Mining-driven
Manufacturing Process Optimization,” In: Proceedings of the World
Congress on Engineering 2012 Vol III, WCE 2012. Hong Kong:
Newswood 2012, pp. 1475-1481.
K. Wang, S. Tong, B. Eynard, L. Roucoules and N. Matta ,“ Fuzzy
systems and knowledge discovery,” FSKD, 2007.
P. Michalik, J. Štofa and I. Zolotová, “Testing the properties of Kmeans algorithm for data mining applications,” In: LINDI 2013 : 5th
IEEE International Symposium on Logistics and Industrial Informatics :
Proceedings : September 5-7, 2013, Wildau, Germany. - Piscataway :
IEEE, 2013 P. 99-102. - ISBN 978-1-4799-1257-5.
G. Köksal, İ. Batmaz and M. C. Testik, “A review of data mining
applications for quality improvement in manufacturing industry,”Expert
Systems with Applications, 38 (10) (2011), pp. 13448–13467.
Rexer Analytics, “Data miner survey – 2013 survey summary report,“
2014,
[cit.
20.06.2015].
Available
online:
http://www.rexeranalytics.com/Data-Miner-Survey-Results-2013.html.
KDnuggets, “Data Mining Community‘s Top Resource,“ 2014, [cit.
23.06.2015]. Available online: http://www.kdnuggets.com.
B. Vorhies, “The big deal about big data: what’s inside – structured,
unstructured, and semi-structured data in data magnum blog,“ 2013. [cit.
25.06.2015] Available online: http://data-magnum.com/the-big-dealabout-big-data-whats-inside-structured-unstructured-and-semistructured-data/.
X.Z Wang, “Data mining and knowledge discovery for process
monitoring and control advances in industrial control,“ Springer Science
& Business Media, 2012. 251p. ISBN 978-1-44710-421-6.
X. Zhu, “Knowledge discovery and data mining: challenges and
realities,“ Challenges and Realities. 2007. Idea Group Inc (IGI). 290p.
ISBN 978-1-59904-252-7.
H. Chen, R. H. L. Chiang, and V. C. Storey, “Business intelligence and
analytics: from big data to big impact ,“ MIS Q. 36, 4 (December 2012),
1165-1188.
P. Chapman, P. Kerber, J. Clinton, J. Khabaza, T. Reinartz, C. Shearer
and R. Wirth, “The CRISP-DM Process Model,”. Discussion Paper. 0503-99. Marec 1999.
“What is the CRISP-DM Methodology,” [cit. 20.06.2015]. Available
online: http://www.sv-europe.com/crisp-dm-methodology/.
J.A. Harding, M. Shahbaz, S. Srinivas and A. Kusiak, “Data mining in
manufacturing: a review,” Journal of Manufacturing Science and
EngineeringTransactions of ASME, 128(4), 969–976.2006.
A.K. Choudhary, M.K. Tiwari and J.A. Harding, “Data mining in
manufacturing: a review based on the kind of knowledge. ” In: Journal
of Intelligent Manufacturing. Leicestershire: Loughborough University´s
Institutional Repository. 20 (5), s. 501 – 521. 2009.
KDnuggets, “Algorithms for data analysis/data mining. Which methods/
algorithms did you use for data analysis?,” [cit. 19.06.2015]. Available
online: http://www.kdnuggets.com/polls/2011/algorithms-analytics-datamining.html.
T.H. Davenport and J. Dyché, “Big data in big companies,” Thomas H.
Davenport and SAS Institute Inc May 2013.
W.Fan and A. Bifet, “Mining big data: current status, and forecast to the
future,” SIGKDD Explor. Newsl. 14, 2 (April 2013), 1-5.
M. Rouse, “Big data analytics,” [cit. 20.06.2015]. Available online:
http://searchbusinessanalytics.techtarget.com/definition/big-dataanalytics.
S. Zhang, C. Zhang and Q. Yang, “Data preparation for data mining,
Applied Artificial Intelligence,” An International Journal, Volume 17,
Issue 5-6, 2003. Taylor & Francis, 2003. doi: 10.1080/713827180.