Download Centralized Management and Processing Policy for Log Files

Centralized Management and Processing Policy for Log Files
Qiao-Ping SUN1,2, Xiao-Ming ZHAO1
1
Department of Computer, Taizhou University, Taizhou, Zhejiang 317000, P.R.China
2
Department of Computer Science and Engineering, Shanghai Jiao Tong University, Shanghai 200030,
P.R.China
E-mail: sunqp887@ sohu.com
∗
Abstract: The management mechanism for log files is important when we establish the policy of
network information security, since the log files record user’s activities on systems and network, playing
an important role in network security. This paper analyzes the effects and features of the log files, and
presents an integrated policy to process and administrate them. Some methods to analyze the log files
are also introduced.
Key words: log files; network security; intrusion activity; integrality safeguard
1 Introduction
With the rapid development of the information technology, computers and networks become
indispensable to our daily life. But illegal intrusion and attack to computers and networks are rampant
increasingly, threatening the security of information resource every moment. Thus network security
problem attracts more and more attention. As the primary recordings of systems and network user’s
activities, logs are greatly important for finding intrusion activities, recovering systems, reporting usage
statuses of system resources and offering electronic evidences to crack down on computer crimes. When
a system is attacked, we can find system holes by analyzing log files, fix on the weak link of the
network, find out the probable attack, and apply necessary measures to strengthen network management.
Therefore we should protect system log files from modifying or deleting by intruders. It’s of vital
importance to system maintenance and system safety.
2 Characteristics and applications of log files
Log files consist of Windows operation system log files, Unix/Linux server log files, application
server log files, intrusion detection system(IDS) log files, fire wall log files, router log files, and other
application tools log files. In the computer security field, the applications of log files include monitoring
system resource and network throughput to detect network problems, monitoring users’ actions by
recording service conditions to prevent users from exceeding their authority and warn of questionable
conduct, diagnosing abnormal events, determining the intrusion range, providing help for system
recovery, generating report, and providing evidence for computer crimes.
Log file information has three characteristics as follows.
(1) Log files are different in source, and their formats differ from each other. There is no standard
log file format internationally. Although most log files are stored in text files, different operation
systems, different application software, and different network servers generate log files with different
formats. Even the same server, such as IIS, may record log information in various formats. Furthermore,
different system developers and different network equipment manufacturers define different log file
formats as they need.
(2) The data is very large. Various security equipments, network equipments, mainframe systems
and application systems produce large quantity of log information every day. As a result, log files are
very large, especially the log files generated by external service such as Web service log, firewall log,
IDS log, database log and server log, ranging from several megabytes to dozens of gigabytes.
(3) Log files are easy to be maliciously destroyed, modified, and even forged. Most systems store
∗
This paper is sponsored by the Provincial Natural Science Foundation of Zhejiang(104521)
1254
the sensitive log files in plaintext in the system-generating log files, or send the log files to database
through syslog mechanism. The frangibility of this log file management mechanism gives chances to
invaders, who can cover their secret attacks by modifying or deleting log information which contains
their crime trace.
The characteristics above bring great difficulties to the log file management and analysis.
3 Secure storage and transmission policy of log files
To keep log files integrity and log data validity, and to make the stored log data easy to analyze,
three measures should be taken when one stores and transmits log files.
3.1 Establishing knowledge base of log file formats
According to the variety of log files, the log file formats of various systems, the positions of log
files stored and the names should be formally analyzed before dealing with the log files. A complete
format knowledge base of log files should be established. Note that it is easy to classify log files and
refresh the knowledge base according to new formats.
3.2 Storing log data in database by classification
When a log file is collected, we can analyze the log file according to the rules of the log file
knowledge base, determine the type and format of the log file, find out matched log data list, and store
log data in database in consolidation form after classification, as show in fig.1.
Fig.1 Flow process chart of storing log files
3.3 Integrity and encryption-based protection
One-way hash functions and message authentication codes can be adopted to protect log file
integrity [1]. We can operate the original log files with hash functions, generate fixed-length check codes,
and store them in database. Generally, one-way hash functions without keys should be adopted so that
every one can verify hashed values. MD4, MD5 and SHA are the widely-used one-way hash functions.
Sensitive log files can be protected by Message Authentication Code with encryption keys, and be stored
by compression or medium bake-up, shown below.
Fig.2 Original log file integrity and encryption-based protection
1255
4 Integrated management mechanism for log files
Operating environments vary in different parts of a network system, and the log file formats are
different as well. Therefore it is inefficient and time-consuming to conduct regular checks on every
target log file only by the network administrator. And it is difficult to respond to security events as a
whole for every part of the network if there is no uniform rule [2]. It is also difficult to analyze log data.
Here we present an integrate management mechanism. All log files on network are collected in a log
server for integrated management, shown in fig.3.
Fig.3 The topology of integrated log management
From Fig. 3 we can see that the integrated log management structure is a trilaminated model. The
bottom layer is the target equipments which generate various log files. The intermediate layer is a log
management console, which is the core of the management mechanism. The top layer is a log storage
server and a log database server.
4.1 Log management console
The log management console uses two network cards to physically insulate the bottom network
equipment layer from the top log server layer. One network card communicates with the bottom target
equipment. The other communicates with the top log server, and performs the functions below.
(1) Controlling the target equipment to send original log files. That is to say, the target equipment is
demanded to send the encrypted log files to log console buffers through authentification mechanism.
(2) Controlling the log storage server to receive, store and send log files. The log console decrypts
the encrypted log files received by the buffers and send the original log files to the log storage server.
The log console extracts log data from log files, store log data in log database, regularly checks the
storage capacities of log storage servers, and deletes the earliest log files after backup if it is necessary.
(3) Controlling log database server to receive, process, and analyze log data. It analyzes the log
data in log database, and then generates a statistical report.
4.2 Log storage server
Log storage server collects various original log files received from the log console. The log files are
stored with directory structures of the log resource, the log type, and the date. To the security of log files
stored in the log storage server, log storage server boots up with CD driver, and does not provide any
application service.
4.3 Log database server
Log database server stores all kinds of log data of various log files for the administrator to query,
count and analyze. Because the data stored in log database is very large and the data structure is very
complex, a large relational database management system should be taken, such as ORACLE.
1256
5 To extract and analyze log files
The administrator can query and analyze the log data, and make statistics after the log files are
stored. In order to do well to the data stored in the log files, one should do the three jobs as below.
5.1 To filtrate and extract
The large amount of log data always confuse the administrator, so we should filter the large amount
of log data, and then extract the log data worthy of being analyzed to the assay. In fact, we mainly
extract the related data of login and logout(such as IP address, login time, login account, password and
logout time et al) and the log information regarding with system confusion state (such as CPU, memory,
external equipment, etc.).
5.2 To check the coherence of log data
We can find out the obviously abnormal events in time when checking the coherence of log data,
and then we can take actions on emergency. The coherence checking includes such contents. Is the
structure of original logs according with the system set or not? Are the regular events absent or abundant?
Is any log file missing before its deletion time?
5.3 To analyze the log data
We also use statistical methods in data analysis. That is to establish characteristic functions
according to the frequency, average amount, and square difference of the characteristic variables when
the system is running normally. When the log files differ from the normal characteristic functions
significantly, we conclude that the system is in broken. Besides, we can adopt the rule based expert
system and machine learning techniques [3] etc.
6. Conclusion
Although log administration is a critical component in network administration, it is often ignored.
The log formats vary greatly. The data is very large, and easy to be destroyed. These bring great
difficulties to the log management and analysis. The integrated management mechanism presented in
this paper realizes the log collecting, sorting, and pigeonholing, guaranteeing the security of log files in
storage and transfer. This strategy works effectively in practice.
References
[1] FANG Hangfeng, WANG Haihang. Design and Implentation of Log Distill and Analysing System.
Computer Engineering. 2004 30(14):108-110
[2] Yu Shaohua, Guan Yong,Dai Yigi. The Application of Data Mining on Log Management. Computer
Engineering and Applications. 2004.15:178-181
[3] Wang Wei, Peng Qinke. Analysis of Host Audit Trails and Their Applications to Intrusion Detection.
Computer Engineering and Applications. 2002.13:35-37
1257