Download Security monitoring through log analysis

Security Monitoring Thru Log Analysis
Sam.NG@PISA
What the hacker did
Startup script was modified,
a line is added to the rc.local
 ./usr/bin/t0x5mm (I can’t
remember the exact name)
 “ls” the file, nothing showed
 I tried to remove the file by
“rm”, “rm” ran without any
error
 Many months later, I knew
that’s something call “rootkit”

Log was gone!




Some of the log files are truncated
I wanted to know how the hacker
got in, and I redirected syslog to
“/dev/lp0” and ran for a couple of
weeks
Several inches of paper printed,
but there is more than I could
handle
I can’t remember how, but finally I
knew the hacker got in by exploiting
an IMAP4 vulnerability
Lesson Learnt




You will need the log in sometime
And better still, store it in a safe
place
Write once read many (WORM) is
good, but searching capability is
even more important
And the most important: you have
to prepare it
beforehand
But how can I use the log for
security monitoring?
The Challenge
“I don’t have the staff
to handle 140,000
alerts. I don’t even
have the staff to handle
12,000 alerts”
David MacLeod, Ph.D., CISSP
The Regence Group CISO
Reference:
Counterpane’s Security
Monitoring Service
Bruce Schneier
Block Diagram
Phase I
Log
Collection
Phase II
Noise
Reduction
Phase III
Data
Mining
Phase IV
Alert/Ticket
Management
Alert/Ticket
Management
Noise
Reduction
Log Collectors
Data Mining
Log Collectors
Send your log to DATABASE
 Can be direct (open database connection
from the client) or in-direct (e.g. post data
to a web application)
 Each different log may need a different log
collector
 Many logs have built-in support to store
data in database

Unix syslog







syslog can be redirected
Can be redirected to printer! (“/dev/lp0” in Linux)
Can be redirected to remote syslog host
But redirect to program (pipe) seems not
working!!??
syslog-ng support logging directly to database
Win32 solutions to accept remote syslog records
and log to database
Plain text format, easy to write a program to
watch the data and then send to database
Demo

A smiple “tail” base perl script to monitor
new records in syslog and send it to remote
database
Windows Eventlog
Microsoft logparser can log to database
 WMI interface allows you to query remote
eventlog
 Many third party solutions support logging
to database

Demo

A smiple C# program to monitor new
records in Eventlog and send it to remote
database
Quiz 1
Which of the following Windows
Server will pass Microsoft
Baseline Analyzer (MSBA) 2.0
auditing requirements with
default install?
A.
B.
C.
D.
Windows NT
Windows 2000
Windows 2003
None of the above
DO YOU KNOW?



You can pass MSBA 2.0 if you enabled Success
and Failure auditing for the Account Logon Events
But indeed, it recommends you to enable
Success and Failure auditing: Account Logon
Events, Account Management, Policy Change,
System Events
Failure auditing: Directory Service Access, Object
Access
Data volume may be quite high, a server config in
this way may generate a eventlog for every 2
seconds (actual volume depends on your server)
Microsoft IIS Web Server
Native support log to
ODBC (but seems to be
only available on server
platform only)
Microsoft IIS Web Server (2)






Microsoft does not recommend logging to
database if the IIS is busy (Q245243)
But nowadays most web pages are generated with
SQL queries
Test ODBC logging on your own environment
PrepWebLog (Q296093) convert lIS log to SQL
insertion statements in plain text format, but still
have to run these SQL insertion by some other
means
Still, the best would be realtime logging to
database
Writing a “tail -f” similar program in IIS is difficult
DO YOU KOWN?
IIS log file is updated in 64k chunks. On
servers that do not have a high usage rate,
the statistics will not be up-to-date because
of the delayed write (Q142557)
 When it flush, may be only first 20k of the
chunk contains data, the rest 44k data is ‘\0’
and will be filled in next flush
 Obviously a mmapped file, may be due to
performance consideration… but

Suggestions to programmer
In C/C++, STDERR, cerr are not buffered
 In fact, error log should be send out
immediately, should not be cached/buffered

# man stderr
……
……
CONSIDERATIONS
The stream stderr is unbuffered. The stream stdout is linebuffered when it points to a terminal. Partial lines will
not appear until fflush(3) or exit(3) is called, or a
newline is printed. This can produce unexpected results,
especially with debugging output.
……
……
J2EE Application Server
Most application server support log4j or
java.util.logging (JDK 1.4+)
 log4j natively support logging to database
thru JDBCAppender

My Experience







I have a program developed since JDK1.3
At that time, Java don’t have java.util.logging
And I don’t know much about log4j
I wrote my own log handling routine (similar to
syslog) to insert my own records into database
But is not configurable/adaptable, and is not
consolidated with the Application Server’s log
Since logging is widely spread all over the codes, it
is very difficult for me to change the program to
use these new technology
Log architecture should be planned in DESIGN
PHASE
Snort IDS
Comes with database
output plugin to send
alerts to database
 The packet analysis thread
is also responsible for
database insertion

My Experience
I have experience in using Snort to monitor
a ~20M Internet link, with database output
plugin, default rules
 CPU is just about 30%, seems good 
 When I changed to log to local file, CPU
sharp jump to 100%
 Barnyard seems to be a solution but it does
not support database!!

And many others
Checkpoint FW1: thru LEA enabled product
http://www.opsec.com/
 Microsoft ISA: default support ODBC logging
(KB838710)
 Microsoft Exchange: same as Eventlog
 Apache httpd: mod_log_sql

Noise Reduction

Very important

Noise will kill the system


1.
2.
3.
4.
5.
And to improve overall performance
Can be done on several layers
ignore certain message at the log collection sender
ignore certain message at the log collection
receiver end
delete (mark as ignore) at the database
fine tune your IDS rule, firewall logging policy etc..
exception cases for data mining phase
Data Mining
“Data processing using sophisticated
data search capabilities and statistical
algorithms to discover patterns and
correlations in large preexisting
databases; a way to discover new
meaning in data”
From http://www.tfd.com
Data Mining (2)




The heart of Security
Monitoring
A board term, a
general concept
Utilize database
queries to get the
information you want
Can be an external
program, can be a SQL
server schedule job
Data Mining Techniques
Data Mining
Attack Detection
Attack Definition
Event Correlation
Statistical Analysis
Anomaly Detection
Normal Definition
Event Correlation
Statistical Analysis
Attack Definition
IIS Unicode Directory Traversal (cmd.exe) attack strings
GET
GET
GET
GET
GET
GET
GET
GET
GET
GET
GET
GET
GET
GET
GET
GET
GET
GET
GET
GET
/..%c0%af../winnt/system32/cmd.exe?/c+dir+c:\
/MSADC/root.exe?/c+dir
/_mem_bin/..%255c../..%255c../..%255c../winnt/system32/cmd.exe?/c+dir
/_vti_bin/.%252e/.%252e/.%252e/.%252e/winnt/system32/cmd.exe?/c+dir+c:\
/_vti_bin/..%255c../..%255c../..%255c../winnt/system32/cmd.exe?/c+dir
/_vti_bin/..%c0%af..%c0%af..%c0%af..%c0%af..%c0%af../winnt/system32/cmd.exe?/c+dir+c:\
/_vti_bin/..%c0%af../..%c0%af../..%c0%af../winnt/system32/cmd.exe?/c+dir+c:\
/_vti_cnf/..%c0%af..%c0%af..%c0%af..%c0%af..%c0%af../winnt/system32/cmd.exe?/c+dir+c:\
/adsamples/..%c0%af..%c0%af..%c0%af..%c0%af..%c0%af../winnt/system32/cmd.exe?/c+dir+c:\
/cgi-bin/..%255c../..%255c../..%255c../winnt/system32/cmd.exe?/c+dir+c:/
/cgi-bin/..%c0%af..%c0%af..%c0%af..%c0%af..%c0%af../winnt/system32/cmd.exe?/c+dir+c:/
/cgi-bin/..%c0%af..%c0%af..%c0%af..%c0%af..%c0%af../winnt/system32/cmd.exe?/c+dir+c:\
/cgi-bin/..%f0%80%80%af../winnt/system32/cmd.exe?/c+dir+c:/
/iisadmpwd/..%c0%af..%c0%af..%c0%af..%c0%af..%c0%af../winnt/system32/cmd.exe?/c+dir+c:\
/iisadmpwd/..%c0%af../..%c0%af../..%c0%af../winnt/system32/cmd.exe?/c+dir+c:\
/msadc/.%252e/.%252e/.%252e/.%252e/winnt/system32/cmd.exe?/c+dir+c:\
/msadc/..%c0%af../..%c0%af../winnt/system32/cmd.exe?/c+dir+c:\
/msdac/root.exe?/c+dir+c:\
/scripts/..%e0%80%af../winnt/system32/cmd.exe?/c+dir+c:\
/scripts/..%f0%80%80%af../winnt/system32/cmd.exe?/c+dir+c:\
Vulnerability characteristic
≠attack characteristic






CodeRed I [/default.idq?NNNNN……]
CodeRed II [/default.idq?XXXXX……]
In fact, the vulnerability can be exploited if the
variable name is around 240 bytes
Regular Expression [\/default\.idq\?.{240,}]
Difficult to write an effective and accurate definition
And not all attack leave audit trail
Event Correlation


E.g. 1000 login failures followed
by ONE successful login from the same IP
E.g.
IF http_response_code = 500; THEN
find_all_other_url_accessed();
ENDIF
Anomaly Detection
Normal Definition
You define what is normal and then monitor
it
 E.g. Operators login time should be
corresponding to their shift duty
 E.g. All server services should not be
restarted unless necessary (ignore service
start within 3minutes of system startup)

Statistical Analysis




E.g. on average a event
occurs 10 time a day, with a
standard deviation of 2.3. But
today we have 2000 records.
E.g. anything happens more
than 200 times in the past 30
minutes
E.g. a event never seen in the
past 7 days
E.g. “TOP 10”
events/users/hosts, etc…
Alert/Ticket Management






Works like Bug Tracker
Save the alert as a
“Ticket”
A ticket is a something
like an outstanding job
Assign the ticket to a
staff to follow up
Escalate it if remain
unresolved for some
time
And don’t forget People Management
(Time and Skill)
Alert/Ticket Management (2)



According to my experience,
at least 90% of the alerts
generated by data mining
are still FALSE ALARMS
People will get use to it and
tend to think EVERY alerts
are false alarms
If possible, fine tune the
system to eliminate the
false alarm from occurring
again
Couterpane 2003 Results
DO YOU KNOW?

TSL providing Email
to pager service for
about HK$80/mth
How should I start?



Do it step by step, phase by
phase, EventLog, syslog are
easy to start with
A group brain storming
section would give you at
least 10 such data mining
rules, and is a good starting
point
Security Monitoring is a (long
term) process, do not regard
it as a single one-shot install
and forget project
Contact: [email protected]