Download SQL Log File: Spelunking

SQL Log File:
Spelunking
(for anyone, really, that wants to
recover lost data or find out
whodunnit)
Welcome to my lo-fi
presentation
• (boycott slideshows!)
• I am Scott Ellis, kCura’s Infrastructure
Engineering Architect
The Scenario!
You (or your client) got HACKED
Symptoms and Verification
• Important tables are
missing
• Clients are missing
• There is a giant smiley
face painted on your
application home page
• There are no database
errors
• DBCC checkDB checks OK
• You verify certain tables
are, in fact, missing.
• You restore from backup
and now you want to roll
forward up to the
moment of disaster.
What now?
• Log file analysis
Wait!
What’s a log file?
• (SQL calls them "ldf"s)
• It's a rewind file for the database.
• a record of everything, circular log, blah blah
blah
• They are often poorly maintained and retain
massive amounts of data.
• Unchecked, they can get HUGE (terabytes in
size)
Types of database logging
• ACID compliant databases have logs.
• MS SQL
– BULK
– SIMPLE
– FULL
– Emergency
When should you look for Log files?
• Any database that is ACID compliant will use
them, though the structure won't be the
same.
– Atomic
– Consistent
– Isolated
– Durable
Where are they?
• Large, improperly maintained log files are not
only common, they are prevalent. They are
anywhere that an ACID database is.
• (on your phones!)
• On a live database, you can easily see where
the logs are kept by viewing its properties
So now you want to know:
“How do we go about finding out what's in them?”
• You can also search through log backups (If you have them - the absence
of log backups is not suspicious if the transaction log is quite large and the
maintenance plan for log backups has been failing.)
• For live database log files, uncommitted transactions:
Use the undocumented fn_dblog function.
• for committed that are still sitting in the log file:
DBCC TraceOn (2536) then
fn_dblog
• From log backup files
fn_dump_dblog
• BEWARE: as of at least 8/13/2013: every time
fn_dump_dblog is called, it creates a new
hidden SQLOS scheduler and up to three
threads, which will never go away and never
be reused. It's a bug.
• Here's the thing about undocumented
features: DON'T USE THEM ON YOUR
PRODUCTION SYSTEM!
DISCLAIMER
• (If you have Relativity, and you break your
production system by executing anything I
show you tonight, I will deny ever having been
here.)
DEMO Time!
--First, let's make some backups...
ALTER DATABASE readytest SET RECOVERY FULL
BACKUP DATABASE [readyTest] TO DISK = N'J:\ReadyTestDemo_Full.bak' WITH INIT;
GO
BACKUP LOG [readyTest] TO DISK = N'J:\ReadyTestDemoLog_Log1.bak' WITH INIT;
GO
--Now do the bad thing:
DROP TABLE eddsdbo.[4n6ArtifactWords]
--Now pretend to rebuild the tables (next three tabs) (I did this ealier because it takes a long time, this is the Bloomberg data)
--now, when was the table dropped, and who did it?
--First, let's make some backups...
ALTER DATABASE readytest SET RECOVERY FULL
BACKUP DATABASE [readyTest] TO DISK = N'J:\ReadyTestDemo_Full.bak' WITH INIT;
GO
BACKUP LOG [readyTest] TO DISK = N'J:\ReadyTestDemoLog_Log1.bak' WITH INIT;
GO
--Now do the bad thing:
DROP TABLE eddsdbo.[4n6ArtifactWords]
demo continued
--Now pretend to rebuild the tables (next three tabs) (I did this earlier because it takes a long time, this is the Bloomberg data)
--now, when was the table dropped, and who did it?
SELECT Top 10
suser_sname([Transaction SID]) as 'Login',
[Transaction Name],
[Begin Time],
[Current LSN],
[Operation],
[Context],
[Transaction ID],
[Description]
FROM
fn_dblog (NULL, NULL),
(SELECT
[Transaction ID] AS [tid]
FROM
fn_dblog (NULL, NULL)
WHERE
[Transaction Name] LIKE '%DROPOBJ%') [fd]
WHERE
[Transaction ID] = [fd].[tid];
GO
--If you happen to bee googling around and run across this SQL: SELECT suser_sname(convert(varbinary,"Transaction SID")) as 'Login'
--it is wrong. Don't use it.
-•
• Performance Tip
Disk, and CPU will get slammed by this procedure.
• Google tip: just the word Fn_dblog will turn up a lot of
info. I suggest that you set aside a day and plan
stepping through what you find, and create your own
documentation about how it works. For examole,
Dimitri's info, while useful, doesn't come along with
him saying that he actually tested doing this. If you
want to recover the log file from a backup of the
database that included the tail of the log, then you
should try it and make sure this works.
• Useful other tools (if you don't want to carve bytes)
Light speed, Toad, Apex
• Tips: use a machine that has a lot of disk IO and CPU
Some Questions:
• Can each of these tools work with just a log file, or do
they need a functional database behind them?
• What if I carve out what I believe is a log file, will it
work?
• Where will you find SQL Log files?
• More information can be found on SQLskills.com
http://www.sqlskills.com/blogs/paul/using-fn_dblogfn_dump_dblog-and-restoring-with-stopbeforemark-to-an-lsn/
• From dimitri furman (msn blog)
http://blogs.msdn.com/b/dfurman/archive/2009/11/05/readingdatabase-transaction-log-with-fn-dump-dblog.aspxUpdate 2012-02-08:
I just found out that the function can also work against database
backups, not just log backups. A database backup contains a portion of
the log that will be rolled forward on restore to make the restored
database transactionally consistent, and that portion of the log can be
viewed with the fn_dump_dblog() function. This is potentially useful to
discover the LSN and the timestamp of the checkpoint that occurs
during the backup - look for LOP_END_CKPT in the Operation column.
FINAL Question:
• Shutting down a database server: shut down normally or
pull the plug? If a database really is ACID, should it matter
if I kill the power to it?
• Also, if you shut it down normally, it's going to finish
executing whatever transactions it's doing!
• What if it is a cleanup where the log gets truncated and
then wiped (because the dude you are after was able to
kick of his "Destroy everything" maintenance task before he
was tackled.)
• Wellllll......
I asked an expert "Is it true that
unplugging a database server can
corrupt it?" and here is what he said:
"In theory, if it was perfectly configured, it won't, but that's rarely true. People
leave write caching on at the RAID controller, for example. When you pull the
plug, the data is (hopefully) written to non-volatile RAM, and then when
power comes back, it finishes writing the data out to the relevant drives.
Perfect example of problems though - we had a datacenter-wide AC outage,
and the sysadmins yanked cables to shut down servers faster. When we went
to power back up, some of them suffered RAID array failures - when you stop
drives that have been running for years, sometimes they don't come back up.
So we pulled the drives, put them into another server, and I had corrupt
databases because SQL Server's writes hadn't all been done to the transaction
log. Some arrays had them, some didn't."
~well-known SQL expert Brent Ozar
Evidence Tracking Application
Client Information
Evidence Tracking Application
Project Information
Evidence Tracking Application
Company Information
Evidence Tracking Application
Custodian Information
Evidence Tracking Application
Evidence Information
Evidence Tracking Application
Forensic Image Information
Evidence Tracking Application
Examiner Information