Download Operating_System_Performance

Operating System
Performance
prepared by
Jason Meyer
Presented by
Group 4
Agenda
• Mindcraft Benchmarks
–
–
First Benchmark
Open Benchmark
• Linux Optimizations
a.
b.
c.
d.
e.
f.
General Tips
Hardware Tuning
Network Tuning
File Serving
Web Serving
Mail Serving
• References
Operating System Performance
• Mindcraft’s first Benchmark
– April 13, 1999
– Performed a File and Web Server comparison
between Windows NT and RedHat Linux 5.2
(kernel 2.2.2)
– Used Dell PowerEdge 6300m with 4x400MHz
Pentium II Xeon processors, 4 GB RAM, and a
RAID disk.
Operating System Performance
• Mindcraft’s first Benchmark
– Linux:
• Samba 2.0.3 as the SMB file server
• Apache 1.3.4 as the Web server.
– Windows NT:
• embedded SMB file server
• Internet Information Server 4.0 Web server.
Operating System Performance
• Mindcraft’s first Benchmark
– File server performance tested with Ziff-Davis
Benchmark Operation NetBench 5.01
• Measures throughput in Bytes/second
• A number of test systems read and write files on
a server
• NetBench test suite made up of a number of
mixes.
– A mix is a particular configuration of NetBench
parameters.
– Each mix increases the number of test systems
• Mindcraft used 10 mixes, ranging from 1 to 144
test systems (clients).
Operating System Performance
• Mindcraft’s first Benchmark
– Web server performance tested with
Ziff-Davis Benchmark Operation
WebBench 2.0
• A number of test systems request URLs.
• Measures 2 things:
– The number of HTTP GET requests per second.
– The number of bytes per second that a Web
server sends to all test systems
Operating System Performance
• Mindcraft’s first
Benchmark
Results:
– Windows NT Server 4.0
was 2.5 times faster
than Linux as a File
Server and 3.7 times
faster as a Web Server
Operating System Performance
• Outcry from Linux Community
– Mindcraft did not reveal they were funded by
Microsoft, worked in a Microsoft lab, and had
help from Microsoft to tune NT
– Mindcraft claimed to have attempted to tune
Linux but couldn’t get any help
– Many people in the Linux community believed
they purposefully tried not to tune Linux, or at
the least made no effort to tune Linux.
Operating System Performance
• Mindcraft’s Response
– Re-ran the benchmark (but never released
results).
– Mindcraft offered to do another “Open
Benchmark”.
– Leaders from the Linux community were
invited to offer tips for improving performance
and were allowed to be present for the actual
test.
– Took place June 30, 1999
Operating System Performance
• Open Benchmark results:
– Overall, results were similar to the first
benchmark.
– Windows NT was about 2 times faster as a File
Server and a Web Server, depending on the
setup used.
Operating System Performance
Open Benchmark results:
Operating System Performance
• Linux Community response:
– Some downplayed these results for various
reasons
• Claimed benchmarks not reflective of real world
• Claimed Microsoft had unfair advantage in
tuning performance.
– At the least, this showed that Linux was
not miles ahead of NT or at least had room
for improvement.
– The following website was developed to
aid in improving performance on Linux:
http://linuxperf.nl.linux.org/
Operating System Performance
• General Tips:
– The following tips can be relevant no
matter what the primary purpose of a
machine is.
– If the system seems to be running slow,
the first step is to track down why it is
slow, and then find a fix for the problem.
– Even if no problems are noticeable, the
following tips could help improve
performance.
Operating System Performance
• Common Problems that hinder
performance:
– Swapping: Problem if process needs to
be continuously mapped back into main
memory.
– Can check with “top” and seeing the
process ‘kswapd’ takes a large
percentage of the CPU for several
seconds.
– Can also check with vmstat and iostat.
Operating System Performance
• Common Problems that hinder performance:
– Example of top where swapping is not a problem:
2:22pm up 21 days, 5:39, 2 users, load average: 0.34, 0.08, 0.02
54 processes: 53 sleeping, 1 running, 0 zombie, 0 stopped
CPU0 states: 0.0% user, 6.1% system, 0.0% nice, 93.3% idle
CPU1 states: 6.2% user, 2.1% system, 0.0% nice, 91.0% idle
Mem: 1545264K av, 1188708K used, 356556K free, 0K shrd, 28944K buff
Swap: 1542232K av,
0K used, 1542232K free 967872K cached
PID
4997
764
4933
4962
1
2
3
4
5
USER
fcsl
root
root
fcsl
root
root
root
root
root
PRI
17
9
9
10
8
8
19
18
9
NI SIZE RSS SHARE STAT %CPU %MEM
0 31256 30M 9896 D
8.7 2.0
0
0
0
0 SW
0.3 0.0
0
644 644
540 S
0.1 0.0
0 1064 1064
856 R
0.1 0.0
0
520 520
452 S
0.0 0.0
0
0
0
0 SW
0.0 0.0
19
0
0
0 SWN
0.0 0.0
19
0
0
0 SWN
0.0 0.0
0
0
0
0 SW
0.0 0.0
TIME
0:04
0:00
0:00
0:00
0:15
0:00
0:00
0:00
0:00
COMMAND
otbsaf
rpciod
in.telnetd
top
init
keventd
ksoftirqd_CPU0
ksoftirqd_CPU1
kswapd
Operating System Performance
• Common Problems that hinder performance:
– Example of vmstat where swapping is not a problem
[fcsl@seneca ~]$ vmstat
procs
memory
swap
r b w swpd
free buff cache si so
0 0 0
0 309944 28944 984756 0
0
bi
0
io
system
bo in
cs
0 6
2
cpu
us sy id
0 0 6
where
swap:
si: Amount of memory swapped in from disk (kB/s).
so: Amount of memory swapped to disk (kB/s).
Operating System Performance
• Common Problems cont’d.:
– Accessing the disk too hard
– Can check by running `ps axlw` and the
STAT field will contain a “D” when the
process is waiting on the disk.
– Not much to help, perhaps use a faster
disk or RAID.
Operating System Performance
• Common Problems cont’d.:
– Example of ps alwx without anything waiting on
disk access:
[fcsl@ute OTBSAF]$ ps alwx
F
UID
PID PPID PRI NI
100
0
1
0
8
0
040
0
2
1
9
0
040
0
3
0 19 19
040
0
4
0 19 19
040
0
5
0
9
0
040
0
6
0
9
0
040
0
7
0
9
0
040
0
10
1
9
0
040
0
11
1
9
0
. . .
VSZ
1412
0
0
0
0
0
0
0
0
NOTE, for the STAT field,
S = sleeping
W = has no resident pages
N = low-priority task
RSS
468
0
0
0
0
0
0
0
0
WCHAN
do_sel
contex
ksofti
ksofti
kswapd
bdflus
kupdat
down_i
down_i
STAT
S
SW
SWN
SWN
SW
SW
SW
SW
SW
TTY
?
?
?
?
?
?
?
?
?
TIME
0:18
0:00
0:09
0:01
1:21
0:00
0:10
0:00
0:00
COMMAND
init [5]
[keventd]
[ksoftirqd_CPU0]
[ksoftirqd_CPU1]
[kswapd]
[bdflush]
[kupdated]
[scsi_eh_0]
[scsi_eh_1]
Operating System Performance
• Common Problems cont’d.:
– All time spent in User processes
• A user-level program chews up all the CPU
• Use top to find the process that is the culprit
• Optimize this process or move it to another machine.
Operating System Performance
• Common Problems cont’d.:
– Example of top with the process “guimain” chewing up
all the CPU
2:57pm up 21 days, 6:12, 11 users, load average: 0.15, 0.03, 0.02
92 processes: 85 sleeping, 7 running, 0 zombie, 0 stopped
CPU0 states: 0.1% user, 0.2% system, 0.0% nice, 99.7% idle
CPU1 states: 99.9% user, 0.1% system, 0.0% nice, 0.0% idle
Mem: 1545264K av, 1491272K used, 53992K free, 0K shrd, 23536K buff
Swap: 1542232K av,
26592K used, 1515640K free 1222088K cached
PID USER
32021 fcsl
6997 fcsl
. . .
PRI
17
9
NI SIZE RSS SHARE STAT %CPU %MEM
0 20292 18M 3932 R
99.9 1.2
0 3072 3068 1928 S
0.1 0.1
TIME COMMAND
0:10 guimain
0:00 xterm
Operating System Performance
• Common Problems cont’d.:
– All time spent in Kernel processes
• If the machine spends a large amount of time in
system mode, then a process is probably wasting
system calls.
• Use `ps axlw` to see what function call a process
is running at any given time
• May be able to reduce the impact of the system
calls by adding memory or upgrading the CPU.
Operating System Performance
• Common Problems cont’d.:
– Network Problems
• Improve Service Speed (time to respond to a
query)
– Fewer concurrent connections, which reduces the server load.
• DNS Lookup Time
– Trying to find the hostname associated with an IP address can
be slow.
– Process is usually idle while this lookup occurs.
– As more requests arrive, more process will be waiting
(sleeping)
– Can check with `ps axlw` for processes with an “S” or with
`strace –p`.
Operating System Performance
• Common Problems cont’d.:
– Kernel Problems
• System may spend all of its time in
system mode for no reason.
• Can try to use kernel profiling to find the
slowest kernel function.
• Change lilo.conf so “profile=2” is passed
to the kernel so that the kernel will keep
statistics.
Operating System Performance
• Common Problems cont’d.:
For example:
image=/boot/2.2.7-sard
label=linux
append="profile=2“
– Make sure/etc/System.map matches your
kernel, run lilo, and reboot.
– Use `readprofile` to view statistics on each
of the kernel's functions to find the
slowest.
Operating System Performance
• Hardware Tuning
– Use hdparm to optimize IDE hard drives.
– Software raid may improve performance.
– Tune the disk I/O elevators
• Useful if system continuously does a large amount of disk
I/O
• Changes how long the I/O scheduler will let a request sit in
the queue before it has to be handled.
• I/O scheduler collapses some requests together, which can
increase throughput.
Operating System Performance
• Network Tuning
– Try to use 100BTX instead on 10BTX
– If building a router, check the "Optimize as
router" box in the kernel configuration.
– ip_frag_low_thresh and ip_frag_high_thresh
should be increased in some
situations, especially NFS and Samba servers.
• Server takes far more connections and puts them
together in memory before having to ditch their
fragments.
• Can safely increase these to several megabytes
without any problems.
Operating System Performance
• Tuning for particular applications:
• The following slides provide more detailed tips
for particular applications
• File Server
• Web Server
• Mail Server
Operating System Performance
• Tuning a File Server:
• Increase default packet sizes used by NFS clients
• With larger packet sizes, fewer requests needed
to transfer the same amount of data.
For example:
mount -o rsize=8192,wsize=8192 -t nfs host:/dir
/mnt/point
Or on a Samba server in smb.conf :
socket options = SO_SNDBUF=4096 SO_RCVBUF=4096
Operating System Performance
• Tuning a File Server:
– Use larger block sizes
• Helpful for file systems dedicated to fairly large
files
• Increase the block size from 1024 bytes to
4096 bytes.
• Leads to less fragmentation, faster fsck’s,
faster deletes, and faster raw read speed, due
to the reduced number of seeks
Operating System Performance
• Tuning a File Server:
– Unfortunately, must re-format the drive:
mke2fs -b 4096 -m 1 /dev/whatever
where the –m 1 adds an additional improvement
by only reserving 1% of the drive for root instead
of 5%.
– Must be careful, though, if there are many small
files – this can lead to much wasted disk space.
Operating System Performance
• Tuning a File Server:
• Don’t use the atime attribute
 This records the last access time (separate
from the last modified time) and is not very
useful.
 This leads to significant improvements when
running `find` as well as accessing frequently
changing files like the contents of
/var/spool/news
Operating System Performance
• Tuning a File Server:
– To turn off the atime attribute:
chattr -R +A /var/spool (for a whole directory
tree)
Or mount an entire partition with no access time
updating, by editing /etc/fstab:
# device mount point type options dump frequency fsck pass number
/dev/sdb1 /var/spool/news ext2 defaults,noatime 1 2
Operating System Performance
• Tuning a Web Server:
– Apache is a web server used by Linux.
– MaxRequestsPerChild should be set high (10000).
• This sets the number of requests a server child process
will handle before it dies.
• If this value is too low, apache will be constantly
restarting httpds.
– Increase the number of available httpd processes
• Default is 256, which can be exceeded
• add -DHARD_SERVER_LIMIT=4000 to CFLAGS in Configuration
• Or modify the apache source code
– This may require modifying the kernel to support
more processes.
Operating System Performance
• Tuning a Web Server:
– Use Efficient Server-side scripting
• If you do a lot of CGI in perl, consider using mod_perl.
• The Apache::Registry module can in most cases speed up your
CGI requests with no actual changes to your code.
• Similar apache modules for most popular scripting languages
(python, Ruby, Php, tcl).
– Work around the process model
• Scalability problem with apache: one process per connection.
• This can require a large amount of resources: RAM, schedular
slots, ability to grab locks, database connections, file descriptors.
Operating System Performance
• Tuning a Web Server:
– Two methods to help:
• Use a dedicated server for static files, if there are lots of
static files.
– Can use a very light apache setup, or something like thttpd, boa,
khttpd, or TUX.
• Use a reverse-proxy
– In a reverse proxy, clients make requests for content in the namespace of the reverse proxy. The reverse proxy then decides where to
send those requests and returns the content as if it was itself the
origin.
– Advantages include content caching, load balancing, and the
prospect of moving slow connections to lighter weight servers.
Operating System Performance
• Email Server Tuning:
– Mail servers comprised of two parts:
• Mail transfer agent, MTA (also called an SMTP server)
• Mail retrieval agent, MRA
– Limiting factor seems to be disk I/O
• MTAs queue messages to a queue directory (sendmail uses
/var/spool/mqueue)
• MRAs continually access the mail store (sendmail uses
/var/spool/mail)
• Put these directories on different filesystems that are on different
disks to minimize the MTA and MRA from fighting for bandwidth
for the same set of disks.
Operating System Performance
• Email Server Tuning
– MTA Tuning:
– Don’t allow mail server to be used as an open relay
• Only allow outgoing messages if from correct LAN/WAN
• Otherwise, spammers will use the ail server to deliver
their messages.
– MRU Tuning:
• IMAP vs. POP3
– POP3 is older and faster
– IMAP is newer and more complex (more features), so slower.
• 3 ways to store mail: single file, directory hierarchy, and database
– Performance depends on how the particular system is used.
Operating System Performance
• Repeat of History
– In April 2003, Microsoft had VeriTest run another
benchmark, comparing Windows Server 2003 and
Linux
– Results claimed Windows 2003 was about twice as
fast as a File Server.
– Linux community again up in arms about validity
of results
• Again, much support for Microsoft to fine tune Windows
Server 2003, with little to no support to tune Linux.
Operating System Performance
• Repeat of History
Results of Veritest benchmarks:
Operating System Performance
• Caveat
– Most of these performace improvements taken
from http://linuxperf.nl.linux.org
– There are no dates on this website, so cannot tell
how recent or up-to-date these recommendations
are (though it says it was created as a direct result
of the Mindcraft benchmarks in 1999).
– In newer versions of Linux, some of these
recommendations may have already been
implemented by default.
Operating System Performance
• Conclusions of these benchmarks
– Highly tuned Microsoft Windows products (NT,
Windows Server 2003) may or may not be faster
than Linux essentially straight out of the box.
– Microsoft products only measured faster in
benchmarks that don’t have much applicability to
the real world (though there aren’t many better
alternatives for benchmarking).
– Need to be aware of why Operating Systems
perform well or poorly on benchmarks, to be able
to find ways to improve their performance.
References
• http://www.mindcraft.com/whitepapers/nts4rhlinux.html
(links to results from original benchmark and the open
benchmark).
• http://www.kegel.com/mindcraft_redux.html (lists
several issues responsible for the poor Linux performance
in the benchmarks).
• http://linuxperf.nl.linux.org/ (performance tips for
Linux)
• “Windows 2000 Performance Tuning”, available at
http://www.microsoft.com/windows2000/server/ev
aluation/performance/reports/perftune.asp
References
• “Microsoft Windows Server 2003 vs. Linux
Competitive File Server Performance Comparison,”
available at:
http://www.veritest.com/clients/reports/microsoft/
default.asp
• http://www.linuxworld.com/story/32673.htm
(response to above comparison).
• http://perl.apache.org/ (information on mod_perl)
• http://people.redhat.com/alikins/system_tuning.html
(system tuning Linux)