Download Lecture 6

CS 245
Lecture 7:
More on Operating Systems
*nix Shell Scripting
John Hurley
Cal State LA
Course Schedule
• Part 1 of the midterm will be administered after
tonight's lecture. It consists of two Assembly
programming questions that are similar to
problems from the last two labs.
• There is no new lab for this week.
• Next week's quiz will be long and will cover
various operating systems concepts. Review the
material from lectures 4 on.
• Next week's lab will cover shell programming
(introduced later in this lecture). Part II of the
midterm, administered in week 8, will be very
much like the problems in next week's lab. I
recommend that you experiment with shell
programs this week while you are
procrastinating from studying for your other
midterms.
Operating System Services
•
•
Operating systems provide an environment for execution of programs and
services to programs and users
One set of operating-system services provides functions that are helpful to
the user:
– User interface - Almost all operating systems have a user interface (UI).
• Varies between Command-Line (CLI), Graphics User Interface
(GUI), Batch
– Program execution - The system must be able to load a program into
memory and to run that program, end execution, either normally or
abnormally (indicating error)
– I/O operations - A running program may require I/O, which may
involve a file or an I/O device
Operating System Services (Cont.)
•
One set of operating-system services provides functions that are helpful to the user
(Cont.):
– File-system manipulation - The file system is of particular interest. Programs
need to read and write files and directories, create and delete them, search
them, list file Information, permission management.
– Communications – Processes may exchange information, on the same
computer or between computers over a network
• Communications may be via shared memory or through message passing
(packets moved by the OS)
– Error detection – OS needs to be constantly aware of possible errors
• May occur in the CPU and memory hardware, in I/O devices, in user
program
• For each type of error, OS should take the appropriate action to ensure
correct and consistent computing
• Debugging facilities can greatly enhance the user’s and programmer’s
abilities to efficiently use the system
Operating System Services (Cont.)
•
Another set of OS functions exists for ensuring the efficient operation of the system
itself via resource sharing
– Resource allocation - When multiple users or multiple jobs running
concurrently, resources must be allocated to each of them
• Many types of resources - CPU cycles, main memory, file storage, I/O
devices.
– Accounting - To keep track of which users use how much and what kinds of
computer resources
– Protection and security - The owners of information stored in a multiuser or
networked computer system may want to control use of that information,
concurrent processes should not interfere with each other
• Protection involves ensuring that all access to system resources is
controlled
• Security of the system from outsiders requires user authentication,
extends to defending external I/O devices from invalid access attempts
A View of Operating System Services
Operating-System Operations
• Dual-mode operation allows OS to protect
itself and other system components
– User mode and kernel mode
– Mode bit provided by hardware
• Provides ability to distinguish when system is
running user code or kernel code
• Some instructions designated as privileged, only
executable in kernel mode
• System call changes mode to kernel, return from call
resets it to user
• Increasingly CPUs support multi-mode
operations
– i.e. virtual machine manager (VMM) mode for
guest VMs
Silberschatz, Galvin, and Gagne, 2013
Transition from User to Kernel Mode
• Example: Timer to prevent infinite loop / process hogging
resources
–
–
–
–
–
Timer is set to interrupt the computer after some time period
Keep a counter that is decremented by the physical clock.
Operating system set the counter (privileged instruction)
When counter zero generate an interrupt
Set up before scheduling process to regain control or terminate
program that exceeds allotted time
Silberschatz, Galvin, and Gagne, 2013
Computer-System Architecture
There are two main types of OS architecture.
These are archetypes; many actual operating
systems are hybrids of the two approaches:
• Monolithic: most of the OS runs in kernel
space
• Microkernel: modular design with only a
minimum core os the OS running in kernel
space and many OS services running in user
space
Computer-System Architecture
Microkernel System Structure
Application
Program
File
System
messages
Interprocess
Communication
Device
Driver
user
mode
messages
memory
managment
microkernel
hardware
CPU
scheduling
kernel
mode
Layered Approach
• The operating system is
divided into a number of
layers (levels), each built
on top of lower layers.
The bottom layer (layer
0), is the hardware; the
highest (layer N) is the
user interface.
• With modularity, layers
are selected such that
each uses functions
(operations) and
services of only lowerlevel layers
Virtual Machine
App1 App2 App3
System calls
I/O instructions
Linux
Windows NT
FreeBSD
VMware
VMware
VMware
Calls to simulate I/O
“Real” I/O instructions
Linux
Bare hardware
• First widely used in VM/370 with CMS
• Available today in VMware
– Allows users to run any x86-based OS on top of Linux or NT
• “Guest” OS can crash without harming underlying OS
– Only virtual machine fails—rest of underlying OS is fine
• “Guest” OS can even use raw hardware
– Virtual machine keeps things separated
Ahmed Amer, Ethan L. Miller and Scott A. Brandt, University of Pittsburgh
Computer-System Architecture
• Older systems used a single general-purpose processor
per system
– Most systems have special-purpose processors as well
• Multiprocessors systems growing in use and
importance
– Also known as parallel systems, tightly-coupled systems
– Advantages include:
1. Increased throughput
2. Economy of scale
3. Increased reliability – graceful degradation or fault tolerance
– Two types:
1. Asymmetric Multiprocessing – each processor is assigned a specie
task.
2. Symmetric Multiprocessing – each processor performs all tasks
Silberschatz, Galvin, and Gagne, 2013
Symmetric Multiprocessing Architecture
Silberschatz, Galvin, and Gagne, 2013
Clustered Systems
• Like multiprocessor systems, but multiple systems
working together
– Usually sharing storage via a storage-area network (SAN)
– Provides a high-availability service which survives failures
• Asymmetric clustering has one machine in hot-standby mode
• Symmetric clustering has multiple nodes running applications,
monitoring each other
– Some clusters are for high-performance computing (HPC)
• Applications must be written to use parallelization
– Some have distributed lock manager (DLM) to avoid
conflicting operations
Silberschatz, Galvin, and Gagne, 2013
Clustered Systems
Silberschatz, Galvin, and Gagne, 2013
Protection and Security
• Protection – any mechanism for controlling access of
processes or users to resources defined by the OS
• Security – defense of the system against internal and
external attacks
– Huge range, including denial-of-service, worms, viruses,
identity theft, theft of service
• Systems generally first distinguish among users, to
determine who can do what
– User identities (user IDs, security IDs) include name and
associated number, one per user
– User ID then associated with all files, processes of that user
to determine access control
– Group identifier (group ID) allows set of users to be
defined and controls managed, then also associated with
each process, file
– Privilege escalation allows user to change to effective ID
with more rights
Process Management
• A process is a program in execution. It is a unit of work within
the system. Program is a passive entity, process is an active
entity.
• Process needs resources to accomplish its task
– CPU, memory, I/O, files
– Initialization data
• Process termination requires reclaim of any reusable resources
• Single-threaded process has one program counter specifying
location of next instruction to execute
– Process executes instructions sequentially, one at a time, until
completion
• Multi-threaded process has one program counter per thread
• Typically system has many processes, some user, some operating
system running concurrently on one or more CPUs
– Concurrency by multiplexing the CPUs among the processes /
threads
Silberschatz, Galvin, and Gagne, 2013
Process Management Activities
The operating system is responsible for the following activities in
connection with process management:
• Creating and deleting both user and system
processes
• Suspending and resuming processes
• Providing mechanisms for process
synchronization
• Providing mechanisms for process
communication
• Providing mechanisms for deadlock handling
Silberschatz, Galvin, and Gagne, 2013
Memory Management
• To execute a program all (or part) of the instructions must
be in memory
• All (or part) of the data that is needed by the program
must be in memory.
• Memory management determines what is in memory and
when
– Optimizing CPU utilization and computer response to users
• Memory management activities
– Keeping track of which parts of memory are currently being
used and by whom
– Deciding which processes (or parts thereof) and data to
move into and out of memory
– Allocating and deallocating memory space as needed
Silberschatz, Galvin, and Gagne, 2013
Storage Management
• OS provides uniform, logical view of information
storage
– Abstracts physical properties to logical storage unit - file
– Each medium is controlled by device (i.e., disk drive, tape
drive)
• Varying properties include access speed, capacity, datatransfer rate, access method (sequential or random)
• File-System management
– Files usually organized into directories
– Access control on most systems to determine who can
access what
– OS activities include
•
•
•
•
Creating and deleting files and directories
Primitives to manipulate files and directories
Mapping files onto secondary storage
Backup files onto stable (non-volatile) storage media
Migration of data “A” from Disk to Register
• Multitasking environments must be careful to use most
recent value, no matter where it is stored in the storage
hierarchy
• Multiprocessor environment must provide cache coherency
in hardware such that all CPUs have the most recent value in
their cache
• Distributed environment situation even more complex
– Several copies of a datum can exist
System calls
• Programs want the OS to perform a service
– Access a file
– Create a process
– Others…
• Accomplished by system call
– Program passes relevant information to OS
– OS performs the service if
• The OS is able to do so
• The service is permitted for this program at this time
– OS checks information passed to make sure it’s OK
• Don’t want programs reading data into other programs’
memory!
Ahmed Amer, Ethan L. Miller and Scott A. Brandt, University of Pittsburgh
System Calls
• Programming interface to the services provided by the
OS
• Typically written in a high-level language (C or C++)
• Mostly accessed by programs via a high-level Application
Programming Interface (API) rather than direct system
call use
• Three most common APIs are Win32 API for Windows,
POSIX API for POSIX-based systems (including virtually all
versions of UNIX, Linux, and Mac OS X), and Java API for
the Java virtual machine (JVM)
Note that the system-call names used throughout this
text are generic
Silberschatz, Galvin, and Gagne 2013
Example of System Calls
• System call sequence to copy the contents of
one file to another file
Example of Standard API
System Call Implementation
• Typically, a number associated with each system call
– System-call interface maintains a table indexed
according to these numbers
• The system call interface invokes the intended
system call in OS kernel and returns status of the
system call and any return values
• The caller need know nothing about how the
system call is implemented
– Just needs to obey API and understand what OS will do
as a result call
– Most details of OS interface hidden from programmer
by API
• Managed by run-time support library (set of functions built
into libraries included with compiler)
API – System Call – OS Relationship
System Call Parameter Passing
• Often, more information is required than simply
identity of desired system call
– Exact type and amount of information vary according to
OS and call
• Three general methods used to pass parameters to
the OS
– Simplest: pass the parameters in registers
• In some cases, may be more parameters than registers
– Parameters stored in a block, or table, in memory, and
address of block passed as a parameter in a register
• This approach taken by Linux and Solaris
– Parameters placed, or pushed, onto the stack by the
program and popped off the stack by the operating
system
– Block and stack methods do not limit the number or
length of parameters being passed
Types of System Calls
• Process control
–
–
–
–
–
–
–
–
–
–
create process, terminate process
end, abort
load, execute
get process attributes, set process attributes
wait for time
wait event, signal event
allocate and free memory
Dump memory if error
Debugger for determining bugs, single step execution
Locks for managing access to shared data between
processes
Types of System Calls
• File management
–
–
–
–
create file, delete file
open, close file
read, write, reposition
get and set file attributes
• Device management
–
–
–
–
request device, release device
read, write, reposition
get device attributes, set device attributes
logically attach or detach devices
Types of System Calls (Cont.)
• Information maintenance
– get time or date, set time or date
– get system data, set system data
– get and set process, file, or device attributes
• Communications
– create, delete communication connection
– send, receive messages if message passing model to
host name or process name
• From client to server
– Shared-memory model create and gain access to
memory regions
– transfer status information
– attach and detach remote devices
Types of System Calls (Cont.)
• Protection
– Control access to resources
– Get and set permissions
– Allow and deny user access
Examples of Windows and Unix System Calls
Standard C Library Example
• C program invoking printf() library call,
which calls write() system call
Disable Root SSH Logins
• see a list of the last 20 unsuccessful login attempts:
– (sudo) lastb | head -n 20
• My VM gets hundreds or thousands of failed ssh login attempts
per day. Some of these are caused by configuration mistakes in
applications on other systems, but the fact that they almost all
use the login name root shows that these are mostly attempts
to guess the password and get control of the system.
• Here is how to disable root ssh logins:
– Make sure you have another sudoer set up to do admin work!
– sudo vi /etc/ssh/sshd_config
• change PermitRootLogin from yes to no
• sudo systemctl restart sshd
• There are at least two open source utilities, fail2ban and
denyhosts, that blacklist particular addresses after
several failed login attempts. You can try them out on
your own if you'd like.
grep
• grep searches text for a substring and returns
every line in which the substring occurs
[godzilla@fedora-1gb-sfo1-01 ~]$ cat mary
Mary had
a little lamb
little lamb
little lamb
and also some mint jelly
[godzilla@fedora-1gb-sfo1-01 ~]$ cat mary | grep lamb
a little lamb
little lamb
little lamb
*nix Shell History
• In addition to using the scroll keys to get back to past
commands, the history command is useful for this
purpose. For example, if I need to recall the syntax for
blacklisting an ip address:
[godzilla@fedora-1gb-sfo1-01 ~]$ history | grep DROP
7 sudo iptables -I INPUT -s 183.2.202.255 -j DROP
11 sudo iptables -I INPUT -s 183.2.202.173 -j DROP
20 sudo iptables -I INPUT -s 159.226.63.186 -j DROP
622 history | grep DROP
1005 history | grep DROP
the second to last entry is from a previous time I ran history |
grep DROP; the last one is the current command
Shell Programming
• A Shell like Bash can be viewed as a programming
language. Shell commands and utilities can be seen as
statements. Shells also offer control structures like loops
and conditionals.
• We can write executable files that consist of shell
commands. These are often called shell scripts.
– "script" has many different meanings in programming. In
this case it signifies that these are a) relatively small
programs and b) interpreted, not compiled.
• Programs like this are most often used for administering
the OS, but they can be used for other useful purposes
too.
Shell Programming
• Scripts sometimes have no file name extension and sometimes use the
filename extension .sh. Opinions differ about best practices:
– Scripts are executable, and other *nix executables usually have no filename
extension.
– On the other hand, scripts are also source files, and most other source files have
filename extensions that indicate the programming language
• Scripts usually start with a directive that says which shell to use, including
the path to the bash executable. This allows you to run a shell script from
outside the shell (say, from a GUI or another shell instance)
• For bash, this form is usually correct:
#!/bin/bash
with or without a space after the !
•
•
The directive is sometimes called "shebang" for reasons that should be
obvious.
The rest of a script consists of shell commands, but bash and other shells
include commands that are commonly used only in scripts
–
control flow
•
•
if / then
loops
Shell Programming
#!/bin/bash
echo "Hello, World!"
Shell Programming
No need to compile; you can view the shell as an interpreter
for this scripting language.
However, you do need to set file permissions:
[godzilla@fedora-1gb-sfo1-01 shellprog]$ ./hello
-bash: ./hello: Permission denied
[godzilla@fedora-1gb-sfo1-01 shellprog]$ ll
total 4
-rw-rw-r--. 1 godzilla godzilla 33 Apr 30 18:26 hello
[godzilla@fedora-1gb-sfo1-01 shellprog]$ chmod 700 hello
[godzilla@fedora-1gb-sfo1-01 shellprog]$ ./hello
Hello, World!
Shell Programming
No need to compile; you can view the shell as an interpreter
for this scripting language.
However, you do need to set file permissions:
[godzilla@fedora-1gb-sfo1-01 shellprog]$ ./hello
-bash: ./hello: Permission denied
[godzilla@fedora-1gb-sfo1-01 shellprog]$ ll
total 4
-rw-rw-r--. 1 godzilla godzilla 33 Apr 30 18:26 hello
[godzilla@fedora-1gb-sfo1-01 shellprog]$ chmod 700 hello
[godzilla@fedora-1gb-sfo1-01 shellprog]$ ./hello
Hello, World!
Shell Programming
Set variable values:
• Godzilla=king
no whitespace around the equals sign!
To define a string that includes whitespace, use quotes:
• Godzilla="King of the Monsters"
To dereference a variable, prepend a $ to its name:
[godzilla@fedora-1gb-sfo1-01 shellprog] echo Godzilla
Godzilla
but
[godzilla@fedora-1gb-sfo1-01 shellprog] echo $Godzilla
King of the Monsters
To capture the output from an executable (like a utility program) for a variable value
or in output, use $(executable):
[smogmonster@fedora-1gb-sfo1-01 lab3]$ currtime=$(date)
[smogmonster@fedora-1gb-sfo1-01 lab3]$ echo $currtime
Sun May 1 21:32:13 PDT 2016
Shell Programming
Two more simple examples:
#!/bin/bash
Godzilla="King of the Monsters"
echo Hello, $Godzilla!
output:
[godzilla@fedora-1gb-sfo1-01 shellprog] ./hello
Hello, King of the Monsters!
#!/bin/bash
name=$(whoami)
echo Hello, $name!
output (logged in as godzilla):
[godzilla@fedora-1gb-sfo1-01 shellprog] ./hello
Hello, godzilla!
if…then…elif…else…fi
#!/bin/bash
g_epithet="King of the Monsters"
s_epithet="Smoggy Lord"
identity=$(whoami)
if [ $identity = "godzilla" ]
then
echo "Greetings, $g_epithet"
elif [ $identity = "smogmonster" ]
then
echo "Greetings, $s_epithet"
else
echo "Greetings, lesser monster"
fi
output (logged in as godzilla):
Greetings, King of the Monsters
output (logged in as smogmonster):
Greetings, Smoggy Lord
output (logged in as mothra):
Greetings, lesser monster
The style of the indentation and line breaks follows *nix convention, though it may seem awkward. Note that it is easy to see
where the block ends.
Quotes
Quotation mark usage in *nix is hard to get used to. The last example would
also work like this:
#!/bin/bash
g_epithet="King of the Monsters"
s_epithet="Smoggy Lord"
identity=$(whoami)
if [ $identity = godzilla ]
then
echo Greetings, $g_epithet
elif [ $identity = smogmonster ]
then
echo Greetings, $s_epithet
else
echo Greetings, lesser monster
fi
Quotes
The example would not work like this:
#!/bin/bash
g_epithet="King of the Monsters"
s_epithet="Smoggy Lord"
identity=$(whoami)
if [ $identity = godzilla ]
then
echo 'Greetings, $g_epithet'
elif [ $identity = smogmonster ]
then
echo 'Greetings, $s_epithet'
else
echo Greetings, lesser monster
fi
Variable names are not dereferenced ("expanded") when inside strings
defined with single quotes.
While…do…done
evaluate a mathematical expression: $((expression))
inequalities are abbreviated as -lt, -gt, -le, -ge, etc.
#!/bin/bash
counter=4
factorial=1
while [ $counter -gt 0 ]
do
factorial=$(( $factorial * $counter ))
counter=$(( $counter - 1 ))
done
echo $factorial
http://www.cyberciti.biz/faq/bash-while-loop/
Command Line Parameters
Command line parameters are accessed using 1-based indexing and the $ operator:
$1, $2, etc.
#!/bin/bash
counter=$1
factorial=1
while [ $counter -gt 0 ]
do
factorial=$(( $factorial * $counter ))
counter=$(( $counter - 1 ))
done
echo $factorial
output:
[godzilla@fedora-1gb-sfo1-01 shellprog]$ ./factorial 5
120
http://www.cyberciti.biz/faq/bash-while-loop/
Command Line Parameters
#!/bin/bash
for ((i=$1; i<=$2; i++))
do
echo $i
done
output:
[godzilla@fedora-1gb-sfo1-01 shellprog]$ ./fordemo 3 9
3
4
5
6
7
8
9