Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
File Systems Tanenbaum Chapter 4 Silberschatz Chapters 10, 11, 12 cs431-cotter 1 File Systems Essential requirements for long-term information storage: • • • It must be possible to store a very large amount of information. The information must survive the termination of the process using it. Multiple processes must be able to access the information concurrently. Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639 cs431-cotter 2 File Structure • None: – File can be a sequence of words or bytes • Simple record structure: – Lines – Fixed Length – Variable Length • Complex Structure: – Formatted documents – Relocatable load files • Who decides? cs431-cotter 3 File Systems Think of a disk as a linear sequence of fixed-size blocks and supporting reading and writing of blocks. Questions that quickly arise: • • • How do you find information? How do you keep one user from reading another’s data? How do you know which blocks are free? cs431-cotter Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639 4 File Naming Figure 4-1. Some typical file extensions. cs431-cotter Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639 5 File Access Methods • Sequential Access – Based on a magnetic tape model – read next, write next – reset • Direct Access – – – – cs431-cotter Based on fixed length logical records read n, write n position to n relative or absolute block numbers 6 File Structure Figure 4-2. Three kinds of files. (a) Byte sequence. (b) Record sequence. (c) Tree. cs431-cotter Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639 7 File Types cs431-cotter Figure 4-3. (a) An executable file. (b) An archive. Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639 8 File Attributes Figure 4-4a. Some possible file attributes. cs431-cotter Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639 9 File Operations The most common system calls relating to files: • • • • • • Create Delete Open Close Read Write • • • • • Append Seek Get Attributes Set Attributes Rename cs431-cotter Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639 10 Example Program Using File System Calls (1) ... Figure 4-5. A simple program to copy a file. cs431-cotter Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639 11 Example Program Using File System Calls (2) Figure 4-5. A simple program to copy a file. cs431-cotter Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639 12 Directory Structure • Collection of nodes containing information on all files F2 F1 cs431-cotter F4 F5 F3 13 Information in a Device Directory • • • • • • • • • File name: File Type: Address: Current Length Maximum Length Date Last accessed (for archiving) Date Last updated (for dumping) Owner ID Protection information cs431-cotter 14 Directory Operations • • • • • • Search for a file Create a file Delete a file List a directory Rename a file Traverse the file system cs431-cotter 15 Objectives for a Directory System • Make it efficient – It should be easy to locate a file quickly • Make file (and directory) naming convenient – Allow 2 users to have the same name for different files – Allow the same file to have more than 1 name • Allow logical grouping of files – All word processing files together – All c++ files together – etc. cs431-cotter 16 Alternative Directory Structures • Single-Level Directory cat bo a test data mail cont hex word calc • Issues: – Naming – Grouping cs431-cotter 17 Alternative Directory Structures • Two-Level Directory User1 cs431-cotter User2 User3 18 Tree-Structured Directory cs431-cotter 19 Path Names Figure 4-8. A UNIX directory tree. cs431-cotter Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639 20 File System Structure • A data structure on a disk that holds files • Disk Storage attributes: – Allows direct access to any given block of information – Supports re-writing in place (copy block, modify, rewrite) • Organized in Layers: – – – – – – Application Programs Logical file system file-organization module basic file system I/O control devices cs431-cotter 21 File System Organization • Devices – Peripheral equipment- disk drives, tapes, etc. • I/O Control – device drivers and interrupt handlers – Communicates directly with peripheral devices – Starts and closes out I/O request • Basic File System – Requires knowledge of physical device (track, cylinder, head, etc.) – Manages data transfer at block level (buffering, placement) cs431-cotter 22 File System Organization • File-organization Module – Maps logical block structure of file to physical block structure of disk – Includes free space manager • Logical file system – Manages file directory information – Provides security, protection • Applications – Maintains basic data about the file – Allows users to access file information cs431-cotter 23 File System Structure • Access to File – file control block (generic) – file descriptor (UNIX) – file handle (Windows) • File System Requirements – – – – cs431-cotter boot mechanism file system information file information file data 24 Allocation Methods Contiguous Allocation • Each file occupies a set of contiguous blocks on the disk. • Number of blocks needed identified at file creation – May be increased using file extensions • Advantages: – Simple to implement – Good for random access of data • Disadvantages – Files cannot grow – Wastes space cs431-cotter 25 Contiguous Allocation FileA 0 1 2 3 4 5 6 7 8 FileB 9 10 11 12 13 14 15 16 17 18 FileC 19 20 21 22 23 FileE 24 25 26 27 FileD 28 29 30 31 33 34 cs431-cotter 32 File Allocation Table File Name FileA FileB FileC FileD FileE Start Block Length 2 9 18 30 26 3 5 8 2 3 26 Allocation Methods Linked Allocation • Each file consists of a linked list of disk blocks. data ptr data ptr data ptr data Null • Advantages: – Simple to use (only need a starting address) – Good use of free space • Disadvantages: – Random Access is difficult cs431-cotter 27 Linked Allocation 0 FileB 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 cs431-cotter File Allocation Table File Name ... FileB ... Start Block End ... 1 ... ... 28 ... 28 Linked Allocation 0 FileB 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 cs431-cotter File Allocation Table File Name ... FileB ... Start Block End ... 1 ... ... 28 ... 29 Allocation Methods Indexed Allocation • Collect all block pointers into an index block. Index Table • Advantages: – Random Access is easy – No external fragmentation • Disadvantages – Overhead of index block cs431-cotter 30 Indexed Allocation File Allocation Table 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 cs431-cotter File Name Index Block Jeep 24 1 3 8 14 28 31 Indexed Allocation File Allocation Table 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 cs431-cotter File Name Index Block Jeep 24 1 8 3 14 28 32 UNIX File System File Types • • • • • cs431-cotter Regular File Directory Block-oriented Device Character-oriented Device Symbolic Link 33 UNIX File Ownership read write execute owner 1 1 0 group 1 0 0 others 1 0 0 (for directories: read, write, search) -rwx-----drwxr-xr-x cs431-cotter 1 rsmith 2 rsmith is is 785 Feb 11 1994 send_exe 512 Apr 16 13:49 sysmon_server 34 File Access Model • open -- read -- write -- close fdesc = open (*filename, r / w / a / +); n = read (fdesc, buff, 24); n = write (fdesc, buff, 24); lseek (fdesc, 100L, L_SET); (L_SET, L_CUR, L_END) close (fdesc); cs431-cotter 35 Concurrent File Access • From separate processes – OPEN generates a new file descriptor. – Allows independent access to the same file. – No inherent blocking in access • From parent / child processes – fork duplicates all active file descriptors. – all processes access a common “current” pointer • File locking – flock, lockf cs431-cotter 36 UNIX File Sharing - unrelated processes process table entry fd 0 fd flags ptr File Table file status flags current file offset v-node ptr V-node Table v-node info I-node info file size process table entry fd 0 fd flags cs431-cotter ptr file status flags current file offset v-node ptr 37 UNIX File Sharing - related processes process table entry fd 0 fd flags ptr File Table file status flags current file offset v-node ptr V-node Table v-node info I-node info file size process table entry fd 0 fd flags cs431-cotter ptr file status flags current file offset v-node ptr 38 UNIX i-node mode owners(2) timestamps(3) size block count data data data direct blocks single indir double indir triple indir cs431-cotter 39 UNIX i-node mode owners(2) timestamps(3) size block count data data data direct blocks single indir double indir : : data data triple indir cs431-cotter 40 UNIX i-node mode owners(2) timestamps(3) size block count data data data direct blocks single indir double indir triple indir cs431-cotter : : data data : : : : : : data data data data 41 I-node File Sizes 1 KB block 2KB block 4KB block Direct 12 k bytes 24 k bytes 48 k bytes Single indirect Double indirect Triple indirect 256 k bytes 1 M bytes 4 M bytes 64 M bytes 512 M bytes 16 G bytes 256 G bytes 2 G bytes cs431-cotter 2 T bytes 42 Disk Space Management Block Size (1) Figure 4-20. Percentage of files smaller than a given size (in bytes). cs431-cotter Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639 43 Disk Drive Layout (simplified) Disk Drive Partition Block Group partition Block group partition Block group Block group directory blocks and data blocks I-list I-node cs431-cotter partition I-node I-node I-node 44 Sample filesystem: Creating the directory “testdir” directory block i-list i-node cs431-cotter i-node 1267 i-node 2549 i-node 45 Sample filesystem: Creating the directory “testdir” directory block i-list i-node i-node 1267 i-node 2549 i-node 1267 cs431-cotter • 46 Sample filesystem: Creating the directory “testdir” directory block i-list i-node i-node 1267 i-node 2549 i-node 1267 2549 cs431-cotter • testdir 47 Sample filesystem: Creating the directory “testdir” directory block i-list i-node i-node 1267 i-node 2549 i-node 1267 2549 cs431-cotter • testdir 48 Sample filesystem: Creating the directory “testdir” directory block i-list i-node i-node 1267 i-node 2549 directory block i-node 2549 1267 • • • 1267 2549 cs431-cotter • testdir 49 Sample filesystem after creating the directory “testdir” directory block i-list i-node i-node 1267 i-node 2549 directory block i-node 2549 1267 • • • 1267 2549 cs431-cotter • testdir 50 Viewing directory structure • ls –ai / –2 –: – 1730 . home • ls –ai /home – – – – 1730 2 : 199 cs431-cotter . .. rcotter • ls –ai /home/rcotter – – – – 199 . 1730 . . : 241224 cs431 • ls –ai ~/cs431 – – – – 241224 . 199 .. : 245436 exec.cpp 51 Shared Files (1) Figure 4-16. File system containing a shared file. ln –s shared_file new_link cs431-cotter Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639 52 Shared Files (2) Figure 4-17. (a) Situation prior to linking. (b) After the link is created. (c) After the original owner removes the file. cs431-cotter Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639 53 Linux File System Structure • Linux uses a Virtual File System (VFS) – Defines a file object – Provides an interface to manipulate that object • Designed around OO principles – File system object – File object – Inode object (index node) • Primary File System - ext2fs – Supports (or maps) several other systems (MSDOS, NFS (network drives), VFAT (W95), HPFS (OS/2), etc. cs431-cotter 54 Virtual Filesystem • A kernel software layer that handles all system calls related to a standard UNIX filesystem. • Supports: – Disk-based filesystems • IDE Hard drives (UNIX, LINUX, SMB, etc.) • SCSI Hard drives • floppy drives – Network filesystems • remotely connected filesystems – Special filesystems • /proc cs431-cotter 55 VF Example inf = open (“/floppy/test”, O_RDONLY, 0); outf = open (“/tmp/test”, O_WRONLY|O_CREATE|O_TRUNC, 0600); do { cnt = read(inf, buf, 4096); write (outf, buf, cnt); } while (cnt); close (outf); close (inf); cp VFS ext2 cs431-cotter MS-DOS 56 Virtual file system Model • Superblock object – metadata about a mounted filesystem • inode object – general information about a specific file • file object – information about process and open file interaction • dentry object – directory entry information cs431-cotter 57 Virtual Filesystem and Processes disk file superblock Process 1 inode object file object dentry object Process 2 file object Process 3 file object cs431-cotter dentry object 58 Ext2fs History • Minux • extfs • ext2fs - 1994 cs431-cotter 59 Ext2fs Blocks • 1k default • 2k, 4k possible • choice based on file sizes cs431-cotter 60 Ext2fs Disk Data Structures Boot Block Super Block Block Group 0 Group Data block Inode Inode Descriptors Bitmap Bitmap Table Block Group N Data Blocks • Each Block Group contains: – – – – – – A copy of the Super Block A copy of the group of block group descriptors A data block bitmap (for this group) An inode bitmap (for this group) A group of inodes (for this group) Data blocks cs431-cotter 61 Disk Data Structures • SuperBlock – Contains metadata about the block group – # free blocks, – # free inodes, – block size, – times (last mount, last write), – check status, – blocks to pre-allocate, – alignments, etc. (~38 items) cs431-cotter 62 Disk Data Structures • Block Groups – Block group limited to 8 * block size by data block bitmap. – 1k block limited to 8,192 blocks or 8 mbytes per block group. 4k block = 128 mbytes • Group descriptor - 24 bytes per group – block numbers for bitmaps, start of tables, etc. cs431-cotter 63 Disk Data Structures • Inode Table – Inodes - 128 bytes – File type and access rights – owner – length – timestamps (last access, last change, etc.) – hard links counter – pointers to data blocks, – etc. cs431-cotter 64 Disk Data Structures • Items cached in memory on filesystem mount – Superblock - always cached – Group descriptor - always cached – Block bitmap - fixed limit - most recent only – Inode bitmap - fixed limit - most recent only – Inode - dynamic – Data block - dynamic cs431-cotter 65 File Blocks Grouped • • • • Physically and logically Inodes distributed throughout disk Data blocks allocated close to inode Pre-allocation of free blocks to minimize frag. cs431-cotter 66 Summary Features • • • • Fast Symbolic Links pre-allocation of data blocks file-updating strategy immutable files or append only files cs431-cotter 67 Journal File Systems • Designed to speed file system recovery from system crashes. • Traditional systems use an fs recovery tool (e.g. fsck) to verify system integrity • As file system grows, recovery time grows. • Journal File Systems track changes in meta-data to allow rapid recovery from crashes cs431-cotter 68 Journal File System • Uses a version of a transaction log to track changes to file system directory records. • If a system crash occurs, the transaction log can be reviewed back to a check point to verify any pending work (either undo or redo). • For large systems, recovery time goes from hours or days to seconds. cs431-cotter 69 Journal File System - Examples • XFS – – – – – – – Commercial port by SGI (IRIX OS) Based on 64 bit system (ported to 32 bit) Supports large systems 2TB -> 9 million TB Uses B+trees for improved performance Journals file system meta-data Supports Quotas, ACLs. Supports filesystem extents (contiguous blocks) cs431-cotter 70 Journal File System - Examples • JFS – – – – – – – Commercial Port by IBM. Used in OS/2 64 bit system ported to 32 bits. Journal support of meta-data. System Size 2TB -> 32PB Built to scale on SMP architectures Uses B+trees for improved performance Supports use of extents. cs431-cotter 71 Journal File System - Examples • ReiserFS – – – – – – Designed originally for Linux (32 bit system) Supports file systems to 2TB -> 16TB Journal support of meta-data Btree structure supports large file counts Supports block packing for small files No fixed inode allocation - more flexible. cs431-cotter 72 Journal File System - Examples • Ext3fs – Simple extension of ext2fs that adds journaling – All virtual file system operations are journaled. – Other limitations of ext2fs remain. cs431-cotter 73 Journal File System - Examples • ext4fs – Next generation of file system development, incorporating improvements and changes to ext3fs. – Journalling filesystem • Improvements is journal checksumming – Larger file systems – 1 MTB (exabyte) • Larger files – 16 TB – Delayed block allocation, multi-block allocator • Allows files to be written as contiguous sequences of blocks: improves read performance of streaming data files. • Poses the risk of lost data if system crashes before data is written. – Use of extents • Large (<= 128 MB) contiguous physical blocks. cs431-cotter 74 Free Space Management • Bit Vector management 0 1 1 0 ..... 1 • One bit for each block – 0 = free; 1 = occupied • Use bit manipulation commands to find free block • Bit vector requires space – block size = 4096 = 2 12 – disk size = 1 gigabyte = 2 30 cs431-cotter – bits = 2 (30-12) = 2 18 = 32k bytes 75 Free Space Management • Bit vector (advantages): – Easy to find contiguous blocks • Bit vector (disadvantages): – Wastes space (bits allocated to unavailable blocks) • Issues: – Must keep bit vector on disk (reliability) – Must keep bit vector in memory (speed) – cannot allow memory != disk (memory = 1, disk = 0) cs431-cotter 76 Free Space Management • Linked List management – Use linked list to identify free space • Advantages: – no wasted space • Disadvantages: – harder to identify contiguous space. • Issues: – Must protect pointer to free list cs431-cotter 77 Free Space Management • Grouping of blocks Boot block cs431-cotter File File system descriptors descriptor 78 Reducing Disk Arm Motion Figure 4-29. (a) I-nodes placed at the start of the disk. (b) Disk divided into cylinder groups, each with its own blocks and i-nodes. cs431-cotter Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639 79 Keeping Track of Free Blocks (1) Figure 4-22. (a) Storing the free list on a linked list. (b) A bitmap. cs431-cotter 80 Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639 Disk Quotas Figure 4-24. Quotas are kept track of on a per-user basis in a quota table. cs431-cotter Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639 81 File System Backups (1) Backups to tape are generally made to handle one of two potential problems: • • Recover from disaster. Recover from stupidity. cs431-cotter Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639 82 File System Backups (2) Figure 4-25. A file system to be dumped. Squares are directories, circles are files. Shaded items have been modified since last dump. Each directory and file is labeled by its i-node number. cs431-cotter Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639 83 File System Backups (3) Figure 4-26. Bitmaps used by the logical dumping algorithm. cs431-cotter Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639 84 File System Consistency Figure 4-27. File system states. (a) Consistent. (b) Missing block. (c) Duplicate block in free list. (d) Duplicate data block. cs431-cotter Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639 85 Caching (1) Figure 4-28. The buffer cache data structures. cs431-cotter Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639 86 Caching (2) • Some blocks, such as i-node blocks, are rarely referenced two times within a short interval. • Consider a modified LRU scheme, taking two factors into account: •Is the block likely to be needed again soon? •Is the block essential to the consistency of the file system? cs431-cotter Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639 87 Example File Systems • CD-ROM – ISO 9660 – Rock Ridge – Joliet • MS-DOS cs431-cotter 88 The ISO 9660 File System Figure 4-30. The ISO 9660 directory entry. cs431-cotter Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639 89 Rock Ridge Extensions Rock Ridge extension fields: • • • • • • • • PX - POSIX attributes. PN - Major and minor device numbers. SL - Symbolic link. NM - Alternative name. CL - Child location. PL - Parent location. RE - Relocation. TF - Time stamps. cs431-cotter Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639 90 Joliet Extensions Joliet extension fields: • • • • Long file names. Unicode character set. Directory nesting deeper than eight levels. Directory names with extensions cs431-cotter Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639 91 The MS-DOS File System (1) Figure 4-31. The MS-DOS directory entry. cs431-cotter Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639 92 The MS-DOS File System (2) Figure 4-32. Maximum partition size for different block sizes. The empty boxes represent forbidden combinations. cs431-cotter Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639 93 Summary • • • • • File Structures Directory Structures File System Implementation File System Management Example File Systems cs431-cotter 94 Questions • • • • • • Two primary access methods are used to manage files. Discuss these two methods, including which types of physical devices each is normally associated with. What is the difference between a tree structured file directory and an acyclic graph directory? Please give an example Operating System used today that uses each. Discuss 3 ways in which files might be allocated to file stores (disks). What are the advantages or disadvantages of each? How might a file system manage its free space so that it can easily assign space for a new file when needed? Give at least 2 approaches. What can an OS designer to improve the efficiency of a file system? What factors should be considered? How do those affect the efficiency of the system? Discuss at least 2 mechanisms that might be used to ensure the recovery of information stored in a file system.95 cs431-cotter