Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download File Management - Cal Poly computer science
Document related concepts
Object storage wikipedia , lookup
Library (computing) wikipedia , lookup
MTS system architecture wikipedia , lookup
Plan 9 from Bell Labs wikipedia , lookup
Windows NT startup process wikipedia , lookup
Commodore DOS wikipedia , lookup
Spring (operating system) wikipedia , lookup
Burroughs MCP wikipedia , lookup
File locking wikipedia , lookup
Transcript
Course Overview Principles of Operating Systems Introduction Computer System Structures Operating System Structures Processes Process Synchronization Deadlocks CPU Scheduling © 2000 Franz Kurfess Memory Management Virtual Memory File Management Security Networking Distributed Systems Case Studies Conclusions File Systems 1 Chapter Overview File Management Motivation Objectives Files structure operations implementation Directories structure operations implementation © 2000 Franz Kurfess Secondary Storage Management file allocation free space management Disk Scheduling Important Concepts and Terms Chapter Summary File Systems 2 Motivation large amounts of information must be stored on a permanent basis in the computer system convenient access to stored information is needed by the user files provide a convenient mechanism for storing closely related information items the file system helps with the organization and management of files © 2000 Franz Kurfess File Systems 3 Objectives be aware of the need for large capacity, persistent storage of information understand the concept, usage, and implementation of files know the basic techniques to implement files and directories understand the usage of hard disks as storage medium for files and directories © 2000 Franz Kurfess File Systems 4 Necessity of a File System storage of large amounts of information persistent storage information must continue to exist after the termination of a process convenient access to data should adapt to the needs of the user some items must be accessible to different users efficient utilization of resources here in particular hard disk as instance of secondary storage © 2000 Franz Kurfess File Systems 5 File System collection of algorithms and data structures that perform the translation from logical file operations (system calls) to actual physical storage of information more convenient abstraction than the underlying devices not strictly necessary for the operation of a computer system not considered part of the kernel in a modern OS © 2000 Franz Kurfess File Systems 6 File System Diagram directories files © 2000 Franz Kurfess File Systems 7 File System Objectives convenient storage and manipulation of data support of I/O devices standard user interface consistency of data performance optimization system user multiple users integration with networks © 2000 Franz Kurfess File Systems 8 File System Architecture. generally layered user level file management level I/O system level physical devices relies on standard I/O device management device © 2000 Franz Kurfess drivers, DMA, interrupts, etc. File Systems 9 Requirements create, modify, delete files for the user organization and arrangement of files directories properties access control for a user’s own files for files of other users back up and recovery of files © 2000 Franz Kurfess File Systems 10 Elements of a File System a file system consists of files directory structure affiliated system calls, management routines, scheduling algorithms, device drivers, etc. © 2000 Franz Kurfess File Systems 11 File System Usage user: interactively through the user interface file manager system programs user via file programs system calls system utilities special system programs that help with the administration and usage of the file system compression,encryption system calls standard system interface used by all programs for accessing file system services © 2000 Franz Kurfess File Systems 12 File System Levels T he Layers of the File System. The User System Utilities cd ls mkdir System Calls The Filesystem. open read close Logical I/O System Basic I/O Supervisor Physical I/O System Device Drivers. © 2000 Franz Kurfess File Systems 13 File System Levels user process initiates file file operations via system calls system translates the logical file operations requested by users via system calls into instructions to the I/O system level I/O system OS portion responsible for managing I/O devices placement of blocks onto devices, access control, I/O buffers, scheduling decisions device driver communication © 2000 Franz Kurfess with the actual physical device File Systems 14 File file concept logical file structure file size and granularity file naming © 2000 Franz Kurfess File Systems 15 File Concept a file is a named collection of related information files often have an internal structure imposed by the items contained in them information about a file is referred to as meta-information files are usually stored on secondary storage most often on hard disks from the systems perspective, a file is a sequence of bytes, possibly with an internal structure © 2000 Franz Kurfess File Systems 16 Logical File Structure the internal structure of a file can be handled by the operating systems or by applications using the file generic structures are flat file sequence of bytes records each record contains one item, which can consist of several subitems records can have fixed or variable length © 2000 Franz Kurfess File Systems 17 File Size and Granularity user and programmer want unlimited file size impractical due to size restrictions of the storage media difficult to manage in practice, there are file size limits so large that it is sufficient for almost all cases frequently larger than the storage capacity of current technology file granularity the internal structure of storage media allow file sizes only in certain increments, e.g. disk blocks often not visible to the user © 2000 Franz Kurfess File Systems 18 File Meta-Data information about a file file name file type file size owner access privileges creation, access and modification dates internal file management information accessible © 2000 Franz Kurfess via system calls and system programs File Systems 19 File Naming file names are usually sequences of (ASCII) characters special characters (blank, ^, &, !, ?, etc) are sometimes not allowed the length of a file name is often restricted in modern OSs, files are organized in hierarchical file systems this leads to path names for the access to files absolute path names start from the root (top level) and specify the intermediate levels explicitly relative path names start at the current level (directory) and indicate the location and name of the file relative to © 2000 Franz Kurfess File Systems 20 Access Methods random access arbitrary items in a file can be accessed directly based on the disk model used for most files sequential access the items within a file can only be accessed one after the other to access a particular item, all its predecessors have to be accessed first based on the tape operation model © 2000 Franz Kurfess File Systems 21 File Operations create delete open close read write append seek get attributes set attributes rename © 2000 Franz Kurfess File Systems 22 File Structure logical structure from the user’s perspective, a file consists of records records contain fields physical structure the system handles a file as a sequence of blocks blocks are the unit of information transfer from disk usually fixed size, e.g. 512 Bytes, 1KB, 4KB © 2000 Franz Kurfess File Systems 23 File Implementation logical file organization internal file structure according to access method of contents determined by the user or application physical organization placement and tracking of file contents on a physical device typically hard disk, floppy, CD-ROM, tape © 2000 Franz Kurfess File Systems 24 Organization Criteria rapid access ease of update economic storage simple maintenance reliability © 2000 Franz Kurfess File Systems 25 File Organizations flat file (byte organization) sequence of records indexed tree © 2000 Franz Kurfess File Systems 26 Flat File file is organized as a sequence of bytes no further internal organization application programs perform the interpretation of the contents properties very flexible higher overhead for user programs used by Unix and MS-DOS © 2000 Franz Kurfess File Systems 27 Sequence of Records files are stored as fixed or flexible length records file operations work on records properties simple to implement, especially fixed length records less flexible CP/M © 2000 Franz Kurfess File Systems 28 Indexed Tree files consist of a tree of records efficient access is provided via indexing records must include a key field for indexing file system supports operations based on the key field properties reasonably flexible efficient operations provided by the OS secure main-frame © 2000 Franz Kurfess operating systems File Systems 29 File Types regular files (ASCII or binary) directory files character special files block special files © 2000 Franz Kurfess File Systems 30 File System Structure file system objects and operations data structures files and i/o devices © 2000 Franz Kurfess File Systems 31 File System Objects and Operations files hold open persistent data files allow access to the contents of files directories provide © 2000 Franz Kurfess naming and location of files File Systems 32 Open File intermediate level of abstraction that allows flexible treatment of files and I/O devices files must be opened before operations on them can be performed opening a file creates an internal OS object affiliated with that file similar objects can be created for devices (terminal, window, network connection) or programs (e.g. pipes) uniform © 2000 Franz Kurfess method for treating I/O operations File Systems 33 Files and I/O Devices in some operating systems, I/O devices are logically integrated into the file system access through paths and file names e.g. /var/dev/tty_01 operations are the same as for files © 2000 Franz Kurfess File Systems 34 File System Organization descriptors contain file the essential information about files systems and disk partitions file systems can span several disk partitions different disk partitions may be used by the file systems of different operating systems combining file systems integration of file systems across computer systems homogeneous vs. heterogeneous file systems and networks file systems span a whole local or global area network © 2000 Franz Kurfess requires the integration of different file systems File Systems 35 usually File System Data Structures process descriptor contains open file table lists file all files currently open in the system descriptor table contains file pointers to open files file descriptor of all open files descriptors information about files kept on secondary storage © 2000 Franz Kurfess File Systems 36 File System Implementation File Descriptors Files on Secondary Storage © 2000 Franz Kurfess File Systems 37 Memory-Mapped Files a file is mapped into the address space of a running process access to the file is mapped to the same operations as memory access works best in a system that supports segmentation makes programming easier, eliminates I/O © 2000 Franz Kurfess File Systems 38 Directories a directory is itself a file contains one entry for each file in the directory © 2000 Franz Kurfess File Systems 39 Directory Structure single-level directory two-level directory one tree directory per user structure hierarchically widely organized used © 2000 Franz Kurfess File Systems 40 Single Directory all files of all users are under the same directory very easy to implement may cause naming conflicts file organization difficult impractical for multiple users root dir. files © 2000 Franz Kurfess File Systems 41 Two-Level Directory separate directories for each user naming conflicts reduced, but still possible file organization not very convenient root directory user directories © 2000 Franz Kurfess files File Systems 42 Tree-Structured Directory multiple directories for each user hierarchical directories naming conflicts eliminated flexible file organization root directory user directories © 2000 Franz Kurfess files File Systems 43 Links and Aliases shortcuts for files or directories frequently used shared more than one directory entry points to the same file or directory only one instance of the file/directory exists potential problems cycles in the graph that corresponds to the file system access conflicts for shared files © 2000 Franz Kurfess File Systems 44 Path Names absolute a path from the root to the file relative a path relative to the working directory © 2000 Franz Kurfess File Systems 45 Directory Operations create delete opendir closedir readdir rename link and unlink © 2000 Franz Kurfess File Systems 46 Secondary Storage Management mapping of the logical file structure to physical devices hard disk, floppy disk, tape, etc allocation of blocks to files keeping track of free blocks © 2000 Franz Kurfess File Systems 47 File Allocation allocation of secondary storage space to files quick access to file contents efficient utilization of disk space pre-allocation, or allocation on demand final size file size is often unknown of the allocated sections large vs. small fixed vs. flexible © 2000 Franz Kurfess File Systems 48 File Allocation Methods contiguous linked indexed © 2000 Franz Kurfess File Systems 49 Contiguous Allocation each file occupies a set of contiguous blocks on the disk blocks allocated at file creation time (preallocation) leads to internal fragmentation if the file doesn’t reach its full size FAT (file allocation table) contains file name, start block, length similar to the dynamic memory allocation problem in memory management first fit and best fit are commonly used problems © 2000 Franz Kurfess File Systems 50 Indexed Allocation a file is implemented as a linked list of blocks each block contains a pointer to the next block blocks may be anywhere on disk directory contains a pointer to the first (and sometimes last) block of the file easy extension of files no external fragmentation overhead for pointers only sequential access reliability problems if a pointer is lost © 2000 Franz Kurfess File Systems 51 File Allocation Table variation on the linked allocation each disk (or partition) has a FAT table the table has one entry for each disk block the entry for each block specifies the block number of the next block in the file The directory entry contains the number of the first block in the file. a special value indicates unused blocks random access can be supported by caching the FAT © 2000 Franz Kurfess File Systems 52 Indexed Allocation all pointers for one file are collected in one location (index block) much better for random access this index block may be kept in main memory overhead for pointers index block size good random access © 2000 Franz Kurfess File Systems 53 Indexed Allocation for Large Files link several index blocks use a multilevel index a combination of the first two © 2000 Franz Kurfess File Systems 54 Unix File Allocation Hard Disk Unix Inode other info direct blocks ... single indirect double indirect triple indirect © 2000 Franz Kurfess ... ... ... ... .. . ... ... ... ... ... ... ... ... ... File Systems 55 Free Space Management. disk allocation table is used to keep a track of available blocks common methods bit tables linked free portions indexing © 2000 Franz Kurfess File Systems 56 Protection mainly prevention of unauthorized or improper access to stored information restricted physical access to storage media only practical for single-user systems or high-security domains access control to files and directories permissions are given to single users or groups of users may be specific to certain operations read, write, execute append, delete, list, etc. trade-off © 2000 Franz Kurfess between flexibility and overhead File Systems 57 Reliability protection often special from physical damage through duplicates of files purpose systems computer RAIDs systems designed for high reliability (redundant arrays of inexpensive disks) several disks are combined into one large “virtual” disk redundancy at various levels increases reliability design allows easy replacement of faulty components (“hot swapping”) © 2000 Franz Kurfess File Systems 58 Bad Block Management due to manufacturing tolerances, there are practically always some bad blocks or sectors on a hard disk bad block management allows the OS to avoid bad blocks or sectors spare blocks replace the defective ones special file that contains all the bad blocks © 2000 Franz Kurfess File Systems 59 Backups protection physical data against loss of data failure of storage devices (magnetic disks) head crash, bad block, switched bit on a disk is usually backed up to a magnetic tape incremental backups can minimize the amount of copying done in backups. © 2000 Franz Kurfess File Systems 60 Consistency determines the simultaneous usage of a file by several users especially relevant for modifications (write) visibility of changes to other users of the file immediately single physical instance of the file no simultaneous modifications after (Unix) the file has been closed (Andrews) multiple instances of the file simultaneous modifications may lead to consistency problems (multiple versions of a file) © 2000 Franz Kurfess File Systems 61 File System Consistency usage of files a disk block is typically read into memory, modified, and written out to the disk at a later time problems when a system crash occurs file blocks that were not written out inconsistent state of the file system system check inconsistencies can be repaired if there is some redundancy in the file system © 2000 Franz Kurfess File Systems 62 File Consistency contents of directories are checked against the existing files © 2000 Franz Kurfess File Systems 63 Block Consistency comparison in four of the appearance of a block a file and in the free list cases consistent each block appears either in a file or in the free list missing block a block appears neither in a file nor in the free list duplicate a block appears twice in the free list duplicate free block data block a block appears twice in the list of files © 2000 Franz Kurfess File Systems 64 File System Performance to improve performance disk blocks in use are kept in a buffer cache in memory a block replacement algorithm is used (e.g. LRU, FIFO, etc.) when should blocks be written to the disk? periodically (e.g. every 30 seconds) with every write (write-through cache) © 2000 Franz Kurfess File Systems 65 Performance Improvement reducing disk arm movement clustering blocks that are likely to be accessed in sequence are place close to each other (i.e. same cylinder) put inodes in the middle of the disk divide disk into cylinder groups, each with its own inodes, blocks, and free-list © 2000 Franz Kurfess File Systems 66 Block Size transfer between many I/O devices and main memory is done in blocks more economical than transfer of individual bytes or words faster because blocks can be cached on local buffers of the device controller relevant for the size of the allocation unit sector track cylinder page size © 2000 Franz Kurfess File Systems 67 Block Size small block size efficient use of space most systems have many small files larger overhead for transferring large amounts of information large block size higher internal fragmentation I/O time for small quantities of information is relatively large independent of the amount of information used, at least one block will be transferred © 2000 Franz Kurfess File Systems 68 Magnetic Disks main secondary storage device persistent storage good capacity for reasonable price practically random access long average access time millisecond range © 2000 Franz Kurfess File Systems 69 Logical Disk Structure tracks concentric circles on a platter typically several hundred to several thousand per disk cylinder same tracks on different platters sectors sections within a track typically 10 to 100 per track can be fixed or variable length blocks units of transfer between disk and memory © 2000 Franz Kurfess File Systems 70 Physical Disk Structure mechanical components platters store information on their magnetized surfaces spindle rotational movement of the platters actuator with read/write heads reading and writing of information to the platters electronic component (the controller) conversion of analog signals from the disk heads into digital signals arrangement of bits into larger units Bytes, words, blocks © 2000 Franz Kurfess File Systems 71 Hard Disk Diagram tracks spindle disk platter arm arm actuator read/write head © 2000 Franz Kurfess actuator File Systems 72 Hard Disk Access Times seek time time to move the disk head over the right track/cylinder typically several milliseconds (2-10 ms) latency time time until the right sector is under the read/write head depends on the rotational speed 7200 rpm = 120 rps = 1/120 sec/revolution = 8.3 ms/revol. transfer linear to the amount of information transferred for 100 blocks per track, it is 1/100 of a revolution per block average time access time av. seek time + av. latency + av. transfer time = 6 ms + 4 ms + 0.4 ms ≈ 10 ms © 2000 Franz Kurfess File Systems 73 Disk Scheduling Algorithms First-Come-First-Served (FCFS) Shortest Seek Time First (SSTF) SCAN C-SCAN © 2000 Franz Kurfess File Systems 74 The FCFS Algorithm disk requests are served in the order of their arrivals easy to implement FIFO queue fair performance not very good for frequently accessed disks sufficient for most single-user systems © 2000 Franz Kurfess File Systems 75 Comparison of Algorithms evaluation based on a series of disk requests total distance the read/write head has to travel time needed linear with respect to the distance traveled latency and transfer time are neglected beyond the control of the OS depend on the hardware and amount of data transferred initial configuration must be known starting position travel direction of the head for some algorithms reference starting © 2000 Franz Kurfess sequence: 39 11 75 50 85 52 66 position: track 45, speed 0.1 tracks/ms File Systems 76 FCFS Example 0 11 39 45 50 52 66 75 85 Distance Time + 6 + 28 + 64 reference sequence: + 50 + 39 11 75 50 85 52 66 Total head movement 230 © 2000 Franz Kurfess 99 35 + + 33 14 + 6 0.6 34 3.4 98 9.8 148 1.48 183 1.83 216 2.16 230 2.30 File Systems 77 SSTF Algorithm shortest seek time first selects the request with the shortest seek time from the current head position performance usually better than FCFS problem starvation © 2000 Franz Kurfess of some requests is possible File Systems 78 SSTF Example 0 11 39 45 + 5 + 50 52 66 75 85 Distance + 2 5 + 7 13 27 reference sequence: 20 47 + 56 9 39 11 75 50 85 52 66 + 10 66 + + © 2000 Franz Kurfess 99 74 140 Total head movement: 140 File Systems 79 The SCAN Algorithm the head moves from one end to the other before reversing direction a variation (LOOK) travels only as far as needed to satisfy pending requests fixed upper bound on the total motion of the head twice the number of tracks/cylinders also known as elevator algorithm performance very good for heavy loads bias towards recent arrivals, requests for inner- or outermost tracks © 2000 Franz Kurfess File Systems 80 SCAN Example 0 11 39 45 + 5 50 52 66 99 + 2 + 5 7 + reference sequence: 21 9 39 11 75 50 85 52 66 Total head movement 142 + 85 Distance 14 + 75 30 + 10 40 + 14 60 + 54 114 142 28 © 2000 Franz Kurfess File Systems 81 The C-SCAN Algorithm variation of the SCAN algorithm scanning always occurs in the same direction eliminates more reversals of direction uniform wait time than the SCAN algorithm © 2000 Franz Kurfess File Systems 82 C-SCAN Example 0 11 39 45 + 5 50 52 66 5 7 + Total head movement 192 © 2000 Franz Kurfess + 30 + 10 40 + 14 99 28 21 9 39 11 75 50 85 52 66 + 99 + 2 + reference sequence: 11 85 Distance 14 + 75 + 54 153 164 192 File Systems 83 Selection of an Algorithm performance depends on the number and types of requests location of directories and index blocks is important frequently used items are often placed in the middle tracks file allocation influences the performance oversimplification since only seek time is considered low-level scheduling algorithms are sometimes implemented in the disk controller disk I/O optimization is not the only factor for the OS © 2000 Franz Kurfess File Systems 84 Important Concepts and Terms alias backup block compression compaction consistency contiguous file directory disk block disk scheduling algorithms extension file file allocation table (FAT) file descriptor file system file system descriptor © 2000 Franz Kurfess flat file hard disk hard link index block inode latency logical block mounting path name persistent storage physical block RAID redundancy relative path name seek time symbolic link transfer time File Systems 85 Chapter Summary the file system manages persistent storage of large quantities of information files and directories are the main components convenient and fast access are important for the user efficient resource utilization (storage capacity) and fast access are main considerations of the OS hard disks are the most important secondary storage devices disk scheduling is used to improve access times © 2000 Franz Kurfess File Systems 86
