Download Open systems solutions to large file requirements

Open Systems Solutions
to Large File
Requirements
DtaPaper Title
Jochen Kirsten
Open Systems Technical Support
SAS Institute GmbH, Europe
Problem

32-bit integer architectur





231-1 = 2 147 483 647 = 2G
Downsizing
DW initiatives
Large databases
Need for files larger than 2GB
S
3 Possible Solutions

64-bit integer architecture





263-1 = 9 223 372 036 854 775 807 = 9?
3 trillion US dollars!
Will this suffice for the next years?
Do both 32-bit and 64-bit support
Do nothing about large files
S
Standard OS Support




DEC Unix
Convex OS
“Native” 64-bit file pointer support
with system calls like lseek etc.
“Just buy big enough disk drives
and roll with VLFS”
S
“Extended” OS Support






Intel ABI
MIPS ABI
Native 32-bit pointers with lseek etc.
sas
Additional set of calls that have 64bit parameters, eg. lseek64
sas -largefile sasvlfs
S
No OS Support






Partitioned LIBNAME syntax
LIBNAME big (‘/disk1’ ,’/disk2’, ...)
type=partition partsize=1800M;
/disk1/ds.ssd01, /disk2/ds.ssd01.1, ...
All SAS file types supported
Take advantage of LVMs
sas -partsize <size>
S
Performance




Does not seem to depend on the
method
Fast disk-I/O required
Use LVMs
Sorting a dataset requires at least
2.5 times the size of the datset in the
WORK library plus an additional
(temporary) copy in its own library!
S
Availability


Fully available for the SAS System on
ConvexOS and DEC Unix
Experimental
 -largefile sasvlfs
 -partsize <size>
options with the Orlando Release on
Intel ABI and MIPS ABI
S
Conclusion






Full hardware exploitation
Migration path
Databases
Data warehousing
Data mining
Unlimited possibilities
S
Acknowledgment
Special thanks to






Tom Truscott (SUGI presenter)
Carl Zeigler (LF summit meetings)
Bob Huemmer (SAS port to DEC Unix)
Deborah Morgan (subroutine calls)
Larry Noe (partitioned libname engine)
Eric Coates (testing)
S
Thank you for
your attention
DtaPaper Title
The SAS® System for Open Systems