Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
EMu Technical Requirements Document Version 3.2 EMu version 5.1 Contents SECTION 1 SECTION 2 ArchitectureandConnectivity 1 Server Workstations Web server Network Open Systems Licensing Multimedia 2 5 5 6 6 7 8 Examplesystemconfigurations 11 Index 17 Architecture and Connectivity SECTION1 ArchitectureandConnectivity EMu is a collections management system that provides both client‐server and web browseraccesstoacollection. AtypicalhardwareconnectivitysetupforanEMuinstallationisillustratedbelow.In thissetupamulti‐userservermachineprovidesdatabaseserverfacilitiestoboth: LocalandremoteEMuclientworkstations and Localandremotebrowser‐basedInternet/Intranetusers EMuConnectivityOverview Large EMu sites in particular may have a dedicated web server located outside the corporatefirewall.InthiscaseEMuwebservicesresideontheexternalwebserverand gainaccesstotheEMuserverviaasecureconnectionthroughthecorporatefirewall. EMu Technical Requirements 1 Architecture and Connectivity Server Operatingsystem TheEMuservermustrunoneofthefollowingoperatingsystems: Linux/Unix AllofthemostpopularUnixandLinuximplementationscanbeused,including: CentOS RedHatES Ubuntu Solaris FreeBSD Windows Server 2008 and 2012 are supported via a Linux virtual machine running throughahypervisor. WerecommendUnixorLinuxforlargeEMuimplementations(over500,000recordsor morethan10concurrentusers).ItisestimatedthatthereshouldbeoneserverCPU coreforevery2‐3concurrentusersforheavyusage,orevery4‐5concurrentusersfor lightusage.EitherUnixorWindowsissuitableforsmallerimplementations. Wecanprovideadviceonthebestservertomeetyourrequirements. Authentication EachEMuusermusthavearegistereduseraccountontheEMuserver.Thefollowing userdatabasesaresupported: 2 Unixpasswordfile NIS(NetworkInformationServices) WindowsActiveDirectory LDAP(LightweightDirectoryAccessProtocol) EMu Technical Requirements Architecture and Connectivity Installedsoftware ThefollowingsoftwarepackagesarerequiredbytheEMuServer: Perl5.8.8orgreater‐perlscriptinglanguage Thefollowingperlpackagesarerequired: XML::Parser::Expat File::FcntlLock The following extra software packages are required by the EMu web server (IMu) if installed: ImageMagick‐imagemanipulationpackage PerlMagick‐perlinterfacetoImageMagickpackage Thefollowingperlpackagesarerequired: HTML::Parser Image::ExifTool PDF::APi2 Backup EMurequiresnospecialbackupsoftwareanddoesnotuserawpartitions.Anystandard backup and restore software provided with the operating system or a third party product(e.g.ARCserve)maybeused. Storage ThediskspacerequirementfortheEMuback‐end(withoutrecorddataormultimedia) islessthan2GB(thisdoesnotincludespacerequiredbytheoperatingsystem). Otherstoragerequirementsdependheavilyontheanticipatedamountofmultimedia whichistobestoredwithinthesystem,andrecommendationsvaryaccordingly. Memory Abasememorylevelof3GBshouldbeallocatedtotheserver. The additional memory requirement for theserveris dependenton the numberand rolesoftheconnectedusers.Generally,theservermemoryrequirementperconcurrent uservariesbetween150and300MBforartandculturalhistoryorganizations,and256 and512MBforlarge,multidisciplinarynaturalhistorymuseums. EMu Technical Requirements 3 Architecture and Connectivity Virtualization The EMu server may be virtualized. There is a performance overhead in running virtualizationhoweverandifperformanceiscritical,itisrecommendedthatEMuisrun onrawhardwareifatallpossible.Theoverheadwithvirtualizationistypically5to10% butcanbeashighas15%. Inanenvironmentrunningvirtualizationdirectlyonrawhardware,VMWareESXior CitrixXenServerisrecommended. IfWindowsbasedvirtualizationisrequired,werecommendVMWareoverHyperV. In all instances, we recommend either RedHat Linux or CentOS Linux as the guest operatingsystem. 4 EMu Technical Requirements Architecture and Connectivity Workstations EMuworkstationsmustrunWindows10,8,7,Vista,2012,2008or2003.Thediskspace requirementontheclientcomputerislessthan2GB(withoutcachingofmultimedia andreports),althoughatleast10GBofavailablefreediskspaceisrecommended. Minimum requirements include a video card capable of at least 800x600 resolution, withatleast16bitcolor.Otherminimum(andrecommended)requirementsare: Pentium2.0GHzDualCoreCPU 2048MBRAM(4096for64bitWindows) AppleMacintosh ItispossibletoruntheEMuWindowsclientonAppleworkstationsthroughtheuseof Windowsemulationtools.Sometoolsavailableinclude: Parallels(http://www.parallels.com/products/desktop/) VMware(http://www.vmwar.com/products/fusion/) Pleasecontactusforfurtherdetails. TerminalServices Windows users may also use Terminal Services or Citrix, which can be useful for providing high speed EMu access over low speed connections to remote sites or for teleworkers. Clientinstallation TheEMuclientcanbeinstalledinoneoftwoways: Local ThefullEMuclientisinstalledoneachPC.EachPCwillneedto beupdatedforEMuclientupgrades. Network The major portion of the EMu client is installed on a network (recommended) drivetowhichallEMuusershaveaccess.OnlyasmallEMuclient needstobeinstalledoneachPC,andinmostinstancesonlythe networkdriveneedstobeupdatedforEMuclientupgrades. Webserver ThewebservermustrunPHP5.3orlater. EMu Technical Requirements 5 Architecture and Connectivity Network All data and multimedia reside on the EMu server and are transferred between the server and each client PC as required. Multimedia can also be stored on a separate serverornetworkattachedstoragedevice(SAN).RecorddatatransferredbetweenPCs andtheserveronlyplacesasmallloadonthenetwork. MultimediafilestransferredbetweenPCsandtheservermayplacesignificantloadon thenetworkdependingonthesizeofthefilesandnetworkthroughput. ThenetworkmustsupportTCP/IP. For direct connection between the EMu client and server a minimum network bandwidth of 1GB/s is required. If thin client technology is in place (e.g. Citrix or TerminalServices),thenetworkbetweentheenduserandthinclientgatewaymaybe substantiallyslowerthanthis(withCitrix,evenaconnectionacrossadial‐upmodemis feasible). OpenSystems TheEMudatabaseengine,Texpress,hasanumberofstandardtechnologiesbuiltinto the DBMS for extracting data. These technologies include http, XML (the two most commonstandardsusedintheworldtoday)andSQL. TechnologiessuchastheSQLinterfacearedesignedonthestandardsavailable.TheSQL interface is designed to SQL92 with many enhancements for subsequent design advances. All of the information required to use this tool is published. Many people outsideofAxiellhavesignificantexpertiseinthisandotherinterfacestoTexpress. With EMu's data export tool it is possible to export data in a wide range of formats, includingXML,CSV,Blankpadded,andavarietyofothers. 6 EMu Technical Requirements Architecture and Connectivity Licensing Licensing for EMuisby concurrentuser access totheEMu server. Thus EMu can be madeaccessibletoawideuserbasewiththelicenceonlyaffectingthenumberofusers whorunthesystemsimultaneously. EMu incorporates a fully integrated web interface suitable for Internet and Intranet access.DependingontheexpectedInternet/Intranetloadasitecanchoosetorunone ormoreEMuwebservers(inthenatureoftheweboneEMuwebservercanservice severalusersapparentlysimultaneously).Note,however,thateachEMuwebserveris consideredaconcurrentuserintermsoflicensing. Thenumberoflicencesthatshouldbeallocatedtowebserversdependsontheexpected numberofwebsearches.Ageneralguidelineistodedicateonelicenceoutofeveryfive totenlicencestoservicingwebsearches.Theallocationoflicencesforwebusemaybe changedatanytime.Themorelicencesdedicatedtowebuse,themorewebsearches canbeservicedconcurrently. Otherthanoperatingsystemsoftwarenoothersoftwareisnecessaryinordertouse EMu.However,manyothersoftwarepackagescanbeusedinconjunctionwithEMu,in particularawebbrowser,emailsystem,MicrosoftWordandExcelandCrystalReports Developer. EMu Technical Requirements 7 Architecture and Connectivity Multimedia EMucanstoreanymultimediaresource.Itisabletoshow/playmostcommonimage, videoandaudioformatsdirectly,andcaninvokeaseparate"helper"applicationforany otherformats. Multimedia resources are addedinteractively tothe EMu Multimedia modulefrom a localornetworkdriveaccessiblefromaclientPC.Whenamultimediarecordissaved, EMu automatically creates a thumbnail and any other derivatives as required. The multimediaresource(andderivativesifany)arethenuploadedandstoredontheEMu server. Once the multimedia record is saved, EMu does not keep a reference to a multimediaresourceontheclientPC. MultimediaresourcesaredownloadedtoeachclientPCondemand.Amultimediacache can be used on each PC to reduce network traffic. EMu client options can be set to configurethemultimediadownloadandcachefacilities. ImageSupport EMusupportsover100imagefileformats,including: BMP DCX DNG GIF JP2 JPEG PCD PCX PNG TGA TIFF MicrosoftWindowsBitmapImage ZSoftIBMPCmulti‐pagePaintbrush DigitalNegative CompuServeGraphicsInterchangeFormat JPEG2000 JointPhotographicExpertsGroup PhotoCD ZSoftIBMPCPaintbrush PortableNetworkGraphics TruevisionTargaImage TaggedImageFileFormat EMu’s image capabilities are extensible, automatically supporting any image formats registeredbyapplicationsinstalledonacomputer. 8 EMu Technical Requirements Architecture and Connectivity ImageMetadataSupport EMusupportsthefollowingmetadatastandards: EXIF ExchangeableImageFileFormat(Version2.3,April2010) IPTC InternationalPressTelecommunicationsCouncil(Version1.1,July2009) XMP ExtensibleMetadataPlatform(RevisionDecember2008) Metadataisautomaticallyextractedfromthefollowingfileformats: Standard Formats supported EXIF JPEG,TIFF IPTC JPEG,TIFF,PICT,PS,PSD XMP JPEG,TIFF AudioSupport AudioformatssupportedbyEMuinclude: AIFF AU MIDI MP3 RMI SND WAV WMA AudioInterchangeFileFormat AudioFileFormat MusicalInstrumentDigitalInterface MPEGAudioStream,LayerIII MIDIinRIFFFileFormat SoundFileFormat WaveformAudio WindowsMediaAudioFile EMu’s audio capabilities are extensible, automatically supporting any MCI compliant audioCODECsinstalledonacomputer. EMu Technical Requirements 9 Architecture and Connectivity VideoSupport VideoformatssupportedbyEMuinclude: ASF AVI M1V M2V MP4 M4V MODJVC MPEG WMV AdvancedStreamingFormat AudioVideoInterleaveFile MPEG‐1VideoFile MPEG‐2VideoFile MPEG‐4Part14 EverioGZ‐MG20UDigitalVideoFile MPEG1SystemStream WindowsMediaFile EMu’s video capabilities are extensible, automatically supporting any MCI compliant videoCODECsinstalledonacomputer. 10 EMu Technical Requirements Example system configurations SECTION2 Examplesystemconfigurations In this section we describe a range of EMu system configurations suitable for an institution with approximately 110 concurrent users. But first we examine what resourcesaretypicallyrequiredtorunEMuinanenvironmentwithapproximately110 concurrentusers. CPU CPU usage varies considerably depending on what activities users undertake: for instance,searchesuselittleCPUresourcescomparedtodatamanipulation(e.g.sorting andreporting).Ifusersperformlotsofdatamanipulation,itisimportanttohavehigh speedCPUsandpreferablyenoughcoressothatoneprocessisnotblockedbyanother. Whilemultiplethreadspercoreisusefulasitallowsformoreconcurrentprocesses, distinct cores will give superior performance as load increases (in other words, it is bettertohavelotsofcoresratherthanasinglecorewithmultiplethreads). To guarantee that processes are not CPU bound (processes going slow because they cannotgetenoughoftheCPU),thenumberofcoresshouldbesimilartothenumberof activeusers.Ingeneral,ofcourse,notalluserswillproduceaheavyloadatthesame time and it is reasonable to have less cores than the number of users. Furthermore, processesneedtostopfromtimetotimetowaitfordisks,sootherprocessescanuse thecoreatthistime.Iffewercoresareavailable,thenmultiplethreadspercorebecome usefulwhentheloadpeaks(multiplethreadsleadtoaslowerdegradationofmachine performance). In a system with 110 concurrent users and a high load, 100 cores would be recommended.Inreality,itishighlyunlikelythatalluserswillbeproducingaheavy loadsimultaneouslyanditisfeasibletohavesignificantlylesscoresandtoallowtwo threadspercoretohandlepeakloads.Inthiscase,maybe50coreswithtwothreads percorewouldbeadequate. Cache Naturally,thelargertheCPUcache,thebetter.EMuisadatabasesystemsothehitrate onthedatacachecanbecomelowwhenmanipulatingalotofdata(asthedataisspread outovermemory).However,thelargerthecache,thebetterthehitrate.Alsoalarge cache allows more of the EMu executable to reside in the instruction cache, hence improvingCPUperformance.Thisisusefulwhentwothreadsarerunningonasingle coreandmultiplecoresarelocatedontheonechipasallthreadsarelikelyrunningthe sameexecutable(theEMuserver). EMu Technical Requirements 11 Example system configurations NumberofCPUs ItisbettertohavemorephysicalCPUseachwithlesscoresthanoneCPUwithalarge numberofcores. Unfortunately the more physical CPUs, the greater the cost; however, the benefits outweighthecost.Inparticular,theduplicationofmemoryaccesshardwarerequired by each CPU allows greater throughput. Also less cores and more CPUs means more distinctcaches(oneperCPU)whichincreasesthehitrateaslessprocessesareusing thesamecache(allowingeachprocesstousemoreofthecache). Tosummarize:forEMuitisworthlookingformorephysicalCPUswithlesscoresand dualthreadspercore.AlargecacheisalsobeneficialandthehighertheCPUspeed,the better. SPARCvsIntel UntilrecentlySPARCbasedsystemswerethearchitectureofchoiceforlargemachines duetothewellthoughtoutdesignofSPARCsub‐systems(memory,disk,network,etc.). Designedtocommunicatesimultaneously,thisprovidesmaximumusageofhardware atanypointintime.Incontrast,IntelbasedsystemsweredesignedassuperPCs,where theneedforsimultaneousoperationswaslow. Inrecent yearsIntelandAMDhave movedinto theserver marketandnowproduce hardwarethathandlessimultaneousoperationsaseffectivelyasSPARCbasedsystems. Intel/AMDCPUsprovidebetterthroughputattheCPUlevelthanSPARCbasedCPUs, anddosoatalowercostanditisourviewthatanIntel(x86_64)basedsolutionwill nowadays provide better computing power for a lower cost than a SPARC based solution. Memory EMu can be fairly memory intensive when a process is accessing a large number of tables.InordertoprovidebetterperformanceEMudoesnotfreememoryassociated with tables after the table is no longer required. As the overhead of reloading the schema,etc.ishigh,itisbettertopaythepricewithmemoryratherthanprocessing poweranddiskaccesses(requiredifthetableneedstobereloaded).Thismeansthat EMuprocessestendtogrowasusersaccessmoretables. TheEMuserverprocessstartsatabout6MBbeforeanytablesareloaded.Theamount ofmemoryrequiredforafullyloadedservervariesdependingonthetablesinstalledin aninstitution.Atypicalrangeisfrom200MBto400MB.Ifweassumeaworstcaseof 500MBandthereare110users,thenabout55GBRAMwouldberequiredbyEMuitself. Tothisweneedtoaddmemoryforallothersystemprocesses,theoperatingsystem andthefilesystemcache:roughly500MBforotherEMuprocess(backgroundloads, etc.)and500MBforsystemandoperatingsystemprocesses. 12 EMu Technical Requirements Example system configurations Theamountofdiskcachedependsonthefilesystemused.IfusingZFS(recommended forSolarissystems),thebiggerthecache,thebetter.Asizeofabout4‐5GBisuseful. Thismeansthatabout64GBwouldberequiredtohandletheexistingload.Thisdoes notallowformoreusersinthefuture,norfortheadditionofmorewebservices,etc. Wewouldrecommendincreasingthisnumberto96GBtoallowforfutureexpansion. IdeallythememoryshouldbesplitintobanksbasedonthenumberofphysicalCPUsin themachine.Thus,iftherearefourCPUs,thenfourbanksof24GBeachshouldbeused. ThisallowseachCPUtoworkonitsownsetof24GBwithminimalneedforexpensive crossmemorybankreads. Disk As a database system, EMu can be disk intensive when a large number of records is beingmanipulated(sorting,reporting,globaledits,etc.).Sincethedisksmaybecomea bottleneck(asallprocessesneedtoaccessthem),itisworthgettingthefastestdisk subsystemavailable. Iflookingatlocaldisks,itishardtogopastSAS(SerialAttachedSCSI)drives.Ifpossible, use15,000rpmdrives,otherwise10,000rpmdrives.Inordertoimprovethroughput, usingRAID0(striping)isrecommended.Ideallythenumberofdisksstripedshouldbe afactoroftheblocksizeofthefilesystem.AgoodfilesystemblocksizeforEMuis16 KB.Thiscouldbeconfiguredasfourdisksinthestripewhereeachdiskhasa4096byte sectorsize.Thismeansthateach16KBreadrequiresreadingonlyonesectorfromeach disk.Asthishappensinparallelacrossthefourdisks,thethroughputgainislarge. Ifdataredundancyisrequired,RAID1isrecommended.ChecksumbasedRAIDisnot recommendedduetotheslowwritetimesexperienced(e.g.RAID5).If1.2TBofdiskis requiredforinstance,then8x300GBdriveswouldbeideal,assumingRAID10isused (thatisRAID0and1combined).Ifthisisnotpossible,then4x600GBdrivescouldbe usedwithafilesystemblocksizeof8K. Ifnon‐localdisks(SAN)aretobeused,muchthesameappliestotheconfigurationof the SAN. Ideally the data should be striped over a large number of disks to provide betterthroughputviasimultaneousread/writes.Alsothelargerthediskcacheonthe SAN,thebetter.OnceagainchecksumbasedRAIDconfigurationsarenotrecommended astheymayimpactwriteperformance. Ingeneral,localdisksarepreferabletoanon‐localdiskSANaslocaldisksdonothave tocompetewithothermachinesaccessingtheSAN.Ofcourse,iftheSANisdedicatedto the EMu server, then it functions much like a local disk subsystem (provided the connectingtechnologyisfast,e.g.fibreoptic,andtheSANoperatingsystemistunedfor databaseusage). EMu Technical Requirements 13 Example system configurations Hybrid disk technologies are appealing. They provide the appearance of faster disk accessesthroughtheuseoffrontendSSDs.SoprovidedallinformationisintheSSD cache,very fastreadsarepossible; similarlyfast writesare possible(SSDswrite2‐4 faster than high speed SAS drives). Such solutions are more expensive, but for disk intensivesystemslikeEMu,significantbenefitscanbeachieved. Finally,SSDonlybaseddrivesmaybeused.Theseprovidesuperiordiskthroughputfor bothreadsandwrites.Howeverthethroughputcomesataprice.SSDsareexpensive, butifthefastestsolutionisrequired,theyshouldbeconsidered. SSDsdohavealimitednumberofwritecycles,howeverthelimitformostcontemporary drives is around 10,000 writes and it is extremely unlikely that this limit would be reachedinafiveyeartimeframe(even,indeed,inunder10years). OperatingSystem For a medium to large institution (> 10 users) we recommend running under some variantofUNIXratherthanWindows. ThetwomostpopularUNIXversionswesupportareLinuxandSolaris.Forverylarge institutions (>=50 users) we would recommend Solaris 10/11 rather than Linux as SolarissupportsmultiplevirtualCPUs(whereavirtualCPUisathreadonacoreona physical CPU) better than Linux when the number of virtual CPUs grows above ten. HoweverEMurunswellonLinuxandachoicebetweenthetwoUNIXversionsreally dependsonthepreferenceofyourSystemAdministrator. Forsmallernumbersofusers(<50)Linuxprovidesaverynicesolution. 14 EMu Technical Requirements Example system configurations SuggestedConfigurations The following configurations are suggestions only but should provide some idea of configurationssuitableforEMu.Whichconfigurationissuitable(andwhethertweaking isrequired)willdependonaninstitution'ssize,requirementsandloads. The machine specifications below are for a site with approximately 110 concurrent users. Configuration1 SunX4‐4Server 4x15coreE7‐8895v2,2.80Ghzx86_64CPU 37.5MBSharedL3CacheperCPU 128GB(4x2x16GB)DDR3‐ECCmemory 6xSASdrives(300GB) Solaris10/11x86_64 ZFSfilesystem ThisconfigurationusesZFSastheunderlyingfilesystemwithRAID10.Itisamiddle solution(intermsofcostandperformance).ZFSdoesnotsupporttheuseofSSDsfor data,howeveritdoessupportthemforitsZILandARC(caches).Theideahereisto purchaseextramemorytoensurethatmostdatablocksarekeptincorebyZFS(hence the128GBRAM).InthiscaseSSDdrivesarenotrequired.Youcouldlowerthememory to96GBandincludetwoSSDsfortheZFScaches,butthiswouldaffectthestripingas thenumberofdisksinthestripewouldreducefromthreetotwo.Ifahigherendserver (andmoreexpensiveserver)isrequired,theSunx4‐8shouldbeconsideredwiththe aboveconfiguration,exceptthat8xSASdriveswouldprovidebetterdiskstriping. Configuration2 SunX4‐4Server 4x15coreE7‐8895v2,2.80Ghzx86_64CPU 37.5MBSharedL3CacheperCPU 128GB(4x2x16GB)DDR3‐ECCmemory 6xSSDdrives Linuxx86_64 XFS/Ext4fsfilesystem Thisconfigurationissimilartoconfigurationone.TheonlydifferenceistheuseofLinux insteadofSolaris.ThereasonforthechangeisthatLinux'sXFSandExt4fsfilesystems bothprovidedirectsupportforSSDdrives.InthisconfigurationitispossibletouseSSD drivesexclusivelytoprovidesuperiordiskthroughput.Theaboveconfigurationwould provide a faster disk subsystem, however it is debatable how much of the disk subsystemisusedgiventhelargeRAMcachepossiblewith128GBofmemory. EMu Technical Requirements 15 Example system configurations Configuration3 DELLC6220IIServer 4x12CoreE5‐2697v2,2,70Ghzx86_64CPU 30MBSharedL3CacheperCPU 128GB(4x2x16GB)DDR3‐1866memory 8x300GBSASdrives(15K)or8x300GBSSDdrives Linuxx86_64 XFS/Ext4fsfilesystem TheaboveconfigurationallowsforeitherSSDorSASdrives.Costwillprobablydecide thebestwaytogo.Giventhelargememorycache,thefilesystemcouldresideonSAS drivesandtheextrabenefitsofSSDmaythereforebeminimal. Configuration4 DELLR920Server 4x15CoreE7‐8890v2,2.80Ghzx86_64CPU 37.5MBSharedL3CacheperCPU 128GB(4x2x16GB)DDR3‐1600memory 8x146GBSASdrives,4x149GBSSDdrives Linuxx86_64 XFS/Ext4fsfilesystem TheaboveconfigurationwouldallowtheSSD/HDDsolution,providingaveryfastdisk subsystemthroughtheuseofSSDdrivestofrontendtheSASdrives. 16 EMu Technical Requirements Index Operating system • 3 Operating System • 14 A Apple Macintosh • 5 S Server • 3 SPARC vs Intel • 12 Architecture and Connectivity • 1 Storage • 4 Audio Support • 10 Suggested Configurations • 15 Authentication • 3 B Backup • 4 C Client installation • 5 T Terminal Services • 5 V Video Support • 10 Virtualization • 4 Configuration 1 • 15 W Configuration 2 • 15 Configuration 3 • 16 Web server • 7 Configuration 4 • 16 Workstations • 5 CPU • 11 D Disk • 13 E Example system configurations • 11 I Image Metadata Support • 9 Image Support • 9 Installed software • 3 L Licensing • 7 M Memory • 4, 12 Multimedia • 9 N Network • 7 O Open Systems • 7