* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download NetBackup User Group Session Part I - NetBackup 6.5.2 Skip Farmer
Survey
Document related concepts
Transcript
NetBackup User Group Session Part I - NetBackup 6.5.2 Part II – Disk Based Backups Skip Farmer Agenda 1 Introducing NetBackup 6.5.2 2 Windows Features 3 Troubleshooting, Supportability, & Manageability Features 4 Proliferations 5 Disk and Related Features Symantec CONFIDENTIAL Symantec Sales Conference 2007 Part I NetBackup 6.5.2 2 NetBackup 6.5.2: Features & Benefits Symantec CONFIDENTIAL 3 NetBackup 6.5.2: Features & Benefits Symantec CONFIDENTIAL 4 NetBackup 6.5.2: Features & Benefits Symantec CONFIDENTIAL 5 NetBackup 6.5.3 Features & Benefits Show under NDA Only (revenue recognition risk) Symantec CONFIDENTIAL 6 NetBackup 6.5.2 Windows Features 7 SharePoint 2007 / 2003 Document Recovery 8 SharePoint 2007 Document Recovery Highlights PDF SharePoint Server • Database backup yields database restore… • AND yields restore of any document within a Document Library • Dual restore for the price of one! – Reduces backup time, load, and storage Symantec CONFIDENTIAL 9 SharePoint 2007 Document Recovery Licensing & Policy • Licensing – Std. Client + Database & Application Pack – Enterprise Client • MS-SharePoint policy • Requires Disk-Storage-Unit – Not yet supported from tape – Backup images must first be duplicated from tape to disk • Enable document restore checkbox – Uses Shadow Copy Snapshot (COW) – Does not require Snapshot Client license Symantec CONFIDENTIAL 10 SharePoint 2007 Document Recovery User Interface • Restore uses NBU Client BAR GUI • Simply select document(s) to restore Symantec CONFIDENTIAL 11 SharePoint 2007 Document Restore What is a “Document”? • Can be any file within – Document Library – Picture Library • Limitations – Redirection not yet supported – File revisions not yet supported • Restores most recent file from backup image • SharePoint 2007 Trash Can – Available for user-level recovery • Future plans for 6.5.x – Additional granular capabilities: entire hierarchy of sub-sites, lists, discussions, surveys, etc. – Restore any version of a document – Redirect document restore to D:\folder Symantec CONFIDENTIAL 12 SharePoint 2007 Document Recovery Granular Access Architecture SQL Server SharePoint Application NBU Agent Windows NFS NFS File Read Media Server Win/UNIX NFS NBU NFS NBU DSU • NFS is used for document recovery (yes, NFS!) – Enables support for Windows, Linux, UNIX media servers • OS support for NFS is required on – SharePoint SQL Server(s) – NetBackup Media Server with DSU for SharePoint image • NFS for Windows Server – Install “Microsoft Services for NFS” distributed with WS2003 & 2008 – Windows Server 2003 R2 SP2 is minimum requirement Symantec CONFIDENTIAL 13 SharePoint 2007 Document Recovery Supported Platforms Windows Server SPPS 2003 Farm MOSS 2007 Farm SharePoint Server SQL Server 2003 2003 x86 2000 SP4 x86 2000 SP4 x64 2003 2008 2007 x86 2007 x64 2005 x86 2005 x64 • See NBU 6.x Database Compatibility Listing for details Symantec CONFIDENTIAL 14 SharePoint 2007 Document Recovery Documentation Documentation at http://go.symantec.com/nbu > “Documentation” Download: 1.Support for Microsoft SharePoint Portal Server 2007 and Windows SharePoint Services 3.0 2.Veritas NetBackup for Microsoft SharePoint Portal Server 2003 Administrator’s Guide 3.NetBackup 6.x Database Compatibility List (search for “SharePoint” entries) Symantec CONFIDENTIAL 15 Exchange 2007 / 2003 Instant Recovery Continuous Replication 16 Exchange Instant Recovery Feature Highlights • Snapshot Client protection using MSFT VSS provider Snapshot Client – Supports VSS system, software, or hardware providers – Snapshot is retained on primary disk, restore is directly from the snapshot Enterprise Client & Exchange Agents Primary Snapshots – A specified number of snapshots is retained – When the max is reached oldest snapshot is deleted before creating new one Basic Tape Clustered Media Servers • Two configuration options for IR: – Snapshot-only option Shared Commodity Disk • Skips backing up the snapshot to a storage unit • Not suited for Disaster Recovery – Snapshot with storage unit backup option • Quick recovery backed by standard snapshot backup • No off-host backup in 6.5.2 Symantec CONFIDENTIAL 3 Array or software snapshots 3 Fully-automated 3 No load on the client 17 Exchange Instant Recovery Supported Environments • Windows Server – 2003 – 2008 • Exchange – 2003 (x86) – 2007 (x64) • VSS Providers – Windows Server default provider (COW) – Veritas Storage Foundations for Windows 5.0 or later – IBM DS6000/8000 array family – Other providers to be qualified in future 6.5.x releases Symantec CONFIDENTIAL 18 Exchange Instant Recovery User Interface - Configuration • Check perform snapshot backups. • Check “Retain snapshots for Instant Recovery” • Don’t check perform off-host backup or enable document restore • Click Snapshot Client Options • VSS is the only snapshot method for Exchange Server. • Set the provider type. • Set the maximum number of snapshots to retain. Symantec CONFIDENTIAL 19 Exchange Continuous Replication Continuous Replication - Background LCR diagram CCR diagram • Microsoft introduced “built-in” replication with Exchange 2007 – Local Continuous Replication (LCR) – Cluster Continuous Replication (CCR) – Asynchronously replicates Exchange DB/Logs for failover (outside of NetBackup) • Continuous Replication Highlights – LCR and CCR operate essentially the same way – LCR replicates between separate volumes on the same host – CCR replicates between separate volumes on MSCS nodes – Replication includes both databases and logs • FYI - “Single Copy Clustering” (SCC) is new name for old “shared volume” clustering Symantec CONFIDENTIAL 20 Exchange Continuous Replication Feature Highlights • 6.5.1 – Does not recognize the replicated data – Simply backs up the primary database • New choices in client host properties for Exchange Server – Replica first, primary if failure – Replica only – Primary only • Supports all existing schedule types • No Snapshot Client support – Off-host not needed, since CCR passive node acts as off-host – Instant recovery from snapshot not yet supported Symantec CONFIDENTIAL 21 SQL Server VMware VCB Windows Server 2008 22 SQL Server Checkpoint/Restart Overview • Similar name & benefit of filesystem checkpoint/restart – Retry of failed backup resumes at the point-of-failure • Different implementation: entirely within SQL agent – Agent defines checkpoint sets • In this example, MAXRESTARTSETS = 3 • Every set is separate backup job • Transaction logs are automatically included in last set – If a failure occurs, a script is generated to resume at the failed set – Other parameters in original script (*.bch) determine retry behavior – Ignored for Instant Recovery (entire DB is snapshot) Symantec CONFIDENTIAL 23 SQL Server Checkpoint/Restart User Interface Do not resume unsuccessful backup Standard behavior – no agent retry Retry from the beginning Retry entire job – no checkpoint Save work and restart at point of failure Retry from last checkpoint Symantec CONFIDENTIAL 24 NetBackup for VMware VCB • File-level incremental backup now supported – 6.5.1 – Flashbackup full backup of VCB snapshot – 6.5.2 - File incremental since last full or cumulative backup • This is a standard file incremental, not a Flashbackup incremental • VCB snapshot is not used for incremental backup – faster & less data transfer • Simply use incremental schedule within same policy • Support Windows 2003 x64 VCB Proxy – Added x64 in addition to existing x86 proxy platform support • Future Release(s) – Snapshot Client supported within a VM (VMware Raw Device Mapping) – VSS quiesce of Windows VM (Exchange, SQL Server, SharePoint) – Linux RedHat file restore from image backup (Flashbackup for Linux VM) – Usability: Validation of settings from UI (pre-backup validation) Symantec CONFIDENTIAL 25 Windows Server 2008 Initial Support • 6.5.2 is “Works With Windows 2008” certified – Entry-level certification; full logo certification in next major NBU release • Scope of client support – Windows Server 2008 x86 & x64 platforms, including WS2008 cluster – Exchange, SharePoint, SQL Server agents – “Server Core” option: client-only; silent install; no GUIs; remote admin • Scope of media server support – As much x64 HCL support as possible (popular models) – BasicDisk, AdvancedDisk, SSO, NOM • Important notice – NBU Server – No master server until next major NBU release (mid-2009) – No NBU server for x86 or IA-64 WS2008 platforms • Only support for x64 NBU server Symantec CONFIDENTIAL 26 NetBackup 6.5.2 Troubleshooting, Supportability & Manageability 27 Encryption Key Management for Tape Highlights • Description – Encryption key management for new tape drives with built-in encryption – Limited feature set in the 6.5.2 unlicensed (no charge) version • Maximum of 2 Key Groups 10 Key Records for each Key Group • Simple security: no cryptographic security between master and media server (keys are obscured) • Keystore DR is process-driven, not automated, requires some planning – Encryption based on tape volume pool name – Key Store resides on master server, allows centralized key management – CLI for configuration • Supported tape drives – LTO4 – IBM TS1120 (3592) – T10000, T9840D (STK/Sun) do not yet support the SCSI T10 spec Symantec CONFIDENTIAL 28 Encryption Key Management for Tape Highlights • KMS Benefits – Interoperability with all SCSI T10 encryption compatible tape drives – Supported with Al tape libraries NetBackup supports – NetBackup aware of which backup images are encrypted and admin can see that and generate reports – A single application handles backups and encryption for all supported drives – Free in NetBackup 6.5.2 • KMS Operational Considerations – Keys obscured from key store to media server (NBAC addresses this if necessary) – Keys transferred from media server to drive (over SCSI or FC) in the clear (we will support the SCSI standard when available) – Not FIPS certified – Only works with encrypted tape drives, not other end points (disk, laptops) – Exporting and importing of keys not yet supported Symantec CONFIDENTIAL 29 Encryption Key Management for Tape User Interface • Key Tag provided in NetBackup Admin Console and reports (or via bpimagelist command) indicate which backup images are encrypted Symantec CONFIDENTIAL 30 NOM Enhancements • NOM server failover support – VCS for Windows and Solaris • Infrastructure optimizations for improved performance – Enhanced server response and resource utilization • Improved alert policy configuration – Multi-select enabled in alert policy definition for more granular alert conditions • New alert manager component – Community string for SNMP alerts now configurable – Enables expanded NOM server platform support (planned for future release) • Enhanced job monitoring & control – State details: Displays reason for extended job queue or limited job progress – Job priority: Displays current relative priority for obtaining backup resources – Job priority management: Change priority of queued or active jobs from NOM Symantec CONFIDENTIAL Reverse Name Lookup control Highlights • Prior to 6.5.2, NetBackup required functioning DNS reverse host name lookup – Used for SERVER list verification and logging – Sometimes customers disable DNS or have it misconfigured – Workarounds: fix DNS or use local hosts files • bp.conf (Registry) entry – REVERSE_NAME_LOOKUP = ALLOWED | RESTRICTED | PROHIBITED • Network Settings Host Properties Symantec CONFIDENTIAL 32 Reverse Name Lookup Control What the Values mean • ALLOWED: – NetBackup requires reverse host name lookup (same as 6.0 and earlier) – If reverse name lookup fails, the connection broken • RESTRICTED: – NetBackup attempts to perform reverse host name lookup as in ALLOWED – If reverse lookup fails, don’t break connection, continue as in PROHIBITED • PROHIBITED: – NetBackup does not attempt reverse host name lookup at all – NetBackup resolves the host names of the known SERVER list to IP addresses using forward lookups and compares the address of the connecting server to the list of known server IP addresses – If this comparison fails, the connection fails Symantec CONFIDENTIAL 33 Activity Monitor Job State Details Feature Highlights Problem: • Large numbers of queued jobs create frustration. – No quick way to determine the root cause of the queuing and take corrective action. – Easy to assume something is broken or hung • Many support calls and escalations Solution: • Show why a job is queued • Show what specific resource is causing jobs to queue • …and show info at a glance • E.g., no need to drill down into dozens or hundreds of job’s details Symantec CONFIDENTIAL 34 Activity Monitor Job State Details User Interface Why are 167 jobs queued? Symantec CONFIDENTIAL • Easy to see at a glance what resources are causing the bottlenecks. 35 Activity Monitor Job State Details Operational Considerations Be Patient The job is waiting on a limit or a busy resource. Informational ‘Be Patient’ Examples: • Drives are in use in storage unit (stu) • Maximum job count has been reached for the storage unit (stu) • Waiting for mount of disk volume (disk_volume, disk_pool) • Required disk volume is unmounting (disk_volume, disk_pool) • Required disk volume is in use (disk_volume, disk_pool) • Waiting for fibre transport for client (client_name) • Waiting for shared tape drive scan to stop • Required media is in use (media_id) Fix Problem e.g., resource is offline. Symantec CONFIDENTIAL ‘Fix Problem’ Examples: • Local drives are down • A pending action is associated with the resource request. User intervention may be required. • Required media server is offline for disk (server) • Required media server is offline for tape (server) • Required robotic library is down on media server (robot, server) • Lack of connectivity detected between media server and master server (media_server) 36 Improved Usability of Job Priorities Highlights and User Interface • 6.0 introduced new capabilities to control resource allocation for all types of jobs as well as a priority scheme (bpsched PEM, JM, RB) – Prior to 6.5.2, job priority was set only in the policy configuration with “Job Priority” setting • In 6.5.2, administrators can manage job priority in various ways – Set default job priority for each job type via NOM and Admin Console – Override default priority on job by job basis – Change the priority of queued jobs via NOM and Admin Console – Specify job priority via the CLI for scripts and 3rd party schedulers • Command lines with new –priority option: – bprestore – bprecover – bpverify – bpimport – bpduplicate Symantec CONFIDENTIAL 37 Database Administration Tool for NetBackup RDBMS • Stand-alone interactive, menu-driven tool available on both UNIX and Windows • Simplifies essential DBA tasks on the NetBackup databases • On Windows the tool is in the form of a GUI – Executable \Netbackup\bin\NbDbAdmin.exe • On UNIX/Linux the tool is in the from of a MUI – Executable /openv/db/bin/dbadm Symantec CONFIDENTIAL 38 Database Administration Tool Supported Functions • Select either the NBDB or BMRDB database – Report on the status, consistency, and high-level space utilization • Report on database space – Full and table level reports – Full and table level reorganize – Add free space to database • Manage the transaction log – Truncate the transaction log – Toggle the transaction log mode (full/partial logging) • Check for database consistency and rebuild – Standard and full database validation – Rebuild database • Move the dbspaces and transaction log and create a mirror log • Unload the database • Backup and restore the database • Change the database password and start/stop the databases • Manage the database server cache memory settings Symantec CONFIDENTIAL 39 NetBackup 6.5.2 Proliferations 40 6.5.2 Feature Summary: Proliferations • Oracle 11 including compatibility with 11g compression and Data Pump • Lotus Domino 64-bit with Windows 2003 x64 and AIX. • Lotus 7/8 with AIX 6.1/Red Hat 5 • BE tape reader for 10d • BMR master on Windows 2003 x64 • BMR master on HP-UX 11.23 PARISC • BMR co-existence with EMC PowerPath with Windows 2003 x86 • SharedDisk on RHEL 4.0 update 5 • SharedDisk on Suse 9 SP3 Symantec CONFIDENTIAL • AIX 6.1 Client • Mac OSX 10.5 Client • HP-UX 11.31 NBAC • OpenAFS 1.4.4 Client with Solaris 8 • SAP Agent with HP-UX 11.31, AIX 6.1 • Informix 11 • SunCluster 3.2 on Master/Media servers with Solaris 10 Sparc/x64 • SAN Client with Windows 2008 x64 • Hitachi DF600/700 array support for Snapshot Client • IBM DS6000/DS800 array support for Snapshot Client 41 Part II NetBackup Disk Based Backups Symantec CONFIDENTIAL 42 NetBackup 6.5.2 Disk Features Symantec CONFIDENTIAL 43 RealTime Protection Integration RealTime Background A continuous data protection (CDP) solution Protects every write at the LUN level Data Loss Backup 1 day Snapshots 6-12 hours Replication Latest only CDP Zero 2d Symantec CONFIDENTIAL 1d 44 RealTime Protection Integration Solution Components Interactions Example: 2 4 Virtual Servers Database Email File Server NetBackup (RealTime) Client •Mark application consistent points in time • Copies writes to RealTime Server (steps 1-5) • Part of NBU Standard Client • Works w/NBU Application Agents NetBackup Client NetBackup Client NetBackup Client 3 NetBackup Client •Perform off-host backups • Minimal overhead on application host (steps 1-6) NBU Server RealTime Server 1 Fibre Channel SAN 1 Set NetBackup policies for protection with RealTime RealTime Server NetBackup places application into • SAN 2 attached server (Intel architecture) hot backup mode 5 • Presents TimeImage for recovery NetBackup Inserts a marker to easily • Out3of performance pathidentify the Media current point in time 4 RealTime Storage Primary Storage 6 NetBackup takes application out of hot backup mode RealTime Storage 5 NetBackup updates catalog • Mirror of primary & RealTime journal Perform off-host backup using the marked point in time 6 • Requires 1.3-1.5x production storage • Can be 1 tier lower than production Symantec CONFIDENTIAL 45 RealTime Protection Integration Supported Environments and Functions • Protected System (NBU/RealTime Client) – Solaris 9, 10 (SPARC) • Via new CDP FIM • UFS, VxFS, VxVM – Windows 2003 32 bit • RealTime 4.5 Server/Storage – RealTime storage • SF CFS 4.1 HCL – RealTime Server hardware • Intel system • QLogic 2462 HBA • via VSS • NTFS, VxVM – Primary Storage • All active/active fiber channel arrays • Active/passive arrays supported in single path mode only – Applications • TBD (Oracle on Solaris first) • NetBackup 6.5.2 – All supported media/master server platforms that work with Snapshot Client – Requires Enterprise Client license – Copy-back Instant Recovery from snapshots (roll-back is future) – Regular recovery from STUs – FlashBackup on Solaris Symantec CONFIDENTIAL 46 (Remote) NDMP to Disk Feature Highlights • Disk storage units can now be used in NDMP policies – Previously, only tape was supported for NDMP jobs – Backup, restore, duplicate, verify, import to/from disk all supported • Allows NDMP to leverage current disk technologies – Disk staging, Storage Lifecycle Policies – All disk types supported (Basic Disk, Flexible Disk, OpenStorage, PureDisk) • Operational Considerations – Compatible with all NAS devices supporting remote NDMP – Supported on all OS platforms for which NDMP is supported • All major OS platforms including HP-UX IA64 and Solaris 10 x64 – Data path is from the NAS device across LAN to media server to disk – Does not support checkpoint restart, multiplexing, synthetics, inline copy, compression, encryption Symantec CONFIDENTIAL 47 Specify restore/duplication server Feature Highlights • Description – Enables user to specify which media server(s) are favored (or required) for duplication or restore operations – Applies to media servers sharing SAN-attached disk (SharedDisk) or NFS-attached disk (AdvancedDisk) via the Flexible Disk Option – Feature not yet available for PureDisk, NearStore and OpenStorage • Customer use cases (benefits) – Route restores through a specific high performance server – Keep restore traffic off servers dedicated to backups or duplication – Isolate activity to a given server or set of servers • Directs load balancing to push operations to specific ‘zones’ • Fewer servers need direct access to both source and destination storage. Fewer points of ‘server sharing overlap’ is required to avoid network transfers. Symantec CONFIDENTIAL 48 Specify restore/duplication server Use Case Illustrations • Set a preference (or requirement) to route restores through a specific server. Backup • Isolate duplication activity to a given server or set of servers that have direct access to both source and destination storage. Only 2 media servers are connected to both source and destination storage. Clients Disk Pool Storage Unit Restore Preferred Restore Server Disk Pool Required Duplication Servers for the pool Tape Library SAN-attached disk pool (SharedDisk) NFS Symantec CONFIDENTIAL Use cases are not limited to what is shown. Could be NFS. Could be SAN-attached. 49 Specify restore/duplication server User Interface • Nbdevconfig CLI • Identify which server and for which type of disk pools. • Indicate how NBU should treat server for allocations – “PrefRestore” – prefer the marked server(s) for restores – “ReqRestore” – NBU required to use the marked server(s) for restores – “ReqDuplication” – NBU required to use the marked server(s) for duplications. • Translation of example: – When NBU starts a restore job for an image residing on an AdvancedDisk disk volume, if daloa is one of several servers with access to that volume, NBU should prefer daloa over all other servers. Symantec CONFIDENTIAL 50 Managing Snapshots within Lifecycles Feature Highlights • Description – Customers can now use Storage Lifecycle Policies to automatically manage their have their snapshot based images • Benefits – Automatic backup/duplication of snapshot images to multiple tape or disk locations – Specific retention period applied to snapshots and subsequent backups/duplications – A complete view of all backup copy locations • The feature can be used with any NetBackup-managed snapshot – Snapshot Client – RealTime Protection (CDP) Symantec CONFIDENTIAL 51 Managing Snapshots within Lifecycles Operational Considerations Only fixed retention is permitted for snapshots* The backup destination creates a tar-based image suitable for duplication • * Note: The snapshot retention period could be shorter than specified if the number of snapshots exceeds the configured limit. NBU will “try to keep” the snapshot for the specified retention period, but if the maximum number of snapshots are reached, the earlier snapshots will be deleted. Symantec CONFIDENTIAL 52 Managing Snapshots within Lifecycles Operational Considerations • Snapshot and backup copy creation is a combined operation – If the backup fails, the snapshot is not re-used when the backup is retried. The snapshot will be deleted and a new snapshot will be taken when the backup retried. – (Being smarter and re-using the snapshot is an area for future optimization.) Best Practice: If you want frequent snapshots (e.g., hourly), and backups less frequently (e.g., daily), then create two schedules in the backup policy. In this example, the first schedule triggers the lifecycle copies. The second schedule triggers the hourly snapshots. Symantec CONFIDENTIAL 53 Managing Snapshots within Lifecycles Configuration Create Optionally, Add theaother Storage Policy create destinations that Lifecycle theuses second the to Policy the Storage schedule Storage with Lifecycle afor destination Lifecycle frequent Policy Policy snapshots of Snapshot Symantec CONFIDENTIAL 54 SAN Client 4 Gb HBA and Tape Support What is a SAN Client? • Uses a Fibre Channel (SAN) as a high performance alternative to a LAN to move large amounts of backup data between the NetBackup Client and NetBackup FT Media Server. – Provides an alternative to deploying SAN Media Servers on critical file/application servers. • Provides a means to share Disk Storage Units with high performance over the SAN. LAN Fibre Channel Initiator Fibre Channel Initiator SAN Fibre Transport Pipe SAN SAN Client Client Target Mode Driver (TMD) Symantec CONFIDENTIAL Fibre MediaTransport Server Media Server Backup Storage (Disk) 55 SAN Client 4 Gb HBA and Tape Support What’s New? • SAN Client now use tape storage units theare backend storage. 4 Gb/s HBAcan support: supported QLogic chipas sets – –– – 2312 (PCI-x)and 2Gbit/s used on QLA-243x supported Multiplexed Non-Multiplexed backup cards jobs using FT in 6.5 GA and later 2422 (PCI-x)and 4Gbit/s used on QLA-246x added in 6.5.2 Multiplexed Non-Multiplexed restorescards usingsupport FT 2432 (PCI-e) 4Gbit/s used on QLE-246x cards support added in 6.5.2 – – – – FT Backups from different clients may not be multiplexed together in the same QLA2340, 2344FT backups from a single client may be multiplexed in a MPX group;2342, e.g. only single MPX2362 group QLA2460, Different clients QLE2460, 2462 can still be backed up to the same FT media server using FT, but they will be writing to different tape drives (different MPX groups). • Multiplexing Limitations • Some part numbers for supported HBAs using these are: • Sun, HP, Dell, and others resell these QLogic HBAs with minor – FT and changes LAN backups (from or different clients) all will of notthese be multiplexed firmware and oursame intent is to support together in the same MPX group. SAN FT SAN Zone FT Media Servers Symantec CONFIDENTIAL 56 Thank You! © 2007 Symantec Corporation. All rights reserved. THIS DOCUMENT IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY AND IS NOT INTENDED AS ADVERTISING. ALL WARRANTIES RELATING TO THE INFORMATION IN THIS DOCUMENT, EITHER EXPRESS OR IMPLIED, ARE DISCLAIMED TO THE MAXIMUM EXTENT ALLOWED BY LAW. THE INFORMATION IN THIS DOCUMENT IS SUBJECT TO CHANGE WITHOUT NOTICE. Symantec CONFIDENTIAL