Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
2013 Esri International User Conference July 8–12, 2013 | San Diego, California Technical Workshop Planning: Enterprise Geodatabase Solutions John Alsup Jeff DeWeese Esri UC2013 . Technical Workshop . Agenda • Overview • Database Design • Data Maintenance • Infrastructure Design • Data Distribution and Infrastructure Security Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions What is a Geodatabase? • A database or file structure used to store, query and manipulate spatial data. • Data and functionality • Three types: - - File Geodatabase Personal Geodatabase ArcSDE Geodatabase - DB2 - Informix - Oracle - PostgreSQL - SQL Server Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Networks Surveys Addresses Vectors 27 Main St. Annotation ABC 3D Objects Attributes 107’ Topology Dimensions CAD Drawings Terrain Images Enterprise GIS • GIS technology regarded by users and IT as key to business operations - • May be considered mission critical Mainstream IT – deployed and managed like any other IT system - Architecture, Interfaces, Development tools, Deployment strategies, Standards • Integrated with other enterprise systems • Requires a higher level of planning, integration, testing and support Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions What is an Enterprise Geodatabase? • Data • • • • • • Serves data promptly and efficiently Supports multiple users and departments concurrently Provides seamless data Centralized data management Data integrity Functionality • • • • • • SQL support Collaborative editing, and long transactions Quality control and quality assurance Infrastructure for distributing and replicating data Integrates spatial and business data with other systems Leverages existing GIS and IT skills and resources Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Enterprise Geodatabase Business tables Feature tables Spatial Index tables A and D tables ArcObjects ArcMap ArcCatalog ArcGIS Server Raster tables Topology tables User Schemas ArcGIS Engine Geometric Network Tables Miscellaneous Tables Log files Searching Spatial Processing Temp ArcGIS Runtime Non-Spatial Business Table Native SQL Geodatabase System Schema GDB_ tables ArcSDE tables Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Organizational GIS Configurations Centralized Data Warehouse IT Utilities Department File Servers Utilities ArcSDE Parks Parks Assessor Assessor WAN WAN ArcGIS Desktops ArcGIS Desktops Data Warehouse Departmental GIS Operations Centralized Data Sharing Distributed Client/Server Departmental GIS Centralized Database IT Utilities ArcSDE ArcGIS Server/Terminal Servers (server consolidation) Parks Assessor WAN ArcGIS Desktops, Terminals and Browsers Centralized Database Enterprise GIS Operations Centralized Data Administration Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Why Plan an Enterprise Geodatabase? • Some key reasons: • Foundation for enterprise-wide use of GIS. • Geodatabase projects are complex • Enterprise Geodatabases and GIS application design requires diligent alignment • Large geodatabase projects span organizational groups and disciplines • Impacts almost every part of an enterprise GIS solution Spatial data is a key component of an enterprise GIS architecture . . . . . . delivery of spatial data must be fast, and this requires planning. Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Geodatabase Project Scales • Larger Multi-phased Approach • Elaborate, large databases • Custom applications • Large user base • Potentially outsourced, dedicated project management • Lighter Workgroup Approach • Evolve the geodatabase, gradually upgrade data and applications • COTS application functionality where possible • Built in-house, part-time project management Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Agenda • Overview • Database Design • Data Maintenance • Infrastructure Design • Data Distribution and Infrastructure Security Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Challenges and Risks • Application development has a critical dependency • Normalization in the data model • Updating the model “downstream” is expensive • Mass updates are expensive in an Enterprise Environment • Thorough review of model among teams • Optimizing for publication and maintenance Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Geodatabase Design • Elements of good Geodatabase design - Data model reflects requirements Scalable Avoids redundant storage of data items Efficient access to data Maintains data integrity over time Clearly documented Provides for analysis and behavior Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Data Modeling Methodology Conceptual Design Tasks: Conceptual Model Three Stages • • • Physical Model Physical Design Tasks: • Create and implement model design • Generate physical schema in the DBMS • Testing and validation • Document Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions • Logical Model • Identify business needs Identify thematic layers Identify required applications Leverage data model template Document Logical Design Tasks: • Define tabular database structure • Define relationships • Determine spatial properties • Document Conceptual Model • Identify and Document: • Business needs - requirements • Thematic layers • Required applications and system interfaces • Leverage existing model templates • Pre-designed schema of data objects • Best practices Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions ArcGIS Data Models Web site: http://support.esri.com/downloads/datamodel • Over 25 industry-specific data models • Conceptual and logical diagrams, sample Geodatabase schemas • Case studies • Tips and Tricks documents • Developed and maintained by user and industry communities Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Logical Model Design • Refine conceptual model based on documented requirements • Define and clarify all feature classes, tables, attributes and relationship classes • Use subtypes to control object behavior - Ex. Geometric Network can enforce behavior • Attribute domains and complex coding • Define network and topological properties and rules • Define spatial reference properties • Map placement considerations Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Logical Model Design • Projection - • Projection on the fly can be expensive All feature classes in the same Geometric Network must use spatial reference Density of Features - High vertex count can be expensive Can adversely affect functionality and usability. • Spatial placement vs. Logical placement • Data update cycle - Replacement vs. editing Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Physical Model Design • Implementing the physical Geodatabase prototype, test, review, and refine • Documenting the design for distribution and efficient updating • Test, refine and tune data model design for deployment Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions RDBMS Geometry Storage Format RDBMS Geometry Storage DB2 ST_Geometry, SDEBinary Informix ST_Geometry, SDEBinary SQL Server Geometry, Geography, , SDEBinary Oracle ST_Geometry, SDO, SDEBinary PostgreSQL ST_Geometry or Geometry Netezza VarChar (Shape) Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Important Considerations • Field Names • Geometry Storage Types • RDBMS’s used • External systems and interfaces – key for enterprise GIS • • • CRM, WMS,SAP, other Financials, Reporting Number of interfaces depends upon the organization Consider data sharing - field data types, naming and length Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions External System Interface • ETL • Database Level, duplicating data - • Triggers Update tables Database Views - Joins data from same or different databases Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Creating Structure • Look to existing tools - - • CASE and UML tools – Visio, Rational Rose, etc. Other tools (some free) and samples may work depending on approach Inheritance, re-use of objects through abstract and concrete classes Physical Model Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions XMI (XML Design) Geodatabase Data Modeling Tools • Visio • Rational Rose • Enterprise Architect • Free Esri Tools on ArcScripts: - ArcGIS Diagrammer GDB Xray Geodatabase Diagrammer Geodatabase Designer Free Tools are not supported… Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Mixed RDBMS Environments • For consideration: - IT Field Names, length and keywords Field Data Types and Lengths Database behaviors Utilities SQL Express Parks SQL Enterprise Assessor WAN DB2 Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Oracle Mixed RDBMS GDB License Levels • For consideration: - IT Utilities Domain authentication Field Data Types and Lengths Database behaviors GDB Enterprise Parks GDB Workgroup Assessor GDB Workgroup Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions GDB Enterprise WAN Testing and Refining • Small pilot data migration with sample data • Application testing – Test workflows • - Functionality - Performance - Flexibility and consistency Team review and demonstration - Show how tasks are performed using GIS - Show maps, reports, online demos Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Data Planning • Migration and Conversion - • Data procurement - • Migration deals with moving existing geospatial data between different GIS environments or platforms Conversion refers to development of new data by creating new digital geospatial data Conversion is typically more significant and costly than migration Landbase Imagery Data loading - Tools – In-house or outsourced Procedures Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Agenda • Overview • Database Design • Data Maintenance • Infrastructure Design • Data Distribution and Infrastructure Security Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Overview of Data Maintenance • Plan and manage the maintenance workflow in the geodatabase • Key Tasks - Analyze and build on business process requirements QA/QC Design your maintenance strategy Plan for versioning Define maintenance workflows Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Consider QA / QC • Ensure data is captured, loaded and maintained accurately • Quality Assurance - • Quality Control - • Review data to discover errors and perform data cleaning activities to improve quality. Ensure data products are designed to meet or exceed data requirements. QA/QC Plan - Versioning Manual and automated procedures Validations Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Versioning and Multiuser Geodatabase • Defining versioning specifications and workflows: - Versioning structure - Reconcile, post, compress regimes - Edit volumes, version durations All impact performance… Non-Versioned Editing Versioned Editing DEFAULT DEFAULT Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions DEFAULT Considerations for Versions • Decide how versions will be handled: - - • Lifespan Reconciling Conflict management Naming conventions Structure - Staging or QC version between user versions and DEFAULT - Security - Versions for groups or departments Workflow Management Systems for Handling Versions - Can provide workflows and efficiencies , some examples: - Job Tracking for ArcGIS (JTX) - ArcFM and Network Engineer – In the Utility Area Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Advanced GDB Functionality • Relationship Classes - • Geometric Network - • Persisted vs. temporal Performance implications Topology Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions User Workflows • • Document with Use Cases • A description of the task you need to perform: • “Add new parcel”, “Update new asset” • • • “Add new service” Evaluate business needs: • • Use case What data needs to be edited and in what order Tracking of data changes Conflict detection and resolution Security – user roles, etc. QA/QC steps – enforced through application or database Version update Geodatabase Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Data Performance and Scalability • Essential Tasks - - - Review anticipated data loads - Volume (data file growth management) - Volatility (storage partitioning) Identify key business transactions - Maintenance operations - Publication operations Identify performance requirements for key business transactions - Response time - Initial and scheduled user loads - Throughput - Testing Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Performance • Geodatabase designs - Potential performance issues related to database design - • Relationships - Both # and Type - Schema Cache can help reduce performance cost - Size of data stored in records - Projection on the fly - Number of records returned in a query - Density of data, both number of features and number of vertices Application design - Can have a significant affect on performance; e.g., - Frequently opening a table - Retrieving features one at a time vs. bulk Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Agenda • Overview • Database Design • Data Maintenance • Infrastructure Design • Data Distribution and Infrastructure Security • Database Maintenance & Performance Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Infrastructure Design Key Questions • Is it available enough? • Is it big enough (i.e., capacity)? • Is it continuous enough? • Is it performant enough? • Have constraints been removed? Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions System Availability • Define availability requirements • The Business defines their needs and states system availability as a non-functional requirement. • Example: "System should be available and online from 5am 10 pm PST 7 days a week with peak time 6am - 6pm." • IT responds by providing a standards-based technical solution. • Balance between benefits and costs • More servers or more complex servers • More servers means more software • More servers means more administration • Consider maintenance windows • Compress / DB statistics • Reconcile / Post services • Database schema changes / software patching • Database integrity checks post-restore Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions High-Availability DB Solutions • • • • Virtual Server Clusters* - Provides “basic HA” - Recovery time can be tens of minutes Active/Passive Fail-Over Clusters - Services fail-over to stand-by node - Down time measured in minutes - Semi-complex Active/Active Clusters (i.e., Oracle RAC) - Services fail-over to remaining active nodes - Down time measured in under a minute - Costly / Complex Fault Tolerant Clusters - Provides seamless failover - Zero down time - Costly / Complex Esri UC2013 . Technical Workshop . *Caution: Virtualizing “large” database servers not recommended. DB Server Processing Capacity • Processing capacity is a function of: - - CPU service time Throughput Max allowed CPU% Relative performance of the hardware Proper capacity is required to support expected peak user loads while maintaining reasonable performance. Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions DB Server Memory Capacity • Memory capacity is a function of: - - Number of DB instances Memory per connection Number of MXD layers Number of connections Database size Index size Providing adequate memory for the database server is critical for scalability and performance. Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Disaster Recovery & Business Continuity • • • High-availability addresses minor outages in a short time frame with largely automated means. - Server component failure - Storage failure Disaster recovery addresses major outages that are expected to last for a significant time period. - Flood / Fire / Earthquake - Core network failure - Major power outage DR is addressed by additional GIS computing infrastructure with replicated data in a secondary data center. Business continuity requirements dictate if GIS should participate in DR plans. Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Technology Selection and Performance • Technology selection is key to optimal performance. • Important to keep up with server advancements Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions • Client / Server processing distribution is typically 90% / 10% (depends upon spatial data type) • Client technology is typically a larger factor but don’t ignore the DB server Removing Constraints • The infrastructure can only be as good as constraints allow - - Is the server hardware adequate? Is the network adequate? Has the DB been tuned? Are the workflows reasonable? Is the storage architecture bottleneck free? Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Agenda • Overview • Database Design • Data Maintenance • Infrastructure Design • Data Distribution and Infrastructure Security Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Data Distribution and Infrastructure Security • Geodatabase connection architectures • Data distribution • Infrastructure security Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Geodatabase Connection Architectures SQL Queries Direct Connect Spatial Data types ` ` ArcSDE Connect (“Application Server”) ` RDBMS Client RDBMS Client ArcSDE Libraries ArcSDE Libraries RDBMS Client Geodatabase (Database Server) Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions ArcSDE (Application Server) Why Direct Connect Architecture? • It aligns with Esri’s long-term development strategy - Recent database technology has only been supported via direct connections - • • • It can perform faster - Assuming client or application server processors are faster - Network traffic can be reduced. It increases the scalability of the database server - Off-loads GSRVR processing to the client side - Reduces server memory needs It enables the ability to take advantage of database client-to-server security features - • e.g., Oracle RAC, ArcGIS Server Workgroup, IBM’s DB2 on z/OS, and 10.1 support of Oracle Exadata Database Machine e.g., Oracle Security It reduces deployment cost of ArcGIS for Server Enterprise - e.g., do not need to license cores on the DB server if 100% Direct Connect Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Data Distribution Options • • • • Copy/Paste - Export to FGDB / Import - Can be very time consuming - Does not synchronize GUIDs and Object IDs Source Target Database export/import - Target DB has to be stopped for the update - Can be very time consuming (entire DB export) DBMS level replication - Snapshot / Multi-master/ Merge / Transactional / Hybrid - Limited since NOT geodatabase or version aware! - Does not know how to properly replicate advanced geodatabase objects - Cannot edit DBMS replica using ArcGIS…only parent can be edited Geodatabase replication - See next page…. Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Geodatabase Replication • Leverage geodatabase replication when possible - Built on top of geodatabase versioning Supports the full geodatabase data model including topologies, networks, terrains, relationships, etc. Avoids limits or complexities associated with other methods Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Geodatabase Replication Use Cases • Mobile • Publication • Distributed Sites • Hierarchical Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Replication - Mobile Operations • Supports Mobile Operations - Leverages checkout/check-in replication Enables field editing in a disconnected state ArcGIS Mobile solution can drive database design decisions (e.g., check-in/check-out vs. ArcGIS Mobile). Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Replication - Publication • Supports Publication Environments - - Leverages one-way replication Typical scenario is data publication for web configurations Architecture supports varying NFRs (i.e., availability requirements) between data maintenance and data publication May have practical limits with very complex data sets (i.e., hundreds of relationship classes, large geometric networks, etc.) Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Replication – Distributed Operations • Supports distributed operations - - - Often based on twoway replication Typical scenario is data publication for distributed user sites Trend is moving away from distributed configurations towards centralization Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Replication – Hierarchical / Federated • Supports hierarchical configurations - - Leverages one-way replication Typical scenario state agency level rolling up to national level Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Data Publication Additional Benefits • • • Improves Performance / Scalability - Publication is often read-only nonversioned or FGDB - Requires versioned target for complex data types - Hardware isolation of editors and viewers Enhances Security - Separation of duties - Allow only a subset of data and attributes to be seen - Support for Internet / Cloud publication Supports Maintenance Activities - Separates DB server maintenance and data publication activities Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions ArcGIS.com Data Replication Network Strategies • • • Connected Synchronization - Fast Network (e.g., 1 Gbps) - Connects directly to the geodatabase - Low volume of changes Geodata Service Synchronization - Internet or WAN - Uses HTTP/S Web service - Low volume of changes - Supports check-out/check-in Disconnected Synchronization - Slow, latent, or no network - Large volume of changes - Uses check-out/check-in - Uses zip files Esri UC2013 . Technical Workshop . 58 Data Replication Essential Tasks • • • Define replication architecture - Check-out/Check-in - One-way / Two-way - Versioned / Non-versioned Target Identify data refresh requirements - Hourly? Nightly? Weekly? - Impacts architecture design decisions Understand replication constraints - Not all data types can be replicated (e.g., terrain data structures) - No imagery replication - Very complex data sets may be challenging (e.g., hundreds of relationship classes) Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Data Replication Essential Tasks • Identify data sets to be replicated - • • Identify specific data to be replicated - Don’t need to replicate the entire DB! - Options: Complete / By area / By attribute Source Select appropriate replication target data format - - • Check constraints FGDB - Fast performance - Simplified HA (multiple copies) - Consider file locking issues - Consider loss of functionality EGDB - Supports online sync - Supports functionality such as joins - More complex HA Prototype and test architecture (crucial) Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions .zip FGDB EGDB Target Data Sources Security 101 • Goals: - - Prevent unauthorized access, extracts, or editing of the Geodatabase Ensure data integrity Ensure data availability Ensure physical, policy, and technical controls are in place Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions DBMS Authentication Schemes • • • • Integrated with OS and network domain security - Windows: Simple integration with AD - Unix/Linux: Requires AD integration via LDAP Standard DBMS security - Requires logon using DBMS credentials - Option to save username and password with connection information Mixed mode - Combines Integrated and DBMS security - Recommended for SQL Server Connection Files - Can store user credentials for connecting to data sources - Define a strategy for managing and sharing connection files Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Database Encryption • Data can be encrypted “in-flight” - • Requires Direct Connect Oracle Advanced Security SQL Server SSL Data can be encrypted “at-rest” - TDE database standard (Oracle, SQL Server, etc.) Newer hardware helps reduce performance overhead Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Data Ownership • Use a generic account for data owner - • Use different accounts for different schemas - • Data can be manipulated and processed by a named user but then migrated to a generic user i.e., landbase vs. electric Avoid using the SDE account for data ownership Separation of duties - Only the SDE admin should have the SDE password - Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Data Backup & Recovery Considerations • Key considerations - System availability - Backup sizes and how to minimize down time - Speed of recovery - Transportability - Acceptable loss of edits - Consistency - Affects on performance Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Question & Answer Contact info: John Alsup [email protected] Jeff DeWeese [email protected] Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions Esri UC2013 . Technical Workshop . Planning: Enterprise Geodatabase Solutions