Download Wisconsin - Lessons learned through Node implementation and use

Node Lessons Learned
James Hudson
Wisconsin Department of
Natural Resources
Hudson, 3/1/2005
Nodes Evolve
• Phase 1: Getting the Node Working Done
• Phase 2: Making Data Flow Done
• Phase 3: Current
– Reproducible Implementation of Multiple Flows
– Incoming Data Flows
– Node Management and Administration
• Phase 4: Future
– New Standards (attachments, authentication, WSDL
changes, BPEL and orchestration)
– New Technology (Enterprise Service Bus, etc.)
2
Hudson, 3/1/2005
Wisconsin’s Perspective
• We were Not in the Beta or in Node 1.0
• “Early Adopter,” right after those states
• Built our own node (there were no DNCs)
• Built our own FRS flow (no FCDs)
• Continue to enhance our processing
• The node is key technology for our agency
and state
3
Hudson, 3/1/2005
Challenges
•
•
•
•
Node Management
Environment Changes
Designing for a Reproducible Process
Being able to implement necessary
changes
• Who has the correct data?
4
Hudson, 3/1/2005
The Node Management Challenge
• The Node Specification describes the Web
•
•
•
•
•
Services
The Flow Configurations and schemas describe
the Data and Requests
But there is no standard for what information to
store about your node
Or how to manage that node
Solution so far
– We built our own tools, as have others
Can this be Standardized?
5
Hudson, 3/1/2005
The Environment Challenge
• So far, Wisconsin has been through 2 version
•
•
changes in the Oracle Application Server
We’re moving to Websphere since that seems to
be the enterprise choice for the state, and will
give us a better service level agreement
Any change in environment takes time and effort
and can get in the way of other work
• Solution: invest the time
6
Hudson, 3/1/2005
The New Flows Challenge
• We have a stable node
• We want to implement at least 8 flows in the
•
•
next 2 years. And possibly up to 25.
How do we make those flows easy to build, easy
to change, and high performance?
Our strength is the database
– Data is there already
– So are our skills
– And our processing horsepower
7
Hudson, 3/1/2005
Wisconsin’s Version 1: Java
–
–
–
–
–
–
–
–
–
Configured Node using XML files
Logged to files on the Application Server
HTTP and SSL on the App Server
Used a utility in Java to query Oracle and extract the
XML
Converted relational to XML in Java
XSLT in Java
Schema Validation in Java
ZIP in Java
DIME in Java
8
Hudson, 3/1/2005
What We Learned from Version 1
• Application Server was hard for our staff
to administer
• Java performance was slow and required
too much memory for large operations
(XSLT on 100MB of XML, for example)
• Needed Expert Java Developers
• New flows, especially incoming flows,
would require writing too much code
9
Hudson, 3/1/2005
Wisconsin’s Version 3
• Node functions still in Java
– Configuration and logging stored in Oracle
– SOAP, ZIP, DIME all in Java
– May move most validation to CDX’s web services,
based on the presentation yesterday
• Flows nearly all in Oracle
– Java program stores the Request and calls a database
package
– Results come back to the Java program for packaging
and shipping (header still built in Java, for now)
10
Hudson, 3/1/2005
How Does the Flow Work?
• For each new flow:
– Convert Schema to Oracle Types
– Map to existing database tables by Object-Relational
Views (the hard part)
• We use database built-ins to
–
–
–
–
run query and convert to an XML document
parse using Document Object Model
XSLT
store the results
11
Hudson, 3/1/2005
What Does This Gain Us?
• Performance: FRS Solicit, consolidated schema,
•
•
•
•
•
40,000 facilities:
• 3.5 hours on production server with version 1
• 25 minutes on development server, version 2
Standard Steps for building a new flow
Less need to Redeploy the Java app
Better skills match
Better able to handle two-way flows
Less dependent on the application servers
12
Hudson, 3/1/2005
“What Data is Best?” Challenge
• This has been an issue since states started
reporting to EPA
– Can users trust the national data?
– Can they trust the state date?
– How can they tell why the data are different?
• With the Exchange Network, we may be
able to finally deal with this problem
13
Hudson, 3/1/2005
Final Comments
• The technology is getting much more
stable
• We’re dealing with data issues, like multiagency consistency, that we could never
address before
• It won’t be cheap or routine for a while,
and you’ll need programmers/contractors
• But we’re making a difference
20