* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Download Publishing Dynamic Information
Extensible Storage Engine wikipedia , lookup
Oracle Database wikipedia , lookup
Concurrency control wikipedia , lookup
Microsoft Access wikipedia , lookup
Microsoft SQL Server wikipedia , lookup
Relational model wikipedia , lookup
Team Foundation Server wikipedia , lookup
Microsoft Jet Database Engine wikipedia , lookup
Open Database Connectivity wikipedia , lookup
Database model wikipedia , lookup
Reference: 1.0, Issued (2nd September 2004) Publishing Dynamic Information A study of methods and three examples The purpose of this project is to investigate technology that can publish, specifically, dynamic information, using a common interface such as a Web Browser. An important factor in doing this project is to help the Author’s Cost Centre, the Data and IP Integration and Test Centre which is part of BT Exact. Graham Butcher Publishing Dynamic Information Graham Butcher Contents Chapter 1 : Introduction and Objectives .............................................................................. 1 1.1 Project and Work Objectives .................................................................................... 1 1.2 Document structure ............................................................................................... 1 1.3 Readership ........................................................................................................... 2 1.4 Technical Assumptions ........................................................................................... 2 1.5 Stakeholders of the report, Motivation for writing and Business Case ............................. 2 Chapter 2 : Overview of available technologies and assessment of their suitability ..................... 4 2.1 Introduction ......................................................................................................... 4 2.2 Web Server Side Coding Methods ............................................................................ 4 2.2.1 CGI Type Solutions ................................................................................ 4 2.2.2 Solutions using Apache, PHP and MySQL ................................................. 6 2.2.3 Simple Microsoft/ASP type solutions using IIS, PWS, MS Access ................... 8 2.3 Total Development Environment Methods ............................................................... 11 2.3.1 Cold Fusion – Server and Development Environment ................................ 11 2.3.2 Advanced MS solutions using Visual Studio.NET ...................................... 18 2.3.3 Third Party, Software Development Tools .............................................. 22 2.4 Enterprise Wide Systems ...................................................................................... 24 2.5 Other topics relevant to producing Database driven Web Pages ................................. 25 2.5.1 2.5.2 2.5.3 2.5.4 Client versus Server side scripting and programs ...................................... 25 Use of ODBC for connecting to databases ............................................... 25 What’s Oracle’s contributions to the discussion?...................................... 26 Security ............................................................................................. 26 2.6 A comparison of different solutions ........................................................................ 26 Chapter 3: An overview of the three working examples produced during the project ................. 29 3.1 Inventory and Calibration System – Concert ............................................................. 29 3.2 Enhanced Lab Equipment, Asset Register – UKIP...................................................... 31 3.3 Enhanced Document Management System – XGH1 .................................................. 33 Chapter 4: Details of the first example, Concert Test Equipment & Calibration Details solution .. 36 4.1 Requirements ..................................................................................................... 36 4.2 Background. ....................................................................................................... 36 4.3 Problem analysis ................................................................................................. 37 4.4 Functional analysis of web-access to DBs ................................................................ 41 4.5 Practical advice ................................................................................................... 41 4.6 Heuristics ........................................................................................................... 41 4.7 High level development ........................................................................................ 41 4.8 Scalability .......................................................................................................... 41 4.9 Ease of maintenance ............................................................................................ 41 4.10 Technologies v non-functional requirements .......................................................... 41 4.11 Design and Development of Solution .................................................................... 41 Chapter 5: Details of second example, the UKIP Asset Register ............................................. 43 5.1 Requirements ..................................................................................................... 43 5.2 Background. ....................................................................................................... 43 ii Publishing Dynamic Information Graham Butcher 5.3 Problem analysis ................................................................................................. 43 5.4 Functional analysis of web-access to DBs ................................................................ 47 5.5 Practical advice ................................................................................................... 47 5.6 Heuristics ........................................................................................................... 47 5.7 High level development ........................................................................................ 47 5.8 Scalability .......................................................................................................... 47 5.9 Ease of maintenance ............................................................................................ 47 5.10 Design and Development of Solution .................................................................... 47 Chapter 6: Details of third example, Centre’s Documentation Management System ................. 50 6.1 6.2 6.3 6.4 Requirements ..................................................................................................... 50 Background ........................................................................................................ 50 Problem analysis ................................................................................................. 50 Design and Development of Solution ..................................................................... 53 Chapter 7: Evaluation of Solutions .................................................................................... 55 7.1 Evaluation of solution 1 ........................................................................................ 55 7.2 Evaluation of solution 2 ........................................................................................ 55 7.3 Evaluation of solution 3 ........................................................................................ 56 Chapter 8: Recommendations .......................................................................................... 58 8.1 8.2 8.3 8.4 General Recommendations ................................................................................... 58 Recommendations from Solution 1 ........................................................................ 58 Recommendations from Solution 2......................................................................... 58 Recommendations from Solution 3......................................................................... 59 Chapter 9: Conclusions ................................................................................................... 61 9.1 9.2 9.3 9.4 Conclusions from Solution 1 .................................................................................. 61 Conclusions from Solution 2 .................................................................................. 61 Conclusions from Solution 3 .................................................................................. 61 Overall Conclusions ............................................................................................. 61 9.4.1 Use of Total Development Environments ...................................................... 61 9.4.2 Reliability and Robustness of Solutions Produced .......................................... 62 9.4.3 Use of Skills Developed during the Project for the benefit of the Test Centre……62 Chapter 10: Acknowledgements ....................................................................................... 63 Chapter 11: References ................................................................................................... 63 Chapter 12: Appendix A – Source Code for all Solutions ....................................................... 64 Chapter 13: Appendix B – Glossary ................................................................................... 64 Chapter 14: Appendix C – CodeCharge Product Feature Details ............................................ 66 Application Builder..................................................................................................... 66 Integrated Development Environment (IDE) ................................................................... 66 Form and Component Builders ..................................................................................... 66 Visual Query Builder ................................................................................................... 67 Security Management/Role-based Access Control ........................................................... 68 Support for Multiple Programming Languages ............................................................... 68 Extensibility .............................................................................................................. 68 Multiple Database Connections ................................................................................... 68 Microsoft FrontPage Integration ................................................................................... 68 Integration with Other Products ................................................................................... 69 iii Publishing Dynamic Information Graham Butcher Extensive Database Support ........................................................................................ 69 Application Templates (Solutions) ................................................................................ 70 Custom Code and Actions ........................................................................................... 70 Code Generation Engine ............................................................................................. 71 Sample Databases ..................................................................................................... 72 Chapter 15: Appendix D – ColdFusion Features ................................................................... 72 Figures Figure 1 - Example of an Application that uses CGI and Perl, Autosave Router Configurations ...... 5 Figure 2 - An example of WODA ......................................................................................... 6 Figure 3 - Web interface for MySQL management and administration ....................................... 8 Figure 4 - An example of ASP 3.0 ........................................................................................ 9 Figure 5 - MacroMedia's own description of ColdFusion plus the Price .................................... 11 Figure 6 - An Example of CFML ......................................................................................... 14 Figure 7 - CFML page ...................................................................................................... 16 Figure 8 - CFML with error ............................................................................................... 17 Figure 9 - Microsoft's pricing for Visual Studio.Net .............................................................. 18 Figure 10 - Web Matrix, design window ............................................................................. 20 Figure 11 - SQL Server control panel ................................................................................. 20 Figure 12 - Details of CodeCharge Studio's pricing structure ................................................ 22 Figure 13 - CodeCharge Studio IDE ................................................................................... 23 Figure 14 - PHP - ASP.Net comparison table from ref [5] ...................................................... 27 Figure 15 - Author's comparison table ............................................................................... 28 Figure 17 - View of Assets Table ....................................................................................... 30 Figure 18 - Login, obtain password page ............................................................................ 30 Figure 19 - Simple search page of UKIP Inventory ................................................................ 31 Figure 20 - Power search of UKIP Inventory ........................................................................ 32 Figure 21 - Documentation System, database chooser ......................................................... 33 Figure 22 - Documentation System, search for "Peakflow dos".............................................. 34 Figure 23 - Show versions of "Peakflow DOS upgrade ..." ..................................................... 34 Figure 24 - Requested document retrieved to user’s web browser .......................................... 35 Figure 25 - Starting point, existing NVT database - MS access ............................................... 37 Figure 26 - Existing database, view of Assets ...................................................................... 37 Figure 27 - Flowchart showing choices made for first solution ................................................ 39 Figure 28 – Features provided with Easily.co.uk Web hosting ................................................ 40 Figure 29 – Original user interface to UKIP Inventory database .............................................. 44 Figure 30 - Analysis of tables and relationships in UKIP Inventory database ............................. 44 Figure 31 - Flowchart showing decisions made for solution 2 ................................................. 46 Figure 32 - Analysis of FileStore "back end" database .......................................................... 51 iv Publishing Dynamic Information Graham Butcher Figure 33 - Flowchart to show decisions made for solution 3 .................................................. 52 Figure 34 - CodeCharge Query Builder, covering 3 tables ..................................................... 53 v Publishing Dynamic Information Graham Butcher Copyright © British Telecommunications plc, 2004. All rights reserved. BT maintains that all reasonable care and skill has been used in the compilation of this publication. However, BT shall not be under any liability for loss or damage (including consequential loss) whatsoever or howsoever arising as a result of the use of this publication by the reader, his servants, agents or any third party. All third-party trademarks are hereby acknowledged. Document history Revision Author Date Notes 0.A Graham Butcher 21st October 2003 First Draft after presentation 0.B Graham Butcher 31st May 2004 Second Draft 0.C Graham Butcher 22nd August 2004 Third Draft 1.0 Graham Butcher 2nd September 2004 Issued version Distribution Distribution will be controlled by: Graham Butcher Orion 3rd Floor, Adastral Park, Ipswich IP5 3 RE [email protected] tel: +44(0) 1473 649059 vi Publishing Dynamic Information Graham Butcher Abstract The skills required to be able to publish dynamic information on the Internet or an Intranet are as important today (2004) as using a word processor or business software was in the late 1980s. Those individuals and organisations that had (and have) these skills command a distinct advantage over their competitors in the business world. The purpose of this project is to investigate technology that can publish, this kind of information, using a common interface such as a Web Browser. The main objective of the project however, is to provide tools and skills that can benefit the Author’s work area - the Data and IP Integration and Test Centre which is part of BT Exact at Adastral Park. Within the Test Centre there are many teams doing a variety of activities – these include providing the test environment, testing itself, managing resources and dealing and communicating with end customers. Normally information relating to Inventories, Booking Systems, Configurations, Documentation, Ownership and other related topics is held in databases. If this information were permanently and instantly available, everyone would benefit. The aim of this project is therefore to look at different methods of implementing the display of non static information held in databases by using dynamic web pages. Three examples were produced for the project, adding value to the Author’s workplace. vii Publishing Dynamic Information Graham Butcher Declaration This dissertation is a result of my independent investigation. The portions of the report that are indebted to other sources have been referenced in the normal way. This dissertation has not been accepted in substance for any degree and is not concurrently submitted in candidature for any degree. Signed: Date: ________________________________ __________________________ viii Publishing Dynamic Information Graham Butcher Chapter 1 : Introduction and Objectives This section covers such things as separating the project objectives from normal work objectives, the document structure, the readership the document is aimed at and the type of people who would benefit from reading the report – the stakeholders. 1.1 Project and Work Objectives Project Objectives There are three main project objectives. The first is to find ways of publishing information within the Author’s work areas. This information will be specifically dynamic information. Publishing can be defined in this case as “making information available to a chosen audience through a common means” - without specialised software or dedicated programs. If specialised software or programs were to be used, they might need specialised installation and upgrading. In practice what is meant by a “common means” is a Web Browser. By dynamic information, the meaning for the purpose of this project is “information held in databases that is updated by a means separate from its publication and presentation”. In practice, this could mean, for example, displaying and searching a live inventory database (to a chosen audience) where updating that database would be kept separate from its display. The viewers would always see the latest information whereas the updaters of the information would not have to think about the presentation of the information ie they would not have to update the web page or web site. The second objective is to step back from the Author’s environment and look into all methods available - at the time of writing. Also investigated are reasons why one method is preferred over another, depending on the situation. The final objective is to produce some working examples – which can be used in the Author’s work area – hopefully to the benefit of all. Work Objectives During the period when the project was being researched the Author had several roles. These ranged from Managing Test Labs and Reference Models, to Testing International SDH Bandwidth and, finally, Co-ordinating Test Teams in the UKIP area. One common denominator to all these roles however, was a need to convey information. The tools that were expected to be used were such things as email, Word documents, databases and spreadsheets. Another common requirement – in all three situations - was keeping equipment inventories. A common situation that arose was for people to wish to know the whereabouts of test equipment so items could be borrowed when not in use or booked for future testing. To summarise, the Author’s work objectives were: Manage Test Labs – bookings, configurations, inventories, safety etc. Carry out Global Testing of High Bandwidth SDH, “super-highways” – as well doing the testing, keep track of test equipment and the calibration of instruments. Co-ordinate infrastructure for Test Teams in the UKIP test area – improve documentation and flow of information across the area. 1.2 Document structure This introduction includes the Project Objectives separated from normal Work objectives, Readership Aims, Technical Assumptions, who the Stakeholders of the Report are and the Motivation for writing the report. Section 2 steps back from the Project and looks at categories of solutions making suggestions about choices of technologies such as languages, platforms, web servers and Development Environments. Section 3 is an overview of the three working examples 1 of 81 Publishing Dynamic Information Graham Butcher produced. Sections 4, 5 and 6 document in more detail, the production, problem solving and decisions made to produce each of these solutions. Sections 7, 8 and 9 cover the Evaluations, Recommendations and Conclusions of the three respective solutions. The final Chapters, sections 11-15 cover such things as References, Source Code, Glossary of Terms used and Product Features of CodeCharge Studio and ColdFusion MX. 1.3 Readership In the early part of the project the readership was aimed at other Lab Managers who frequently needed to display information about such things as lab test equipment and bookings. Later on as the scope of the project broadened it became apparent that there were many other uses of the skills and techniques investigated that could benefit a wider audience. The readership of this report is therefore aimed at: Lab Managers wishing to make information available to colleagues and customers. Managers or Project Leaders wishing to commission bespoke publicising/publishing Systems. Database specialists and developers who seek to explore new presentational methods. Web site managers who wish to enhance their web pages with more information ie database based and searchable. Anyone wishing to publicise dynamic information on an intranet or the internet. Anyone wishing to Manage information – Minimising holding, Maximising Access. 1.4 Technical Assumptions A high level of technical knowledge is not assumed for use of this document. A basic knowledge of the following will be required: An elementary knowledge of HTML, tags and mark-up languages in general. A basic knowledge of the components that make up Databases – tables, queries, records, views, links etc Some knowledge of programming ie the difference between a compiled language and an interpreted one. Some knowledge of Operating Systems such as Microsoft Windows, Solaris and Linux 1.5 Stakeholders of the report, Motivation for writing and Business Case Stakeholders The main stakeholders of this report are: 1. Work colleagues wishing to publish (dynamic information) within their work environment. 2. People looking for an overview of techniques. “Time is money” and both could be saved if information was made, more readily available. Motivation The motivation for this research came from the period when the Author’s main job was managing Test Laboratories. There seemed to be a common need amongst Lab Managers to have a System that would not only cover bookings, inventories, loans, configurations and the like but also make the information available to users and managers alike. There were also issues about different versions of the same document “floating around” in emails. Sometimes people got left out of “the loop” and hence had wrong versions. 2 of 81 Publishing Dynamic Information Graham Butcher During this time various Systems were tried but were not highly successful because the information they provided was too difficult to access. The Systems were not aimed at the larger audiences. Another observation made was that when specialists were brought in they only provided one type of solution ie there was little choice available. The Author visualised having the skills within his work area where people could quickly build intuitive, flexible, open Systems that would allow the free flow of information. Business case The business case is fairly simple. If suitable Systems were produced there would savings in the following areas: 1. Email storage – only one copy, the latest version, would be published with emails just carrying the links to the location of the information. 2. Time – to find documents. Searching is more efficient with dedicated search tools and less time is wasted looking for information. 3. Version control – a definitive version of a piece of information or document would be available. 3 of 81 Publishing Dynamic Information Graham Butcher Chapter 2 : Overview of available technologies and assessment of their suitability 2.1 Introduction This section attempts to stand back from the project and tasks in hand and tries to give an overview of the techniques, methods and products that can help produce database driven web pages and sites. In the Author’s view, there are roughly 3 categories of solutions: Simple Web Server side coding methods. These involve writing and running programs on a web server that, when called, go off and perform tasks like connecting to databases, querying them and outputting the results in HTML to the user’s browser. In the old days this was done by CGI scripts but nowadays dedicated scripting languages such as PHP, ASP and CFML are frequently used. A knowledge of one of the PHP, ASP or CFML languages is required for this type of solution or some knowledge of how CGI works. Total Development Environment methods. Three such Systems are looked at: a) The Microsoft contribution - Visual Studio.Net b) The Macromedia (DreamWeaver) environment version - Cold Fusion. c) An independent company solution like the CodeCharge Studio product from YesSoftware. This third example, unlike the first two, is technology independent, producing solutions to most platforms using code and databases of the user’s choice. The product generates the code. These types of solutions tend to be for more time-complex situations and, in the case of the first two examples, more proprietary. The third example is slightly different in that it is not tied to a platform. In all three, there are varying degrees of extra cost involved. The three examples given are by no means the only ones available but probably reflect a reasonable cross section of the market. Enterprise Wide Systems. These are not covered in this discussion as they are usually multi-million pound corporate wide solutions. They are ordered at board level and make use of technologies such as Java, XML and heavyweight distributed databases. They could involve a 3 level model – including an intermediate application server with corporate strategy implemented. These solutions are considered beyond the scope of this report. Since starting this project BT has adopted several such Systems – the Niku Enterprise System and the LiveLink document management System. There are of course grey areas and overlaps between these artificial segregations of categories. First we will look at simple Server Side Scripting Solutions. 2.2 Web Server Side Coding Methods 2.2.1 CGI Type Solutions In the early days before dedicated web scripting languages such as ASP and PHP were developed programmers and developers had to produce their own code, native to the operating System hosting the web service. They used existing languages supported on the web server. These specialised pieces of code were usually placed in a directory called the CGI bin – the Common Gateway Interface, binaries folder or directory. Although any language, compiled or interpreted is acceptable, the common choices were: I. Report writing language of PERL. II. A scripting language such as csh – C shell scripting language III. A compiled language such as C or C++ The only requirements for this method are: The program must work on the web server. It must be capable of accessing the data or database, hosted locally or remotely 4 of 81 Publishing Dynamic Information Graham Butcher It must also be capable of outputting the results of the queries or searches in HTML so that the user can view them in his/her browser. This method is still used for complex or specialised solutions. Sometimes, historically the bulk of the coding has already been written from a previous proprietary solution and can be re-used for a more open and efficient web based solution. An example of CGI and Perl, used in the Author’s work area is the Autosave tool that captures and displays various configuration information from equipment such as Cisco Routers. Figure 1 - Example of an Application that uses CGI and Perl, Autosave Router Configurations A second example of using CGI and Perl to produce Database Web Pages One of the first examples of a complete Web based database System is WODA (Web Oriented Database), which is claimed by the Designer, Prof Ziga Turk, to have originated around 1995. The System is completely written in Perl. Turk also comments “if in 1994 concepts like active server pages and database Systems built right into the HTTPD servers were available, as easily as today, the evolution of WODA would perhaps be different” ref [6]. Turk claims that WODA is still used internally by NASA, Siemens and Amazon.com. 5 of 81 Publishing Dynamic Information Graham Butcher Figure 2 - An example of WODA An example of a Dynamic Database Web site produced exclusively using WODA Summary (of CGI method) A couple of disadvantages of using CGI programs are: a) all the commands used to contact and connect to databases must be known and included in the CGI code. With the dedicated languages such as PHP and ASP, the details are built into the languages and less knowledge is required. b) if five different people submit queries via CGI, five copies of the program have to run to fulfil the requests. This can make a popular web site slow down significantly. An advantage of using CGI is that existing non Web based solutions can be ported to the Web using the original code with minimal alterations required. This would only be a short term solution however, that doesn’t take advantage of latest developments in the Web, Database arena. 2.2.2 Solutions using Apache, PHP and MySQL This type of solution appeals to people on a tight budget or people familiar with the UNIX/Solaris/Linux platforms. All three components – Apache, PHP and MySQL - that make up the solution are effectively free. Apache is a freely available complete Web Server that is usually deployed on a UNIX Operating System such as Solaris or Linux. Apache accounts for approximately 65% of the worlds Web Servers [ref1]. As well as supplying complete Web Hosting Services, additional free components called modules can be added. These modules supply the following services [ref1]: CGI script handling Site activity logging Dynamic HTML Proxy services Direct Access to Perl User authentication Image maps Encryption and security User and activity tracking Java servlets 6 of 81 Publishing Dynamic Information Graham Butcher Commercially produced modules or components can also be added. Apache originally came from the NCSA (National Centre for Super computing Applications) organisation. Its name came from the fact that, because the earlier versions had so many software “patches”, the product subsequently became known as “Apache”. Apache is available free of charge at www.apache.org or ftp.apache.org PHP (Hypertext pre-processor) is a freely available scripting language specially designed for web applications and database access. It is available at www.php.net. The advantages of PHP are: It’s fast. Because it is embedded in HTML code, the response time is short It’s inexpensive – free in fact It’s easy to use – similar constructs as other popular languages such as C and C shell Runs on many platforms – Linux/UNIX, Windows and Mac OS Technical support is widely available through discussion groups It’s secure – the user does not see the PHP code It’s designed to support databases – most supported It’s customisable – the open source licence allows programmers to modify PHP for their own specific needs or environments if necessary. PHP works by embedding scripting code in normal HTML using extensions to the HTML tag method. A piece of PHP code will start with the <?PHP tag and finish with the ?> tag. As a result all HTML statements are sent to the browser “as is” but PHP statements are pre-processed and outputted in HTML before sending to the browser. Eg <?PHP echo “<p>Hello World”; ?> Will be processed to <p>Hello World which is HTML – Hello World is displayed in the user’s browser. The consequence of this is that the end user never sees the PHP code - making it transparent and secure. MySQL is a freely available full RDBMS (Relational Database Management System) similar to Oracle. It is available at www.mysql.com. MySQL is developed, marketed, and supported by a Swedish company MySQL AB. There are two types of licence: I. Open Source Software – available via the GNU General Public Licence (GPL). Anyone who meets the GNU requirements can use the product for free, even if they are making money from it. The disadvantage is that developments have to be made public. II. Commercial Licence – MySQL is available at a modest price for developers who want to produce and sell new products that use MySQL rather than release them under GPL. MySQL comes with a complete Web based Management and Administration System. 7 of 81 Publishing Dynamic Information Graham Butcher Figure 3 - Web interface for MySQL management and administration Putting the three products – Apache, PHP and MySQL - together makes an attractive, open, low budget solution for implementing web based database Systems. 2.2.3 Simple Microsoft/ASP type solutions using IIS, PWS, MS Access or SQL Server The reason why this section is called “Simple Microsoft …” solution is to distinguish it from solutions provided in section 2.2.2 where a more advanced MS solution is described. The solutions that are described in this section use a proprietary Microsoft language called ASP – Active Server Pages. This language is very similar to PHP, described in 2.2.2, in that it was designed specifically for web applications and connecting to databases. It would probably not be an exaggeration to say that ASP, sometimes known as ASP 3.0, is the most common language used to produce web pages with dynamic contents. ASP will run automatically on all Microsoft based web servers - IIS and PWS – but will not work on Apache or other web servers. This is because there is no MS install program available. ASP is built into MS web servers. To get round this problem however, a third party company called Chilisoft produced an “open” version of ASP, called Chilisoft ASP. This product is fully compatible with non-Microsoft platforms and servers. Chilisoft ASP used to be freely available at www.chilisoft.com When that URL is now accessed though, you are re-directed to Sun Java System, Active Server Pages 4.0 so it looks as though Sun have bought out the company and distributes it as part of its web server products. Chilisoft ASP is basically an “open” version of Microsoft’s language designed to run on all platforms. Chilisoft ASP is particularly useful for porting legacy Systems from a MS/ASP environment into an open one. 8 of 81 Publishing Dynamic Information Graham Butcher Microsoft’s main Web Server product is called IIS – Internet Information Service. The product comes free with various versions of Windows – usually known as “professional” eg Windows XP, Professional. With Windows 98 and 2000 there is a free optional installable product called PWS – Personal Web Server. This is a light weight Web Server, useful for testing out ASP code and Web pages on your PC. Microsoft’s two database offerings are Access and SQL Server. The former is more aimed at the single user customer with the latter aimed at the multi-user heavy weight end of the market. Many web based database solutions however, run quite happily using Access as their database. A simple example of ASP (written in VB script – Visual Basic Script) code would be: <html> <body> <center> <% Dim ftsize, ftcolor%> <% For ftsize = 1 to 7 %> <font size = <%=ftsize%>> Hello and Welcome!<br> <% Next %> </center> </body> </html> NB This is actually a mixture of HTML and ASP. The ASP code, embedded in the HTML and highlighted starts with <% and finishes with %>. In this way it works like PHP and is not visible to the end user. The result of calling this short bit of code would be to display the words “Hello and Welcome!” in 7 different font sizes. Figure 4 - An example of ASP 3.0 By using, for example, IIS, ASP and Access, a quick and simple web based database System can be built. A knowledge of the ASP language will however be required. Alternatively, if additional tools are used like Microsoft’s “Front Page” – their web page and site generator - then wizards and short cuts can also be used for simple solutions without any knowledge of ASP. 9 of 81 Publishing Dynamic Information Graham Butcher Also, a dedicated code generation tool such as CodeCharge Studio (see section 2.3.3) can be used which removes the need to learn any languages. 10 of 81 Publishing Dynamic Information Graham Butcher 2.3 Total Development Environment Methods The second category of solutions is Total Development Environments category. A sample of three is given here – Cold Fusion, Visual Studio.Net and CodeCharge Studio. 2.3.1 Cold Fusion – Server and Development Environment The first example in this category is ColdFusion. It is generally considered that Dreamweaver is the best Web Page Development Tool. The manufacturers, Macromedia, also have a strong hold on Web graphics with their Flash and Fireworks products. Their Integrated Development Environment (IDE) includes a product called ColdFusion that integrates database and data sources into Web pages. It is possible to download the complete ColdFusion package for a month’s free trial. When the trial expires it is possible to keep the development environment for use with a single IP address. This allows developers to effectively continue to develop pages on their own computer and deploy them on other existing, licenced ColdFusion servers. The price of the development and server software is listed below - $1299 for the Standard Edition (correct in July 2004). This is considered a little too expensive for the Project. It is also felt that ColdFusion’s influence is now slightly on the wane see ref [2] “ColdFusion had become less popular. Its embedded tag has also been embraced in different formats by other Web technologies…” The following discussion – obtained from ref [3], www.macromedia.com explains how ColdFusion works and how you develop database driven dynamic web pages using the Macromedia ColdFusion environment. Figure 5 - MacroMedia's own description of ColdFusion plus the Price What is ColdFusion MX? ColdFusion MX is a powerful web application server that lets you create robust sites and applications without a long learning curve. ColdFusion MX does not require coding in traditional programming languages (for example, C/C++, Java, XML), although it supports these traditional programming languages. ColdFusion MX consists of the following core components: ColdFusion server ColdFusion Markup Language (CFML) ColdFusion Administrator 11 of 81 Publishing Dynamic Information Graham Butcher The ColdFusion application server The ColdFusion application server is itself a web application that typically resides on the same computer as your web server software ie it can co-exist with IIS and Apache. It is the program that parses (reads and interprets) and processes supplied instructions. These instructions are passed to ColdFusion through ColdFusion pages, which use a CFM or CFC file extension. A ColdFusion page looks like an HTML page but contains special tags that instruct the ColdFusion server to perform specific operations. In this way CFML is similar to PHP and ASP 3.0. How ColdFusion processes pages The ColdFusion server looks at the content of the page and searches for the following ColdFusion instructions: Tags that begin with cf. Variables and functions that are always surrounded by hash signs (#). If the ColdFusion server finds any HTML or plain text in the page, the ColdFusion server returns it to the web server untouched. The ColdFusion server processes all the ColdFusion instructions found, and returns any remaining results to the web server. The web server then sends the entire output to the browser. The ColdFusion Markup Language ColdFusion Markup Language (CFML) is a tag-based language similar to HTML that uses special tags and functions. With CFML you can enhance your standard HTML files with database commands, conditional operators, and high-level formatting functions, and rapidly produce easyto-maintain web applications. CFML looks similar to HTML: it includes start and end tags, and each tag is enclosed in angle brackets. All ending tags are preceded with a forward slash (/) and all tag names are preceded with cf; for example: <cfstarttagname> </cfendtagname> Building applications with CFML You build ColdFusion applications as a series of pages that use CFML. Developers can extend this language by creating their own custom tags or user-defined functions (UDF), or by integrating COM, C++, and Java components (such as JSP tag libraries). Interacting with data sources ColdFusion applications can interact with any database that supports a JDBC technology-based driver. A JDBC technology-based driver uses an Application Programming Language (API) to execute SQL statements to databases on most platforms. However, ColdFusion is not limited to JDBC data sources. You can also interact with existing Open Database connectivity (ODBC) data sources by using ODBC Socket, a driver that interacts with an existing ODBC driver. Development tools While you can code your ColdFusion application with NotePad or any HTML editor, Macromedia recommends that you build your applications using Macromedia Dreamweaver MX. Dreamweaver MX offers features and wizards that enhance ColdFusion development. 12 of 81 Publishing Dynamic Information Graham Butcher The ColdFusion MX Administrator You use the ColdFusion MX Administrator to configure and maintain the ColdFusion Application Server – the Web Server. It is a web-based application that you can access using any web browser, from any computer with an Internet connection. You can manage the following configuration options with the ColdFusion MX Administrator: ColdFusion data sources Debugging output Server settings Application security Creating a ColdFusion page Creating a ColdFusion page involves using tags and functions. The best way to understand this process is to create a ColdFusion page. In the following procedure, you will create a simple ColdFusion page by using HTML tags, one ColdFusion tag, and two ColdFusion functions. The following table briefly explains the ColdFusion tags and functions: Element Description. Now() A function supported in CFML that you can use to retrieve information from your System. You will use the Now ( ) function in the following procedure to return the current date that is retrieved from your System. DateFormat() A function that instructs ColdFusion to format the date returned by the Now() function. cfoutput A ColdFusion tag that you use to return dynamic data (data retrieved from a database) to a web page. You will use the cfoutput tag in the following procedure to display the current date retrieved from your System. Note: ColdFusion tags and functions are considered primary elements of CFML. To create a ColdFusion page: Open your editor and create a blank file. Enter the following code on the page: <html> <head> <title>A ColdFusion Page</title> </head> <body> <strong>Hello world, this is a ColdFusion page.</strong> <br> <cfoutput> Today's date is #DateFormat(Now())# </cfoutput> </body> </html> 13 of 81 Publishing Dynamic Information Graham Butcher The following figure shows the cfpage.cfm in the browser: Figure 6 - An Example of CFML Understanding CFML elements CFML consists of two primary language elements: tags and functions. Tags let you perform operations such as accessing a database. Functions can return data and do other operations like retrieving the System date. Almost everything you want to accomplish with ColdFusion will involve using tags and functions. You will use another important element known as a variable. Variables are an important part of most programming languages and are equally important with CFML. Variables let you store information in memory and enable you to pass data. The following sections describe how to use these three elements. Tags You can think of tags as commands that you use to instruct the ColdFusion server to perform operations. These operations might include selecting data from a database, reading a file that resides on the server, or showing the results of processing. Tag syntax ColdFusion tags are similar to HTML tags. ColdFusion tags are enclosed in angle brackets and often have a start and end tag. The start tag encloses the tag name in brackets, like this: <tagname> Most often the end tag encloses the tag name in brackets and includes a slash (/), like this: </tagname> The information processed by ColdFusion is placed between the start and end tag, like this: <tagname> info to be processed ... 14 of 81 Publishing Dynamic Information Graham Butcher </tagname> ColdFusion tags, for the most part, share these common characteristics: All start with cf. A start and end tag. Use of attributes (like html tags), and most attributes have values. Some ColdFusion tags, such as cfset, omit the closing tag. This type of tag uses one set of angle brackets and places all the required information between the left (<) and right (>) angle brackets, like this: <cfset name="bob"> Tag attributes Tag attributes instruct the ColdFusion server about the details of an operation. For example, to update a database table, the server needs to know specifics about the database, such as the database name and the table name. The code required to write this type of statement might look like this: <cfupdate datasource="mydb" tablename="mytable"> where datasource and tablename are attributes of the cfupdate tag and "mydb" and "mytable" are attribute values. Functions Typically, a function acts on data. It can generate a value or a set of values, usually from some input. You can perform the following operations (actions) with functions: Manipulate data and time values Examine a value or variable Display and format information Manipulate string data Retrieve System information and resources Perform mathematical operations Using functions on values Usually, a function performs an operation on a value, and the value can include the value of a variable. For example: to format the value of a variable containing a value in dollars, the code to write this statement might look like this: #DollarFormat(price)# The DollarFormat function returns a value as a string and formats that value with two decimal places, thousand separator, and dollar sign. The hash signs (#) around the function instruct ColdFusion to evaluate the content between the hash signs and display the value. Functions and parentheses All functions have parentheses, regardless of whether the function acts on data. Consider the following function: #Now()# 15 of 81 Publishing Dynamic Information Graham Butcher If you put anything inside the parentheses of the Now() function, an error would occur. The Now() function returns an unformatted date and time. However, you can format the results of this function with other functions, such as the DateFormat() or TimeFormat() functions. Nesting functions Functions can generate data as well as act on data. Consider the following example: #DateFormat(Now(), "mm/dd/yyyy")# In this example, the Now() function generates the date, and then the DateFormat function formats the date. Functions and hash signs You use hash signs (#) with functions to display the results of a function on the page. Hash signs tell the ColdFusion server to evaluate the content between the hash signs and display the value, for example: <cfoutput> Hello world, <br> Today's date is #DateFormat(Now(), "mm/dd/yyyy")# </cfoutput> The following figure shows the output of this example: Figure 7 - CFML page If you did not include the hash signs around the DateFormat(Now(),"mm/dd/yyyy") function, ColdFusion would not evaluate the function and the previous example would display your source code, as follows: 16 of 81 Publishing Dynamic Information Graham Butcher Figure 8 - CFML with error Variables Variables let you store data in memory on the server - they always have a name and a value. You can assign a value to a variable, or you can instruct ColdFusion to assign variable values based on data that it retrieves from a data source, such as a database table. Summary ColdFusion was probably the first robust integrated development environment introducing features such as server side scripting with CFML (ColdFusion Markup Language) using embedded declarative tags. It is now losing popularity a little to cheaper more versatile solutions. CFML requires a ColdFusion Web Server to host the site, applications and CFML web pages. Yue and Ding ref [2] writing from Design and Evolution of an Undergraduate Course on Web Application say “The initial inclusion of ColdFusion in 1998 and its exclusion in 2000 may serve of an example …” ColdFusion would suit the developer that works exclusively with Macromedia tools and has a ColdFusion Web Server to publish on. 17 of 81 Publishing Dynamic Information Graham Butcher 2.3.2 Advanced MS solutions using Visual Studio.NET The second example of Total Development Environment category of solutions is Visual Studio.Net. Visual Studio.Net is Microsoft’s current development environment for Web, Windows and Mobile applications. It costs $1079. Figure 9 - Microsoft's pricing for Visual Studio.Net The Integrated development environment contains the following elements: A common IDE for the supported languages: o Visual Basic.Net o Visual C# o Visual FoxPro o Visual C++.Net Easy ASP.Net programming ie WYSIWYG (what you see is what you get) and Code Aware Editors The .Net initiative started to take off around the middle of 2000. It came about because of the increasingly important role of the Internet, more collaboration between developers and the gradual shift from desktop computing to distributed computing. It’s method of communicating between applications, environments and platforms is XML (eXstensible Mark-up Language). As people have some knowledge of the older MS languages of Visual Basic and Visual C++ this discussion will just focus on the new elements of the package – ASP.Net and C# languages. 18 of 81 Publishing Dynamic Information Graham Butcher ASP.Net is Microsoft’s next version of (ASP) Active Server Pages – the earlier version, ASP 3.0 is mentioned in section 2.2.3 of this report. The main difference between ASP 3.0 and ASP.Net is that the latter is object oriented and compiled. It can be produced within Visual Studio.Net IDE using any of the languages mentioned earlier. The advantages of ASP.Net over ASP 3.0 are: Unlike ASP 3.0, ASP.Net is compiled therefore applications will run faster. Just In Time (JIT) binding is possible. ASP.Net provides DataGrid and DataList controls tailored for efficient database access. State management is handled better. ASP.Net offers three types of states for Web applications: application, session and user. Improved caching. A complete caching API is implemented. One of the benefits is improved performance when interacting with databases through better caching. Better updating. With 3.0 it’s necessary to shutdown the Web server to update the ASP with ASP.Net it’s not. Multiple audience targeting. The same page can deliver “text only” for a mobile phones or rich-dynamic content for IE 6.0. This is a built-in function that detects the browser being used. Object oriented. Better re-use of code is achieved. Upgrading from ASP 3.0 to ASP.Net is achieved in one of two ways: 1. Rewrite the code in ASP.Net 2. Change the file extension from .asp to .net Visual C# is Microsoft’s latest language that combines the strengths of C++ with its low level controls with the simplicity and RAD features of Visual Basic. It also has the added benefit of being able to convert applications produced in C# into XML web services available to a range of platforms and other environments. To encourage people to move to the .Net method of development, Microsoft are giving away, free, certain elements of Visual Studio. One of these is a product called Web Matrix, an editor for producing and compiling ASP.Net pages. It is a visual editor with elements such as DataGrid and DataList “drag and droppable” onto Web pages. Also provided free is a run time version of their Database System, SQL Server. More information is available at their ASP.Net community site www.asp.net ref [7]. 19 of 81 Publishing Dynamic Information Graham Butcher Figure 10 - Web Matrix, design window Figure 11 - SQL Server control panel 20 of 81 Publishing Dynamic Information Graham Butcher Summary Using .Net technology is an investment for the future as the older Microsoft ASP 3.0 is likely to be phased out and unsupported some time in the future. The visual Studio.Net environment also has the advantage that it can develop for the Web, for Windows and for Mobile devices such as PDAs. This means that applications written for one platform can be ported across relatively easily to another. The prohibitive cost of the product – Visual Studio.Net – and lack of free trial period prevented an in-depth study in this project. 21 of 81 Publishing Dynamic Information Graham Butcher 2.3.3 Third Party, Software Development Tools (CodeCharge Studio from Yes software) The third example in this “Total Development Environment” category is CodeCharge Studio. Figure 12 - Details of CodeCharge Studio's pricing structure – NB, The Personal Edition does not cover all languages This type of solution has created quite a buzz in the industry as companies like Yes Software are now challenging the big names – Microsoft and Macromedia – in the field of Rapid Application Development of dynamic database driven web pages. At the start of the project this type of solution wasn’t considered important but as the project progressed, this particular tool became more and more important. A simple internet search revealed CodeCharge Studio as being a method of developing code for database web applications. The full blown product is available, free of charge, for one month and consists of a 25 MB download. After 3 months of trials (on 3 separate computers) the Author was so impressed that the product was purchased at the full price of $180. Subsequently the price has risen to $499 see above. CodeCharge’s own sales information is quoted as saying “CodeCharge Studio is a visual application builder and code generator that provides a feature-rich environment for rapid application development for the web. It is the most productive way available today to create powerful, scalable, and secure web applications quickly. Based on a sophisticated code generation engine, CodeCharge Studio opens up new possibilities for web developers by automating the creation of virtually all web application components and by generating robust, professional-level server code in any of the following programming languages: ASP.NET (C# and VB), ASP 3.0, PHP 4.0, Java Servlets 2.2, JSP 1.1, ColdFusion 4.01, and PERL 5.0” ref [4]. 22 of 81 Publishing Dynamic Information Graham Butcher How CodeCharge works CodeCharge Studio is functionally and visually similar to other RAD tools, such as Microsoft Studio.Net and Macromedia ColdFusion with the ability to maximize developer productivity while keeping the cost of development to a minimum. While visual programming tools are traditionally used for developing desktop and client applications, CodeCharge Studio is ideal for developing sophisticated web applications. Figure 13 - CodeCharge Studio IDE Project development using CodeCharge The following steps demonstrate how a basic application would be created using CodeCharge: 1. Start CodeCharge Studio. 2. Create a new project. 3. Name the project and select Application Builder. 4. Specify project settings such as the programming language and path to the web server. 5. Specify the database connection type and establish a database connection. 6. Configure site authentication by specifying the database table and columns that contain user login and password information as well as security groups and roles. 7. Select the database tables based upon which the web application will be created. 8. Specify the type of web page to be created for each of the previously selected database tables. Specify if you want to create Search, Grid, and Record maintenance pages for each table. 9. Specify a theme to be applied to the site. The theme defines the basic layout and appearance of different elements within a page. 10. The Application Builder will automatically create a project with sets of pages that make up the web application. You can then publish the pages to the server and test them. 23 of 81 Publishing Dynamic Information Graham Butcher 11. If you wish, you can make further modifications to the created application by using the integrated development environment (IDE). CodeCharge Studio does not help you develop the underlying databases that drive the applications. It does however, let you connect to existing databases during development and to test your application with real database connections during testing. CodeCharge Studio Features The following is a summary of the CodeCharge sales information available at www.codecharge.com ref [4]. The main features are: 1. Application Builder 2. Integrated Development Environment 3. Form and Component Builder 4. Visual Query Builder 5. Security Management/Role-based Access Control 6. Support for multiple languages 7. Extensibility 8. Multiple Database Connections 9. Integration with other products 10. Extensive Database Support 11. Application Templates (Solutions for customisation) 12. Custom Code Actions 13. Code Generation Engine The details of these features are included verbatim in Appendix C. Benefits of using CodeCharge Studio The following benefits are obtained by using CodeCharge Studio: 1. CodeCharge generates structured, object oriented code in the language of your choice. 2. As the code is well structured it is easily maintainable and can be extended manually if necessary. If this done within CodeCharge’s own editors the new code is not over-written during subsequent use of the automatic code generator. 3. The code produced separates the programming logic from the presentation layer ie the files .asp, .php, .jsp, etc files are separate from their corresponding HTML files. In large scale developments this allows separate teams to work on programming and visual design. Summary CodeCharge Studio offers an attractive and flexible way of rapidly producing Dynamic Database Web pages. It would be the solution that the Author would recommend that people, new to the arena, look at first. 2.4 Enterprise Wide Systems As stated earlier, this type of solution was considered beyond the scope of the project and is only included here for the sake of completion. 24 of 81 Publishing Dynamic Information Graham Butcher 2.5 Other topics relevant to producing Database driven Web Pages This section includes additional pieces of information, critical to the understanding of web based database technologies. 2.5.1 Client versus Server side scripting and programs Up until now all the programming languages and code have been designed to work on the Web Server side of the Client/Server model used for discussing web pages. One (or two) important languages that run instead in the client’s browser are Java script (and the MS version JScript). These are important because they, for example, can verify data input to a form. This saves time as the browser doesn’t need to access the server to do this. Basic checks for data integrity of things like telephone numbers and email addresses are easily achieved. The only problem with this client side programming is that unlike HTML, there are few open, independent standards and different browsers behave in different ways to interpret the same code. A way to get round this is to make the scripting program first check the browser type and version before re-directing to the required code for that browser. Also worth mention are Java Applets and Java Servlets. These are programs, written in Java, a complex language based on C++, that run on the Client (PC, Mac or other) or the Server respectively. Obviously, if they run on the client PC (or other) they have the possibility of doing a lot of damage. This could be things like installing a Trojan, a Virus or merely carrying out the command “format C:\” on a PC machine. This means that with normal browser settings the user is first asked if he wants to accept the downloaded program before it can run. 2.5.2 Use of ODBC for connecting to databases ODBC is the “glue” that allows different applications to access different databases from different manufacturers in a consistent way. It existed long before web based databases were thought of. ODBC originated from Microsoft’s Windows Open Systems Architecture (WOSA), which provides a series of application program interfaces (APIs) to simplify and provide standards for various programming activities. The goal is to have all applications communicating through the same set of APIs. ODBC is just one piece of the WOSA picture. Other components include telephone services (TAPI), messaging services (MAPI), and open data services (ODS). Today it is generally treated as an “open” standard. Normally before accessing an ODBC database, you must install on your System the appropriate ODBC driver for that database (if it’s not installed already). A Data Source Name (DSN) is what ODBC uses to allow you to associate a database with a driver. You use the ODBC Administrator (in Control Panel on PCs) to configure and give the DSN a unique name and then associate it with both a database and a driver. When you open a connection to ODBC, you specify what DSN you want to work with. ODBC takes care of all the little details involved in opening that database. Before you can access ODBC databases, you must configure the ODBC data source names, the ODBC drivers, and the configuration values used in ODBC.INI. You also should understand the structure of an ODBC driver and the ODBC API as well as some ODBC-related terminology. In order for any application to access data in a database, it must establish a connection to that database through its corresponding ODBC driver. To do this, the application must request a connection from the ODBC Driver Administrator, specifying the data source desired. Although ODBC is the oldest method it is by no means the only way of connecting to databases. Other methods are JDBC, ADO (Active Data Objects) and Jet (MS Access database engine). A tool like CodeCharge Studio will allow you to choose the type and sort out the necessary code for connecting to databases automatically. 25 of 81 Publishing Dynamic Information Graham Butcher 2.5.3 What’s Oracle’s contributions to the discussion? Oracle is the world’s largest database company. Along with Oracle v7 and possibly v8, there used to be a free Web Server and Wizards that allowed you to develop dynamic web pages. Although this appears to be supported with the latest version, Oracle’s support seems to be withdrawing and they generally seem to be recommending other proven methods such as PHP and ASP.Net. See Hull, ref [5]. 2.5.4 Security Web and Database security are massive subjects in their own right. The discussion in this report is limited to a few mistakes that can be made when designing Database Web pages and sites: 1. Access database (or other single file database) can be downloaded. This actually happened where a holiday company made available all their customers credit card and other details stored in a single access database. The hacker types something like: http://www.holiday-company.com/customers.mdb and the browser asks if the file should be opened or saved. Even if the hacker doesn’t know the name of the database file, it can be found by a “trial and error” program. The way to protect against this is to obviously, locate all data files in an area that can be reached by the code but not browsers ie do not use the root directory for any data. 2. Permissions on all files on the Web Server have to be correct. On some web sites, typing an incorrect URL gives you a listing of all files held their. There is then the opportunity to download files that were not meant to be downloaded. Sometimes database usernames and passwords have been known to be stored in text files accessed by the code. 3. All Web servers have a default file or default filenames that are activated even though not specified. The most common is index.html or index.htm. The developer needs to be aware of all the Web Server settings when designing the code to be used. 2.6 A comparison of different solutions This is one of the hardest sections to write as there are so many possible combinations of different products and technologies. When making choices the starting point is usually what is available to you at the start. This may be anything from a Web Server, knowledge of a certain language or use of a certain development tool. In the case of this project, the Author started with virtually nothing so using an external hosting company with many built in features appeared to be an attractive choice. More information is given in details, assessments and conclusions of the three solutions in this report – particularly in figures 27, 31 and 33. The following table however, does try to compare two elements and focuses on comparing PHP with ASP.Net. It is from an article by Sean Hull, PHP and ASP.Net go head to head, Ref [5] PHP 4 PHP 5 ASP.NET Software price free free free Platform price free free $$ Speed strong strong weak Efficiency strong strong weak 26 of 81 Publishing Dynamic Information Graham Butcher Security strong strong strong Platform strong strong weak (IIS only) win32 Source available yes yes no Exceptions no yes yes OOP weak strong strong Figure 14 - PHP - ASP.Net comparison table from ref [5] Hull’s conclusion is that PHP 5 is best overall but with the caveat that it’s not so good at Object Orientation (as .Net). 27 of 81 Publishing Dynamic Information Graham Butcher The following is a summary of the Author’s findings regarding some of the products and technologies discussed so far: PHP 4 PHP 5 ASP.Net ASP 3.0 CGI MySQL Access/ SQL Server Apche IIS CFusn Web Server CFML CCharge Studio Software price free free free free free free $$ free $$ $$$$ free $$ Platform price free free $$ $$ free free $$ free $$ $$$$ $$$$ $$ Speed strong strong weak strong strong strong strong strong strong strong strong strong Efficiency strong strong weak strong strong strong strong strong strong strong strong strong Security strong strong strong strong strong strong strong strong strong strong strong strong Platform strong strong weak (IIS only) Weak (IIS only) strong strong Weak (IIS only) strong weak strong strong Weak (PC only) Source available yes yes no no yes no no no no no no no Exceptns no yes yes no possible no no no no no no no OOP weak strong strong weak possible weak strong weak strong strong strong strong Figure 15 - Author's comparison table Of course some of these comments such as Apache Web Server’s capability of supporting Object Orientation are a bit meaningless. Some dedicated CGI programmers would swear that you can make CGI do what you want! Later in this report, there is a flow chart mapping a developer’s choices when choosing the elements to make up a working dynamic web database System. See figures 27, 31 and 33. 28 of 81 Publishing Dynamic Information Graham Butcher Chapter 3 : An overview of the three working examples produced during the project The following is a short description of the three solutions produced during the project. Later in the report, more details are given. In chronological order they are: 1. A complete Test Equipment Inventory and Calibration Record System (for Concert Global Networks). 2. An enhanced Lab Equipment Asset Register (for XGH1, UKIP). 3. An enhanced Document Management System (for XGH1, the Author’s Cost Centre). 3.1 Inventory and Calibration System – Concert In this System, details of Concert Test Equipment, Network Trials and Calibration Information were stored in a manner available to all 3 Companies involved – Concert and its parents BT and AT&T. The reason why the past tense is used here is that Concert is no longer. The technology used for this solution was: A commercially hosted external web site Apache, PHP and MySQL. MS FrontPage for buttons and navigation pages Access was controlled by session-based passwords and all usage of the System, logged in the database itself. The following screen shots are from the solution: Figure 16 - Page for selecting views of tables 29 of 81 Publishing Dynamic Information Graham Butcher Figure 17 - View of Assets Table Figure 18 - Login, obtain password page 30 of 81 Publishing Dynamic Information Graham Butcher 3.2 Enhanced Lab Equipment, Asset Register – UKIP This section gives an overview of the second working example, produced to add value to the Author’s working area. An existing Database Inventory System was made available to all in the UKIP test teams by producing an open web based query tool. Previously this data was only available to two or three people updating the data. The data is searchable by two different methods – “simple” or “power”. The technology used was: IIS, ASP and MS Access Code developed by CodeCharge IDE Figure 19 - Simple search page of UKIP Inventory 31 of 81 Publishing Dynamic Information Graham Butcher Figure 20 - Power search of UKIP Inventory Figure 19 – Display of single Asset showing all fields 32 of 81 Publishing Dynamic Information Graham Butcher 3.3 Enhanced Document Management System – XGH1 This section gives an overview of the third example produced to add value to the Author’s work area. This solution provides a multi-database Document Retrieval System allowing search and retrieval of any registered document within the Data and IP Integration Centre directly into the users’ browser. The technology used was: IIS, ASP and MS Access Multiple databases CodeCharge development IDE IE5 browser as document viewer and editor Figure 21 - Documentation System, database chooser 33 of 81 Publishing Dynamic Information Graham Butcher Figure 22 - Documentation System, search for "Peakflow DOS" Figure 23 - Show versions of "Peakflow DOS upgrade ..." 34 of 81 Publishing Dynamic Information Graham Butcher Figure 24 - Requested document retrieved to users web browser 35 of 81 Publishing Dynamic Information Graham Butcher Chapter 4 : Details of the first example, Concert Test Equipment & Calibration Details solution In the following three sections each solution is dealt with in more detail. The first one is the Concert Test Equipment and Calibration Details solution. The Concert Company, however, did not last for more than a few years and as a result the solution produced in this area didn’t get past the working prototype stage. The lessons learned were passed on to the second and third examples, discussed later. 4.1 Requirements The requirements for this example were as follows: The existing data, in the Access database, must be used. The System should be used through the Web using a Web browser. The data must be available to people from a variety of backgrounds and locations ie within the 3 companies involved – Concert, BT and ATT&T. The offices covered 3 continents. Database creator (owner) and administrator don't manage this database as their primary job function. Therefore the creation and management of the database should not take much time. Database owner is not a programmer or a database specialist. It should be possible to define and manage the database without programming, as much as possible. There isn’t a high demand for changing and updating the data. It is presently carried out by one person – the Author. Hopefully in the future, the System could be adopted by Concert’s own Apache based web sites. Home-workers should be able to use the System via a narrowband modem connection. End users might enjoy the following: The generated HTML or any other code (Java, JavaScript) should be portable and run on as many browsers as possible. Minimal requirement are second generation browsers with tables support. Data types common to the Web should be supported, such as text areas, radio buttons, URLs, email addresses... Speed and scalability. Users on the Web are expected to wait a second or two for a reply from a server. The System should respond within that time frame. Security. The System will not be used to handle financial transactions but may contain details of equipment and locations. 4.2 Background. Concert was a joint venture company formed from its two parents BT and AT&T. It lasted for 3-4 years before it was divided up and given back to the original founding companies. One of the main businesses of Concert was to supply and manage Global Bandwidth. The Author’s role in the company was to test, point to point “superhighways” carrying SDH and SONET traffic, sometimes supplied by outside companies such as Level 3, Quest or Global Crossing. To do this, test equipment had to be moved round the world and put in place at the two ends of the circuits. The tests of the circuits supplied by the third party companies were known as “trials” and had to be planned well in advance. The test equipment used had to calibrated regularly. 36 of 81 Publishing Dynamic Information Graham Butcher 4.3 Problem analysis The starting point of this solution was the existing NVT (Network Verification Testing) Concert Database: Figure 25 - Starting point, existing NVT database - MS access Figure 26 - existing database, view of Assets The main problems to be overcome in designing this information System were: 1. A choice of technologies (language and interface or database connection) had to be chosen 2. A suitable platform had to be found to host the service – available to a wide audience 3. The existing data had to be incorporated 4. Some new skills had to learned to implement the solution 37 of 81 Publishing Dynamic Information Graham Butcher The starting point for the solution was choosing the platform. It was decided that, instead of setting up a working platform incorporating the required 3 components – web server, database and language – a commercial hosting company would be used. After looking at several the best choice seemed to be: www.easily.co.uk Their publicity stated the following inclusive features (see figure25) next page. The cost was £15 per year for the service with additional charges for uploads and downloads – amounting to another £5. The next stage was to decide on the database. As there was an existing Access database holding current data it was decided to go with Access and incorporate that into the solution. The final decision was to choose the language. Chilisoft ASP was chosen because it was already included in the hosting package and ASP appeared to integrate well with Access. This is where the first discovery was made. Access databases consist of a single *.mdb file. By placing this single file on a UNIX/Linux Apache server the database became unusable. This is because the Microsoft file structure is incompatible with UNIX file Systems apart from mere storage. At this point the solution had to be re-designed. The solution, mark II, consisted of the same commercially hosted web server (Apache) with MySQL and PHP used instead of Access and ASP. The contents of the Access database had to be exported to text based CSV (comma separated variable) files – one for each table. This was done via Microsoft Excel as an intermediate stage. These text based files were then used to load the data into the MySQL database, supplied by the hosting company. 38 of 81 Publishing Dynamic Information Graham Butcher Figure 27 - Flowchart showing choices made for first solution 39 of 81 Publishing Dynamic Information Graham Butcher Product Included 100 MB Diskspace FTP Php 4 MySQL FrontPage 2000 Extensions Sun Chilisoft ASP Own CGI-bin Mod Perl Graphical Website Statistics Quick Use Templates Extra Diskspace Available HTML, Graphics & Javascript Flash Dreamweaver 3 Compatible HotMetal Pro Compatible Helpdesk Support Linux & Apache Based Free Software Downloads Free Links to Development Sites "Pay As You Go" Data Transfer Internet Connection via 100 MB line No Hidden Extra Costs Figure 28 – Features provided with Easily.co.uk Web hosting 40 of 81 Publishing Dynamic Information Graham Butcher 4.4 Functional analysis of web-access to DBs By hosting the database on the same server as the web server, connections were simplified. Additionally the PHP language is designed to connect to databases so the commands to make the connections are simpler – all source code used in the 3 solutions is available on request. The problem with using a remote and externally managed database during development is that the debug, edit, upload, re-test, cycle takes longer. 4.5 Practical advice Having a dummy MySQL database available locally on a development PC would have been useful. This would have allowed full control of the database and would have made requests for the live System easier to formulate. When importing data into MySQL a command Line Interface is used. These commands are best stored as text script files. By doing this a record is kept, re-use is possible and “typos” are easily detected and corrected. A knowledge of Oracle CLI (Command Line Interface) is useful as MySQL CLI is similar. 4.6 Heuristics When solving problems, just typing the question into a modern Internet Search Engine such as Google often brings up answers quicker than expected – quite often quicker than delving through the relevant text books. 4.7 High level development The High Level Development consisted, in this case, of deciding to host the Service on the Internet rather than the three local Intranets (BT, AT&T and Concert). Although this does have security issues for a live System, it does simplify the problem of not having to request proxy servers and simultaneous access to three Intranets. These days it is not that uncommon to use the public Internet for commercially sensitive information, albeit with the use of encryption, tunnelling and other technologies. 4.8 Scalability Scalability should not be a problem with this solution as disk space is expandable on request. PHP and MySQL are designed to take multiple requests. High request rates are handled a lot better on this platform than CGI for example. 4.9 Ease of maintenance With this solution the maintenance and administration of Apache, PHP and MySQL are handled externally. Although this appears to be a good thing, the downside is that there is obviously also a forfeiting of control. Internally the MySQL account still has to be managed so that data is meaningful. This can be time consuming. 4.10 Technologies v non-functional requirements This type of technical solution gives rather a bland type of presentation of the data. These days people’s expectations are of all the latest graphic displays and complex “soft” logic applied to search techniques such as “Google”. This solution does not meet these expectations. 4.11 Design and Development of Solution The strategy used to produce the solution was: 1. Port the data – from Access to MySQL 2. Re-use existing code – modified from a variety of sources - for displaying the data. 41 of 81 Publishing Dynamic Information Graham Butcher 3. Produce Front-end, user friendly pages to display the PHP code written – Microsoft Front Page was used. 4. Protect the data with password protected pages 5. Look for ways of managing the database – user management, disk space, table repairs etc These stages were all completed before the Company (Concert) was disbanded and development stopped. A couple of extra stages would have produced a more complete solution: 1. More complicated “Views” of the data could have been written taking into account the links in the original Tables. 2. An online update System could have been produced that used forms and matrices for updating the MySQL data. The method used in the prototype was to update the data by using the built-in MySQL Management facility. 3. A more compact and cosmetically attractive way of displaying the data could be found. This was achieved in the second and third working examples. 42 of 81 Publishing Dynamic Information Graham Butcher Chapter 5 : Details of second example, the UKIP Asset Register This section documents in detail the second working example produced - the enhanced Database Inventory System as used by different groups making up the UKIP Test Labs. This existing database System holds details of serial number, owner status location, purchase request number etc of all UKIP Assets – full list of fields supplied in figure 19. 5.1 Requirements The requirements for this example were as follows: The existing data, in the Access database, must be used. The new System should be used through the Web using a Web browser – no specialised software required. The new System should be read only – the database is administered by the infrastructure team The new System must not corrupt the data The new System must not force the infrastructure team to work in a different way. End users might enjoy the following: The generated HTML or any other code (ASP, Java, JavaScript) should be portable and run on as many browsers as possible. Minimal requirement are second generation browsers with tables support. The search syntax must be simple for novices but advanced users must be able to define complex search expressions. If the results are sortable by all columns, that would be an advantage Data types common to the Web should be supported, such as text areas, radio buttons, URLs, ordering by columns with a single click etc. Home-workers should be able to use the System via a narrowband modem connection. Currently the database size is 10MB and having to download it in its entirety is not good way to work. Database queries through a “thin” client web browser would be much more efficient. Speed and scalability. Users on the Web are expected to wait a second or two for a reply from a server. The System should respond within that time frame. Security. The System will not be used to handle financial transactions. It will hold information about equipment and locations and should be shown outside BT. 5.2 Background. The UKIP Test area of the Author’s Centre consists of several teams sharing a range of equipment used for simulating, testing and integrating new products and features. These will be used in different areas of the live BT IP Networks. The current Access database is held on a shared drive and is accessible by using the interface shown in figure 26 below. 5.3 Problem analysis The starting point for this solution was looking at the existing Access database. This database holds details of all Assets for the UKIP area of the Test Centre. At that point the database was only used by the infrastructure team. The interface to the database looked like: 43 of 81 Publishing Dynamic Information Graham Butcher Figure 29 – Original user interface to UKIP Inventory database This interface did not give away many clues about the internals of the database. The designer of the database was consulted to find out more details. The next task was to analyse in detail, the structure of the database by looking at the tables and their relations. MS Access itself provides a useful tool for doing this under Tools/relationships from the pull-down menus. The analysis showed: Figure 30 - Analysis of tables and relationships in UKIP Inventory database 44 of 81 Publishing Dynamic Information Graham Butcher This analysis revealed there were 7 tables in total and surprisingly, that there were no links or relationships in place between the tables. All the Asset data was in one table. This made the design of the solution considerably easier as the task could be reduced to displaying one table – the “1 Network Devices” table. The main remaining problems to be overcome were: 1. A choice of technologies/languages and interfaces/connections had to be chosen 2. A suitable platform had to be found to host the service – available to all in UKIP area of the Test Centre. 3. The existing data had to be incorporated. 4. Very importantly – the new System should not interfere with the mission critical, existing System managed by the Infrastructure Team. It should not even be seen to slow down or change the ways of working of the existing methods. 5. The new presentation of the data – the web pages – should be Read Only. New data should be added and modified using the existing Access interface which has full auditing built-in. 6. Some new skills had to learned to implement the solution At this point in time the CodeCharge Studio Development Environment and automatic code generation System had been “discovered” and it was decided to put it to use using an IIS Webserver with ASP 3.0 to interface to the existing Access database. 45 of 81 Publishing Dynamic Information Graham Butcher Figure 31 - Flowchart showing decisions made for solution 2 46 of 81 Publishing Dynamic Information Graham Butcher 5.4 Functional analysis of web-access to DBs Another decision that was made was that a System that used the live (single) database file directly was too risky. A batch job was to be setup that would copy the live database file to the chosen web server 4 times a day. This enabled the database manager to effectively see no difference to the way he had worked previously but ensured that the web pages were never more than a couple of hours out of date. 5.5 Practical advice Diagram to show how choices are made are shown in figure 31 above. 5.6 Heuristics When dealing with a new tool such as CodeCharge Studio it is well worth investing some time in finding how to use it properly. Fortunately the product came with several tutorial examples that were studied to take full advantage of the product’s features. These tutorials are available free of charge from the web site www.codecharge.com ref [4]. 5.7 High level development The high level development decisions made in this example were things such as: Realising the importance of the existing System and building round that System with a low risk enhancement. Choosing the interface. In contrast to the original interface, see figure 29, the web based interface is extremely simple. For the “simple” search a single text box is used to search for any record containing any supplied string – see figure 19. For example if just the letter “e” was entered the result would be a list of all records. If the word “Peakflow” is entered, 4 records are displayed all containing “Peakflow” in at least one field. The “power” search allows more specific information. In figure 20 all records that satisfy “building=b48, room=9, owner=james, status=in use” are displayed. Choosing a development environment and automatic code generation tool for the first time. 5.8 Scalability With a maximum of 40 or 50 people working in the UKIP area, the chances of numerous simultaneous queries occurring are low so using Access as a database should not be a problem. That the current size of the database file is 10 MB, having grown from 6 MB two years ago, means that there shouldn’t be a problem with disk space on the server. The System implemented should therefore be adequate for hosting present and future needs. 5.9 Ease of maintenance All database management and data content is handled by the infrastructure team. The IIS Webserver is also managed by another team. The only check that has to be made is that the files are being copied across to the web server as expected. If this fails, the database web pages will still work but the data won’t be completely up to date. To summarise, the designed System needs very little maintenance. 5.10 Design and Development of Solution The design and development of the solution followed the following stages: 1. Decide on the 3 key elements – database, platform and language. In this case the existing Access database was preserved, hosted on an IIS server with ASP 3.0 used as the scripting language. The reason for using ASP 3.0 was that the IIS web server was relatively old and there was a fear that ASP.Net might not be supported. 47 of 81 Publishing Dynamic Information Graham Butcher 2. Next, the development environment was set up on a single PC. The PC was running XP Professional so had a built in IIS web server that had to be enabled. Windows XP does not enable IIS by default. CodeCharge Studio was installed as the development environment. 3. A copy of the live database was used in the development environment. Next the stages of Project Development described in 2.3.3, were applied and the Application was built. The first version used just a simple search. This effectively created an SQL select command of the form: Select {field1},{field2},{field3}… FROM {Table name} WHERE {field1} LIKE '%{s_keyword}%' OR {field2} LIKE '%{s_keyword}%' OR {field3} LIKE '%{s_keyword}%' …. The actual command was: SELECT [Asset reg no], Description, [Part No], itemname, [Serial No], Building, Room, Rack, Owner, Comments, Status FROM [1 Network Devices] WHERE Description LIKE '%{s_keyword}%' OR itemname LIKE '%{s_keyword}%' OR Building LIKE '%{s_keyword}%' OR Room LIKE '%{s_keyword}%' OR Rack LIKE '%{s_keyword}%' OR Owner LIKE '%{s_keyword}%' OR Status LIKE '%{s_keyword}%' OR Comments LIKE '%{s_keyword}%' OR [Asset reg no] LIKE '%{s_keyword}%' OR [Serial No] LIKE '%{s_keyword}%' OR [Part No] LIKE '%{s_keyword}%' The application was then tested on a few key users and feedback obtained. This SQL command was generated automatically by CodeCharge Studio. In the early releases of the software however, the square brackets around field names like “Serial No” had to be added manually! 4. A second type of search, the “power search” was then added using SQL of the form: Select * FROM {Table name} WHERE {field1} LIKE '%{field1_entry_form}%' AND {field2} LIKE '%{field2_entry_form}%' AND {field3} LIKE '%{field3_entry_form}%' …. The actual command was SELECT * FROM [1 Network Devices] WHERE [Asset reg no] LIKE '%{s_Asset_reg_no}%' AND Description LIKE '%{s_Description}%' AND [Part No] LIKE '%{s_Part_No}%' AND itemname LIKE '%{s_itemname}%' AND [Serial No] LIKE '%{s_Serial_No}%' AND Building LIKE '%{s_Building}%' AND Room LIKE '%{s_Room}%' AND Rack LIKE '%{s_Rack}%' AND Owner LIKE '%{s_Owner}%' AND Comments LIKE '%{s_Comments}%' AND Status LIKE '%{s_Status}%' 48 of 81 Publishing Dynamic Information Graham Butcher This was also tested on a few key users before being released along with the links between the two pages holding the searches. 5. The final stage was to make the URL to find the web pages “more friendly”. This was done by adding a DNS entry on the BT intranet so that URL used was: http://xgh1.nat.bt.com/asset register The development time for this solution was significantly shorter than the time taken for the first, Concert example. The reasons are: More experience. Better Methodology. Use of Development and Automatic Code Generating Tools. 49 of 81 Publishing Dynamic Information Graham Butcher Chapter 6 : Details of third example, Centre’s Documentation Management System The third and final example was the enhancements made to the Centre Document Management System. The lessons learnt from first two example were applied and a working System was produced relatively quickly. The technology used for the second solution seemed to work well so was also adopted for the third. This means that the third has many similarities and only obvious differences are highlighted. To understand this section completely the reader should read the previous section – details of second solution. 6.1 Requirements The requirements for this example were as follows: The System should allow web based access to all the Centre’s registered documents 1. The System should remove the need for the existing dedicated software to view and download documents. The System should be read only, in that it should not be used for registering and storing documents The System should work with specified browsers. As the browser has the dual purpose of locating the documents and also displaying them, the specified browser will be MS IE5 or later. This is because MS Office is used for most documentation within the Centre. If a different type of browser is used it will be able to locate documents but not display them – unless they are TEXT, JPEG, HTML or other web supported formats. As some documents can have many Authors – writing different versions – there should be two types of searches available. One is a document oriented search and the second an Author oriented search. Search results should be “orderable” by clicking on column headings. 6.2 Background XGH1, the Author’s work Centre has had its own documentation System for a number of years. The System which is MS Access based was developed by Paul Wigens, a work colleague. There have been a number of releases of this dedicated software. The System is used for storing and retrieving documents and managing their change control. Within XGH1 there are a number of work areas, each has its own database of stored documents, the documents themselves being stored on various different file servers. Most people in XGH1 only have access to two or three areas, their own work area and the Centre’s common area. The common area holds general information like templates used by the Centre. This System is also seen as “mission critical” as, if for example, documents (or details about documents) were lost, it could have a catastrophic economic affect on the Centre’s finances. For this reason the Author again developed a “safe” System that would not risk the integrity of the existing data and files. Copies are made, 4 times daily, of all database files to the web server, with the live FileStore files left untouched. The new System is Read Only so does not risk any database corruption. 6.3 Problem analysis Again with this solution the deployment method had to be chosen. Because the second solution had worked so well with choices made it was decided to make the same choices for this one. Not everyone has access to all documents in all areas. The new System should allow the same people the same access that they would have had through the existing System. 1 50 of 81 Publishing Dynamic Information Graham Butcher The analysis of the database structure behind FileStore showed that the product consisted of two layers of databases. The first database held information about the areas of work covered and pointed to the other databases that actually held the information about the documents, Authors and locations where the documents were stored. The databases are known as “front end” and “back end.” An analysis of a single “back_end database” showed that there were 6 tables with the following relations. Figure 32 - Analysis of FileStore "back end" database The problem therefore resolved to producing two types of search on two different web pages – the Document oriented search and the Author oriented search. The next page documents, with the help of a flowchart, the platform choices made. 51 of 81 Publishing Dynamic Information Graham Butcher Figure 33 - Flowchart to show decisions made for solution 3 52 of 81 Publishing Dynamic Information Graham Butcher 6.4 Design and Development of Solution 1. Decide on the 3 key elements – database, platform and language. In this case the existing Access database was preserved, hosted on an IIS server with ASP 3.0 used as the scripting language. The reason for using ASP 3.0 was that the IIS web server was a relatively old and there was a fear that ASP.Net might not be supported. 2. Next, the development environment was set up on a single PC. The PC was running XP Professional so had a built in IIS web server that had to be enabled. Windows XP does not enable IIS by default. CodeCharge Studio was already installed as the development environment. 3. A copy of the live database was used in the development environment. Next the stages of Project Development described in 2.3.3, were applied and the Application was built. In this case there was more of a challenge in building the query to cover three tables. The CodeCharge Query builder was used for the document search page: Figure 34 - CodeCharge Query Builder, covering 3 tables The resulting SQL query was: SELECT TblDocuments.*, Type, TblDocTypes.DocCatID AS TblDocTypes_DocCatID, TblDocCat.DocCatID AS TblDocCat_DocCatID, DocCat, FileRef AS FileRef FROM TblDocuments INNER JOIN (TblDocTypes LEFT JOIN TblDocCat ON TblDocTypes.DocCatID = TblDocCat.DocCatID) ON TblDocuments.DocTypeID = TblDocTypes.DocTypeID WHERE Title LIKE '%{s_keyword}%' OR FileRef LIKE '%{s_keyword}%' OR Keywords LIKE '%{s_keyword}%' 53 of 81 Publishing Dynamic Information Graham Butcher OR Type LIKE '%{s_keyword}%' OR DocCat LIKE '%{s_keyword}%' This would have been extremely difficult to formulate manually. 4. Next a separate page was produced for making “Author searches”. 5. Then the version page was created. All versions of a selected document were found and displayed. 6. The next challenge was to return the location of any stored file directly into the web browser. This stretched the capabilities of CodeCharge Studio but was eventually achieved. This means that when the user clicks on the selected document version, the document is retrieved directly into the browser. Luckily IE 5 supports most document types so there was no need to call up any other programs like Word, Excel or Notepad to display the requested document. The only Microsoft document that couldn’t be displayed appears to be MS Project documents. 7. The final page to be produced was one that displayed the contents of the “front end” database. This was used to allow users to pick the database (or work area) they wanted. 8. The last stage was to produce a friendly URL to the application, the one used is: xgh1.nat.bt.com/filestore 54 of 81 Publishing Dynamic Information Graham Butcher Chapter 7 : Evaluation of Solutions To keep things simple, the three separate solutions are looked at separately. 7.1 Evaluation of solution 1 The main lesson learnt from this exercise was that Microsoft files, Access files specifically, are not useable on a UNIX platform. In addition to this the main difficulties encountered were: 1. Learning to write PHP to high enough to standard to allow enough customisation and features to be implemented. 2. Uploading code and pages from behind BT Firewalls. This was possible when the project started but increasing security standards prevented it latterly. 3. Exporting from Access and importing into MySQL is not a straightforward process. All loading scripts should be kept for future use and for modifications of any “typos”. The good point that came out of this solution was: 1. The use of a server that all components already loaded saved a lot of time. Installing and configuring Apache, MySQL, PHP and Chilisoft ASP would have been time consuming. Summary This solution never got past the working prototype stage mainly because the users it was designed to support disappeared – along with the company. Of the three solutions it was the only one built without development tools. This probably added to the development time as all the coding had to be done by hand. The use of a commercially managed server has good and bad points. The good are that it will most likely be in a stable working state with all standard configurations correct and in place. The bad side is that obviously, a certain amount of control is forfeited and certain restrictions are in place. 7.2 Evaluation of solution 2 The main lesson learnt from designing and implementing this solution was how useful the development environment and automatic code generator was. The solution produced, not only had a simple and pleasing interface but performed efficiently even over 56k links. This was all achieved by using supplied templates and style sheets combined with a query generator. Using a MS proprietary data connection, ADO – Active Data Objects, the web-database link was optimised. The difficulties encountered were: 1. Having to learn how to use a powerful but complicated tool – CodeCharge Studio 2. CodeCharge Studio, in its early days, having a few bugs and having to manually “tweak” the generated code to make it work. This helped to understand ASP coding however. The good points that came out of this solution were: 55 of 81 Publishing Dynamic Information Graham Butcher 1. An automatic code generator – CodeCharge Studio in this case – is good news for at least four reasons : a. A Knowledge of ASP (or other languages is not necessary) b. The code generated is well structured and easily maintainable. For example, all local variables such as paths and web locations are held in a single file (common.asp in this case). If the application is ported or moved to a different location it is easy to modify this single file. c. Language sensitive editors made editing the code easier. Reserved words are highlighted in different colours along with comments etc. d. Cascading Style sheets are built in so the resulting application has a consistent “look and feel”. Summary The resulting product produced by the tool – CodeCharge Studio – was significantly quicker in the making and had a more professional “look and feel” and interface to it. After having used the product it would be difficult to go back to producing by hand coding and building from first principles. This finding was put into practice with the third example as almost the same choices were made as far as database, language, server and development tool were concerned. As a result the third example was produced even quicker. 7.3 Evaluation of solution 3 Although solution 3 applies to a completely different area than 2, technically it is quite similar. A lot of the comments applied to 2 also apply to 3. One of the main achievements of this solution was to not only locate the document as a URL, but return the URL back to the browser so that the required document could be displayed (in the browser). One of the biggest challenges was to link several tables into a query that would display the required data. The CodeCharge Studio query builder certainly helped with this task. The solution works particularly well when used over a 56k dial up link. The previous, Access program based solution forces you to down load the whole FileStore (back end) database before it can supply any information. This can take over an hour to download over 56k, telephone link. The new web based version retrieves information in a matter of seconds and documents, depending on size, in a matter of moments. Another advantage of the System is that documents can now be “described” by their URL. The URL can be sent or distributed rather than the document itself. For example if a large document is to be distributed to a group of people, it can first be stored in FileStore and then the URL can be cut and pasted into an email to the distribution list. This has two advantages: The recipients don’t get their email Inboxes clogged up with large attachments If the version of the document changes or the document is updated, the URL is still valid and the latest version is found. Surprisingly this facility wasn’t picked up within the Centre. FileStore seemed to keep its perceived function as a repository for documents rather than a collaboration tool or an email “un-clogger”. The final advantage is that the web based “front end” allows users to switch between databases very easily. This is achieved by using the “database chooser” page. The old Access based System forced a download of a complete database to the users’ PCs before displaying any information. There are of course problems with retrieving documents directly in to ones browser as this technique is totally dependent on the state of the network. Sometimes the web browser would 56 of 81 Publishing Dynamic Information Graham Butcher time out before the document arrived because the local LAN was overloaded. Using the refresh button seems to help because the first half of the file transfer gets cached and refresh usually completes the transfer. Even cutting and pasting the URL into a new browser window has been known to work. As mentioned earlier MS Project files did not appear to be viewable in MS browsers even when the application – MS Project - was loaded on the recipients PC. The contents of Zip files were viewable but required the application to be installed. During the course of the development of this third solution the Author was asked to design a web based uploading System. The reasons for not doing this were fairly simple: It would take too long and extend the project beyond its recommended time-span and effort. It had the risk of not working properly, corrupting the databases and losing, mission critical information. The FileStore System was due to phased out in a matter of years anyway because of the universal adoption of the “bought in” solution called LiveLink. LiveLink is a commercial product “rolled out” to the whole of BT with a few additional features. Documents can be uploaded and downloaded through its web interface, permissions can be controlled more tightly, communities and group ownerships can be created and it covers the whole of the Company. More information at http://www.opentext.com/products/livelink/enterprise-server/ Difficulties and Good Points: These are much the same as for Solution 2 but without the bad points! The skills in using CodeCharge Studio had already been developed and the CodeCharge product is now stable, mature and robust. Summary The full potential of the enhancement to FileStore was probably not achieved. This could be due to a couple of reasons: The Author did not publicise the benefits well enough Users weren’t ready for a new way of working and didn’t have the skills to use the new interface. An example of a comment received was “you can’t browse certain areas or categories of documents in a hierarchical way that you could with the old software”. This is not in fact true as the “orderable” columns allow you to separate categories and types. It needs practice, though. One place the new software is used is in the test labs where testers are away from their desktop PCs and require documents quickly. Being able to retrieve these without the usual dedicated program is a definite advantage. There was initially some concern that the whole of BT would be able to see all documents in FileStore but the fileservers that hold the documents prevent this. All that people can retrieve is the location, version and a few other details. 57 of 81 Publishing Dynamic Information Graham Butcher Chapter 8 : Recommendations General recommendations are given before specific ones for each solution. 8.1 General Recommendations 1. Consider the Security of the solution first. This can affect the location of files and permissions and is harder to modify later in the development. See the section on security 2.5.4 Security. 2. Consider the risks involved with the data. Who should see it, who shouldn’t? When building a new System from scratch, what is the worst case scenario that can happen? The same applies to enhancing an existing System – but probably more so. Design accordingly. 8.2 Recommendations from Solution 1 The following recommendations are: 1. Choose a web server or ISP where applications such as PHP and MySQL are already installed. 2. Search the Web for existing code that can be adapted for your needs instead of writing everything from scratch. Get involved with communities that discuss and help each other overcome problems. 3. Don’t expect Microsoft Database files (eg *.mdb) to be readable (or “openable”) on a UNIX, Apache Server. 4. Use a development environment and software tools as much as possible – as in solutions 2 and 3. Future Improvements and Enhancements If this solution were to be continued the following improvements are recommended: 1. Writing a web page to enter data. As mentioned earlier, the current System uses the built-in Administration tool to add data. A web page (or pages) dedicated to updating the data would be a more efficient way of doing things. There are two ways, or levels of difficulty, of doing this: a. The first is fairly simple and straightforward – one table is updated at a time using a simple and separate web form for each table. b. The second type of web page would be one that takes account of links between tables. For example if the Assets table were being updated a pull-down list of locations, extracted from the locations table could be displayed. If a suitable location didn’t exist it could be added at that point without going to a form specific for adding locations. 2. A second enhancement is something that would have to be done before b) above. That is, establish links between the tables by writing the necessary SQL. This would allow deployment of the database as it was originally designed in its Access format ie with linked tables. The links were lost when the data was ported from Access to MySQL. 3. The third improvement or enhancement recommended would be to present the data in a more attractive and compact way. See Figure 17 - View of Assets Table for the current method of presentation. 8.3 Recommendations from Solution 2 The recommendation and lesson learnt from this solution is “use development tools” as much as possible. This has been mentioned several times in the report. 58 of 81 Publishing Dynamic Information Graham Butcher Future Improvements and Enhancements There are several recommendations for improving this second solution, should the project continue: 1. The first and most obvious one is again, writing the web pages to update the data. This is made a little more complicated with this particular database, as at present, all actions and changes are audited and all changes logged. The security of who was allowed to update and who wasn’t would have to be controlled by a password System. This is not too difficult to do with CodeCharge’s built in “Security Management/Role-based Access Control”. Additional hand crafted code would have to be written to be executed after the “update,” “add” or “delete” buttons were pressed. The Asset table, in this case would be relatively simple to update as it is a single table without any links to complicate things. If this enhancement were implemented, using a web page to update data however, would be significantly quicker than the present method which involves downloading the whole Asset Register database (10MB) to the updater’s PC before any changes can be made. 2. The second recommendation is that instead of having two copies of the database – a live one on the File-server, and a copy on the Web-server, (synchronised every 2 hours) - a single database would be used. This of course is a pre-requisite for recommendation 1 above. There would be two ways of doing this: a. Move the original database to the Web-server b. Make a network connection from the Web-server to the live database file on the File-server. Solution a) would be the easier and more reliable way, as the web-database connection would not have to rely on networks. 8.4 Recommendations from Solution 3 This particular solution was built quite quickly – in a matter of evenings – using the lessons and experience gained from the first two solutions. It is not recommended that the solution be extended and enhanced within the BT environment as a corporate solution, LiveLink, has been bought in and is now used extensively. The work produced during the project ie the enhancement to FileStore is still used however, as users are presently transferring important documents to the new System and have to look in different databases to locate them. Extending/Enhancing the solution If the solution were to be extended and upload pages were to be included, the following advice from DataObjx/CCS Developer Magazine ref [9] could be used. “There are two (2) ways for using the non-COM based file upload component that ships with CodeCharge Studio. One methods allows you to perform a script based upload and the other method allows you to perform an ADO.Stream based file upload. In both cases it appears that the component requires your server to have a temporary directory into which the file is initially loaded and a 'permanent' directory into which the file is subsequently moved if it passes validation. 59 of 81 Publishing Dynamic Information Graham Butcher However, there are times when the application will require that the Upload component must upload the file directly into a field in a database table. The reasons can range from the need for additional security or simply because the ISP that hosts your site is unwilling to allow sufficient permissions to the necessary directories. Furthermore, if you do have sufficient permissions on these directories, you may be concerned that the directories could be discovered and in one way or another compromised. As a result of this concern, you have probably adjusted your component to disallow the uploading of certain types of files, especially .exe, .com, .bat files and so forth. And you would be correct to have done so. Still, there will be times when a System requires a higher level of security or the ability to upload the executable type files. And thus make it harder for a hacker to compromise your web sites security measures. Uploading the file directly to the database and downloading the file directly from the database may by your only choice. Since CodeCharge Studio comes with sufficient information about their upload component, our application is going to use a different one. The component we're going to use is a free Upload component called PureASPUpload. This component is apparently capable of uploading files up to 2 GIG in size. Normally however, this limitation is no limitation at all. This component can be downloaded from http://www.ormacdigital.com/pureaspup load/help/default, htm They also sell another product called HUGE-ASP Upload that allows you to upload files larger then 2 GIG. Note: Downloading documents, images and other binary files from your database can cause a strain on both the database and your bandwidth. Unfortunately, reality often dictates things and we have no choice but to perform uploads directly to a table. Since this component allows files to be uploaded directly to a database field and is free, it's a likely choice to use until the CodeCharge Studio component provides similar functionality.” 60 of 81 Publishing Dynamic Information Graham Butcher Chapter 9 : Conclusions This section covers the conclusions from the 3 examples and also a few overall conclusions. 9.1 Conclusions from Solution 1 This type of solution using Apache, PHP and MySQL must be the cheapest. It also gives considerable scope for expansion and Scalability. The main thing that went against it was the overhead of having to learn PHP. PHP is not a difficult language but as, like most languages, it has a few “quirks” that need getting used to. After having developed the second and third solutions using the CodeCharge Studio development tool, this solution was re-visited with a view of improving the code and adding updating pages. It was found that a live development environment was not easy to setup with the remote server managed by a third party. As a result the CodeCharge Studio tool could not connect directly to the deployed database and test during development. There was no local copy of MySQL to work with so development by this method was abandoned. 9.2 Conclusions from Solution 2 Several unexpected findings came out of this solution: One reaction of to the solution was “what are the printing capabilities of the solution?” This demonstrated that people are not always ready for new ways of working. People don’t always want their data publicised. This might be for several reasons – it might not be as up-to-date as they would like, it may result in more empowerment to outside groups or they simply don’t like changes to the “status quo”. The main surprise from this solution was, that after some initial teething problems with the Development Tool, how easy it was to produce a simple working System that met the design requirements. 9.3 Conclusions from Solution 3 The main conclusions drawn from producing solution number three were: It would have been “nice” to have also produced pages for uploading documents instead of just the Read Only solution produced. The Author believes however, that the correct decision was made in not doing this, as the risks were too high. The solution is still in use but will have a limited life time as FileStore is being phased out to make way for LiveLink. This will happen in the next 4-6 months during which time the enhanced FileStore can be used for finding and “porting over” documents and files to the new System. 9.4 Overall Conclusions These can be divided into 3 separate areas: 9.4.1 Use of Total Development Environments A Total Development Environment type solution is definitely recommended. It appears that the more you spend the more functionality you get – in the way of built-in utilities. With the 3 Development Environments looked at – Visual Studio, ColdFusion and CodeCharge, CodeCharge the latter came out as the most useful, mainly on the basis of a cost to functionality ratio. The Author was fortunate in a way though, by buying the product in its early days when it 61 of 81 Publishing Dynamic Information Graham Butcher was less expensive. The down-side of that was that in the early days it had more bugs (or undocumented features) that had to be corrected and manually adjusted in the ASP Code before it would work as expected. Sometimes the generated code had to be manually “tweaked” to make it work eg putting brackets or quotes around complex SQL queries. These modifications would then be over-written with the next iteration of the automatically generated code. These types of problems now appear to have been fixed in the latest released version. 9.4.2 Reliability and Robustness of Solutions Produced The Author can honestly report that, since producing the three solutions no-one has found any ”bugs” or undocumented features in the code. This is not surprising with the first solution as it was not well publicised – only reaching the working prototype stage of development and the Concert Company reverting back to its parents. Several people have offered to try and “break” the second and third solutions but no-one has yet succeeded. This is testament to the professional level code produced by CodeCharge Studio. 9.4.3 Use of Skills Developed during the Project for the benefit of the Test Centre, XGH1 The Author now feels confident that he could produce bespoke Systems fairly quickly to help the flow of information within his Centre, XGH1. This is only half of the story though, as one of the objectives of this project is to spread the knowledge and experience gained, to other people in the Centre and BT in general. This could be achieved by distribution of the report to people interested in furthering the same goal – of improving information flow. It is reassuring that since starting the project over 2 years ago, BT has adopted several such Company wide Systems – the Niku Enterprise System, managing Projects, Finance and Workloads and also the FileStore Document Management System. 62 of 81 Publishing Dynamic Information Graham Butcher Chapter 10 : Acknowledgements Special thanks are given to my two supervisors, Stefan Poslad, Queen Mary College and Alan Wheeler, BT Exact. Also thanks to Paul Wigens and Bob Reason for their help and encouragement with FileStore development. Thanks to John Williams, Alex Workman and Mike Carter with the document review process. Chapter 11 : References 1 Dr Phill Edwards Apache Web Course Learning Tree - Course 541 2003 2 Kwok-Bun Yue and Wei Ding Design and Evolution of An Undergraduate Course on Web Application Development ITICSE, Leeds, UK June 2830, 2004 3 Not Applicable www.macromedia.com Not Applicable Not Applicable 4 Not Applicable www.codecharge.com Not Applicable Not Applicable 5 Sean Hull PHP and ASP.Net go head to head www.oracle.com/technology/pu b/articles/hull_asp.html August 2004 6 Prof Ziga Turk WODA - A slim Web Oriented Database. Internet Database Conference, Hong Kong http://www.zturk.com/data/wor ks/robots/6dae.htm July 1999 7 Not Applicable www.ASP.Net Not Applicable Not Applicable 8 Janet Valade PHP and MySQL for Dummies Hungry Minds Inc, ISBN: 07645-1650-7 2002 9 Author Unknown Building a Document Management System CCS Developer, Issue 1, Page 12. August 2004 10 Bill Hatfield Active Server Pages for Dummies 2nd Edition Hungry Minds Inc, ISBN: 07645-0603-X 1999 11 Nitin Pandey, Yesh Singhal and Mridula Parihar Visual Studio.Net All-in-One Reference for Dummies Hungry Minds Inc, ISBN: 07645-1626-4 2002 12 Ben Forta and Nate Weiss ColdFusion MX, Web Application Construction Kit, Fifth Edition MacroMedia Press, ISBN 0321-12516-9 2003 63 of 81 Publishing Dynamic Information Graham Butcher Chapter 12 : Appendix A – Source Code for all Solutions All the source code for the 3 solutions is available from the Author on request. Chapter 13 : Appendix B – Glossary ADO Active Data Object - MS proprietary term Apache A Freeware Web Server API Application Programming Interface (or Language) ASP Active Server Page (Language) - not to be confused with Application Server Platform CFML ColdFusion Markup Language CGI Common Gateway Interface ColdFusion Macromedia's Server, Database Development Environment and Language CSV Comma separated variables (files) DB Database GNU "Not Unix" GPL General Public Licence HTML Hypertext Mark up Language IDE Integrated Development Environment IIS Internet Information System - Microsoft's Web Server Java A compiled language, similar to C++ - originates from Sun Computers JavaScript An interpreted, scripting language, similar to Java - originates from Sun Computers - runs within user's browser - is browser dependent JDBC Java API for executing SQL statements - trademark not acronym JScript Microsoft's version of JavaScript LAN Local Area Network Linux Linus Torvald's UNIX - flavour of Unix MS Microsoft NCSA National Centre for Supercomputing Applications NVT Network Verification Testing PERL Practical Extraction and Reporting Language PHP Pre - Hypertext Processing language RAD Rapid Application Development RDBMS Relational Database Management System SDH Structured Digital Hierarchy - a transport mechanism 64 of 81 Publishing Dynamic Information Graham Butcher Solaris The name of Sun Computer's Operating System SQL Structured Query Language UKIP The UK IP part of BT WODA Web Oriented Database (System – Prof Ziga Turk, see ref [6]) WOSA Microsoft’s Windows Open Systems Architecture XGH1 The Author's Cost Centre within BT Exact - like a Department XML eXtensible Mark up Language 65 of 81 Publishing Dynamic Information Graham Butcher Chapter 14: Appendix C – CodeCharge Product Feature Details Most of the following was obtained from [4] www.codecharge.com Application Builder The Application Builder is perhaps the most advanced web development automation tool available to web developers. With just a few clicks, users can create complex database-enabled Web applications with dozens-or even hundreds-of web pages for data management and administration. For example, a user with a database of employees, projects and tasks, can use the Application Builder to automatically create a Task List page, Employee Search page, Employee Maintenance page, Project Maintenance page, and complete an entire application project in a few short minutes. Integrated Development Environment (IDE) CodeCharge Studio combines visual interface design tools with an integrated development environment (IDE) to enable developers to create powerful and sophisticated web applications in record time. CodeCharge Studio features powerful HTML and code editors, an HTML design component, project explorer, property browser, dockable tool windows, and many other features desired by professional developers. The built-in HTML editor lets users quickly design their program's user interface and add database-connected grids, record maintenance forms, text boxes, list boxes, labels, buttons, etc. CodeCharge Studio gives users full access to the generated programming code through a full-featured, customizable code editor with syntax highlighting and smart indenting to help distinguish different source code elements. Developers can modify any part of the generated code without losing their customizations during subsequent code generations. This translates to having complete control of the generated programs and eliminates the need to use external code editors. The code editor also allows users to attach code to objects on the web page, for example, to send a confirmation email when some information is submitted via the web. Form and Component Builders Component Builders are key constituents of CodeCharge Studio and help users assemble Web pages from multiple components such as Grid, Editable Grid, Record, Directory, Login, or Search. A Builder is a dynamic wizard that creates the appropriate component on the page by allowing the developer to specify the database table and fields that are used to display, validate or accept information. 66 of 81 Publishing Dynamic Information Graham Butcher Visual Query Builder Developers can easily select tables and stored procedures or custom SQL as the data source for their data-aware forms. A Visual Query Builder is provided for visual creation of SQL statements. For example, a user can create a query that displays the list of employees, along with corresponding departments and roles. Developers can also specify filtering criteria, for example, to display only the employees with names that match search criteria entered in another page or form. 67 of 81 Publishing Dynamic Information Graham Butcher Security Management/Role-based Access Control CodeCharge Studio goes the extra mile in protecting generated Web applications by implementing additional security features that prevent users from externally modifying forms and submitting invalid data to the server. CodeCharge Studio allows developers to define secure role-based access to each page or form. Web users who don't have sufficient privileges to access a page will be automatically redirected to the login page. The users who are authorized to access a page but are not permitted to view or update a particular form will see the page without the form or will not see the Insert/Update/Delete buttons needed to update the form. Support for Multiple Programming Languages CodeCharge Studio can generate programming code in C# and VB. Net (ASP.NET), VBScript (ASP), ColdFusion, Java (Servlets or JSP), PHP, and PERL. Since the project model is stored in XML format, the programming language can be changed at any time and the same project can be regenerated in multiple programming languages. Extensibility CodeCharge Studio has an open architecture and is fully extensible. Most of the functionality is implemented as HTML, JavaScript, XML, and XSL. Users can customize almost any part of the built-in functionality or develop their own components, such as Builders, Components, Actions, and Themes. In the future, users will be able to obtain the CodeCharge SDK and develop their own code generation templates. Multiple Database Connections CodeCharge Studio supports multiple database connections to allow Web applications to utilize multiple data sources, for example, to validate user access against an external user database. Microsoft FrontPage Integration CodeCharge Studio can be installed as an Add-In for Microsoft FrontPage. The Add-In extends Microsoft FrontPage using a set of code-generating Builders, a project explorer, properties browser and code editor. FrontPage users can use all of CodeCharge Studio's features directly within their familiar environment. 68 of 81 Publishing Dynamic Information Graham Butcher Integration with Other Products Web pages created with CodeCharge Studio can be opened with any web editor, either by opening the HTML files saved by CodeCharge Studio on the disk, or by right-clicking on a page within the CodeCharge Studio IDE and selecting Open Externally. After a page is modified and saved, it is automatically available back within CodeCharge Studio. For example, users can create a simple web page with CodeCharge Studio, then open it in Macromedia DreamWeaver and add sophisticated graphics and layout features. Users can also copy and paste HTML code from any existing web page into a CodeCharge Studio web page. Extensive Database Support CodeCharge Studio supports most existing databases via JET and ODBC to establish the design-time connection. The server-side connection can be made using JET, ODBC, JDBC, ADO, DBI and PHPLib to connect to databases during run-time. For example, the connection from a JSP project to an Oracle database can be established using the Oracle JDBC driver. CodeCharge Studio goes as far as implementing bug fixes to PHPLib library for reliable use of PHP with Oracle. 69 of 81 Publishing Dynamic Information Graham Butcher Application Templates (Solutions) CodeCharge Studio comes with ten templates and examples of pre-built web applications that can serve as starting points for your applications. These applications are ready for customization and adaptation for organizational needs. The templates include intranet applications such as Bug Tracking and ask Management Systems as well as an Employee Directory. Internet application examples include an Online Store, Community Portal, Classifieds, Yellow Pages, Discussions Forum, and an Event Management System. Custom Code and Actions Applications generated with CodeCharge Studio can be easily extended by adding custom code or by using any of the predefined Actions. Actions are user-definable code components that are inserted into events. Many actions are provided with the product, while additional actions can be created by users and shared with others. Internally, actions consist of XML and XSL code that can be easily customized. 70 of 81 Publishing Dynamic Information Graham Butcher Unlike manually edited code that cannot always be recovered without the use of a versioning System or a backup, actions can be regenerated at any time. Code Generation Engine CodeCharge Studio users create web applications by connecting to a database and placing data-aware forms and components on a page. During the page design process, the representation of the project and all of its pages is saved in XML format in addition to HTML code that contains the page design. During project publishing, the HTML files are copied to the server while the XML files are used to generate server code using XSL code templates. This flexible architecture allows users to regenerate their web application in any programming language at any time. For example, users can generate a C# application then regenerate it in Java. More advanced users can even create their own XSL processing templates that automatically generate the documentation, UML diagrams, or test scripts for their applications. On the server, the generated code is fully separated from the HTML. During execution, the code files load corresponding HTML files into memory, replace static content with database values and output the final HTML to the browser. This method of separating the code from the HTML design content allows designers to make future changes to the site without having to modify the code. 71 of 81 Publishing Dynamic Information Graham Butcher Sample Databases Several sample databases and applications are provided with the distribution to help you get started with CodeCharge Studio. Chapter 15: Appendix D – ColdFusion Features 72 of 81 Publishing Dynamic Information Graham Butcher < End of document > 73 of 81