Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
T H E TO L LY G R O U P No. 200230 January 2001 NetScreen Technologies, Inc. Test Summary NetScreen-5 versus SonicWALL, Inc. SOHO/50 and WatchGuard Technologies, Inc. SOHO Competitive Evaluation of SOHO Internet Security Devices N etScreen Technologies, Inc. commissioned The Tolly Group to evaluate the performance of its NetScreen-5 v. 2.00r3, an Internet security appliance integrating firewall and virtual private networking (VPN) in a SOHO environment. NetScreen requested that The Tolly Group evaluate the NetScreen-5 along with the following Internet appliances: a SonicWALL, Inc. SOHO/50 v. 5.0.2 and a WatchGuard Technologies, Inc. SOHO versions 1.5.8 and 2.1.3.1 In all IPSec gateway tests traffic was forwarded within a single IPSec Security Association (SA) and was encrypted using DES-3 (triple Data Encryption Standard) and SHA-1 authentication Encapsulating Security Payload (ESP) encryption scheme with a pre-shared secret key with the exception of the SonicWALL device that used MD-5 authentication. The Tolly Group conducted tests of devices as IPSec tunnels for application and zeroloss throughput. Tolly engineers also tested 1 Tolly engineers tested two WatchGuard Technologies, Inc. SOHO devices because one of the devices became nonfunctional after the first set of tests and engineers continued testing with a new device in an upgraded version. © 2001 The Tolly Group Test Highlights m Forwards 5.6 Mbit/s of FTP throughput in an IPSec tunnel as compared to 0.1 Mbit/s from SonicWALL SOHO/50 and 0.9 Mbit/s from WatchGuard SOHO m Delivers 3.9 Mbit/s of SAP R/3 traffic in an IPSec tunnel as compared to 0.1 Mbit/s from SonicWALL SOHO/50 and 0.7 Mbit/s from WatchGuard SOHO m Sends zero-loss throughput across an IPSec tunnel at 30% of the theoretical maximum in tests of 512-byte packets and 45% in tests of 1,024-byte packets and 30% in tests using 1,518-byte packets m Demonstrates 50% greater packet throughput than its competitors in firewall tests of 512-byte packets, 40% more in tests of 1,024-byte packets and 10% more when using 1,518-byte packets Application Throughput Across an IPSec (DES-3) Tunnel: Bidirectional Chariot Traffic, 10 Mbit/s Half-duplex Fast Ethernet (IP Packets) 8.8 10 Average throughput Average throughput (Mbit/s) (Mbit/s) Premise: Managers of small office/home offices (SOHOs) updating their network security infrastructure with access media beyond dial-up to xDSL routers and cable modems need to verify the performance of Internet security appliances integrated with firewall and point-to-point VPN features. IT managers accustomed to wire-speed network LAN infrastructures need to ensure that network performance will not degrade when implementing a device providing both security and encryption of sensitive information within their own corporate sites, to branch offices, and to telecommuters. 8 6 6.9 5.6 3.9 4 0.9 2 0.1 0.1 0.7 0 NetScreen-5 SonicWALL WatchGuard Baseline SOHO/50 SOHO (No tunnel) Application type: FTP SAP R/3 * The stations, switches and devices were all set for half duplex and the test bed was configured so that each side of the tunnel transmitted traffic. Source: The Tolly Group, January 2001 Figure 1 Page 1 NetScreen Technologies, Inc. Test results show that out of all the devices tested in IPSec tunnel configurations using application data, the NetScreen-5 devices demonstrated the highest throughput. Furthermore, the NetScreen-5 devices also demonstrated the highest throughput in IPSec tunnel zero-loss throughput tests. In firewall tests, the NetScreen-5 demonstrated throughput equal to, or significantly better than, both of the other devices. Results IPSec Tunnel Application Throughput Tolly engineers configured a pair of NetScreen-5 gateway devices to create an IPSec tunnel. Results show that the NetScreen-5 was capable of a higher throughput rate than the SonicWALL SOHO/50 and the WatchGuard SOHO. These systems were tested in the same configuration. In tests using FTP traffic, the NetScreen-5 IPSec tunnel throughput was 5.6 Mbit/s while the SonicWALL SOHO/50 averaged 0.1 Mbit/s and the WatchGuard SOHO showed an average of 0.9 Mbit/s. The NetScreen-5 tunnel also showed that in tests of SAP R/3 traffic, it performed better than any of its competitors at 3.9 Mbit/s. The SonicWALL SOHO/50 and the WatchGuard SOHO results showed 0.1 Mbit/s and 0.7 Mbit/s, respectively. See figure 1. IPSec Tunnel Zero-loss Throughput Using the same IPSec tunnel configuration, engineers tested for zero-loss throughput and results show that the NetScreen-5 IPSec tunnel demonstrated the same or a higher percentage of the theoretical maximum than all other devices under test in the same configuration using 64- through 1,518-byte packets. In tests of 64-byte © 2001 The Tolly Group NetScreen-5 Zero-loss Throughput Across an IPSec (DES-3) Tunnel: Bidirectional SmartBits, 10 Mbit/s/Half-duplex Ethernet (UDP Packets) % of Theoretical % of theoretical maximum Maximum each device as a firewall and measured the zero-loss throughput when using UDP packets. For zero-loss performance tests, The Tolly Group measured steady-state throughput at 0.001%, the same metric The Tolly Group uses to test Layer 2 and Layer 3 networking devices. Testing was performed July through November 2000. 100% 80% 45% 60% 30% 30% 40% 5% 5% 5% 64-byte 512-byte 1,024-byte 20% 0% 0% 1,518-byte Packet sizes NetScreen-5 SonicWALL SOHO/50 WatchGuard SOHO Note: All percentages at 5% indicate minimum value tested, but may not indicate zero loss. SonicWALL SOHO/50 tunnel results show 99.99% loss for 64-byte packets, 99.24% loss for 512-byte packets, and 99.85% loss for 1,024-byte packets. The WatchGuard SOHO tunnel results show 99.99% loss for 64-byte packets, 6.69% loss for 512-byte packets, and 0.005% loss for 1,024-byte packets. Both the SonicWALL SOHO/50 and WatchGuard SOHO were unable to forward 1,518-byte packets. Source: The Tolly Group, January 2001 Figure 2 Zero-loss Throughput Across a "Single Rule" Firewall: Bidirectional SmartBits Traffic, 10 Mbit/s Half-duplex Ethernet (UDP Packets) 100% % of theoretical maximum The Tolly Group 85% 85% 75% 80% 75% 60% 60% 60% 45% 40% 20% 25% 10% 5% 0% 64-byte 512-byte 1,024-byte 1,518-byte Packet sizes NetScreen-5 SonicWALL SOHO/50 Source: The Tolly Group, January 2001 packets, all of the systems under test forwarded 5% of the offered traffic load. In tests of 512-byte packets, the NetScreen-5 IPSec tunnel forwarded WatchGuard SOHO Figure 3 30% of the theoretical maximum while the SonicWALL SOHO/50 and WatchGuard SOHO devices forwarded 5% of the theoretical maximum. Page 2 The Tolly Group NetScreen Technologies, Inc. Results for both competitive devices did not change in tests of 1,024-byte packets but the NetScreen-5 IPSec tunnel forwarded 45% of the offered load. Finally, the SonicWALL SOHO/50 and the WatchGuard SOHO were unable to forward 1,518-byte packets. The NetScreen-5 gateway forwarded 30% of the offered load. See figure 2. All percentages at 5% indicate maximum value tested but may not indicate zero loss. The WatchGuard SOHO when offered 5% (i.e., 500 Kbit/s) of the theoretical maximum load for 64-byte packets on 10 Mbit/s Ethernet discards 99.95% of the packets; for 512-byte packets, it discards 8.099% packets; and for 1,024-byte packets, it discards 0.005% of the packets. The WatchGuard SOHO was unable to pass 1,518-byte packets through the VPN tunnel because it must fragment the packets.2 The SonicWALL SOHO/50 when offered 5% of the theoretical maximum load for 64-byte packets on 10 Mbit/s Ethernet discards 99.988% of the packets; for tests using 512-byte packets, the device discards 99.924% of the packets, and it discards 99.852% of the packets when using 1,024-byte packets. The SonicWALL SOHO/50 was unable to pass 1,518-byte packets through its IPSec tunnel.3 Firewall Zero-loss Throughput The Tolly Group engineers configured the NetScreen-5 to serve as a firewall with single-rule processing applied to both inbound and outbound UDP traffic in a half-duplex Ethernet environment. Test results show that when transmitting 64-, 512-, 1,024and 1,518-byte packets, the NetScreen-5 forwarded a higher percentage of the offered load than the other devices under test. In tests of 64-byte packets, the NetScreen-5 system forwarded 10% of the offered load while the SonicWALL SOHO/50 and the WatchGuard SOHO forwarded 5% of the offered load and still did not attain zero-loss. In tests of 512-byte packets, the NetScreen-5 forwarded 75% of the offered load. This was far more than its competitors who each forwarded 25%. When engineers used 1,024-byte packets, the NetScreen-5 firewall forwarded 85% of the offered load. The SonicWALL SOHO/50 forwarded 60% of the offered load and the WatchGuard SOHO forwarded 45%. Finally, in tests using 1,518-byte packets, the NetScreen-5 firewall forwarded 85% of the offered load. The SonicWALL SOHO/50 and the WatchGuard SOHO forwarded 75% and 60%, respectively, of the offered load. See figure 3. Analysis Baseline tests of the network demonstrate that without IPSec tunnel or firewall appliances, application throughput of bidirectional IP packets at 10 Mbit/s halfduplex demonstrated a baseline of 6.9 Mbit/s for FTP traffic and 8.8 Mbit/s for SAP R/3 traffic. The baseline cannot reach the theoretical maximum because traffic was forwarded in both directions and packets that collided were discarded. When IPSec and firewall appliances are added to a network for security purposes, the encryption and decryption functions take processing power and time. The encapsulation of packets steals bandwidth. In fact, more data than is reported in these tests actually went through the appliances under test and across the network, but The Tolly Group only counted unencrypted traffic on both the ingress and egress ports. The NetScreen-5 achieves speeds almost comparable to Ethernet (some DSL and cable speed). This is very important because a home office or branch office that begins to get accustomed to the higher speeds of DSL and cable can still have those speeds with an IPSec tunnel. Furthermore, customers know that they have encryption and authentication at a rate equivalent to high speed DSL access without having 2 The Tolly Group contacted WatchGuard Technologies, Inc. and asked why the IPSec tunnel of their devices had to fragment 1,518-byte packets but WatchGuard representatives did not respond. 3 SonicWALL says it has a non-shipping firmware to allow for fragmentation, but it was not publicly available at the time of testing. © 2001 The Tolly Group NetScreen-5 NetScreen Technologies, Inc. NetScreen-5 Competitive Evaluation NetScreen Technologies, Inc. NetScreen-5 Product Specifications* Performance • 1000 concurrent sessions • 960 new sessions/second • 10 Mbit/s firewall performance** • 10 Mbit/s 3DES performance** • 100 policies • 256 schedules Mode of operation • Transparent mode • NAT (Network Address Translation) • PAT (Port Address Translation) IP address assignment • Static • DHCP client • PPPoE client • Internal DHCP server VPN • 56-bit DES (IPSec) • 168-bit Triple DES (IPSec) • SHA-1 • MD5 • X.509 digital certificates ο Verisign ο Entrust ο Microsoft User authentication • Built-in (internal) database (100 user limit) • RADIUS (external) database Traffic management • Guaranteed bandwidth • Maximum bandwidth • Eight bandwidth priority levels Logging/monitoring • Syslog • WebTrends • SNMP • E-mail (2 addresses) • Traceroute • VPN tunnel monitor • Websense URL filtering **Performance achieved with 400-byte and larger UDP packets For more information contact: NetScreen Technologies, Inc. 2860 San Tomas Expressway Santa Clara, CA 95051 (408) 330-7800 (408) 330-7850 URL: http://www.netscreen.com *Vendor-supplied information not verified by The Tolly Group Page 3 The Tolly Group NetScreen Technologies, Inc. to worry about unencrypted information passing through the Internet. Also, authentication provides additional security because the network managers know exactly who is allowed to access the network. When testing with 1,518-byte packets, the SonicWALL SOHO/50 and WatchGuard SOHO reply to the device that is sending the packets to put an "allow fragmentation" in the packet so that the system under test can forward the packet across the tunnel. The Chariot endpoints allow this when sending 1,518-byte packets by putting an "allow fragmentation" bit in the packet. SmartBits does not respond to the system under test and cannot "allow fragmentation." Therefore, the SonicWALL SOHO/50 and WatchGuard SOHO are unable to pass 1,518-byte packets because the test system (SMB-200) will not adjust itself as a workstation might (adjusting its TCP/IP stack). Due to this, the SonicWALL SOHO/50 and WatchGuard SOHO cannot fragment packets. In IPSec tunnel (DES-3, SHA-1) performance tests, the NetScreen-5 delivered approximately half the packet-per-second (pps) performance than it did in a firewall configuration, and loses only 1 Mbit/s of throughput for batch traffic and 2 Mbit/s of throughput for interactive traffic. The NetScreen-5's performance is better than all others with all types of traffic tested. Competitors show a dramatic performance decrease in tests of 64byte packets. At only 5% of the theoretical maximum the results show a 99% loss and with 1,518-byte packets, traffic could not be forwarded at all. Effective data throughput is reduced to below 1 Mbit/s for the WatchGuard SOHO, and effective data throughput is approximately 0.1 Mbit/s for the SonicWALL SOHO/50. Although the NetScreen-5 does not perform at wire speed, the performance does exceed the available bandwidth of a T1 WAN as both a firewall and IPSec tunnel. Given the many variations of xDSL, the NetScreen-5 as a firewall can perform at the high speed of ADSL © 2001 The Tolly Group NetScreen-5 (Asymmetric DSL) at 6.1 Mbit/s downstream. The NetScreen-5 can provide both security and secure communications for remote offices connected at high WAN speeds and for internal departments at Ethernet LAN speeds. Also, if given a specific rule that would apply to a traffic type, the NetScreen-5 is capable of forwarding either IP or UDP packets at the same rate regardless of the protocol, whereas the competitors tested could not. Internet security appliances, both model WG2500. WatchGuard SOHOs version 2.1.3 were used for IPSec tunnel tests and version 1.5.8 was used for firewall tests. The NetScreen-5 as a firewall exceeds its competitors’ raw IP pps zero-loss throughput rates, especially for 512-, 1,024-, and 1,518-byte packet sizes; and it has a zero-loss rate using 64byte packets, whereas the competitors cannot sustain zero-packet loss at 5% of the theoretical maximum. The NetScreen-5's batch traffic throughput (6.13 Mbit/s) only shows a slight increase in effective throughput over the SonicWALL SOHO/50 (6.335 Mbit/s) and a slight decrease to the WatchGuard SOHO (6.620 Mbit/s); and interactive business traffic throughput (6.045 Mbit/s) for smaller packet sizes and loss for pps correlates to the better performance of the NetScreen-5. In order to test all of the Internet security appliances in an IPSec tunnel configuration, engineers tested a pair of the Internet security devices under test. One of the two Internet security devices under test was connected to a 3Com SuperStack II 3300 24-port Ethernet Switch version 2.60 P/N 3C16980 and the other was connected to a 3Com CoreBuilder 3500 Layer 3 Ethernet Switch version 2.10 P/N 3C35100. In between each tunnel under test was an Acterna DominoPlus DA-360 hardware-based network analyzer running DominoCore software version 2.6 and hardware version BN 9316/04 with DominoFastEthernet line interface 2.6 configured for 100Base-TX pass thru full duplex. Two identical DominoPlus DA-360 network analyzers were configured in-line between each Internet security appliance and each 3Com switch. When comparing UDP pps zero-loss throughput, the NetScreen-5 values remain the same for all tested packet sizes. The SonicWALL SOHO/50 and WatchGuard SOHO forward UDP packets at a much higher rate than IP packets and can sustain zero-loss for almost all packet sizes. Test Configuration and Methodology Devices Under Test NetScreen Technologies, Inc. used two NetScreen-5 Internet security appliances, both software version 2.00r3 and hardware version 2010. The purpose-built systems each have two 10Base-T Ethernet ports and weigh one pound. The NetScreen-5s have the following dimensions: 5 x 6.2 x 2.1 inches and have a built-in Web server that can be managed from any Web browser (password protected). Engineers also used a pair of purposebuilt WatchGuard SOHOs from WatchGuard Technologies, Inc. Finally, engineers tested a pair of SonicWALL, Inc. SonicWALL SOHO/50 Internet security appliances both model W50 version 5.0.2. IPSec Tunnel Test Bed Configuration Each 3Com switch also connected to a Spirent Communications SmartBits SMB-200 Advanced Multiport Performance Tester/Analyzer/ Simulator, a four-port network traffic simulator firmware version 6.63 00004 equipped with two ML-7710 10/100 Mbit/s Ethernet interfaces. A 200-MHz Intel Pentium IBM clone with 32 Mbytes of RAM, a PCI bus card and 2.0 Gbytes of fixed-disk space served as the Chariot console, the DominoPlus console and ran Spirent SmartWindows 6.53. This PC ran Microsoft Windows NT Workstation 4.0 SP5 and ran Chariot 3.2 and Domino NAS 1.0. The console was equipped with a Compaq Netelligent 10/100 Mbit/s Ethernet PCI bus card. Page 4 The Tolly Group NetScreen Technologies, Inc. NetScreen-5 IPSec Tunnel Test Bed Spirent Communications SmartBits SMB-200 Chariot Endpoint Chariot Endpoint 3Com SuperStack II 3300 NetIQ Chariot/ Spirent SmartWindows/ Acterna Domino Chariot NAS Console Endpoint System Under Test System Under Test Acterna DominoPlus Acterna Acterna Internetworking Analyzer DominoPlus DominoPlus DA-360 Internetworking Internetworking Analyzer DA-360 Analyzer DA-360 Chariot Endpoint Source: The Tolly Group, January 2001 The following devices ran Chariot Endpoint 3.5 software: a K6/400-MHz Advanced Micro Devices, Inc. IBM clone with 64 Mbytes of RAM with a PCI bus card and 6.0 Gbytes of fixeddisk space, equipped with a 10/100 Mbit/s Compaq Computer Corp. Netelligent PCI adapter with a NetFlex-3 v. 4.25m SP4 driver; a K6/400-MHz Advanced Micro Devices, Inc. IBM clone with 64 Mbytes of RAM and a fixed-disk space of 6.0 Gbytes, equipped with a 3Com 3C905C-Tx 10/100 Mbit/s Ethernet PCI-bus card with driver version EL90xBC4.sys 1.60.00.0000; a 200-MHz Intel Pentium IBM clone with 32 Mbytes of RAM and 2.0 Gbytes of fixed-disk space, equipped with an Intel Corp. PRO/100+ Server 10/100 Mbit/s PCI-bus card with driver version 4.02.25.0000; and a 200MHz Intel Pentium IBM clone with 64 Mbytes of RAM and a fixed-disk space of 2.0 Gbytes, equipped with an IBM Netfinity 10/100 Mbit/s Ethernet PCI-bus card with driver version 3.37.14.0002. All clients were running Microsoft Windows NT Server 4.0 SP5. See figure 4. IPSec Tunnel Test Methodology The Tolly Group engineers tested the systems under test in IPSec tunnel configurations for both application and zero-loss throughput results. For application tests, engineers configured Chariot to generate bidirectional FTP © 2001 The Tolly Group 3Com CoreBuilder 3500 Figure 5 and SAP R/3 traffic. All traffic was encrypted for DES-3 and the NetScreen-5 and WatchGuard SOHOs used SHA-1 authentication and a shared secret key. The SonicWALL SOHO/50 system used MD-5 authentication and a shared secret key. Chariot measured the throughput as effective user/application data in Mbit/s. The two DominoPlus DA-360 devices that were outside the IPSec tunnel verified packet sizes and utilization. The DominoPlus DA-360 configured in-line with the IPSec tunnel verified the encapsulation of each packet. For steady-state, zero-loss, bidirectional packet-per-second tests, engineers measured the percent each system could forward when offering increments of 5% of the theoretical maximum load. SmartBits generated 64-, 512-, 1,024- and 1,518-byte packets in separate tests with each system under test configured in an IPSec tunnel via a 10/100 Mbit/s halfduplex Ethernet link. The Tolly Group considers aggregate zero-loss packetper-second throughput to be equal to, or less than, 0.001% of the total transmitted packets. SmartBits measured the percent of traffic forwarded by the tunnel under test. The two DominoPlus DA-360 devices that were outside the IPSec tunnel verified packet sizes and utilization. The DominoPlus DA-360 configured in-line with the IPSec tunnel verified the encapsulation of each packet. Firewall Test Bed Configuration To test the systems under test for firewall throughput, engineers removed one of the two Internet security appliances in each test system and the DominoPlus DA-360 located in between these appliances. The remaining test bed was the same as the IPSec tunnel test bed. The Tolly Group engineers measured the percent each device forwarded when offered 100% of the theoretical maximum load. SmartBits generated 64-, 512-, 1,024and 1,518-byte UDP packets in separate tests with each device under test. The Tolly Group considers aggregate zero-loss packet-persecond throughput to be equal to, or less than, 0.001% of the total transmitted packets. SmartBits measured the percent of traffic forwarded by the firewall under test that was configured for a single rule. The two DominoPlus DA-360 devices that were on either side of the firewall verified packet sizes and utilization. Equipment Acquisition and Support The SonicWALL SOHO/50 and WatchGuard SOHO were acquired through normal product distribution channels. The Tolly Group contacted executives at SonicWALL, Inc. and WatchGuard Technologies Inc. and invited them to provide a higher level of support than available through normal channels. Both companies Page 5 The Tolly Group NetScreen Technologies, Inc. NetScreen-5 IPSec Tunnel Test Bed Spirent Communications SmartBits SMB-200 Chariot Endpoint NetIQ Chariot/ Spirent SmartWindows/ Acterna Domino NAS Console 3Com SuperStack II 3300 Chariot Endpoint System Under Test Acterna DominoPlus Internetworking Analyzer DA-360 Acterna DominoPlus Internetworking Analyzer DA-360 Source: The Tolly Group, January 2001 accepted and provided phone technical support to assist Tolly engineers to configure/tune the devices for the test suites executed by The Tolly Group. The Tolly Group verified product release levels and shared test configurations with the vendors in order to give them an opportunity to optimize their devices for the tests. Chariot Endpoint 3Com CoreBuilder 3500 Chariot Endpoint Figure 5 Results were shared with the competitive vendors and the vendors acknowledged their accuracy. For a more complete understanding of the interaction between The Tolly Group, and SonicWALL and WatchGuard, refer to the Technical Support Diary for Competitive Products Tested posted on The Tolly Group's World Wide Web site at http://www.tolly.com (see document 200230). The Tolly Group gratefully acknowledges the providers of test equipment used in this project. Vendor Product Web address Acterna Corp. DominoFastEthernet DA-360 http://www.acterna.com NetIQ Chariot 4.0 http://www.netiq.com Spirent Communications SmartBits SMB-200 http://www.spirentcom.com Since its inception, The Tolly Group has produced highquality tests that meet three overarching criteria: All tests are objective, fully documented and repeatable. We endeavor to provide complete disclosure of information concerning individual product tests, and multiparty competitive product evaluations. As an independent organization, The Tolly Group does not accept retainer contracts from vendors, nor does it endorse products or suppliers. This open and honest environment assures vendors they are treated fairly, and with the necessary care to guarantee all parties that the results of these tests are accurate and valid. The Tolly Group has codified this into the Fair Testing Charter, which may be viewed at http://www.tolly.com. Project Profile Sponsor: NetScreen Technologies, Inc. Document number: 200230 Product Class: Internet security appliance Products under test: • NetScreen-5 v. 2.00r3 • SonicWALL SOHO/50 v. 5.0.2 • WatchGuard SOHO versions 1.5.8 and 2.1.3 Testing window: July through September 2000 Additional information available: • Technical Support Diary For more information on this document, or other services offered by The Tolly Group, visit our World Wide Web site at http://www.tolly.com, send E-mail to [email protected], call (800) 933-1699 or (732) 528-3300. Internetworking technology is an area of rapid growth and constant change. The Tolly Group conducts engineering-caliber testing in an effort to provide the internetworking industry with valuable information on current products and technology. While great care is taken to assure utmost accuracy, mistakes can occur. In no event shall The Tolly Group be liable for damages of any kind including direct, indirect, special, incidental, and consequential damages which may result from the use of information contained in this document. All trademarks are the property of their respective owners. The Tolly Group doc. 200230 rev. kco 04 Jan 01 © 2001 The Tolly Group Page 6