Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Parametric Bottlenecks Testing Catalogue (POSCA) Yu Yang 20161201 Contents • Goals and scope of POSCA testsuite • Landscape of POSCA testsuite • • • Work Plan for POSCA testsuite • • Interaction with Yardstick for POSCA Metrics & Tools for POSCA POSCA in Bottlenecks D release Draft Demo Parametric Bottlenecks Testing Catalogue in Bottlenecks (POSCA) 3. Upstream Develop DB DB DB DB Yardstick VSPERF SFC Cperf StorPerf 1. Classified bottlenecks IPv6 Bottlenecks SDNVPN Installers Test Cases Network Storage Test Results Compute Midware APP Bottlenecks Testing Results OVS4NFV 4. Performance Improvement 2. Feedback bottlenecks OPNFV Reference Platform 3 Interaction with Yardstick for POSCA Task.iter() Metric Overall Aver. Res. Time Compute Max Num. User Network Throughput Storage Requests/Sec Bottlenecks N S C M A Midware APP Latency Yardstick Metric Compute SW Tuning Logics HW Tuning Protocol Opt. SLA Decoupling Bottlenecks Restful Storage Network Locate Bottlenecks Midware APP … 4 Max Value Min. Value Aver. Value Metrics & Tools Discussion • Target – Metrics Set for Specific Bottlenecks – Feature testing results could be organized into different metrics sets to find the bottlenecks • Monitoring – – – – – Compute: latency, utilization of CPU, cache size, etc. Network: throughput, number of connection, packet delay, etc. Storage: memory available mbytes, pages/sec, idle time, etc. Midware: concurrent request, response speed, throughput, etc. APP: throughput, latency, request/user, etc. Metrics from Yardstick 6 A Brief List of Metrics and Tools Category Bottlenecks M&T List Compute Metrics Set Description https://cloud.google.com/monitoring/api/metrics http://www.slac.stanford.edu/xorg/nmtf/nmtf-tools.html http://www.applicationperformancemanagement.org/network-monitoring/network-monitoring-tools/ Short of Processor (System\%Total processor time, Processor %Processor Time, system\Processor Queue Length) latency reponse time throughput (reponse time, %package loss) Short of Memory (Memory Available MBytes) (Page Reads/Sec, Page/Sec) PS is not necessarily lack of memory when it is high, maybe an application sequentially reading a memory mapped file (%Disk Time, Page Reads/Sec, Avg.Disk Queue Length) Short of memory will cause using Disk Cache memory leak (Memory Available MBytes) (Process\Private Bytes, Process\Working Set, Memory\Available bytes) PPB an PWS are couters that increase when MAB decreases I/O (PhysicalDisk/%Disk time,PhysicalDisk/%Idle Only DT is hight, then Disk is not the bottlenecks. Time,Physical Disk\ Avg.Disk Queue Length, PRS is to avoid memory shortage Disk sec/Transfer) Network Storage Metrics 2 is for SQL Server PPT is to avoid memory shortage SPQL is to trace LB of processors Metrics 1 is for web server Metrics 2 is for where the network congestion occur and throuput reaches it bottleneck More are under discussion and planed to develop Work Plan for D Release • Adding testing suite to Bottlenecks projects – – • Determine metrics set and tools for the initial setup – – – • Jenkins job and proposed test suite Code structure in the Bottlenecks repo Compute: Short of Processor Network: bandwidth, latency and throughput Storage: Short of Memory, memory leak, I/O Determine test story/test cases – – – Factor test: base test cases that Feature test and Optimization will be dependant on Feature test: test cases for features/scenarios Optimization test: test to tune the system parameter • RESTful API – • CLI framework – • Flask + Swagger Click Dashboard for test result – Kibana & Elasticsearch Work Plan for D release Proposed Test Suite POSCA Test Cases Discussion & Current Status Bottlenecks VSTF Rubbos Posca Factor Test Test Case Description System Bandwidth Throughput Compute Burdon CPU Load TX/RX Cache Size Cache Size TX/RX Package Size Package Size Protocol Limit Feature Test Test Case Description Optimization Test POSCA Demo Jan 2016 11 Demo Contents • Bottlenecks CLI Framework • • • • CLI virtual environment setup POSCA testcase run CLI virtual environment clearup POSCA Demo • • • • Docker-compose for POSCA setting up dockers for different components Calling Yardstick to run testcases Bottlenecks for system bandwidth Kibana & Elasticsearch for testing results Demo Steps • Step 1: Build docker images • Step 2: Docker-compose setting up SUT • Step 3: Login bottlenecks docker and set CLI virtual enviroment • Step 4: Run testcase • Step 5: Results sent to designed EK • Step 6: Clearup CLI virtual enviroment, logout bottlenecks docker and clearup docker resources Thank You Jan 2016 14 Some Storage Metrics • Capacity utilisation: in terms of percent/GB of space used, as well as subcategories such as raw, formatted, free, allocated or allocated not used • I/O per second (IOPS) • Bandwidth • Latency • Access time: read, write, random • Energy usage: from macro (subsystem) to micro (device or component) • Mean time between failure (MTBF) • Mean time to repair or replace (MTTR) failed subsystems/components • Recovery point objective (RPO): The point in time to which you want data restored • Recovery time objective (RTO): The time period in which data to the point required by the RPO must be restored.