Download OPNFV PlugFest

Parametric Bottlenecks
Testing Catalogue (POSCA)
Yu Yang
20161201
Contents
• Goals and scope of POSCA testsuite
• Landscape of POSCA testsuite
•
•
•
Work Plan for POSCA testsuite
•
•
Interaction with Yardstick for POSCA
Metrics & Tools for POSCA
POSCA in Bottlenecks D release
Draft Demo
Parametric Bottlenecks Testing Catalogue in Bottlenecks (POSCA)
3. Upstream Develop
DB
DB
DB
DB
Yardstick
VSPERF
SFC
Cperf
StorPerf
1. Classified bottlenecks
IPv6
Bottlenecks
SDNVPN
Installers
Test Cases
Network
Storage
Test Results
Compute
Midware
APP
Bottlenecks Testing Results
OVS4NFV
4. Performance Improvement
2. Feedback bottlenecks
OPNFV Reference
Platform
3
Interaction with Yardstick for POSCA
Task.iter()
Metric
Overall
Aver. Res. Time
Compute
Max Num. User
Network
Throughput
Storage
Requests/Sec
Bottlenecks
N
S
C
M
A
Midware
APP
Latency
Yardstick
Metric
Compute
SW Tuning
Logics
HW Tuning
Protocol Opt.
SLA
Decoupling
Bottlenecks
Restful
Storage
Network
Locate Bottlenecks
Midware
APP
…
4
Max Value
Min. Value
Aver. Value
Metrics & Tools Discussion
• Target
– Metrics Set for Specific Bottlenecks
– Feature testing results could be organized into different metrics sets to find the
bottlenecks
• Monitoring
–
–
–
–
–
Compute: latency, utilization of CPU, cache size, etc.
Network: throughput, number of connection, packet delay, etc.
Storage: memory available mbytes, pages/sec, idle time, etc.
Midware: concurrent request, response speed, throughput, etc.
APP: throughput, latency, request/user, etc.
Metrics from Yardstick
6
A Brief List of Metrics and Tools
Category
Bottlenecks
M&T List
Compute
Metrics Set
Description
https://cloud.google.com/monitoring/api/metrics
http://www.slac.stanford.edu/xorg/nmtf/nmtf-tools.html
http://www.applicationperformancemanagement.org/network-monitoring/network-monitoring-tools/
Short of Processor
(System\%Total processor time, Processor
%Processor Time, system\Processor Queue
Length)
latency
reponse time
throughput
(reponse time, %package loss)
Short of Memory
(Memory Available
MBytes)
(Page Reads/Sec, Page/Sec)
PS is not necessarily lack of memory when it is
high, maybe an application sequentially reading a
memory mapped file
(%Disk Time, Page Reads/Sec, Avg.Disk Queue
Length)
Short of memory will cause using Disk Cache
memory leak
(Memory Available
MBytes)
(Process\Private Bytes, Process\Working Set,
Memory\Available bytes)
PPB an PWS are couters that increase when MAB
decreases
I/O
(PhysicalDisk/%Disk time，PhysicalDisk/%Idle
Only DT is hight, then Disk is not the bottlenecks.
Time，Physical Disk\ Avg.Disk Queue Length，
PRS is to avoid memory shortage
Disk sec/Transfer)
Network
Storage
Metrics 2 is for SQL Server
PPT is to avoid memory shortage
SPQL is to trace LB of processors
Metrics 1 is for web server
Metrics 2 is for
where the network congestion occur and
throuput reaches it bottleneck
More are under discussion and planed to develop
Work Plan for D Release
•
Adding testing suite to Bottlenecks projects
–
–
•
Determine metrics set and tools for the initial setup
–
–
–
•
Jenkins job and proposed test suite
Code structure in the Bottlenecks repo
Compute: Short of Processor
Network: bandwidth, latency and throughput
Storage: Short of Memory, memory leak, I/O
Determine test story/test cases
–
–
–
Factor test: base test cases that Feature test and
Optimization will be dependant on
Feature test: test cases for features/scenarios
Optimization test: test to tune the system parameter
•
RESTful API
–
•
CLI framework
–
•
Flask + Swagger
Click
Dashboard for test result
–
Kibana & Elasticsearch
Work Plan for D release
Proposed Test Suite
POSCA Test Cases Discussion & Current Status
Bottlenecks
VSTF
Rubbos
Posca
Factor Test
Test Case
Description
System Bandwidth
Throughput
Compute Burdon
CPU Load
TX/RX Cache Size
Cache Size
TX/RX Package Size
Package Size
Protocol
Limit
Feature Test
Test Case
Description
Optimization Test
POSCA Demo
Jan 2016
11
Demo Contents
• Bottlenecks CLI Framework
•
•
•
•
CLI virtual environment setup
POSCA testcase run
CLI virtual environment clearup
POSCA Demo
•
•
•
•
Docker-compose for POSCA setting up dockers for different components
Calling Yardstick to run testcases
Bottlenecks for system bandwidth
Kibana & Elasticsearch for testing results
Demo Steps
•
Step 1: Build docker images
•
Step 2: Docker-compose setting up SUT
•
Step 3: Login bottlenecks docker and set CLI virtual enviroment
•
Step 4: Run testcase
•
Step 5: Results sent to designed EK
•
Step 6: Clearup CLI virtual enviroment, logout bottlenecks docker and
clearup docker resources
Thank You
Jan 2016
14
Some Storage Metrics
•
Capacity utilisation: in terms of percent/GB of space used, as well as subcategories such as raw, formatted, free, allocated or
allocated not used
•
I/O per second (IOPS)
•
Bandwidth
•
Latency
•
Access time: read, write, random
•
Energy usage: from macro (subsystem) to micro (device or component)
•
Mean time between failure (MTBF)
•
Mean time to repair or replace (MTTR) failed subsystems/components
•
Recovery point objective (RPO): The point in time to which you want data restored
•
Recovery time objective (RTO): The time period in which data to the point required by the RPO must be restored.