Download IoEHowToTest-Softtest

Paul Gerrard
[email protected]
gerrardconsulting.com
@paul_gerrard
How to Test the
Internet of Everyting
Background
• The IoE will dominate our industry for 10 to 20 years
• Right now, it is very confusing
• Standards emerging, but current applications are
bleeding edge, speculative, exploratory (often simple)
• Security and privacy risks dominate today
• But functional and integration complexity challenges
must also be overcome
• There are also broader societal risks
• Government, companies and individuals need to pay
attention.
Intelligent Definition and Assurance
Slide 2
Testing the Internet of Everything
An article and webinar series
1. "Internet of Everything – What is it and how
will it affect you?"
2. "Internet of Everything – Architecture and
Risks"
3. "Internet of Everything – Test Strategy"
All available at:
http://gerrardconsulting.com/?q=node/652
Intelligent Definition and Assurance
Slide 3
What is the IoE?
Devices, makers, corporates, smart
cities, drones, 3-D printed cars
Some cheap devices
Slide 5
My Robot (and a better one)
https://www.youtube.com/watch?v=f0CluQiwLRg
Slide 6
Slide 7
Smart City World Congress
Slide 8
Slide 9
Test this?
Slide 10
Architecture(s)
A Layered Model of the IoE
• No standard for networking cheap devices
• No standard/conventional architecture for IoE
• But...
– some patterns are emerging and the architecture
can be split into logical layers
– Standards are emerging (slowly)
– The 7 layer model describes the features of
elements in an IoT architecture.
Intelligent Definition and Assurance
Slide 12
Seven layer architecture
internet connected
Local, metro
network
Data-Centre
Based
Layers are often
combined into
one object/system
Internet
connected
Not every system
has seven
separate layers.
Not Internet
connected
Local
Network
Slide 13
Drivers
Introduction
• IoE test strategy – for testing everything?
– Can only paint a picture
– Dimensions, influences, and opportunities
– Suggest what testers need to think about
• The challenge we face is somewhat different
to what we have tested before.
Intelligent Definition and Assurance
Slide 15
What's in Scope?
• The scope of IoE is growing and includes:
– Vast numbers of static devices, mostly sensors
– Mobile devices: cars, buses, trains, planes, ships, satellites...
• It will include:
– Objects in homes, cities, work, hospitals, social venues
– Wearable, human-embedded devices, phones and tablets
• IoE integrates with 100s of services, payment and legacy
systems
• Eventually, IoE will be called the Internet
• The test strategy for the IoE is the test strategy for everything
• Once more with feeling...
Intelligent Definition and Assurance
Slide 16
The test strategy for
the IoE is the test
strategy for everything
The Biggest Thing in 30 Years?
• Continuous delivery and DevOps bring factory
automation processes into software
• After Waterfall and Agile, CD and DevOps
become 'the third way'
• The IoE will affect everyone on the planet.
• From 50-700bn devices over the next 5 to 20
years; no one knows
• We are on a journey that could increase the scale
of the internet by one hundred times
• How will we test it?
Intelligent Definition and Assurance
Slide 18
The Testing Journey
(from the 1980s to the present)
• Screen based applications running on dumb terminals –
1980s
• GUI applications and Client/Server – 1990s
• The emergence of the Internet - 1995
• 'Mobile Revolution' - today
• Internet of Everything - emerging
– Smart phones, buildings, cars, ships, planes and cities
– BigData, pervasive, wearable computing
– Massive scale
• IoE brings new levels of complexity and scale
• Non-functional risks are well-understood – what's new?
• Functional testing and simulation at very large scale.
Intelligent Definition and Assurance
Slide 19
If CMOs hold the budget…
• There is a shift in power in the mobile space
• If marketers hold the money for building IoE:
– IT must meet short and long term business goals
– Software teams must deliver, experiment and learn
• Market testing and experimentation to drive
rapid evolution
• DevOps, Continuous Delivery, high levels of
automation and simulation
• Test analytics* blended with production analytics.
* Thinking Big: Introducing Test Analytics, http://gerrardconsulting.com/?q=node/630
Intelligent Definition and Assurance
Slide 20
Connecting the Unconnected
• 'Internet.org by Facebook' wants to provide
cheaper access to data (perhaps 1/100th of the
current price)
• Developing economies will probably lead the way
in implementing the IoE with mobile internet
• But what networking technology/protocols will
be used for local connectivity?
"No technology dominates yet. We are waiting for
technologies to improve in range and performance. The
standard technology for connecting the IoT hasn't been
invented yet."
• We are still at the start of the IoE journey
Intelligent Definition and Assurance
Slide 21
The Scope of IoE Testing
Functional testing
• Hardware-Level functionality
– Most of this testing will be performed by manufacturers
• Object and Server level functionality
– We'll have to test simple web-apps to systems with many
complex sub-systems
• Integration and Interaction at all levels
– Companies, testers will have to use judgement on deciding
what to test and how much
• Collaboration confusion
– Use expectations and user testing of collaborating systems
will be difficult.
Intelligent Definition and Assurance
Slide 23
Testing @ Scale
• Scalability
– We'll have to do some large-scale load,
performance and stress testing
• Wearables and embedded
– We will need to simulate and test thousands of
devices carried by or embedded in people
• Everything connected
– We'll need to do location-based tests at very large
scale.
Intelligent Definition and Assurance
Slide 24
Networks
• Moving networks
– We'll have to test security and resilience of our systems against
alien or rogue networks
• Network security risks at multiple levels
– Networks will need to be hardened and tested at all levels, not
just 'perimeter'
• Device registration, provisioning, failure and security
– Power-down, power-up and automated authentication,
configuration and registration processes will need to be tested
• Mobile objects
– Power, interference, network strength, roaming and jamming
issues will all need testing.
Intelligent Definition and Assurance
Slide 25
Big Data
• Logistics
– A substantial data storage service will have to be
tested (performance, backup/recovery etc.)
• Analysis and visualisation
– These outputs and processes will need to be
tested
• Personal and corporate privacy
– Testing will need to embrace the ethical use of
data and it's protection from misuse.
Intelligent Definition and Assurance
Slide 26
The Scope of IoE Testing
• Not all IoE systems will be huge or complex
• But we face a wider range of concerns than
ever before:
– Most bring a new technology and risk profile
– The range of testing issues we need to consider
– The scale of the testing required has increased
• We'll need new approaches to tackle the risks
of failure.
Intelligent Definition and Assurance
Slide 27
Test Strategy for the IoE
Modelling, Testing and Test Data
• Tools to model the IoE are only just emerging
• We'll need to use Big Data test techniques
• 'Pattern-based test design'
– Generate tests from patterns to simulate scenarios
– 'Heuristic oracles' will derive from conventions or
common-sense
– Personas as patterns of user or system behaviour
• Developers will need to make systems testable
through High Volume Automated Testing.
Intelligent Definition and Assurance
Slide 29
Test Environments; Testing in the
Field
• Test environments will vary widely
– A test lab for a home could be set up in any office
– An urban environmental management system will be
harder to create
• Scale of field testing will vary widely, but will become a
common (and possibly mandatory) testing stage
• Barcelona has set up an urban-scale testing facility
– Barcelona Urban Lab is positioned as a field test
environment to prove the viability of innovative systems
• Cities around the world will offer similar services.
Intelligent Definition and Assurance
Slide 30
Slide 31
Tool Support
• Automated support will be essential and
probably dominate testing
• Most components tested through APIs, web or
other services
• New protocols emerging; custom drivers may
be required
• The challenge:
"How do we design the thousands or millions of
tests to feed the tools?"
Intelligent Definition and Assurance
Slide 32
Test Analytics
• DevOps processes and production analytics
monitor system vital signs (like sensors)
– These analytics are essentially the same
• Production systems are platforms for
experimentation
– The code that serves marketers can also serve
developers and testers
• Analytics code is your sensor network
• DevOps processes are 'things', too.
Intelligent Definition and Assurance
Slide 33
Performance Testing and Test Data
• Local network protocols
– IP based (XMPP, MQTT); few tools, but cloud services emerging
– Low Power/Lossy Network (ZigBee, 6LowPAN, DASH7/RFID,
Bluetooth and NFC); only need simple drivers to 'chirp'
• Coherent data may be a problem, e.g. location data
– Cars must follow the road network
– People can only walk through certain places in a shopping mall
– So you will have to capture the GPS data using a tool perhaps
• To create unique paths, you could edit together segments
of intersecting paths.
Intelligent Definition and Assurance
Slide 34
The Future for Testing
and Testers
We need to think a little differently
A New Model for Testing
Manual v Automated Testing?
• Automation will not make testing easy; it will
make testing possible
• More and more, DevOps processes are a source
of data for analysis just like production
• You might not need to be a programmer but you
may need to:
– Write your own utilities, download, configure and
adapt open source tools or create automated tests
– Have informed conversations with developers
• Should testers learn to code? Probably
Intelligent Definition and Assurance
Slide 36
New Model Testing
29 page paper: http://dev.sp.qa/download/newModel
Slide 37
Shift-Left
• Every company seems to be 'shifting left'
– Developers responsible for testing – no testers
– Testers embedded in development teams
– Testers morphing into business or systems analysts
• Testing thinking is being moved left and earlier in
the process
• The New Model identifies these thinking activities
• Testers need to learn how to 'let go' of testing, so
view this as an opportunity, rather than a threat.
Intelligent Definition and Assurance
Slide 38
Close
• The strategy for testing the IoE is the strategy for
testing everything
• Some of the tools we need don't yet exist
– Test modelling, test pattern/data generators and
automated oracles
– Simulation/high volume automation/visualisation tools
• Testing in the lab and experimentation in the field
become one discipline informed by Test Analytics
• New test approaches will be required
• The New Model aims to trigger some new
thinking in this area.
Intelligent Definition and Assurance
Slide 39
Slide 40
Slide 41
Slide 42
Paul Gerrard
[email protected]
gerrardconsulting.com
@paul_gerrard
How to Test the
Internet of Everything