Download Powerpoint

Azure ML in SSIS
An introduction to Azure Machine Learning
Through the eyes of an SSIS developer
David Söderlund – SolidQ Nordic
[email protected]
Twitter: @QuadmanSWE
LinkedIn: https://se.linkedin.com/in/soderlunddavid
Sponsors
Using and improving Azure ML in SSIS
 Introduction to Azure Machine Learning




Predictive analysis
Fully managed in Azure
Pay as you go
Free trials available with limited functionality
 Interacting with Azure ML Predictive
experiments through SSIS
 Custom component for batch execution
 Azure Feature Pack for blob storage interaction
3 | 2015-09-05 | SQL Saturday #433, Gothenburg
Machine Learning / Data mining
 Consider machine learning if




There is a pattern you would like to find
There is no apparent mathematical solution
There is enough data
Senior data scientists get involved in larger
projects but everyone should know the essentials.
 We will learn how to implement machine
learning models in our daily ETL!
4 | 2015-09-05 | SQL Saturday #433, Gothenburg
Machine Learning / Data mining
 What is learning?
 Reinforcement learning by looking at examples,
this is why we need data
 Look at examples with correct labeling and try to
figure out the target function
 Come up with hypothesis function
 Validate
5 | 2015-09-05 | SQL Saturday #433, Gothenburg
Typical Machine Learning project roles
 Domain expert
 The people who know the data for the problem
 Developer
 The people who can transform data to conform to
the models and to the project’s needs
 Data scientist
 Runs the experimental process
 The name comes from applying the scientific
method to data driven decision making.
6 | 2015-09-05 | SQL Saturday #433, Gothenburg
Azure ML compared to SSAS DM
 Getting started, developing models and trying
out ideas is not a huge project in itself.
 To try out a model requires data at hand and a
web browser
 You don’t have to know everything to get
some insights
 SSAS is better for big projects with more
complex models.
 Better visualization tools in visual studio
7 | 2015-09-05 | SQL Saturday #433, Gothenburg
Azure ML short overview (demo soon)
 The final product is a model
 The model can be used to predict values on data
that it hasn’t seen before
 Everything is an experiment
 Not every experiment is successful
 You don’t need to know everything to run it
8 | 2015-09-05 | SQL Saturday #433, Gothenburg
Azure ML short overview (demo soon)
 Training experiments
 Looks like an SSIS package
 When it works we can “compile” the model and
produce a predictive experiment
 Predictive experiment
 Uses a “trained model” and does not retrain every
run
 The inputs are run through the model and
become outputs
9 | 2015-09-05 | SQL Saturday #433, Gothenburg
Demo?
DEMO
10 | 2015-09-05 | SQL Saturday #433, Gothenburg
Executing predictive scoring
 Web service API
 Request – Response Service (RRS)
 Row by agonizing row
 Useful for streaming data
 Batch Execution Service (BES)
 Can only run off of files in azure blob storage
 Therefor we must make sure our data is in that format
before running the batch
 Results are also stored in this format and a relative
location is returned in the execution status
 We can use this to retrieve the results!
11 | 2015-09-05 | SQL Saturday #433, Gothenburg
Batch execution service from SSIS
 Make sure the data is in blob storage
 Run the batch execution by pointing to the
correct container and file
 Read the data from blob storage with the
container/file that execution returned.
12 | 2015-09-05 | SQL Saturday #433, Gothenburg
SSIS requirements for development





SSIS Azure Feature Pack
A custom component for the HTTP-request
Connection manager to our source data
Connection manager to azure blob storage
Connection information to our Azure ML
experiment
 Variables to help us keep track of where the
experiment output is
13 | 2015-09-05 | SQL Saturday #433, Gothenburg
SSIS requirements for development
14 | 2015-09-05 | SQL Saturday #433, Gothenburg
SSIS requirements for development
15 | 2015-09-05 | SQL Saturday #433, Gothenburg
Demo?
DEMO
16 | 2015-09-05 | SQL Saturday #433, Gothenburg
What about updating and improving?
 Experiments can have multiple web service
outputs, so make an output for the score
model module.
 Run the training experiment through it’s API
and catch both the new model object and its
score.
 In SSIS compare the old and the new then
update the predictive experiment with the
new model, if it’s better.
17 | 2015-09-05 | SQL Saturday #433, Gothenburg
Automate!
 Figure out how the components work
 Parameterize the locations and keys that
change with each experiment
 Auto-generate packages from metadata with
BIML
18 | 2015-09-05 | SQL Saturday #433, Gothenburg
Useful links to replicate setup
 Azure trial account (for storage and studio)
 SSIS Azure Feature pack
 Chris Price (BluewaterSQL)
ExecureAzureMLBatch Component
 Some assembly required… get it? 
 How to retrain models programmatically by
Raymond Laghaeian (not part of demo)
 Automate your batch package with BIML
19 | 2015-09-05 | SQL Saturday #433, Gothenburg