Download University of Utah Green Infrastructure Monitoring Database

1
UNIVERSITY OF UTAH GREEN
INFRASTRUCTURE
MONITORING DATABASE
CVEEN 7970 Hydroinformatics
Semester Project
Zachary Magdol, Jai Kanth Panthail, Pratibha
Sapkota, Megan Walsh
2
Outline
• Introduction
• GI sites around the University
• GIRF(Green Infrastructure Research Facility)
• ODM schema for GIRF Database
• Data and Database Requirements
• Implementing MSQL Database
• Challenges of implementing the database for GIRF
• PYTHON for Data Analysis
• Calculations with Python
• Plotting in PYTHON
• DBPlot (Alternative Data Analysis Tool)
• Future Work
3
Green Infrastructure Monitoring
• Green infrastructure sites in and around U of U campus
employ sensors to monitor various parameters for
research and performance evaluation
• Problem:
• No framework for organizing data
• No method for compiling data
• No interface for visualizing/analyzing data
4
Green Infrastructure Monitoring
Variables
• Soil moisture
• Soil temp
• Flow
• Depth
• Water quality
• Climatic data
5
Green Infrastructure Research Facility
(GIRF)
6
GIRF
7
Data
Synthetic Storm
Water
Sensors monitor soil moisture and
soil temperature at different depths
and locations
0.6 m Topsoil
0.6 m Gravel
Infiltration
Drain
2m
Data logger collects soil moisture, soil
temperature and battery voltage
observations every 30 minutes
Battery powers sensors and data
logger
8
Current data management plan
• No set technique for compiling, organizing, and sharing data from the
sensors
• Data collected is downloaded as CSV files, and they are analyzed
using Excel.
Problems:
 Cumbersome process
 Very difficult to find relationships
within data
 Analysis is limited using Excel
Need a proper data management plan to make
life easier for everyone!
9
Sample data:
Table 1: Moisture Data from Campbell Scientific. The first three columns represents Julian Date, the fourth column
shows time, the last column shows the battery volt, and all other columns are moisute percentage in different
gardens
127
2013
252
1200
0.186
0.161
0.185
0.091
0.09
0.085
0.157
0.155
11.66
127
2013
252
1230
0.186
0.161
0.185
0.091
0.09
0.085
0.157
0.155
11.69
127
2013
252
1300
0.187
0.161
0.185
0.091
0.09
0.085
0.156
0.155
11.7
127
2013
252
1330
0.188
0.162
0.185
0.091
0.09
0.086
0.156
0.154
11.7
127
2013
252
1400
0.188
0.162
0.184
0.091
0.09
0.086
0.156
0.154
11.7
127
2013
252
1430
0.188
0.162
0.184
0.091
0.09
0.086
0.156
0.154
11.7
127
2013
252
1500
0.188
0.162
0.184
0.091
0.09
0.086
0.155
0.154
11.7
127
2013
252
1530
0.188
0.162
0.184
0.091
0.09
0.086
0.155
0.153
11.7
127
2013
252
1600
0.188
0.162
0.184
0.091
0.09
0.086
0.155
0.153
11.7
127
2013
252
1630
0.188
0.162
0.184
0.091
0.09
0.086
0.155
0.153
11.71
127
2013
252
1700
0.188
0.163
0.184
0.091
0.09
0.086
0.154
0.153
11.7
127
2013
252
1730
0.188
0.163
0.184
0.091
0.09
0.086
0.154
0.152
11.7
127
2013
252
1800
0.188
0.163
0.184
0.091
0.09
0.087
0.154
0.152
11.7
127
2013
252
1830
0.187
0.163
0.184
0.091
0.09
0.087
0.154
0.152
11.7
127
2013
252
1900
0.187
0.163
0.184
0.091
0.09
0.087
0.154
0.152
11.7
127
2013
252
1930
0.187
0.163
0.184
0.091
0.09
0.087
0.154
0.152
11.7
127
2013
252
2000
0.186
0.163
0.184
0.091
0.09
0.087
0.154
0.152
11.7
127
2013
252
2030
0.186
0.163
0.184
0.091
0.09
0.086
0.154
0.152
11.69
10
GIRF Data Management Plan
Main requirements:
1. Automated data entry and storage
2. Easy querying of data
3. Connections with analysis programs and
code
4. Easy user interface
5. Suitable for future upgrades and
modifications
User Interface
(Hydrodesktop, Custom GUI or
Webpage)
Analysis tools
(Python, R, Matlab, DB Plot etc)
Database
(MySQL, Microsoft SQL etc)
Automated data input
(Wireless links, Telemetry etc)
Data storage
Data retrieval
and input
User
interface
Data Analysis
11
GIRF Database Framework
We required a suitable database schema to suitably store the GIRF data…
• Schema requirements:
1.
2.
3.
4.
5.
6.
Support for relational queries.
Built to be suitable for storing hydrological data.
Sufficient support for storing metadata
Background data like Variables and Units is already present in the
database schema
Compatibility with CUAHSI tools like HydroDesktop and HydroServer Lite
- Future opportunity!
Easy implementation in MySQL
12
Option 1: Custom database Schema 1
Developed by Meg Walsh
13
Option 1: Custom database Schema 2
Developed by Zach Magdol
14
Option 3: ODM schema- Selected option
Source:
http://his.cuahsi.org/images/ODM1_1SchemaDiagram_md.jpg
15
Database implementation
Adding sites
specific to
GIRF
-> Using MySQL Command Line Client and TOAD®
LOCK TABLES `sites` WRITE;
/*!40000 ALTER TABLE `sites` DISABLE KEYS */;
INSERT INTO `sites` VALUES (1,'UOUGIRF1','GIRF-Upland-1',40.76060748,111.830785,105,'Bioretention',1486,'MSL',NULL,NULL,NULL,NULL,'Utah','Salt
County',NULL),(2,'UOUGIRF2','GIRF-Control-2',40.76060748,111.830785,105,'Bioretention',1486,'MSL',NULL,NULL,NULL,NULL,'Utah','Salt
County',NULL),(3,'UOUGIRF3','GIRF-Wetland-3',40.76060748,111.830785,105,'Bioretention',1486,'MSL',NULL,NULL,NULL,NULL,'Utah','Salt
County',NULL),(4,'UOUGIRF4','GIRF-Control-4',40.76060748,111.830785,105,'Bioretention',1486,'MSL',NULL,NULL,NULL,NULL,'Utah','Salt
County',NULL),(5,'UOUGIRF5','GIRF-Upland-5',40.76060748,111.830785,105,'Bioretention',1486,'MSL',NULL,NULL,NULL,NULL,'Utah','Salt
County',NULL),(6,'UOUGIRF6','GIRF-Wetland-6',40.76060748,111.830785,105,'Bioretention',1486,'MSL',NULL,NULL,NULL,NULL,'Utah','Salt
County',NULL),(7,'UOUGIRF7','GIRF-Wetland-7',40.76060748,111.830785,105,'Bioretention',1486,'MSL',NULL,NULL,NULL,NULL,'Utah','Salt
County',NULL),(8,'UOUGIRF8','GIRF-Upland-8',40.76060748,111.830785,105,'Bioretention',1486,'MSL',NULL,NULL,NULL,NULL,'Utah','Salt
County',NULL),(9,'UOUGIRF9','GIRF-Control-9',40.76060748,111.830785,105,'Bioretention',1486,'MSL',NULL,NULL,NULL,NULL,'Utah','Salt
County',NULL);
/*!40000 ALTER TABLE `sites` ENABLE KEYS */;
UNLOCK TABLES;
Lake
Lake
Lake
Lake
Lake
Lake
Lake
Lake
Lake
Challenge 1:
•The ODM schema was not originally designed to
store data from bioretention plots. At the University of
Utah, each plot (a site) has individual bioretentions
(sub-sites).
 Each bioretention plot was considered to be a site
and was added separately to the ‘Sites’ table.
16
--- Dumping data for table `variables`
--
Adding
variables
specific to
GIRF
LOCK TABLES `variables` WRITE;
/*!40000 ALTER TABLE `variables` DISABLE KEYS */;
INSERT INTO `variables` VALUES (1,'U1','UOUVolt','Not Applicable',168,'Other','Field
Observation',1,30,102,'Minimum','Instrumentation',-9999),(2,'U2','UOUTemp5','Not
Applicable',97,'Soil','Field Observation',1,30,102,'Continuous','Climate',9999),(3,'U3','UOUTemp25','Not Applicable',97,'Soil','Field
Observation',1,30,102,'Continuous','Climate',-9999),(4,'U4','UOUTemp50','Not
Applicable',97,'Soil','Field Observation',1,30,102,'Continuous','Climate',9999),(5,'U5','UOUMoist10','Not Applicable',1,'Soil','Field
Observation',1,30,102,'Continuous','Climate',-9999),(6,'U6','UOUMoist30','Not
Applicable',1,'Soil','Field Observation',1,30,102,'Continuous','Climate',9999),(7,'U7','UOUMoist50','Not Applicable',1,'Soil','Field
Observation',1,30,102,'Continuous','Climate',-9999),(8,'U8','UOUMoist25','Not
Applicable',1,'Soil','Field Observation',1,30,102,'Continuous','Climate',-9999);
/*!40000 ALTER TABLE `variables` ENABLE KEYS */;
UNLOCK TABLES;
Challenge 2:
•Each bioretention plot (a ‘Site’ in the GIRF database)
had sensors at multiple depths. For example, moisture
sensors existed at depths of 10cm, 25cm, 30cm and
50cm.
 Four new variables representing the moisture at
individual depths had to be created. This was
required to preserve the dissimilarity of data from
each depth.
17
Sample data added using TOAD
 Data for GIRF-Upland-1, GIRF-Control-2 and GIRF-Wetland-3
 Variables consisted of UOUVolt, UOUTemp5, UOUMoist10 and UOUMoist25
 Total of approximately 15000 values were added to the ‘datavalues’ table
18
Data Analysis
Python
19
Python
• MySQLdb
• Python interface
• Free!
• connects to the database with the
connection command, or “cnxn”.
20
Python
• Matplotlib.pyplot
• 2D plotting library
• Free!
• Numpy
• enables complex scientific
computing to be performed in
python
• Free!
Source: http://matplotlib.org/
Source: http://www.numpy.org/
21
Python Analysis & Results
Get Values
Average Daily Temperature
22
Python
• Advantages
• Free
• Easy to use
• Easy to connect with MySQL databases
• Plotting capabilities: user-defigned plot preferences
• Disadvantages
• Indentations
• Must download/import appropriate modules
• Numerous lines of code
23
Alternative Data Analysis Tool
• DBPlot (http://sourceforge.net/projects/dbplot/)
• GUI to visualize and manipulate data
• Supports MySQL, MS SQL, SQLite
Query script
Query results
Query status
24
DBPlot
• Compute and plot daily average, maximum and minimum
soil moisture at 10 cm depth in Site 1
select EXTRACT(day from localdatetime) AS theday, EXTRACT(year from localdatetime) AS theyear,
EXTRACT(month from localdatetime) AS themonth, avg(datavalue) AS avg_per_cent, min(datavalue) AS
min_per_cent, max(datavalue) AS max_per_cent
from datavalues where siteid=1 and variableid=5 group by EXTRACT(day from localdatetime) order by theday
25
DBPlot
• Advantages:
• Easy to use
• Same language as database
• Rapidly query and visualize data in one tool
• Disadvantages:
• Poor plot quality and customization capabilities
• Limited analyses
26
Conclusion
• Successfully loaded GI data into ODM schema
• Data uploaded to MySQL using Toad
• Explored data analysis alternatives
• Python
• DBPlot
• R
27
Future Work
• Expand current database to include data from all GI sites
in the University
• Automate data acquisition and analysis
• Customize user-friendly interface to allow
data visualization
• Create a GI website, and connect to the database for data
sharing