Survey
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
* Your assessment is very important for improving the workof artificial intelligence, which forms the content of this project
Python Scripting for ArcGIS Paul Zandbergen Department of Geography p g p y University of New Mexico Outline of Topics Outline of Topics • Introduction d – Examples, Python and ArcGIS, Python versions • Fundamentals of geoprocessing Fundamentals of geoprocessing in ArcGIS in ArcGIS • Python language fundamentals – – – – Where to run Python code Data types: numbers, strings, lists Functions and modules Controlling workflow g • ArcPy: Geoprocessing using Python – Using tools, functions, classes – Describing data, listing data, working with lists b d l d k hl • Creating custom tools – Script tools, tool parameters p , p • Resources Workshop Materials Posted Workshop Materials Posted posted until October 24 posted until October 24 h // http://www.paulzandbergen.com/workshops l db / kh Forthcoming Book Forthcoming Book • • • • Python Scripting for ArcGIS Esri Press Sometime in 2012 Updated for ArcGIS 10.1 • Sample exercises posted (for 10.0) Introduction Prior Knowledge and Experience Prior Knowledge and Experience • Using ArcGIS 9.3 or 10.0? – Workshop is for 10.0 p • Prior Python experience? – I’m not assuming any g y • Other programming experience? Oh i i ? – I’m not assuming any Example 1 Example 1 • Script to copy all shapefiles in a folder into a geodatabase import arcpy from arcpy import env env.overwriteOutput = True env.workspace k = "c:/workshop/ex01" " / k h / 01" fclist = arcpy.ListFeatureClasses() for fc in fclist: py ( ) fcdesc = arcpy.Describe(fc) arcpy.CopyFeatures_management(fc, "c:/workshop/ex01/study.mdb/" + fcdesc.basename) Example 2 Example 2 • Script tool to generate a k‐nearest neighbor table • Runs an existing ArcGIS tool multiple times, writes the result import arcpy from arcpy import env env.overwriteoutput it t t = True T infc = arcpy.GetParameterAsText(0) output = arcpy.GetParameterAsText(1) k = arcpy.GetParameter(2) arcpy GetParameter(2) n = 1 f = open(output, "w") while n <= < k: result = arcpy.CalculateDistanceBand_stats(infc, n) f.write(str(n) + " " + str(result[1])+ "\n") n = n + 1 f.close() Example 3 Example 3 • Script tool to run Huff model • Sophisticated analysis not available in ArcGIS Example 3 Example 3 What is Python Scripting? What is Python Scripting? • Add functionality to ArcGIS Add f i li A GIS – – – – Integrated into ArcGIS interface B ild Builds upon existing functionality i i f i li Automates repetitive tasks Expands analysis options Expands analysis options • Share new functionality Sh f ti lit – Script tools work just like regular tools – Can be integrated into models, tools C b i t t di t d l t l – Easy to share with others (free) Why Python? Why Python? • • • • • • Free, open source Object oriented Basic scripting AND complex object‐oriented programming “Batteries included” Embraced by geospatial community, including ESRI Many libraries Python Community Python Community http://www.python.org Python and ArcGIS Python and ArcGIS • Python is the preferred scripting language for ArcGIS P h i h f d i i l f A GIS 1. You can run Python from within ArcGIS – Python Window works like an interactive interpreter 2. All tools in ArcToolbox can be accessed from Python – Import ArcPy to get full library of tools 3 Python scripts can be made into tools 3. P th i t b d i t t l – Extend functionality of ArcGIS 4 Support for other scripting languages will go away 4. Support for other scripting languages will go away – VBScript and JScript being replaced by Python Python Versions and ArcGIS Python Versions and ArcGIS • Versions: – – – – Current version of Python is 3.2.2 Python that works with ArcGIS 10.0 is 2.6.x Python that works with ArcGIS 10 0 is 2 6 x Python that works with ArcGIS 10.1 is 2.7.x Move to Python 3.x likely only with ArcGIS y y y 11 • ArcGIS only works with a specific version of Python: y p y – Use the one that comes installed with ArcGIS – Don’t install your own version of Python Installing Python Installing Python • Remove any existing installations of Python • Install ArcGIS 10.0 – Python 2.6.5 will be installed by default • Install a Python editor Install a Python editor • Configure the editor to work with ArcGIS • Note: You can run different versions of Python on one machine – however, a clean install of Python2.6.5 with ArcGIS 10.0 is recommended Demo: Check ArcGIS and Python installation Fundamentals of Geoprocessing Fundamentals of Geoprocessing in ArcGIS Geoprocessing Tools Tool Organization Tool Organization Tool Dialogs Tool Dialogs Tool Parameters Tool Parameters • Parameters – Required q – Optional • Errors • Warning Environment Settings Environment Settings Geoprocessing Options Demo: Geoprocessing Fundamentals Running Python Code Two ways to run Python Code Two ways to run Python Code 1. Using an Interactive Interpreter – Code is executed directly line‐by‐line y y 2 By running a script 2. B i i t – Code saved in a .py file – Run from within a Python editor or directly from p g y operating system Where to type and run Python code? Where to type and run Python code? 1. Python window in ArcGIS – – Built into any ArcGIS Desktop application Good for testing code, very short scripts 2. Python editor – – – IDLE installed by default IDLE installed by default Many others, PythonWin is a good one to start Good for more complex code saving scripts Good for more complex code, saving scripts Python Window in ArcGIS Python Window in ArcGIS Python Window in ArcGIS Python Window in ArcGIS • Works with current map document • Interactive interpreter: – Executes code directly line‐by‐line • Good for testing short code Good for testing short code • Code can be saved • No error checking / debugging N h ki / d b i Python Editor IDLE Python Editor ‐ Python Editor PythonWin Python Editor ‐ Python Editor Python Editor • Stand‐alone – outside of ArcGIS • Interactive interpreter: Interactive interpreter: – Executes code directly line‐by‐line • Save code as script files (.py) ( ) • Good for organizing more complex code Good for organizing more complex code Demo: Running simple Python code Python Documentation Python Documentation Python Documentation http://www.python.org Python Documentation Python Documentation Version specific! p http://docs.python.org Python Beginners Guide Python Beginners Guide http://wiki.python.org/moin/BeginnersGuide Python Books Python Books Version specific! Version specific! N None of these books including anything on ArcGIS f th b k i l di thi A GIS or geoprocessing! i ! Python Language Fundamentals Python Data Types Python Data Types • • • • • Number (integer and float) String List Tuple Dictionary • Strings, lists and tuples are sequences • Strings, numbers and tuples are immutable • List and dictionaries are mutable Numbers • Integers – Whole number, i.e. no decimals – e.g. ‐34 • Floats – Decimal point – e.g. ‐34.8307 g Numerical Operators Numerical Operators Operator Description * / % + ‐ Multiplication Division Modulus Addition Subtraction Integer Example Result 9 * 2 18 9 / 2 4 9 % 2 1 9 + 2 11 9 ‐ 2 7 Floating‐point Example Result 9 * 2.0 18.0 9 / 2.0 4.5 9 % 2.0 1.0 9 + 2.0 11.0 9 – 2.0 7.0 Demo: Numerical Operators Strings • A set of characters surrounded by quotes is called a string literal • To create a string variable, assign a string literal to it >>> mytext = "Crime hotspot maps are cool." >>> print mytext Crime hotspot maps are cool. Quotes in Python Quotes in Python • In Python single and double quotes are the same • "NIJ" is the same as 'NIJ' >>> print "I I said: 'Let's Let s go!'" go! • Quotes in Python are straight‐up • "text" or 'text', not “text” or ‘text’ • Be aware of copy/paste and auto‐formatting Be aware of copy/paste and auto‐formatting Variables • Python scripts use variables to store information • To work with variables use an assignment statement >>> x = 17 >>> x * 2 34 Variables • Python uses dynamic P h d i assignment i >>> x = 17 >>> type(x) <type 'int'> int > >>> x = "GIS" >>> type(x) <type 'str'> • No need to declare variables • Value defines the type Variable Names Variable Names • Rules – Letters, digits and underscores – Cannot start with a digit – Don’t use keywords (print, import, etc.) • Recommendations – Be descriptive (count instead of c) – Keep it short (count Keep it short (count instead of count_of_records) instead of count of records) – Follow convention: all lowercase, use underscores Statement and Expressions Statement and Expressions • A Python expression is a value >>> 2 * 17 34 • A Python statement is an instruction to do something >>> x = 2 * 17 Working with Strings Working with Strings • Concatenate strings >>> >>> >>> >>> GIS x = "G" y = "I" I z = "S" print x + y + z Converting to String Converting to String >>> temp = 100 >>> print "The temperature is " + temp + " degrees" TypeError: cannot concatenate 'str' str and 'int' int objects >>print "The temperature is " + str(temp) + " degrees" • Converting the value of a variable from one type to another is known as casting Lists • A Python list is an ordered set of items • The list of items is surrounded by square brackets [ ], and the it items are separated by commas (,) t db () • Items can consist of numbers, strings and other data types mylist = [1, 2, 4, 8, 16, 32] mywords ["jpg", j "bmp", "tif"] i • Lists are very widely used in geoprocessing: – e.g. list of feature classes, list of records, list of fields, etc. Python Functions Python Functions • A function carries out a certain action • Python has many built‐in functions <function>(<arguments>) >> pow(2,3) 8 • Using a function is referred to as calling a function • Additional functions can be accessed using modules Python Methods Python Methods • A method is a function that is closely coupled to some object <object>.<method>(<arguments>) >>> topic = "Crime Mapping" >>> topic.count("i") 2 • Many of Python’s data types have methods String Indexing String Indexing • Python strings have an index positioning system P th ti h i d iti i t >>> mystring = "Crime Crime Mapping" Mapping >>> mystring[0] 'C' >>> mystring[-1] 'g' • Strings can be sliced into smaller strings using slicing >>> mystring[0:5] 'Crime’ Working with List Working with List • Python lists have an index positioning system h l h d >>> crimes = [ ["arson", arson , "burglary", burglary , "robbery"] robbery ] >>> cities[1] 'burglary' • There are many list methods >>> crimes.append("homicide") >>> crimes.remove( crimes.remove("arson") arson ) >>> crimes ['burglary', 'robbery', 'homicide'] Working with Pathnames Working with Pathnames • Pathnames are critical when writing scripts: P h ii l h ii i – Example workspace: c:\data\results – Example shapefile: c:\data\results\streams.shp E l h fil \d \ l \ h • In Python a backslash (\) is an escape character • Pathnames in Python should therefore look like one of the following "c:/data" "c:\\data" \\ r"c:\data" (raw string) Python Modules Python Modules • M Modules d l are like extensions that can be imported into Python lik i h b i di P h to extend its capabilities >>> import time • A typical module contains a number of specialized functions which can be called once the module has been imported <module>.<function> >>> time.localtime() Conditional Statements Conditional Statements • Branching can be used to control workflow import random x = random.randint(0,6) print x if x == 6: print = "You win!“ • Syntax: keyword if, followed by a condition, followed by (:) Indentation in Python Indentation in Python • Indented code is referred to as a block • Use tabs or spaces Use tabs or spaces – be consistent be consistent • Recommended: 4 spaces • Tip: Tip: be careful with copy/paste from other be careful with copy/paste from other applications More Conditional Statements More Conditional Statements • Use of elif f and else d is optionall import random x = random.randint(0,6) print x p if x == 6: print "You win!" elif x == 5: print "Try again!" else: print "You lose!" Loop Structures: While Loop Structures: While • LLoop structures allow you to repeat a certain part of your ll i f code • A while A while loop repeats until a particular condition is reached loop repeats until a particular condition is reached i = 0 while i <= 10: print i p i += 1 • The while statement uses a sentry variable in the exit condition Loop Structures: For Loop Structures: For • A for loop repeats a block of code for each element of a l bl k f d f h l f sequence mylist = ["A", "B", "C", "D"] for letter in mylist: print letter • In the example, letter is the name of a variable and for each iteration of the loop this varaible is assigned a different value ArcPy: Geoprocessing using Python What is ArcPy? What is ArcPy? • A ArcPy P was introduced with ArcGIS i d d i h A GIS 10.0 10 0 • ArcPy is a collection of modules, classes and f functions which give access to all the geoprocessing i hi h i ll h i tools in ArcGIS from within Python • Most geoprocessing M i scripts will start with: i ill ih import arcpy • Note: ArcPy replaces the older arcgisscripting module Setting Current Workspace Setting Current Workspace • After importing ArcPy, most scripts start with setting a workspace to retrieve and store files import arcpy arcpy.env.workspace k = "c:/workshop" " / k h " • In the code above env h d b i is a class and workspace l d i is a property of this class arcpy.<class>.<property> Using Tools Using Tools • ArcPy gives you access to all tools in ArcToolbox • All tools are provided as functions arcpy.<toolname_toolboxalias>(<parameters>) • Example: import arcpy arcpy env workspace = "c:/data" arcpy.env.workspace arcpy.Clip_analysis("streams.shp", "study.shp", "result.shp") Tool Parameters Tool Parameters • A good understanding of tool parameters is essential • Parameters have properties: – – – – Name Type (feature class, integer, etc.) Direction (input or output) Direction (input or output) Required or optional • Example: Clip tool p p Tool Syntax Tool Syntax Tool dialog: Python syntax: Clip_analysis(in_features, clip_features, out_feature_class, {cluster tolerance}) {cluster_tolerance}) Example: Clip_analysis("streams.shp","study.shp", " "result.shp") lt h ") Optional Parameters Optional Parameters • Required tool parameters are listed first • Optional tool parameters can be left out – But what if some need to be set? Buffer_analysis (in_features, out_feature_class buffer_distance_or_field, {line_side}, {line_end_type}, {dissolve_option}, { p }, { {dissolve_field}) }) arcpy.Buffer_analysis("roads", "buffer", "100 METERS", "", LIST , "Code") Code ) "", "LIST" arcpy.Buffer_analysis("roads", "buffer", "100 METERS", di dissolve_option=LIST, l ti LIST dissolve_field=Code) di l fi ld C d ) Hard coded Parameters Hard‐coded Parameters • Consider the example import arcpy arcpy.env.workspace = "c:/data" arcpy.Clip_analysis("streams.shp", "study.shp", "result.shp") • How can we make this code more usable? Using Variables for Parameters Using Variables for Parameters import arcpy arcpy.env.workspace = "c:/data" infc = "streams.shp" clipfc = "study.shp" outfc = "result.shp" arcpy.Clip_analysis(infc, clipfc, outfc) Variables Provided by a User Variables Provided by a User import arcpy infc = arcpy.GetParameterAsText(0) clipfc = arcpy.GetParameterAsText(1) outfc = arcpy.GetParameterAsText(2) arcpy.Clip_analysis(infc, clipfc, outfc) Result Objects Result Objects • ArcPy returns the output of a tool as a Result object import arcpy arcpy.env.workspace = "c:/data" myresult = arcpy.Clip_analysis("streams.shp“,"study.shp","result.shp") print myresult • This will print the path to the output dataset c:/data/result.shp Multiple Operations using Result Objects Multiple Operations using Result Objects • Result objects can be used as the input into another function import arcpy arcpy.env.workspace = "c:/data/study.gdb" buffer = arcpy.Buffer_analysis("str","str_buf","100 METERS") py _management(buffer) g count = arcpy.GetCount print count • This allows complex geoprocessing operations ArcPy Classes • Some tool parameters are complicated/detailed – e.g. coordinate system • ArcPy classes are used to work with these parameters – Classes are used to create objects j – Classes have properties and methods • General syntax arcpy.<classname>(<parameters>) ArcPy Classes: Example Classes: Example • The following is an example of the contents of a .prj file • To avoid having to work with this actual string, we can use a SpatialReference class ArcPy Classes: Example Classes: Example • Th The following example creates a spatial reference object f ll i l i l f bj based on an existing .prj file ‐ properties of this object can then be used import arcpy prjfile = "c:/data/streams c:/data/streams.prj prj" spatialref = arcpy.SpatialReference(prjfile) myref = spatialRef.name print myRef • This will print NAD_1983_StatePlane_Florida_East_FIPS_0901_Feet ArcPy Classes: Example Classes: Example • The following example creates a spatial reference object and use this to define the coordinate system of a new feature class import arcpy out_path = "c:/data" out_name = "lines.shp" prjfile = "c:/data/streams.prj" spatialref = arcpy.SpatialReference(prjfile) arcpy.CreateFeatureclass management(out path, out_name, arcpy.CreateFeatureclass_management(out_path, out name, "POLYLINE", "", "", "", spatialref) ArcPy Functions • All geoprocessing tools are ArcPy functions • Additional ArcPy functions: – listing data – Retrieving and setting properties g gp p – Many more… • General syntax General syntax arcpy <functionname>(<arguments>) arcpy.<functionname>(<arguments>) ArcPy Functions • • • • • • • • • • • • Cursors C Describing data Environment and settings Fields General General data functions General data functions Getting and setting parameters Licensing and installation Listing data Messaging and error handling Progress dialog Progress dialog Tools and toolboxes Describing and Listing Data Describing Data Describing Data • The Describe function is used to determine properties of dataset Th D ib f ti i dt d t i ti fd t t • General syntax import arcpy <variable> = arcpy.Describe(<input dataset>) • Example: import arcpy desc = arcpy.Describe("c:/data/streams.shp") print desc.shapeType desc shapeType Describing Data: Example Describing Data: Example import i t arcpy arcpy.env.workspace = "c:/data" infc = "streams.shp" clipfc = "study.shp" outfc = "streams_clip.shp" desc = arcpy.Describe(clipfc) arcpy Describe(clipfc) type = desc.shapeType if type == "Polygon": arcpy.Clip_analysis(infc, clipfc, outfc) else: print "The The clip features are not polygons polygons." Listing Data Listing Data • Listing data is very common Li ti d t i • Several different list functions in ArcPy – – – – – – – – – ListFields ListIndexes ListDataset ListFeatureClasses ListFiles ListRasters ListTables ListWorkspaces ListVersions • Similar logic: g – Create a list – Iterate over the list using a for loop Listing Feature Classes Listing Feature Classes • Th The ListFeatureClasses Li tF t Cl f ti function returns a list of t li t f feature classes in the current workspace • General syntax: y ListFeatureClasses ({wild_card}, {feature_type}, {feature dataset}) {feature_dataset}) • Example: import arcpy from arcpy import env env.workspace = "c:/data" fclist = arcpy.ListFeatureClasses() Listing Feature Classes Listing Feature Classes • No filtering: N filt i fclist = arcpy.ListFeatureClasses() arcpy ListFeatureClasses() • Filtering based on wild card Filtering based on wild card fclist = arcpy.ListFeatureClasses("w*") • Filtering based on feature type fclist = arcpy.ListFeatureClasses("", "point") Listing Fields Listing Fields • The ListFields h f function lists the fields in a feature class or l h f ld f l table in a specified dataset. • General syntax: G l t ListFields (dataset, (dataset {wild {wild_card}, card} {field {field_type}) type}) • Example p import arcpy arcpy.env.workspace = "c:/data" fieldlist = arcpy.ListFields("roads.shp") Using Lists in for loops Using Lists in for • The following script creates a list of fields of type String and determines for each text field what the length of the field is import arcpy arcpy.env.workspace = "c:/data" fieldlist = arcpy.ListFields("roads.shp", "", "String") for field in fieldlist: print field.name + " " + str(field.length) Using Lists in for loops Using Lists in for • The following script creates a list of TIFF files and h f ll i i li f fil d iterates through each file in the list to build pyramids id import arcpy from arcpy import env env.workspace = "c:/data" tifflist = arcpy.ListRasters("","TIF") for tiff in tifflist: arcpy.BuildParamids_management(tiff) Creating Custom Tools Ways to Execute a Script Ways to Execute a Script 1 As a stand‐alone script 1. A d l i – The script is executed from the operating system or from within a Python editor such as PythonWin within a Python editor such as PythonWin – When using ArcPy, ArcGIS needs to be installed and licensed – No ArcGIS Desktop application needs to be open 2. As a script tool within ArcGIS – – – A tool dialog is created to execute the script A tool dialog is created to execute the script Script tool looks like any other tool in ArcToolbox Tool execution is controlled from ArcGIS Desktop p Python Scripts as Tools Python Scripts as Tools Why Create Script Tools? Why Create Script Tools? • • • • • • • Tool dialog makes it easier to use Tool dialog validates user inputs Becomes part of all geoprocessing Environment settings are passed on Environment settings are passed on Writes messages to the Results window Easier to share Does not require user to know Python Steps to Create Script Tools Steps to Create Script Tools 1. 2. 3. 4. Create a Python script (.py) Create a custom Toolbox (.tbx) Create a custom Toolbox (.tbx) Add a tool to the Toolbox using Add Script Modify the script with inputs and outputs Example Script: Hardcoded Variables Example Script: Hardcoded Variables import impo t arcpy a cp from arcpy import env env.overwriteoutput = True infc = "c:/data/points.shp" / / output = "c:/data/result.txt" k = 10 n = 1 f = open(output, "w") while n <= k: result = arcpy.CalculateDistanceBand_stats(infc, n) f.write(str(n) + " " + str(result[1])+ "\n") n = n + 1 f.close() Tool Parameters and Dialog Tool Parameters and Dialog Example Script: User Provided Parameters Example Script: User Provided Parameters import impo t arcpy a cp from arcpy import env env.overwriteoutput = True infc = arcpy.GetParameterAsText(0) output = arcpy.GetParameterAsText(1) k = arcpy.GetParameter(2) n = 1 f = open(output, "w") while n <= k: result = arcpy.CalculateDistanceBand_stats(infc, n) f.write(str(n) + " " + str(result[1])+ "\n") n = n + 1 f.close() More ArcPy Functionality More ArcPy Functionality More ArcPy • Cursors to work with rows and geometry – Retrieve, edit, create • arcpy.sa module to work with rasters • a arcpy.mapping cpy. app g module module for map automation for map automation • Creating custom functions and classes Resources for Python Scripting in ArcGIS ArcGIS Desktop Help Desktop Help Virtual Campus Courses Virtual Campus Courses http://training.esri.com ArcScripts http://arcscripts.esri.com ArcGIS Resource Center Resource Center http://resources.arcgis.com ArcGIS Resource Center Resource Center http://resources.arcgis.com/content/geoprocessing/10.0/about Beyond ArcGIS Using PySAL for Spatial Analysis Using PySAL for Spatial Analysis http://geodacenter.asu.edu/pysal PySAL • Python library of spatial analysis methods • ESDA, spatial statistics, geostatistics ESDA, spatial statistics, geostatistics • Growing and expandable Using R for Spatial Analysis Using R for Spatial Analysis • • • • Open source language for data analysis Libraries have been developed for spatial methods Large and active user community Growing and expandable Growing and expandable ArcGIS and R and R Script Tool Script Tool Python script that calls R Python script that calls R Evaluating R Statements Evaluating R Statements Concluding Remarks Concluding Remarks • Python is a relatively easy to learn language • ArcGIS is becoming more “Pythonesque” • Creating time‐savings scripts for repetitive tasks does not take a lot of code tt k l t f d • Easy to share script tools Paul Zandbergen Department of Geography [email protected] db @ d www.paulzandbergen.com p g Workshop Materials Posted Workshop Materials Posted posted until October 24 posted until October 24 h // http://www.paulzandbergen.com/workshops l db / kh