Download Taming the World Bank Data Using SAS

NESUG 2006
Data Manipulation
Data and
Manipulation
Analysis
Taming the World Bank Data Using SAS®
Mike Zdeb, University@Albany School of Public Health, Rensselaer, NY
ABSTRACT
The World Bank web site is a wonderful source of country-specific demographic and economic data. A user can download
data (in the form of an Excel spreadsheet) from multiple countries over multiple year for a variety of variables. Unfortunately,
the downloaded data are not arranged in a format that allows easy use and analysis. A combination of SAS data steps and
SAS PROCs are used to rearrange the data into a SAS data set that is much easier to use than data in the original format.
The SAS code is written in such a way that it makes use of the downloaded data to write a portion of the data steps used in
the job, making the code flexible enough to handle any combination of countries, years, and variables with minimal user
intervention. The SAS code is meant to illustrate the use of various SAS tools for data manipulation and organization done
prior to data use and analysis.
INTRODUCTION
When looking for data for use in either research projects or teaching, two qualities that one looks for are rich content and
good organization. While it is easy to find data with rich content on the web, finding data that is also organized in a fashion
that makes it easy to use is not as common. This is sometimes a bad news/good news situation. The bad news is that poorly
organized data are difficult to use. The good news is that such data
can be used as a teaching tool, showing students that a good
portion of some research projects is often spent organizing data
prior to analysis and how to use a variety of SAS tools for data
organization and management.
The data that are available on the World Bank web site area good
example of rich content. If one goes to the World Bank web site, a
few mouse clicks will lead you to a set of free, WDI (World
Development Indicators)...
http://devdata.worldbank.org/data-query/
and a portion of that web site is shown on the right.
The data are available for 226 countries. After one selects one or
more countries, subsequent web pages allow you to select up to 54
WDIs (for example, data on school enrollment, immunization,
energy use), and data from any or all of the years 2000 through
2005. After making all the necessary selections, the data are
displayed on the screen. At the bottom of the screen, you are
presented with the option of exporting the selected data to either an
ASCII or an Excel file. The Excel file is not a true spread sheet,
rather it is a tab-delimited file that one can open in Excel.
A portion of the tab-delimited file (opened in Excel) is shown below. The default organization of the file is: columns with
country name, ind1-desc (the World Development Indicators, or WDIs), years; rows with entries for each country/WDI
combination. On the web site, one is given the option of changing the screen view from WDIs with a given country to
countries within a given WDI. However, changing the screen display has no effect on the row order of the exported file. It is
always as shown below, WDIs within counties.
-1-
NESUG 2006
Data Manipulation
Data and
Manipulation
Analysis
The content is rich in that there are many development indicators for many countries. If all one wanted to do was to use the
data as a reference file (look up information for given countries or WDIs), the default organization is fine. However, if one
wanted to convert the data to a SAS data set and try to answer a question such as "how is GDP related to other WDIs", the
data are not well organized. A better layout for the data would be a column for each WDI within a country and one row per
year within each country. In SAS data set terminology, we would like to have: observations - one for each country/year
combination; variables - country, year, and one for each WDI.
TAMING (REARRANGING) THE DATA
SAS code was written to convert the tab-delimited data file exported from the World Bank web site into a SAS data. The
various steps taken to read and transform the data are described in the following sections. Though the various sections of
SAS code are specific to the World Bank data, hopefully there are techniques and tips that are useful with other data.
In looking at the spreadsheet on the bottom of page 1, the text that describes each WDI does not qualify as a SAS variable
name, however that text would serve just fine as variable labels. The following steps describe the steps taken to create a
SAS data set that has: one variable for each WDI with simple variable names such x1, x2, x3, etc.; labels for each variable
that are derived from the text in that data file; one observation fore each country/year combination. If PROC CONTENTS is
run using the final data set, the result should look something like the following...
#
1
2
41
19
27
35
.
.
.
37
48
49
56
Variable
country
x1
x2
x3
x4
x5
Type
Char
Num
Num
Num
Num
Num
x52
x53
x54
year
Num
Num
Num
Num
Len
30
8
8
8
8
8
8
8
8
8
Label
COUNTRY
AGRICULTURAL LAND (% OF LAND AREA)
AGRICULTURE, VALUE ADDED (% OF GDP)
BIRTHS ATTENDED BY SKILLED HEALTH STAFF (% OF TOTAL)
CASH SURPLUS/DEFICIT (% OF GDP)
CO2 EMISSIONS (METRIC TONS PER CAPITA)
TIME REQUIRED TO START A BUSINESS (DAYS)
TOTAL DEBT SERVICE (% OF EXPORTS OF GOODS, SERVICES AND INCOME)
WORKERS' REMITTANCES AND COMPENSATION OF EMPLOYEES, RECEIVED (US$)
YEAR
The goal is to create such a data set using just SAS code that does not have any points at which one stops and manually
makes changes to any of the files.
Reading a Tab-Delimited Data File
The first step is to use a data step to read the tab-delimited file.
data temp;
infile "j:\world_bank_2000_2006.xls" firstobs=2 dsd dlm='09'x;
input
country
: $upcase30.
í
var
: $upcase200.
(x2000-x2005) (??)
î
;
run;
ì
An INFILE statement ì is used to accomplish a number of tasks: specify the location of the file; skip the first row (column
headers using the FIRSTOBS option; indicate that two consecutive delimiters indicate missing data using the DSD option;
specify that the delimiter is a tab using the DLM option with a hex character value of 09. List input is used in the INPUT
statement with informats and colon format modifiers. All character data are converted to uppercase with UPCASE í
informats. In the data file, missing data are represented by two consecutive periods, not the single period that SAS
recognizes as representing missing data. The "??" î format modifier is used to suppress the invalid data and error
messages written to SAS LOG when reading the two consecutive periods.
Once this data step is run, the SAS LOG shows...
NOTE: The infile "j:\world_bank_2000_2005.xls" is:
File Name=j:\world_bank_2000_2005.xls,
RECFM=V,LRECL=256
NOTE: 10036 records were read from the infile "j:\world_bank_2000_2005.xls".
The minimum record length was 41.
The maximum record length was 170.
NOTE: The data set WORK.TEMP has 10036 observations and 8 variables.
-2-
NESUG 2006
Data Manipulation
Data and
Manipulation
Analysis
Remember that there were data for 226 countries and 54 WDIs downloaded from the World Bank web site. That should
produce a data set with 12,204 observations. There are only 10,036 observations in the data set and that occurs since each
WDI is not available in each country. Also notice that there are no invalid data or error messages (remember the "??" format
modifier).
Determine the Unique WDIs in the SAS Data Set
Once the data are in a SAS data set, the real work of rearranging the data can begin. The next step is store the values of the
variable VAR (the names of the WDIs, column two in the spreadsheet at the bottom of page 1) in a SAS data set. This is
done using PROC FREQ...
proc freq data=temp;
table var / noprint out=tvar (keep=var); ì
run;
A SAS data set named TVAR is created using a TABLE statement in PROC FREQ ì. The creation of a table in the output
window is suppressed using a NOPRINT option. When printed, a portion of that data set looks as follows...
Obs
1
2
3
4
5
.
.
.
50
51
52
53
54
var
AGRICULTURAL LAND (% OF LAND AREA)
AGRICULTURE, VALUE ADDED (% OF GDP)
BIRTHS ATTENDED BY SKILLED HEALTH STAFF (% OF TOTAL)
CASH SURPLUS/DEFICIT (% OF GDP)
CO2 EMISSIONS (METRIC TONS PER CAPITA)
SERVICES, ETC., VALUE ADDED (% OF GDP)
SURFACE AREA (SQ. KM)
TIME REQUIRED TO START A BUSINESS (DAYS)
TOTAL DEBT SERVICE (% OF EXPORTS OF GOODS, SERVICES AND INCOME)
WORKERS' REMITTANCES AND COMPENSATION OF EMPLOYEES, RECEIVED (US$)
Create a Format from the WDI Text Strings
Though the reason might not be obvious at this point in the process, the next step is to use each of the text strings in the
above as part of a format. Each text string will serve as a format value and the position of the text string in the data file (you
can think of that as the observation number shown above) will be a format label. A data step is used to create a CNTLIN data
set (named TVAR, replacing the data set created with PROC FREQ) and a macro variable named &NVARS that contains the
number of unique WDIs (there are 54) in the data set.
data tvar;
retain fmtname "$var2num"; ì
set tvar (rename=(var=start)) end=last;
label = left(put(_n_,2.)); î
if last then call symputx('nvars',_n_);
run;
í
ï
The format will be named $VAR2NUM ì (that name indicates that the format will be used to convert a text string to a variable
number). As the text strings are read, they are stored in the variable START (the text that appears to the left of the = sign
when one creates a format). The text that normally appears to the right of the = sign in a format, the variable named LABEL
in the CNTLIN data set, is provided by the data step iteration counter _N_ î. After the last observation is read, the number of
observations (also the number of unique WDIs in the original data and the number of variables of the form Xn that are to be n
the final data set) is stored in a macro variable ï.
The format is created using PROC FORMAT...
proc format cntlin=tvar;
select $var2num; ì
run;
Since a SELECT statement is used in PROC FORMAT ì, a
description of the format appears in the output window and a
portion of that output is shown on the right. Each WDI text
string is now associated with a variable number.
-3-
NESUG 2006
Data Manipulation
Data and
Manipulation
Analysis
Create a LABEL Statement from the WDI Text and Variable Numbers
All the text needed to create a LABEL statement is in the data set TVAR. Again, the reason might not be obvious at this
point, but PROC SQL is used to create a macro variable (&LABEL) containing most of the text needed in a LABEL statement
that associates a WDI text string with each of the X1 through X54 variables in the final set.
proc sql noprint;
select catt(' x',label,'="',start,'"') ì
into :label í
separated by ' ' î
from tvar; ï
quit;
The CATT function is used to combine: the letter "x" with the value of the variable label (1 through 54); an = sign; the value
of the variable start (the text that describes each WDI) placed in quotes ì. All that text is placed in the macro variable
&LABEL í. The text strings are separated by spaces î. The values for all the variables used in PROC SQL are found in the
data set TVAR ï.
A portion of the macro variable &LABEL looks as follows...
x1="AGRICULTURAL LAND (% OF LAND AREA)" x2="AGRICULTURE, VALUE ADDED (% OF GDP)" x3="BIRTHS
ATTENDED BY SKILLED HEALTH STAFF (% OF TOTAL)" x4="CASH SURPLUS/DEFICIT (% OF GDP)" x5="CO2
EMISSIONS (METRIC TONS PER CAPITA)" x6="ELECTRIC POWER CONSUMPTION (KWH PER CAPITA)"
.
.
.
x50="SERVICES, ETC., VALUE ADDED (% OF GDP)" x51="SURFACE AREA (SQ. KM)" x52="TIME REQUIRED
TO START A BUSINESS (DAYS)" x53="TOTAL DEBT SERVICE (% OF EXPORTS OF GOODS, SERVICES AND
INCOME)" x54="WORKERS' REMITTANCES AND COMPENSATION OF EMPLOYEES, RECEIVED (US$)"
Transpose the Data Set
The next step will show why the format was created earlier. That format can be used to convert the WDI text string in each
observation to a variable number as follows...
data temp;
set temp;
varnum = put(var,$var2num.);
run;
ì
The original SAS data set is read with SET statement and the format $VAR2NUM is used in a PUT statement to create a new
variable named VARNUM ì. The reason for adding this variable to the SAS data set is shown in the following SAS code...
proc sort data=temp;
by country; ì
run;
proc transpose data=temp out=world_bank prefix=x; í
var x2000-x2006; î
id varnum; ï
by country; ð
run;
The data are sorted in order by the variable COUNTRY (though this step may not be necessary given that the data exported
from the World Bank web site appear to be in country order) ì. PROC TRANSPOSE is used to create a new data set named
WORLD_BANK from data set TEMP í. Most of the new variables in data set WORLD_BANK will have prefix "x" (variables
X1 through X54). The data within each year is transposed and will now appear in across rather than within observations î.
The newly created variable VARNUM is used in a ID statement and provides the last part of the variable names X1 through
X54 ï. Finally, the variable COUNTRY is used to determine how observations are filled with variables ð.
-4-
NESUG 2006
Data Manipulation
Data and
Manipulation
Analysis
A portion of the new WORLD_BANK data set looks as follows...
Obs
223
224
225
226
227
228
1279
1280
1281
1282
1283
1284
country
CHAD
CHAD
CHAD
CHAD
CHAD
CHAD
UNITED
UNITED
UNITED
UNITED
UNITED
UNITED
STATES
STATES
STATES
STATES
STATES
STATES
_NAME_
x2000
x2001
x2002
x2003
x2004
x2005
x2000
x2001
x2002
x2003
x2004
x2005
x1
38.56
38.62
38.62
38.62
.
.
44.99
44.81
44.71
44.69
.
.
x2
39.20
37.97
38.73
45.58
.
.
1.23
1.18
1.03
1.19
.
.
x3
16.3
.
.
.
14.4
.
.
.
.
.
.
.
This data set is very close to the final version. Only a few changes are needed.
Add Variable Labels and Do Some _NAME_ Changing
The data set needs a variable named YEAR and the variable _NAME_ contains values that can be used to produce that new
variable, the last four characters of each value. The data set also needs labels for each variable since it is difficult to
remember what WDIs variables X1 through X54 represent. Now it is time to use the macro variable &LABEL created earlier.
data world_bank;
set world_bank;
year = input(substr(_name_,2,4),4.); ì
label &label year='YEAR' country='COUNTRY';
drop _name_; î
run;
í
A SUBSTR function is used to extract the value for the variable YEAR from the variable _NAME_ ì. An INPUT function
makes YEAR a numeric variable. A LABEL statement adds a label to each variable and most of the text for the LABEL
statement is provided by a macro variable í. The variable _NAME_ is no longer need and is dropped.
PROC CONTENTS run using the final WORLD_BANK data set now shows...
#
1
2
41
19
27
35
.
.
.
47
36
37
48
49
56
Variable
country
x1
x2
x3
x4
x5
Type
Char
Num
Num
Num
Num
Num
x50
x51
x52
x53
x54
year
Num
Num
Num
Num
Num
Num
Len
30
8
8
8
8
8
8
8
8
8
8
8
Label
COUNTRY
AGRICULTURAL LAND (% OF LAND AREA)
AGRICULTURE, VALUE ADDED (% OF GDP)
BIRTHS ATTENDED BY SKILLED HEALTH STAFF (% OF TOTAL)
CASH SURPLUS/DEFICIT (% OF GDP)
CO2 EMISSIONS (METRIC TONS PER CAPITA)
SERVICES, ETC., VALUE ADDED (% OF GDP)
SURFACE AREA (SQ. KM)
TIME REQUIRED TO START A BUSINESS (DAYS)
TOTAL DEBT SERVICE (% OF EXPORTS OF GOODS, SERVICES AND INCOME)
WORKERS' REMITTANCES AND COMPENSATION OF EMPLOYEES, RECEIVED (US$)
YEAR
EXAMINING THE "TAMED" DATA
Now that the data are reorganized, it is easy to look at how much data are available for each WDI. For example, if one
wanted to see how much data are present in the year 2000, PROC MEANS can be used to count the number of missing and
non-missing values...
proc means data=world_bank n nmiss;
var x1-x&nvars; í
where year eq 2000; î
run;
ì
-5-
NESUG 2006
Data Manipulation
Data and
Manipulation
Analysis
PROC MEANS is asked only to count the number of non-missing and missing variable values ì. One does not even have to
know how many WDIs there are in the data set since that number was placed in the macro variable &NVARS earlier in the
SAS code í. Only data from the year 2000 are analyzed î. A portion of the output looks as follows...
Variable
x1
x2
x3
x4
x5
.
.
.
x51
x52
x53
x54
Label
AGRICULTURAL LAND (% OF LAND AREA)
AGRICULTURE, VALUE ADDED (% OF GDP)
BIRTHS ATTENDED BY SKILLED HEALTH STAFF (% OF TOTAL)
CASH SURPLUS/DEFICIT (% OF GDP)
CO2 EMISSIONS (METRIC TONS PER CAPITA)
N
215
192
72
79
206
Miss
11
34
154
147
20
SURFACE AREA (SQ. KM)
TIME REQUIRED TO START A BUSINESS (DAYS)
TOTAL DEBT SERVICE (% OF EXPORTS OF GOODS, SERVICES AND INCOME)
WORKERS' REMITTANCES AND COMPENSATION OF EMPLOYEES, RECEIVED (US$)
216
0
123
170
10
226
103
56
CONCLUSION
There are many sources of data on the web. Though much of the data are organized in a manner that makes it difficult to use
SAS analysis tools such as the various SAS PROCs, there are many tools available within SAS that allow you organize and
manage data. Some of the tools used to rearrange the World Bank data included CNTLIN data sets, macro variables, PROC
SQL, and PROC TRANSPOSE. The wealth of tools makes the limiting factor on organizing data your own resourcefulness
and creativity.
ACKNOWLEDGMENTS
SAS and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of SAS Institute Inc. in
the USA and other countries. ® indicates USA registration. Other brand and product names are registered trademarks or
trademarks of their respective companies.
CONTACT INFORMATION
The author can be contacted using e-mail... [email protected]
-6-