Download The tale of two trainings… - Baltimore Workforce Development Board

Baltimore’s Workforce System at Work
Training Investment Analyses
________________________________________________________________________________________________
2.8 Results of the Training Investment Study
In this section we focus on the question “what are we getting for our public investment in
training?” We review the strategies available, and the results of studies measuring the dollar
value. We then give the results of our own analyses on the customized training and ITA-funded
training given by the local public workforce investment system to 216 individuals in Baltimore.
We present results to both a cost-effectiveness analysis, and a return on investment analysis. We
conclude this section with some recommendations for future action, based on these findings.
Why is this important? There is a need to determine the costs and benefits of
public investments in workforce development in a systematic fashion, so that
comparisons can be made within the same program over time and between
different investment options. Cost-effectiveness approaches and training investment analyses can demonstrate the value of workforce development efforts in
terms that the business and legislative communities understand.
2.8.1 The value of training: what do we know, and how do we know it?
Training magazine has estimated that American businesses budget almost $60 billion a year for
formal training.1 The American Society for Training and Development (ASTD) calculates each
year how that total
amount spent emerges
How much training do U.S. firms do?
in terms of several “key
ratios of training” (see
Total training expenditures per training-eligible employee:
$761
Total training expenditures as % of payroll:
1.9%
sidebar).
Percent of training-eligible employees trained:
78.5%
Training-eligible employees to trainer ratio:
367
In 2001, U.S. firms
Percent of training time via classroom:
77.1%
provided almost 24
Percent of training time via learning technologies:
10.5%
hours of training to each
Payments to outside companies as % of total training expenditures: 20.9%
employee, and spent
Total training hours per training-eligible employee:
23.7
about $761 per
Source: ASTD, 2002
employee, which is
equivalent to about 2%
of their total payroll. Yet for all this activity and spending, there are only a handful of studies that
have attempted to establish the generic “value of training” and then to put a specific dollar value
on it.
Studies of the value of training:
One of the earliest studies of publicly-funded training was by Holzer et al,2 who explored the
effects of the Michigan Job Opportunity Bank Upgrade program. This was a state-financed
training program for manufacturing firms that between 1986 and 1990 administered over 400
grants averaging $16,000 per firm. Using data from both successful and unsuccessful applicants
1
“1997 Industry Report”, Training magazine, October.
2
“Are training subsidies for firms effective? The Michigan Experience”, by Holzer H, Block R, Cheatham M, and
Knott J, Industrial and Labor Relations Review, v46(4), July 1993, pp625-635.
1
Baltimore’s Workforce System at Work
Training Investment Analyses
________________________________________________________________________________________________
to the program, the authors were able to look at total hours of training in the firms and the product
scrap rate. They found receipt of training grants was associated with a large and significant – but
one-time – increase in training hours, and also with a more lasting reduction in scrap rates.
Specifically, they calculated every additional hour of training that was stimulated cost roughly $6
to $7 of government funds.
Bartel looked beyond these specific indices to consider the impact of training programs on labor
productivity as a whole.3 She looked at personnel policies and economic characteristics of 595
manufacturers from a 1986 Columbia Business School survey, to measure the impact of formal
training programs on labor productivity. She found that businesses operating below their expected
labor productivity levels in 1983 but which also implemented new employee training programs
after that date, realized significant increases in their rate of productivity growth between 1983 and
1986. This increase was enough to
bring those businesses up to the
Employers! Here’s what these research studies have
labor productivity levels of
shown training can do for you:
comparable organizations.
 reduce scrap rates
 increase productivity and the rate of productivity growth
Black and Lynch attempted to link
 complement your other investments in new physical capital
human capital investments and
 increase your total shareholder return
productivity, using data from 1990
 increase your profit margins
and 1993 on over 2,900
 increase company income per employee
establishments from the National
 increase price-to-book ratios.
Center on the Educational Quality
of the Workforce’s “National
Employers Survey”.4 They found that the average educational level within the establishment has
significant positive effects within both manufacturing and non-manufacturing sectors, and they
estimated that a 10% increase in “average educational level” would lead to 8.5%-12.7% increases
in productivity. Their other findings were that past training increases current productivity, that
off-the-job training has a greater impact than other types, that computer-skills development has a
positive impact regardless of industry, and that employers who focus on grade levels when hiring
also experience significantly higher productivity than competitors.
Lynch and Black next used data from a 1994 survey to examine how the incidence, content, and
extent of employer-provided training, were linked to workplace practices and characteristics and
physical capital investments.5 They found substantial variation by employer size and industry in
the provision of formal training, and that the percentage of workers given training was highest in
establishments that had made large investments in physical capital, or had adopted new forms of
work organization. They suggested their findings show employer-provided training complements,
rather than substitutes for, investments in physical capital and education.
3
“Productivity Gains from the Implementation of Employee Training Programs”, by Bartel A, Industrial
Relations, v33(4), October 1994, pp411-425.
4
“Human Capital Investments and Productivity”, by Black S and Lynch L, AEA Papers and Proceedings, v86(2),
May 1996, pp263-267.
5
“Beyond the Incidence of Employer-Provided Training”, by Lynch L and Black S, Industrial and Labor
Relations Review, v52(1), Oct. 1998, pp64-79.
2
Baltimore’s Workforce System at Work
Training Investment Analyses
________________________________________________________________________________________________
Bassi et al 6 attempted to answer the question “do firms’ investment in education and training
payoff?”. Using 1996-98 data on 575 publicly-traded U.S. firms in the ASTD Benchmarking
Service database, they found that training investments are significantly related to later total
shareholder return (TSR). Specifically, an increase of $680 of training expenditures per employee
was associated with, on average, a 6% improvement in total shareholder return the following
year, even after controlling for other factors like industry type, company size, and prior financial
performance. Firms with above median training investments for the group had a TSR that was
86% higher than firms below, and 45% higher than the stock market average. Firms in the top
quartile also enjoyed significantly higher profit margins, income per employee, and price-to-book
ratios, than the firms in the bottom quartile. The authors suggest these results can be used inside
firms to justify training compared to other potential investments, and outside of firms as a way for
stock market analysts to predict share values.
How do organizations evaluate training?
Human resources, organizational development, and “workplace learning for performance”
professionals, commonly draw on the “Kirkpatrick model” when structuring their evaluations of
training.7 This scheme has four “levels”, and the share of all organizations in the ASTD
Benchmarking Service database8 attempting each different level in 2001 is shown in the bar chart
below.
KIRKPATRICK LEVEL
Use of evaluation strategies for training
"Reaction" (level 1)
91%
"Learning" (level 2)
36%
"Behavior" (level 3)
17%
"Results" (level 4)
9%
0%
20%
40%
60%
80%
100%
Share of organizations attempting
Source: ASTD, 2002
Level 1 is the “Reaction” level, measuring the learner customer’s satisfaction with the course,
content, delivery method, and instructor. This is the easiest method of evaluation, requiring only a
simple survey form administered at completion, and so is undertaken in over 90% of
organizations. However, it is also in some ways the least informative method.
6
7
8
Profiting from learning: do firms’ investment in education and training payoff? Research White Paper by Lauri
Bassi, Jens Ludwig, Dan McMurrer, and Mark Van Buren, for ASTD and SABA, Alexandria, VA, Sep. 2000.
Kirkpatrick D (1998) Evaluating Training Programs, 2nd edition, Berrett-Koehler Publishers, San Francisco.
In this total, 88% are private corporations and 12% from the public sector.
3
Baltimore’s Workforce System at Work
Training Investment Analyses
________________________________________________________________________________________________
The second level is the "Learning" level, focusing on trainee mastery and retention of content.
This usually requires a final exam or rating of the trainee by the supervisor, and is used by just
over one-third of all organizations.
The third level is the "Behavior" level, which focuses on whether the trainee did anything
different or performed better, as a result of the training. This method has seen a surge in the
number of organizations using it in recent years, because of the shift away from passive
acceptance of exam results and towards a more dynamic “demonstration of competencies”
approach, and because of the growing availability of process-monitoring technologies at the
individual work-station level. Level 3 evaluation is now used by about 17% of organizations.
The fourth and final level in Kirkpatrick’s original scheme is the “Results” level. This focuses on
the “bottom-line” implications of training: productivity increases, the value of additional units
made, or sales achieved, the reductions in costs through lower error and scrap rates, reduced staff
turnover, increased customer satisfaction, repeat customer business, and so on. This fourth level
produces the kind of hard business information most often sought by managers, but it also
requires a more sophisticated study design, more detailed measurement and tracking of activities
and costs, and more time and expertise to complete, than do the other three levels. As a result,
Level 4 evaluations are usually only undertaken for large volume/high cost/high visibility training
initiatives, and so appear in under 10% of all organizations.
More recently, Phillips has supplemented this basic four-level Kirkpatrick framework with a new
“Level 5”, which he calls “return on investment”. This focuses on comparison of specifically the
financial costs and benefits of training, using
the formula shown in the sidebar.
How is ROI calculated?
The ROI calculation itself is but is one piece
ROI is “earnings divided by investment”, and can be
of a multi-step process (see sidebar).9 First
considered a variant of traditional cost–benefit
comes planning the ROI study, including
where:
specification of its objectives, scope,
Benefit:cost ratio =
Program Benefits
methodology, and the data to be collected.
---------------------This step involves identifying appropriate
Program Costs
measures, locating data sources, developing
collection instruments, and specifying
Phillips’s ROI process compares net program
timeframes and responsibilities. Next, the
benefits and costs:
data collection needs to be operationalized.
Net Program Benefits
Baseline and post-training data can be
ROI
(%)
=
----------------------------x 100
collected using surveys, questionnaires,
Program
Costs
observation, interviews, focus groups,
assignments, action plans, performance
(where “net program benefits” are “program
contracts, or performance monitoring. The
benefits” minus “program costs”)
effects of the training itself then need to be
isolated from the potential contributions of other factors, using, for example, experimental control
group strategies, trend-line analysis, forecasting models, or professional estimation. The data
then need to be converted to monetary values, using strategies like translating output data into
profit contributions, historical cost comparisons, or management estimates. The resulting
monetary values are then compared to program costs, including the costs of design and
development, participant program materials, and cost of instructors, facilities, expenses and
administrative overhead. The final step is to identify the intangible, non-monetary benefits, such
9
Phillips J (1967) Return on Investment, Gulf Publishing/Butterworth-Heinemann, MA
4
Baltimore’s Workforce System at Work
Training Investment Analyses
________________________________________________________________________________________________
as increased job satisfaction, improved organizational commitment, improved teamwork,
improved customer service, and reduced complaints and conflicts. These are converted to
monetary value where possible and
credible, recognizing that some costs
The Phillips multi-step approach to ROI
and benefits may have to be left outside
1. Plan the evaluation.
the ROI calculation as “intangibles”.
2. Collect the baseline, treatment, and post-event data.
3. Isolate the effects of the training.
4. Convert the hard and soft data to monetary value.
5. Tabulate program costs.
6. Calculate the ROI.
7. Identify the remaining intangible benefits.
The Phillips “ROI Process” has grown
in popularity and use recently, as the old
justification for training being inherently
“a ‘good’ thing for its own sake” has
given way to the modern performance
perspective, wherein corporations have to justify every additional dollar and staff position for
training in terms of its return on investment. Phillips’s “ROI Institute” has developed protocols, a
curriculum and certifications, and has built a critical mass of practitioners who now use Phillips’s
process, steps, and operational strategies. This movement has thus brought some consistency to
approaches and some comparability to study outcomes.
Applying this approach directly to training done by the public sector, however, runs into some
additional problems noted by Phillips.10 These include the general absence of true revenues and
profits for government activity, the relative paucity of “hard” data, the need to serve multiple
constituencies when arriving at and communicating results, the restricted range of options open to
managers for acting on any ROI findings, and the belief that government services are “essential”
public goods that may not ever be discontinued even if they show poor financial returns.
While these issues may indeed make the ROI task harder, they are more issues of discipline and
appetite for adaptation. We suggest there is a far more serious structural issue with attempting
ROI in the public sector: namely, one of “scope”. The ROI processes developed for private
sector training deal with costs and benefits internalized to the enterprise. This situation gives
natural boundaries for deciding which costs and benefits should go into the calculation. Public
sector activity, in contrast, owes its very existence to the fact of externalized costs and benefits.
These by definition preclude private sector actors from capturing all the benefits and remove
incentive for private provision, leaving the public sector to step in and raise taxes to fund the
activity. In the public sector, therefore, the choice of which -- and whose -- costs and benefits to
include in any ROI calculation is a very much more open one.
We suggest this is a serious enough difference to go beyond the present process, and conceive of
a new “Level 6” type of evaluation, preserving some of the relevant operational steps of Level 5,
but dealing with activities whose impacts are both individual and societal, public and private,
internal and external. For the local public workforce system context, these issues are manifested
in the costs being borne by multiple levels of government (and taxpayers) and the benefits
accruing to both the individual client (through hopefully higher individual income received) and
multiple levels of government – and ultimately the taxpayer -- through the client being in a
different tax and welfare situation.
10
Phillips J and Phillips P (eds, 2002) Measuring ROI in the Public Sector, ASTD Press ‘In Action’ series,
Alexandria, VA.
5
Baltimore’s Workforce System at Work
Training Investment Analyses
________________________________________________________________________________________________
We attempt below the more comprehensive approach needed to account for all these costs and
benefits through a pilot study of two different types of training offered by the local public
workforce system in Baltimore: “customized training”, and “ITA-funded training”. The main
differences between these two types of training are summarized in the sidebar. The key difference
is in the presence of an employer for customized training, versus the individual paths taken by
clients using ITAs to “purchase” their training “retail”.
The tale of two trainings…
FEATURE
CUSTOMIZED TRAINING
ITA-FUNDED TRAINING
Route:
Location:
Content:
Curriculum:
Duration:
Cooperative venture with employer
Can be workplace or institution
Developed for particular needs of employer
Usually modular, tailored
Tailored to needs of employer:
can be short (days or weeks)
WIA with 50% employer match
Usually guaranteed, if completed
Individual path
Usually institution
Developed by training institution
Usually fixed-length course in structured sequence
Usually multiples of 10, 12 or 15 weeks in
quarters or semesters
WIA or other state and local funds
No direct link to employment opportunity.
Funding:
Employment:
We obtained data on 216 clients who started WIA-funded training activities and exited the
program at various dates between May 2000 and June 2002. The “customized training” option
was followed by 101 clients (47% of the total), and the “Individual Training Account (ITA)
funded training” by 115 clients (53%).
We investigate what are the differences between the two groups, in terms of their initial
demographics, the training intervention (cost and time), and their job and earnings outcomes. We
look particularly at their earnings change from before the training to after exit from the program,
and compare this to the cost of training, as a measure of the “cost-effectiveness” of these two
types. Finally, we undertake a more comprehensive ROI analysis. (Details of data and sources,
and detailed findings, are given in Appendix D).
Cost-effectiveness of the different training types:
Analysis of the demographics of the two training groups showed that the customized group had a
higher share of females, of people with families, and of employed people. It was also slightly
younger, with a lower median wage, and a higher share receiving food stamps, prior to training.
The ITA group had a higher share of dislocated workers, of unemployed clients, and of exoffenders, and also a higher median wage, with a lower share receiving food stamps, prior to the
start of training.
The median cost of training per client was half as much for the customized group as for the ITA
group, and the time spent between start of training and exit was shorter for the customized group
than for the ITA group.
The median hourly wage after training for those with wages was roughly the same for both
groups, but for the full 4 quarters following exit the customized group had a higher median wage
than the ITA group. The customized group also had a higher median absolute wage gain in the 4
quarters post-exit compared to the 4 quarters before training. There is no significant correlation
between the dollar cost of the individual’s training and that individual’s wage gain after training.
6
Baltimore’s Workforce System at Work
Training Investment Analyses
________________________________________________________________________________________________
About 1 in 4 trainees exhibited a wage decline after training compared to their situation before:
this number is higher for the ITA group than for the customized group.
Our chosen measure of cost-effectiveness for these two training groups is “aggregate dollars of
client wage gain per $1 of aggregate training cost” (not including employer contributions and
MOED staff time). The recommendations that follow assume this indicator is something we are
trying to maximize: there may be good policy, strategic, or equity reasons for over-riding this
consideration in individual cases. Nevertheless, on this variable alone, customized training is 2.3
times more effective (at $3.55) than ITA-funded training ($1.49), which suggests customized
training is a better route for scarce training dollars (see sidebar below).
Aggregate wage change (4 qtrs pre-training and post-exit) per $1
of aggregate training cost
$ of wage change per $ of
training cost
$4.00
$3.55
$3.50
$3.00
$2.20
$2.50
$2.00
$1.49
$1.50
$1.00
$0.50
$0.00
CUSTOMIZED
ITA-funded
BOTH TYPES
Type of training
This large difference in cost-effectiveness between the two types of training may not all be
attributable to the training type itself, though, because:
(i) There is no statistically significant correlation between the cost of training and the wages in
the 4 quarters after exit, or between the cost of training and the wage change between preand post- periods, for either training type considered alone or for both.
(ii) There is a significant correlation between wages in the 4 quarters before training and in the
4 quarters after, and between wages before training and the value of wages gain. These are
statistically significant for customized training, for ITA-funded training, and for both
groups together.
(iii) There are also differences – and in some cases larger differences than the $3.55 and $1.49
found above – between different client groups: for example, between males ($1.62) and
females ($2.60), and between those in the lowest quartile of the wage distribution before
training ($4.90) versus those in the highest quartile (-$0.97). To the extent that these
7
Baltimore’s Workforce System at Work
Training Investment Analyses
________________________________________________________________________________________________
different client groups are also disproportionately distributed between the two types of
training, then this will account for some of the observed difference between training types,
on this variable.
This finding of lack of correlations does not necessarily mean that training is “unimportant”, “not
effective”, or being “poorly done”, or that we cannot learn anything from the data on training type
itself. It could just mean, for example, that the price of training in general in society is not itself
related to training’s potential value for wage outcome, but perhaps more to the type of institution
through which it is delivered. Some “cheaper” training institutions may be performing better than
the more expensive ones, in terms of getting their customers into higher paying jobs. Some types
of client may have to go into one type of training versus the other.
What it does mean is that, in the
“pre-condition” --> “training intervention” --> “post-condition”
sequence, in order to understand the value of training we need to give as much thought and
emphasis to the “pre-” step (assessment and selection of participants going in), and to the “post-”
step (wage, longevity/retention of employment, likelihood of wage gain over time once in the
job), as to the training type itself.
Attention also has to be given to differences within the third, “post-” stage: median hourly wage
at placement (a single point in time just after training) does not differ between training types, but
entire first year wages post-exit do, with the customized group doing better than the ITA group.
This suggests examining the nature of placements obtained through customized training, and
focusing on maximizing those placements with longer job retention over time, and with good
wage trajectories over time, in order to increase effectiveness. This broadens our focus from our
analytic starting point of just “comparing two training types”.
Within the training choice, there are still some interesting findings suggesting what can be
achieved through each type, because cost-effectiveness is found to differ between training type
for the same client group. For example:
 dislocated workers in both training groups have a higher median wage than adults before
training; this wage difference persists afterwards for dislocated workers in the ITA group, but
in the customized group the adults catch up through training, with a median wage gain almost
double that of the dislocated workers;
 females in the customized group have a higher median wage than males both before and after
training, and females also have a higher median wage gain than males, but within the ITA
group, this situation is reversed and males do better than females; this situation could result
from females tending to be represented in different kinds of jobs with different wage levels
than males;
 high school grads have the biggest median wage gain of the three educational status groups
(dropouts, HS grads, and college grads);
 all but the highest quartile of wages achieve higher incomes through both types of training,
but the poorest group of clients going in (those in the lowest quartile of the pre-training wage
distribution), achieves a higher median wage after training, and a higher median gain in
wages, through the customized route than through the ITA route;
 the client group with the highest gain in median wage, out of 11 groups (with some overlap of
individuals across groups) is that with wages in the lowest quartile prior to training – i.e. the
poorest group to enter the system: the median wage gain is over 70% that of the second
8
Baltimore’s Workforce System at Work
Training Investment Analyses
________________________________________________________________________________________________
highest wage gain group of “all adults”, and also the next, the second lowest quartile of the
wage distribution prior to training;
 the client group with the lowest median wage gain, out of 11 groups (again with some overlap
of individuals across groups) is that with wages in the highest quartile prior to training: their
median wage change is actually negative (-$1,064), and almost half of the 1 in 4 trainees who
experience a lower income after training than before, are in this group;
 the “highest yield” client group, in terms of highest wage gain per $1 of training cost, out of
11 groups (again, with some overlap of individuals across groups) is the client group with
wages in the lowest quartile prior to training: these clients earn afterwards $4.90 more per $1
spent on their training than they did before; this group’s figure is also the highest of any client
group’s in either type of training, reaching $7.45 more for customized and $3.28 more with
ITAs;
 the “lowest yield” client group, in terms of having the lowest wage gain per $1 of training
cost, out of 11 groups (again, with some overlap of individuals across groups) is the client
group with wages in the highest quartile prior to training: they earn afterwards 97 cents less
per $1 spent on their training than before, and this figure is also negative for this group in
each type of training (-$1.53 for customized, and -$0.75 for ITA).
This all broadly confirms, and supports with real numbers, that training is associated with wage
gains for almost all groups of clients, and that the gain effect is highest for those who previously
were the poorest.
Recommendations based on cost-effectiveness findings:
The summary of strategic choices that might increase this training effectiveness even further, as
suggested by these findings, are:
1. Work with employers to develop more customized training opportunities in which to train
clients.
2. Save ITAs for: (a) dislocated workers, because they are the only client group whose wage
gain for each training dollar is higher through the ITA type than through customized
training; (b) those in the lowest quartile of the wage distribution coming into the training
stage, because they this group is twice as cost-effective as other groups in that training
type.
3. Pick less expensive training providers rather than more expensive ones, for the same kind
of training. This choice would not itself hurt client wage outcomes on average, but it
would improve cost-effectiveness because it would save money for the same wage
outcome.
4. If the workforce system has to “ration” training funds and select participants from a
larger group of applicants, and if there is a choice of trainees, then, in general, costeffectiveness (wage gain per $1 of training cost) could be maximized through:
 selecting those who are poorest (i.e. in the lowest quartile) before training;
 avoiding those who are already the highest (i.e. upper quartile) earners: use
additional non-training services instead, to help place them in their next job; if
9
Baltimore’s Workforce System at Work
Training Investment Analyses
________________________________________________________________________________________________
they have to be trained, the ITA route would be more cost-effective than the
customized route;
 picking high school grads over college grads, to receive training;
 routing females, school drop-outs, and those from the lowest income quartile, into
customized training, where their wage gain per $1 of training cost is much higher
than for these same groups in the ITA type;
5. As the above recommendations are based on findings about group medians, recognize
that individuals differ and that these recommendations cannot be blindly applied
prescriptively to everyone, solely on the basis of their group’s characteristics. The few
groupings with data here are demographic and do not fully capture all information about
an individual’s potential. Be prepared to assess individuals for their motivation to
succeed beyond their group medians.
6. Given the low incidence of welfare receipts in these trainees’ records for the 4 quarters
before and after training, combined with the intermittent appearance of benefits in
earlier quarters before the prior year, thought should be given to better coordinating the
timing of income supports with training, so that training progress and success are better
leveraged.
Return on investment (ROI):
The cost-effectiveness findings above show the impact on wages for each $1 spent on training.
This is not an ROI measure, as it shows only the individual’s wage gain in relation to the public’s
cost. A full ROI requires incorporating the other elements, such as taxes paid and welfare
receipts, for both before and after training, and then making some judgment about the persistence
of training benefits over time. Thus we began with the wage record data and the family size and
structure information, and then used TurboTax™ to estimate the following components of a
client’s tax return:
Federal:
 AGI
 Filing status
 Standard deductions
 Exemptions
 Taxable income
 Total tax liability prior to credits
 Child tax credit
 Earned income tax credit
 Total federal income tax paid/refunded
State and local:
 Maryland taxable income
 Maryland income tax liability prior to credits
 Maryland earned income credit
 Maryland poverty level credit
 Local income tax liability prior to credit
 Local earned income tax credit
10
Baltimore’s Workforce System at Work
Training Investment Analyses
________________________________________________________________________________________________



Local poverty level credit
Local refund of earned income credit
State and local income tax paid/refunded.
We then aggregated the value of these components for all 216 trainee clients in both the tax year
prior to start of training activity, and in the tax year after exit from the program, and calculated
the pre/post differences. These are shown in Chart 2.8.2. Clearly visible there are the spikes in
income, and then in taxable income, after training compared to before. What is perhaps surprising
for this population of clients is just how much the values of the income components overshadow
the values of the credits (EITC and CTC at the federal level, and PLC and EITC at the state and
local levels).
The relevant tax components were then selected and plugged in as the “program benefits” portion
of the ROI calculation formula, alongside the “program costs”, in Table 2.8.3. This table shows
how they were used to calculate the ROI for each type of training. The main result is that neither
of the training types yields a positive return on investment. The customized training ROI is minus
9.6%, and the ITA-funded training is minus 57.9%. Combined, both types have an ROI of minus
41.2%. This contrasts sharply with the high earnings impact of training investment found in the
cost-effectiveness analysis above, where each $1 of training investment yielded over double its
value ($2.20) in wage gain. This difference is because the prime weight in the ROI calculation is
given not to the individual’s wage gain as it was in the cost-effectiveness calculation, but instead
to the increases in taxes paid and the reductions in benefits received, as a result of that wage gain.
For just the one year after exit, the combined additional taxes paid and reductions in benefits do
not themselves outweigh the aggregate cost of investment in the training. Even for the customized
group, training costs of over $237,000 still outweighed the value of tax increases and benefit
reduction value at under $215,000.
These negative ROIs have resulted when including just the one year before and one year after
training. Yet clients will likely have their job for more than one year, and thus later years’ costs
and benefits ought to be included in the calculation. Chart 2.8.4 shows the cumulative ROI for
one through ten years post-exit. These calculations assume the client retains this job, or another
with equal earnings, for the years in question, and that retention is a function of training. The
question is how many years to look at.
Different researchers have used varying lengths of time as the period it would be realistic to take
credit for training: MDRC’s evaluation of JTPA training used five years, and Benson’s analysis
of job training in Ohio used eight years, for example. Both these time-spans seem overly
generous, given the increased churning of the economy since they did their work, and the
common assertion that in the knowledge economy the value of what workers know erodes at 20%
a year. An alternative logic would be to say that training earned clients entry to their job, and the
competent use of their training once in the job allowed them to perform their tasks well enough to
pass their first annual performance review. So in effect, training counted for up to the end of two
years. After that length of time, new situations may emerge in the workplace requiring other
skills, or clients’ jobs may have shifted in nature with changing demand and enterprise needs,
making their training prior to employment less relevant. Thus, two years may be a conservative
but realistic time period over which to credit training with any wage, tax, and benefit changes.
With this assumption, Chart 2.8.4 shows what happens to the ROI as it is calculated on a
cumulative basis over subsequent years. The benefits (additional taxes and welfare reductions)
are assumed to continue at their year one level, while the training costs are in for year one only
and do not appear in subsequent years. The net effect is thus to compound the benefits while the
11
Baltimore’s Workforce System at Work
Training Investment Analyses
________________________________________________________________________________________________
costs stay fixed: consequently, the cumulative ROI improves over time. With the second year
after exit, the cumulative ROI for the customized group has changed from a negative 9.6% in the
first year to a positive 81%, but the analogous figure for ITA-funded training is still negative, at
minus 16%. However, by year three, both types have a positive cumulative ROI. Thereafter, the
gap between the customized group’s ROI and the ITA group’s ROI continues to open, reflecting
the direction set by the initially higher wage gains of the customized group. The implications are
that: (1) more than one year in the job is required for either type of training to be a public
investment that pays off, and hence job retention over time is key; (2) time-to-payoff after exit
will be faster for the customized group than for the ITA-funded group.
So what have we learned from this section, and what recommendations does it support?
(TBD)
12
Tax components
13
St
at
e
an
d
lo
in
co
m
e
ta
x
in
c
at
e
t
d
ed
i
di
t
pa
i
cr
cr
e
t
di
t
ed
i
cr
cr
e
ly
e
rty
om
ul
tim
ne
d
ta
x
ov
e
m
e
to
di
t
di
t
cr
e
cr
e
e
ts
om
ed
i
cr
t
id
ed
i
di
t
pa
cr
in
c
ty
ve
r
io
r
pr
lp
o
le
co
m
e
po
in
in
co
x
M
D
Lo
ca
ed
xa
b
x
el
y
ta
pr
io
rt
ta
ta
m
e
di
ts
cr
e
cr
e
ta
x
ul
ti m
at
rn
ed
ea
r
ar
n
of
le
x
ta
x
m
e
ea
l in
co
un
d
ca
l
R
ef
Lo
ca
Lo
ca
ta
M
D
e
co
m
e
om
M
D
in
in
c
in
co
ld
to
$152,800
$219,221
$63,703
$171,741
$15,507
$2,634
$15,745
$1,758
$41,398
$26,116
$38,686
$80,162
$31,655
$2,859
$81,448
$42,740
$63,086
$116,115
$35,536
$111,133
$142,721
$85,544
$10,149
$36,813
$0
M
D
al
de
r
ed
ch
i
io
r
pr
$500,000
Fe
ea
rn
x
co
m
e
$2,626,368
$2,343,030
$1,981,520
$1,444,132
$997,200
$1,000,000
al
ta
in
$1,500,000
de
r
e
bl
e
AG
I
$2,000,000
Fe
om
ta
xa
al
$2,500,000
al
in
c
al
de
r
$3,000,000
de
r
al
de
r
de
r
Fe
$3,851,781
$4,000,000
Fe
Fe
Fe
Baltimore’s Workforce System at Work
Training Investment Analyses
________________________________________________________________________________________________
2.8.2 Aggregate value of tax components (pre and post)
$3,500,000
Pre-training:
Post-exit:
Baltimore’s Workforce System at Work
Training Investment Analyses
________________________________________________________________________________________________
2.8.3 Calculating the ROI by training type……
Basic ROI formula:
CUSTOMIZED TRAINING
Program Program
benefits - costs
ROI =
ITA-FUNDED TRAINING
$214,297 - $237,137
x 100
ROI =
$188,649 - $447,624
x 100 =
Program costs
BOTH TYPES OF TRAINING
-9.6%
ROI =
$402,946 - $684,761
x 100 = -57.9%
$237,137
ROI =
x 100 = -41.2%
$447,624
$684,761
Where "program benefits" includes…
reductions in:
Federal child tax credit
Federal earned income tax credit
MD earned income credit
MD poverty credit
Local poverty credit
Refund of earned income credit
Value of child care vouchers
Food stamps receipts
TANF/TCA receipts
...and increases in:
Federal income tax ultimately paid
State and local income tax ultimately paid
...which sum to:
$13,202
-$22,219
-$11,919
-$14,500
-$7,035
-$6,550
-$13,507
-$54,885
$1,943
$13,462
-$34,958
-$26,789
-$14,296
-$6,952
-$6,323
-$12,064
-$16,093
$172
$26,664
-$57,177
-$38,708
-$28,796
-$13,987
-$12,873
-$25,571
-$70,978
$2,115
-$115,470
-$103,841
-$219,311
$46,503
$52,324
$98,827
$29,094
$55,714
$84,808
$75,597
$108,038
$183,635
$214,297
$188,649
$402,946
14
Baltimore’s Workforce System at Work
Training Investment Analyses
________________________________________________________________________________________________
804%
2.8.4 Cumulative ROI over time, by training type
713%
800%
623%
700%
2
321%
279%
371%
237%
195%
153%
194%
111%
26%
77%
-16%
-41%
1
-58%
-10%
18%
100%
69%
135%
171%
81%
200%
-100%
253%
261%
300%
0%
312%
352%
400%
430%
442%
500%
488%
533%
600%
3
4
5
6
7
8
9
CUST
ITA
Years post-exit
15
BOTH
10
Baltimore’s Workforce System at Work
Training Investment Analyses
________________________________________________________________________________________________
16
Baltimore’s Workforce System at Work
Appendices
________________________________________________________________________________________________
Appendix D
Data for the cost-effectiveness and ROI analyses
17
Baltimore’s Workforce System at Work
Appendices
________________________________________________________________________________________________
Data sources:
The following data was given for the analyses of cost-effectiveness and return on investment:
(1) Client identification and new file ID set-up, prepared by Donna Safely, MOED and David
Stevens, UB-JFI.
(1) Client demographics and training data in starting SPSS file, prepared by Donna Safely,
MOED.
(2) Wage record information for up to and including 4 quarters prior to starting training, and up
to and including 11 quarters after exiting program, prepared by David Stevens, UB-JFI.
(3) Child Care Voucher data from Maryland Child Care Administration, prepared by Jane
Staveley, Maryland Dept. of Human Resources (DHR) and David Stevens , UB-JFI.
(4) TANF/TCA data from DHR, prepared by Jane Staveley, DHR.
(5) Food Stamps data from DHR, prepared by Jane Staveley, DHR.
(6) Annual/modified “Adjusted Gross Income”; federal, state and local income taxes paid;
federal, state, and local earned income credits (EIC); state and local poverty level credit. All
calculated from starting wage record data and family status and size information, following
IRS tax form rules and tax tables, for 1999, 2000, 2001, 2002, and 2003, using TurboTax™ ,
by David Bosser, JOTF, and Lindsey Woolsey, JHU-IPS.
Initial demographics of the two training groups:
 “dislocated workers” are twice as prevalent within the ITA group (57%) compared to in the
customized group (28%);
 females make up three-quarters of the customized group, but only just over half of the ITA
group;
 55% of the customized group and 59% of the ITA group are single persons with no
dependents; 45 other members of the customized group who are not single have 91
dependents, and the 47 non-single members of the ITA group have 78 dependents; thus, the
216 trainees are associated with another 169 individuals economically dependent on their
success: 385 people in total are “affected” by how this training works out;
 the customized group is, on average, younger than the ITA group, with a median age of 35
compared to 40;
 around three-quarters of each group has at least a HS/GED;
 college grads are slightly more prevalent in the ITA group (13%) than in the customized
group (10%);
 the customized group has double the level of “employed” clients compared to the ITA group
(21% compared to 10%);
 a higher share of the ITA group is “unemployed” (84%) compared to the customized group
(64%);
 the share of clients “economically disadvantaged” is higher for the customized group (59%)
than for ITA-funded training group (41%);
 the share of ex-offenders in the study population overall is low (under 7%), but they are
slightly more represented in the ITA group (10%) than in the customized group (3%);
 more customized trainees report an annual family income of $0 than do in the ITA group (40
compared to 28);
 individual median wages (from wage record data) of clients for the four quarters before the
start of training activity was lower for the customized group ($6,526) than for the ITA group
($9,092);
18
Baltimore’s Workforce System at Work
Appendices
________________________________________________________________________________________________
 8% of clients overall had no earnings in any quarter of the four preceding training; each
quarter before training had at least 22% of its clients with a “$0” income;
 about one-third of the customized group received Food Stamps in the four quarters prior to
the start of training, compared to only 14% of the ITA group.
Training intervention: cost and time
 the average cost of training is $2,348 for the customized group, compared to $3,892 for the
ITA group: the ITA option is two-thirds more expensive than the customized; this difference
is even more pronounced if we instead look at the medians ($1,894 for customized vs. $3,992
for ITAs – a 111% difference);
 the aggregate amount of money spent on training (excluding employer contributions to
customized training) for the customized group is $237,000, compared to $448,000 for the
ITAs;
 the median number of days between “start of training activities” and “exit” is 187; this is
shorter for the customized group (with a median of 164 days) than for the ITA group (223
days);
Wages and work-week outcomes:
 the median hourly wage of those reporting any earnings after training was $9.98 for the
customized group and $10.05 for the ITA group;
 the average hours in a work week was lower for the customized group (34.7 hours) compared
to the ITA group (37.9 hours); if we exclude those with “0” hours, this gap is erased (38.9 for
customized and 38.2 for ITAs);
 median earnings in the period for four quarters after exit are $19,120 for the customized
group, compared to $17,191 for ITAs;
Wages before and after training:
 overall, 73% of all clients show a four-quarter wage improvement in the four quarters after
exit compared to the four quarters before start of training activities (no adjustment is made for
average wage gain or inflation, which also could have contributed to the raise); 26% show a
wage decline; the share of clients with wage decline after training is lower for the
customized group (22% with a decline) than for the ITA group (30%);
 the median absolute 4-quarter wage change for all 216 clients is $7,154; the median change is
higher for the customized ($9,669) than for the ITA group ($5,572);
 the aggregate wage gain for all trainees in the customized group (101 cases) was $840,697,
compared to $668,085 for the larger (115 cases) ITA group;
Benefits before and after training:
 TANF/TCA: (TBD)
19
Baltimore’s Workforce System at Work
Appendices
________________________________________________________________________________________________

Food Stamps: 76 of the 216 clients (35%) had been in receipt of food stamps at sometime in
the 37 month period from April 1998 to April 2001. These 76 client recipients averaged 16.3
months of Food Stamp receipts each, with an average monthly receipt of $268.04. The
smallest monthly check was $4, and the largest was $647. During the 37 months, the smallest
total amount received by any individual client in receipt was $25, and the largest was
$18,755. When adding receipts for two periods -- 4 quarters before start of training activity
and the 4 quarters after exit – then in the customized group, 34 clients received Food Stamps,
with a median level of receipts of $1,163, and in the ITA group 16 clients received Food
Stamps, with a median level of receipts of $849. The change in the aggregate value of Food
Stamps by the customized group between pre-training and post-exit was a drop of $54,885,
and for the ITA group the same drop was $16,093. However, because the Food Stamp data
series stops in April 2001, we are missing more post-data than pre-data, so this comparison is
skewed.
 Child Care Vouchers: only 10 of 216 clients had any child care vouchers in either the 4
quarters before starting training or the 4 quarters after exit; the total sum involved before start
of training was $31,948, with individual client child care voucher values varying from $226
to $8,160; 9 of the 120 cases saw a decline in their child care voucher value post-exit, with
the aggregate value falling to $6,377 – an 80% decline.
Taxes before and after training:
(TBD)
Training investment calculations:

dollars of wage change per dollar of training cost, in aggregate, is $2.20 overall; this is much
higher for the customized group ($3.55) than for the ITA group ($1.49);
What else we have learned:
 Limitations of the self-reported annual family income variable. “Annual family income” or
“AFI” (as reported by clients with varying family sizes) may be a very poor measure of actual
income, compared to the quality of data in the wage record database (“WRD”) (which is
reported by the employer, verified by the state, and organized around the individual’s social
security number). The fact that the data we have from these two series are not for exactly
coincidental time periods will inevitably interrupt their numeric relationship, but even so, AFI
should generally be equal to, or greater than, the WRD number because potentially more
earners are present in the AFI numbers than in the WRD numbers. However, for the 79 cases
with non-zero values on both variables, 28 (or 35%) showed a family income less than the
individual’s income. One likely conclusion is that applicants for needs-based services and
benefits have an incentive to under-report, and that many do so.
 With increasing distance downstream from exit, the earnings data in each quarter become
“thinner”. Beyond 4 quarters post-exit (i.e. after just one year later), there is considerable
fall-off in earnings information and an increasing number of cases with “missing” earnings.
By quarter #6 post-exit, over half the clients have “missing” earnings data, and by quarter #8
20
Baltimore’s Workforce System at Work
Appendices
________________________________________________________________________________________________
(i.e. two years after exit), 94/101 of the customized group and 91/115 of the ITA group (or
85% of all 216 cases), are listed as “missing” for earnings. Partly this may reflect the
increasing impact of lags in reporting as we move further downstream from exit and closer to
the present day. The implication is we have relatively more information on older cases than
recent ones, and a systematic bias in results. Since the customized training model has been
relatively constant through this period while the “true-WIA ITA” model has evolved more
recently, then this time question may also bias the results by training group.
 Lack of spousal wage information is a large problem for estimating several components of
the overall budget. Some 44% of all trainees have dependents, and 18% (some of whom may
overlap with the 44%) are likely to be part of a 2-parent family or share the additional
income(s) of other earner(s). Both these variables in turn have large impacts on:
 the type of tax form used (single, joint, separate, 1040/1040A/1040EZ), which we are
trying to replicate;
 the size of “Adjusted Gross Income” for tax purposes, because there may be more than
one income and because of the number of exemptions that can be claimed;
 EITC, where eligibility and dollar value is a function of total family income level and
the number of dependents;
 taxes paid, since EITC is subtracted from the taxable income step.
 Different databases have different lags in compilation, introducing systematic bias between
different components of the budget. Clients in this study exited as late as June 2002. Wage
record data is available for all clients for the 4-quarter period after exit, but Food Stamp data
is only available up to April 2001. Of the 216 trainee cases, 53 received Food Stamps in the
4-quarter period before the end of the Food Stamp data set, but only 36 had a “start of
training activities” date during that period (thereby allowing summing of pre-training Food
Stamp dollars). Only 8 had an “exit” date during that period (allowing summing of post-exit
Food Stamp dollars). The reason why this is important for our overall calculations is that
Food Stamps receipt is not tied to workforce program participation. Food Stamps can carry
on after exit, but projecting beyond the April 2001 cutoff to make up for not having these
data would not be reliable, because the 77-case, 36-month Food Stamp database shows the
general pattern of Food Stamp receipt to be “intermittent”: only 26 cases have uninterrupted
periods of receipt. Thus, we have more complete information on wages than on benefits,
more complete information on “pre” than “post” situations, and more complete information
on earlier starters and exiters than we have on later clients. As a result, any pre- versus postcomparisons would inevitably overstate the “savings” to the budget from “reduced” food
stamp payments. We recommend holding this variable to one side until more complete data
is available.
 The importance of the downstream situation. At placement, i.e. just after exit, the median
hourly wage for the customized group is $9.98, while for the ITA group it is $10.05. This is a
difference of just 7 cents, or only 0.7%. Over the whole four quarters after exit, however, the
customized median wages were $19,120, versus $17,191 for the ITA group: this is a
difference of $1,929, or 10.1%. Thus, the relative value of different types of training may be
seen not so much in the ability to get a job with a given wage at placement, but in being able
to (a) access a job with potential for later earnings growth, and (b) being able to perform in
the job and thereby increase earnings later. This seems to support the case for looking not
only at "placement-based success" measures, but also further downstream at earnings
trajectories, if all the good the workforce system does is to be captured and demonstrated.
21