Download Nikos Kastrinos

Assessing the impact of Framework Programme social sciences and humanities research
on policy: what do we know and where are we heading
"Draft. Do not quote!"
Nikos Kastrinos
European Commission, DG RTD, Social Sciences, Humanities and Foresight
Abstract
There is an increasingly pressing need to evaluate the impact of social sciences and
humanities research programmes on policy. The links between research programmes and
policy making are multiple, complex and mediated by institutions, networks and scientific and
policy-making communities. Furthermore, specific characteristics of social sciences and
humanities, in institutional as well as content terms, have important implications for the ways
in which European research programmes function in Europe’s research scene, and for the
ways in which impact expectations are formed and materialise. The paper combines an
analysis of the mechanisms of knowledge transfer in policy with an analysis of the ways in
which the EU Framework Programme impacts on Europe’s science, economy, and society to
frame a programme of enquiry into the impact of Framework Programme social sciences and
humanities research on policy.
Disclaimer
All views expressed in this paper are the views of the author and do not necessarily reflect the
views of the European Commission.
1. Introduction
The European Union’s Framework Programme of Research and Development is founded on
the obligation to support the Union’s industrial competitiveness as well as the design and
implementation of its policies. Since the 4th Framework Programme (1994-1997) social
sciences and humanities have been consistently and rapidly gaining support from the EU.
Currently they constitute a substantial level of investment (in the latest discussions about
€90M per year over the next 7 years) channelled through thematic programmes targeted on
specific issues. At the same time the Union has been upgrading impact assessment and
evaluation activities, with a view to improve its policy-making. Based on its White Paper on
Good European Governance (COM(2001) 428 final) the Commission has produced a series of
guidelines about a policy-making process that makes good use of ex ante impact assessment,
real-time monitoring and ex-post evaluation (see for example: COM(2002) 276 final). The
rising level of investment in social sciences and humanities and the increasing importance of
impact assessment bring about a need to look at the impact of social sciences and humanities
research.
There is little tradition in ex post evaluation and impact assessment in the social sciences.
Research evaluation activities developed in the 1980’s underpinned by the relative scarcity of
funds which Ziman (1987) coined “science in a steady state”. In that period social sciences
and humanities were not seen as important parts of the government research enterprise. At the
European level around this time, when the Commission’s MONITOR programme was
promoting an evaluation culture throughout Europe and illustrating its value by applying it to
its own programmes, there were no substantial European programmes in the social sciences
and humanities.
The lack of tradition does not mean that the social sciences and humanities need to start from
scratch. They can benefit from the experiences of natural sciences and engineering, and from
the methods of impact assessment applied in technological programmes. There are two sets
of challenges in this transfer of methods and approaches. The first, relates to the “methods to
be transferred”.
Technological programmes are typically assessed for their impact on
innovation and economic competitiveness of enterprises (Vonortas and Hinze 2005). Social
sciences and humanities research programmes are justified at the community level in terms of
their support to policy. Thus assessment methods based on economic criteria and dimensions
of impact may not apply.
The second set of challenges for the use of traditional impact assessment approaches and
methods relates to social sciences and humanities themselves and their links with policy.
Social sciences and humanities comprise a rather diverse set of enterprises, which do not
always relate harmoniously to each-other. They are predominantly national not only in the
ways in which the research is organized but also in the definition of the subjects that they
study. They are strongly associated with the institutionalisation of Universities in the model
of Von Humboldt’s ideals, and thus they have a tendency to identify themselves with free
enquiry and downgrade policy-relevant research. It is notoriously difficult to trace “links of
influence” between social sciences and humanities research and economic, social and political
developments (Georghiou et al 2002). Even when links can be traced, the attribution of
influence is difficult and inescapably tenuous.
And sometimes when influence can be
ascertained, scientists do well to refuse responsibility for it, for as Davies et al (2005) remind
us, impacts may also be negative. In a very touching speech Rom Harré described how he felt
that a book, which he wrote with colleagues describing the ritual aspects of hooliganism,
contributed to the de-ritualisation of sport related violence, and thus to its increased brutality1.
In the face of such difficulties, advocates of social sciences and humanities research
programmes make increasing use “impact events” to argue that the programmes make
important contributions. The category of impact events includes “policy citations” (see EC
2003), workshops and conferences where researchers and practitioners meet and dialogue
(Liberatore 2001), scholars in positions of public influence and case-studies showing
influence circumstances (Commission on the Social Sciences 2003, British Academy 2004).
In particular the British Academy (2004) and the Commission on the Social Sciences (2003)
offer a rich collection of cases where researchers played a very important role in public
debates, which have shaped policy. In all those cases the case of influence describes the
scientist and not the research, and the links between programme intentions and impact are
always very weak. The “impact events” are near to descriptions of the concept of spin-off
The event was a workshop in Bruges on “social sciences for knowledge and decision-making” 28-28 June
2000, and the book in question was See Peter Marsh, Ely Rosser and Rom Harré, The Rules of Disorder,
London, Routledge and Kegan Paul, 1978
1
(Reppy; Ulrich 1988), reflecting the conclusion of the ASIF study (Georghiou et al 2002, p
322): that:
“the demonstration of explicit socio-economic benefits from social science (..)
research requires evaluators to follow long complex chains of events, but also, to be
aware of unexpected consequences”.
Impact events and unanticipated consequences can only be starting points for the assessment
of the impact of European programmes in social sciences and humanities on policy. The
reason is that the programmes aim to support policy, they design research agendas on the
basis of policy-relevance, and they assess proposals on the basis of relevance criteria. Thus
policy impacts cannot be coincidental in the way unanticipated consequences are. While
designing “impact events” and “dissemination strategies” can be seen as an indicator of
impact-intentions, this cannot be equated with impact.
For the purposes of impact
assessment, stories of impact need to be built around “intended consequences” and these are
channelled through the relationship between policy and science. For the purposes of this
paper, the framework in which intended consequences develop into policy-impact comprises
the relationship between policy-making and social sciences and humanities. Critical in this
framework is the interplay between scientific excellence and policy-relevance.
This interplay is discussed in the next section with the help of two stories of impact events.
The discussion leads to an analysis of the relations between social sciences, humanities and
policy-making. This discussion follows the tradition of innovation studies, starting with the
question of whether social sciences and humanities can be seen as sources of radical change in
a Schumpeterian sense, which is discussed in section 3. The question is placed within a broad
analogy of public intellectuals, who are seen as playing key roles in how social sciences and
humanities affect society (see for example British Academy 2004) with Schumpeterian (1934)
entrepreneurs who bring to society fruits of science through innovation. From radical change
the paper then turns, in section 4, to social sciences and humanities in their potential to be
involved in incremental change through policy-learning processes. Section 5 discusses the
diffusion of social sciences and humanities knowledge looking at its characteristics as well as
the ways in which epistemic communities and networks of learning may connect scientific
and policy institutions.
This discussion provides a series of hypotheses and stylized
understandings which section 6 relates to the general arguments about the impact of the socioeconomic impacts of the Framework Programme. Section 7 uses our fairly limited knowledge
of how the EU programmes have affected the social sciences and humanities community to
develop impact assessment questions and perspectives, while section 9 concludes with the
impact on policy.
2. Events of impact: social scientists affecting policy
In March 2000, Maria Joao Rodrigues, the then Chief Economist of the office of the
Portuguese Prime Minister Antonio Guterres who was presiding at the European Council,
organized a small conference in Lisbon.
A number of the academics involved in the
conference had long been involved in European research programmes. The conference has
been seen by some as the launch pad of the Lisbon Strategy document, which was approved
by the heads of State and Government and became the key political declaration on Europe’s
economic future (see Rodrigues 2002)2. Europe’s social sciences and humanities community
celebrated the document. In 2001 the Commission produced the largest ever international
research programme in the social sciences and humanities, with the title “Citizens and
governance in a knowledge based society”. Maria Joao Rodrigues became known as an
architect of the Lisbon Strategy 3and was appointed to lead the European Commission’s
Advisory Group for Social Sciences and Humanities. In the beginning of 2002, in response to
a call for expressions of interest, the Commission received more than 1200 proposals to
implement the theme.
The generic story is as follows. Scientists who take part in a research programme produce
knowledge that plays an important role in setting a policy agenda. The policy agenda plays an
important role in setting a research agenda in the social sciences and humanities and the social
science community responds enthusiastically. This can be now compared to another story. On
21October 2005 the Prime Minister of the UK, Tony Blair, reportedly brainstormed with 10
leading academics from across Europe in preparation for the Hampton Court informal
European Council Summit.
Less than half of the academics present had taken part in
European research programmes4. Both European Summits dealt with the future of Europe’s
economy, in both cases academic scholars from the social sciences and humanities were
strongly involved, and in both cases some of those academics have been supported by
2
Of the 9 contributors to the edited volume 8 were in some way associated with the TSER programme.
See CORDIS 20-10-2005: Focusing on Lisbon fundamentals:
http://aoi.cordis.lu/article.cfm?article=1487&lang=EN
3
European programmes. However, the difference in the number of the scholars involved
which is related to the programme raises some questions: Could it be that, between 2000 and
2005, the European programme lost its excellence, its relevance and its potential for impact?
How would the difference in the sources of expertise affect the difference in the outcomes
between the two Summits?
One could argue that the events show that the policy-relevance of the programme and its
potential for policy impact declined between 2000 and 2005, although, from a dominant
agenda-setting influence in a European Summit, the only way is down.
In terms of
excellence, there are two interesting sets of questions arising. One is empirical and concerns
the criteria whereby expert advisers are chosen and whether these criteria provide results that
are consistent with citation ratings and scientific reputations. The other question concerns the
relationship between scientific excellence and the quality of the outcomes of policy-making
based on scientific advice. Lal (2002) argues that the contributions of sciences could be
measured exactly by this measure. This is for historians to argue about.
In their current institutional set up, social sciences and humanities take no responsibility for
policy, neither for design nor for implementation5.
Social sciences and humanities are
concerned about the quality of science and its excellence as internal matters, and its “impact”
primarily as internal but increasingly also as an external affair (see Davies at al 2005). The
more science becomes concerned with external impacts, the more scientific excellence
becomes involved in dealing with such impacts, either as a prerequisite for impact (e.g. as a
criterion to be fulfilled by policy advisors) or as a guarantor of quality of the advice (e.g. in
risk assessments that affect laws) or both. Excellent science should by definition have more
impact than less excellent science, but one cannot use the impact of science to assess its
excellence, because the impact mechanisms and processes may not be functioning as well as
they should in all cases. A great deal of the effort of this paper is in highlighting those impact
mechanisms, the ways in which they may be structured and function between social sciences,
humanities and policy-making. It is the argument of this paper that the relationships between
social sciences and humanities and policy need to be carefully considered in framing the
enquiry into the impact of the Framework Programme.
4
As report in European Voice 20-26 October 2005by Dana Spinant and Tim King
Some believe that this is where social sciences differ from natural sciences which take responsibility for their
arguments, predictions and their applications (see Lal 2002).
5
Now impact assessment, as depicted in figure 1, can address the following questions: How
much impact does the programme have on “science”? (Arrow A in figure 1). How much
impact does science have on policy? (Arrow B in figure 1). How much of that impact can the
programme claim? (A/B * 100 %). Of course the programme has characteristics other than it
financing science and thus there are impacts which arise outside the scientific nature of the
programme (arrow C in Figure 1). Those impacts will not be addressed in this paper, not
because they can be assumed to be negligible, but because they concern predominantly
integration effects on mechanisms and processes of research policy-making (see Georghiou et
al 1993, Kastrinos 1998, Patricio 2005).
Figure 1: An impact assessment framework starting from the programme
Science
A
C
Research Programme
B
Policy
There are a number of simplifications in the above scheme, most obviously ones concerning
the directions of impact. From a programme perspective it is important to understand that
these are the directions of impact that are most interesting and important, even if they are
diminished or reinforced by influences in the other direction. This is the broad impact
assessment tradition in which this paper will develop.
3. Social sciences and humanities and radical change
One aspect of great importance in the study of learning and change is the dialectic between
events and processes. Events have the power to focus our attention and make us remember
the importance of changes in the histories we live through. The collapse of the Berlin Wall
and the invention of the horseless carriage, are events which signify important processes of
change and through this process the events acquire a timelessness and instantaneity. They
develop boundaries around them, and historians and analysts argue about these boundaries.
From this perspective people often distinguish between important events that signal
“revolutions” and less important events that are part of continuous “evolutions”. There is
always a lot of debate about the level of unpredictability and change that needs to be
attributed to an event before it becomes part of a “revolution”. Yet, study of change always
looks first at obvious change phenomena i.e. revolutions.
Schumpeter attributed economic change to entrepreneurs who destabilise the system with
their innovations. Those innovations then diffuse through the system as imitators follow the
example of entrepreneurs.
Innovations were important “events” that destabilized the
economic system. The source of innovations was seen as being outside the sphere of the
economy (at least in early Schumpeterian papers see Philips 1971). Rosenberg (1976) argued
that as a result of the Schumpeterian heritage, we neglect important aspects of the innovative
process, we overemphasize the contributions of science and we neglect important processes of
adaptation and diffusion. In a similar vein Freeman and Louça (2002) argued that the myth of
the heroic scientists and entrepreneurs was responsible for important misunderstandings in
economic history. In short, in the study of economic change, attention has shifted from
“major events” towards “processes” that are much more piecemeal, mundane parts of
everyday life. .
The view of heroic technological innovations inducing radical changes in the economy and
society is a powerful modernist vision, holding strong still today. For example, the inventioninnovation part of Schumpeter’ (1939) s business cycle is widely known as the “linear model
of innovation”, and is still today the cornerstone of much of science policy around the world.
The idea that developments in science and technology underpin a great deal of change in the
economy, society and policy, coupled with the image of disinterested scientists discovering
the laws of nature, provides a legitimate source of potentially radical innovation in society and
policy.
In the political arena, this dialectic between revolution and evolution has rarely been analysed
to greater depths than in Popper’s “Poverty of Historicism” (1957). Popper argued that the
belief in historical destiny, which underpins many revolutionary movements, is “sheer
superstition”. Utopian movements are likely to fail in their predictions for a number of
reasons. One such reason is that they cannot consider future knowledge about the natural
world and future technology. Thus, promises about a better future may always be rendered
obsolete by future technology. For Popper (1957) knowledge is emancipating, and he argues
that it is impossible to control what people think and it is thus impossible to centralize
knowledge. For these reasons, Karlsson (2005) argues that there is a long-term decline in the
role of visions in political debate and an increasingly differentiated society.
This is a powerful argument which, nonetheless, deserves some discussion. First, scientific
discovery and technological change do not only have emancipating consequences but also
contributes to all kinds of new risks, new inter-dependencies (Beck 1999), and new kinds of
governance questions (Collingridge 1980). It is fair to say that scientific and technological
change have destabilising consequences for policy, and that governments exercise some
control over science although science is not fully controlled by governments. Governments
often set up intermediary agencies to control and direct science and technology towards
desired directions (Johnston and Gummett 1979, Braun 1993). The conquest of space, safe
nuclear energy, and public health, are amongst the missions of such agencies, representing
important policy visions with positive social connotations. Even through such agencies,
government control over science has been limited. Agencies depend on science for the
knowledge of their staff (Braun 1993) and they
“are obliged to proceed on a narrow path that is framed by the long-term time-horizon
of basic research in the scientific system, the social and institutional organisation of
research as well as the cognitive abilities of scientific communities” (p 155).
It is not obvious that Popper (1957) considered social sciences and humanities as
emancipating knowledge about the natural world, or simply introvert thoughts of individuals
that cannot be controlled by government. Yet, there is no doubt that social sciences and
humanities share some of the disruptive potential of natural sciences and technologies, and
that governments often set-up agencies to direct and control them and to benefit from their
development (see Amann 2001, Braun and Benninghoff 2003, Caswill 2003). There is also
no doubt that, following the collapse of Communism, scholars are becoming less and less
fascinated by the prospects of generating normative implications from research results
(Karlsson 2005). For example Louis Uchitelle, quoting Steven Levitt and Arjo Krahmer,
reports in The New York Times that
“.. fewer students than in the 1980’s have read the works of ... Adam Smith, ... Alfred
Marshall and John Maynard Keynes. These eminences painted the economy on a
broad scale and were far more engaged than modern economists in political choice.
Research, theory, anecdotal observation and policy prescriptions were much more
intertwined. This was also true for recent Nobel laureates like Milton Friedman, Paul
Samuelson, James Tobin and Robert Solow, all of whom were young during the
depression.... ‘we have lost our optimism that the tools of economics can be used to
manage the economy’ Levitt said and we have moved to a much more “micro” view of
the world’”6
This reluctance is a blow to the role of “public intellectual” as a source of radical change in
society, and it is a powerful one in a period when there are widespread warnings about the
dangers of over-reliance on distinguished academic specialists “since, with the best will in the
world, what they provide is one particular interpretation of the available evidence”. (Amann
2001 p 74). Yet, it should not be understood that the potential of social sciences and
humanities to nurture ideas that could bring about radical change can now be dismissed. It
could well be that the focus on research agendas of lesser consequence than changing the
world, will emancipate the social scientists and humanists and provide them with the space to
claim more responsibility for the impacts of their work.
4. Policy learning, policy-making and social sciences and humanities
The standard starting reference for policy-making and policy learning is the “rational actor”
model (Richardson and Jordan 1979).
In “rational-actor” settings, learning takes place
through experimentation, within action-feedback loops. The simplest and most powerful
example of a feedback mechanism for democratic governments is the next election. To start
the accumulation of learning experiences, there are problems that need solutions. To follow
the example above, sets of such problems and proposed solutions are defined in pre-election
manifestos. Rational actors define the problem, compare solutions and take action to solve
the problem. In practice, actors are recognizably bounded in their rationality, feedback and
consultation take many forms, and ad hoc trial and error processes by different actors
involved in collective policy-making create a process of “muddling through”, which from the
6
“Yound American economists shun policy wars”, The New York Times, 26 January 2006
outside seems to lack rationality (Lindbloom 1959). Models of policy making of this nature
are called incrementalist models (ibid.).
The incrementalist description of policy-making as muddling through is very powerful in the
ways in which it depicts people’s experience of policy-making. At the same time, by giving
primacy to the interests of different actors, the description suffers from a lack of normative
implications. For example, Lindbloom and Woodhouse (1993) find that policy-making is
“insufficiently intelligent and insufficiently responsive”, and argue that it should make more
use of analysis and more democratic participation. The process of making policy-making
more intelligent is close to ideas of policy-learning as discussed by May (1992) and Barun
and Benninghoff (2003). Braun and Benninghoff (2003 p. 1850) see policy learning as
involving a “confrontation between ‘power’... and ‘puzzling’”. This does not mean that
rationality is absent from the process of interplay between power and puzzling, between
political interests and substantive content.
As Braun and Benninghoff put it:
“the problem solving discourse is constrained by interests but interests are also
tempered by their incorporation in the discourse process” (2003 p 1862).
May (1992) distinguishes between political learning which concerns policy processes and
prospects and takes place within advocacy contexts, and policy-learning which concerns the
construction of policy problems, the means to address these problems and their evaluation.
According to May (1992) policy learning involves instrumental learning and social learning.
The former leads to actors becoming more intelligent about the instruments that they use,
while the latter concerns the broad rethinking about the problems addressed and the means to
address them.
The social learning element of policy learning has been emphasized by the literature on
policy-transfer, which describes a process of diffusion of knowledge from one policy domain
to the next and from one country to another (Kingdon 1984, Rose 1991, Stone 2001). Rose
argues that policy learning is social learning that takes place within “epistemic communities”.
Such communities involve embedded ideas about causal relations between policy, collective
and individual actions, and they are fundamental for the ways in which policy ideas transfer
between places and policy areas.
Yet even in such communities, policy-makers learn
primarily from experience, from observing others and from discussing with colleagues in
networks which concern policy-knowledge (Stone 2004).
Rose (2001 p 15) juxtaposes
“lesson-drawing models” to “theory-based models”.
“The fundamental difference between a lesson drawing model and an economic model
is that the former is abstracted from an actual programme whereas an economic model
can be produced by deductions from pure theory. A lesson-drawing model is therefore
about what is7; its content is not defined by reasoning from ‘landless’ axioms but by
contextual observation”.
The juxtaposition of the models used by scientists to the models used by practitioners is found
also in the field of technology (e.g. see Mackenzie and Wajcman 1985). In that field the
models of scientists have been increasingly important for ways engineers carry out their work.
A similar phenomenon is observable in economic and policy life, where models of
“scientists” who study organization become ever more powerful shapers of the practice of
organized life (see Callon 1998, Mackenzie 2005). Thus, it is possible that the contribution
of “theory”, which is so dear to the hearts of humanists and social scientists (Nowotny 2005),
to policy-learning is (and can be) much more important than analysts seem to consider.
In addition, instrumental policy learning often involves government science, which includes
social sciences and humanities. These sciences often generate information for use in policy,
such as indicators and feedback on policy-choices (Kingdon 1984). A caveat here is that not
all government science is “instrumental policy learning”, as some routinely carried out
government research activities are part of policy implementation but not part of policy
improvement processes. Furthermore, a lot of policy improvement processes are not effective
in changing policy, which according to May (1992) is a key question in determining whether
policy learning has taken place. However, a great deal of applied research commissioned by
government in the form of consultancy is part of policy improvement processes and thus can
be seen as a corollary of government investment in instrumental policy-learning.
To summarize, in incrementalist policy-making substantive rationality does not disappear but
holds a very important place, and is supported by theory and evidence, as well as research
endeavours which may be parts of government science, or commissioned consultancy. Social
sciences and humanities can be greatly involved in this exercise and to some extent are,
7
emphasis from the original
through from a distance. It is rather curious that observers of policy-learning continue to
describe it as an empiricist exercise based on craft models of communication between
practitioners, rather than as an exercise based on scientific research. To understand why, one
has to look at the characteristics of knowledge produced by social sciences and humanities
research and at the composition and functioning of the communities and networks of
practitioners where policy knowledge takes shape.
5. Policy-learning communities and the characteristics of social sciences and humanities
knowledge
In many ways the process whereby knowledge diffuses in policy-making communities
resembles the diffusion of technological innovation in the economy (see Rogers 1962):
through imitation and adaptation (DiMaggio and Powell 1983), complementary resources and
skills (Teece 1986), with the pattern of diffusion depending on the innovation, the adopters
and the environment in which it takes place (Stone 2001). Issues of transmissibility of
knowledge, the media used, trustworthiness of the source and opportunity (see Liberatore
2001) are as important as issues of usability, user-friendliness and usefulness in particular
contexts, requirements for complementary assets and competencies, short-term performance
advantages and competitive gains, as well as interest in terms of allowing the development of
interesting longer terms agendas.
According to Solesbury (1994) social sciences and humanities have five defining
characteristics which affect their capacity to communicate their messages to policy-makers
and thus to affect policy. They are “mundane”, “contingent”, “reflexive”, “non-cumulative”
and “non-appropriable”.
Because of these qualities, communicating the message is
particularly difficult. Mundane messages are often seen as no more than common sense.
Contingent messages are not generalisable. Reflexive messages tell you more about the
sender than about the subject of the message. Contingency and reflexivity combined
undermine the accumulation of knowledge, and as a result such knowledge is not exploited in
innovation process in government and industry. Solesbury (1994) concludes that:

because of these characteristics social sciences and humanities knowledge is a public
good, transferred in the public domain;

social sciences are very present in society and media and that often social scientists are
often uncomfortably drawn into public debate; and

processes of knowledge transfer are not unidirectional from producers to users, but
rather should be interactive, continuous and involve a dialogue between sophisticated
speakers and sophisticated listeners, something in the spirit of what Gibbons et al
(1994) coined “mode 2” research programmes.
On the basis of a similar analysis, although with different emphasis, Manicas (2003) argues
that social sciences suffer a credibility-deficit which undermines their effectiveness in playing
an important role in processes of innovation in the economy, society and policy. This
credibility deficit, he attributes to specific institutional choices of the social sciences, such as:
the radicalisation of debates about the sciences themselves; the use of boring textbooks; the
lack of concern with the interest and quality of students; the increasing specialization in
narrow fields; and the loss of the terrain of public debate to professional journalists.
Manicas’s (2003) credibility deficit is consistent with most points made by Solesbury (1994)
about the nature of social sciences and humanities knowledge. Where the two disagree is on
whether the epistemic communities of social scientists and humanists are too closed to have
any impact (Manicas 2003) or too open to be distinguished from the broader society
(Solesbury 1994). The result in both cases is sub-optimal exploitation of social sciences and
humanities knowledge, either because the cost of translation is too high or because it is
indistinguishable from common sense.
Rather than arguing about the ineffectiveness of social science knowledge in addressing its
messages to policy, Weingart (2001) describes a gradual opening of politics to science and of
science to politics, key players in which are scientists-advisors, whose task is to translate
scientific knowledge into inputs in policy. He describes a process whereby policy-makers
discover that science can be a political resource and compete with each-other about the latest
research-based knowledge.
This exposes the political elements in scientific debate and
knowledge and undermines confidence in science itself
“...the competition for, and inherently inflationary use of, scientific advice for
legitimating (and even instrumental) purposes is self-destructive and de-legitimating.
The paradox arises because, in principle, the competition for the latest, and therefore
supposedly most compelling scientific knowledge, drives recruitment of expertise far
beyond the realm of consensual knowledge right up to the research frontier where
knowledge claims are uncertain, contested and open to challenge” (p 85).
An interesting aspect in Weingart (2001) is that the process is driven by the effort of policymakers to appropriate science, and is fed-back by the effort of scientists to exploit the interest
of policy-makers. Thus, the two communities, the scientists and policy-makers become
entangled in a web of mutual influence and support, a kind of hybrid community, or what Rip
(2001) called a “network of joint learning”. Apparently this entanglement creates problems
for both policy and scientific institutions. Before discussing this in more detail, it is worth
reviewing two ways to involve social sciences and humanities in policy learning without
creating a hybrid community, namely action-research and evidence-based policy.
Action-research is an umbrella term for research methods that use the research process to
improve practice. Action research is participative and the research problem is constructed
iteratively between the practitioners and the researchers, who collectively construct the
solution through the research process. Action research is not “mode 2” research in the social
sciences and humanities. In action research the contribution of research to policy is primarily
procedural. The authority of the scientists stems from procedural knowledge rather than
authoritative knowledge of the subject matter of the problem at hand. In this setting, the
knowledge of the scientists is exploitable through the professional recognition of the
scientists’ skills in process organization. The community of “action researchers” can only be
a professional community and requires some form of professional regulation. At the same
time the community in which substantive issues are discussed can be as broad and pluralistic
as the concept of “public debate”.
A completely different approach comes from the recent UK initiative on “evidence based
policy”. The core idea here is to develop a service that uses research in the social sciences
and humanities to derive evidence about the potential and actual efficacy of different policy
choices, and to use this evidence to guide policy-choices.
This implies that scientific
evidence is usable outside the context in which they were generated, and somekind of a
process of review can provide the requisite validation, if an appropriate organizational
structure is established (see Amann 2001). The relations between the process of “deriving
evidence” and “scientific authority” are critical for the system, which could evolve as a
knowledge management IT tool or as a corporatist authority for the governance of social
sciences and humanities research which talks to policy in the name of science8.
8
Amann (2001) describes it as an information system but gives it corporatist governing qualities
Both action-research and “evidence-based policy” have, so far, limited following. One reason
for the limited following may be that, in both cases, there are important issues to the
appropriation of substantive knowledge by the scientists who create it. Researchers are
unlikely to be satisfied with the role of procedural councilor implied in "action research",
whilst they would probably be suspicious of the corporatist spirit of evidence-based policy.
In the process of interaction described by Weingart (2001) competition between scientists for
policy influence is an important part of the story.
“Policy discourse can create linearity after the fact” and “(social) scientists can push
for a linear model in arguing for the importance of their insights in creating or
modifying policy. The evolution of evidence and policy in relation to controversial
issues (can be best understood) as a network of data, interpretations, theories, values,
interests and strategic positioning, which eventually stabilises and creates robust
evidence and justified policy. This understanding can be extended to evidence and
policy more generally; it is an evolving network rather than the unidirectional
relationship suggested by ‘evidence-based’. Such evolving networks are the carriers
of joint learning.... Joint learning is open-ended (nobody knows the right answers) and
the process can be traced as increasing articulation and alignment, and new processes
of quality control need to be found” (Rip 2001 p 97)
The new processes of quality control need to satisfy the evaluations of scientists about
scientific quality of knowledge as well as the evaluations of policy-makers for the usefulness
of the knowledge offered. And in such conditions the networks of joint learning may evolve
into joint epistemic communities with their own institutions and learning processes. Such
communities may not be as broad and pluralistic as action research or as regulated and holistic
as evidence-based policy. They may evolve in specific fields and topics through specialist
journals, training and epistemic institutions and research programmes9.
9
A classic community of this nature is the one dealing with evaluation of research, which involves practicioners
with policy-making roles and practitioners with academic roles, and its own journals, methodological
discussions, authorities etc.
6. Evaluating the impact of the framework programme: what do we know and where
are we going?
In the classic form of evaluating the impacts of the framework programme, the evaluator will
look for induced changes that could be attributed to the inputs. The most frequently met
approach is to ask beneficiaries:

for the benefits that they accrued from their participation,

whether these benefits met with their expectations and

how they would contrast the situation after their participation in the programme, with
a counterfactual situation in which they would not have participated.
The difference between the perceived and the counterfactual is called additionality, and it is
the most widely used measure of impact (see Vonortas and Hinze 2005). The induced change
may be quantitative, for example at the level of investment, and/or structural or behavioural
(Buisseret et al 1995).
This is the approach that has been applied to the Framework programmes. Research has
found that participants value the funding that they receive as well as the cooperative links
which they form through the projects. Through these two sets of inputs people, organisations
and institutions change their cognitive and social outlook and become more open-minded, and
thus more able to use competencies and assets that they did previously did not have access to,
information about or authority on10.
Of course the assessment needs to compare the additional benefit to the cost of the
programme. The argument here goes as follows. If the additional private benefit for the
participants was higher than the additional private cost, then there would be no need for the
public programme, as participants would finance the projects themselves through market
mechanisms. Thus, the public benefit must be over and above the sum of private benefits
accrued. It must be said here that the private benefit from the additional investment is also
probabilistic, as it requires innovation, competitiveness and improved performance. However
10
This is perhaps an unfair reduction of a great deal of insight from a predominantly management literature on
the benefits of R&D cooperation. For example, Guy et al (2005) used 19 different indicators to examine the
impact of FP5 projects on participants, 18 of which fall comfortably under the category “open-mindedness” (the
19th was “cost-reduction”).
the important point is that the public benefit must be over and above the additionality of the
programme.
There have been a number of hypotheses as to how the public benefit materialises. A first
hypothesis is that the Framework Programme imposes a workable contractual standard in
Europe for collaborative endeavours and thus reduces dramatically the transaction costs
involved in collaborative R&D. A second hypothesis rests on the notion that knowledge, and
especially technological knowledge, is an impure public good, and as such it is best
appropriated within communities (or networks) of a particular size. Watkins (1991) using this
theory argued that Framework Programme projects and programmes help communities of
practice approximate this optimal size for the appropriation of technological knowledge. A
similar hypothesis is that the Framework Programme has provided for increased but
controlled variety of technological options in Europe, through promoting inter-institutional
cooperation and links. Yet another hypothesis could relate to the limited elasticity of R&D
employment in companies (Hall 2002). Because of the importance of tacit knowledge in
R&D organization, companies are reluctant to hire and fire R&D personnel to compensate for
fluctuations in their turnover. An argument can thus be advanced that the flexibility in the
labour market for researchers promoted by the Framework Programme, has helped both
researchers to become more adaptable (by virtue of being more open-minded) as well as
companies train occasional and future R&D personnel.
How do these hypotheses relate to social sciences and humanities? To start we have to clarify
that there seems to be very little private R&D investment in social sciences and humanities,
and thus the issue of additionality does not really apply.
Then the benefits from the
Framework Programme seem to concern predominantly industry and its relations with science
and thus not evidently the social sciences and humanities. Yet, we know that researchers in
the social sciences and humanities enjoy the benefits of open-mindedness brought about by
the funding for research and the international collaboration involved (Kuhn and Remoe 2005).
Thus one could look for the ways in which this open-mindedness may affect the networks and
communities that are formed, and ways in which the characteristics of those networks and
communities affect the performance of the institutions involved. The study reported in Kuhn
and Remoe (2005) illustrated some of the structural effects of the programmes, highlighting
the different ways in which comparative research has been pursued and how these have been
building a European research community concerned with European issues.
The study
examined the basic characteristics of the programme: international collaboration, interdisciplinarity and policy-relevance and provided the richest, so far, source of evidence about
the mechanisms of impact of the Framework programme on social sciences and humanities,
and on the way in which they are organized. Insights from this study can be used to frame
questions about public benefit as well as policy influence.
7. The impact of the framework programme on social sciences and humanities
Kuhn and Remoe (2005) considered that “scientific excellence” was an ex ante criterion
which had been applied in the evaluation of proposals. Ex post assessment of scientific
performance can look into the scientific reputations of participants and their teams before and
after the programme, the careers of individual scientists, and the impact of their work on the
scientific state of the art. Indeed the technical and methodological aspects of such a project
would be quite complex but in principle the scientific quality of research can be ascertained.
For example, there is some discussion about whether results can be attributed to programmes
or simply to the implementing people and institutions, but there is a consensus that the
research team is an appropriate level of analysis and thus the extent to which the programme
has been part of the funding sources of successful teams can be an appropriate method of
attribution (see Laredo and Callon 1990, Laredo 2001). Through explorations of systematic
relationships between participation if the programme and the impact of teams it is feasible to
acquire a picture of the impact of the programme on the “state of the art” in science. It is
important to mention that such an enquiry cannot answer questions of output-additionality and
attribution, i.e. it would still not be possible to say what the state of the art would have been
without the programme. However, it would provide a solid understanding of the relative
influence of the programme, which could then be compared with the relative influence of
other programmes, allowing thus some form of benchmarking (see Kastrinos 2001).
In exploring the impact on the state of the art, one issue that stands out is the belief that
comparative research is more incisive, and provides findings of higher validity and reliability
than non-comparative research, especially in relation to policy relevant questions. Kuhn and
Weidemann (2005) show that comparative research is at the heart of European concepts of
trans-nationality, distinguishing between three types: of research projects:

country case-studies;

country case-studies combined with cross-country thematic analyses; and

thematic analyses involving sub- or meta- country-level units of comparison.
A key element of the quality of comparative research is the validity and reliability of the
comparison. This depends very much on the existence or not of comparative (and thus
comparable) data-sets. The extent to which European projects generate comparative data-sets
is an important empirical question, which may underpin the disparity between expectations of
impact and reality, at least for the researchers interviewed by Kuhn and Weidemann (2005)
who complement their analysis of trans-nationality with an analysis of the problems of
comparison associated with linguistic diversity. Genuinely comparative data will be a major
achievement of European research in terms of usefulness (see Forbes and Abrams 2004). The
award of the 2005 Descartes prize to the European Social Survey, and the political importance
given to the discussion about data in the US, the UK and also in Europe point in that
direction11.
8. From impact on science to impact on policy: policy relevance and joint learning
As part of the same study, Greco et al (2005) provide an analysis of policy relevance in
projects as involving five distinct discourses:

“knowledge for data and model production”: this kind of research takes place within
models applied in policy, and fits well with May’s (1992) instrumental for policylearning;

“knowledge for policy suggestions”; this kind of projects aimed at reframing policy
issues in ways that were meaningful to policy makers but different from the dominant
ones;

“knowledge for complexity reduction”: this kind of research aimed at reducing the
complexity of the world through theory led models (exemplifying Rose’s (2001)
distinction between theory based models and policy lesson-drawing models.

“knowledge for socially relevant purposes”; this kind of research aimed at responding
to the needs of broader communities and involve them in social learning processes;
and

“knowledge for knowledge production”; this discourse was about the distinctiveness
of the scientific enterprise from policy-learning.
These five discourses, which were not necessarily project-specific, reflect the picture drawn
from the policy-learning discussion. Two distinct communities that could communicate and
interact at the levels of both “theory” and “evidence”, yet do not. Considering that the
evidence in Kuhn and Remoe (2005) comes from studying researchers, this is strong evidence
that, at least at the European level, communities of practice spanning social sciences,
humanities and policy-learning are not common, and there is a lack of institutions and real
efforts that could bridge social science and policy-learning. An alternative interpretation may
be that the joint learning networks of European Union policies do not necessarily involve the
coordinators of research projects, who gave some evidence of being frustrated by this
situation (ibid.). Under this interpretation, European research projects need to perform
simultaneously in two respects in order to become parts of “joint learning” processes. The
first is scientific quality and the second is illustrated usefulness.
Scientific quality and illustrated usefulness are necessary, but not sufficient conditions for
“joint learning” in networks that involve researchers and policy-makers. First of all, the
network has to be somehow constituted. In this area, Kuhn and Remoe (2005) argue, EU
programmes have promised scientists a lot, but not delivered. It is also an area where there
are continuing discussions and experimentations with different forms and initiatives. A
promising recent discussion concerns “societal platforms”, wherein stakeholders and
researchers would articulate joint research agendas. Although there are high hopes for this
type of processes, the fundamental problems associated with the limited appropriability of
knowledge will affect the commitment of stakeholders and will not be easy to overcome.
However, the ex ante character of societal platforms (as dialogue spaces before the research)
would mean that impact assessment considerations will need to be formed rather differently.
Depending on established communities, networks and processes of joint learning, the impact
may differ substantially between countries, between policy areas and policy institutions. Yet,
as policy-makers learn from one another, the programme may have substantial impact
indirectly. For example, it could be that that the evidence-based-policy mechanism in the UK,
some programme findings contribute substantially to a policy change in the UK, which is then
imitated elsewhere. Following the rationale of Watkins (1991) this may be an efficient way
of managing knowledge and policy-learning from social sciences and humanities at European
11
The prospects for comparative data sets are to be discussed in a seminar organized in ESOF 2006 jointly by
the EC and the US NSF on “big social science: a transatlantic perspective”
level. On the one hand, the diversity of possibilities for impact increases the possibility of
research-based policy innovation, and on the other hand, where an all inclusive dialogue and a
coordinated process of research-based policy-learning would create information overload and
“congestion” phenomena. In terms of impact assessment such “spin-off” type phenomena are
very difficult to capture, require particular effort at documentation as well as research-friendly
citation practices. EC (2003) showed that it is feasible to illustrate the existence of research
citation by policy documents, but it is very difficult to say anything reliable about the citation
practices of policy-makers.
A more reliable but also less attributive method is to employ co-word analysis (Callon et al
1983). With the help of social network analysis software, co-word analysis is much easier
these days then when it was first advocated (cite the dictionary project). Co-word analysis
maps text as a network between words that are linked with links of co-occurrence in a text.
Such maps may represent knowledge development as is indicated by scientific texts, (for
example research reports), which could be compared with maps of programmatic texts as well
as maps of policy papers. (The diffusion of the practice of “green papers” and “white papers”
in Europe may be very helpful in this respect). As texts are characterized by the time of
publication, comparisons in time between maps may show directions of influence and uncover
the networks of “joint learning” in the sense that Rip (2001) used the word. Such an exercise
would also allow the assessment of the impact of “important events” and place dialogue
initiatives in context.
Needless to say the combination of data on spin-offs, events, and cognitive linkages in time
would constitute a dream machine for impact assessment. Yet, it would miss the potentially
very important “labour-market” effects of the programme, which are not included in the
discussion of scientific quality. There is an urgent need to examine the training effects of the
Framework Programme, its contribution to the creation of a European community of flexible
and open-minded young researchers, and the career paths of those researchers. It is primarily
through people that the experience of trans-national research affects the ways institutions
think, learn and perform, and it is though links between people that networks of individuals
which engage in joint learning are formed. Are the links between researchers from different
countries of today, the policy-learning networks of the future? The more senior managers and
policy-makers are formed through such networks, or are trained by people involved in such
networks, the more likely they are to learn from social sciences and humanities research.
In 1978 Haas at al. interviewed a number of scientists, mostly physicists, who worked in
international organisations, asking them whether they found physics useful in their work.
People were evenly divided, with about half of the respondents believing that their scientific
background helps them a lot with the analysis of the situations they meet at work. Almost 30
years later physicists have become much rarer in international organisations, which are
increasingly staffed with social scientists and people with humanistic education. It would be
interesting to know how many of them find that their studies have been helpful (as compared
to the physicists of Haas et al (ibid)), and how many of them still read academic journals that
impressed them during their postgraduate studies.
References
Albrecht, Ulrich (1988) “Spin-off: A Fundamentalist Approach,” pp. 38-57 in Philip
Gummett and Judith Reppy, eds., The Relations Between Defence and Civil Technologies
Dordrecht: Kluwer Academic Publishers, 1988
Amann, Ron (2001) “Evidence-based policy: taking the vision forward” in OECD Social
Sciences for Knowledge and Decision Making , Paris
Beck , Ulrich (1999) World Risk Society, Polity Press, Cmabridge
Braun, Dietmar (1993) Who governs intermediary agencies? principal – agent relations in
research policy-making, Journal of Public Policy 13 (2) pp 135-162
Braun Dietmar and Martin Benninghoff (2003) “Policy learning in Swiss research policy-the
case of the National Centres of Competence in Research” Research Policy 32 (10) pp. 18491863(15)
Buisseret, T.J,. Cameron, H.M and Georghiou, L.G. (1995),“What Difference Does it Make?
Additionality in the Public Support of R&D in Large Firms”, International Journal of
Technology Management, Vol. 10, Nos. 4/5/6, pp. 587-600
Callon, M (1998) The Laws of the Markets. Oxford: Blackwell
Callon, M., Courtial, J.P., Turner, W.A., Bauin, S.,(1983);From translations to problematic
networks: An introduction to co-word analysis. Social studies of science 22:191-235
Caswill, C. (2003) ‘Principals, agents and contracts’, Science and Public Policy, vol. 30 (5)
pp. 337-346
Collingridge, David.(1980) The Social Control of Technology. New York: St. Martin's Press,
COM(2001) 428 final: “European Governance: A White Paper” European Commission;
Brussels
COM(2002) 276 final “Communication on Impact Assessment” European Commission
Brussels
Commission on the Social Sciences, (2003), “Great Expectations: the Social Sciences in
Britain”; Academy of Learned Societies for the Social Sciences, London
Chris Freeman and Francisco Louçã (2002) As Time Goes By: From the Industrial
Revolutions to the Information Revolution; Oxford, Oxford University Press
Davies, Huw, Nutley, Sandra and Walter, Isabel (2005) Assessing the impact of social
science research: conceptual, methodological and practical issues, A background discussion
paper for ESRC Symposium on Assessing Non-Academic Impact of Research, May, available
from http://www.st-andrews.ac.uk/%7Eruru/publications.htm
DiMaggio, P and Powell, W (1983) “The iron cage revisited: Institutional isomorphism and
collective rationality in organizational fields”; American Sociological Review 48: 147 - 160.
EC (2003) The overall socio-economic dimension of community research in the 5th
Framework Programme (1998-2002) Brussels, EUR 20577
Forbes, Ian and Abrams, Dominic (2004) “International social science research:
craft industry or baby behemoth?” International Social Science Journal 56 (180) pp. 189-192
Georghiou, L., Cameron, H., Stein, J.A., Nevada, M., Janes, M., Yates, J., Piper, M., Boden,
M., Senker, J. (1993) The Impact of EC Policies for Research and Technological
Development upon Science and Technology in the UK, HMSO, London
Georghiou, Luke, Rigby, John and Cameron, Hugh / eds (2002) Assessing the Socio-economic
Impacts of the Framework Programme (ASIF); Report to the European Commission available at
http://www.cordis.lu/fp5/monitoring/studies.htm
Gibbons, Michael, Nowotny, Helga, Limoges, Camille, Trow, Martin, Schwartzman, Simon
(1994) The New Production of Knowledge: The Dynamics of Science and Research in
Contemporary Societies, SAGE London
Guy, Ken; Amanatidou, Effie; Psarra, Foteini (2005 Framework Programme 5 (FP5) impact
assessment: a survey conducted as part of the five-year assessment of European Union
research activities (1999-2003); Science and Public Policy 32 (5) pp. 349-366
Haas, Ernst B, Williams, Mary Pat and Don Babai (1978) Scientists and World Order: The
Uses of Technical Knowledge in International Organizations.. Berkeley: University of
California Press,
Hall, Bronwyn (2002) “The Financing of Research and Development,” NBER Working Paper
No. 8773 (February 2002), available from http://elsa.berkeley.edu/~bhhall/bhpapers.html.
Johnston, Ron and Gummett, Philip (1979) Directing technology : policies for promotion and
control, New York: St. Martin's Press
Karlsson, Rasmus (2005). “Why the far-future matters to democracy today, Futures 37 pp
1095-1103
Kastrinos, Nikos (1998) The EC Framework Programme as Technology Policy: towards an
assessment; (PhD Thesis), University of Manchester
Kastrinos, Nikos (2001) Contribution of socio-economic research to the benchmarking of
RTD policies in Europe; Science and Public Policy, 28 (4) pp. 238-246(9)
Kingdon, JW (1984) Agendas, Alternatives and Public Policy, Boston: Little, Brown,
Kuhn, Michael and Remoe, Svend Otto (2005) (eds) Building the European Research Area:
socio-economic research in practice, Peter Lang Publishers, New York
Kuhn, Michael and Weidemann, Doris (2005) Reinterpreting Transnationality – European
Trans-national Socio-economic Research in Practice, in Kuhn and Remoe (eds) op. cit.
Lal, V (2002) Unhitching the disciplines: history and the social sciences in the new
millennium Futures, 34 (1) pp 1-14
Laredo, P (2001), Benchmarking of R&D Policies in Europe: Research Collectives as an
Entry Point for Renewed Comparative Analyses, Science and Public Policy, 28, (4), 285-294.
Larédo P. & Callon M, (1990) L’impact des Programmes Communautaires de Recherche sur
le Tissu Scientifique et Technique Français, Paris, La Documentation Française,
Liberatore, A (2001) From Science/Policy Interface to Science/Policy/Dialogue, in OECD
Social Sciences for Knowledge and Decision Making, Paris
Lindblom, Charles E. (1959). The science of muddling through. Public Administration
Review, 19, pp 79-88
Lindblom, Charles E. & Woodhouse, Edward J. (1993). The policy-making process (3rd ed.).
Englewood Cliffs: Prentice Hall.
MacKenzie, Donald (2005) Is Economics Performative? Oprion theory and the construction
of derivatives markets, paper presented to the annual meeting of the History of Economics
Society, Tacoma, WA, 25 June
MacKenzie, D. & Wajcman, J., Eds. (1985). The Social Shaping of Technology. Milton
Keynes, UK: Open University Press
Manicas, P. T. (2003) The Social Sciences: Who Needs 'Em?, Futures, 35,pp 609-619
May, Peter J (1992) Policy learning and failure, Journal of Public Policy 12 (4) 331-354
Nowotny, Helga (2005) The researchers perspective; contribution to a workshop on the Role
of the Humanities and Social Sciences in the Funding Portfolio of the future European
Research Council, Budapest, 11-12 February
Patricio, Teresa (2005) The Influence of the Framework Programme on the Social Sciences
and Humanities in Portugal, presentation at a conference on “Social sciences and humanities
in Europe: New challenges, new opportunities” organized by the European Commission,
Brussels, 12-13 December
Philips, A (1971) Technology and market structure; a study of the aircraft industry;
Lexington, Mass., Heath Lexington Books
Popper, Karl R (1957) “Poverty of Historicism” Routledge and Kegan Paul, London, reprint
1974
Richardson, J. J. and Jordan, A G.(1979) Governing under pressure: The policy process in a
post-parliamentary democracy; Oxford: Martin Robertson & Co
Rip, Arie (2001) “In praise of speculation”, in OECD OECD Social Sciences for Knowledge
and Decision Making, Paris
Rodrigues Maria-Joao. / ed (2002), The new knowledge economy in Europe : a strategy for
international competitiveness and social cohesion. Cheltenham: Edward Elgar.
Rogers, Everett M. (1962). Diffusion of Innovations. The Free Press. New York
Rose, Richard (1991) 'What is lesson-drawing?', Journal of Public Policy 11 (1) 3-30
Rose, Richard (2001) Ten Steps in Learning Lessons from abroad, Future Governance Paper
1, available from http://www.hull.ac.uk/futgov/Papers/PubPapers/RRPaper1.pdf
Rosenberg, Nathan (1976) Perspectives on technology; Cambridge University Press
Cambridge Mass
Schumpeter, Joseph A. (1934). The Theory of Economic Development. Cambride: Harvard
University Press
Schumpeter, Joseph. A (1939) Business Cycles (2 Vols.). New York: McGraw-Hill.
Solesbury, William (1994) Knowledge transfer: the medium and the message, Research
Evaluation, 4(1) pp55-58 ...
Stone, Diane (2001) ‘Learning Lessons, Policy Transfer and the International Diffusion of
Policy Ideas’. CSGR Working Paper 69/01: 1-41.available through
http://www.warwick.ac.uk/fac/soc/CSGR/wpapers/wp6901.pdf
Stone, Diane (2004) ‘Transfer Agents and Global Networks in the ‘Transnationalisation’ of
Policy’, Journal of European Public Policy, 11(3) 2004: 545-66.
Teece, David J (1986) Profiting from technological innovation: Implications for integration,
collaboration, licensing and public policy, Research Policy 15 pp 285-305
Vonortas, Nicholas S, and Hinze, Sybille / eds (2005) Special Issue on Evaluation of
European Union Framework Programmes: the 2004 Five-Year Assessment; Science and
Public Policy 32 (5)
Watkins, Todd A., "A Technological Communications Costs Model of R&D Consortia as
Public Policy," Research Policy (1991) 20: 87-107.
Weingart, Peter (2001)Paradoxes of scientific advice to politics, in OECD OECD Social
Sciences for Knowledge and Decision Making, Paris
Ziman, John (1987). Science in a steady state. The research system in transition,
(Science Policy Support Group)