People & Science

A publication of the British Science Association

23/10/2014

Show me content for... +

Show me content for...
Events
Resources
Volunteers
Teachers
Professional development
Families & teenagers (aged 12+)
Families (children aged 12 & under)

Donate

register

Register with us and you can....

  • Sign up to our free e-communications
  • Become a member of the Association
  • Create your own web account, & post comments
  • Be part of British Science Festival
  • Save your favourite items

Register

Keep up to date with the latest news from the British Science Assocation. Sign up to our RSS feeds and take us with you when you are on the move.

You are here

We need to evaluate engagement

It’s a complex social process, argues Eric Jensen

The ‘impact’ component in the UK’s upcoming Research Excellence Framework (REF) is raising interest among scientists about increasing their public engagement activities and ensuring they can evidence their impacts. This interest is well-justified given the positive impacts science engagement can offer. For example, my research with visitors to the Cambridge Science Festival shows it is deeply valued. Visitors appreciate the access to cutting edge scientific knowledge and scientists’ openness, enthusiasm and eagerness to engage. However, my prior research shows that science engagement can also lead to negative outcomes, if poorly designed or executed.

Training for engagement

As an experienced evaluation researcher and social scientist, I have seen the full range of effective and ineffective practices within UK science engagement. Enhancing quality - alongside increases in quantity - is achievable, but requires changes in training and evaluation.

At the level of scientists’ preparation for undertaking public engagement, training in the most important and relevant lessons from across the social sciences could improve the likelihood of positive interactions with publics, while limiting the risk of re-enacting long discredited practices. Incorporating a précis of social scientific knowledge covering media literacy, audience reception, learning, communication and sociology as an essential, if necessarily brief, element of scientists’ qualifications could substantially enhance the quality of science engagement downstream, as well as enriching scientists’ education.

Evaluation needed

While understanding principles of good practice is a reasonable foundation for most scientists delivering quality public engagement, those undertaking science engagement frequently or as a full-time career should also make sure that the impacts of their engagement are rigorously evaluated.

Quality evaluation of audience outcomes not only provides evidence of impact for the REF and other institutional requirements. It can also be a crucial mechanism for avoiding the risk of unforeseen negative outcomes. Good evaluation requires upstream planning and clear objectives, and its results should inform science engagement practice. It also requires additional training in relevant social scientific research methods (for example, survey design).

Standards low

Unfortunately, institutions sponsoring professionally supported or delivered science engagement activities do not consistently require high-quality evaluation of audience impacts, for example, approaching a standard suitable for publication in a peer-reviewed social scientific journal. Moreover, long-term impacts are hardly ever assessed.

My experience of most of the evaluation research routinely undertaken by professionals and consultancies in UK science engagement is that it rarely satisfies even the most basic methodological standards. For example, imbalanced ‘level of agreement’ scales that skew results towards a positive outcome are commonplace. Indeed, I have used such evaluation reports prepared by UK science communication and museum consultancies in my undergraduate ‘Surveys and Statistics’ module at Warwick University to exemplify poor practice. Within minutes, my students can spot fatal flaws that invalidate these expensive professional evaluations.

Understanding engagement

Part of the problem is that, in the relatively rare instances that science engagement projects commission full-scale independent evaluations, tenders are often assessed either by those whose work is being evaluated or by staff at the funding institution without relevant methodological expertise. Thus, even the professional side of science engagement is rife with shoddy methodology or no evaluation of actual audience impact at all (for example, mere attendance counts and a handful of positive audience quotations).

This failure to routinely ensure rigorous, falsifiable evaluations which are widely disseminated to enhance practice is hindering the field’s development and impact. Moreover, the tendency to limit evaluation to the engagement event’s duration (as opposed to long-term research assessing impacts over time) is unrealistically myopic and insensitive to contextual factors that modulate impact.

At every level, effective science engagement should be better understood as the complex social process that it is, with concomitant skill, thought and care taken in planning, delivery and evaluation.

Click for More
Dr Eric Jensen
Dr Eric Jensen is Assistant Professor of Sociology, University of Warwick
Join the debate...
Log in or register to post comments