2020 Fellowships Impact evaluation

Pūkeko Research Ltd are conducting an impact evaluation of Fellowships funded through the Vote Business Science and Innovation funding.

We have commissioned Pūkeko Research Ltd to conduct an impact evaluation of the Crown’s Vote Business, Science and Innovation funding of Fellowships. The seven Fellowships are jointly administered by our partner agencies, the Royal Society Te Apārangi, Fulbright New Zealand and the Health Research Council.

Pūkeko Research Ltd will liaise with Fellowship recipients, applicants and key sector points of contact as required.

Evaluation purpose

The overall purpose of the impact evaluation is to determine the impact of the fellowships funded from Vote Business, Science and Innovation, and their alignment with their original policy intent. This is expected to be carried out within 2 main frames of reference:

  • Individual: To assess the seven Fellowship investments, against their individual terms of reference. This will identify whether the schemes are meeting their current objectives, and determine if the policy settings (i.e. fund amount, eligibility and any restrictions on use) are appropriate and if objectives could be better supported.
  • Portfolio: To determine the collective success of the seven Fellowship investments against their overall policy objectives.

We will use the findings of this impact evaluation to:

  • demonstrate the impacts of the Fellowships
  • report on value-for-money of investment in Fellowships
  • identify learnings to inform the future design changes, and sustainability of the Fellowship programmes (individual and/or portfolio)
  • provide independent advice that will inform MBIE’s future investments
  • demonstrate accountability of the use of funds to the New Zealand public and research sector.

The expected output of this evaluation is a review report, clearly identifying the strengths and any issues relating to the performance and impact of the Fellowship investments. The report is expected to recommend possible solutions to those issues within the context of overall support for early-mid career New Zealand researchers.

Evaluation principles

An impact evaluation is considered to be the most appropriate approach. The evaluation will examine how effectively the fellowship investments have delivered on expected achievements.

Along with our partner agencies we note the need for the evaluation principles to account for the varying measure of impact and excellence that exist between individual fellowship programmes. Accounting for these differences is expected to be achieved by ensuring that the evaluation is fully contextualised against the specific objectives of each fund, through the individual frame of reference approach, earlier described.

Where differing measures of impact are sought by an individual fund, provision for this can be made in the assessment. In 2019 we published an impact of research position paper [PDF, 1.5 MB], and see also our 2018 Diversity in Science statement [PDF, 888 KB].

The impact of the investments will be determined by looking at the schemes’ effectiveness to date, but also considering current context, future needs and alignment with the overall research and governing strategies.

Scope of the evaluation

The Fellowship programmes in scope for this evaluation are:

  • James Cook Research Fellowship programme
  • Rutherford Discovery Fellowship programme
  • Rutherford Foundation Trust New Zealand Postdoctoral Fellowship programme
  • Fulbright New Zealand Science and Innovation Graduate Awards
  • Sir Charles Hercus Health Research Fellowship programme
  • Māori Health Research Post-Doctoral Fellowship programme
  • Pacific Health Research Post-Doctoral Fellowship programme

Fellowships and Scholarships not listed above will not be considered as part of the evaluation activity. The evaluation is also not expected to provide a gap analysis or any detailed policy development for any changes it recommends.

Evaluation process

The evaluation approach is expected to include mixed methodologies to support a robust quantitative and qualitative assessment.

The 4 main areas that will evaluation will focus on, at both the individual and portfolio level, are:

  1. What impact have the fellowship investments had on people working in science, research and health?
  2. What impact have the fellowship investments had on research outputs and new innovations?
  3. What impact have the fellowship investments had on the science, health and research sectors?
  4. What impact have the fellowship investments had on global connections, research links and partnerships for New Zealand’

The questions may be adapted or added to as necessary.

The structural content of the final evaluative report is expected to contain recommendations to outline strengths, weaknesses and best practice for Fellowship design internationally. The report may also provide supporting evidence for the recommendations identified by this evaluation. International exemplars may also provide best practice on evaluation methodologies, and the pool of applicants and awardees from which to generate randomised samples.

We will publish a summary version of the final report which is expected in July 2020.

More information

For more information, contact: media@mbie.govt.nz