7. Overview of methodology
On this page
The quality of the insights that can be drawn from administrative data is influenced by data limitations including lack of data, and inconsistency in data collection and definitions. Note also that the data used in the analysis is self-reported information. Consistent methods for analysing the data within each fund have been applied to mitigate these limitations where appropriate.
7.1 Overview of method
Analysing key research questions through data analysis required an iterative approach, capturing dollars awarded, project counts, and Māori researcher and entity involvement:
- A series of key research questions were framed, and a select number of funds were included in scope to match needs and resources.
- Information and data were sourced taking an iterative approach to identify what insights could be pulled from the funding data. MBIE facilitated the sharing of raw data, responding to a series of data requests.
- Excel and R was used to clean and analyse the data in line with the key research questions, and data available.
Key research questions for this data analysis
How much RSI funding is awarded to projects that explore the following?
- People – funding Māori workforce/s, building the capacity and capability of the Māori RSI workforce
- Knowledge – mātauranga Māori (including kaupapa Māori) and topics that support positive outcomes for Māori
What gaps are there in our monitoring? What should be collected and how should the information be used?
Sources of information
- MBIE’s internal Information Management System (IMS)
- Reports provided by the Royal Society, Callaghan Innovation, and the Health Research Council
- Callaghan Innovation’s User Product Experience Dashboard
- Relevant funding websites
- Direct data requests
Years in scope
The 2018, 2019 and 2020 funding rounds. Application, funding, and reporting years differ slightly across funds, but generally, and for the purposes of this analysis are:
- 2018 funding round = 1st July 2018 to 30th June 2019 = FY19
- 2019 funding round = 1st July 2019 to 30th June 2020 = FY20
- 2020 funding round = 1st July 2020 to 30th June 2021) = FY21
7.2 Method for analysis
The number and value of projects awarded funding
VMCF, Endeavour, NSC, SSIF, and Catalyst – MBIE funds
Data was sourced from MBIE’s funding administrative data (collected through the MBIE’s Integrated Management System).
For each fund, unique projects and the associated amount awarded was found for each funding year. Note that the money may have been awarded in a particular year but the project itself runs over multiple years. The project is only counted in the year it was first awarded money (within the 2018 to 2020 scope).
SSIF and NSC funds are contracted differently to other MBIE funds. Projects may run over many years, but funding is sought each year with a new project ID created. For fair comparison with other funds in scope of this analysis, a unique project count was created (using the ‘original ID’), and the amount that was funded across the 2018 to 2020 years for each unique project was aggregated.
The project and amount awarded over 2018 to 2020 is then only shown in the year that the project first appears.
Growth, Student, and Project Grants – Callaghan Innovation
Data was sourced direct from Callaghan Innovation, using their funding administrative data.
Grant counts are by ‘Contract ID’, assuming each unique contract ID corresponds to a unique grant awarded. ‘Start Date’ was taken as the date for which the grant was awarded, and this was allocated into equivalent funding round years. To enable fair comparison with MBIE funds, the ‘Forecast Value’ figure for the contract was taken as the amount awarded for each new grant (though in practice this might have been expensed over a number of years for the project).
Marsden – Royal Society
Figures were taken from the Royal Society New Zealand Data Report files for the report years 2019, 2020, and 2021 (which are for the 2018, 2019, 2020 funding rounds, respectively). These reports were provided by MBIE and the Royal Society.
Specifically, data on the ‘MF DR’ sheet from the excel workbook that forms part of the report packages was used the analysis. The number of ‘new’ contracts for the relevant year was taken. For consistency with other funds, the ‘total contracted’ figure was used to give the amount awarded for each new contract, for the funding year (even though the project or contract might run over a number of years).
Māori researchers involved in projects awarded
Overview
Ethnicity data has been sourced from MBIE funding-related administrative data, where some funds seek ethnicity information from contract holders (though responses are optional).
For the purposes of this analysis, `Māori researchers’ captures any individual who self identifies as being of Māori ethnicity, and plays a key role in the project as listed in ‘role’ data. Individuals may be of Māori ethnicity but not deem themselves to be a Māori researcher.
Researchers who do not identify their ethnicity as Māori but are considered a Māori researcher, are not identifiable in the data.
The number of Māori researchers was found by linking people and ethnicity data with project identification data, which was linked back to funds and years. Individuals self-identify their ethnicity (selecting one or more ethnicities), or they may choose not to. By default, if one of the ethnicities selected by an individual was ‘Māori’ this analysis deemed them to be Māori.
Ethnicity data started being collected in 2018, and this data collection became more consistent from 2019. Depending on the fund, between 22% to 33% of people listed as having a role on projects, across 2019-2020, did not have ethnicity data. Unless directly obvious, it has been assumed that the same names appearing multiple times are the same individual.
VMCF and Endeavour
The ethnicity of ‘key researchers’, ‘key individuals’, and ‘science leaders’, was grouped into ‘non-Māori’, ‘Māori’, and ‘not stated’, and linked to projects (using the original project ID). Where individuals appeared more than once, the string of data that indicates ethnicity was taken if the other data string did not. If both indicated ethnicity, the most recent identification data was used.
Catalyst
The same method was followed as above, but due to role definitions differing, ‘Māori researchers’ are ‘Lead or co-lead’, ‘Collaborator’ ‘Other contributors’, ‘Technician’ and ‘Supporting staff’.
SSIF and NSC
The same key method and definitions would have been applied as in the Catalyst fund. However, ethnicity data for individuals in the SSIF and NSC funds is not collected so this analysis does not present ethnicity data for these funds.
Identifying ‘unique’ researchers
Spell cleaning was used to identify unique individuals – removing those that exist in a year, and a fund, and are on multiple projects to ensure these individuals are only reported once, as appropriate to the insights being presented.
Projects indicating relevance to Māori RSI, or an alignment with Vision Mātauranga
VMCF, Endeavour, and NSC
In these funds, applicants are asked about the project’s/ programme’s relevance to Māori, noting the proportion (out of 100) of the project that falls into the following categories:
- research not involving and not specifically relevant to Māori
- research specifically relevant to Māori
- research involving Māori
- Māori-centred research
- kaupapa Māori research.
Sometimes there is a preceding question asked: ‘Will this project make a significant difference to Māori research and innovation?’ A ‘yes/no’ answer is available to be selected.
To find the projects indicating relevance to Māori, each of the projects that selected ‘no’ to the above question were removed from the data set. Any project that noted ‘yes’ to the preceding question but had allocated ‘100%’ of their project relevance into the ‘research not involving and specifically relevant to Māori’ category was also removed, as this selection is deemed to mean that the project is not truly relevant to Māori.
With some further analysis to identify unique projects, the remaining count gave the individual number of projects of relevance to Māori RSI.
As a following step, a closer analysis was undertaken to identify projects that noted 50% or more of the project had relevance to ‘kaupapa Māori research’ (as defined in MBIE’s funding profiling categorisation questions). Key details about the projects, such as project title, description, lead organisation, and contract value, were analysed to provide additional insights on the projects that indicated they took this approach.
No profiling categorisation data was available at the time of this analysis for VMCF projects in 2020, so the number of kaupapa Māori projects in the VMCF is likely underestimated over the 2018 to 2020 period.
Note: Projects were only captured in the year they were first seen in the 2018 to 2020 snapshot, for consistency with the rest of the analysis on project numbers.
SSIF
The SSIF asks the same profiling questions around contracts/projects making ‘a significant difference to Māori research and innovation’ as in the VMCF, Endeavour, and NSC funds. However, instead of being able to allocate a percentage effort to each category, SSIF contract holders are asked, ‘Which Vision Mātauranga category [from the 5 categories] best describes the project?’ They are only able to select one category. This has given the number of SSIF projects aligned with each profiling category, including kaupapa Māori research projects.
Marsden
Marsden contract holders are asked whether the research project aligns with one or more of the 4 themes of Vision Mātauranga. They are able to select, none or more than one. Data on each new contract and their VM alignment selections were drawn from the Royal Society New Zealand Data Report files for the report years 2019, 2020, and 2021 (which are for the 2018, 2019, 2020 funding rounds, respectively).
Unable to analyse
We weren’t able to quantitively analyse alignment for the following funds as they do not ask specific profiling questions as in the other MBIE administered funds. Open text analysis would be required to provide a picture of alignment or relevance for these funds.
Marsden contracts that are Māori-led
Marsden
Data was sourced directly from the Royal Society through data requests. The number of Māori-led projects for new contracts awarded in the 2018, 2019 and 2020 funding years was sought. This number has been calculated by the Royal Society, based on a definition of Māori-led as ‘at least 20% of the key applicants (provider institutions) in the project identify as Māori.’
VMCF 2013 to 2020
The same method used to provide insights on the VMCF over the 2018 to 2020 period was simply extended to provide insights over a longer time frame, from 2013 to 2020.
Māori businesses involved in Callaghan grants awarded
Callaghan Innovation project, growth, and student grants
Data was sourced directly from Callaghan Innovation, and through their User Experience Dashboard.
Callaghan Innovation collects information on the types of businesses that apply and are awarded R&D related grants. Entities include: A Limited Partnership registered under the Limited Partnerships Act 2008, a Māori Incorporation or a Trust established under Te Ture Whenua Māori Act 1993, a Trust established on behalf of Māori claimants to receive and manage assets as part of the settlement of a claim under the Treaty of Waitangi, a Māori statutory body, and a business that is controlled by one or more of the above types of Māori entities.
Callaghan provided a list of contracts and noted against each whether it was awarded to a Māori business or a non-Māori business. This figure was then counted across the funding years, and grant types. Businesses that were awarded multiple grants across the years in scope were identified, to give individual business counts.
Callaghan Innovation only recently started collecting data on Māori/non-Māori businesses through the Customer Navigation function, so some data may be underreported for the 2018-2020 period for this analysis.
Experience information was explored using Net Promoter Score information and general comments provided by Māori and non-Māori businesses through Callaghan’s User Product Experience Dashboard, which notes whether a customer/client (and for the purposes of this analysis, successful grant holder), is a Māori business or not.
Note: the administrative contract data, and the experience information from the UPE dashboard were not linked or matched as datasets as part of this analysis.
HRC data – example for comparison purposes
Health Research Council grants
Data was sourced from HRC; we did not do any further analysis. The report provided an overview of the research HRC had funded in the 2020/21 financial year and the research that had relevance to Māori.
Data was drawn from figures presented in tables and summaries in the report, alongside key points on the website. Conclusions have been drawn from the data and supporting information. This analysis did not review the activities of the HRC to substantiate their performance.