Appendix B: Detailed evaluation approach

1. To what extent does current practice in the building consent system align with the objectives of the building consent system and the wider building regulatory system?
   • What are the objectives of the building consent system, and have they changed over time?
   • What do effectiveness, efficiency, and predictability mean in context of the building consent system, now and in the future?
   • How well in practice does the current building consent system align with the objectives of the system and the wider building regulatory system?
2. What are the underlying causes of problems in the building consent system?
   • What evidence is there that suggests these underlying causes are affecting the efficiency, effectiveness, and predictability of the building consent system?
   • Where in the system do they occur, and who is affected by them?
   • Are these underlying causes common across all BCA districts and sector stakeholders, or experienced by specific districts or groups?
   • How frequently do they occur, and how significant are the impacts?

Scope

Given the complexity of the building consent system, and broad nature of these evaluation questions, the project required a clear definition of what was in and out of scope (see Table 1). Further considerations were given to various COVID-19 alert levels, and the unprecedented level of activity in the building consent system at the time fieldwork was being undertaken.

Table 1: Evaluation scope

The relatively tight timeframes for the evaluation fieldwork meant that in-depth research across all regions was not an option. The evaluation team selected regions based on a range of inclusion criteria:

• Both rural and urban regions, and by extension larger and smaller sized BCAs
• Regions where BCAs were known to have faced challenges in the past
• Regions where BCAs were reported to be using innovative approaches.

Methodology

This section provides an overview of the individual methods used, and the approach taken to analysis.

Interviews

Interviews were carried out face-to-face, via telephone, or online using video chat, with a broad range of BCA and sector professionals. The interviews focused on respondents’ understanding of the objectives of the system, and how their experience aligned with that intent. Respondents were asked to consider not only their role, but how they engaged with other stakeholders in the building system. Interviews were conversational in nature, rather than a strict question and answer format, providing respondents opportunities to elaborate on things they felt most strongly about. In line with the key evaluation question, these conversations included exploring the underlying causes of any issues respondents identified.

Fifty-nine individuals were interviewed over 43 interviews (see Table 2 for more detail).

Table 2: Interview count by respondent group

Focus groups

Focus groups provided an opportunity to bring together a diverse group of stakeholders from across the building consent system, providing the opportunity to test out ideas and identify different perspectives. The focus groups started by asking respondents to outline their understanding of the intent of the building consent system, before comparing that with how the system currently functions. This enabled the group to collectively explore where there were issues in the system, and to discuss the underlying causes of these issues.

Five focus groups were held, attended by a total of 41 BCA and sector professionals (see Table 3 for more detail). Focus groups included between four and 12 participants, and were held in Auckland, Wellington, wider Wellington and Hutt Valley region, Christchurch, and Manawatū.

Table 3: Focus group attendees by respondent group

Surveys

Given the nature of the evaluation questions, data collection was primarily qualitative. However, two surveys were carried out to provide some level of quantification for the issues being raised in interviews and focus groups. Two very similar questionnaires were sent to a range of stakeholders involved in the building consent system, one to BCAs, and a second to sector professionals (builders, engineers, architects, and designers).

The objectives of the surveys were to identify key underlying causes of problems in the building consent system, the extent to which they were an issue (a big issue, small issue, no issue), and to get a sense of the most important issues overall. In addition to asking about specific issues, the survey also included open text questions, providing the opportunity for respondents to identify issues not explicitly asked about in the survey.

For further detail about the survey method refer to the Building Consent System Evaluation Survey report.

Building site visits

Members of the evaluation team undertook site visits to both BCAs and a small number of building sites in order to engage with stakeholders in their own space. These site visits were often combined with travel associated with facilitating focus groups. Visits were made to BCA and/or building sites in the following regions: Auckland, Christchurch, Wellington, Hutt Valley, Northland, Masterton, Palmerston North, and Levin.

Site visits gave evaluators a chance to observe how work related to building consents was undertaken, rather than asking respondents to describe it during an interview. This approach was less formal than interviews and focus groups, and allowed for impromptu conversations to happen in a more organic way.

Process analysis

A Business Analyst was brought into the evaluation team to explore how the building consent process functioned in practice. This involved comparing existing building consent process maps based on legislation, with the day-to-day practices in a range of BCAs. This work included a mix of observational work, informal interviews with BCA staff, and review of BCA process documents.

This work was intended to explore how well current practice aligns with that expected by the legislation, and the types of situations that would cause variations in process or ‘work-arounds’.

Research review

Issues with the effectiveness, efficiency, and consistency of the building consent system are not new. This topic has been researched extensively over the years, both within MBIE and by other researchers. This research was reviewed by the evaluation team, to ensure this project focused on addressing information gaps rather than repeating existing work. This review of existing literature helped shape the evaluation questions, particularly the focus on underlying causes of issues rather than simply describing surface-level problems.

For more information about the research reviewed as part of this evaluation see Appendix A.

Limitations

Low survey response rates

The BCA survey was sent to Building Services Managers or similar roles across all BCAs. Of the 71 surveys sent to BCAs, 28 responded, resulting in a 39 per cent response rate. Of the 28 who responded, 20 were classified as urban BCAs and eight rural BCAs. Compared to recent surveys of BCAs this is a relatively low response rate, and likely reflects the currently unprecedented levels of activity.

It is important to note that the sample of sector professionals is not a truly representative sample of building system professionals. The lack of a suitable sample frame for the construction sector meant that the evaluation team needed to take a pragmatic approach. The sector survey was sent out to members of selected professional bodies: the New Zealand Institute of Architects, Designers Association of NZ, Association of Consulting Engineers/Engineering New Zealand and Registered Master Builders Association.

Emails inviting individuals to participate in the survey were sent to approximately 10,000 professionals, and responses were only received from 263 individuals. Some of these mailing lists did not appear to be up to date, including invalid emails and contacts for people no longer operating in the sector. As a result of issues with the mailing lists it is difficult to accurately report response rates, but the overall response rate is estimated to be lower than 5 per cent.

Across both surveys, 291 responses were received. However, given the relatively low response rates findings from these surveys should be considered broadly indicative of trends rather than statistically accurate measurements.

The fieldwork did not include homeowners

There was a decision not to include homeowners in the fieldwork as significant projects that require building consents are typically “one off” or infrequent activities for this group. In addition, homeowners usually lack the ability and expertise to see what is happening “below the surface” of building work, which influences their perception of quality. They are also likely to rely on contractors to manage the processes involved in the building consent system.

The exception to this is the certificate of code compliance, which is the responsibility of the owner to obtain. However, feedback from other participants suggested that this is largely viewed as a tick-box exercise and was not seen to be a particularly concerning aspect of the consent process.

While some previous studies on owner experiences during the building process have been undertaken, specific understanding of the owner experience with the consent process could be an opportunity for further research.

< Appendix A: References and additional resources | Appendix C: Key events in the building consent system >