Use and outputs

Artificial intelligence (AI) tools can be helpful, but they don’t always get things right. That’s why it’s important to think carefully before using AI results to make big decisions – especially ones that affect people’s lives. AI should support the work people do, not replace their judgement. It’s best when humans and AI work together, with people staying responsible for the final choices.

GenAI user inputs

* GenAI tools work by responding to prompts – the questions or instructions you give them. The information included in any prompt (for example the images you provide, or text you type into an LLM) can affect what the tool gives back, and sometimes even how it learns and responds over time.

• * Some GenAI tools, particularly those that are publicly available or free of charge, may share prompt data (including attachments) with external parties (including developers or other users). This means information provided through prompts could be exposed in future through a security breach or the model itself.
• * Even if your prompts or attachments don’t include names or are otherwise anonymised or depersonalised, personal, sensitive or classified information could be joined up with other information over time to re-identify someone.

Use of GenAI outputs

* GenAI can be used in a variety of ways including to improve customer experience, for marketing and content creation, and for productivity gains. However, there are some potential risks when it comes to GenAI outputs and their use:

• * LLMs are designed to generate “statistically probable language patterns”, which means different (though often similar) answers are likely to be given to the same prompt. Businesses should therefore not rely on any specific answer being produced in response to a specific prompt.
• * LLMs may be susceptible to errors, omissions, training bias and factual inaccuracy. While responses can seem well developed and credible, they may confidently present opinion as fact, skip important details depending on the prompt given, and in some instances fabricate information as truth. This tendency is referred to as ‘hallucination’.
• * Depending on the training data and prompt used, outputs may also reflect prevalent, often biased, out-of-date or unethical viewpoints. These tendencies could impact Māori and other Indigenous, minority, or otherwise disproportionately disadvantaged communities (including women, older people, young people and children, disabled people) disproportionately.
• * Some GenAI tools that generate realistic images or videos of a person or their voice, can be used maliciously. These are called ‘deepfakes’ – realistic but fabricated audio, video, or images that convincingly mimic real individuals. Malicious actors can use them to spread misinformation, defame individuals, or enact scams and exploitation. The technology has also been misused to create explicit content.
• * GenAI can be, and is often used, to generate new and novel content and material. However, outputs may lack commercial protection or ownership (including given issues assessing level of ‘human authorship’ and establishing originality). There is also always a risk that model outputs may (intentionally or inadvertently) be substantially similar to existing copyright works (whether or not those works were used to train the model) and therefore be potentially subject to plagiarism and copyright infringement claims.
• * GenAI has been used to replicate components of Māori culture and traditional knowledge without appropriate permission and attribution. New Zealand’s intellectual property laws include provisions for the protection of mātauranga Māori, in relation to patents and trade marks. These provisions help prevent the registration of trade marks or granting of patents that would be considered offensive by Māori or contrary to Māori values. Te Puni Kōkiri (the Ministry of Māori Development) is working to address regulatory barriers to commercialisation of mātauranga Māori (traditional knowledge).

Human-in-the-loop decision-making

AI systems, and particularly those using advanced machine learning methods such as deep learning, are often treated as ‘black boxes’ due to their complex inner workings.

Depending on the way an AI system is being used, some level of real-time human intervention may be needed. In most situations, the level of human review will depend on the proven level of reliability of the system to deliver desired outputs, combined with a business’s assessment of their specific tolerance for the impact of inaccuracy.

If an AI system is doing something low-risk – like suggesting a movie or a product – it might not need a person to check its work. This is often referred to as ‘human-out-of-the-loop’.

If an AI system is helping to make big decisions – like those about money, health, or the law – a person should always be involved. This is called ‘human-in-the-loop’. The level of human involvement will likely depend on the risk associated with those decisions, or the cost of harm of a ‘wrong’ decision. This can range from having a human consistently make the final decision (informed by an AI output), or having human oversight over particularly high-risk or low-confidence AI results.

For example, an AI system may be used to analyse transaction patterns to detect fraud, but given the potentially high cost of incorrect decisions (which could lead to financial losses, regulatory penalties, or reputational damage), high-risk transactions are flagged for human review. This can be considered ‘augmented intelligence’ – the AI assists with scanning transactions for suspicious patterns, but the fraud analyst will make the final decision.

Humans can over-rely on automated systems or algorithms and accept outputs as correct without critical evaluation. This is ‘automation bias,’ and can undermine the purpose of having human oversight as individuals are not effectively acting as an independent check.

Overall, regardless of the supporting technology used, businesses are responsible for their decisions, so robust checking and due diligence, including adequate recordkeeping and documentation, is crucial.

< Data and modelling | Continuing the conversation >