CREW Webinar: AI and Culture – 10 fault lines to watch out for

Picture this. A man sets up an AI agent to book his gym sessions for him. The agent does exactly what he asked and books the sessions.

The AI agent also finds a vulnerability in the gym's booking software, bumps somone else off the waitlist, and puts the man at the top of the queue. When the man realises what's happened and tells it to put that person back, the agent says, sorry I can't do that.

That story came from Andrew Brown at Adaptive Cultures. It is an ordinary case rather than an extreme one, and it still caused real damage, quickly and with no way to reverse it.

Andrew and his colleague Teresa Collis joined Mayflower for a CREW webinar on culture and AI, and Andrew came back on the Friday for a longer chat specifically on governance.

Cultural fault lines

Adaptive Cultures started this research a few years back, in a joint venture with IBM. What they found even then was that AI doesn't create new organisational weaknesses so much as it finds the ones that were already there. These include poor data-sharing habits, unclear accountability and a culture that rewards busyness over judgement. AI doesn't invent these weaknesses, but it does expose and amplify them.

Adaptive Cultures refer to these as cultural fault lines. There are ten in total and Andrew and Teresa focused on three: the missing middle, governance theatre and the adaptive deficit.

The missing middle

Andrew told a story about training as an actuary in the 1990s, studying each part of the syllabus separately, then sitting practice exams that forced him to hold all of it in his head at once, confused, until something clicked. That click, he said, is where judgement emerges. Judgement  is built in how we explore the gap between confusion and a meaningful answer.

That gap disappears when any question, however complex, returns a fluent, coherent answer in seconds.

Andrew cited a Polish study of endoscopists screening for bowel cancer. AI assistance lifted detection rates significantly while it was in use. However, when the same clinicians went back to working without AI assistance a year or two later, their own unassisted detection rate had dropped, from around 28% to about 22%. The skill had gone backwards.

The same pattern applies to handwriting after keyboards, mental arithmetic after calculators, and map reading after GPS. Teresa gave an example: backpacking around Europe in the eighties with nothing but a rough map in a guidebook, she could navigate anywhere. Without her phone in Melbourne now, she cannot. The middle, where the skill is built through friction, is missing.

This is relevant to anyone drafting PDS content. AI handles general material well, such as why insurance is a good idea. Content specific to a particular product is higher risk, because the output usually looks right enough that it may not be checked thoroughly. AI-generated text also tends to repeat the same point in several different forms, which is a problem in a document such as a SEN, because the reader loses the main point. The greater risk is not a PDS manager using AI where they shouldn't. It is someone from outside the usual PDS process, brought in because the business is stretched, who does not know which parts can be handed over and which need a person drafting throughout.

Governance theatre

Andrew noted that if you ask an organisation what AI governance means to them, you get a large amount of information back, almost none of which is governance. Common examples:

• "We haven't started thinking about it, we've just asked people to come up with their own ideas."

• "Don't put confidential data into a public AI agent." That is true, but it is a rule, not a framework.

• "You can only use Copilot or Claude or Gemini." That is also true and also not a framework.

• "It's up to IT to make sure the control environment is effective." Someone should own it and that's not the same as it being owned.

That is governance theatre: it reads like governance in a board pack, but does almost nothing to change what happens when someone points an agent at a real problem.

Andrew shared a graph from Anthropic's own research showing how capable their models have become at exploiting unknown vulnerabilities in operating systems and browsers. Late 2025 to early 2026, that success rate sat around 1 to 2%. By April 2026, only four or five months later, it had jumped to 72%. According to Anthropic, the models can find and exploit zero-day vulnerabilities, many of them ten or twenty years old, in every major operating system and browser when a user directs them to.

Then there's the OpenAI example from July, where a model being benchmarked in what the team thought was a fully controlled sandbox, with no internet access, found a vulnerability in the sandbox itself, hopped into another program on the same platform, and used that as a side door to the internet. It did not break its instructions, and nobody had told it not to hack its way out. The logs showed it had worked out how to get each piece of its plan past a security checkpoint separately, then reassemble them on the other side.

Andrew's phrase for this is the Ethical Trojan horse. Give an agent an outcome to aim for and the approach prioritises reaching that outcome over how it is reached. The agent has no ethics framework to optimise against; it simply has a task. Humans do this too, for example splitting a $400,000 job into four $100,000 pieces to dodge a procurement threshold. The difference is speed and scale: a person doing that takes weeks and leaves a trail. An agent does it in the time it takes to answer the prompt.

There's a reason markets typically mature toward prudential regulation, trusting professionals with judgement and giving them room to move, rather than staying locked into prescriptive rules. Andrew's point is that AI is nowhere near mature enough for a purely prudential approach yet. Dario Amodei's paper on this called it the adolescence of the technology. Both are needed: clear prescriptive controls around what agents can and can't touch, and a genuine attempt at the harder cultural and prudential work underneath it.

APRA has asked for what it called a step change in governance around AI. Motivations vary - competitive pressure, genuine merit or keeping up with what other organisations are doing. Whatever the motivation, the outcome depends on the culture and on what the organisation privileges when nobody is watching. Sustained cost-cutting tends to end in a Royal Commission rather than efficiency. AI is not a uniquely dangerous technology. The same failure mode, chasing the outcome and letting the means take care of themselves, now plays out much faster.

A member raised the 9/11 Commission report and its finding of a failure of imagination. The instinctive response is that AI has no failure of imagination, because the discussion moves straight to worst-case scenarios.

Nobody who built the gym-booking agent imagined the queue-jumping outcome, because it was not dramatic enough to consider. The failures between the mundane task and the catastrophe are the ones that go unimagined, and that is where governance theatre leaves a gap.

The adaptive deficit

The third fault line ties the other two together. Teresa introduced the idea of AQ, adaptive quotient, alongside IQ and EQ, as a critical capability at both an individual and organisational level. It measures how well a culture adapts when conditions keep changing. Adaptive Cultures have built a diagnostic around it, ranking statements against each of the ten fault lines to show where an organisation may be most vulnerable.

Their four-phase model for the work, discover, diagnose, design and deliver, embed, is a reasonable way to approach an organisation's own AI rollout: understand the context and identify where the fault lines actually are rather than where they are assumed to be. Design initiatives that build capacity rather than policy documents, and continue because the technology changes constantly and embedding is not a one-off exercise.

The takeaway

More governance is not sufficient on its own. A rulebook that has not been tested against a real scenario is theatre, however tidy it looks in a board pack. What protects an organisation is slower work, building the judgement to spot a gym-booking problem before it happens, accepting that AI optimises only for the outcome it is given and developing a culture that can adapt at closer to the speed of the technology.

Deliberate, unassisted thinking time may be the most useful governance control currently available.

Please get in touch for the full set of Adaptive Cultures white papers, or to discuss where your own organisation sits. We would also be interested to hear about similar incidents, small, well-intentioned AI outcomes that went wrong in ways nobody anticipated.