Two Mistakes in One Afternoon #

Imagine an ordinary afternoon in an office. Someone pastes a client's full contract into a chat window to "get the risks out of it". Nobody in the room blinks. A few hours later a status report lands on your desk. A machine has clearly summarized it at least twice. It reads well. It is also wrong in the one detail that matters.

Two incidents. Same office, same tools, same afternoon. It is tempting to file them as two separate problems. They are not. They are the same reflex, pointing in two directions. Underneath both sits a third cost that we are only now starting to measure.

We lose data. Then we lose judgment. Then we lose the ability to do the thinking ourselves. Each one is harder to get back than the last.

The Illusion of the Digital Confessional #

Many users treat chat windows like private diaries. They paste sensitive personal matters, internal business details and unchecked information straight into prompt fields.

In practice it looks like this. The customer contract pasted in "to summarize the risks". The production log with real email addresses still in it. The performance review someone wanted help phrasing. The architecture diagram covered by an NDA. The config file with a live key three lines down that nobody scrolled to.

Major AI platforms still use human review loops to evaluate and train models. On consumer tiers, conversations often go into training by default. Reviewers sample chats for quality and abuse. Retention windows outlive the conversation. Memory features carry context from one session into the next on purpose. Most of this happens through tools nobody in the company ever approved: the browser extension, the phone app used after the web version was blocked, the personal account opened because the corporate one felt slow.

Typing sensitive details into a standard chat window is like reading personal records aloud to a stranger on a crowded morning train. The privacy exposure is the same. If you would not share the information across a public carriage, it does not belong in a prompt box.

This has a legal side that people overlook. Trade secret protection in Germany requires appropriate confidentiality measures. You have to take steps to keep the secret a secret. Paste it into an unapproved public chatbot and you hand opposing counsel a decent argument that it never qualified as one. Under GDPR, personal data sent to a tool with no processing agreement is your organization's violation. It is not the vendor's. You do not only risk the leak. You risk the legal status of the thing you leaked.

The exposure runs one way only. You can delete a file. You cannot un-train a model or un-see a human review. You cannot un-index a conversation link that a search engine already crawled. Deleting the chat deletes your view of it. It does not delete the copy.

No privacy setting fixes this. The decision is not made once in a settings menu. You make it fresh, in the half second before you press Enter. My test is crude, and it works: would I be comfortable if this exact prompt showed up as a screenshot in a company-wide channel? If the answer needs thinking about, the answer is no.

The Same Reflex at Both Ends #

Look at what causes that first failure. It is not ignorance about data protection. A window that answers in fluent first person feels like a colleague. It does not feel like a service running on somebody else's hardware. We hand it things we would never hand a stranger, because it does not feel like one.

The same misreading runs the other way. If it feels like a competent colleague, its answer feels like a colleague's answer: already checked, already reasoned through, ready to forward. It is not. It is a plausible sequence of tokens. It comes with the same confidence whether it is exactly right or badly wrong. The model has no tell.

The first mistake costs you data. At least you can describe that loss to a lawyer. The second one costs you something harder to put a number on, and much harder to get back.

The Broken Summary Chain #

AI speeds up repetitive engineering tasks, fast code drafting, and targeted document searches. The risk starts when teams remove human judgment from the workflow.

A bad pattern is spreading across workplaces:

  • Team member A generates an unverified project summary with AI.
  • Team member B takes that summary, uses an AI tool to condense it further, and passes it upward.
  • The client receives a polished slide deck built entirely on unchecked assumptions.

Every summarization step is lossy. The danger is that it is not lossy at random. Caveats, hedges, conditions and dependencies go first, because they sit furthest from the central point of the text. Confidence survives compression. Doubt does not.

Imagine a line that goes in as "integration is blocked pending a decision from legal, best case two weeks". Two machine passes later it comes out as "integration on track". Nobody lied. Nobody made a mistake they could point at. The qualifier did not make the cut, twice in a row. What reached the client was not only inaccurate. It was more certain than anything anyone in the chain believed.

Underneath sits an old management failure with new tooling. Everyone in the chain assumed somebody earlier had checked. Nobody had. Automation bias does the rest. A confident answer from a machine gets less scrutiny than a hesitant one from a person. That is exactly backwards. AI did not invent diffusion of responsibility. It runs it at machine speed, with better typography.

When no one checks the output, errors multiply quietly. AI tools are not deterministic calculators that guarantee truth. They generate plausible token sequences. Relying on them without careful manual review leads straight to flawed business decisions.

Three habits stop the chain. Never summarize a summary. Go back to the primary artifact instead. Keep provenance attached, so the next person can find the source without archaeology. And treat a signature as a signature: whoever's name is on the slide owns every number on it.

Your Brain on Autopilot #

This part is not a process problem. It is a personal one.

In 2025 the MIT Media Lab put 54 people in EEG caps and had them write essays over four months. One group used an LLM, one used a search engine, one used only their own head. The LLM group showed the weakest neural connectivity in the networks tied to memory and attention. They also produced the most generic prose. The finding that should stop you is this: they struggled to quote back sentences from the essay they had finished minutes earlier. They had submitted the work. They had not retained it. The authors call the effect cognitive debt. If you have ever argued about technical debt in a sprint planning, you know how that story ends.

The same study also contains the fix. Some participants wrote unaided first and brought the model in afterwards. They showed stronger recall and wider brain activity than those who started with the model and later had to work alone. The tool is not the problem. The tool going first is the problem.

Microsoft Research and Carnegie Mellon surveyed 319 knowledge workers about 936 real tasks. They found the trade-off stated plainly: the more confidence someone placed in the AI, the less critical thinking they reported doing. Trust and scrutiny move in opposite directions. The better the tool feels, the less of yourself you bring to it.

None of this is entirely new. Back in 2011, Sparrow, Liu and Wegner showed what they called the Google effect. When people expect to have access to information later, they remember where to find it instead of the thing itself. We have outsourced memory to machines for fifteen years and mostly got away with it. What changed is the depth of the outsourcing. With a search engine you still opened the sources, compared them and decided which one to believe. With a chatbot the sources never appear. You end up remembering neither the fact nor where it came from.

If that sounds abstract, look at the version you can already feel. A 2020 study in Scientific Reports found that heavier lifetime GPS use predicts worse spatial memory, dose-dependently. The researchers ruled out the obvious objection. It was not that people with a poor sense of direction used GPS more. Using it made them worse. Many of us struggle to navigate a city we have lived in for a decade without it. That is not a metaphor for what happens to reasoning under heavy AI use. It is a preview.

The mechanism is not mysterious. The struggle is the encoding. Wrestling with a hard problem is not an annoying delay before the answer arrives. It is the process that builds the structure you need next time. Skip the difficulty and you do not get the result faster. You get the result without the learning. So you need the tool again tomorrow, and a little more of it. That is a loop, and it tightens.

The web is filling up with interchangeable, smoothed-out articles that all follow the same rhythm: safe intros, predictable lists, shallow conclusions. It is the same effect at civilizational scale. I have written before about where that road leads. Progress needs friction. When leadership defaults to automated consensus instead of critical reasoning, decisions reflect convenience, not deep competence.

Tools Assist, Humans Decide #

AI belongs in engineering pipelines and everyday operations as a support tool, not as an outsourced brain. Automation should remove repetitive overhead. Full ownership of the final verdict stays with human experts. I am not arguing for less AI. I use it every day. I have made the case that, managed properly, it raises the floor for everyone. I am arguing for keeping ownership of both ends of it.

The rules I hold myself to:

  1. Classify before you paste. Public, internal, never. Decided before the prompt, not after.
  2. Brain first, model second. Form your own answer, then ask, then compare. This is the most valuable habit on the list. It is also the one the EEG data supports.
  3. Never summarize a summary. Go back to the source, or do not send it.
  4. The name on it owns it. If you forward it, you have read all of it.
  5. Use it as an adversary, not an oracle. "Attack this reasoning" beats "what should I do" every time.
  6. Keep one hard thing a week tool-free. Do it on purpose, even when it is slower. It is training. You will notice the difference inside a month.

Dropping systematic review produces clean surfaces with little substance underneath. The discipline at both ends is the same discipline. Stay the one who decides what goes in. Stay the one who decides what comes out.

Sources and further reading #