Building Trust in AI Applications

BREAKING! The FDA just released this draft guidance, titled Artificial Intelligence-Enabled Device Software Functions: Lifecycle Management and Marketing Submission Recommendations, that aims to provide industry and FDA staff with a Total Product Life Cycle (TPLC) approach for developing, validating, and maintaining AI-enabled medical devices. The guidance is important even in its draft stage in providing more detailed, AI-specific instructions on what regulators expect in marketing submissions; and how developers can control AI bias. What’s new in it? 1) It requests clear explanations of how and why AI is used within the device. 2) It requires sponsors to provide adequate instructions, warnings, and limitations so that users understand the model’s outputs and scope (e.g., whether further tests or clinical judgment are needed). 3) Encourages sponsors to follow standard risk-management procedures; and stresses that misunderstanding or incorrect interpretation of the AI’s output is a major risk factor. 4) Recommends analyzing performance across subgroups to detect potential AI bias (e.g., different performance in underrepresented demographics). 5) Recommends robust testing (e.g., sensitivity, specificity, AUC, PPV/NPV) on datasets that match the intended clinical conditions. 6) Recognizes that AI performance may drift (e.g., as clinical practice changes), therefore sponsors are advised to maintain ongoing monitoring, identify performance deterioration, and enact timely mitigations. 7) Discusses AI-specific security threats (e.g., data poisoning, model inversion/stealing, adversarial inputs) and encourages sponsors to adopt threat modeling and testing (fuzz testing, penetration testing). 8) And proposed for public-facing FDA summaries (e.g., 510(k) Summaries, De Novo decision summaries) to foster user trust and better understanding of the model’s capabilities and limits.

Most people think having a human approve an AI decision means the decision is safe. It does not. 👀 There is a term for what actually happens when humans rubber stamp AI outputs under time pressure. Automation bias. It is one of the most documented and underreported risks in enterprise AI right now. After 13 years and 200+ deployments, here is what I have learned about building genuine oversight into AI systems. The human reviewing an output needs three things to actually be in the loop. They need to understand what they are reviewing. They need the context to catch what the model gets wrong. And they need to be genuinely empowered to say no without institutional pressure to simply keep moving. Most organisations have none of those three in place. They have a signature process. That is not the same thing. Before any high-stakes AI output reaches a decision point in your organisation, ask these questions. ➡️ Does the person approving this understand the underlying data well enough to catch an error? ➡️ Is there time built in for genuine review or just enough time to click approve? ➡️ What happens if someone says no? Is that genuinely supported? If the answer to any of those is no… you do not have human oversight. You have automation bias with a human signature attached. What does genuine human oversight look like in your organisation right now? #ai #leadership #futureofwork #artificialintelligence #aistrategy #teamhuman #intellectualatrophy #criticalthinking

𝗡𝗮𝗶𝘃𝗲 𝗥𝗔𝗚 𝘄𝗼𝗿𝗸𝘀 𝗶𝗻 𝗮 𝗱𝗲𝗺𝗼. 𝗜𝘁 𝗳𝗮𝗶𝗹𝘀 𝘁𝗵𝗲 𝗺𝗼𝗺𝗲𝗻𝘁 𝗿𝗲𝗮𝗹 𝘂𝘀𝗲𝗿𝘀 𝘀𝗵𝗼𝘄 𝘂𝗽. Embed → retrieve → generate looks clean in a notebook. Real requirements break it: → Questions whose answer is spread across many documents → Industry terms that embeddings get wrong → Bad chunks the pipeline never catches → Answers that live in how things connect, not in any single chunk → PDFs full of tables and images a text-only index cannot read These 5 architectures are how serious teams stay ahead in the agentic AI era: 𝟬𝟭 𝗛𝘆𝗯𝗿𝗶𝗱 𝗥𝗔𝗚 → Dense vectors find meaning. BM25 finds exact words. → Reciprocal Rank Fusion combines both ranked lists. → A safe baseline for almost every team. 𝟬𝟮 𝗚𝗿𝗮𝗽𝗵𝗥𝗔𝗚 → Pull entities and their relationships into a knowledge graph. → Retrieve subgraphs and community summaries, not chunks. → Best when the answer lives in how things connect. 𝟬𝟯 𝗔𝗴𝗲𝗻𝘁𝗶𝗰 𝗥𝗔𝗚 → A planner agent picks the right tool: vector, web, or SQL. → A reasoner agent keeps trying until the answer is solid. → Retrieval becomes a plan, not a single step. 𝟬𝟰 𝗖𝗼𝗿𝗿𝗲𝗰𝘁𝗶𝘃𝗲 𝗥𝗔𝗚 (𝗖𝗥𝗔𝗚) → Grade every retrieval before you trust it. → Correct → answer. Unclear → rewrite the query. Wrong → search the web. → This is what production RAG actually looks like. 𝟬𝟱 𝗠𝘂𝗹𝘁𝗶𝗺𝗼𝗱𝗮𝗹 𝗥𝗔𝗚 → One embedding model (CLIP, ColPali) for text, images, and tables. → One vector index. One multimodal LLM. → No more separate pipelines for PDFs with charts. I built a runnable example for each of the five patterns. GitHub link in the first comment. The best teams in 2026 do not pick one. They combine them — hybrid retrieval inside an agentic loop, with a corrective grader, over a multimodal index. Naive RAG is a starting point, not a finish line. That is why most enterprise GenAI projects stall at the demo. Which of these five becomes the default RAG stack in the next 18 months — and which stays a specialized tool?

Is Transparency About AI Decisions More Important Than Accuracy? I was experimenting with an AI system this week when something surprising happened. The answer it gave me was so precise and so polished that I felt my brain relax a little too quickly. And that moment made me pause. What if accuracy becomes so good that it quietly replaces our instinct to question? Here is what kept circling in my mind: • People trust outcomes faster than reasoning • When something feels correct, scrutiny dissolves • Transparency becomes a rescue tool rather than a standard Most debates treat transparency as a moral checkbox, but in my opinion, it is something far more practical. It protects the cognitive effort that accuracy tends to erase. I keep thinking of it like this: Accuracy helps us move faster. Transparency keeps us awake while we move. A perspective most people overlook When systems become highly reliable, we start outsourcing judgment without noticing it. This gradual dependency is the real risk, not the model itself. The brain adapts to whatever feels easy, even if that ease comes at the cost of long term reasoning. So instead of choosing one side, here is what I think teams should build for: ✔ Clear explanations that reveal the logic behind decisions ✔ Light touch prompts that encourage users to question even strong answers ✔ Interfaces that show the source, the guess, and the reasoning separately ✔ A daily habit of checking one AI output for hidden assumptions ✔ Systems that preserve curiosity rather than suppress it Accuracy gives us confidence today. Transparency protects our cognition tomorrow. A question I keep returning to If AI becomes flawless, will humans forget how to challenge anything that feels correct? #AIethics #AItrust #CognitiveScience #ResponsibleAI #FutureOfWork #AIliteracy

Using AI every day is not evidence of AI fluency. It proves frequency, not judgement. A better test is whether someone knows when a model is operating within its capabilities and when a polished answer may be hiding a bad one. In the Harvard-led research commonly described as the “jagged technological frontier,” people using AI performed better on tasks suited to the system. But on a task outside that frontier, they were reported to be 19 percentage points less likely to reach the correct answer. The dangerous output was not necessarily confused or obviously broken. It could be complete, confident and wrong. Before relying on an AI answer, ask three questions: What is the cost if this is wrong? Can I check it independently? Can the decision be reversed? Low-cost, reversible work that is easy to verify, such as producing a rough first draft, may need only a quick review. Advice affecting people, money, customers, safety or reputation deserves stronger verification, a second source or relevant human expertise. AI fluency is not the number of prompts someone writes. It is calibrated trust: knowing when to use an answer, when to test it and when not to use AI at all.

Two identical CVs. Both written by AI. Both sent to 1,000 people. The only difference: one was named James, one was named Emily. James’s CV got a 97% approval rating. Emily’s got 76% - and reviewers were TWICE as likely to question her competence. Twenty-two percent more likely to question whether she could even be trusted. The feedback on Emily’s CV: “She can’t even write a CV herself - not sure she has the skills to carry out the job.” The feedback on James’s CV: “He just needed a bit of help putting it together.” Same words. Same AI. Different gender. Different verdict. 🚨🚨🚨🚨 How are we STILL HERE?!?!? The study, by former Meta strategist Zehra Chatoo, was reported in Fortune on 10 May. And the most uncomfortable finding wasn’t from older reviewers. It was from Gen Z men. They were 3.5 times more likely to call Emily’s CV “weak.” The generation that is growing up with AI. The generation telling us AI is the great equaliser. The data says otherwise. Chatoo summarised it in a sentence I have not been able to stop thinking about: “When men use AI, we question their effort. When women use AI, we question their integrity.” This is not one study. Harvard Business School has the AI adoption gender gap at 25%. Brookings has found that 86% of the roles with high AI exposure and low capacity to adapt to displacement are held by women. The pattern is consistent and it is widening. The conclusion most people are drawing from this data is “women should be more confident with AI.” I think that misses the point. The bias isn’t in the technology. It is in the people reading the output. Women are not being irrational when they hesitate to use AI openly - they are reading the room accurately. The reputational cost of being seen to use AI is genuinely higher for them. The data confirms what they already sense. The answer is not to ask women to ignore that. The answer is to fix the people doing the judging. To name what is actually happening when an “Emily” CV gets called weak and a “James” CV gets the benefit of the doubt for the same words. To call out the Gen Z men perpetuating a bias they like to claim their generation has moved past. And for women in leadership reading this - use AI anyway. Lead anyway. Document your AI workflows openly. Train your teams in them. Make your usage visible in the rooms where decisions get made. The cost of stepping back from AI in this moment is far higher than the cost of stepping in. We have the data to prove it now. If this resonated, I write about the AI gender gap, ethics, and practical strategy for women in leadership every week in my newsletter. The link is here: https://lnkd.in/emWjxC9t

The best AI model on this benchmark is wrong nearly half the time. The worst one hallucinates on 93% of its mistakes and never admits it. Now ask yourself which one is in your production stack. That's not an accuracy problem. That's a governance failure and it's already in production. Look at the data closely. Not because of who's winning. Because of what most teams aren't measuring at all. Two charts. Two different questions. Most leaders only ask one. Accuracy: How often is the model right? Hallucination: How dangerous is it when it's wrong? Those are not the same metric. Confusing them is how companies end up with confident AI giving wrong answers to high-stakes decisions. Legal. Compliance. Finance. Medical. One bad hallucination in those domains doesn't cost you a prompt. It costs you a quarter. Years in pharma and manufacturing taught me something most AI leaders miss: The most dangerous model isn't the least accurate one. It's the one that never says "I don't know." A model that admits uncertainty protects you. A model that guesses with confidence exposes you. That distinction is the difference between an AI tool and an AI liability. This is exactly why I built the AI Trust Stack™. Five layers. Bottom to top: Data Quality. Data Governance. Ethical AI. AI Governance. Business Alignment. Hallucination risk lives between layers 1 and 4. Dirty data feeds bad outputs. Missing governance lets bad outputs reach decision-makers unchecked. Most teams are optimizing layer 5 without ever auditing layers 1 through 4. That's building on sand and wondering why the foundation cracks. The benchmark wars miss the point. Leaderboards measure capability. Production measures trust. And trust isn't built by picking the highest-scoring model. It's built by knowing exactly where each model breaks and designing for that failure before it reaches your customer, your board, or your balance sheet. Decisions outlive prompts. Make sure yours are built on trust, not confidence scores. ♻️ Repost this before your team learns this the hard way 🔔 Follow Gabriel Millien for AI transformation insights that cut through the noise Infographic credit: Greg Coquillo

At Amazon, I’ve built pipelines that move thousands of gigabytes of data. At Amazon, I’ve also built platforms used by hundreds of teams across the organization. But do you know how I got the opportunity to do these things? → It was because of one simple mindset shift: I stopped thinking like a pipeline builder. And started thinking like a product builder. Here’s what that shift looks like in real life 👇 1. Optimize for adoption, not just execution A fast Spark job is nice. But a pipeline that any team can deploy, monitor, and debug without you? That’s a game-changer. If your internal users are struggling, that’s a UX bug. 2. Design APIs, not one-off scripts Your Airflow DAGs and Glue jobs should feel like APIs. Versioned, observable, with clear inputs/outputs. That’s how you build trust at scale. 3. Surface friction like a PM If people keep pinging you for creds, schemas, or weird Athena errors, that’s a signal. Treat those moments like product bugs. Fix them once, and fix them for everyone. 4. Metrics = feedback loops In product, you track conversion. In data platforms, track usage: → How many teams use your tools? → How often do they fail? → Who’s stuck? These are your feature requests. 5. Think enablement > control Great platforms don’t block, they enable. Guardrails should guide, not restrict. Make it easy to do the right thing. I’ve learned this the hard way. When you think like a product builder, your work scales. It doesn’t stop at you. It becomes a system that helps others move faster. So next time you're building a data pipeline, ask yourself: What would this look like if it were a product? Let’s build platforms that people actually want to use.

I’m so happy to see this! Yesterday, the ISO published a new standard, ISO/IEC 42001:2023 for AI Management Systems. My suspicion is that it will become as important to the AI world as ISO/IEC 27001 arguably became the most important standard for information security management systems. The standard provides a comprehensive framework for establishing, implementing, maintaining, and improving an artificial intelligence management system within organisations. It aims to ensure responsible AI development, deployment, and use, addressing ethical implications, data quality, and risk management. This set of guidelines is designed to integrate AI management with organisational processes, focusing on risk management and offering detailed implementation controls. Key aspects of the standard include performance measurement, emphasising both quantitative and qualitative outcomes, and the importance of AI systems’ effectiveness in achieving intended results. It mandates conformity to requirements and systematic audits to assess AI systems. The standard also highlights the need for thorough assessment of AI's impact on society and individuals, stressing data quality to meet organisational needs. Organisations are required to document controls for AI systems and rationalise their decisions, underscoring the role of governance in ensuring performance and conformance. The standard calls for adapting management systems to include AI-specific considerations like ethical use, transparency, and accountability. It also requires continuous performance evaluation and improvement, ensuring AI systems' benefits and safety. ISO/IEC 42001:2023 aligns closely with the EU AI Act. The AI Act classifies AI systems into prohibited and high-risk categories, each with distinct compliance obligations. ISO/IEC 42001:2023's focus on ethical AI management, risk management, data quality, and transparency aligns with these categories, providing a pathway for meeting the AI Act’s requirements. The AI Act's prohibitions include specific AI systems like biometric categorisation and untargeted scraping for facial recognition. The standard may help guide organisations in identifying and discontinuing such applications. For high-risk AI systems, the AI Act mandates comprehensive risk management, registration, data governance, and transparency, which the ISO/IEC 42001:2023 framework could support. It could assist providers of high-risk AI systems in establishing risk management frameworks and maintaining operational logs, ensuring non-discriminatory, rights-respecting systems. ISO/IEC 42001:2023 may also aid users of high-risk AI systems in fulfilling obligations like human oversight and cybersecurity. It could potentially assist in managing foundation models and General Purpose AI (GPAI), necessary under the AI Act. This new standard offers a comprehensive approach to managing AI systems, aiding organisations in developing AI that respects fundamental rights and ethical standards.

Breast cancer can now be detected 5 years before it develops thanks to AI: Recent advances in artificial intelligence have shown remarkable potential for early breast cancer detection. AI systems are being developed that can analyze mammograms and identify potential cancer risks up to five years before clinical manifestation. These systems operate through sophisticated deep learning models trained on extensive mammogram databases, enabling them to detect subtle imaging patterns that might escape human notice. Different research teams have taken varied approaches to this challenge. For instance, scientists at MIT and Massachusetts General Hospital created a comprehensive model that examines entire mammogram images for cancer-predictive patterns. Meanwhile, Duke University researchers developed AsymMirai, which takes a more focused approach by analyzing breast tissue asymmetry between left and right breasts, achieving similar accuracy through a more streamlined and transparent method. AI is also proving valuable as a complementary tool for radiologists. The Mia system, currently being tested by Britain's National Health Service, serves as an additional layer of scrutiny, helping identify minute cancerous formations that human reviewers might miss. This capability for earlier detection can lead to more timely interventions and less aggressive treatment options.