Conducting Skills Assessments

Industry expertise makes you credible. It doesn't make you a product manager. I see this pattern quite often. Someone with deep domain expertise gets promoted to VP of Product. They understand customers. They know the industry. But product management is a distinct craft, part art, part science, part psychology. Without learning the craft's methodologies and frameworks, you'll lose credibility with your team fast. A healthcare veteran might know exactly what patients need, but struggle with roadmap prioritization frameworks. A financial services expert understands regulatory requirements but can't facilitate effective product discovery sessions. Domain expertise gives you customer empathy, not product management skills. The companies that get this right invest in teaching new product leaders the actual craft. They don't assume industry knowledge translates to strategic product thinking, cross-functional leadership, or outcome-focused execution. In Episode 240 of the Product Thinking with Melissa Perri, I talked about exactly this challenge. New VPs need to master product methodologies alongside their domain expertise, not instead of it. The best product leaders combine deep customer understanding with strong product craft. One without the other leaves you incomplete. That’s one of the reasons why we launched the Mastering Product Strategy course at Product Institute last year, to help Product Leaders navigate the complexities of strategic Product Management. Have you seen this play out in your organization? How do you help domain experts develop true product leadership skills?

When a leader is appointed to a high-stakes role without a background in that field, the conversation often splits into blind optimism or personal attacks. Both miss the point. We need to look at the rationality of domain expertise. We need to look at the facts. The Statistical Case for the "Expert Leader" The "disruptive outsider" is a popular narrative, but the data tells a different story 🚀 The Performance Premium: A study of 35,000 data points in The Leadership Quarterly found "Expert Leaders" correlate with a 25% to 33% increase in organizational performance. 📊 The Match Quality Gap: Research from 2025 (Lyman & Capron) shows "match quality" between a leader and firm is 78% higher for those with deep-background experience. ⚠️ The Dismissal Rate: The Center for Creative Leadership reports 55% of outsider CEOs are dismissed within 18 months, largely due to a "learning tax" paid when lacking industry nuance. 📉 The Strategic Failure Rate: HBR analysis shows outsiders hired for "turnarounds" fail to improve operational returns in over 50% of cases. For a population of one, any person can break the odds. We should never prejudge an individual; every leader deserves the chance to prove their competence. However, we must distinguish between "prejudice" and "rational skepticism." 🔹 Positive Outliers: They exist, but they are statistically rare. Long-term performance data shows that only 1 in 10 outsiders manages to reach the top 20% of industry performance. 🔹 Negative Outliers: In most fields, you will find more negative outliers than positive ones. Probability distributions show outsiders have a significant "left-tail" risk, meaning they are statistically more likely to cause a massive decline in value than a massive gain compared to industry veterans. The Bottom Line Is it "rational" to have concerns about a total outsider? Yes. That concern is grounded in the statistical reality that background and passion for a field are the primary predictors of success. We can hope for the exception, but we should plan for the average. Sources 📍 The Leadership Quarterly (2015): Expert leaders explain 16% of organizational performance variance. 📍 Insider CEOs: Lucky or Good? (2025): Insiders possess 78% higher firm-specific capital. 📍 Center for Creative Leadership: 55% of external CEO hires fail within 18 months. 📍 HBR (Khurana & Nohria, 2012): Analysis of 850+ successions over 20 years shows outsiders are significantly more likely to represent "extreme negative outcomes" (left-tail risk) than internal experts. 📍 McKinsey & Company (2023 CEO Excellence Study): Performance tracking of 2,400 CEOs reveals that only 10% of outsider appointments achieve "top-quintile" excess shareholder returns over a 5-year period. 📍 Wharton School of Business (Matthew Bidwell, 2011/2012): Research on "External Hires" confirms they are 61% more likely to be laid off or fired and score lower on performance reviews despite being paid 18% more on average.

If you’re an AI engineer trying to understand how reasoning actually works inside LLMs, this will help you connect the dots. Most large language models can generate. But reasoning models can decide. Traditional LLMs followed a straight line: Input → Predict → Output. No self-checking, no branching, no exploration. Reasoning models introduced structure, a way for models to explore multiple paths, score their own reasoning, and refine their answers. We started with Chain-of-Thought (CoT) reasoning, then extended to Tree-of-Thought (ToT) for branching, and now to Graph-based reasoning, where models connect, merge, or revisit partial thoughts before concluding. This evolution changes how LLMs solve problems. Instead of guessing the next token, they learn to search the reasoning space- exploring alternatives, evaluating confidence, and adapting dynamically. Different reasoning topologies serve different goals: • Chains for simple sequential reasoning • Trees for exploring multiple hypotheses • Graphs for revising and merging partial solutions Modern architectures (like OpenAI’s o-series reasoning models, Anthropic’s Claude reasoning stack, DeepSeek R series and DeepMind’s AlphaReasoning experiments) use this idea under the hood. They don’t just generate answers, they navigate reasoning trajectories, using adaptive depth-first or breadth-first exploration, depending on task uncertainty. Why this matters? • It reduces hallucinations by verifying intermediate steps • It improves interpretability since we can visualize reasoning paths • It boosts reliability for complex tasks like planning, coding, or tool orchestration The next phase of LLM development won’t be about more parameters, it’ll be about better reasoning architectures: topologies that can branch, score, and self-correct. I’ll be doing a deep dive on reasoning models soon on my Substack- exploring architectures, training approaches, and practical applications for engineers. If you haven’t subscribed yet, make sure you do: https://lnkd.in/dpBNr6Jg ♻️ Share this with your network 🔔 Follow along for more data science & AI insights

Expertise can hurt leaders. I’ve met leaders who are brilliant in their domains. They know the playbook inside out and can almost instinctively predict what they’re supposed to do next. But they’re so confident in what worked that they fail to see what’s changing. Sector knowledge helps, but only up to a point. The higher you go, the less your technical depth matters. What starts to matter more is your ability to connect the dots across industries, disciplines, contexts and trends. That’s where real innovation comes from. - Leena Nair moved from Unilever to CHANEL. No fashion background.  - Alan Mulally went from Boeing to Ford. No auto experience.  - Angela Ahrendts led Burberry, and later joined Apple. These leaders brought fresh ideas, a fresh set of eyes and fresh questions. Have you sat in a room where the most experienced person was also the most closed? No curiosity. No listening. Did you see new ideas die, not because they were weak, but because nobody had the patience to hear them? Have you been told by the expert leader “but our industry is different,” and felt embarrassed for looking silly? It’s good to know your stuff. Until that very expertise becomes the trap that keeps you on autopilot and away from innovation. Expertise is a good foundation. But as we all know, leadership is about much, much more…..   #leadership #expertise #beyondtheplaybook    

The Hidden Interview Questions You Didn't Know Were Being Asked I spent Tuesday meeting five candidates for a senior sales role. By the time the last one left, I noticed something fascinating. Each person was answering questions I never actually asked. Here's what I mean: When Sarah arrived 15 minutes early, she showed me she values preparation and respects others' time. When Michael kept checking his phone, he told me his priorities might be elsewhere. And when Emma asked thoughtful questions about our company culture, she revealed her interest went beyond just getting a pay cheque. You see, the interview starts well before you sit down. As an experienced headhunter, I can tell you that hiring managers are constantly gathering data points that candidates don't realise are being assessed. Some of these hidden assessment moments include: How you treat the receptionist or junior staff Whether you researched the company properly Your body language while waiting How you handle unexpected hiccups (like a delayed interviewer) The questions you ask at the end I once worked with a client who rejected an otherwise perfect candidate because they were dismissive to the office assistant. That 30-second interaction outweighed an hour of brilliant answers. Think about your last interview. What signals might you have sent without knowing it? That email you took three days to respond to? The thank-you note you forgot to send? The next time you're up for a job, remember that everything from your arrival to your departure is part of the assessment. The most successful candidates understand that actions speak louder than rehearsed answers. #Recruitment #HiringTips #TalentAcquisition

I've been thinking about the importance of leaders with domain expertise. It is tough to have that these days, given the pace of change, but it's vital that leaders understand HOW things get done, not just WHAT gets done on their teams. Leaders get shifted sideways a lot, allowing them to gain experience in a wider set of organizational disciplines. That's smart, but it means that leaders without digital experience can find themselves leading digital teams and people with no #CX background can end up leading #CustomerExperience teams. So often, leaders are used to leading, but leading when you don't understand the domain is a problem. It not only results in bad decisions but frustrates and disempowers the seasoned, capable people on the team. That's why taking over a new and unknown discipline requires a special kind of leadership. The kind of command-and-control leadership that got you to this point is a detriment when taking over a team in an unfamiliar domain. In this circumstance, it becomes more important to listen, trust people, and delegate as you increase your domain knowledge. That means planning carefully and in phases, starting with narrower executive decisions and expanding your control and decision-making as your knowledge grows. Decades ago, I had a mid-sized family-owned client. Over the course of a couple of years, we managed its email campaigns, lifting engagement and measuring improvements. Then, a relative of one of the founders took over as head of marketing, despite having no marketing or digital experience. In one fateful meeting, she told us that she'd read a book on email marketing, and we were doing it all wrong. So, she fired our agency. It was her prerogative to do so, of course. When I think back to that meeting, I frequently consider how she might have approached this differently. Had she expressed concerns or discussed her ideas, we could've had a great dialog. We tried to explore this with her, of course, but she had made her decision. I'm sure that action made her feel (and perhaps look) powerful. We continued to monitor the brand's emails and watched as they became spammier, less informational, poorly targeted to the brand's personas, and less engaging. We privately heard from a former peer at the company that engagement plunged and unsubscribes soared. I'm not proud that I felt a sense of schadenfreude upon hearing that, but more than that, I was frustrated that I wasn't given the chance to help a brand I loved to succeed. By resorting to command-and-control behaviors, this new leader deprived herself of access to talent that was committed, experienced, and capable. The key success factor of leadership, in my opinion, is not your ability to make decisions; it's humility. Knowing what you don't know, recognizing when to trust others and delegate, and adapting your leadership style to your level of domain expertise is vital as you move up and around an organization.

In many boardrooms today, a repeating question is emerging: how much technical depth is enough to effectively govern in a digital world—especially as technology reshapes not just business models, but risk, talent, and long-term value creation? Somewhere along the way, we began to equate effective leadership in a digital world with depth in a single domain. As if technical expertise alone is the differentiator. It is not. What I am seeing instead is a growing need for what we often call T-shaped leaders: Leaders who go deep in one area, but just as importantly can stretch across: -Technology -Strategy -Culture -Talent -Societal impact In fact, leaders should not try to out-code their CTO or out-engineer their product teams. But they should ask better questions, connect dots others miss, and translate complexity into direction. They should be integrators and versatile in how they think. That is the essence of being T-shaped. Not every board seat needs to be held by a technical expert, but a successful board does have the breadth—and integration of thinking—to govern what comes next.

🦜Beyond Stochastic Parrots: Understanding How LLMs Really "Think". Recently, I've been fascinated by the mechanics of how LLMs choose/sample their next words. Some spicy stuff going on in the land of LLM samplers. Here's what I've learned about token selection in LLMs: 🔹For each potential word, the model assigns probabilities across its entire vocabulary ~120,000 tokens 🔹The selection process can be tuned through "temperature": • Low temperature = focused, precise responses • High temperature = more creative, exploratory outputs 🔍 What's exciting is the emergence of dynamic temperature adjustment. The idea is to automatically adapt the creativity level(temperature) of the LLM based on the models confidence(entropy): 🔹If you have high LogProbs → High confidence of next token → Low Entropy State → Increase Temp → We can afford to increase temperature and force the model to be more creative. 🔹If you have low LogProbs → Low confidence of next token → High Entropy State → Decrease Temp → We need to reduce the likelihood of the model going off the rails and rein in the creativity. This adaptive approach allows LLMs to be both precise when needed and creative when appropriate - much like human thinking! The next time someone tells you LLMs are just "stochastic parrots," remember: they're implementing sophisticated probability-based decision making that can dynamically adjust to the task at hand. Resources: 🔸Nice Explanation of LLM Samplers: https://lnkd.in/gjHQ4EVA 🔸Slides on Dynamic Temperature: https://lnkd.in/gBy3V4uX 🔸Notebook on Dynamic Temp Sampler: https://lnkd.in/geTwBwsV 🔸Deepmind Paper on Dynamic Sampling: https://lnkd.in/gyYxHCN2 🔸MinP Sampling Paper: https://lnkd.in/gD3uzj8M 🔸Entropix – Entropy based sampling: https://lnkd.in/gAhxz_e9

If your legal AI makes a “judgment,” it’s not just following rules. 𝗜𝘁’𝘀 𝗲𝘅𝗽𝗿𝗲𝘀𝘀𝗶𝗻𝗴 𝗮 𝘄𝗼𝗿𝗹𝗱𝘃𝗶𝗲𝘄. Most software operates on if/then logic. Input X produces Output Y. Every time. Deterministic. LLMs are fundamentally different. They're probabilistic, and new research shows just how much that matters. A recent paper, "Evaluative Fingerprints" by Wajid N., studied 9 frontier LLMs evaluating the same content using the same rubric. The results are striking: 𝗜𝗻𝘁𝗲𝗿-𝗺𝗼𝗱𝗲𝗹 𝗮𝗴𝗿𝗲𝗲𝗺𝗲𝗻𝘁 𝘄𝗮𝘀 𝗻𝗲𝗮𝗿 𝘇𝗲𝗿𝗼. Yet individual models were remarkably consistent 𝘄𝗶𝘁𝗵 𝘁𝗵𝗲𝗺𝘀𝗲𝗹𝘃𝗲𝘀, just not with each other. The researchers could identify which model produced an evaluation with 𝟴𝟵.𝟵% 𝗮𝗰𝗰𝘂𝗿𝗮𝗰𝘆 based solely on its scoring patterns. Even GPT-4.1 and GPT-5.2 (same provider, different versions) were distinguishable 99.6% of the time. The paper calls this the "reliability paradox": models don't agree on what "good" means, but 𝘁𝗵𝗲𝘆'𝗿𝗲 𝘀𝗼 𝗰𝗼𝗻𝘀𝗶𝘀𝘁𝗲𝗻𝘁 𝗶𝗻 𝗵𝗼𝘄 𝘁𝗵𝗲𝘆 𝗱𝗶𝘀𝗮𝗴𝗿𝗲𝗲 that their evaluation patterns function as fingerprints. 𝗪𝗵𝘆 𝘀𝗵𝗼𝘂𝗹𝗱 𝗹𝗲𝗴𝗮𝗹 𝗰𝗮𝗿𝗲? In many ways, LLMs behave like people, who are also probabilistic. We've always known that different lawyers assess risk differently, interpret contract language differently, prioritize issues differently. Now we're deploying AI with the same characteristics. Consider a legal department implementing an agentic workflow that escalates matters based on risk assessment. This research suggests the choice of model isn't an implementation detail. It's a substantive decision that shapes outcomes. As we build agentic processes that assert judgment (contract review, risk triage, compliance monitoring), we need to understand that model selection is a methodological choice with real consequences. The question isn't whether to use AI for legal judgment. 𝗜𝘁'𝘀 𝘄𝗵𝗲𝘁𝗵𝗲𝗿 𝘄𝗲 𝘂𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱 𝘄𝗵𝗮𝘁 𝘁𝗵𝗲𝗼𝗿𝘆 𝗼𝗳 𝗷𝘂𝗱𝗴𝗺𝗲𝗻𝘁 𝘄𝗲'𝗿𝗲 𝗲𝗻𝗰𝗼𝗱𝗶𝗻𝗴 𝘄𝗵𝗲𝗻 𝘄𝗲 𝗽𝗶𝗰𝗸 𝗮 𝗺𝗼𝗱𝗲𝗹.

You're a PM managing a product in a domain where you're not the expert. Healthcare. Fintech. Developer tools. Whatever. Your team has people who've been in this space for 15 years. And they know it. And you know it. And everyone knows it. Then you're in a product review and you make a recommendation. The domain expert engineer: "That's not how [industry thing] works..." The room shifts. Everyone looks at you. Your credibility is being tested. Most PMs respond in one of three broken ways: 1. Fake expertise: "Actually, I think it does work that way..." (Now you're wrong AND arrogant) 2. Defer completely: "You're right, what do you think we should do?" (Now you're a facilitator, not a PM) 3. Defensive: "Well, the user research shows..." (Now you're dismissing their expertise) All three erode your authority. Your authority as a PM doesn't come from being the smartest person about the domain. It comes from being the best person at making product decisions given incomplete information. Here's what actually works: 1. Separate domain knowledge from product judgment "You're right, I don't know [technical detail] as well as you do. Here's what I do know: [user problem, business context, strategic priority]. Given that, does it change your thinking?" 2. Make your reasoning transparent "Help me understand - if we do it the way you're suggesting, what's the tradeoff? Because from where I sit, I'm optimizing for [X], even if that means compromising on [Y]." 3. Use your lack of expertise as an asset "I don't have your expertise, which means I'm looking at this fresh. What am I missing that you see clearly?" 4. Know when domain trumps product Sometimes the expert is right and you're wrong. The key is knowing when to defer vs. when to push back. Defer when: Technical constraints, regulatory requirements, domain-specific risks Push when: User needs vs expert preferences, strategic priorities, resource tradeoffs Domain experts are really good at what's technically possible. They're often bad at what users actually need. Your job is to hold the tension between "can we" and "should we." The experts handle the "can we." You handle the "should we." Don't try to out-expert the experts. Out-strategize them. How do you handle being the least knowledgeable person in the room? #ProductManagement #Leadership #DomainExpertise #PMLife