Experience Design Metrics

🔍Rethinking How We Measure and Interpret Medication Errors One of the most common misconceptions in medication safety is that the number of reported errors reflects the quality of care. This couldn’t be further from the truth. As a Medication Safety Expert, I’d like to share key insights, supported by evidence, on how we should calculate and interpret medication errors effectively. 🤔 Should Medication Errors Be Calculated? Medication errors should be evaluated using a combination of: 1️⃣ Error Rate per 1,000 Patient Days or Doses Administered: This standardizes reporting and allows for meaningful comparisons over time (National Quality Forum). 2️⃣ Severity and Type of Errors: Focus on errors causing harm (Category E and above as per NCC MERP) and near-misses to address system weaknesses. 3️⃣ Trends Over Time: Use error data to identify systemic patterns, not to penalize individuals or departments. 🤔 Why Error Counts Alone Are Misleading The total number of reported errors reflects the reporting culture more than the actual error rate. Institutions with a robust just culture encourage error reporting and often have higher reported numbers—but this is a sign of transparency, not failure. Reported Errors ≠ True Errors Benchmarking Medication Errors is a Big No-No 👎 Comparing error rates across institutions is flawed. Each facility has unique patient populations, case complexities, and reporting practices. Benchmarking can discourage reporting, as institutions may fear appearing less safe. 🔗 Evidence: The ISMP strongly advises against benchmarking, emphasizing that trends and root causes are far more valuable for improving safety (ISMP Medication Safety Alert, 2018). What Matters Most in Error Communication? Focus on Lessons Learned: Share stories and actionable insights, not just numbers. Highlight Systemic Improvements: Show how errors lead to prevention strategies. Engage Teams: Use multidisciplinary reviews to tackle root causes collaboratively. The Way Forward Medication safety isn’t about counting errors—it’s about creating systems that minimize harm and improve outcomes. Let’s calculate error rates thoughtfully, avoid harmful benchmarking, and focus on actionable insights for prevention. What are your strategies for advancing medication safety in your organization? I’d love to hear your thoughts! #MedicationSafety #PatientSafety #MSO #Quality

💡Essential UX Metrics & KPIs in Product Design 'Measure early, measure often' is the only right strategy for product design. When you measure your design regularly, you minimize the risk of product failure. All UX metrics and KPIs can be divided into two large categories—behavioral and attitudinal. Here are a few of the most popular metrics that apply to many types of products: ✅ Behavioral UX metrics & KPIs Behavioral metrics are derived from users' actions and behaviors during their interaction with a product. ✔ Task success rate. This metric assesses whether or not users can successfully complete a designated task. It's critical for evaluating the usability and functionality of a product. ✔ Time-on-task. The time users take to complete specific tasks. This can reveal how intuitive and efficient a product is. Longer times may indicate usability issues or a steep learning curve. ✔ Error rate. The frequency of errors made by users while interacting with a product. Error rate = Number of errors / Total number of task attempts. High error rates may indicate usability problems in a product. ✔ Error recovery time. How quickly a user can recover from an error while interacting with a product. Long error recovery times can lead to increased frustration and a negative user experience. ✔ Feature adoption rate. Assessment of the acceptance of new features among their user base. Feature adoption rate = number of users who have adopted a specific feature / the total number of users during (week/month) *100. A high feature adoption rate (>80%) typically indicates that the feature meets the needs or preferences of users. ✅ Attitudinal UX metrics and KPIs Attitudinal UX metrics are used to gauge users' attitudes, feelings, and satisfaction levels with a product. Unlike behavioral metrics, which focus on users' actions and behaviors, attitudinal metrics are concerned with users' subjective experiences. ✔ Customer Effort Score (CES): CES measures the ease of user interaction with a product. It asks users to rate the effort required to use the product or to achieve a specific task. ✔ System Usability Scale (SUS): SUS is a 10-item questionnaire with five response options for respondents that aims to assess the usability of products. SUS provides a score out of 100. A SUS score above 68 is considered above average, and anything below 68 is below average. ✔ Customer satisfaction (CSAT): CSAT measures how content a user is with a product on a 100-point scale. Satisfaction can be assessed through surveys, feedback forms, or direct interviews. A high rating (>70) means the customer is satisfied. A middle rating (>50 but <70) means the customer is neutral. A low rating (<50) means the customer is dissatisfied. 📺 Google HEART framework for measuring UX: https://lnkd.in/dhkwy_jN 🖼️ Attitudinal vs behavioural research by Maze #UX #design #uxdesign #measureux #productdesign #uidesign #metrics

Failure Rate: Not Just a Number—It Tells a Story! Whether we’re designing trains, planes, or software systems—failure rate is one of the most critical indicators of reliability What Is Failure Rate (λ)? Failure rate is the expected number of failures per unit time, often expressed in: 1. Failures per hour (F/h) 2. Failures per million hours (FPMH) 3. FIT (Failures In Time) → 1 FIT = 1 failure per 10⁹ hours But here’s the twist: The way we calculate λ depends on the context! 1. Parts Count Method (When detailed design info is not available) Formula: λ_system = Σ (λᵢ × Qᵢ) Add up all component failure rates multiplied by their quality factor. Used early in design using standard failure rate tables like MIL-HDBK-217. 2. Parts Stress Method (When actual operating conditions are known) Formula: λ = λ_b × π_T × π_Q × π_E × π_S Multiply the base rate by various real-world factors: π_T (temperature), π_Q (quality), π_E (environment), π_S (stress) Gives a more accurate estimate for your real design. 3. Empirical / Field Data Method (Based on actual failure data) Formula: λ = N_f / T Divide the number of observed failures (N_f) by total operating time (T). Simple and very accurate if you have real test or field data. 4. Weibull Analysis (When failure rate changes over time) Formula: λ(t) = (β / η) × (t / η)^(β − 1) Used when failure rate isn’t constant (e.g. infant mortality or aging). β = shape of the failure curve η = characteristic life Great for reliability curves and predicting when systems wear out. 5. IEC 61709 Method (Standardized prediction using adjustment factors) Formula: λ = λ_ref × π_cond × π_usage Adjusts a reference failure rate based on environmental and usage conditions. Used in many industries including automotive, telecom, and electronics. Why It Matters Ignoring or guessing λ can lead to… 1. Unexpected failures 2. Higher maintenance costs 3. Safety risks But with the right method, you get… ✅Smarter designs ✅ Reliable systems ✅ Lower life-cycle cost #ReliabilityEngineering #FailureRate #WeibullAnalysis #RAMS #EngineeringForEveryone #SystemDesign

Monitoring Web applications: Should you monitor Error Count or Error Rate? Error Count & Error rate are 2 metrics that tell us about the successes & failures of our web app. So what’s the difference and why does it matter? 👉 Error Count is the total number of errors that occur in your application over a specific period. It is pretty simple to understand and plot on a graph. But I'd argue that it is also pretty useless as a metric. It Doesn’t account for traffic volume. High traffic can naturally lead to a higher error count, even if the application is relatively stable. eg- You see 100 errors/minute in your app. Is this okay or are you in trouble? That depends on the amount of traffic. If your app is serving 1 Million requests / min, 100 errors is not even an issue worth looking into. It is tolerable. But if your app serves only 500 req/min and 100 of them are errors, you need to drop everything and fix the problems. 👉 Error Rate is the percentage of requests that result in errors, calculated as (Error Count / Total Requests) * 100. This is almost always a better metric because it shows you failures relative to the total amount of traffic. Depending on the nature of your app, you can determine the % that's tolerable for you. eg- An API serving memes can afford a 10% error rate but a high-scale payment processing API should probably tolerate < 1% error rate. Error rate actually tells you about user experience - is a large part of your user base facing issues while interacting with your app? Error count will often just mislead you. So unless you have a very good reason, always monitor Error Rate instead of error count. Thoughts? In my video on monitoring Kubernetes apps, I explore error rate in much more detail -> youtu.be/XK7CJlOLuic

Why you shouldn't be using measures that sum across the errors of different time series/products (like MAE, MSE, WAPE, MAE% and others) to measure your forecasting performance... "Children usually learn at an early age the dangers of adding apples to orange (or apples to bananas or whatever). Yet in later life, we often fail to remember this maxim." (Chatfield, 1988) "The scale of the data often varies considerably among series. Series with large numbers might dominate comparisons. Despite this problem, the need for unit-free measures was not widely appreciated in the early 1980s" (Armstrong & Collopy, 1992) "There are some commonly used accuracy measures whose scale depends on the scale of the data. These are useful when comparing different methods applied to the same set of data, but should not be used, for example, when comparing across data sets that have different scales." (Hyndman & Koehler, 2006) 35+ years after the first of these references, still forecasting practitioners and consultants fail to understand this simple principle. Instead of summing the non-scaled errors and metrics across products (i.e., summing up apples with oranges), you should be using unit-free/scale-independent error metrics. My preference is to use relative measures (where the division takes place before summarising across products) which directly show the forecast-value-added (FVA) over a benchmark. It does not matter if some of these metrics (like WAPE or MAE%) apply scaling after summing across the units of products. At that point, the results are already non-sensible. References Chatfield C. (1988) "Apples, oranges and mean square error", International Journal of Forecasting, 4(4), 515-518. Armstrong J.S. & Collopy F. (1992) "Error measures for generalizing about forecasting methods: Empirical comparisons", International Journal of Forecasting, 8(1), 69-80. Hyndman R.J. & Koehler A.B. (2006) "Another look at measures of forecast accuracy", International Journal of Forecasting, 22(4), 679-688.

5G NR – BLER: IBLER & RBLER Explained Through Q & A Q1 : What is BLER, IBLER and RBLER in 5G NR? => BLER (Block Error Rate): fraction of Transport Blocks (TBs) failing CRC check at the receiver. It is the core KPI driving CQI reporting, MCS selection and HARQ operation. ** IBLER (Initial BLER): error rate measured on the first transmission only (before HARQ retransmissions). It reflects how accurately the scheduler/MCS matched the instantaneous channel. * High IBLER → aggressive MCS, channel overestimation. * Low IBLER → conservative scheduling, under-utilization. - Operators tune Outer Loop Link Adaptation (OLLA) to keep IBLER ≈ 10%. ** RBLER (Residual BLER): error rate measured after all HARQ retransmissions are complete. It reflects the final reliability seen by higher layers (RLC/PDCP/application). * For eMBB, target ≤ 1%. * For URLLC, target is far stricter since retransmission budget is limited. ** My notes : Together * IBLER = how good was the first shot? * RBLER = what’s the final outcome after HARQ? * BLER (generic) = umbrella KPI, but must always be clear whether it refers to initial or residual. Q2 : Why do operators monitor both IBLER and RBLER? => IBLER shows if the scheduler & MCS selection are tuned correctly. * RBLER shows if the system is delivering SLA-grade reliability. ** Together, they answer: “Was it a bad first guess or did the retransmission chain fail?” Q3 : In real deployments what causes IBLER ≫ RBLER? => Aggressive MCS due to optimistic CQI reporting. * Works fine for eMBB → HARQ catches errors. * But wastes PRBs and latency budget → not good for URLLC. ** Fix: tune OLLA step size, cap MCS delta for critical 5QI. Q4 : What if RBLER is also high? => This points to HARQ failure rather than just bad MCS. Common root causes: * Poor PUCCH → NACK/ACK misdetection. * Late ACK → HARQ deadline missed. * Limited HARQ processes → buffer overflow. * Strong interference → soft combining ineffective. ** Operators check: HARQ round distribution, PUCCH BLER, per-RB SINR heatmaps. Q5 : How do mobility events impact BLER? => Around handovers, CQI reports are stale → wrong MCS → IBLER spike. * If HO command is delayed or PDCCH coverage poor → RBLER also rises. ** Mitigation: * Stronger PDCCH aggregation on neighbors. * Adjust A3/A5/TTT mobility thresholds. * DRX alignment to prevent missing HO command. Q6 : Why do drive tests often report different IBLER/RBLER than OSS counters? => Drive tests measure air-interface BLER (layer-1 CRC). * OSS counters may measure post-HARQ BLER (MAC PDUs) or even RLC retransmissions. * Some Vendor say "Post-HARQ BLER” and other defines as IBLER. ** takeaway * Target IBLER ≈ 10% → maximum spectral efficiency. * Keep RBLER <1% (stricter for URLLC). * Always split KPIs into pre-HARQ vs post-HARQ buckets; otherwise tuning will mislead. * BLER is not just a number: it reflects LA, HARQ robustness, scheduler design, interference management and mobility tuning all at once. #5GNR #BLER #IBLER #RBLER

Bit Error Rate (BER) in PAM4 based designs: SERDES (SERializer - DESerializer) serves as a critical component in modern chip-to-chip communication. The evolution of high-speed interfaces has seen a shift from NRZ (Non-Return to Zero) signaling in earlier generations to the more advanced PAM4 signaling in recent designs. In evaluating SERDES performance, the Bit Error Rate (BER) stands out as a key metric. While NRZ-based signaling maintains a BER of 1e-12, PAM4 signaling operates at a BER of only 1e-6. The Signal-to-Noise Ratio (SNR) emerges as a pivotal factor influencing PAM4 BER due to its four voltage levels, resulting in a narrower eye opening compared to NRZ signaling. This reduced margin amplifies the impact of noise, elevating the likelihood of bit errors. The intricacies of PAM4, with its closer voltage levels, render it more vulnerable to noise and timing jitter, making signal decoding error-prone even with minor disruptions. Additionally, PAM4 signals encounter more significant attenuation in transmission channels at higher frequencies compared to NRZ, leading to further signal quality degradation. How BER is improved in PAM4 based designs? Enhancing the BER to 1e-12 or better in PAM4 involves leveraging advanced equalization techniques, clock recovery mechanisms, and channel optimizations. Digital strategies such as Forward Error Correction (FEC) codes, Gray coding, and pre-coding play a crucial role in error detection and correction, thus enhancing overall link reliability. Gray Coding ensures that successive symbol values differ by only one bit, while Pre-coding mitigates burst errors, crucial in high radiation environments. Implementing Reed-Solomon (RS) codes, commonly found in standards like PCIe Gen6, 100G Ethernet, and USB4 v2, adds an extra layer of error correction. These codes append parity codes to data at the transmitter end, aiding in error correction at the receiver's FEC decoder. Stay tuned for more insights on these essential blocks in my upcoming posts.

📘DWDM Learning Series – Part 12: Understanding the Link Quality Chain When we talk about the health of an optical network, three key metrics define performance: OSNR, Q-Factor, and BER. Think of them as links in a chain, if one weakens, the entire chain fails. 📍The Link Quality Chain Every DWDM link is judged by three interconnected measures: 1️⃣OSNR (Optical Signal-to-Noise Ratio): How clean your signal is in the optical domain. 2️⃣Q-Factor: How well the receiver distinguishes between ones and zeros in the electrical domain. 3️⃣BER (Bit Error Rate): The actual count of errors seen at the system level. A weak OSNR leads to a poor Q-Factor, which inevitably increases BER. That’s why we call it a “chain.” 📍The Café Conversation Analogy To make it simple: imagine you’re talking in a noisy café. ▪️OSNR is like the clarity of the speaker’s voice versus background noise. ▪️Q-Factor is how well your brain interprets the words you hear. ▪️BER is whether you understood the message correctly. ▪️In short: a clear voice (high OSNR) → accurate understanding (Q-Factor) → fewer mistakes (low BER). 📍OSNR: Optical Signal-to-Noise Ratio ▪️OSNR is the ratio of signal power to noise power in a specific optical bandwidth. ▪️Measured in dB, it shows how “clean” your signal is. ▪️OSNR can degrade due to fiber loss, amplifier noise, and crosstalk. ▪️A strong OSNR means your signal is ready for long-haul transmission. 📍Q-Factor: Electrical Signal Quality ▪️Derived from the eye diagram at the receiver. ▪️A high Q-Factor means a clear separation between 1s and 0s → fewer mistakes. ▪️A low Q-Factor means the “eye” is closing, making errors more likely. ▪️Strongly correlated with BER before FEC (Forward Error Correction). ▪️Think of Q-Factor as the receiver’s confidence in deciding what bit it just saw. 📍BER: Bit Error Rate ▪️BER is the ratio of incorrectly received bits to the total transmitted bits. ▪️Pre-FEC BER shows raw link performance — must stay within FEC correction range. ▪️Post-FEC BER ideally approaches zero after corrections. ▪️This is the final truth test of network quality: can your system deliver error-free data? 📍OSNR vs. Q-Factor vs. BER Here’s how they compare: ▪️Domain: OSNR = Optical | Q-Factor = Electrical | BER = System. ▪️Use: OSNR helps in planning, Q-Factor in signal analysis, BER in final performance verification. ▪️Key takeaway: Measure all three to get the full picture of your DWDM link health. 💡 Final Note: If OSNR, Q-Factor, and BER form a chain ➡️ OSNR is the foundation, Q-Factor is the decision-making, and BER is the outcome. Together, they define how strong (or weak) your DWDM link really is.

When we run usability tests, we often focus on the qualitative stuff — what people say, where they struggle, why they behave a certain way. But we forget there’s a quantitative side to usability testing too. Each task in your test can be measured for: 1. Effectiveness — can people complete the task? → Success rate: What % of users completed the task? (80% is solid. 100% might mean your task was too easy.) → Error rate: How often do users make mistakes — and how severe are they? 2. Efficiency — how quickly do they complete the task? → Time on task: Average time spent per task. → Relative efficiency: How much of that time is spent by people who succeed at the task? 3. Satisfaction — how do they feel about it? → Post-task satisfaction: A quick rating (1–5) after each task. → Overall system usability: SUS scores or other validated scales after the full session. These metrics help you go beyond opinions and actually track improvements over time. They're especially helpful for benchmarking, stakeholder alignment, and testing design changes. We want our products to feel good, but they also need to perform well. And if you need some help, i've got a nice template for this! (see the comments) Do you use these kinds of metrics in your usability testing? UXR Study