SQL, Python, Stats, and BI Interview Questions
SQL, Python, Stats, and BI Interview Questions
Outliers can be identified using the IQR method by computing the IQR as the difference between the 75th (Q3) and 25th (Q1) percentiles of the data. Outliers fall below Q1 - 1.5*IQR or above Q3 + 1.5*IQR. Treatment involves capping, transformation, or removal which should be judiciously considered as this can skew results or omit influential data points. Therefore, understanding the context of data and the reason for outliers is important before proceeding with their treatment .
Using `DISTINCT` is generally more straightforward as it scans the result set to remove duplicates. However, when optimizing or maintaining detailed control over duplicate removal, utilizing GROUP BY with HAVING clauses can provide additional flexibility and better performance tuning based on indices. `DISTINCT` tends to compute all columns' distinctness directly, which can be less efficient on large tables, whereas techniques like groupings allow optimization via specific index uses .
Type I error, or false positive, occurs when a true null hypothesis is rejected. It's critical in fields with high costs of error, such as medical testing. Controlling Type I error is typically done by setting a lower alpha level (e.g., 0.05 or 0.01). Type II error, or false negative, occurs when a false null hypothesis is not rejected, impacting again fields with significant downside from inaction, like failing to detect a malfunctioning drug. Balancing these errors is crucial for maintaining rigorous and reliable outcomes, directly influencing the confidence in the conclusions drawn from hypothesis tests .
`apply()` allows for the application of functions along an axis of the DataFrame (rows or columns) and is flexible for custom functions, but it can be slower since it operates serially. `map()` is generally used for substituting values in a Series and is faster than `apply()` since it's a vectorized form for specific substitution tasks. Vectorization directly utilizes numpy operations under the hood which are optimized for performance, making vectorized operations the fastest in Pandas for suitable operations dealing with large datasets .
Verification of normal distribution in a dataset can be accomplished through both visual and statistical methods. For a visual assessment, histograms and Q-Q plots can provide immediate graphical representation of the data's skewness and kurtosis. On the statistical front, tests like the Shapiro-Wilk test and the Kolmogorov-Smirnov test are commonly used to objectively assess normality, though they should be used with an understanding of their power dependencies on sample size. Python libraries like SciPy offer easy implementation of these tests .
To analyze a 20% drop in sales, employ a combination of tools and strategies including trend analysis through time series visualization in BI tools like Tableau or Power BI to identify any temporal patterns. Use SQL queries to segment and compare sales by various dimensions such as product categories, regions, or customer demographics to pinpoint specific areas of decline. Further exploration can involve hypothesis testing for external and internal factors alongside regression analysis to assess predictive factors affecting sales decline. It's also beneficial to utilize visual dashboards to continually monitor KPIs and adjust based on real-time data feedback .
To find the second-highest salary in an employee table without using subqueries, one can take a creative approach using window functions such as ROW_NUMBER() or DENSE_RANK() in conjunction with ORDER BY clauses as SQL providers like PostgreSQL support window function approaches. Another alternative is by using a self-join on the table to filter out the top salary and then finding the maximum of the remaining values .
`INNER JOIN` returns rows when there's at least one match in both tables, useful for intersecting data. `LEFT JOIN` returns all rows from the left table and matched rows from the right table; unmatched rows will have NULL in the right table columns. It's useful when you need all records from the left table (e.g., maintaining all employees with their department details). `RIGHT JOIN` is the converse, returning all rows from the right table. `FULL JOIN` returns all records when there's a match in either left or right table records. It's useful in scenarios where completeness of data with join keys from both sides is necessary .
In Pandas, you can merge two DataFrames using the `merge()` function by specifying the `on` parameter or `left_on` and `right_on` parameters for distinct keys. When merging on multiple columns, ensure that you handle cases of overlapping column names and potential key mismatches by validating the `how` parameter ('inner', 'outer', 'left', 'right') for the suitable join type. Performance considerations include memory usage for large datasets, so processing in chunks or filtering relevant columns before merging can help maintain efficiency while preserving data integrity .
The Central Limit Theorem (CLT) states that the distribution of sample means approximates a normal distribution as the sample size becomes large, regardless of the population’s distribution. This theorem underpins inferential statistics by justifying the application of normal probability tests and confidence intervals which depend on normality. It allows for assumptions, like approximation of binomial distributions to normal for large samples, to simplify complex analyses—a cornerstone for techniques ranging from hypothesis testing to regression analysis .