Sensitivity Power Analysis 민감도 검증력 분석

Sensitivity Power Analysis 민감도 검증력 분석

민감도 검증력 분석을 하는 이유

X가 Y에 영향을 준다고 했을 때, 실질적으로 그 영향이 유의미하더라도 아주 작아 별로 관련이 없거나, 다른 사건에 의해 쉽게 제거될 수 있음. 따라서 p 값이 .05 미만으로 유의미하다는 것만으로는 소용이 없는 경우가 많음. 즉 실질적으로 중요한 영향인지를 알아보는 것이 민감도 검증력 분석임

the sensitivity power analysis signifies  "the smallest effect size you care about".

연구를 계획할 때 연구자는 먼저 실용적/이론적 바탕을 기반으로 연구 질문에 대해 최소한의 중요한 효과 크기 (effect size)를 결정해야함. 여기서 말하는 "최소한의 중요한"이란, 연구하는 현상에 따라 달라질 수 있음 (케바케라는 말..). 

이상적으로는, 연구자들은 일반적인 검증력 분석(power analysis)을 통해 효과크기를 고려한 표본 크기를 결정하게 됨. 하지만, 실질적으로 연구 할 때 제약이 있는 경우가 있음. 예를 들면 자원(인력, 시간, 돈)이나 연구 모집단부터 크기가 작은 경우(표본을 구하기 힘든 경우)를 예로 들 수 있음. 이러한 경우 민감도 검증력 분석을 사용해, 탐지할 수 있는 최소 효과 크기를 계산하고 샘플로부터 관측된 효과크기를 비교할 수 있음.

여러 경우의 수..

1. Your observed effect size is greater than the predicted minimal detectable effect size, and your test indicates a statistically significant effect

You reject the null-hypothesis and claim evidence in favor of the effect. You can now focus on discussing how far your observed and minimum detectable effect sizes were above or below the lower bound of practical/theoretical interest (minimal detectable/interesting effect size), focusing on practical significance (this is more important than p>.05 or "statistical significance"). You may argue that your study was over-/under-powered depending on whether the minimum detectable effect size was below/above the threshold of practical/theoretical interest. You may argue that the relevance of your finding was high or low, depending on whether the observed effect size was above or below the threshold of practical/theoretical interest, respectively. 

1. Your observed effect size is greater than the predicted minimal detectable effect size, your test result is not statistically significant 

This case should not occur if power analysis and statistical testing were based on the same alpha and n. However, power analysis is often performed before the study. The realized final sample may be smaller than planned. Further, if you expressed your minimal detectable effect size in terms of non-standardized effect sizes (so NOT Cohen's d, eta^2, r etc.), the variance in measurements factors in (in case of repeated measurements the correlation between measurements, as well). You may have assumed a lower SD in your power sensitivity analysis (which is based on standardized effect sizes) than what was observed in the sample. You can discuss this result, and additionally provide an updated power sensitivity analysis demonstrating the "realized" sensitivity of the study. >> see case 4

1. Your observed effect size is smaller than the predicted minimum detectable effect size, your test indicates a statistically significant effect

In other words, our observed effect size was smaller than the smallest "true" effect size that your study was set to detect "reliably", nevertheless the statistical test indicated statistical significance. This scenario is possible, especially when a high beta-value was chosen. You reject the null-hypothesis and claim evidence in favor of the effect. You should discuss that despite evidence for an effect, the observed effect size was below the bound of what your study was set to detect "reliably". This may put the reliability of results in question, depending on the standards you've set. Beyond that, see Case 1 for discussing practical/theoretical significance.

1. Your observed effect size is smaller than the minimal detectable effect size, and you found a null effect.
   • If the minimal detectable effect size is above the minimum of practical/theoretical interest, your results are inconclusive and your study was underpowered. A true effect with an effect size of interest may (or may not) exist, but your study did not provide enough information to be reasonably certain. 
   • If the minimal detectable effect size was at or below the threshold of minimum practical/theoretical interest you can argue that your study was powered sufficiently and that the true effect in question is likely smaller than what is practically/theoretically relevant. 
2. Your observed effect size was below the smallest "true" effect size that your study was set to detect "reliably" and no evidence for an effect was found. Here, the results of your sensitivity analysis are most interesting: A negative test result may equally mean "no effect", or "indeed an effect, but too small to be certain, given this sample size". The sensitivity power analysis can help you to better interpret these results. It depends on what your minimal detectable effect size represents:

In summary, it is good scientific practice to both report minimum detectable effect size and the minimum effect size of practical/theoretical interest, regardless of your test outcomes. These metrics help to interpret your results by focusing on effect sizes instead of statistical significance and therefore bridge the argumentative gap between "statistical" and "practical" significance.

 

출처: https://stats.stackexchange.com/questions/446995/how-to-interpret-sensitivity-power-analyses#:~:text=The%20sensitivity%20power%20analysis%20can,and%20your%20study%20was%20underpowered.