Precision - Role of Confidence Level
The confidence level is an index of certainty. For example (With N=93 per group) we might report that the treatment improves the response rate by 20 percentage points, with a 95% confidence interval of plus/minus some 13 points (7 to 33). This means that in 95% of all possible studies, the confidence interval computed in this manner will include the true effect. The confidence level is typically set in the range of 99% to 80%.
The 95% confidence interval will be wider than the 90% interval, which in turn will be wider than the 80% interval. For example, compare Figure 4, which shows the expected value of the 80% confidence interval, with Figure 3 which is based on the 95% confidence interval. With a sample of 100 cases per group the 80% confidence interval is plus/minus some 9 points (11 to 29) while the 95% confidence interval is plus/minus some 13 points (7 to 34).
The researcher may elect to report the confidence interval for more than one level of confidence, for example "The treatment improves the cure rate by 10 points (80% confidence interval 11 to 29, and 95% confidence interval 7 to 34. It has also been suggested that the researcher use a graph to report the full continuum of confidence intervals by as a function of confidence levels. (See Poole, 1987a,b,c; Walker, 1986a,b)
Precision - Role of Tails
The researcher may elect to compute two-tailed or one-tailed bounds for the confidence "interval". A two-tailed confidence interval extends from some finite value below the observed effect to another finite value above the observed effect. A one-tailed confidence "interval" extends from minus infinity to some value above the observed effect, or from some value below the observed effect to plus infinity (the logic of the procedure may impose a limit other than infinity, such as 0 and 1 for proportions). A one-tailed confidence interval might be used if were concerned only with effects in one direction. For example, we might report that a drug increases the remission rate by 20 points with a 95% lower limit of 15 points (the upper limit is of no interest).
For any given sample size, dispersion and confidence level, a one-tailed confidence "interval" is "narrower" than a two tailed interval in the sense that the distance from the observed effect to the computed boundary is smaller for the one-tailed interval (the one-tailed case is not really an interval, since it has only one boundary). As was the case with power analysis, however, the decision to work with a one-tailed procedure rather than a two-tailed procedure should be made on substantive grounds, rather than as a means for yielding a more precise estimate of the effect size.