*** Updated 10/27/11: Original text appended in strike. ***
Bill Bolstad's response to Xi'an's review of his book Understanding Computational Bayesian Statistics included the following comment, which I found interesting:
Frequentist p-values are constructed in the parameter dimension using a probability distribution defined only in the observation dimension. Bayesian credible intervals are constructed in the parameter dimension using a probability distribution in the parameter dimension. I think that is more straightforward.
Classical statistics is concerned with the distribution of statistics that estimate a fixed population parameter. And, statistics are clearly constructed in the observation dimension. But, consider that certain statistics may come very close in value to a population parameter as the sample grows. In this sense, a classical procedure may not need to consider the parameter dimension.
Classical statistics is concerned with the distribution of statistics that estimate a fixed population parameter. And, statistics are clearly constructed in the observation dimension. But, consider that a statistic evaluated at the population level (i.e., computed using all members of a finite population, or the limit in an infinite sequence of observations) is also a population parameter. In this sense, there is no distinction between the observation and parameter dimensions.
It seems natural to ask: "What is the region that contains the value of quantity X with some level of confidence?". Equivalently, the "value of quantity X" may be the "value of a parameter", or the "value of a statistic in large samples or a census". The Bayesian might construct a posterior predictive density for the value of a statistic on 'new' observations.
It seems natural to ask: "What is the value of a statistic if computed at the population level?" Classical statistics offer an answer. The Bayesian analog is not immediately clear.
Perhaps 'classical statistics' should be just 'statistics', and 'Bayesian statistics', just 'parameters'. 😉