Goodness of fit test - overview

• H₀: the population proportions in each of the $J$ conditions are $\pi_1$, $\pi_2$, $\ldots$, $\pi_J$

• H₀: the probability of drawing an observation from condition 1 is $\pi_1$, the probability of drawing an observation from condition 2 is $\pi_2$, $\ldots$, the probability of drawing an observation from condition $J$ is $\pi_J$

• H₀: the population medians for the $I$ groups are equal

• H₀: the population scores in any of the $I$ groups are not systematically higher or lower than the population scores in any of the other groups

• H₀: P(an observation from population $g$ exceeds an observation from population $h$) = P(an observation from population $h$ exceeds an observation from population $g$), for each pair of groups.

• H₀: P(first score of a pair exceeds second score of a pair) = P(second score of a pair exceeds first score of a pair)

• H₀: the population median of the difference scores is equal to zero

• H₀: $\beta_1 = \beta_2 = \ldots = \beta_K = 0$

• H₀: $\beta_k = 0$
  or in terms of odds ratio:
• H₀: $e^{\beta_k} = 1$

• H₀: $\beta_k = 0$
  or in terms of odds ratio:
• H₀: $e^{\beta_k} = 1$

• H₀ for main and interaction effects together (model): no main effects and interaction effect
• H₀ for independent variable A: no main effect for A
• H₀ for independent variable B: no main effect for B
• H₀ for the interaction term: no interaction effect between A and B

• H₁: the population proportions are not all as specified under the null hypothesis

• H₁: the probabilities of drawing an observation from each of the conditions are not all as specified under the null hypothesis

• H₁: not all of the population medians for the $I$ groups are equal

• H₁: the poplation scores in some groups are systematically higher or lower than the population scores in other groups

• H₁: for at least one pair of groups:
  P(an observation from population $g$ exceeds an observation from population $h$) $\neq$ P(an observation from population $h$ exceeds an observation from population $g$)

• H₁ two sided: P(first score of a pair exceeds second score of a pair) $\neq$ P(second score of a pair exceeds first score of a pair)
• H₁ right sided: P(first score of a pair exceeds second score of a pair) > P(second score of a pair exceeds first score of a pair)
• H₁ left sided: P(first score of a pair exceeds second score of a pair) < P(second score of a pair exceeds first score of a pair)

• H₁ two sided: the population median of the difference scores is different from zero
• H₁ right sided: the population median of the difference scores is larger than zero
• H₁ left sided: the population median of the difference scores is smaller than zero

• H₁: not all population regression coefficients are 0

• H₁: $\beta_k \neq 0$
  or in terms of odds ratio:
• H₁: $e^{\beta_k} \neq 1$
  If defined as Wald $ = \dfrac{b_k}{SE_{b_k}}$ (see 'Test statistic'), also one sided alternatives can be tested:
• H₁ right sided: $\beta_k > 0$
• H₁ left sided: $\beta_k < 0$

• H₁: $\beta_k \neq 0$
  or in terms of odds ratio:
• H₁: $e^{\beta_k} \neq 1$

• H₁ for main and interaction effects together (model): there is a main effect for A, and/or for B, and/or an interaction effect
• H₁ for independent variable A: there is a main effect for A
• H₁ for independent variable B: there is a main effect for B
• H₁ for the interaction term: there is an interaction effect between A and B

• Sample size is large enough for $X^2$ to be approximately chi-squared distributed. Rule of thumb: all $J$ expected cell counts are 5 or more
• Sample is a simple random sample from the population. That is, observations are independent of one another

• Group 1 sample is a simple random sample (SRS) from population 1, group 2 sample is an independent SRS from population 2, $\ldots$, group $I$ sample is an independent SRS from population $I$. That is, within and between groups, observations are independent of one another

• Sample of 'blocks' (usually the subjects) is a simple random sample from the population. That is, blocks are independent of one another

• Difference scores are normally distributed in the population
• Sample of difference scores is a simple random sample from the population of difference scores. That is, difference scores are independent of one another

• Scores are normally distributed in the population
• Sample is a simple random sample from the population. That is, observations are independent of one another

• Sample of pairs is a simple random sample from the population of pairs. That is, pairs are independent of one another

• In the population, the relationship between the independent variables and the log odds $\ln (\frac{\pi_{y=1}}{1 - \pi_{y=1}})$ is linear
• The residuals are independent of one another

• Variables are measured without error

• Multicollinearity
• Outliers

• Within each of the $I \times J$ populations, the scores on the dependent variable are normally distributed
• The standard deviation of the scores on the dependent variable is the same in each of the $I \times J$ populations
• For each of the $I \times J$ groups, the sample is an independent and simple random sample from the population defined by that group. That is, within and between groups, observations are independent of one another
• Equal sample sizes for each group make the interpretation of the ANOVA output easier (unequal sample sizes result in overlap in the sum of squares; this is advanced stuff)

$H = \dfrac{12}{N (N + 1)} \sum \dfrac{R^2_i}{n_i} - 3(N + 1)$

Here $N$ is the total sample size, $R_i$ is the sum of ranks in group $i$, and $n_i$ is the sample size of group $i$. Remember that multiplication precedes addition, so first compute $\frac{12}{N (N + 1)} \times \sum \frac{R^2_i}{n_i}$ and then subtract $3(N + 1)$.

• $X^2 = D_{null} - D_K = \mbox{null deviance} - \mbox{model deviance} $
  $D_{null}$, the null deviance, is conceptually similar to the total variance of the dependent variable in OLS regression analysis. $D_K$, the model deviance, is conceptually similar to the residual variance in OLS regression analysis.

• Wald $ = \dfrac{b_k^2}{SE^2_{b_k}}$
• Wald $ = \dfrac{b_k}{SE_{b_k}}$
  

• $X^2 = D_{K-1} - D_K$
  $D_{K-1}$ is the model deviance, where independent variable $k$ is excluded from the model. $D_{K}$ is the model deviance, where independent variable $k$ is included in the model.

• $F = \dfrac{\mbox{mean square model}}{\mbox{mean square error}}$

• $F = \dfrac{\mbox{mean square A}}{\mbox{mean square error}}$

• $F = \dfrac{\mbox{mean square B}}{\mbox{mean square error}}$

• $F = \dfrac{\mbox{mean square interaction}}{\mbox{mean square error}}$

For large samples, approximately the chi-squared distribution with $I - 1$ degrees of freedom.

For small samples, the exact distribution of $H$ should be used.

• chi-squared distribution with $K$ (number of independent variables) degrees of freedom

• If defined as Wald $ = \dfrac{b_k^2}{SE^2_{b_k}}$: approximately the chi-squared distribution with 1 degree of freedom
• If defined as Wald $ = \dfrac{b_k}{SE_{b_k}}$: approximately the standard normal distribution

• chi-squared distribution with 1 degree of freedom

• $F$ distribution with $(I - 1) + (J - 1) + (I - 1) \times (J - 1)$ (df model, numerator) and $N - (I \times J)$ (df error, denominator) degrees of freedom

• $F$ distribution with $I - 1$ (df A, numerator) and $N - (I \times J)$ (df error, denominator) degrees of freedom

• $F$ distribution with $J - 1$ (df B, numerator) and $N - (I \times J)$ (df error, denominator) degrees of freedom

• $F$ distribution with $(I - 1) \times (J - 1)$ (df interaction, numerator) and $N - (I \times J)$ (df error, denominator) degrees of freedom

• Check if $X^2$ observed in sample is equal to or larger than critical value $X^{2*}$ or
• Find $p$ value corresponding to observed $X^2$ and check if it is equal to or smaller than $\alpha$

• Check if $X^2$ observed in sample is equal to or larger than critical value $X^{2*}$ or
• Find $p$ value corresponding to observed $X^2$ and check if it is equal to or smaller than $\alpha$

• Check if $X^2$ observed in sample is equal to or larger than critical value $X^{2*}$ or
• Find $p$ value corresponding to observed $X^2$ and check if it is equal to or smaller than $\alpha$

• Check if $t$ observed in sample is at least as extreme as critical value $t^*$ or
• Find two sided $p$ value corresponding to observed $t$ and check if it is equal to or smaller than $\alpha$

• Check if $t$ observed in sample is equal to or larger than critical value $t^*$ or
• Find right sided $p$ value corresponding to observed $t$ and check if it is equal to or smaller than $\alpha$

• Check if $t$ observed in sample is equal to or smaller than critical value $t^*$ or
• Find left sided $p$ value corresponding to observed $t$ and check if it is equal to or smaller than $\alpha$

• Check if $t$ observed in sample is at least as extreme as critical value $t^*$ or
• Find two sided $p$ value corresponding to observed $t$ and check if it is equal to or smaller than $\alpha$

• Check if $t$ observed in sample is equal to or larger than critical value $t^*$ or
• Find right sided $p$ value corresponding to observed $t$ and check if it is equal to or smaller than $\alpha$

• Check if $t$ observed in sample is equal to or smaller than critical value $t^*$ or
• Find left sided $p$ value corresponding to observed $t$ and check if it is equal to or smaller than $\alpha$

• Check if $W$ observed in sample is in the rejection region or
• Find two sided $p$ value corresponding to observed $W$ and check if it is equal to or smaller than $\alpha$

• Check if $W$ observed in sample is in the rejection region or
• Find right sided $p$ value corresponding to observed $W$ and check if it is equal to or smaller than $\alpha$

• Check if $W$ observed in sample is in the rejection region or
• Find left sided $p$ value corresponding to observed $W$ and check if it is equal to or smaller than $\alpha$

• Check if $z$ observed in sample is at least as extreme as critical value $z^*$ or
• Find two sided $p$ value corresponding to observed $z$ and check if it is equal to or smaller than $\alpha$

• Check if $z$ observed in sample is equal to or larger than critical value $z^*$ or
• Find right sided $p$ value corresponding to observed $z$ and check if it is equal to or smaller than $\alpha$

• Check if $z$ observed in sample is equal to or smaller than critical value $z^*$ or
• Find left sided $p$ value corresponding to observed $z$ and check if it is equal to or smaller than $\alpha$

• Check if $X^2$ observed in sample is equal to or larger than critical value $X^{2*}$ or
• Find $p$ value corresponding to observed $X^2$ and check if it is equal to or smaller than $\alpha$

• If defined as Wald $ = \dfrac{b_k^2}{SE^2_{b_k}}$: same procedure as for the chi-squared tests. Wald can be interpret as $X^2$
• If defined as Wald $ = \dfrac{b_k}{SE_{b_k}}$: same procedure as for any $z$ test. Wald can be interpreted as $z$.

• Check if $F$ observed in sample is equal to or larger than critical value $F^*$ or
• Find $p$ value corresponding to observed $F$ and check if it is equal to or smaller than $\alpha$

• Proportion variance explained $R^2$:
  Proportion variance of the dependent variable $y$ explained by the independent variables and the interaction effect together:
  $$ \begin{align} R^2 &= \dfrac{\mbox{sum of squares model}}{\mbox{sum of squares total}} \end{align} $$ $R^2$ is the proportion variance explained in the sample. It is a positively biased estimate of the proportion variance explained in the population.
  
  
• Proportion variance explained $\eta^2$:
  Proportion variance of the dependent variable $y$ explained by an independent variable or interaction effect:
  $$ \begin{align} \eta^2_A &= \dfrac{\mbox{sum of squares A}}{\mbox{sum of squares total}}\\ \\ \eta^2_B &= \dfrac{\mbox{sum of squares B}}{\mbox{sum of squares total}}\\ \\ \eta^2_{int} &= \dfrac{\mbox{sum of squares int}}{\mbox{sum of squares total}} \end{align} $$ $\eta^2$ is the proportion variance explained in the sample. It is a positively biased estimate of the proportion variance explained in the population.
  
  
• Proportion variance explained $\omega^2$:
  Corrects for the positive bias in $\eta^2$ and is equal to:
  $$ \begin{align} \omega^2_A &= \dfrac{\mbox{sum of squares A} - \mbox{degrees of freedom A} \times \mbox{mean square error}}{\mbox{sum of squares total} + \mbox{mean square error}}\\ \\ \omega^2_B &= \dfrac{\mbox{sum of squares B} - \mbox{degrees of freedom B} \times \mbox{mean square error}}{\mbox{sum of squares total} + \mbox{mean square error}}\\ \\ \omega^2_{int} &= \dfrac{\mbox{sum of squares int} - \mbox{degrees of freedom int} \times \mbox{mean square error}}{\mbox{sum of squares total} + \mbox{mean square error}}\\ \end{align} $$ $\omega^2$ is a better estimate of the explained variance in the population than $\eta^2$. Only for balanced designs (equal sample sizes).
  
  
• Proportion variance explained $\eta^2_{partial}$: $$ \begin{align} \eta^2_{partial\,A} &= \frac{\mbox{sum of squares A}}{\mbox{sum of squares A} + \mbox{sum of squares error}}\\ \\ \eta^2_{partial\,B} &= \frac{\mbox{sum of squares B}}{\mbox{sum of squares B} + \mbox{sum of squares error}}\\ \\ \eta^2_{partial\,int} &= \frac{\mbox{sum of squares int}}{\mbox{sum of squares int} + \mbox{sum of squares error}} \end{align} $$

• One sample $t$ test on the difference scores.
• Repeated measures ANOVA with one dichotomous within subjects factor.

• McNemar's test: $W = b$, and $z^2 = X^2$
• Friedman test with two related groups

• Put your categorical variable in the box below Test Variable List
• Fill in the population proportions / probabilities according to $H_0$ in the box below Expected Values. If $H_0$ states that they are all equal, just pick 'All categories equal' (default)

• Put your dependent variable in the box below Test Variable List and your independent (grouping) variable in the box below Grouping Variable
• Click on the Define Range... button. If you can't click on it, first click on the grouping variable so its background turns yellow
• Fill in the smallest value you have used to indicate your groups in the box next to Minimum, and the largest value you have used to indicate your groups in the box next to Maximum
• Continue and click OK

• Put the $k$ variables containing the scores for the $k$ related groups in the white box below Test Variables
• Under Test Type, select Cochran's Q test

• Put the two paired variables in the boxes below Variable 1 and Variable 2

• Put your variable in the box below Test Variable(s)
• Fill in the value for $\mu_0$ in the box next to Test Value

• Put the two paired variables in the boxes below Variable 1 and Variable 2
• Under Test Type, select the Sign test

• Put your dependent variable in the box below Dependent and your independent (predictor) variables in the box below Covariate(s)

• Put your dependent (quantitative) variable in the box below Dependent Variable and your two independent (grouping) variables in the box below Fixed Factor(s)

• Put your categorical variable in the box below Variable
• Click on Expected Proportions and fill in the population proportions / probabilities according to $H_0$ in the boxes below Ratio. If $H_0$ states that they are all equal, you can leave the ratios equal to the default values (1)

• Put your dependent variable in the box below Dependent Variables and your independent (grouping) variable in the box below Grouping Variable

• Put the $k$ variables containing the scores for the $k$ related groups in the box below Measures

• Put the two paired variables in the box below Paired Variables, one on the left side of the vertical line and one on the right side of the vertical line
• Under Hypothesis, select your alternative hypothesis

• Put your variable in the box below Dependent Variables
• Under Hypothesis, fill in the value for $\mu_0$ in the box next to Test Value, and select your alternative hypothesis

• Put the two paired variables in the box below Measures

• Put your dependent variable in the box below Dependent Variable and your independent variables of interval/ratio level in the box below Covariates
• If you also have code (dummy) variables as independent variables, you can put these in the box below Covariates as well
• Instead of transforming your categorical independent variable(s) into code variables, you can also put the untransformed categorical independent variables in the box below Factors. Jamovi will then make the code variables for you 'behind the scenes'

• Put your dependent (quantitative) variable in the box below Dependent Variable and your two independent (grouping) variables in the box below Fixed Factors