# Agreement for polytomous outcomes

This document describes the use of the Agree package for two data examples that are used in the paper on specific agreement on polytomous outcomes in the situation of more than two raters . The first data example is an example of ordinal ratings and the second example of nominal rating.

``library(Agree)``
``````##
## Attaching package: 'Agree'``````
``````## The following object is masked from 'package:base':
##
##     kappa``````

## Ordinal data example

For the ordinal data example we use data from a study by . This data is based on photographs of breasts of 50 women after breast reconstruction. The photographs are independently scored by 5 surgeons, the patients, and three mothers. They each rated the quality of the reconstruction on a 5 point ordinal scale with the verbal anchors on the left side ‘very dissatisfied’ on the left end and on the right end ‘very satisfied’ on the right end. They specifically rated the volume, shape, symmetry, scars and nipple. In this paper we use the data of 4 surgeons because one surgeon had some missing values and we look at the rates for symmetry. Data set 1 is used for the example of ordinal categories.

``````data(breast)

variable <- "symmetry"
raters <- c("PCH1", "PCH2", "PCH3", "PCH4")
ratersvars <- paste(raters, variable, sep="_")
data1 <- data.frame(breast[ratersvars])

``````##       PCH1_symmetry  PCH2_symmetry PCH3_symmetry PCH4_symmetry
## 1         satisfied very satisfied     satisfied     satisfied
## 2           neutral        neutral  dissatisfied       neutral
## 3         satisfied        neutral       neutral       neutral
## 4      dissatisfied        neutral  dissatisfied  dissatisfied
## 5 very dissatisfied      satisfied  dissatisfied     satisfied
## 6         satisfied        neutral     satisfied     satisfied``````

### Agreement table

First the agreement table are summed for all rater combinations into one agreement table. Then the off diagonal cells are averaged to obtain symmetry agreement tables.

``sumtable(data1,offdiag = FALSE) %>% kable()``
very dissatisfieddissatisfiedneutralsatisfiedvery satisfied
very dissatisfied61020
dissatisfied0192381
neutral01839274
satisfied04282816
very satisfied0023638
``sumtable(data1,offdiag = TRUE) %>% kable()``
very dissatisfieddissatisfiedneutralsatisfiedvery satisfied
very dissatisfied6.00.50.01.00.0
dissatisfied0.519.020.56.00.5
neutral0.020.539.027.53.0
satisfied1.06.027.528.026.0
very satisfied0.00.53.026.038.0

### Agreement

From the agreement table we can calculate the agreement. And we can calculate the confidence interval around this agreement.

``agreement(data1)``
``````## overall agreement
##         0.4333333``````
``agreement(data1, confint = TRUE)``
``````## overall agreement             lower             upper
##         0.4333333         0.3286321         0.5434725``````

### Specific agreement

The specific agreement for polytomous data, can be defined in two ways: the agreement of for one category versus not that category (e.g. very satisfied versus all other categories) or the agreement for one category versus any other (e.g. very satistfied versus satisfied). Below the Confidence intervals for the specific agreements are bootstrapped.

``agreement(data1, specific="satisfied", confint = TRUE)``
``````##                                       p     lower     upper
## overall agreement             0.4333333 0.3286321 0.5434725
## specific agreement: satisfied 0.3163842 0.2083333 0.4021164``````
``agreement(data1, specific=c("satisfied", "very satisfied"), confint = TRUE)``
``````##                                                         p     lower     upper
## overall agreement                               0.4333333 0.3286321 0.5434725
## specific agreement: satisfied vs very satisfied 0.5185185 0.3749708 0.6465067``````
``agreement(data1, specific= c("satisfied","neutral"), confint = TRUE)``
``````##                                                  p     lower     upper
## overall agreement                        0.4333333 0.3286321 0.5434725
## specific agreement: satisfied vs neutral 0.5045045 0.3683503 0.6346415``````

### Conditional probability

We can calulate the probability of any other outcome conditional on an specific outcome.

``conditional.agreement(data1) %>% kable()``
prevalenceproportionvery dissatisfieddissatisfiedneutralsatisfiedvery satisfied
very dissatisfied7.50.0250.8000.0670.0000.1330.000
dissatisfied46.50.1550.0110.4090.4410.1290.011
neutral90.00.3000.0000.2280.4330.3060.033
satisfied88.50.2950.0110.0680.3110.3160.294
very satisfied67.50.2250.0000.0070.0440.3850.563

### Weighted agreement

For ordinal data it might also be useful to look at the agreement when they may be one category off. So the agreement plus or minus one category, that categories is weighted (default weight=1).

``weighted.agreement(data1)``
``## [1] 0.93``
``weighted.agreement(data1, weight=0.5)``
``## [1] 0.6816667``

## nominal data example

For the nominal data example we use a data set that was used in a paper by . In this data patients are diagnosed in 5 categories: Depression, Personality Disorder, Schizophrenia, Neurosis, and Other by 6 raters.

``````data(diagnoses)
data2 <- data.frame(lapply(diagnoses,as.factor), stringsAsFactors = TRUE)

levels(data2\$rater1) <- c("Depression", "Pers disord.", "Schizophrenia", "Neurosis", "Other")
levels(data2\$rater2) <- c("Depression", "Pers disord.", "Schizophrenia", "Neurosis", "Other")
levels(data2\$rater3) <- c("Depression", "Pers disord.", "Schizophrenia", "Neurosis", "Other")
levels(data2\$rater4) <- c("Depression", "Pers disord.", "Schizophrenia", "Neurosis", "Other")
levels(data2\$rater5) <- c("Depression", "Pers disord.", "Schizophrenia", "Neurosis", "Other")``````

### Agreement table

First the agreement table are summed for all rater combinations into one agreement table. Then the off diagonal cells are averaged to obtain symmetry agreement tables.

``sumtable(data2,offdiag = FALSE) %>% kable()``
DepressionPers disord.SchizophreniaNeurosisOther
1. Personality Disorder
1. Schizophrenia
1. Neurosis
1. Other
Depression231112014398
Pers disord.023617503138
Schizophrenia74360206212
Neurosis1016053100246
Other85624300022
1. Personality Disorder
100000000
1. Schizophrenia
003000000
1. Neurosis
0111000000
1. Other
103370000
``sumtable(data2,offdiag = TRUE) %>% kable()``
DepressionPers disord.SchizophreniaNeurosisOther
1. Personality Disorder
1. Schizophrenia
1. Neurosis
1. Other
Depression23.00.59.015.04.52.51.54.54.5
Pers disord.0.523.05.016.55.00.01.57.04.0
Schizophrenia9.05.036.00.04.00.04.51.57.5
Neurosis15.016.50.053.01.50.00.017.04.5
Other4.55.04.01.543.00.00.00.014.5
1. Personality Disorder
2.50.00.00.00.00.00.00.00.0
1. Schizophrenia
1.51.54.50.00.00.00.00.00.0
1. Neurosis
4.57.01.517.00.00.00.00.00.0
1. Other
4.54.07.54.514.50.00.00.00.0

### Agreement

From the agreement table we can calculate the agreement. And we can calculate the confidence interval around this agreement.

``agreement(data2, confint = TRUE)``
``````## overall agreement             lower             upper
##         0.3955556         0.2805873         0.5200202``````

### Specific agreement

The specific agreement for polytomous data, can be defined in two ways: the agreement of for one category versus not that category (e.g. Depression versus all other categories) or the agreement for one category versus any other (e.g. Depression versus Schizophrenia). The confidence intervals for specific agreement are bootstrapped.

``agreement(data2, specific="Depression", confint = TRUE)``
``````##                                        p     lower     upper
## overall agreement              0.3955556 0.2805873 0.5200202
## specific agreement: Depression 0.3538462 0.1270909 0.5400135``````
``agreement(data2, specific="Pers disord.", confint = TRUE)``
``````##                                          p     lower     upper
## overall agreement                0.3955556 0.2805873 0.5200202
## specific agreement: Pers disord. 0.3680000 0.1881961 0.5478514``````
``agreement(data2, specific="Schizophrenia", confint = TRUE)``
``````##                                           p     lower     upper
## overall agreement                 0.3955556 0.2805873 0.5200202
## specific agreement: Schizophrenia 0.5333333 0.3748558 0.6571849``````
``agreement(data2, specific="Neurosis", confint = TRUE)``
``````##                                      p     lower     upper
## overall agreement            0.3955556 0.2805873 0.5200202
## specific agreement: Neurosis 0.4930233 0.3857011 0.5887080``````
``agreement(data2, specific="Other", confint = TRUE)``
``````##                                   p     lower     upper
## overall agreement         0.3955556 0.2805873 0.5200202
## specific agreement: Other 0.5931034 0.3110606 0.7345909``````

### Conditional agreement

``conditional.agreement(data2) %>% kable()``
prevalenceproportionDepressionPers disord.SchizophreniaNeurosisOther
1. Personality Disorder
1. Schizophrenia
1. Neurosis
1. Other
Depression65.00.14444440.3540.0080.1380.2310.0690.0380.0230.0690.069
Pers disord.62.50.13888890.0080.3680.0800.2640.0800.0000.0240.1120.064
Schizophrenia67.50.15000000.1330.0740.5330.0000.0590.0000.0670.0220.111
Neurosis107.50.23888890.1400.1530.0000.4930.0140.0000.0000.1580.042
Other72.50.16111110.0620.0690.0550.0210.5930.0000.0000.0000.200
1. Personality Disorder
2.50.00555561.0000.0000.0000.0000.0000.0000.0000.0000.000
1. Schizophrenia
7.50.01666670.2000.2000.6000.0000.0000.0000.0000.0000.000
1. Neurosis
30.00.06666670.1500.2330.0500.5670.0000.0000.0000.0000.000
1. Other
35.00.07777780.1290.1140.2140.1290.4140.0000.0000.0000.000

## References

de Vet, H. C. W., M. G. Mullender, and I Eekhout. 2018. “Specific Agreement on Ordinal and Multiple Nominal Outcomes Can Be Calculated for More Than Two Raters.” Journal of Clinical Epidemiology 96: 47–53. https://www.jclinepi.com/article/S0895-4356(16)30837-X/abstract.
Dikmans, R. E., L. Nene, M. B. Bouman, H. C. W. de Vet, M. Mireau, M. E. Buncamper, H. Winters, M. Ritt, and M. G. Mullender. 2017. “The Aesthetic Items Scale: A Tool for the Evaluation of Aesthetic Outcome After Breast Reconstruction.” Plastic and Reconstructive Surgery. Global Open. 5 (3): e1254.
Fleis, J. L. 1971. “Measuring Nominal Scale Agreement Among Many Raters.” Psychologiclal Bulletin 76 (5): 378–82.
##### Iris Eekhout, PhD
###### Statistician

Iris works on a variety of projects as methodologist and statistical analyst related to child health, e.g. measuring child development (D-score) and adaptive screenings for psycho-social problems (psycat).