Agreement for polytomous outcomes

This document describes the use of the Agree package for two data examples that are used in the paper on specific agreement on polytomous outcomes in the situation of more than two raters (de Vet, Mullender, and Eekhout 2018). The first data example is an example of ordinal ratings and the second example of nominal rating.

library(Agree)

Ordinal data example

For the ordinal data example we use data from a study by Dikmans et al. (2017). This data is based on photographs of breasts of 50 women after breast reconstruction. The photographs are independently scored by 5 surgeons, the patients, and three mothers. They each rated the quality of the reconstruction on a 5 point ordinal scale with the verbal anchors on the left side ‘very dissatisfied’ on the left end and on the right end ‘very satisfied’ on the right end. They specifically rated the volume, shape, symmetry, scars and nipple. In this paper we use the data of 4 surgeons because one surgeon had some missing values and we look at the rates for symmetry. Data set 1 is used for the example of ordinal categories.

data(breast)

variable <- "symmetry"
raters <- c("PCH1", "PCH2", "PCH3", "PCH4")
ratersvars <- paste(raters, variable, sep="_")
data1 <- data.frame(breast[ratersvars])

data1 %>% head()
##       PCH1_symmetry  PCH2_symmetry PCH3_symmetry PCH4_symmetry
## 1         satisfied very satisfied     satisfied     satisfied
## 2           neutral        neutral  dissatisfied       neutral
## 3         satisfied        neutral       neutral       neutral
## 4      dissatisfied        neutral  dissatisfied  dissatisfied
## 5 very dissatisfied      satisfied  dissatisfied     satisfied
## 6         satisfied        neutral     satisfied     satisfied

Agreement table

First the agreement table are summed for all rater combinations into one agreement table. Then the off diagonal cells are averaged to obtain symmetry agreement tables.

sumtable(data1,offdiag = FALSE) %>% kable()
very dissatisfieddissatisfiedneutralsatisfiedvery satisfied
very dissatisfied61020
dissatisfied0192381
neutral01839274
satisfied04282816
very satisfied0023638
sumtable(data1,offdiag = TRUE) %>% kable()
very dissatisfieddissatisfiedneutralsatisfiedvery satisfied
very dissatisfied6.00.50.01.00.0
dissatisfied0.519.020.56.00.5
neutral0.020.539.027.53.0
satisfied1.06.027.528.026.0
very satisfied0.00.53.026.038.0

Agreement

From the agreement table we can calculate the agreement. And we can calculate the confidence interval around this agreement.

agreement(data1)
## [1] 0.4333333
CIagreement(data1,m=4,n=nrow(data1))
##     CIlow agreement    CIhigh 
## 0.3286321 0.4333333 0.5434725

Specific agreement

The specific agreement for polytomous data, can be defined in two ways: the agreement of for one category versus not that category (e.g. very satisfied versus all other categories) or the agreement for one category versus any other (e.g. very satistfied versus satisfied). Below the Confidence intervals for the specific agreements are bootstrapped.

specific.agreement(data1, cat1="satisfied")
## [1] 0.3163842
CIagreement(data1, cat1="satisfied")
##   BCIlow.2.5%     agreement BCIhigh.97.5% 
##     0.2183856     0.3163842     0.3980597
specific.agreement(data1, cat1="satisfied", cat2="very satisfied")
## [1] 0.5185185
CIagreement(data1, cat1="satisfied", cat2="very satisfied")
##   BCIlow.2.5%     agreement BCIhigh.97.5% 
##     0.3888754     0.5185185     0.6371872
specific.agreement(data1, cat1="satisfied", cat2="neutral")
## [1] 0.5045045
CIagreement(data1, cat1="satisfied", cat2="neutral")
##   BCIlow.2.5%     agreement BCIhigh.97.5% 
##     0.3655425     0.5045045     0.6285875

Conditional probability

We can calulate the probability of any other outcome conditional on an specific outcome.

conditional.agreement(data1) %>% kable()
prevalenceproportionvery dissatisfieddissatisfiedneutralsatisfiedvery satisfied
very dissatisfied7.50.0250.8000.0670.0000.1330.000
dissatisfied46.50.1550.0110.4090.4410.1290.011
neutral90.00.3000.0000.2280.4330.3060.033
satisfied88.50.2950.0110.0680.3110.3160.294
very satisfied67.50.2250.0000.0070.0440.3850.563

Weighted agreement

For ordinal data it might also be useful to look at the agreement when they may be one category off. So the agreement plus or minus one category, that categories is weighted (default weight=1).

weighted.agreement(data1)
## [1] 0.93
weighted.agreement(data1, weight=0.5)
## [1] 0.6816667

nominal data example

For the nominal data example we use a data set that was used in a paper by Fleis (1971). In this data patients are diagnosed in 5 categories: Depression, Personality Disorder, Schizophrenia, Neurosis, and Other by 6 raters.

data(diagnoses) 
data2 <- data.frame(lapply(diagnoses,as.factor), stringsAsFactors = TRUE)

 levels(data2$rater1) <- c("Depression", "Pers disord.", "Schizophrenia", "Neurosis", "Other")
 levels(data2$rater2) <- c("Depression", "Pers disord.", "Schizophrenia", "Neurosis", "Other")
 levels(data2$rater3) <- c("Depression", "Pers disord.", "Schizophrenia", "Neurosis", "Other")
 levels(data2$rater4) <- c("Depression", "Pers disord.", "Schizophrenia", "Neurosis", "Other")
 levels(data2$rater5) <- c("Depression", "Pers disord.", "Schizophrenia", "Neurosis", "Other")

Agreement table

First the agreement table are summed for all rater combinations into one agreement table. Then the off diagonal cells are averaged to obtain symmetry agreement tables.

sumtable(data2,offdiag = FALSE) %>% kable()
DepressionPers disord.SchizophreniaNeurosisOther
Depression23111201
Pers disord.0236175
Schizophrenia743602
Neurosis10160531
Other856243
sumtable(data2,offdiag = TRUE) %>% kable()
DepressionPers disord.SchizophreniaNeurosisOther
Depression23.00.5915.04.5
Pers disord.0.523.0516.55.0
Schizophrenia9.05.0360.04.0
Neurosis15.016.5053.01.5
Other4.55.041.543.0

Agreement

From the agreement table we can calculate the agreement. And we can calculate the confidence interval around this agreement.

agreement(data2)
## [1] 0.5933333
CIagreement(data2)
##     CIlow agreement    CIhigh 
## 0.4832393 0.5933333 0.6959584

Specific agreement

The specific agreement for polytomous data, can be defined in two ways: the agreement of for one category versus not that category (e.g. Depression versus all other categories) or the agreement for one category versus any other (e.g. Depression versus Schizophrenia). The confidence intervals for specific agreement are bootstrapped.

specific.agreement(data2, cat1="Depression")
## [1] 0.4423077
CIagreement(data2, cat1="Depression")
##   BCIlow.2.5%     agreement BCIhigh.97.5% 
##     0.1428571     0.4423077     0.6572465
specific.agreement(data2, cat1="Pers disord.")
## [1] 0.46
CIagreement(data2, cat1="Pers disord.")
##   BCIlow.2.5%     agreement BCIhigh.97.5% 
##     0.2272727     0.4600000     0.6818561
specific.agreement(data2, cat1="Schizophrenia")
## [1] 0.6666667
CIagreement(data2, cat1="Schizophrenia")
##   BCIlow.2.5%     agreement BCIhigh.97.5% 
##     0.4687500     0.6666667     0.8200000
specific.agreement(data2, cat1="Neurosis")
## [1] 0.6162791
CIagreement(data2, cat1="Neurosis")
##   BCIlow.2.5%     agreement BCIhigh.97.5% 
##     0.4838575     0.6162791     0.7317286
specific.agreement(data2, cat1="Other")
## [1] 0.7413793
CIagreement(data2, cat1="Other")
##   BCIlow.2.5%     agreement BCIhigh.97.5% 
##     0.3823529     0.7413793     0.9242424

Conditional agreement

conditional.agreement(data2) %>% kable()
prevalenceproportionDepressionPers disord.SchizophreniaNeurosisOther
Depression520.17333330.4420.0100.1730.2880.087
Pers disord.500.16666670.0100.4600.1000.3300.100
Schizophrenia540.18000000.1670.0930.6670.0000.074
Neurosis860.28666670.1740.1920.0000.6160.017
Other580.19333330.0780.0860.0690.0260.741

References

de Vet, H. C. W., M. G. Mullender, and I Eekhout. 2018. “Specific Agreement on Ordinal and Multiple Nominal Outcomes Can Be Calculated for More Than Two Raters.” Journal of Clinical Epidemiology 96: 47–53. https://www.jclinepi.com/article/S0895-4356(16)30837-X/abstract.
Dikmans, R. E., L. Nene, M. B. Bouman, H. C. W. de Vet, M. Mireau, M. E. Buncamper, H. Winters, M. Ritt, and M. G. Mullender. 2017. “The Aesthetic Items Scale: A Tool for the Evaluation of Aesthetic Outcome After Breast Reconstruction.” Plastic and Reconstructive Surgery. Global Open. 5 (3): e1254.
Fleis, J. L. 1971. “Measuring Nominal Scale Agreement Among Many Raters.” Psychologiclal Bulletin 76 (5): 378–82.
Iris Eekhout, PhD
Iris Eekhout, PhD
Statistician

Iris works on a variety of projects as methodologist and statistical analyst related to child health, e.g. measuring child development (D-score) and adaptive screenings for psycho-social problems (psycat).

Related