Cohen's kappa and weighed kappa for multiple raters, adapted from the psych::cohen.kappa function (Revelle, 2020). Light's method is used to combine the scores from multiple raters.

kappa(
data,
weight = NULL,
confint = FALSE,
alpha = 0.05,
k = NULL,
n = NULL,
...
)

## Arguments

data

A data.frame or table with equal number of columns and rows. Or data.frame that contains the scores for each rater in each column.

weight

matrix of weights for the weighed kappa.

confint

Logical indicator for confidence interval

alpha

Confidence interval level, default = 0.05.

k

number of raters; default k = ncol(data).

n

sample size; default n = nrow(data).

...

options for sumtable if is.data.frame(data)

vector

## References

Revelle, W. (2020) psych: Procedures for Personality and Psychological Research, Northwestern University, Evanston, Illinois, USA, https://CRAN.R-project.org/package=psych Version = 2.0.12,. Light, R. J. (1971) Measures of response agreement for qualitative data: Some generalizations and alternatives, Psychological Bulletin, 76, 365-377.

## Examples

df <- data.frame(r1=factor(c(1,0,1,0,0,1,1,0,0,0,1,1,0,1,1)),
r2=factor(c(1,1,1,1,0,1,1,0,0,0,1,1,0,1,0)),
r3=factor(c(1,1,1,0,0,0,1,1,1,0,0,1,0,1,1)),
r4=factor(c(1,1,1,1,1,1,1,1,1,1,1,1,1,1,1)))
table <- sumtable(df=df, ratings=c("r1", "r2", "r3", "r4"), levels=c("0","1"))
kappa(df)
#>          kappa weighted kappa
#>      0.2028986      0.2028986
kappa(table)
#>          kappa weighted kappa
#>      0.2028986      0.2028986

df <- data.frame(r1=factor(c(1,2,2,0,3,3,1,0,3,0,2,2,0,3,1)),
r2=factor(c(1,1,1,0,3,3,1,0,1,0,2,2,0,2,1)),
r3=factor(c(1,1,1,3,3,2,1,0,1,0,2,2,0,3,1)),
r4=factor(c(1,2,1,0,3,3,1,0,3,0,2,2,0,2,1)))
table <- sumtable(df=df, ratings=c("r1", "r2", "r3", "r4"), levels=c("0","1", "2", "3"))
kappa(df)
#>          kappa weighted kappa
#>      0.6858639      0.7213395
kappa(table)
#>          kappa weighted kappa
#>      0.6858639      0.7213395
kappa(df, confint = TRUE)
#>                    kappa     lower     upper
#> kappa          0.6858639 0.5691753 0.8025525
#> weighted kappa 0.7213395 0.5689155 0.8737636
kappa(table)
#>          kappa weighted kappa
#>      0.6858639      0.7213395