Iota Reliability Concept of the Second Generation : Measures for Content Analysis Done by Humans or Artificial Intelligences

Iota Reliability Concept of the Second Generation Measures for Content Analysis Done by Humans or Artificial Intelligences

In educational settings, analyzing textual data via content analysis is a popular research method. The data is a valuable source of information as it offers deep insights into learning and learning outcomes. In practice, it can be used to improve classroom diagnostics and instruction. Nowadays, tech...

Descripción completa

Detalles Bibliográficos
Otros Autores: Berding, Florian, author (author), Pargmann, Julia, author
Formato: Libro electrónico
Idioma:Inglés
Publicado: Berlin, Germany : Logos Verlag Berlin GmbH 2022.
Colección:Berufs- und Wirtschaftspädagogik.
Materias:
Artificial intelligence.
Content analysis (Communication)
Ver en Biblioteca Universitat Ramon Llull:https://discovery.url.edu/permalink/34CSUC_URL/1im36ta/alma991009720294406719
Tabla de Contenidos:
  • 1 Introduction. 1
  • 2 Summarizing the Iota Concept of the First Generation. 7
  • 3 Introducing Iota Concept of the Second Generation. 11
  • 3.1 Refining the Assignment Error Matrix, the Alpha and the Beta Elements11
  • 3.2 Introducing the Chance-Correction for Alpha and Beta Elements13
  • 3.3 Refining Iota 15
  • 3.4 Reliability on the Scale Level18
  • 3.5 Assumption of Weak Superiority .20
  • 3.6 Comparing the Iota Concepts 21
  • 4 Estimation and Log-Likelihood . 25
  • 5 Simulation Study I 31
  • 5.1 Hypotheses and Design of Simulation Study I31
  • 5.2 Results of Simulation Study I .34
  • 5.2.1 Data Description and Preparation 34
  • 5.2.2 Accuracy of the Estimated Assignment Error Matrix and Categorical Sizes.35
  • 5.2.3 Accuracy of the Derived Reliability Measures .40
  • 5.3 Summary of Simulation Study I47
  • 6 Simulation Study II. 49
  • 6.1 Research Questions and Design of Simulation Study II 49
  • 6.2 Results of Simulation Study II 56
  • 6.2.1 Overview.56
  • 6.2.2 Analyses of the Deviation Between True and Estimated Sample Association/Correlation.62
  • 6.2.3 Analyses of Type I Errors.68
  • 6.2.4 Analyses of Type II Errors72
  • 6.2.5 Analysis of Correct Classification of Effect Sizes74
  • 6.3 Summary of Simulation Study II.78
  • Iota Reliability Concept of the Second Generation 7 Simulation Study III 85
  • 7.1 Design of the Study85
  • 7.2 Results of Simulation Study III .86
  • 7.2.1 Overview.86
  • 7.2.2 Potential Cut-off Values and Certainty of Reliability Effects for Deviation.89
  • 7.2.3 Potential Cut-off Values and Certainty of Reliability Effects for Type I Errors.92
  • 7.2.4 Potential Cut-off Values and Certainty of Reliability Effects for Classifying Effect Sizes 95 7.3 Summary of Simulation Study III97
  • 8 Discussion 101
  • 8.1 Conclusions101
  • 8.2 Examples for Practical Applications of iotarelr.105
  • 8.2.1 Overview.105
  • 8.2.2 Checking the Quality of Codings of New Raters 106
  • 8.2.3 Checking for Bias and Different Guidance of a Coding Scheme .110
  • 8.2.4 Improving the Quality of Codings112
  • 8.3 Limitations and Further Directions 113
  • References. 114
  • Appendix A - Confidence Intervals
  • Appendix B - Illustrations of the Relationship Between Reliability and the Deviation Between the True and Estimated Association/Correlation
  • Appendix C - Global Indices of Model Fit in Simulation Study II
  • Appendix D - Global Indices of Model Fit in Simulation Study III.

Ejemplares similares