Automatically characterizing resource quality for educational digital libraries

Steven Bethard, Philipp Wetzler, Kirsten Butcher, James H. Martin, Tamara Sumner

Research output: Chapter in Book/Report/Conference proceedingConference contribution

22 Scopus citations


With the rise of community-generated web content, the need for automatic characterization of resource quality has grown, particularly in the realm of educational digital libraries. We demonstrate how identifying concrete factors of quality for web-based educational resources can make machine learning approaches to automating quality characterization tractable. Using data from several previous studies of quality, we gathered a set of key dimensions and indicators of quality that were commonly identified by educators. We then performed a mixed-method study of digital library curation experts, showing that our characterization of quality captured the subjective processes used by the experts when assessing resource quality for classroom use. Using key indicators of quality selected from a statistical analysis of our expert study data, we developed a set of annotation guidelines and annotated a corpus of 1000 digital resources for the presence or absence of these key quality indicators. Agreement among annotators was high, and initial machine learning models trained from this corpus were able to identify some indicators of quality with as much as an 18% improvement over the baseline.

Original languageEnglish (US)
Title of host publicationJCDL'09 - Proceedings of the 2009 ACM/IEEE Joint Conference on Digital Libraries
Number of pages10
StatePublished - 2009
Externally publishedYes
Event2009 ACM/IEEE Joint Conference on Digital Libraries, JCDL'09 - Austin, TX, United States
Duration: Jun 15 2009Jun 19 2009

Publication series

NameProceedings of the ACM/IEEE Joint Conference on Digital Libraries
ISSN (Print)1552-5996


Other2009 ACM/IEEE Joint Conference on Digital Libraries, JCDL'09
Country/TerritoryUnited States
CityAustin, TX


  • Educational digital library
  • Learning resource
  • Machine learning
  • Quality

ASJC Scopus subject areas

  • General Engineering


Dive into the research topics of 'Automatically characterizing resource quality for educational digital libraries'. Together they form a unique fingerprint.

Cite this