Types of Quality Metrics

This is a synthesis of possible criteria for quality metrics based on a wide array of frameworks and literature (see citations), organized roughly into the seven categories identified by the Bruce/Hillmann framework. Individual criteria could be used to create measurable benchmarks depending on the priorities and technical capabilities of a specific digital collection/organization for “better-than-minimal” benchmarks.

Additionally, the scope of the metadata benchmarks focuses specifically on descriptive metadata values, but this list includes aspects contributing to overall quality, such as the role of data structures, sharability, technical maintenance and other components of digital collection description and maintenance. However, note that technical specifications are still primarily within the context of descriptive metadata (e.g., may include technical criteria related to displaying values but not general UI/UX of digital libraries)

  • Technical specifications: relate to system requirements, modeling, shareability, and other aspects outside of descriptive metadata values

  • Metadata values: applies to values in metadata fields, primarily descriptive metadata, unless otherwise noted

Accessibility

TECHNICAL SPECIFICATIONS

METADATA VALUES

  • The schema or element set describes the items in a way that is understandable to its designated communities

  • Infrastructure:

    • has a stable source of access (e.g., a service provider guarantee)

    • can be maintained with available local resources (monetary, technical, personnel, etc.)

    • is extensible to add features when needed

  • Technical components for sharing metadata, such as URIs and OAI endpoints, are stable and and reliable

  • Metadata is clearly licensed – with machine- and human-readable licenses – regarding harvest, republishing, and other applicable uses

  • Metadata records are shareable, preferably as a bulk download

  • Each metadata record has a unique, persistent identifier

  • User displays have human-readable labels for fields and for values represented with machine-readable text (e.g., URIs)

  • Metadata is indexed and searchable

  • Values use terminology and a reading level that is understandable and appropriate for the audience or user community

  • Shortened data values – e.g., abbreviations, acronyms, initialisms, etc. – include definitions or fuller values for clarification

Accuracy

TECHNICAL SPECIFICATIONS

METADATA VALUES

  • The schema or element set truthfully describes the items

    • Both properties and values are defined

  • Any metadata creation, including extraction, migration, transformation, etc. results in appropriate records according to the local schema

  • Measures are in place for validation of:

    • Record structure

    • Values (when applicable)

    • Schema properties

  • All relevant fields/information are free of formatting, spelling, and other kinds of typographical errors that impact findability

  • “Noise” in elements is absent (e.g., tag fragments)

  • Any default values are correct and appropriately used

Completeness

TECHNICAL SPECIFICATIONS

METADATA VALUES

  • The schema or element set is able to fully describe the items

  • Controlled vocabularies or authorities contain all values needed for records

  • All relevant fields/information are included in all item records

    • Required/mandatory fields have values (locally-required and/or required by consortia)

    • Relevant optional fields are included

    • Overall field values-per-record are appropriate and provide sufficient information

    • Entries-per-field meet requirements or expectations of the local schema

      (e.g., a schema could require or prefer multiple subject-type entries)

  • Field values contain all expected information and syntactic components (e.g., qualifiers)

  • Enough unique information is included in an item record to be useful

  • Values that support findability are included (e.g., field values connected to user interface functionality)

Conformance to expectations

TECHNICAL SPECIFICATIONS

METADATA VALUES

  • The data model describes what it claims to

  • Controlled vocabularies are aligned with the needs of the user community or audience

  • Data models or schemas that have been brought together from different sources have been reconciled

  • Data from different sources can be merged without duplication or variants within a controlled field

  • Compromises in schema usage are documented and align with the needs of the user community

  • Values in the record describe the associated item (e.g., vs. a different item or a collection of items when not applicable)

  • Data from varied sources has been normalized in a way that is appropriate to the audience or user group

Consistency

TECHNICAL SPECIFICATIONS

METADATA VALUES

  • Whenever possible, controlled vocabularies or system tools are used to enforce consistency within field values

  • The metadata schema is clearly defined and applied to every applicable record in the same way

  • The data model is

    • well-structured and logical

    • in line with similar models from the same domain

    • uses consistent relationship directions

  • Rules about “blank” (i.e., non-populated/null) or non-relevant elements are applied consistently (e.g., left blank, filled with prescribed values like N/A, etc.)

  • Values for the same referent (e.g., a particular city, person, etc.) are identical

  • Metadata values are formatted in alignment with local guidelines

  • Values are consistent with similar usage in the user community

Provenance

TECHNICAL SPECIFICATIONS

METADATA VALUES

  • Information is documented about how records can be shared via OAI

  • Data is protected from unauthorized changes

  • Security measures, such as digital signatures, can be used to verify data authenticity

Administrative/preservation metadata

  • The names of metadata creators and editors are documented, including anyone responsible for harvesting or normalizing records from another source

  • All changes to a record are documented

  • Methods of creating, extracting, or transforming metadata are documented

  • Attribution for the source of the metadata record is included

  • Appropriate vocabularies are used for documenting provenance information

Timeliness

TECHNICAL SPECIFICATIONS

METADATA VALUES

  • Controlled vocabularies are managed and updated as needed

  • “Age” of records and last edits/updates are recorded

  • Record values are updated regularly, e.g.:

    • As resources change

    • As new information is available

    • As local standards change

    • To meet changing needs within the user community or audience

    • As general practice or industry standards shift

  • Information supports current search and usage activities for the user community or audience