Questions to Answer Before Benchmarking

This section is intended to help organizations consider various aspects of metadata assessment and benchmarking to facilitate planning. The example answers and additional clarifications are meant to provide more guidance about the kinds of directions an organization might take. Not every question will be appropriate for every organization and each local group may answer the same question differently according to their needs or resources.

Purpose

Questions	Example Answers/Additional Clarification
Why are you doing assessment/benchmarking at your organization?	To create definitions or aspirational goals for the future To improve metadata (generally, or in specific ways) To get a general sense of current quality for future planning and resource allocation
What is your desired outcome of assessment/benchmarking?
What are your local priorities? E.g.:
How will metadata assessment address critical issues? What metadata characteristics should be the focus in the short-term, medium-term, or long-term? Are there expectations from administrators at your institution?	Prioritize completeness over timeliness or accuracy (i.e., make sure fields have values first and then control or normalize them) Focus first on issues easiest to identify or measure with available system functionality or tools already in use Address local concerns related to reparative metadata To get metadata ready for a migration or harvesting project, or to align with a consortial partner Add information needed for internal users assisting researchers Maximize findability for users based on information pulled into discovery layers, or more likely to return hits in open web searches To improve records that will be publicized for an event or anniversary, used in a digital exhibit, etc.

Scoping

Questions	Example Answers/Additional Clarification
How are you defining “quality” at your organization?	There are various metadata frameworks and papers describing aspects of quality that could be applicable to your institution, depending on local context For example, “completeness” could be the existence of only required fields, or could include additional fields (and may vary by collection or project) Quality could be based on individual review of records (i.e., all values in each record) or on specific metrics that can be determined at an aggregate level
Is this project abut assessing an entire body of metadata, or a specific sub-set/collection?	For a specific resource type, there may be general expectations that wouldn’t apply elsewhere, e.g., all theses and dissertations should have a known creator (probably only 1), but for cultural heritage items (e.g, photographs) creators may not always be known Some collections have a large amount of information that is known (or findable through research) and others may not; at a collection level, you might set some standards high if they meet reasonable expectations Assessing an entire digital collection or system may mean finding the lowest common denominator for quality (compared to specific groups of materials that have more information or different needs)
What policies and procedures govern metadata creation and quality; how do these fit into the project?	Do changes or remediation plans need to be approved by administrators, an IT department, or others? If editing access is restricted (due to software limitations or local procedures), it could affect the number of people who are able to work on a project
Are the benchmarks intended for a specific project, or long-term planning?	Long-term benchmarking could be used for planning purposes or iterative enhancement/remediation projects A particular collection may be targeted for improvements, e.g., to support a research project, before promoting an event or anniversary, use in a digital exhibit, etc.
Will these benchmarks affect other collections, projects, or organizations?	If these collections are aggregated or part of a consortial project, do those needs or expectations need to be considered? Benchmarks can be tailored for a specific collection or project; is there an expectation that they should work for all local projects?
Is there a specific focus for the benchmarking project?	To bring all records in a set up to a “minimal” level of completeness To gather more information for local best practices (e.g., related to reparative metadata) To make metadata more accessible or shareable To incorporate new authority control or vocabularies

Measurability

Questions	Example Answers/Additional Clarification
How do you expect the benchmarks to be locally operational?	To plan a remediation project or request specific resources To outline expectations to guide ongoing work, or work on a specific collection
What level of granularity is useful for your collections?	The Bruce-Hillmann framework has seven general concepts; other frameworks have more specific criteria or more technical metrics Some metrics are general – e.g., field usage across a collection (are there creator values, date values, etc.) – vs. specific value formatting or accuracy
How important is it to be able to gauge specific levels of quality – at a system, collection, or item level?	To gauge completeness, getting a sense of fields-used-per-record could be easy at an item or collection level, but not at a system level (depending on local capabilities) Some quality aspects are relatively easy to identify at an item level but relatively difficult to assess at scale (e.g., the information matches the item; the term “trail” is appropriate to the item and not a misspelling of “trial”) High-level goals could range from bringing “all” items up to a minimal or better-than-minimal level to targeting specific records and making them “ideal”
Based on locally-available tools, what metrics can be used to show when record/s have moved from one quality level to the next?	Counting-type tools or validation that can show whether fields contain values (e.g., if required/expected/preferred fields are missing values) OpenRefine-type tools that can cluster similar values may highlight areas that need authority control (especially for fields using vocabularies, or with a limited set of expected values)

Local Resources

Links to Related Metadata Working Group Resources

• Tools list (for potential assessment tools): http://dlfmetadataassessment.github.io/projects/tools/

• Skill-building resources (related to using command-line and similar concepts): http://dlfmetadataassessment.github.io/projects/skill-building-resources/

• Metadata documentation from other institutions: http://dlfmetadataassessment.github.io/projects/metadata-application-profiles/

Questions	Example Answers/Additional Clarification
What tools do you have available to evaluate records and assess quality levels, e.g.:
Is there functionality in the system that can be used for assessment? Are there other tools that can be used? How much evaluation can be done automatically (or semi-automated)?	Some systems have validation for certain fields or similar functionality to facilitate assessment Values connected to public interface features can also highlight issues with normalization (e.g., if there are lists of terms for filtering search results)
What documentation governs metadata at the organization, e.g.:
Is there a local MAP, or do records need to align with multiple MAPs (such as the DPLA, Europeana, or other consortial MAP)? What other documentation is available describing syntax/semantics of metadata values? Does any of the documentation need to be reviewed or updated? Is the documentation clear enough to create benchmarks (i.e., does the documentation have enough information to determine whether usage/values meet local requirements and expectations)?	If a collection is under the auspices of multiple MAPs or schemas (e.g., local and consortial), determine if there are any discrepancies and the overall requirements to meet expectations for both Many organizations use local documentation for quality control (i.e., values have to match stated requirements); look at local guidelines to see how these may translate into benchmarks or how to update documentation to make it clearer when a value does/does not meet standards
What personnel are required, e.g.:
Who are the people needed to create benchmarks? How many people are available to do any remediation or enhancement work identified by assessment? What technical personnel (or expertise) are needed to use assessment tools or to edit records?	Benchmarking likely needs metadata librarians or specialists, possibly subject experts or technical personnel Some kinds of remediation work could be done by student workers or non-metadata experts (e.g., change all date formats from ## mo YYYY to YYYY-MM-DD; add spaces between initials in names; etc.) If additional tools or functionality are needed – such as a batch edit feature – are there people who would need to be consulted or brought in to work on the project?
Are there time limits or constraints that affect resources?	Funding periods for projects (e.g., work that has to be done within a fiscal year or grant timeframe) or personnel (e.g., a position only funded for a specific amount of time) Project expectations (e.g., completion of work before a pre-arranged migration or harvest) A contract for specific software or tools