======================================= Questions to Answer Before Benchmarking ======================================= This section is intended to help organizations consider various aspects of metadata assessment and benchmarking to facilitate planning. The example answers and additional clarifications are meant to provide more guidance about the kinds of directions an organization might take. Not every question will be appropriate for every organization and each local group may answer the same question differently according to their needs or resources. Purpose ======= +-----------------------------------------------------------------------------------+-------------------------------------------------------------------------------+ |Questions |Example Answers/Additional Clarification | +===================================================================================+===============================================================================+ |Why are you doing assessment/benchmarking at your organization? |- To create definitions or aspirational goals for the future | +-----------------------------------------------------------------------------------+ | |What is your desired outcome of assessment/benchmarking? |- To improve metadata (generally, or in specific ways) | | | | | |- To get a general sense of current quality for future planning and resource | | | allocation | +-----------------------------------------------------------------------------------+-------------------------------------------------------------------------------+ |What are your local *priorities*? E.g.: | +-----------------------------------------------------------------------------------+-------------------------------------------------------------------------------+ | |- Prioritize completeness over timeliness or accuracy (i.e., make sure fields | | | have values first and then control or normalize them) | | - How will metadata assessment address critical issues? |- Focus first on issues easiest to identify or measure with available system | | - What metadata characteristics should be the focus in the short-term, | functionality or tools already in use | | medium-term, or long-term? |- Address local concerns related to reparative metadata | | - Are there expectations from administrators at your institution? |- To get metadata ready for a migration or harvesting project, or to align | | | with a consortial partner | | |- Add information needed for internal users assisting researchers | | |- Maximize findability for users based on information pulled into discovery | | | layers, or more likely to return hits in open web searches | | |- To improve records that will be publicized for an event or anniversary, used| | | in a digital exhibit, etc. | +-----------------------------------------------------------------------------------+-------------------------------------------------------------------------------+ Scoping ======= +-----------------------------------------------------------------------------------+-------------------------------------------------------------------------------+ |Questions |Example Answers/Additional Clarification | +===================================================================================+===============================================================================+ |How are you defining "quality" at your organization? |- There are various metadata frameworks and papers describing aspects of | | | quality that could be applicable to your institution, depending on local | | | context | | | | | | - For example, "completeness" could be the existence of only required | | | fields, or could include additional fields (and may vary by collection | | | or project) | | | | | |- Quality could be based on individual review of records (i.e., all values in | | | each record) or on specific metrics that can be determined at an aggregate | | | level | +-----------------------------------------------------------------------------------+-------------------------------------------------------------------------------+ |Is this project abut assessing an entire body of metadata, or a specific |- For a specific resource type, there may be general expectations that | |sub-set/collection? | wouldn’t apply elsewhere, e.g., all theses and dissertations should have a | | | known creator (probably only 1), but for cultural heritage items (e.g, | | | photographs) creators may not always be known | | | | | |- Some collections have a large amount of information that is known (or | | | findable through research) and others may not; at a collection level, you | | | might set some standards high if they meet reasonable expectations | | | | | |- Assessing an entire digital collection or system may mean finding the lowest| | | common denominator for quality (compared to specific groups of materials | | | that have more information or different needs) | +-----------------------------------------------------------------------------------+-------------------------------------------------------------------------------+ |What policies and procedures govern metadata creation and quality; how do these fit|- Do changes or remediation plans need to be approved by administrators, an IT| |into the project? | department, or others? | | |- If editing access is restricted (due to software limitations or local | | | procedures), it could affect the number of people who are able to work on a | | | project | +-----------------------------------------------------------------------------------+-------------------------------------------------------------------------------+ |Are the benchmarks intended for a specific project, or long-term planning? |- Long-term benchmarking could be used for planning purposes or iterative | | | enhancement/remediation projects | | |- A particular collection may be targeted for improvements, e.g., to support a| | | research project, before promoting an event or anniversary, use in a digital| | | exhibit, etc. | +-----------------------------------------------------------------------------------+-------------------------------------------------------------------------------+ |Will these benchmarks affect other collections, projects, or organizations? |- If these collections are aggregated or part of a consortial project, do | | | those needs or expectations need to be considered? | | |- Benchmarks can be tailored for a specific collection or project; is there an| | | expectation that they should work for all local projects? | +-----------------------------------------------------------------------------------+-------------------------------------------------------------------------------+ |Is there a specific focus for the benchmarking project? |- To bring all records in a set up to a “minimal” level of completeness | | |- To gather more information for local best practices (e.g., related to | | | reparative metadata) | | |- To make metadata more accessible or shareable | | |- To incorporate new authority control or vocabularies | +-----------------------------------------------------------------------------------+-------------------------------------------------------------------------------+ Measurability ============= +-----------------------------------------------------------------------------------+-------------------------------------------------------------------------------+ |Questions |Example Answers/Additional Clarification | +===================================================================================+===============================================================================+ |How do you expect the benchmarks to be locally operational? |- To plan a remediation project or request specific resources | | | | | |- To outline expectations to guide ongoing work, or work on a specific | | | collection | +-----------------------------------------------------------------------------------+-------------------------------------------------------------------------------+ |What level of granularity is useful for your collections? |- The Bruce-Hillmann framework has seven general concepts; other frameworks | | | have more specific criteria or more technical metrics | | | | | |- Some metrics are general -- e.g., field usage across a collection (are there| | | creator values, date values, etc.) -- vs. specific value formatting or | | | accuracy | +-----------------------------------------------------------------------------------+-------------------------------------------------------------------------------+ |How important is it to be able to gauge specific levels of quality -- at a system, |- To gauge completeness, getting a sense of fields-used-per-record could be | |collection, or item level? | easy at an item or collection level, but not at a system level (depending on| | | local capabilities) | | | | | |- Some quality aspects are relatively easy to identify at an item level but | | | relatively difficult to assess at scale (e.g., the information matches the | | | item; the term “trail” is appropriate to the item and not a misspelling of | | | “trial”) | | | | | |- High-level goals could range from bringing “all” items up to a minimal or | | | better-than-minimal level to targeting specific records and making them | | | “ideal” | +-----------------------------------------------------------------------------------+-------------------------------------------------------------------------------+ |Based on locally-available tools, what metrics can be used to show when record/s |- Counting-type tools or validation that can show whether fields contain | |have moved from one quality level to the next? | values (e.g., if required/expected/preferred fields are missing values) | | | | | |- OpenRefine-type tools that can cluster similar values may highlight areas | | | that need authority control (especially for fields using vocabularies, or | | | with a limited set of expected values) | +-----------------------------------------------------------------------------------+-------------------------------------------------------------------------------+ Local Resources =============== **Links to Related Metadata Working Group Resources** - Tools list (for potential assessment tools): http://dlfmetadataassessment.github.io/projects/tools/ - Skill-building resources (related to using command-line and similar concepts): http://dlfmetadataassessment.github.io/projects/skill-building-resources/ - Metadata documentation from other institutions: http://dlfmetadataassessment.github.io/projects/metadata-application-profiles/ +-----------------------------------------------------------------------------------+-------------------------------------------------------------------------------+ |Questions |Example Answers/Additional Clarification | +===================================================================================+===============================================================================+ |What tools do you have available to evaluate records and assess quality levels, e.g.: | +-----------------------------------------------------------------------------------+-------------------------------------------------------------------------------+ |- Is there functionality in the system that can be used for assessment? |- Some systems have validation for certain fields or similar functionality to | |- Are there other tools that can be used? | facilitate assessment | |- How much evaluation can be done automatically (or semi-automated)? |- Values connected to public interface features can also highlight issues with| | | normalization (e.g., if there are lists of terms for filtering search | | | results) | +-----------------------------------------------------------------------------------+-------------------------------------------------------------------------------+ |What documentation governs metadata at the organization, e.g.: | +-----------------------------------------------------------------------------------+-------------------------------------------------------------------------------+ |- Is there a local MAP, or do records need to align with multiple MAPs (such |- If a collection is under the auspices of multiple MAPs or schemas (e.g., | | as the DPLA, Europeana, or other consortial MAP)? | local and consortial), determine if there are any discrepancies and the | |- What other documentation is available describing syntax/semantics of | overall requirements to meet expectations for both | | metadata values? |- Many organizations use local documentation for quality control (i.e., values| |- Does any of the documentation need to be reviewed or updated? | have to match stated requirements); look at local guidelines to see how | |- Is the documentation clear enough to create benchmarks (i.e., does the | these may translate into benchmarks or how to update documentation to make | | documentation have enough information to determine whether usage/values meet | it clearer when a value does/does not meet standards | | local requirements and expectations)? | | +-----------------------------------------------------------------------------------+-------------------------------------------------------------------------------+ |What personnel are required, e.g.: | +-----------------------------------------------------------------------------------+-------------------------------------------------------------------------------+ |- Who are the people needed to create benchmarks? |- Benchmarking likely needs metadata librarians or specialists, possibly | |- How many people are available to do any remediation or enhancement work | subject experts or technical personnel | | identified by assessment? |- Some kinds of remediation work could be done by student workers or | |- What technical personnel (or expertise) are needed to use assessment tools | non-metadata experts (e.g., change all date formats from ## mo YYYY to | | or to edit records? | YYYY-MM-DD; add spaces between initials in names; etc.) | | |- If additional tools or functionality are needed -- such as a batch edit | | | feature -- are there people who would need to be consulted or brought in to | | | work on the project? | +-----------------------------------------------------------------------------------+-------------------------------------------------------------------------------+ |Are there time limits or constraints that affect resources? |- Funding periods for projects (e.g., work that has to be done within a fiscal| | | year or grant timeframe) or personnel (e.g., a position only funded for a | | | specific amount of time) | | |- Project expectations (e.g., completion of work before a pre-arranged | | | migration or harvest) | | |- A contract for specific software or tools | +-----------------------------------------------------------------------------------+-------------------------------------------------------------------------------+