Why was this change made?

Adds missing lang keys

How was this change tested?

local transform

Which documentation and/or configurations were updated?

n/a

Why was this change made?

Fixes #759

QNL is the only collection where we were inserting bad data into the agg_preview hash conditionally if a iiif manifest is not available. By inserting non-urls into agg_preview.wr_id blacklight broke attempting to render bad URLs, this resolves that.

How was this change tested?

Ran in dev, manual analysis, and existing integration tests.

Which documentation and/or configurations were updated?

Why was this change made?

Fixes some errors in the config.

How was this change tested?

Local transform.

Which documentation and/or configurations were updated?

n/a

Fixes #1

This:

• [x] Creates the initial docker-compose.yml and Dockerfile
• [x] Adds initial (example) DAGs to establish structure
• [x] Add initial README with startup instructions

Add a configurable notifier that will be included in each DAG to email success and failure notifications.

This is an uber-proxy ticket as resources are determined for adding the airflow infrastructure to AWS.

Add a configurable XML harvester that will be called by individual DAGs

Add a configurable OAI harvester that will be called by individual DAGs

Add a configurable JSON harvester that will be called by individual DAGs

Add a configurable CSV based harvester that will be called by individual DAGs.

... and document in the readme getting up and running with local development.

@jacobthill I ran a test on the QNL data where we do not add the agg_preview if there isn't a manifest, this has resolved the breaking issue after re-transforming/indexing.

However - as stated elsewhere I think this is only a half measure. For discussion, what do we want to include in agg_preview (if anything) when there is no thumbnail URL or it is invalid?

My suggestion, since the agg_preview.wr_id field is the thumbnail field used in blacklight here (https://github.com/sul-dlss/dlme/blob/main/app/controllers/catalog_controller.rb#L63) and a default thumbnail is available, we add a validation at the above point (and any other collections that may need it) that then does not include the agg_preview.wr_id if it is not a valid URL.

I'm going to run a test or two in dev.

@jacobthill at least part of the issue is a bug in the QNL transform:

https://github.com/sul-dlss/dlme-transform/blob/main/traject_configs/qnl_config.rb#L152

here, if there isn't a iiif_manifest we insert the ID into agg_preview.wr_id and not a formatted URL, this is why you see the id based pattern above and not a URL.

Should we update the transform to inject a standard/generic thumbnail URL? Or not include this particular field? I'm not sure what the implications of that would be in the UI however.

As the DLME data manager I need to normalize cho_has_type values during transform. These values are mapped to a controlled vocabulary from terms in the contributor metadata using translation maps. Occasionally, the contributor data will contain multiple terms found in the translation map–one precise and one vague or too broad (e.g. the terms weapons & silver might both be used to describe an axe containing silver; weapons is mapped to the specific category Armaments and silver is mapped to a generic category Other Objects). When multiple values are found in the cho_has_type field and one of them is specific, the generic values should be removed.

generic_values = ['Other Image', 'Other Object', 'Other Text', 'Other Video', Other etc.]

• [x] during transform, check length of cho_has_type array; if two or more objects, check each for string pattern 'Other *' . If all match string pattern, pass. If one does not match string pattern (the specific value), delete all values that do match string pattern, leaving only the specific value/s.
• [x] should be called from within the normalize_has_type macro so that all configs don't require updating or, if this isn't possible, will need to update all configs.

Why was this change made?

closes #701

also, some opportunistic touchups in the second commit (discovered the unused let vars while looking for testing patterns i could copy).

How was this change tested?

unit tests

Which documentation and/or configurations were updated?

n/a