profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/aaron-collier/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.
Aaron Collier aaron-collier Stanford University Palo Alto, CA Conference Chaos Monkey

aaron-collier/sword 2

A SWORD server for Samvera (specifically Hyrax)

aaron-collier/2017.code4lib.org 0

Code4lib 2017 Los Angeles

aaron-collier/2018.code4lib.org 0

site for the 2018 code4lib conference in Washington, DC

aaron-collier/c4l18-keynote-statement 0

Code4Lib Community Statement in Support of Chris Bourg

push eventsul-dlss/dlme-transform

Jacob Hill

commit sha cf718c93a4444c9e5cc6599686b2c2abc76853fa

add missing lang keys

view details

Aaron Collier

commit sha df80d83009801e1c637097f1c59cb6a9390f8980

Merge pull request #766 from sul-dlss/lang add missing lang keys

view details

push time in 18 hours

delete branch sul-dlss/dlme-transform

delete branch : lang

delete time in 18 hours

PR merged sul-dlss/dlme-transform

add missing lang keys

Why was this change made?

Adds missing lang keys

How was this change tested?

local transform

Which documentation and/or configurations were updated?

n/a

+11 -11

0 comment

3 changed files

jacobthill

pr closed time in 18 hours

PullRequestReviewEvent

push eventsul-dlss/dlme-transform

Aaron Collier

commit sha fa8346858a08117069e1da6dd5a0e823ca5cadf8

Do not insert bad data when a iiif manifest is not available for QNL

view details

push time in a day

PR opened sul-dlss/dlme-transform

Do not insert bad data when a iiif manifest is not available for QNL

Why was this change made?

Fixes #759

QNL is the only collection where we were inserting bad data into the agg_preview hash conditionally if a iiif manifest is not available. By inserting non-urls into agg_preview.wr_id blacklight broke attempting to render bad URLs, this resolves that.

How was this change tested?

Ran in dev, manual analysis, and existing integration tests.

Which documentation and/or configurations were updated?

+6 -6

0 comment

1 changed file

pr created time in a day

create barnchsul-dlss/dlme-transform

branch : 759-validate-urls

created branch time in a day

PR merged sul-dlss/dlme-transform

update shahre farang mapping

Why was this change made?

Fixes some errors in the config.

How was this change tested?

Local transform.

Which documentation and/or configurations were updated?

n/a

+12 -9

0 comment

3 changed files

jacobthill

pr closed time in a day

push eventsul-dlss/dlme-transform

Jacob Hill

commit sha 888a0bc8e6de25844245af1cb519421e90d2f5a5

update shahre farang mapping

view details

Aaron Collier

commit sha 44c10363e099d475da77c1c3f7a7d74da1f34ddc

Merge pull request #764 from sul-dlss/shahre update shahre farang mapping

view details

push time in a day

delete branch sul-dlss/dlme-transform

delete branch : shahre

delete time in a day

PullRequestReviewEvent

PR opened sul-dlss/dlme-airflow

Initial docker infrastructure

Fixes #1

This:

  • [x] Creates the initial docker-compose.yml and Dockerfile
  • [x] Adds initial (example) DAGs to establish structure
  • [x] Add initial README with startup instructions
+346 -0

0 comment

10 changed files

pr created time in 2 days

push eventsul-dlss/dlme-airflow

Aaron Collier

commit sha dcf1195d851e11aba4ecca3f8b280e4a2507aabd

Start README

view details

push time in 2 days

create barnchsul-dlss/dlme-airflow

branch : initial-docker-infrastructure

created branch time in 2 days

issue openedsul-dlss/dlme-airflow

Add notifier for success and failure notifications

Add a configurable notifier that will be included in each DAG to email success and failure notifications.

created time in 2 days

issue openedsul-dlss/dlme-airflow

Add terraform resources for deployment

This is an uber-proxy ticket as resources are determined for adding the airflow infrastructure to AWS.

created time in 2 days

issue openedsul-dlss/dlme-airflow

Add XML harvester

Add a configurable XML harvester that will be called by individual DAGs

created time in 2 days

issue openedsul-dlss/dlme-airflow

Add an OAI harvester

Add a configurable OAI harvester that will be called by individual DAGs

created time in 2 days

issue openedsul-dlss/dlme-airflow

Add JSON harvester

Add a configurable JSON harvester that will be called by individual DAGs

created time in 2 days

issue openedsul-dlss/dlme-airflow

Add csv harvester

Add a configurable CSV based harvester that will be called by individual DAGs.

created time in 2 days

issue openedsul-dlss/dlme-airflow

Create initial Dockerfile and docker-compose.yml

... and document in the readme getting up and running with local development.

created time in 2 days

create barnchsul-dlss/dlme-airflow

branch : main

created branch time in 5 days

created repositorysul-dlss/dlme-airflow

This is a new repository to capture the work related to the DLME ETL Pipeline and establish airflow

created time in 5 days

issue commentsul-dlss/dlme-transform

Bad urls should not break application

@jacobthill I ran a test on the QNL data where we do not add the agg_preview if there isn't a manifest, this has resolved the breaking issue after re-transforming/indexing.

However - as stated elsewhere I think this is only a half measure. For discussion, what do we want to include in agg_preview (if anything) when there is no thumbnail URL or it is invalid?

jacobthill

comment created time in 5 days

issue commentsul-dlss/dlme-transform

Bad thumbnails in QNL

My suggestion, since the agg_preview.wr_id field is the thumbnail field used in blacklight here (https://github.com/sul-dlss/dlme/blob/main/app/controllers/catalog_controller.rb#L63) and a default thumbnail is available, we add a validation at the above point (and any other collections that may need it) that then does not include the agg_preview.wr_id if it is not a valid URL.

I'm going to run a test or two in dev.

jacobthill

comment created time in 7 days

issue commentsul-dlss/dlme-transform

Bad thumbnails in QNL

@jacobthill at least part of the issue is a bug in the QNL transform:

https://github.com/sul-dlss/dlme-transform/blob/main/traject_configs/qnl_config.rb#L152

here, if there isn't a iiif_manifest we insert the ID into agg_preview.wr_id and not a formatted URL, this is why you see the id based pattern above and not a URL.

Should we update the transform to inject a standard/generic thumbnail URL? Or not include this particular field? I'm not sure what the implications of that would be in the UI however.

jacobthill

comment created time in 7 days

push eventsul-dlss/dlme-transform

Johnathan Martin

commit sha acc6e606c18efe9f27ffd3c534f01525113e3688

Remove generic cho_has_type value when specific value present (#701)

view details

Johnathan Martin

commit sha 1e6ebee5cf03d316f4e962b9e145c4070f09fce1

touchups: make normalize_has_type translation_map list more readable, remove unused let vars

view details

Aaron Collier

commit sha eee2bc3d21a5f7acef1ad5dc61b4bd0a68fcaff7

Merge pull request #760 from sul-dlss/remove-generic-cho_has_type-vals-if-possible Remove generic cho_has_type value when specific value present

view details

push time in 7 days

delete branch sul-dlss/dlme-transform

delete branch : remove-generic-cho_has_type-vals-if-possible

delete time in 7 days

issue closedsul-dlss/dlme-transform

Remove generic cho_has_type value when specific value present

As the DLME data manager I need to normalize cho_has_type values during transform. These values are mapped to a controlled vocabulary from terms in the contributor metadata using translation maps. Occasionally, the contributor data will contain multiple terms found in the translation map–one precise and one vague or too broad (e.g. the terms weapons & silver might both be used to describe an axe containing silver; weapons is mapped to the specific category Armaments and silver is mapped to a generic category Other Objects). When multiple values are found in the cho_has_type field and one of them is specific, the generic values should be removed.

generic_values = ['Other Image', 'Other Object', 'Other Text', 'Other Video', Other etc.]

  • [x] during transform, check length of cho_has_type array; if two or more objects, check each for string pattern 'Other *' . If all match string pattern, pass. If one does not match string pattern (the specific value), delete all values that do match string pattern, leaving only the specific value/s.
  • [x] should be called from within the normalize_has_type macro so that all configs don't require updating or, if this isn't possible, will need to update all configs.

closed time in 7 days

jacobthill

PR merged sul-dlss/dlme-transform

Remove generic cho_has_type value when specific value present

Why was this change made?

closes #701

also, some opportunistic touchups in the second commit (discovered the unused let vars while looking for testing patterns i could copy).

How was this change tested?

unit tests

Which documentation and/or configurations were updated?

n/a

+51 -26

0 comment

6 changed files

jmartin-sul

pr closed time in 7 days