profile
viewpoint
If you are wondering where the data of this site comes from, please visit https://api.github.com/users/jmartin-sul/events. GitMemory does not store any data, but only uses NGINX to cache data for a period of time. The idea behind GitMemory is simply to give users a better reading experience.
Johnathan Martin jmartin-sul Stanford University Libraries

camillevilla/release-drafter-test 0

For testing out https://probot.github.io/apps/release-drafter/

LD4P/sinopia_exporter 0

A simple CLI tool for exporting RDF from a Sinopia group

push eventsul-dlss/argo

Peter Mangiafico

commit sha f4955a6a614c505b85e78a90292eb9a78ade7b88

allow large solr queries to work by using post instead of get

view details

Johnathan Martin

commit sha 75b0172b9b28452e24528747270bdcf7a6d7636c

Merge pull request #2793 from sul-dlss/use-post-instead-of-get-with-blacklight allow large solr queries to work by using post instead of get

view details

push time in 6 hours

delete branch sul-dlss/argo

delete branch : use-post-instead-of-get-with-blacklight

delete time in 6 hours

PR merged sul-dlss/argo

allow large solr queries to work by using post instead of get

Why was this change made?

Andrew was having problems with the Bulk Action page. Traced to a large solr query being sent over GET.

See discussion in slack: https://stanfordlib.slack.com/archives/C09M7P91R/p1628200332093000

How was this change tested?

Existing tests

Which documentation and/or configurations were updated?

+4 -0

0 comment

1 changed file

peetucket

pr closed time in 6 hours

PullRequestReviewEvent

push eventsul-dlss/dlme-transform

Jacob Hill

commit sha 183d039df1563fe9d2b48d1c5cab54d21c561cee

fix language extraction macro and test

view details

Jacob Hill

commit sha 94fd7122c93b4a5369b457b88d89253a696afdc4

same changes on arabic lang macro

view details

Jacob Hill

commit sha 1d486d08256101234bf810b39ecd5856f87e9412

draft test

view details

Aaron Collier

commit sha 8e5be22f26746ed2bdecfa7e40df4484b81e3a19

Add rubocop execption for spec and clean up styling

view details

Jacob Hill

commit sha 389597c3d65439e1716840f92bcfb90cca33c9fe

add latest

view details

Jacob Hill

commit sha 0e53198b77ca11f173e65dcd29d104f276c2e18f

fix test

view details

Johnathan Martin

commit sha 5a2a1639392986cd52be3370564fed24983ed538

EachRecord.convert_to_language_hash: log an error when there's no language key small refactor to DRY up error message copypasta using a string template

view details

Johnathan Martin

commit sha d3f5c331cb23b2529f1f84a6a692060091982818

EachRecord.convert_to_language_hash: raise an error when there's no language key also a rubocop config tweak, and re-gen .rubocop_todo.yml

view details

push time in 9 hours

Pull request review commentsul-dlss/dlme

Delete all resources

+# frozen_string_literal: true++# service class for removing a resource or multiple resources+class ResourceRemover+  def self.remove_all_resources+    new.remove_all_resources+  end++  # @return [Integer] the number of records removed+  def remove_all_resources+    # we could run a count query, but if there are a ton of records, actual deleted count could+    # exceed initial count, because find_each runs in batches non-transactionally.+    removal_count = 0+    Spotlight::Resource.find_each do |resource|+      ResourceRemover.remove_resource(resource: resource)+      removal_count += 1+    end+    removal_count+  end++  def self.remove_resource(resource:)+    new.remove_resource(resource: resource)+  end++  # @param [Spotlight::Resource] resource to be un-indexed and destroyed+  def remove_resource(resource:)

the new remove_all_resources test implicitly exercises this too

jmartin-sul

comment created time in 9 hours

PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commentsul-dlss/dlme

Delete all resources

+# frozen_string_literal: true++# service class for removing a resource or multiple resources+class ResourceRemover+  def self.remove_all_resources+    new.remove_all_resources+  end++  # @return [Integer] the number of records removed+  def remove_all_resources+    # we could run a count query, but if there are a ton of records, actual deleted count could+    # exceed initial count, because find_each runs in batches non-transactionally.+    removal_count = 0+    Spotlight::Resource.find_each do |resource|+      ResourceRemover.remove_resource(resource: resource)+      removal_count += 1+    end+    removal_count+  end++  def self.remove_resource(resource:)+    new.remove_resource(resource: resource)+  end++  # @param [Spotlight::Resource] resource to be un-indexed and destroyed+  def remove_resource(resource:)

i didn't add tests for this method to the new service class' specific tests, since the existing DeleteResourcesJob tests cover it, but can revisit that if others think that'd be a good idea.

jmartin-sul

comment created time in 9 hours

PullRequestReviewEvent

Pull request review commentsul-dlss/dlme

Delete all resources

 def perform(url, exhibit)      resp = Faraday.get(url) -    resources = NdjsonNormalizer.normalize(resp.body, url)-    resources.each do |item|-      remove_resource(item: item, exhibit: exhibit)-    end--    logger.info("#{resources.count} records were removed from #{url}.")-  end--  private--  def remove_resource(item:, exhibit:)-    resource = DlmeJson.find_by(url: item['id'], exhibit: exhibit)-    return unless resource+    resource_hashes = NdjsonNormalizer.normalize(resp.body, url)+    resource_hashes.each do |resource_hash|+      resource = DlmeJson.find_by(url: resource_hash['id'], exhibit: exhibit)+      next unless resource -    # This approach will work for DLME where we only have one exhibit, but-    # will need to be made more complicated for Spotlight in general because-    # one resource could be part of more than one exhibit.-    blacklight_solr.delete_by_query(-      "{!terms f=#{Spotlight::Engine.config.resource_global_id_field}}#{resource.to_global_id}"-    )-    blacklight_solr.commit-    resource.destroy-  end

refactored the rest of the private method functionality into a service class, to re-use in the new rake task

jmartin-sul

comment created time in 9 hours

PullRequestReviewEvent

Pull request review commentsul-dlss/dlme

Delete all resources

 def perform(url, exhibit)      resp = Faraday.get(url) -    resources = NdjsonNormalizer.normalize(resp.body, url)-    resources.each do |item|-      remove_resource(item: item, exhibit: exhibit)-    end--    logger.info("#{resources.count} records were removed from #{url}.")-  end--  private--  def remove_resource(item:, exhibit:)-    resource = DlmeJson.find_by(url: item['id'], exhibit: exhibit)-    return unless resource

pulled this into the loop in #perform

jmartin-sul

comment created time in 9 hours

Pull request review commentsul-dlss/dlme

Delete all resources

 # frozen_string_literal: true -# Load JSON resources from a url, persist them, and index them+# Load JSON resources from a url, then remove them from the index and delete them

this was just a fix to what i think was some old copypasta

jmartin-sul

comment created time in 9 hours

PullRequestReviewEvent

PR opened sul-dlss/dlme

Delete all resources

Why was this change made?

closes sul-dlss/dlme-transform#702

Was the documentation (README, API, wiki, ...) updated?

a description was added for the new rake task that makes this new functionality available. can also add to the README, if desired. i didn't see anything similar in the README, but this also seems like the app's first custom rake task?

+105 -31

0 comment

5 changed files

pr created time in 9 hours

create barnchsul-dlss/dlme

branch : delete-all-resources

created branch time in 10 hours

PullRequestReviewEvent

push eventsul-dlss/preservation_catalog

Johnathan Martin

commit sha 0acae4a8e8d4ed734ceb8de548e138e6fd97f9fc

ObjectsController#checksums: simplify logic, respond with 500 instead of 409 for unexpected errors

view details

Johnathan Martin

commit sha c46e1bebc9d0eb1968310e03d72d5142ef9c6b64

add the new rubocop rules that the codebase already passes

view details

Johnathan Martin

commit sha b3dc69edd4bed07908e5616299330ee8704ca777

Merge pull request #1739 from sul-dlss/checksum-report-error-handling-touchup Checksum report error handling touchup

view details

push time in 2 days

delete branch sul-dlss/preservation_catalog

delete branch : checksum-report-error-handling-touchup

delete time in 2 days

PR merged sul-dlss/preservation_catalog

Checksum report error handling touchup

Why was this change made?

small follow on to #1738 implementing the changes i'd requested, plus bonus rubocop touchup (added the new rules that didn't necessitate any other changes)

How was this change tested?

CI

Which documentation and/or configurations were updated?

openapi definition

+24 -10

0 comment

4 changed files

jmartin-sul

pr closed time in 2 days

Pull request review commentsul-dlss/preservation_catalog

Checksum report error handling touchup

 paths:           description: Bad request (e.g., malformed druid(s))         '406':           description: Unsupported response format-        '409':-          description: Conflict - one or more missing or error-state objects+        '500':+          description: Unexpected error generating checksum report for one or more druids

this might even be the first time i remembered to do it before the test suite reminded me that i had to 😆

jmartin-sul

comment created time in 2 days

PullRequestReviewEvent

pull request commentsul-dlss/preservation_catalog

allow checksum job to return for multiple checksums when some not found

Thanks @jmartin-sul - those all sound great. Happy to review the follow up PR when ready.

follow up PR: https://github.com/sul-dlss/preservation_catalog/pull/1739

peetucket

comment created time in 2 days

PR opened sul-dlss/preservation_catalog

Checksum report error handling touchup

Why was this change made?

small follow on to #1738 implementing the changes i'd requested, plus bonus rubocop touchup (added the new rules that didn't necessitate any other changes)

How was this change tested?

CI

Which documentation and/or configurations were updated?

openapi definition

+24 -10

0 comment

4 changed files

pr created time in 2 days

MemberEvent

Pull request review commentsul-dlss/preservation_catalog

allow checksum job to return for multiple checksums when some not found

 def checksum   # note: this is deliberately allowed to be a POST to allow for a large number of druids to be passed in   # GET OR POST /v1/objects/checksums?druids[]=druid1&druids[]=druid2&druids[]=druid3   def checksums-    checksum_list, missing_druids, errored_druids = generate_checksum_list--    bad_recs_msg = "\nStorage object(s) not found for #{missing_druids.join(', ')}" if missing_druids.any?-    bad_recs_msg = (bad_recs_msg || '') + "\nProblems generating checksums for #{errored_druids.join(', ')}" if errored_druids.any?+    checksum_list, missing_druids_list, errored_druids = generate_checksum_list+    bad_recs_msg = "\nProblems generating checksums for #{errored_druids.join(', ')}" if errored_druids.any?     if bad_recs_msg.present?       render(plain: "409 Conflict - #{bad_recs_msg}", status: :conflict)

i think this might more appropriately be a 500 now that we really don't expect this to happen (since missing druid was our only known error condition, and iirc the reason we chose 409, despite it not being a perfect fit semantically)

peetucket

comment created time in 2 days

Pull request review commentsul-dlss/preservation_catalog

allow checksum job to return for multiple checksums when some not found

 def checksum   # note: this is deliberately allowed to be a POST to allow for a large number of druids to be passed in   # GET OR POST /v1/objects/checksums?druids[]=druid1&druids[]=druid2&druids[]=druid3   def checksums-    checksum_list, missing_druids, errored_druids = generate_checksum_list--    bad_recs_msg = "\nStorage object(s) not found for #{missing_druids.join(', ')}" if missing_druids.any?-    bad_recs_msg = (bad_recs_msg || '') + "\nProblems generating checksums for #{errored_druids.join(', ')}" if errored_druids.any?+    checksum_list, missing_druids_list, errored_druids = generate_checksum_list+    bad_recs_msg = "\nProblems generating checksums for #{errored_druids.join(', ')}" if errored_druids.any?     if bad_recs_msg.present?

i think this could be simplified to just check errored_druids.any? now that bad_recs_msg isn't built with two different conditions.

peetucket

comment created time in 2 days

Pull request review commentsul-dlss/preservation_catalog

allow checksum job to return for multiple checksums when some not found

 def checksum   # note: this is deliberately allowed to be a POST to allow for a large number of druids to be passed in   # GET OR POST /v1/objects/checksums?druids[]=druid1&druids[]=druid2&druids[]=druid3   def checksums-    checksum_list, missing_druids, errored_druids = generate_checksum_list--    bad_recs_msg = "\nStorage object(s) not found for #{missing_druids.join(', ')}" if missing_druids.any?-    bad_recs_msg = (bad_recs_msg || '') + "\nProblems generating checksums for #{errored_druids.join(', ')}" if errored_druids.any?+    checksum_list, missing_druids_list, errored_druids = generate_checksum_list+    bad_recs_msg = "\nProblems generating checksums for #{errored_druids.join(', ')}" if errored_druids.any?

i think the leading newline can go

peetucket

comment created time in 2 days