profile
viewpoint

faye/faye 4307

Simple pub/sub messaging for the web

faye/faye-websocket-ruby 917

Standards-compliant WebSocket client and server

faye/faye-websocket-node 526

Standards-compliant WebSocket client and server

faye/websocket-driver-ruby 185

WebSocket protocol handler with pluggable I/O

faye/websocket-driver-node 97

WebSocket protocol handler with pluggable I/O

faye/faye-redis-ruby 81

Redis engine backend for Faye

faye/faye-redis-node 63

Redis engine backend for Faye

faye/websocket-extensions-node 48

Generic extension management for WebSocket connections

faye/websocket-extensions-ruby 22

Generic extension management for WebSocket connections

faye/permessage-deflate-node 19

Per-message DEFLATE compression extension for WebSocket connections

pull request commentUKGovernmentBEIS/beis-report-official-development-assistance

(1066) Users can see a list of invalid activities in the index page

@mec My commit message explains why we were observing weird effects from Bullet in terms of what it's doing behind the scenes. I discussed this with @MariluzHerrera and my decision went as follows.

Using Bullet.add_whitelist in both the environment config and in individual tests is not advisable since it can lead to conflicts, where a test turns a particular whitelisted thing on and off, changing the base state of the app's config, and those are hard to debug, as we've seen here. The Bullet.{add,delete}_whitelist API does not seem designed for this type of use as it does not provide a reliable way of restoring whatever the prior state was. We need to pick one or the other.

Putting it in individual tests means people would need to add Bullet.{add,delete}_whitelist config to any test that hits a problematic controller action. This particular action is hit by a lot of tests, and it won't be obvious that one needs to add this config in order to silence the warnings Bullet produces, or even if that is a good idea. Getting warnings about code you didn't write is confusing.

Putting it in the environment config means we're globally ignoring a certain kind of optimisation problem -- you cannot. In general, I would rather be alerted to these, but I think ignoring an unused eager load is much less dangerous than ignoring an N+1 query, so I'm happy with the trade-off. In this case, the unused eager load comes from the fact that not all Activity objects use the parent association during validation -- fund activities don't require a parent. The fact that some but not all loaded activities use the parent association causes Bullet to raise an error whether you eager load the association or not -- it's impossible to make this code pass Bullet inspection.

I'm open to making the opposite decision and putting Bullet whitelist logic in individual tests, and not in the environment config, I just think we need to pick one or the other.

MariluzHerrera

comment created time in 6 days

issue commentfaye/faye

Unexpected error while processing request: uninitialized constant EventMachine::Deferrable::Rails

None of the lines in this backtrace refer to the constant Rails, but a lot of them are concerned with running callbacks that will yield to user-supplied code. Your Faye server setup code does not appear to reference anything from Rails, or load application code, and does not set up any event listeners or extensions:

require "faye"
faye_server = Faye::RackAdapter.new(:mount => '/faye', :timeout => 45)
run faye_server

The one problem I can see with this is that it sets the timeout to 45, which is greater than the Thin request timeout limit of 30. This may cause weird results.

What command are you using to start the server in faye.ru?

imi56

comment created time in 6 days

issue commentfaye/faye

uninitialized constant EventMachine::Rails (NameError)

Neither EventMachine nor Faye refers to Rails anywhere in their codebases. Have you bound any event listeners to Faye that depend on anything from Rails or your application?

imi56

comment created time in 6 days

issue commentfaye/faye

Faye requesting an insecure script <URL>

What's the code you're using to start the connection to the Faye server?

imi56

comment created time in 7 days

push eventjcoglan/dotfiles

James Coglan

commit sha ba910a185eb39cf23b4559c3eb5365c030ba8cab

Function for installing ruby with openssl@1.0

view details

push time in 8 days

push eventjcoglan/dotfiles

James Coglan

commit sha b99b9659f79f12dd3c3ce451bd340dd914ea263d

Run all the shell/scripts files through shellcheck

view details

push time in 11 days

push eventjcoglan/dotfiles

James Coglan

commit sha c1d9b2bac90430b1ddd4e4420ce2ea0033fc56fe

Refactor Homebrew functions to expose packages, dependencies, and the ability to check installed packages against arbitrary Brewfiles

view details

push time in 11 days

push eventUKGovernmentBEIS/beis-report-official-development-assistance

dependabot-preview[bot]

commit sha c724bfc5608c757e7470ed181dd14632a52628f7

Bump data_migrate from 6.3.0 to 6.5.0 Bumps [data_migrate](https://github.com/ajvargo/data-migrate) from 6.3.0 to 6.5.0. - [Release notes](https://github.com/ajvargo/data-migrate/releases) - [Changelog](https://github.com/ilyakatz/data-migrate/blob/master/Changelog.md) - [Commits](https://github.com/ajvargo/data-migrate/commits) Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

view details

James Coglan

commit sha 0a4c2d3cec7e999dce9fbc94622c1f2bd2f4aa23

Merge pull request #687 from UKGovernmentBEIS/dependabot/bundler/data_migrate-6.5.0 Bump data_migrate from 6.3.0 to 6.5.0

view details

push time in 12 days

delete branch UKGovernmentBEIS/beis-report-official-development-assistance

delete branch : dependabot/bundler/data_migrate-6.5.0

delete time in 12 days

PR merged UKGovernmentBEIS/beis-report-official-development-assistance

Bump data_migrate from 6.3.0 to 6.5.0 dependencies

Bumps data_migrate from 6.3.0 to 6.5.0. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/ilyakatz/data-migrate/blob/master/Changelog.md">data_migrate's changelog</a>.</em></p> <blockquote> <h1>Changelog</h1> <h2>6.4.0</h2> <p>Add primary key to data_migrations table <a href="https://github.com/aandis">aandis</a></p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li>See full diff in <a href="https://github.com/ajvargo/data-migrate/commits">compare view</a></li> </ul> </details> <br />

Dependabot compatibility score

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


<details> <summary>Dependabot commands and options</summary> <br />

You can trigger Dependabot actions by commenting on this PR:

  • @dependabot rebase will rebase this PR
  • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
  • @dependabot merge will merge this PR after your CI passes on it
  • @dependabot squash and merge will squash and merge this PR after your CI passes on it
  • @dependabot cancel merge will cancel a previously requested merge and block automerging
  • @dependabot reopen will reopen this PR if it is closed
  • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
  • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
  • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
  • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language
  • @dependabot badge me will comment on this PR with code to add a "Dependabot enabled" badge to your readme

Additionally, you can set the following in your Dependabot dashboard:

  • Update frequency (including time of day and day of week)
  • Pull request limits (per update run and/or open at any time)
  • Automerge options (never/patch/minor, and dev/runtime dependencies)
  • Out-of-range updates (receive only lockfile updates, if desired)
  • Security updates (receive only security updates, if desired)

</details>

+3 -3

1 comment

1 changed file

dependabot-preview[bot]

pr closed time in 12 days

PullRequestReviewEvent

push eventUKGovernmentBEIS/beis-report-official-development-assistance

dependabot-preview[bot]

commit sha 5836c59cd63838239bc541e69ab3357336c9046b

Bump auth0 from 4.15.0 to 4.16.0 Bumps [auth0](https://github.com/auth0/ruby-auth0) from 4.15.0 to 4.16.0. - [Release notes](https://github.com/auth0/ruby-auth0/releases) - [Changelog](https://github.com/auth0/ruby-auth0/blob/master/CHANGELOG.md) - [Commits](https://github.com/auth0/ruby-auth0/compare/v4.15.0...v4.16.0) Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

view details

James Coglan

commit sha 1d812c2c27e93e8b784d58f846673005a4a75179

Merge pull request #686 from UKGovernmentBEIS/dependabot/bundler/auth0-4.16.0 Bump auth0 from 4.15.0 to 4.16.0

view details

push time in 12 days

delete branch UKGovernmentBEIS/beis-report-official-development-assistance

delete branch : dependabot/bundler/auth0-4.16.0

delete time in 12 days

PR merged UKGovernmentBEIS/beis-report-official-development-assistance

Bump auth0 from 4.15.0 to 4.16.0 dependencies

Bumps auth0 from 4.15.0 to 4.16.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/auth0/ruby-auth0/releases">auth0's releases</a>.</em></p> <blockquote> <h2>v4.16.0</h2> <h2><a href="https://github.com/auth0/ruby-auth0/tree/v4.16.0">v4.16.0</a> (2020-10-02)</h2> <p><a href="https://github.com/auth0/ruby-auth0/compare/v4.15.0...v4.16.0">Full Changelog</a></p> <p><strong>Added</strong></p> <ul> <li>New Email Verification Fields <a href="https://github-redirect.dependabot.com/auth0/ruby-auth0/pull/237">#237</a> (<a href="https://github.com/davidpatrick">davidpatrick</a>)</li> </ul> <p><strong>Security</strong></p> <ul> <li>Bump actionview from 6.0.3.2 to 6.0.3.3 <a href="https://github-redirect.dependabot.com/auth0/ruby-auth0/pull/236">#236</a> (<a href="https://github.com/apps/dependabot">dependabot[bot]</a>)</li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/auth0/ruby-auth0/blob/master/CHANGELOG.md">auth0's changelog</a>.</em></p> <blockquote> <h2><a href="https://github.com/auth0/ruby-auth0/tree/v4.16.0">v4.16.0</a> (2020-10-02)</h2> <p><a href="https://github.com/auth0/ruby-auth0/compare/v4.15.0...v4.16.0">Full Changelog</a></p> <p><strong>Added</strong></p> <ul> <li>New Email Verification Fields <a href="https://github-redirect.dependabot.com/auth0/ruby-auth0/pull/237">#237</a> (<a href="https://github.com/davidpatrick">davidpatrick</a>)</li> </ul> <p><strong>Security</strong></p> <ul> <li>Bump actionview from 6.0.3.2 to 6.0.3.3 <a href="https://github-redirect.dependabot.com/auth0/ruby-auth0/pull/236">#236</a> (<a href="https://github.com/apps/dependabot">dependabot[bot]</a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/auth0/ruby-auth0/commit/54c8d67b323397ff9176076e88b7b9ac814c0dee"><code>54c8d67</code></a> Bump to 4.16.0</li> <li><a href="https://github.com/auth0/ruby-auth0/commit/392acd280afd2e572cce65d23086ff3d9033144c"><code>392acd2</code></a> Bump actionview from 6.0.3.2 to 6.0.3.3 (<a href="https://github-redirect.dependabot.com/auth0/ruby-auth0/issues/236">#236</a>)</li> <li><a href="https://github.com/auth0/ruby-auth0/commit/dc1a6b52f6c679db7dfdad3e84d74f46dbe721d7"><code>dc1a6b5</code></a> New Email Verification Fields (<a href="https://github-redirect.dependabot.com/auth0/ruby-auth0/issues/237">#237</a>)</li> <li><a href="https://github.com/auth0/ruby-auth0/commit/a7ffeb50a095041bc3263aeb0de87cc02594d85a"><code>a7ffeb5</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/auth0/ruby-auth0/issues/235">#235</a> from fossabot/add-license-scan-badge</li> <li><a href="https://github.com/auth0/ruby-auth0/commit/79b83f33aaeef83622fd63eaa30952963bc8665d"><code>79b83f3</code></a> Add license scan report and status</li> <li><a href="https://github.com/auth0/ruby-auth0/commit/94efce954ee914938294f866791f96011cc2a96b"><code>94efce9</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/auth0/ruby-auth0/issues/234">#234</a> from auth0/release-4.15.0</li> <li><a href="https://github.com/auth0/ruby-auth0/commit/8ee30e04dfed003c790d034d0827d9a49afae30f"><code>8ee30e0</code></a> Update CHANGELOG.md</li> <li><a href="https://github.com/auth0/ruby-auth0/commit/9ed9dd3b6b386907ef131a77b4120d17933ca8bb"><code>9ed9dd3</code></a> Update gems</li> <li>See full diff in <a href="https://github.com/auth0/ruby-auth0/compare/v4.15.0...v4.16.0">compare view</a></li> </ul> </details> <br />

Dependabot compatibility score

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


<details> <summary>Dependabot commands and options</summary> <br />

You can trigger Dependabot actions by commenting on this PR:

  • @dependabot rebase will rebase this PR
  • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
  • @dependabot merge will merge this PR after your CI passes on it
  • @dependabot squash and merge will squash and merge this PR after your CI passes on it
  • @dependabot cancel merge will cancel a previously requested merge and block automerging
  • @dependabot reopen will reopen this PR if it is closed
  • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
  • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
  • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
  • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language
  • @dependabot badge me will comment on this PR with code to add a "Dependabot enabled" badge to your readme

Additionally, you can set the following in your Dependabot dashboard:

  • Update frequency (including time of day and day of week)
  • Pull request limits (per update run and/or open at any time)
  • Automerge options (never/patch/minor, and dev/runtime dependencies)
  • Out-of-range updates (receive only lockfile updates, if desired)
  • Security updates (receive only security updates, if desired)

</details>

+3 -3

1 comment

2 changed files

dependabot-preview[bot]

pr closed time in 12 days

PullRequestReviewEvent

push eventUKGovernmentBEIS/beis-report-official-development-assistance

James Coglan

commit sha 26cc9e771b4429d321712c850b59993176a68f4f

Stop modifying the Bullet whitelist during tests A few of our tests have this configuration that seems to designed to make Bullet temporarily ignore certain optimisations: before :all do Bullet.add_whitelist(type: :unused_eager_loading, class_name: "Activity", association: :parent) end after :all do Bullet.delete_whitelist(type: :unused_eager_loading, class_name: "Activity", association: :parent) end This particular setup makes Bullet ignore unused eager loads of the `Activity#parent` association. The problem is that, if this assoication was already ignored before these tests ran, then the call to `delete_whitelist` will cause it _not_ to be ignored in any tests that run later. This is because of how Bullet stores its whitelist. It indexes associations by optimisation type and class name, so if we have this in the application init code: Bullet.add_whitelist(type: :unused_eager_loading, class_name: "Activity", association: :parent) Bullet.add_whitelist(type: :unused_eager_loading, class_name: "Activity", association: :organisation) Then `Bullet.whitelist` will contain: { :unused_eager_loading => { "Activity" => [:parent, :organisation] } } The association names are stored in an Array, not a Set, so it's possible for them to appear twice. When we run the `before` block above, `:parent` is added again to this list: { :unused_eager_loading => { "Activity" => [:parent, :organisation, :parent] } } When we run the `after` block, `delete_whitelist` removes _all_ occurrences of the given association from the whitelist, so the state ends up containing: { :unused_eager_loading => { "Activity" => [:organisation] } } The `:parent` association is no longer ignored and so all subsequent tests may trigger warnings about it. Because RSpec randomises test order, this means we'll see inconsistent results with Bullet causing different tests to fail each time. It doesn't look like Bullet provides a safe API for this use case of temporarily ignoring a certain warning type. `delete_whitelist` itself is undocumented so it's unclear we should even be using it, or what it's intended to do, but it doesn't provide the inverse of `add_whitelist` as it removes _all_ occurrences of the given association from the list. `add_whitelist` doesn't provide any indication that the association we've just added was already in the whitelist, and therefore we shouldn't remove it after the test. The safest thing to do is probably to only use `Bullet.add_whitelist` in the application config, and not in individual tests.

view details

push time in 13 days

push eventUKGovernmentBEIS/beis-report-official-development-assistance

James Coglan

commit sha e6ea93dcf0b1ee2bdef03dc8b9db9a52b2cb382b

Downgrade better_errors to 2.8.1 There is a bug in better_errors 2.8.2, where it uses its VERSION constant without having loaded it first [1]. This is preventing our app from booting so I'm reverting this version bump until that issue is fixed. This reverts commit a6e2172a42abefaf4e2cf5f50762d0d52999def2, reversing changes made to ce4f953ada4ef1b01db0289f9a161bfe6b20706b. [1]: https://github.com/BetterErrors/better_errors/issues/483

view details

James Coglan

commit sha 01448e08874b979f985c8a1257be487d4027ee1f

Merge pull request #685 from UKGovernmentBEIS/fix/downgrade-better-errors Downgrade better_errors to 2.8.1

view details

push time in 18 days

delete branch UKGovernmentBEIS/beis-report-official-development-assistance

delete branch : fix/downgrade-better-errors

delete time in 18 days

PR merged UKGovernmentBEIS/beis-report-official-development-assistance

Downgrade better_errors to 2.8.1

There is a bug in better_errors 2.8.2, where it uses its VERSION constant without having loaded it first 1. This is preventing our app from booting so I'm reverting this version bump until that issue is fixed.

This reverts commit a6e2172a42abefaf4e2cf5f50762d0d52999def2, reversing changes made to ce4f953ada4ef1b01db0289f9a161bfe6b20706b.

+1 -1

0 comment

1 changed file

jcoglan

pr closed time in 18 days

issue commentBetterErrors/better_errors

uninitialized constant BetterErrors::Middleware::VERSION (NameError)

I believe this change is the source of the problem: referring to VERSION without having explicitly loaded that constant. It's stopping our Rails app from booting.

https://github.com/BetterErrors/better_errors/compare/v2.8.1..master#diff-5e3c1d57ea649185a86b730f951f5a18R43

seb-sykio

comment created time in 18 days

issue commentBetterErrors/better_errors

Version 2.8.2 breakage

I believe this change is the source of the problem: referring to VERSION without having explicitly loaded that constant. It's stopping our Rails app from booting.

https://github.com/BetterErrors/better_errors/compare/v2.8.1..master#diff-5e3c1d57ea649185a86b730f951f5a18R43

thomasbalsloev

comment created time in 18 days

PR opened UKGovernmentBEIS/beis-report-official-development-assistance

Downgrade better_errors to 2.8.1

There is a bug in better_errors 2.8.2, where it uses its VERSION constant without having loaded it first 1. This is preventing our app from booting so I'm reverting this version bump until that issue is fixed.

This reverts commit a6e2172a42abefaf4e2cf5f50762d0d52999def2, reversing changes made to ce4f953ada4ef1b01db0289f9a161bfe6b20706b.

+1 -1

0 comment

1 changed file

pr created time in 18 days

pull request commentUKGovernmentBEIS/beis-report-official-development-assistance

Bump puma from 4.3.6 to 5.0.2

As mentioned in #655, I'd like to hold off on this until Puma 5.0 has been out a little longer.

dependabot-preview[bot]

comment created time in 18 days

delete branch UKGovernmentBEIS/beis-report-official-development-assistance

delete branch : dependabot/bundler/better_errors-2.8.2

delete time in 18 days

push eventUKGovernmentBEIS/beis-report-official-development-assistance

dependabot-preview[bot]

commit sha 7cd47bdd70d4a961f8ea1d9bebdd14ee97969ca2

Bump better_errors from 2.8.1 to 2.8.2 Bumps [better_errors](https://github.com/BetterErrors/better_errors) from 2.8.1 to 2.8.2. - [Release notes](https://github.com/BetterErrors/better_errors/releases) - [Commits](https://github.com/BetterErrors/better_errors/commits) Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

view details

James Coglan

commit sha a6e2172a42abefaf4e2cf5f50762d0d52999def2

Merge pull request #682 from UKGovernmentBEIS/dependabot/bundler/better_errors-2.8.2 Bump better_errors from 2.8.1 to 2.8.2

view details

push time in 18 days

PR merged UKGovernmentBEIS/beis-report-official-development-assistance

Bump better_errors from 2.8.1 to 2.8.2 dependencies

Bumps better_errors from 2.8.1 to 2.8.2. <details> <summary>Commits</summary> <ul> <li>See full diff in <a href="https://github.com/BetterErrors/better_errors/commits">compare view</a></li> </ul> </details> <br />

Dependabot compatibility score

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


<details> <summary>Dependabot commands and options</summary> <br />

You can trigger Dependabot actions by commenting on this PR:

  • @dependabot rebase will rebase this PR
  • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
  • @dependabot merge will merge this PR after your CI passes on it
  • @dependabot squash and merge will squash and merge this PR after your CI passes on it
  • @dependabot cancel merge will cancel a previously requested merge and block automerging
  • @dependabot reopen will reopen this PR if it is closed
  • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
  • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
  • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
  • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language
  • @dependabot badge me will comment on this PR with code to add a "Dependabot enabled" badge to your readme

Additionally, you can set the following in your Dependabot dashboard:

  • Update frequency (including time of day and day of week)
  • Pull request limits (per update run and/or open at any time)
  • Automerge options (never/patch/minor, and dev/runtime dependencies)
  • Out-of-range updates (receive only lockfile updates, if desired)
  • Security updates (receive only security updates, if desired)

</details>

+1 -1

1 comment

1 changed file

dependabot-preview[bot]

pr closed time in 18 days

PullRequestReviewEvent

push eventUKGovernmentBEIS/beis-report-official-development-assistance

dependabot-preview[bot]

commit sha 2eab06329564cced5275e7dfb57ea645eb903a5a

Bump brakeman from 4.9.1 to 4.10.0 Bumps [brakeman](https://github.com/presidentbeef/brakeman) from 4.9.1 to 4.10.0. - [Release notes](https://github.com/presidentbeef/brakeman/releases) - [Changelog](https://github.com/presidentbeef/brakeman/blob/main/CHANGES.md) - [Commits](https://github.com/presidentbeef/brakeman/compare/v4.9.1...v4.10.0) Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

view details

James Coglan

commit sha ce4f953ada4ef1b01db0289f9a161bfe6b20706b

Merge pull request #677 from UKGovernmentBEIS/dependabot/bundler/brakeman-4.10.0 Bump brakeman from 4.9.1 to 4.10.0

view details

push time in 18 days

delete branch UKGovernmentBEIS/beis-report-official-development-assistance

delete branch : dependabot/bundler/brakeman-4.10.0

delete time in 18 days

PR merged UKGovernmentBEIS/beis-report-official-development-assistance

Bump brakeman from 4.9.1 to 4.10.0 dependencies

Bumps brakeman from 4.9.1 to 4.10.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/presidentbeef/brakeman/releases">brakeman's releases</a>.</em></p> <blockquote> <h2>4.10.0</h2> <ul> <li>Add SARIF report format (<a href="https://github.com/swinton">Steve Winton</a>)</li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/presidentbeef/brakeman/blob/main/CHANGES.md">brakeman's changelog</a>.</em></p> <blockquote> <h1>4.10.0 - 2020-09-28</h1> <ul> <li>Add SARIF report format (Steve Winton)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/presidentbeef/brakeman/commit/8f696e3568523563f83e8af4512a30f3bda70866"><code>8f696e3</code></a> Bump to 4.10.0</li> <li><a href="https://github.com/presidentbeef/brakeman/commit/2beaac0ff9d9cfdd58d59e81b4b38f7720049587"><code>2beaac0</code></a> Update CHANGES</li> <li><a href="https://github.com/presidentbeef/brakeman/commit/5daa392b570f2aa5588ad1b983ca21cb5ebbbfbd"><code>5daa392</code></a> Add SARIF output format</li> <li><a href="https://github.com/presidentbeef/brakeman/commit/4cb7ad25839a30a28ac991f020c8353bca529310"><code>4cb7ad2</code></a> Fix permissions during gem build</li> <li>See full diff in <a href="https://github.com/presidentbeef/brakeman/compare/v4.9.1...v4.10.0">compare view</a></li> </ul> </details> <br />

Dependabot compatibility score

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


<details> <summary>Dependabot commands and options</summary> <br />

You can trigger Dependabot actions by commenting on this PR:

  • @dependabot rebase will rebase this PR
  • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
  • @dependabot merge will merge this PR after your CI passes on it
  • @dependabot squash and merge will squash and merge this PR after your CI passes on it
  • @dependabot cancel merge will cancel a previously requested merge and block automerging
  • @dependabot reopen will reopen this PR if it is closed
  • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
  • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
  • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
  • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language
  • @dependabot badge me will comment on this PR with code to add a "Dependabot enabled" badge to your readme

Additionally, you can set the following in your Dependabot dashboard:

  • Update frequency (including time of day and day of week)
  • Pull request limits (per update run and/or open at any time)
  • Automerge options (never/patch/minor, and dev/runtime dependencies)
  • Out-of-range updates (receive only lockfile updates, if desired)
  • Security updates (receive only security updates, if desired)

</details>

+1 -1

0 comment

1 changed file

dependabot-preview[bot]

pr closed time in 18 days

PullRequestReviewEvent

Pull request review commentUKGovernmentBEIS/beis-report-official-development-assistance

(1036) BEIS Users can download Programme (Level B) xml

 def call(eager_load_parent: true)       .includes(eager_load_associations)       .order("created_at ASC") -    programmes = if organisation.service_owner-      programmes.all-    else-      programmes.where(extending_organisation_id: organisation.id)-    end-    programmes+    return programmes if organisation.service_owner++    programmes.where(extending_organisation_id: organisation.id).where(fund_query_condition(fund_id))

None of these snippets execute a query -- where does not trigger a database request on its own. The query is only executed when you attempted to read the result, e.g. by calling each or first. So you can build a query by incrementally adding where calls to a chain, and it effectively merges their conditions into a single query.

mec

comment created time in a month

PullRequestReviewEvent

Pull request review commentUKGovernmentBEIS/beis-report-official-development-assistance

(1036) BEIS Users can download Programme (Level B) xml

 def show      @organisation_presenter = OrganisationPresenter.new(organisation) -    @project_activities = FindProjectActivities.new(organisation: organisation, user: current_user).call(eager_load_parent: false).publishable_to_iati-    @third_party_project_activities = FindThirdPartyProjectActivities.new(organisation: organisation, user: current_user).call(eager_load_parent: false).publishable_to_iati+    fund_ids_for_organisation_programmes = Activity.where(+      level: :programme,+      extending_organisation_id: organisation.id+    ).pluck(:parent_id)+    @funds = Activity.find(fund_ids_for_organisation_programmes)++    @project_activities = FindProjectActivities.new(+      organisation: organisation,+      user: current_user+    ).call(eager_load_parent: false).publishable_to_iati++    @third_party_project_activities = FindThirdPartyProjectActivities.new(+      organisation: organisation,+      user: current_user+    ).call(eager_load_parent: false).publishable_to_iati

These variables are well-named, and it's probably worth pulling these queries out into methods so the show method becomes a high-level description of what's being displayed, rather than the implementation details of fetching that data.

mec

comment created time in a month

PullRequestReviewEvent

push eventUKGovernmentBEIS/beis-report-official-development-assistance

Laura Porter

commit sha c4309d7e6929b48c79ec0a4c0ee55005c0e8dcc8

Show an error message if the user does not select a recipient country Previously we were not showing an error message if the user did not select a recipient country. If no country was selected, we were attempting to find the associated recipient region despite the country being blank. This was causing an error. How, if the country is blank, we assign it to the activity, thereby forcing a validation error to be shown. We do not try to associate the blank country value to a region.

view details

James Coglan

commit sha 89e5fcce0ac5e8a0c70d795c00c63b4505518f45

Merge pull request #670 from UKGovernmentBEIS/fix/872-bug-no-error-message-shown-when-you-dont-select-a-country Show an error message if the user does not select a recipient country

view details

push time in a month

delete branch UKGovernmentBEIS/beis-report-official-development-assistance

delete branch : fix/872-bug-no-error-message-shown-when-you-dont-select-a-country

delete time in a month

Pull request review commentUKGovernmentBEIS/beis-report-official-development-assistance

(1036) BEIS Users can download Programme (Level B) xml

 def call(eager_load_parent: true)       .includes(eager_load_associations)       .order("created_at ASC") -    programmes = if organisation.service_owner-      programmes.all-    else-      programmes.where(extending_organisation_id: organisation.id)-    end-    programmes+    return programmes if organisation.service_owner++    programmes.where(extending_organisation_id: organisation.id).where(fund_query_condition(fund_id))

If you're worried about this being too magical, another option would be to make the whole .where call conditional on having a fund_id, i.e.:

programmes = programmes.where(extending_organisation_id: organisation.id)
programmes = programmes.where(parent_id: fund_id) if fund_id.present?

programmes

Or, dynamically build the conditions, as in:

conditions = { extending_organisation_id: organisation.id }
conditions[:parent_id] = fund_id if fund_id.present?

programmes.where(conditions)
mec

comment created time in a month

PullRequestReviewEvent

push eventUKGovernmentBEIS/beis-report-official-development-assistance

James Coglan

commit sha 6d77febc1f2cfc286f785a2012403e44dd32d1ab

Bump omniauth-auth0 from 2.3.1 to 2.4.0 Bumps [omniauth-auth0](https://github.com/auth0/omniauth-auth0) from 2.3.1 to 2.4.0. - [Release notes](https://github.com/auth0/omniauth-auth0/releases) - [Changelog](https://github.com/auth0/omniauth-auth0/blob/master/CHANGELOG.md) - [Commits](auth0/omniauth-auth0@v2.3.1...v2.4.0) Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

view details

James Coglan

commit sha a8b8fa88d463fc2ba6f2c4cf3b494974aafaee1f

Merge pull request #668 from UKGovernmentBEIS/dependabot/bundler/omniauth-auth0-2.4.0 Bump omniauth-auth0 from 2.3.1 to 2.4.0

view details

push time in a month

delete branch UKGovernmentBEIS/beis-report-official-development-assistance

delete branch : dependabot/bundler/omniauth-auth0-2.4.0

delete time in a month

push eventUKGovernmentBEIS/beis-report-official-development-assistance

dependabot-preview[bot]

commit sha 97ec43b642041a4c743e838c94b350a8eceb674e

Bump omniauth-auth0 from 2.3.1 to 2.4.0 Bumps [omniauth-auth0](https://github.com/auth0/omniauth-auth0) from 2.3.1 to 2.4.0. - [Release notes](https://github.com/auth0/omniauth-auth0/releases) - [Changelog](https://github.com/auth0/omniauth-auth0/blob/master/CHANGELOG.md) - [Commits](https://github.com/auth0/omniauth-auth0/compare/v2.3.1...v2.4.0) Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

view details

James Coglan

commit sha a923ae9a1e824594c017447c17bc774ada6a9f0c

Merge branch 'develop' into dependabot/bundler/omniauth-auth0-2.4.0

view details

James Coglan

commit sha 61128885d8eb139461431f0089ca3db53207bb8e

Merge pull request #663 from UKGovernmentBEIS/dependabot/bundler/omniauth-auth0-2.4.0 Bump omniauth-auth0 from 2.3.1 to 2.4.0

view details

push time in a month

delete branch UKGovernmentBEIS/beis-report-official-development-assistance

delete branch : dependabot/bundler/omniauth-auth0-2.4.0

delete time in a month

PR merged UKGovernmentBEIS/beis-report-official-development-assistance

Bump omniauth-auth0 from 2.3.1 to 2.4.0 dependencies

Bumps omniauth-auth0 from 2.3.1 to 2.4.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/auth0/omniauth-auth0/releases">omniauth-auth0's releases</a>.</em></p> <blockquote> <h2>v2.4.0</h2> <h2><a href="https://github.com/auth0/omniauth-auth0/tree/v2.4.0">v2.4.0</a> (2020-09-22)</h2> <p><a href="https://github.com/auth0/omniauth-auth0/compare/v2.3.1...v2.4.0">Full Changelog</a></p> <p><strong>Security</strong></p> <ul> <li>Bump rack from 2.2.2 to 2.2.3 <a href="https://github-redirect.dependabot.com/auth0/omniauth-auth0/pull/107">#107</a> (<a href="https://github.com/dependabot">dependabot</a>)</li> <li>Update dependencies <a href="https://github-redirect.dependabot.com/auth0/omniauth-auth0/pull/100">#100</a> (<a href="https://github.com/Albalmaceda">Albalmaceda</a>)</li> </ul> <p><strong>Added</strong></p> <ul> <li>Add support for screen_hint=signup param <a href="https://github-redirect.dependabot.com/auth0/omniauth-auth0/pull/103">#103</a> (<a href="https://github.com/bbean86">bbean86</a>)</li> <li>Add support for <code>connection_scope</code> in params <a href="https://github-redirect.dependabot.com/auth0/omniauth-auth0/pull/99">#99</a> (<a href="https://github.com/felixclack">felixclack</a>)</li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/auth0/omniauth-auth0/blob/master/CHANGELOG.md">omniauth-auth0's changelog</a>.</em></p> <blockquote> <h2><a href="https://github.com/auth0/omniauth-auth0/tree/v2.4.0">v2.4.0</a> (2020-09-22)</h2> <p><a href="https://github.com/auth0/omniauth-auth0/compare/v2.3.1...v2.4.0">Full Changelog</a></p> <p><strong>Security</strong></p> <ul> <li>Bump rack from 2.2.2 to 2.2.3 <a href="https://github-redirect.dependabot.com/auth0/omniauth-auth0/pull/107">#107</a> (<a href="https://github.com/dependabot">dependabot</a>)</li> <li>Update dependencies <a href="https://github-redirect.dependabot.com/auth0/omniauth-auth0/pull/100">#100</a> (<a href="https://github.com/Albalmaceda">Albalmaceda</a>)</li> </ul> <p><strong>Added</strong></p> <ul> <li>Add support for screen_hint=signup param <a href="https://github-redirect.dependabot.com/auth0/omniauth-auth0/pull/103">#103</a> (<a href="https://github.com/bbean86">bbean86</a>)</li> <li>Add support for <code>connection_scope</code> in params <a href="https://github-redirect.dependabot.com/auth0/omniauth-auth0/pull/99">#99</a> (<a href="https://github.com/felixclack">felixclack</a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/auth0/omniauth-auth0/commit/860bd451eeb8390ba33a7fcf8fcb969e214a7236"><code>860bd45</code></a> Bump omniauth-auth0 to 2.4.0 (<a href="https://github-redirect.dependabot.com/auth0/omniauth-auth0/issues/108">#108</a>)</li> <li><a href="https://github.com/auth0/omniauth-auth0/commit/7cfc4faa7b4aed9126082ecf6b7e3d8b0fb238f0"><code>7cfc4fa</code></a> Bump rack from 2.2.2 to 2.2.3 (<a href="https://github-redirect.dependabot.com/auth0/omniauth-auth0/issues/107">#107</a>)</li> <li><a href="https://github.com/auth0/omniauth-auth0/commit/fb70eacc6bfa211b138781330a9b818871f167e7"><code>fb70eac</code></a> Add support for screen_hint=signup param (<a href="https://github-redirect.dependabot.com/auth0/omniauth-auth0/issues/103">#103</a>)</li> <li><a href="https://github.com/auth0/omniauth-auth0/commit/7f415614e4219ed157a2587a5dab2cc3b4ef6086"><code>7f41561</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/auth0/omniauth-auth0/issues/106">#106</a> from fossabot/add-license-scan-badge</li> <li><a href="https://github.com/auth0/omniauth-auth0/commit/c3eb541a3d9f7e2140952628919d0182a6e085f1"><code>c3eb541</code></a> Add license scan report and status</li> <li><a href="https://github.com/auth0/omniauth-auth0/commit/9ed2ce50652b97459c0f97bf911bd170912e081c"><code>9ed2ce5</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/auth0/omniauth-auth0/issues/104">#104</a> from auth0/davidpatrick-patch-1</li> <li><a href="https://github.com/auth0/omniauth-auth0/commit/55a4dc09df60b5a3e0f48d840af81f935fecef5c"><code>55a4dc0</code></a> Update README.md</li> <li><a href="https://github.com/auth0/omniauth-auth0/commit/fafaba32fbddc1daf9c833883851b17de5f27baf"><code>fafaba3</code></a> Add security disclaimer for using library</li> <li><a href="https://github.com/auth0/omniauth-auth0/commit/ed9f4e4905d351c54a653b27804be08068ed7f6c"><code>ed9f4e4</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/auth0/omniauth-auth0/issues/99">#99</a> from felixclack/connection-scope</li> <li><a href="https://github.com/auth0/omniauth-auth0/commit/1cf2b95c7d15e448c3d84d05196843170f2f47d4"><code>1cf2b95</code></a> Merge branch 'master' into connection-scope</li> <li>Additional commits viewable in <a href="https://github.com/auth0/omniauth-auth0/compare/v2.3.1...v2.4.0">compare view</a></li> </ul> </details> <br />

Dependabot compatibility score

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


<details> <summary>Dependabot commands and options</summary> <br />

You can trigger Dependabot actions by commenting on this PR:

  • @dependabot rebase will rebase this PR
  • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
  • @dependabot merge will merge this PR after your CI passes on it
  • @dependabot squash and merge will squash and merge this PR after your CI passes on it
  • @dependabot cancel merge will cancel a previously requested merge and block automerging
  • @dependabot reopen will reopen this PR if it is closed
  • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
  • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
  • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
  • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language
  • @dependabot badge me will comment on this PR with code to add a "Dependabot enabled" badge to your readme

Additionally, you can set the following in your Dependabot dashboard:

  • Update frequency (including time of day and day of week)
  • Pull request limits (per update run and/or open at any time)
  • Automerge options (never/patch/minor, and dev/runtime dependencies)
  • Out-of-range updates (receive only lockfile updates, if desired)
  • Security updates (receive only security updates, if desired)

</details>

+6 -6

0 comment

2 changed files

dependabot-preview[bot]

pr closed time in a month

push eventUKGovernmentBEIS/beis-report-official-development-assistance

Robert Lee-Cann

commit sha f73aba0b189569d287b3455ccbe42db1dec0d746

Extract updating model into save_changes! method This will be re-used for saving transactions as well as activities.

view details

Robert Lee-Cann

commit sha 8df3b563a36f761a34476a191f91913edbf57fa1

Ingest transactions when CSV has 'transaction_type' If the input CSV contains a `transaction_type` column, we assume that we're ingesting transactions only, so the Transaction associated with an Activity is modified, rather than the Activity itself. We assume that the combination of delivery_partner_identifier and description is unique enough to allow existing transactions to be overwritten with new data. When creating a transaction we attempt to associate it with the correct report. Skipping the validation check for report means we can still save the transaction on the occasions one doesn't exist. Since we only know the providing organisation side of these transactions, we also skip the validation check on receiving_organisation_name and receiving_organisation_type so we can save the record.

view details

Robert Lee-Cann

commit sha 6aea9f3835510ed3ce6a98ddc2831adc23679bb9

IngestCsv processes transaction columns - transaction_type - date - currency - value - providing_organisation_type NB: description, providing_organisation_name and providing_organisation_reference are ingested as-is.

view details

Robert Lee-Cann

commit sha ddbe3fb4e6778d46b6b1d0776226142db83fa7f7

IngestCsv ignores some additional validations Ignore validation errors on the following attributes: - date (so we can ingest future dated transactions) - collaboration_type (existing activities may not have this set)

view details

Robert Lee-Cann

commit sha 18d2eeb3a104de0578c2737e350a017fbc63793e

Add special cases for Sector and Country lookups There were a small number of ocassions where the CSV contained sectors or countries that were not exact matches for those specified in the IATI approved codelists. Therefore, some additional mappings were manually made to allow ingest to complete successfully. NB: Some forthcoming work will ensure that the codelists are updated in RODA so that activities can be assigned correctly.

view details

Robert Lee-Cann

commit sha df98c32a6f09ebbd499d08de9e884906bcf3ed4f

Merge pull request #653 from UKGovernmentBEIS/feature/986-ingest-financials-from-csv (986) Ingest financials from CSV

view details

Laura Porter

commit sha 970ecef727dceb48cf131f2dd50e2b8f6f10a674

Swap order of Actuals & Forecast so Forecast comes first in the CSV columns

view details

Laura Porter

commit sha 486f280dd85cf99487163960c638fc2a273a6c27

Merge pull request #664 from UKGovernmentBEIS/fix/swap-order-of-fields-in-report-csv Swap order of Actuals & Forecast so Forecast comes first in the CSV columns

view details

James Coglan

commit sha 45bfba4496f9ea50d3e730e2c54037bdcf3d56a4

Reject non-numeric values in bulk transaction import Monetary values entered in the `Value` column in bulk upload are passed through the `CreateTransaction` service, which uses `Monetize.parse` to interpret them. This is much more permissive than we want; it allows various currency symbols and many different kinds of formatting, whereas we want to specifically allow this format: - an optional `£` symbol, followed by - an optional minus (`-`) symbol, followed by - at least 1 digit, with comma (`,`) allowed for readability - an optional decimal point followed by 2 digits We also cannot just remove the Monetize call, as passing the string directly through to ActiveRecord results in truncation; a string like "3a45" is truncated to the value 3.0. We would like any non-numeric characters in the input treated as an error, as they may represent a mistake on the user's part. This is especially important in bulk upload, where a user is adding a lot of data at once and not necessarily checking all of it afterwards. The ConvertFinancialValue service implements the desired format, and ImportTransactions can then report any parsing errors back to the user.

view details

James Coglan

commit sha 6fa5b18d7fb400e79091e3b6c218308547ad69fc

Merge pull request #662 from UKGovernmentBEIS/feature/check-financial-values Reject non-numeric values in bulk transaction import

view details

Laura Porter

commit sha e5b9cabe3ce476c9454c6fd5d92b5bdeed14d83b

Modify ConvertFinancialValue to handle a single decimal place The existing tests for CreateTransaction expect a number with a single decimal place to be handled, so amend ConvertFinancialValue to respect single decimal places (but reject three or more decimal places)

view details

Laura Porter

commit sha 190ee3da0ac2550399f7b5ce2dc881414a627f96

Do not automatically strip letters from a monetary value Previously, letters accidentally entered in the `value` fields for budgets, transactions and planned disbursements were being stripped and the numbers accepted. We would prefer these entries to be rejected as an error instead of silently stripped and passed through. Use the `ConvertFinancialValue` to sanitize the values before they are added to their parent entities, and show a relevant error message if any letters are present.

view details

Laura Porter

commit sha 1e5c006a3e4a3a82d6376d7651a9bbba6797e4b2

Merge pull request #665 from UKGovernmentBEIS/fix/1014-when-a-monetary-value-is-supplied-with-letters-in-it-do-not-strip-lettters (1014) Do not strip alphabetical characters from values, reject the value instead

view details

James Coglan

commit sha a923ae9a1e824594c017447c17bc774ada6a9f0c

Merge branch 'develop' into dependabot/bundler/omniauth-auth0-2.4.0

view details

push time in a month

PullRequestReviewEvent
PullRequestReviewEvent
PullRequestReviewEvent
PullRequestReviewEvent

Pull request review commentUKGovernmentBEIS/beis-report-official-development-assistance

(1014) Do not strip alphabetical characters from values, reject the value instead

 def call(attributes: {})     result   end -  private def editable_report_for_activity(activity:)+  private++  def editable_report_for_activity(activity:)     Report.find_by(organisation: activity.organisation, fund: activity.associated_fund, state: Report::EDITABLE_STATES)   end -  private def sanitize_monetary_string(value:)-    Monetize.parse(value)+  def convert_and_assign_value(budget, value)+    budget.value = ConvertFinancialValue.new.convert(value.to_s)+    budget

Cool cool :shipit:

lozette

comment created time in a month

PullRequestReviewEvent

pull request commentUKGovernmentBEIS/beis-report-official-development-assistance

(861) Sign out link not active on Users page

@mec That's fair enough, I think if this was a behaviour bug e.g. people not being able to log out, a test would be necessary but it's more of a judgement call for presentational concerns. Do go ahead and merge :)

mec

comment created time in a month

Pull request review commentUKGovernmentBEIS/beis-report-official-development-assistance

(1014) Do not strip alphabetical characters from values, reject the value instead

 def call(attributes: {})     result   end -  private def editable_report_for_activity(activity:)+  private++  def editable_report_for_activity(activity:)     Report.find_by(organisation: activity.organisation, fund: activity.associated_fund, state: Report::EDITABLE_STATES)   end -  private def sanitize_monetary_string(value:)-    Monetize.parse(value)+  def convert_and_assign_value(budget, value)+    budget.value = ConvertFinancialValue.new.convert(value.to_s)+    budget

What's the reason for returning the record in each of these methods? The result of convert_and_assign_value does not appear to be used.

lozette

comment created time in a month

PullRequestReviewEvent

Pull request review commentUKGovernmentBEIS/beis-report-official-development-assistance

(1014) Do not strip alphabetical characters from values, reject the value instead

 class ConvertFinancialValue-  VALUE_FORMAT = /^(?:£ *)?((?:- *)?[0-9,]+(?:\.[0-9]{2})?)$/+  VALUE_FORMAT = /^(?:£ *)?((?:- *)?[0-9,]+(?:\.[0-9]{1,2})?)$/

Apologies, I overlooked this as a requirement. Thanks for correcting it!

lozette

comment created time in a month

PullRequestReviewEvent

push eventUKGovernmentBEIS/beis-report-official-development-assistance

James Coglan

commit sha 45bfba4496f9ea50d3e730e2c54037bdcf3d56a4

Reject non-numeric values in bulk transaction import Monetary values entered in the `Value` column in bulk upload are passed through the `CreateTransaction` service, which uses `Monetize.parse` to interpret them. This is much more permissive than we want; it allows various currency symbols and many different kinds of formatting, whereas we want to specifically allow this format: - an optional `£` symbol, followed by - an optional minus (`-`) symbol, followed by - at least 1 digit, with comma (`,`) allowed for readability - an optional decimal point followed by 2 digits We also cannot just remove the Monetize call, as passing the string directly through to ActiveRecord results in truncation; a string like "3a45" is truncated to the value 3.0. We would like any non-numeric characters in the input treated as an error, as they may represent a mistake on the user's part. This is especially important in bulk upload, where a user is adding a lot of data at once and not necessarily checking all of it afterwards. The ConvertFinancialValue service implements the desired format, and ImportTransactions can then report any parsing errors back to the user.

view details

James Coglan

commit sha 6fa5b18d7fb400e79091e3b6c218308547ad69fc

Merge pull request #662 from UKGovernmentBEIS/feature/check-financial-values Reject non-numeric values in bulk transaction import

view details

push time in a month

delete branch UKGovernmentBEIS/beis-report-official-development-assistance

delete branch : feature/check-financial-values

delete time in a month

PR merged UKGovernmentBEIS/beis-report-official-development-assistance

Reject non-numeric values in bulk transaction import

Monetary values entered in the Value column in bulk upload are passed through the CreateTransaction service, which uses Monetize.parse to interpret them. This is much more permissive than we want; it allows various currency symbols and many different kinds of formatting, whereas we want to specifically allow this format:

  • an optional £ symbol, followed by
  • an optional minus (-) symbol, followed by
  • at least 1 digit, with comma (,) allowed for readability
  • an optional decimal point followed by 2 digits

We also cannot just remove the Monetize call, as passing the string directly through to ActiveRecord results in truncation; a string like "3a45" is truncated to the value 3.0. We would like any non-numeric characters in the input treated as an error, as they may represent a mistake on the user's part. This is especially important in bulk upload, where a user is adding a lot of data at once and not necessarily checking all of it afterwards.

The ConvertFinancialValue service implements the desired format, and ImportTransactions can then report any parsing errors back to the user.

Changes in this PR

Screenshots of UI changes

Before

After

Next steps

  • [ ] Is an ADR required? An ADR should be added if this PR introduces a change to the architecture.
  • [x] Is a changelog entry required? An entry should always be made in CHANGELOG.md, unless this PR is a small tweak which has no impact outside the development team.
  • [ ] Do any environment variables need amending or adding?
  • [ ] Have any changes to the XML been checked with the IATI validator? See XML Validation
+81 -2

0 comment

6 changed files

jcoglan

pr closed time in a month

push eventUKGovernmentBEIS/beis-report-official-development-assistance

Robert Lee-Cann

commit sha f73aba0b189569d287b3455ccbe42db1dec0d746

Extract updating model into save_changes! method This will be re-used for saving transactions as well as activities.

view details

Robert Lee-Cann

commit sha 8df3b563a36f761a34476a191f91913edbf57fa1

Ingest transactions when CSV has 'transaction_type' If the input CSV contains a `transaction_type` column, we assume that we're ingesting transactions only, so the Transaction associated with an Activity is modified, rather than the Activity itself. We assume that the combination of delivery_partner_identifier and description is unique enough to allow existing transactions to be overwritten with new data. When creating a transaction we attempt to associate it with the correct report. Skipping the validation check for report means we can still save the transaction on the occasions one doesn't exist. Since we only know the providing organisation side of these transactions, we also skip the validation check on receiving_organisation_name and receiving_organisation_type so we can save the record.

view details

Robert Lee-Cann

commit sha 6aea9f3835510ed3ce6a98ddc2831adc23679bb9

IngestCsv processes transaction columns - transaction_type - date - currency - value - providing_organisation_type NB: description, providing_organisation_name and providing_organisation_reference are ingested as-is.

view details

Robert Lee-Cann

commit sha ddbe3fb4e6778d46b6b1d0776226142db83fa7f7

IngestCsv ignores some additional validations Ignore validation errors on the following attributes: - date (so we can ingest future dated transactions) - collaboration_type (existing activities may not have this set)

view details

Robert Lee-Cann

commit sha 18d2eeb3a104de0578c2737e350a017fbc63793e

Add special cases for Sector and Country lookups There were a small number of ocassions where the CSV contained sectors or countries that were not exact matches for those specified in the IATI approved codelists. Therefore, some additional mappings were manually made to allow ingest to complete successfully. NB: Some forthcoming work will ensure that the codelists are updated in RODA so that activities can be assigned correctly.

view details

Robert Lee-Cann

commit sha df98c32a6f09ebbd499d08de9e884906bcf3ed4f

Merge pull request #653 from UKGovernmentBEIS/feature/986-ingest-financials-from-csv (986) Ingest financials from CSV

view details

Laura Porter

commit sha 970ecef727dceb48cf131f2dd50e2b8f6f10a674

Swap order of Actuals & Forecast so Forecast comes first in the CSV columns

view details

Laura Porter

commit sha 486f280dd85cf99487163960c638fc2a273a6c27

Merge pull request #664 from UKGovernmentBEIS/fix/swap-order-of-fields-in-report-csv Swap order of Actuals & Forecast so Forecast comes first in the CSV columns

view details

James Coglan

commit sha 45bfba4496f9ea50d3e730e2c54037bdcf3d56a4

Reject non-numeric values in bulk transaction import Monetary values entered in the `Value` column in bulk upload are passed through the `CreateTransaction` service, which uses `Monetize.parse` to interpret them. This is much more permissive than we want; it allows various currency symbols and many different kinds of formatting, whereas we want to specifically allow this format: - an optional `£` symbol, followed by - an optional minus (`-`) symbol, followed by - at least 1 digit, with comma (`,`) allowed for readability - an optional decimal point followed by 2 digits We also cannot just remove the Monetize call, as passing the string directly through to ActiveRecord results in truncation; a string like "3a45" is truncated to the value 3.0. We would like any non-numeric characters in the input treated as an error, as they may represent a mistake on the user's part. This is especially important in bulk upload, where a user is adding a lot of data at once and not necessarily checking all of it afterwards. The ConvertFinancialValue service implements the desired format, and ImportTransactions can then report any parsing errors back to the user.

view details

push time in a month

push eventUKGovernmentBEIS/beis-report-official-development-assistance

James Coglan

commit sha c7623f332fb7132218e3321b32afdab652285b1c

Reject non-numeric values in bulk transaction import Monetary values entered in the `Value` column in bulk upload are passed through the `CreateTransaction` service, which uses `Monetize.parse` to interpret them. This is much more permissive than we want; it allows various currency symbols and many different kinds of formatting, whereas we want to specifically allow this format: - an optional `£` symbol, followed by - an optional minus (`-`) symbol, followed by - at least 1 digit, with comma (`,`) allowed for readability - an optional decimal point followed by 2 digits We also cannot just remove the Monetize call, as passing the string directly through to ActiveRecord results in truncation; a string like "3a45" is truncated to the value 3.0. We would like any non-numeric characters in the input treated as an error, as they may represent a mistake on the user's part. This is especially important in bulk upload, where a user is adding a lot of data at once and not necessarily checking all of it afterwards. The ConvertFinancialValue service implements the desired format, and ImportTransactions can then report any parsing errors back to the user.

view details

push time in a month

PR opened UKGovernmentBEIS/beis-report-official-development-assistance

Reject non-numeric values in bulk transaction import

Monetary values entered in the Value column in bulk upload are passed through the CreateTransaction service, which uses Monetize.parse to interpret them. This is much more permissive than we want; it allows various currency symbols and many different kinds of formatting, whereas we want to specifically allow this format:

  • an optional £ symbol, followed by
  • an optional minus (-) symbol, followed by
  • at least 1 digit, with comma (,) allowed for readability
  • an optional decimal point followed by 2 digits

We also cannot just remove the Monetize call, as passing the string directly through to ActiveRecord results in truncation; a string like "3a45" is truncated to the value 3.0. We would like any non-numeric characters in the input treated as an error, as they may represent a mistake on the user's part. This is especially important in bulk upload, where a user is adding a lot of data at once and not necessarily checking all of it afterwards.

The ConvertFinancialValue service implements the desired format, and ImportTransactions can then report any parsing errors back to the user.

Changes in this PR

Screenshots of UI changes

Before

After

Next steps

  • [ ] Is an ADR required? An ADR should be added if this PR introduces a change to the architecture.
  • [ ] Is a changelog entry required? An entry should always be made in CHANGELOG.md, unless this PR is a small tweak which has no impact outside the development team.
  • [ ] Do any environment variables need amending or adding?
  • [ ] Have any changes to the XML been checked with the IATI validator? See XML Validation
+79 -2

0 comment

5 changed files

pr created time in a month

pull request commentUKGovernmentBEIS/beis-report-official-development-assistance

(861) Sign out link not active on Users page

If this was a bug, is it possible to add a test that confirms the expected behaviour?

mec

comment created time in a month

PullRequestReviewEvent

push eventUKGovernmentBEIS/beis-report-official-development-assistance

James Coglan

commit sha 651f7542ad763f7d442a4caead023260ced6285e

Add Activity#roda_identifier This method lets you get the RODA identifier for an activity while hiding the implementation details of the `roda_identifier_fragment` and `roda_identifier_compound` fields. It returns the "compound" identifier, i.e. the cached ID formed by concatenating the fragments of all the ancestor activities.

view details

James Coglan

commit sha f0b4aedb0fbaa3a61b893e3e0c1a47c534a63d4f

Use Activity#roda_identifier where applicable Now that `Activity#roda_identifier` exists to return the RODA identifier, we should use this everywhere we were previously using the `roda_identifier_compound` field directly, to clarify the code and signal how devs _should_ get the RODA identifier for an activity.

view details

James Coglan

commit sha 8ffc8ab558d9390324f948a304a5db7923cbb941

Merge pull request #659 from UKGovernmentBEIS/chore/roda-identifier-alias Add Activity#roda_identifier

view details

push time in a month

delete branch UKGovernmentBEIS/beis-report-official-development-assistance

delete branch : chore/roda-identifier-alias

delete time in a month

PR merged UKGovernmentBEIS/beis-report-official-development-assistance

Add Activity#roda_identifier

Changes in this PR

This is refactoring triggered by a conversation I had with @lozette, to make it clearer how one should get the RODA identifier for an activity.

Next steps

  • [ ] Is an ADR required? An ADR should be added if this PR introduces a change to the architecture.
  • [ ] Is a changelog entry required? An entry should always be made in CHANGELOG.md, unless this PR is a small tweak which has no impact outside the development team.
  • [ ] Do any environment variables need amending or adding?
  • [ ] Have any changes to the XML been checked with the IATI validator? See XML Validation
+40 -17

0 comment

10 changed files

jcoglan

pr closed time in a month

PR opened UKGovernmentBEIS/beis-report-official-development-assistance

Add Activity#roda_identifier

Changes in this PR

This is refactoring triggered by a conversation I had with @lozette, to make it clearer how one should get the RODA identifier for an activity.

Next steps

  • [ ] Is an ADR required? An ADR should be added if this PR introduces a change to the architecture.
  • [ ] Is a changelog entry required? An entry should always be made in CHANGELOG.md, unless this PR is a small tweak which has no impact outside the development team.
  • [ ] Do any environment variables need amending or adding?
  • [ ] Have any changes to the XML been checked with the IATI validator? See XML Validation
+40 -17

0 comment

10 changed files

pr created time in a month

issue commentfaye/websocket-driver-ruby

Unhandled server-side disconnects

@tjad Just to confirm: websocket-driver should always send a pong message if it receives a ping. It will only send a ping message if explicitly instructed.

dblock

comment created time in a month

PullRequestReviewEvent

pull request commentUKGovernmentBEIS/beis-report-official-development-assistance

Bump puma from 4.3.6 to 5.0.0

We discussed this and agreed we should perform this upgrade when we have more time to thoroughly test it on staging. The release notes are worth reading -- it sounds like this is mostly opt-in experiemental performance improvements, but as it's a major version change I'd like to thoroughly test it before going to production.

dependabot-preview[bot]

comment created time in a month

push eventUKGovernmentBEIS/beis-report-official-development-assistance

James Coglan

commit sha 794678187dd4ae0f397a613c265ef9a4ee983f11

Display the parent activity's RODA Identifier When entering a RODA Identifier, there's nothing to tell the user where they are, and the data they enter here is final, so it's both easy and very costly to make a mistake and enter the wrong thing. As a quick improvement, we can add the parent activity's RODA identifier to the page, if the activity has a parent. You can only enter a RODA ID if the parent activity has one completed, so this is guaranteed to always have a value to display.

view details

James Coglan

commit sha 33a94698ff928bcf35cbfd8bf284565eef92a371

Merge pull request #650 from UKGovernmentBEIS/feature/998-show-roda-identifier-prefixes (998) Display the parent activity's RODA Identifier

view details

push time in a month

delete branch UKGovernmentBEIS/beis-report-official-development-assistance

delete branch : feature/998-show-roda-identifier-prefixes

delete time in a month

PR merged UKGovernmentBEIS/beis-report-official-development-assistance

(998) Display the parent activity's RODA Identifier

When entering a RODA Identifier, there's nothing to tell the user where they are, and the data they enter here is final, so it's both easy and very costly to make a mistake and enter the wrong thing.

As a quick improvement, we can add the parent activity's RODA identifier to the page, if the activity has a parent. You can only enter a RODA ID if the parent activity has one completed, so this is guaranteed to always have a value to display.

Screenshots of UI changes

Screenshot 2020-09-17 at 16 18 11

Next steps

  • [ ] Is an ADR required? An ADR should be added if this PR introduces a change to the architecture.
  • [x] Is a changelog entry required? An entry should always be made in CHANGELOG.md, unless this PR is a small tweak which has no impact outside the development team.
  • [ ] Do any environment variables need amending or adding?
  • [ ] Have any changes to the XML been checked with the IATI validator? See XML Validation
+9 -2

1 comment

3 changed files

jcoglan

pr closed time in a month

push eventUKGovernmentBEIS/beis-report-official-development-assistance

dependabot-preview[bot]

commit sha 421ed74e742c4f60d9ac20c834026db9d93a6d80

Bump better_errors from 2.8.0 to 2.8.1 Bumps [better_errors](https://github.com/BetterErrors/better_errors) from 2.8.0 to 2.8.1. - [Release notes](https://github.com/BetterErrors/better_errors/releases) - [Commits](https://github.com/BetterErrors/better_errors/compare/v2.8.0...v2.8.1) Signed-off-by: dependabot-preview[bot] <support@dependabot.com>

view details

Meyric Rawlings

commit sha 1fd7fb05eb192217c04d3078a295df06a3336170

Merge pull request #654 from UKGovernmentBEIS/dependabot/bundler/better_errors-2.8.1 Bump better_errors from 2.8.0 to 2.8.1

view details

James Coglan

commit sha 794678187dd4ae0f397a613c265ef9a4ee983f11

Display the parent activity's RODA Identifier When entering a RODA Identifier, there's nothing to tell the user where they are, and the data they enter here is final, so it's both easy and very costly to make a mistake and enter the wrong thing. As a quick improvement, we can add the parent activity's RODA identifier to the page, if the activity has a parent. You can only enter a RODA ID if the parent activity has one completed, so this is guaranteed to always have a value to display.

view details

push time in a month

push eventUKGovernmentBEIS/beis-report-official-development-assistance

James Coglan

commit sha e648afc8007da20676b43556761d7655200bb5a1

Import a single transaction from a CSV This begins the work of importing transactions in bulk from a CSV input. Most of the machinery will like in the ImportTransactions service, which takes an array of hashes rather than a CSV file directly, to make it easier to test. The initial implementation maps all the bulk import columns to their places in the database, but does not do any additional validation or conversion beyond that performed by the model. It does provide some defaults for mandatory fields: - The currency is GBP - The transaction type is 'Disbursement' - The 'providing organisation' information is inherited from the Activity

view details

James Coglan

commit sha 5d771ec8eaae97569c176863ffc982d391954670

Validate the activity as part of a bulk transaction upload This is the first piece of validation that ImportTransactions does, so this also sets up a bunch of machinery to support error reporting. Errors report the row offset, column name, cell value, and error message, so errors can be easily located in a spreadsheet. Because this class doesn't assume CSV input per se but takes an array of hashes, the row offset is zero-based. It can be adjusted if the caller knows more about the input format. This first validation checks that the `Activity RODA Identifier` field refers to an existing activity, otherwise an error is produced and the transaction is not imported.

view details

James Coglan

commit sha 3ff7786d8e2840d528ddf593ee65610fe076fc71

Validate dates on bulk-uploaded transactions Here we introduce support for reporting errors that result from conversions from the data format supported by the bulk upload CSV service. The conversion methods can raise exceptions, which will be caught and stored on the ImportTransactions instance. The first such validation is that Date must parse as ISO-8601, according to the Ruby implementation of this format. The value reported in the error is the original value from the imported row, not the value after any conversion is applied.

view details

James Coglan

commit sha 896a4237f5121939df084fbb685362a7ca41dc12

Refactor error handling in ImportTransactions The `import_row` method is getting fairly complex, so here we break it into smaller pieces, in particular not mixing error handling in with the rest of the logic quite so much. For each row, errors are collected in a hash, which can be updated incrementally or import all the errors from another object, as in errors.update(converter.errors) At the end of `import_row` we convert the hash's contents into `Error` objects with row numbers attached.

view details

James Coglan

commit sha acb2ee4e8416419aed8f74dc9a73a4b0c1cf2930

Validate dates at the model layer in ImportTransactions If the `Date` field of an imported row is empty, we return `nil` from the conversion function rather than an empty string. This triggers a more meaningful validation in the model. Our imported should only validate the _formatting_ of the date, not its content -- any semantic validation should be done by the model, this class is merely a parser/translator. To effect this, the result of `CreateTransaction` is extracted and we copy any ActiveRecord validation errors into our own error set, using the attribute name to fetch the raw value from the input.

view details

James Coglan

commit sha b56dcf9ba11ddca9436ac6e129ec47554e8bb0f5

Test validation of values during transaction bulk upload Having integrated model-layer validation into ImportTransactions in the previous commit, we add tests that the `Value` field is correctly interpreted. The logic for converting these values is already handled by the model.

view details

James Coglan

commit sha 5605cbb903b615b4b47b4ec6d5db3ca53450c36e

Validate the receiving organisation during bulk transaction upload The transaction bulk upload CSV schema contains three fields about the receiving organisation: - `Receiving Organisation Name` - `Receiving Organisation Type` - `Receiving Organisation IATI Reference` The former two are required, and the Type must be one of the codes in the IATI Organisation Type codelist: https://iatistandard.org/en/iati-standard/203/codelists/organisationtype/ We validate this using our existing codelist file.

view details

James Coglan

commit sha 5ea9fa486db5ee7f03347ff6e1cacad34b5a987d

Test that IATI references are saved during bulk transaction upload Here we're just adding a test for the `Receiving Organisation IATI Reference` field's behaviour, after dealing with the other Receiving Organisation fields in the previous commit. This field is not validated, we just want to check that it gets saved.

view details

James Coglan

commit sha 5cdb600433efec79f0c1a681b0921d22b751a7f4

Validate the disbursement type during bulk transaction upload Much like the `Receiving Organisation Type` field in the CSV upload, `Disbursement Type` must be a valid code from the following IATI codelist: https://iatistandard.org/en/iati-standard/203/codelists/disbursementchannel/ Again, we can use our existing copies of these codelists to validate the provided value.

view details

James Coglan

commit sha 27087c6e040b39ee806881c7362a972a946a7b09

Refactor codelist validations in ImportTransactions::Converter We know have two very similar methods in this class that take a value and check it against an IATI codelist. By factoring our the name of the codelist we can reuse more of the logic and remove a little duplication here.

view details

James Coglan

commit sha b9d3af4fbb62c8ef9bdb4c5ee8ccbce99c23b555

Refactor CreatePlannedDisbursement#report and CreateTransaction#report These two classes both infer the currently active report that a planned disbursement or transaction should be added to. During bulk transaction upload, this report will be provided explicitly, because we'll launch the upload process from the page for a specific report. When ImportTransactions is calling CreateTransaction, I would rather pass the Report in, rather than have CreateTransaction infer the current Report and possibly disagree with the caller. Therefore I've factored the logic shared by these two classes into `Report#editable_for_activity`, and made both classes call this if they're not given a Report explicitly. The one difference introduced here is that the previous implementation looked only for `active` reports. We actually want to find any report the delivery partner can currently edit, which means it may be `active` or `awaiting_changes`.

view details

James Coglan

commit sha 70fb4fb95a6034f9b403c0cb681fdd702fcdf47f

Pass Report explicitly from ImportTransactions to CreateTransaction The plan is that the bulk transaction upload process will be initiated from a report's page, and so we'll pass the report in question into the ImportTransactions class so it knows which report to add transactions to. We do this by passing a Report into ImportTransactions, and forwarding this on to CreateTransaction following the refactor in the previous commit. This is not just a correctness improvement (making sure all transactions are added to the same report without needing to lock anything), it's a performance improvement too. Without this, each transaction added via CreateTransaction would trigger a new query to look the Report for its Activity up. All of these should result in the same Report record, so doing this for every row is wasteful.

view details

James Coglan

commit sha 1b11710e69cdd8ad1df72c350a7acb22a6b93611

Generate a default description for bulk-uploaded transactions In bulk upload, we want the `Description` field to be optional, whereas in the data model it's mandatory. If `Description` is blank in the CSV, then we'll generate a default description for the transaction based on the name of the activity and the description of the report.

view details

James Coglan

commit sha 6dd411cea771edfe6557990ba3531940fb0c50aa

Authorise the user for each bulk upload transaction In normal controller actions we only need to authorise the user for a single resource/action. In bulk upload, the uploaded CSV might name any activity, so we need to make sure the current user is allowed to add data for that activity, and that the activity is under the scope of the current report. To check the uploader is authorised to create this transaction, I'm checking ActivityPolicy#create?, which is the same thing that Staff::TransactionsController#create checks. To check the activity is part of the current report, I'm checking that their organisation and fund match, which is what would cause `Report.editable_for_activity` to link the two objects.

view details

James Coglan

commit sha 526a833c5472a3ca0ad30d3e479ff6cf0b872034

Refactor `import_row` to a RowImporter object We had built up a lot of logic in the methods of ImportTransactions that were passing the same objects between each other. This suggested pulling the whole process of importing a row into a distinct object, so that the various objects involved can live in its instance variables and not have to be passed around explicitly. This looks like a large change but it's not meant to change any logic. It just moves a set of logic into a new class while replacing function parameters with instance variables.

view details

James Coglan

commit sha cc6ec2d5d35ffc2656a99586571c7be5207ef9b6

Import multiple transactions in ImportTransactions All the commits leading up to this have focussed on the logic for importing a single row: translating CSV columns to fields in the database, parsing and validating inputs, and so on. We now implement support for importing multiple rows at the same time. For ease of use, we want a CSV upload to be atomic: either all rows are imported, or none are. Partial success would mean the user being told which rows succeeded or failed, then retrying with a new spreadsheet of just the failed rows. If one failure leads to nothing being imported, the user just needs to amend their existing CSV and try the whole thing again. We acheive this by wrapping the import loop in a database transaction, and rolling it back if any of the rows fail to validate or save.

view details

James Coglan

commit sha bd455d9bd5fa909ec150e481e3fd92c7c3b48a56

Refactor Activity.projects_and_third_party_projects_for_report The implementation of this method would be problematic if used for a bulk operation involving lots of activities under the same report. It loads all the level C/D Activity records for a report's Organisation, and then further filters them by checking whether the Activity's `associated_fund` matches that of the Report. `Activity#associated_fund` requires 2 or 3 further queries to look up the fund under which the level C/D activity sites. So if the initial query returned 500 activities, this would add an additional 1,000-1,500 queries to load the fund for each result, which will often be the same object. Instead, we invert this as follows. Given the fund ID from the Report, we can get the IDs of all programmes (level B) under this fund. From these we can get the IDs of all projects (level C) under those programmes. From these two lists of IDs, we can load all the level C and D activities that have one of them as a parent. The query ends up looking like this? SELECT * FROM activities WHERE organisation_id = ? AND (level = 'project' AND parent_id IN (<programme_ids>)) OR (level = 'third_party_project' AND parent_id IN (<project_ids>)) This reduces the lookup for all relevant activities to three queries.

view details

James Coglan

commit sha 941349e1a987e427533311e4f5247840ff493e4e

Include organisations when loading report variances While writing tests for bulk transaction uploads, accessing this page cause a test to fail because of Bullet complaining about an unoptimised N+1 query. This was its suggestion for fixing the problem.

view details

James Coglan

commit sha 50c1d4755b326d5a87c77cc04ff021f9172cda14

Download a CSV template to begin bulk transaction upload To initiate the bulk upload process, a user downloads a CSV that contains the column headers we require in ImportTransactions. It lists out the RODA Identifier for each activity that should appear in the current report, meaning all the level C and D activities for the report's organisation and fund, that have a RODA Identifier. (The Delivery Partner Identifier and Name are also included in the download for informational purposes and are ignored on upload.) The tests include a third project that belongs to the same organisation, but to a different fund from the other two, to make sure it's not included in the download for this report.

view details

James Coglan

commit sha ca47d7c5a00b2a321f96a267c7cc2b7f60c0bc53

Upload transactions in bulk as CSV This exposes the functionality of ImportTransactions via a file upload that accepts CSV. The CSV is parsed in the controller and the resulting rows fed into the importer, which itself is thoroughly unit-tested. If any errors occur, then no transactions are imported, and the errors are displayed to the user. If all the rows are valid, then the transactions are saved and a success message is displayed.

view details

push time in a month

push eventUKGovernmentBEIS/beis-report-official-development-assistance

James Coglan

commit sha e648afc8007da20676b43556761d7655200bb5a1

Import a single transaction from a CSV This begins the work of importing transactions in bulk from a CSV input. Most of the machinery will like in the ImportTransactions service, which takes an array of hashes rather than a CSV file directly, to make it easier to test. The initial implementation maps all the bulk import columns to their places in the database, but does not do any additional validation or conversion beyond that performed by the model. It does provide some defaults for mandatory fields: - The currency is GBP - The transaction type is 'Disbursement' - The 'providing organisation' information is inherited from the Activity

view details

James Coglan

commit sha 5d771ec8eaae97569c176863ffc982d391954670

Validate the activity as part of a bulk transaction upload This is the first piece of validation that ImportTransactions does, so this also sets up a bunch of machinery to support error reporting. Errors report the row offset, column name, cell value, and error message, so errors can be easily located in a spreadsheet. Because this class doesn't assume CSV input per se but takes an array of hashes, the row offset is zero-based. It can be adjusted if the caller knows more about the input format. This first validation checks that the `Activity RODA Identifier` field refers to an existing activity, otherwise an error is produced and the transaction is not imported.

view details

James Coglan

commit sha 3ff7786d8e2840d528ddf593ee65610fe076fc71

Validate dates on bulk-uploaded transactions Here we introduce support for reporting errors that result from conversions from the data format supported by the bulk upload CSV service. The conversion methods can raise exceptions, which will be caught and stored on the ImportTransactions instance. The first such validation is that Date must parse as ISO-8601, according to the Ruby implementation of this format. The value reported in the error is the original value from the imported row, not the value after any conversion is applied.

view details

James Coglan

commit sha 896a4237f5121939df084fbb685362a7ca41dc12

Refactor error handling in ImportTransactions The `import_row` method is getting fairly complex, so here we break it into smaller pieces, in particular not mixing error handling in with the rest of the logic quite so much. For each row, errors are collected in a hash, which can be updated incrementally or import all the errors from another object, as in errors.update(converter.errors) At the end of `import_row` we convert the hash's contents into `Error` objects with row numbers attached.

view details

James Coglan

commit sha acb2ee4e8416419aed8f74dc9a73a4b0c1cf2930

Validate dates at the model layer in ImportTransactions If the `Date` field of an imported row is empty, we return `nil` from the conversion function rather than an empty string. This triggers a more meaningful validation in the model. Our imported should only validate the _formatting_ of the date, not its content -- any semantic validation should be done by the model, this class is merely a parser/translator. To effect this, the result of `CreateTransaction` is extracted and we copy any ActiveRecord validation errors into our own error set, using the attribute name to fetch the raw value from the input.

view details

James Coglan

commit sha b56dcf9ba11ddca9436ac6e129ec47554e8bb0f5

Test validation of values during transaction bulk upload Having integrated model-layer validation into ImportTransactions in the previous commit, we add tests that the `Value` field is correctly interpreted. The logic for converting these values is already handled by the model.

view details

James Coglan

commit sha 5605cbb903b615b4b47b4ec6d5db3ca53450c36e

Validate the receiving organisation during bulk transaction upload The transaction bulk upload CSV schema contains three fields about the receiving organisation: - `Receiving Organisation Name` - `Receiving Organisation Type` - `Receiving Organisation IATI Reference` The former two are required, and the Type must be one of the codes in the IATI Organisation Type codelist: https://iatistandard.org/en/iati-standard/203/codelists/organisationtype/ We validate this using our existing codelist file.

view details

James Coglan

commit sha 5ea9fa486db5ee7f03347ff6e1cacad34b5a987d

Test that IATI references are saved during bulk transaction upload Here we're just adding a test for the `Receiving Organisation IATI Reference` field's behaviour, after dealing with the other Receiving Organisation fields in the previous commit. This field is not validated, we just want to check that it gets saved.

view details

James Coglan

commit sha 5cdb600433efec79f0c1a681b0921d22b751a7f4

Validate the disbursement type during bulk transaction upload Much like the `Receiving Organisation Type` field in the CSV upload, `Disbursement Type` must be a valid code from the following IATI codelist: https://iatistandard.org/en/iati-standard/203/codelists/disbursementchannel/ Again, we can use our existing copies of these codelists to validate the provided value.

view details

James Coglan

commit sha 27087c6e040b39ee806881c7362a972a946a7b09

Refactor codelist validations in ImportTransactions::Converter We know have two very similar methods in this class that take a value and check it against an IATI codelist. By factoring our the name of the codelist we can reuse more of the logic and remove a little duplication here.

view details

James Coglan

commit sha b9d3af4fbb62c8ef9bdb4c5ee8ccbce99c23b555

Refactor CreatePlannedDisbursement#report and CreateTransaction#report These two classes both infer the currently active report that a planned disbursement or transaction should be added to. During bulk transaction upload, this report will be provided explicitly, because we'll launch the upload process from the page for a specific report. When ImportTransactions is calling CreateTransaction, I would rather pass the Report in, rather than have CreateTransaction infer the current Report and possibly disagree with the caller. Therefore I've factored the logic shared by these two classes into `Report#editable_for_activity`, and made both classes call this if they're not given a Report explicitly. The one difference introduced here is that the previous implementation looked only for `active` reports. We actually want to find any report the delivery partner can currently edit, which means it may be `active` or `awaiting_changes`.

view details

James Coglan

commit sha 70fb4fb95a6034f9b403c0cb681fdd702fcdf47f

Pass Report explicitly from ImportTransactions to CreateTransaction The plan is that the bulk transaction upload process will be initiated from a report's page, and so we'll pass the report in question into the ImportTransactions class so it knows which report to add transactions to. We do this by passing a Report into ImportTransactions, and forwarding this on to CreateTransaction following the refactor in the previous commit. This is not just a correctness improvement (making sure all transactions are added to the same report without needing to lock anything), it's a performance improvement too. Without this, each transaction added via CreateTransaction would trigger a new query to look the Report for its Activity up. All of these should result in the same Report record, so doing this for every row is wasteful.

view details

James Coglan

commit sha 1b11710e69cdd8ad1df72c350a7acb22a6b93611

Generate a default description for bulk-uploaded transactions In bulk upload, we want the `Description` field to be optional, whereas in the data model it's mandatory. If `Description` is blank in the CSV, then we'll generate a default description for the transaction based on the name of the activity and the description of the report.

view details

James Coglan

commit sha 6dd411cea771edfe6557990ba3531940fb0c50aa

Authorise the user for each bulk upload transaction In normal controller actions we only need to authorise the user for a single resource/action. In bulk upload, the uploaded CSV might name any activity, so we need to make sure the current user is allowed to add data for that activity, and that the activity is under the scope of the current report. To check the uploader is authorised to create this transaction, I'm checking ActivityPolicy#create?, which is the same thing that Staff::TransactionsController#create checks. To check the activity is part of the current report, I'm checking that their organisation and fund match, which is what would cause `Report.editable_for_activity` to link the two objects.

view details

James Coglan

commit sha 526a833c5472a3ca0ad30d3e479ff6cf0b872034

Refactor `import_row` to a RowImporter object We had built up a lot of logic in the methods of ImportTransactions that were passing the same objects between each other. This suggested pulling the whole process of importing a row into a distinct object, so that the various objects involved can live in its instance variables and not have to be passed around explicitly. This looks like a large change but it's not meant to change any logic. It just moves a set of logic into a new class while replacing function parameters with instance variables.

view details

James Coglan

commit sha cc6ec2d5d35ffc2656a99586571c7be5207ef9b6

Import multiple transactions in ImportTransactions All the commits leading up to this have focussed on the logic for importing a single row: translating CSV columns to fields in the database, parsing and validating inputs, and so on. We now implement support for importing multiple rows at the same time. For ease of use, we want a CSV upload to be atomic: either all rows are imported, or none are. Partial success would mean the user being told which rows succeeded or failed, then retrying with a new spreadsheet of just the failed rows. If one failure leads to nothing being imported, the user just needs to amend their existing CSV and try the whole thing again. We acheive this by wrapping the import loop in a database transaction, and rolling it back if any of the rows fail to validate or save.

view details

James Coglan

commit sha bd455d9bd5fa909ec150e481e3fd92c7c3b48a56

Refactor Activity.projects_and_third_party_projects_for_report The implementation of this method would be problematic if used for a bulk operation involving lots of activities under the same report. It loads all the level C/D Activity records for a report's Organisation, and then further filters them by checking whether the Activity's `associated_fund` matches that of the Report. `Activity#associated_fund` requires 2 or 3 further queries to look up the fund under which the level C/D activity sites. So if the initial query returned 500 activities, this would add an additional 1,000-1,500 queries to load the fund for each result, which will often be the same object. Instead, we invert this as follows. Given the fund ID from the Report, we can get the IDs of all programmes (level B) under this fund. From these we can get the IDs of all projects (level C) under those programmes. From these two lists of IDs, we can load all the level C and D activities that have one of them as a parent. The query ends up looking like this? SELECT * FROM activities WHERE organisation_id = ? AND (level = 'project' AND parent_id IN (<programme_ids>)) OR (level = 'third_party_project' AND parent_id IN (<project_ids>)) This reduces the lookup for all relevant activities to three queries.

view details

James Coglan

commit sha 941349e1a987e427533311e4f5247840ff493e4e

Include organisations when loading report variances While writing tests for bulk transaction uploads, accessing this page cause a test to fail because of Bullet complaining about an unoptimised N+1 query. This was its suggestion for fixing the problem.

view details

James Coglan

commit sha 50c1d4755b326d5a87c77cc04ff021f9172cda14

Download a CSV template to begin bulk transaction upload To initiate the bulk upload process, a user downloads a CSV that contains the column headers we require in ImportTransactions. It lists out the RODA Identifier for each activity that should appear in the current report, meaning all the level C and D activities for the report's organisation and fund, that have a RODA Identifier. (The Delivery Partner Identifier and Name are also included in the download for informational purposes and are ignored on upload.) The tests include a third project that belongs to the same organisation, but to a different fund from the other two, to make sure it's not included in the download for this report.

view details

James Coglan

commit sha ca47d7c5a00b2a321f96a267c7cc2b7f60c0bc53

Upload transactions in bulk as CSV This exposes the functionality of ImportTransactions via a file upload that accepts CSV. The CSV is parsed in the controller and the resulting rows fed into the importer, which itself is thoroughly unit-tested. If any errors occur, then no transactions are imported, and the errors are displayed to the user. If all the rows are valid, then the transactions are saved and a success message is displayed.

view details

push time in a month

delete branch UKGovernmentBEIS/beis-report-official-development-assistance

delete branch : feature/876-upload-transactions-as-csv

delete time in a month

PR merged UKGovernmentBEIS/beis-report-official-development-assistance

(876) Upload transactions as CSV

Changes in this PR

This allows delivery partner users to upload transactions for a report by uploading a CSV. They work by filling out a template we give them, which contains the required column headings, and the IDs of all the activities they need to report on. The RODA ID is used to identify activities on upload, while the Delivery Partner ID and Name are there for the user's information and are ignored on upload.

This is quite a large PR but the code is mostly fairly boring validation and translation of CSV fields onto database columns, plus a little glue at the end to expose the functionality via the service. I'd recommend reading each commit rather than all the changes at once.

I'd like to check that all the logic for validation and translation the fields is correct, that the right associations between transactions, activities and reports are created, and that any permissions-related checks are correct.

Screenshots of UI changes

When viewing a report, a new button is available in the menu to "Upload actuals":

Screenshot 2020-09-17 at 09 56 58

This links to a page where the user can download a CSV template, and upload their finished spreadsheet:

Screenshot 2020-09-17 at 09 58 16

The download gives them a CSV with the required column headings, and the IDs of each activity they need to provide actuals for in this report.

Screenshot 2020-09-17 at 09 57 52

The user fills this spreadsheet out and re-exports as CSV. If what they upload contains any errors, then the transactions are not created, and the errors are displayed.

Screenshot 2020-09-17 at 09 58 33

If they don't upload a file at all, an error message is shown:

Screenshot 2020-09-17 at 14 52 35

If all the data is valid and the transactions are saved, a success message is shown.

Screenshot 2020-09-17 at 14 54 24

Next steps

  • [ ] Is an ADR required? An ADR should be added if this PR introduces a change to the architecture.
  • [x] Is a changelog entry required? An entry should always be made in CHANGELOG.md, unless this PR is a small tweak which has no impact outside the development team.
  • [ ] Do any environment variables need amending or adding?
  • [ ] Have any changes to the XML been checked with the IATI validator? See XML Validation
+994 -22

3 comments

18 changed files

jcoglan

pr closed time in a month

PR opened UKGovernmentBEIS/beis-report-official-development-assistance

(998) Display the parent activity's RODA Identifier

When entering a RODA Identifier, there's nothing to tell the user where they are, and the data they enter here is final, so it's both easy and very costly to make a mistake and enter the wrong thing.

As a quick improvement, we can add the parent activity's RODA identifier to the page, if the activity has a parent. You can only enter a RODA ID if the parent activity has one completed, so this is guaranteed to always have a value to display.

Screenshots of UI changes

Screenshot 2020-09-17 at 16 18 11

Next steps

  • [ ] Is an ADR required? An ADR should be added if this PR introduces a change to the architecture.
  • [ ] Is a changelog entry required? An entry should always be made in CHANGELOG.md, unless this PR is a small tweak which has no impact outside the development team.
  • [ ] Do any environment variables need amending or adding?
  • [ ] Have any changes to the XML been checked with the IATI validator? See XML Validation
+9 -2

0 comment

3 changed files

pr created time in a month

pull request commentUKGovernmentBEIS/beis-report-official-development-assistance

(876) Upload transactions as CSV

@mec don't feel bad :) You've already caught some things I missed, any feedback is useful!

Also, if any of the team don't understand something I've done, I'd rather make it clearer than just leave something unmaintainable for others, so do shout if you have questions.

jcoglan

comment created time in a month

Pull request review commentUKGovernmentBEIS/beis-report-official-development-assistance

(876) Upload transactions as CSV

+# frozen_string_literal: true++require "csv"++class Staff::TransactionUploadsController < Staff::BaseController+  include Secured++  before_action :authorize_report++  def new+    @report_presenter = ReportPresenter.new(@report)+  end++  def show+    response.headers["Content-Type"] = "text/csv"+    response.headers["Content-Disposition"] = "attachment; filename=transactions.csv"++    csv = CSV.generate { |table|

Cool, I've amended that, thank you :)

jcoglan

comment created time in a month

PullRequestReviewEvent

push eventUKGovernmentBEIS/beis-report-official-development-assistance

James Coglan

commit sha 50c1d4755b326d5a87c77cc04ff021f9172cda14

Download a CSV template to begin bulk transaction upload To initiate the bulk upload process, a user downloads a CSV that contains the column headers we require in ImportTransactions. It lists out the RODA Identifier for each activity that should appear in the current report, meaning all the level C and D activities for the report's organisation and fund, that have a RODA Identifier. (The Delivery Partner Identifier and Name are also included in the download for informational purposes and are ignored on upload.) The tests include a third project that belongs to the same organisation, but to a different fund from the other two, to make sure it's not included in the download for this report.

view details

James Coglan

commit sha ca47d7c5a00b2a321f96a267c7cc2b7f60c0bc53

Upload transactions in bulk as CSV This exposes the functionality of ImportTransactions via a file upload that accepts CSV. The CSV is parsed in the controller and the resulting rows fed into the importer, which itself is thoroughly unit-tested. If any errors occur, then no transactions are imported, and the errors are displayed to the user. If all the rows are valid, then the transactions are saved and a success message is displayed.

view details

James Coglan

commit sha 62a06d775838b657784562251b6a9b82e388ff11

Changelog: Upload transaction data in bulk as CSV

view details

push time in a month

more