faye/faye 4307
Simple pub/sub messaging for the web
Standards-compliant WebSocket client and server
Standards-compliant WebSocket client and server
faye/websocket-driver-ruby 185
WebSocket protocol handler with pluggable I/O
WebSocket protocol handler with pluggable I/O
Redis engine backend for Faye
Redis engine backend for Faye
faye/websocket-extensions-node 48
Generic extension management for WebSocket connections
faye/websocket-extensions-ruby 22
Generic extension management for WebSocket connections
faye/permessage-deflate-node 19
Per-message DEFLATE compression extension for WebSocket connections
pull request commentUKGovernmentBEIS/beis-report-official-development-assistance
(1066) Users can see a list of invalid activities in the index page
@mec My commit message explains why we were observing weird effects from Bullet in terms of what it's doing behind the scenes. I discussed this with @MariluzHerrera and my decision went as follows.
Using Bullet.add_whitelist in both the environment config and in individual tests is not advisable since it can lead to conflicts, where a test turns a particular whitelisted thing on and off, changing the base state of the app's config, and those are hard to debug, as we've seen here. The Bullet.{add,delete}_whitelist API does not seem designed for this type of use as it does not provide a reliable way of restoring whatever the prior state was. We need to pick one or the other.
Putting it in individual tests means people would need to add Bullet.{add,delete}_whitelist config to any test that hits a problematic controller action. This particular action is hit by a lot of tests, and it won't be obvious that one needs to add this config in order to silence the warnings Bullet produces, or even if that is a good idea. Getting warnings about code you didn't write is confusing.
Putting it in the environment config means we're globally ignoring a certain kind of optimisation problem -- you cannot. In general, I would rather be alerted to these, but I think ignoring an unused eager load is much less dangerous than ignoring an N+1 query, so I'm happy with the trade-off. In this case, the unused eager load comes from the fact that not all Activity objects use the parent association during validation -- fund activities don't require a parent. The fact that some but not all loaded activities use the parent association causes Bullet to raise an error whether you eager load the association or not -- it's impossible to make this code pass Bullet inspection.
I'm open to making the opposite decision and putting Bullet whitelist logic in individual tests, and not in the environment config, I just think we need to pick one or the other.
comment created time in 6 days
issue commentfaye/faye
Unexpected error while processing request: uninitialized constant EventMachine::Deferrable::Rails
None of the lines in this backtrace refer to the constant Rails, but a lot of them are concerned with running callbacks that will yield to user-supplied code. Your Faye server setup code does not appear to reference anything from Rails, or load application code, and does not set up any event listeners or extensions:
require "faye"
faye_server = Faye::RackAdapter.new(:mount => '/faye', :timeout => 45)
run faye_server
The one problem I can see with this is that it sets the timeout to 45, which is greater than the Thin request timeout limit of 30. This may cause weird results.
What command are you using to start the server in faye.ru?
comment created time in 6 days
issue commentfaye/faye
uninitialized constant EventMachine::Rails (NameError)
Neither EventMachine nor Faye refers to Rails anywhere in their codebases. Have you bound any event listeners to Faye that depend on anything from Rails or your application?
comment created time in 6 days
issue commentfaye/faye
Faye requesting an insecure script <URL>
What's the code you're using to start the connection to the Faye server?
comment created time in 7 days
push eventjcoglan/dotfiles
commit sha ba910a185eb39cf23b4559c3eb5365c030ba8cab
Function for installing ruby with [email protected]
push time in 8 days
push eventjcoglan/dotfiles
commit sha b99b9659f79f12dd3c3ce451bd340dd914ea263d
Run all the shell/scripts files through shellcheck
push time in 11 days
push eventjcoglan/dotfiles
commit sha c1d9b2bac90430b1ddd4e4420ce2ea0033fc56fe
Refactor Homebrew functions to expose packages, dependencies, and the ability to check installed packages against arbitrary Brewfiles
push time in 11 days
push eventUKGovernmentBEIS/beis-report-official-development-assistance
commit sha c724bfc5608c757e7470ed181dd14632a52628f7
Bump data_migrate from 6.3.0 to 6.5.0 Bumps [data_migrate](https://github.com/ajvargo/data-migrate) from 6.3.0 to 6.5.0. - [Release notes](https://github.com/ajvargo/data-migrate/releases) - [Changelog](https://github.com/ilyakatz/data-migrate/blob/master/Changelog.md) - [Commits](https://github.com/ajvargo/data-migrate/commits) Signed-off-by: dependabot-preview[bot] <[email protected]>
commit sha 0a4c2d3cec7e999dce9fbc94622c1f2bd2f4aa23
Merge pull request #687 from UKGovernmentBEIS/dependabot/bundler/data_migrate-6.5.0 Bump data_migrate from 6.3.0 to 6.5.0
push time in 12 days
delete branch UKGovernmentBEIS/beis-report-official-development-assistance
delete branch : dependabot/bundler/data_migrate-6.5.0
delete time in 12 days
PR merged UKGovernmentBEIS/beis-report-official-development-assistance
Bumps data_migrate from 6.3.0 to 6.5.0. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/ilyakatz/data-migrate/blob/master/Changelog.md">data_migrate's changelog</a>.</em></p> <blockquote> <h1>Changelog</h1> <h2>6.4.0</h2> <p>Add primary key to data_migrations table <a href="https://github.com/aandis">aandis</a></p> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li>See full diff in <a href="https://github.com/ajvargo/data-migrate/commits">compare view</a></li> </ul> </details> <br />
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
<details> <summary>Dependabot commands and options</summary> <br />
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot mergewill merge this PR after your CI passes on it@dependabot squash and mergewill squash and merge this PR after your CI passes on it@dependabot cancel mergewill cancel a previously requested merge and block automerging@dependabot reopenwill reopen this PR if it is closed@dependabot closewill close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually@dependabot ignore this major versionwill close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this minor versionwill close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this dependencywill close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)@dependabot use these labelswill set the current labels as the default for future PRs for this repo and language@dependabot use these reviewerswill set the current reviewers as the default for future PRs for this repo and language@dependabot use these assigneeswill set the current assignees as the default for future PRs for this repo and language@dependabot use this milestonewill set the current milestone as the default for future PRs for this repo and language@dependabot badge mewill comment on this PR with code to add a "Dependabot enabled" badge to your readme
Additionally, you can set the following in your Dependabot dashboard:
- Update frequency (including time of day and day of week)
- Pull request limits (per update run and/or open at any time)
- Automerge options (never/patch/minor, and dev/runtime dependencies)
- Out-of-range updates (receive only lockfile updates, if desired)
- Security updates (receive only security updates, if desired)
</details>
pr closed time in 12 days
pull request commentUKGovernmentBEIS/beis-report-official-development-assistance
Bump data_migrate from 6.3.0 to 6.5.0
@dependabot rebase
comment created time in 12 days
push eventUKGovernmentBEIS/beis-report-official-development-assistance
commit sha 5836c59cd63838239bc541e69ab3357336c9046b
Bump auth0 from 4.15.0 to 4.16.0 Bumps [auth0](https://github.com/auth0/ruby-auth0) from 4.15.0 to 4.16.0. - [Release notes](https://github.com/auth0/ruby-auth0/releases) - [Changelog](https://github.com/auth0/ruby-auth0/blob/master/CHANGELOG.md) - [Commits](https://github.com/auth0/ruby-auth0/compare/v4.15.0...v4.16.0) Signed-off-by: dependabot-preview[bot] <[email protected]>
commit sha 1d812c2c27e93e8b784d58f846673005a4a75179
Merge pull request #686 from UKGovernmentBEIS/dependabot/bundler/auth0-4.16.0 Bump auth0 from 4.15.0 to 4.16.0
push time in 12 days
delete branch UKGovernmentBEIS/beis-report-official-development-assistance
delete branch : dependabot/bundler/auth0-4.16.0
delete time in 12 days
PR merged UKGovernmentBEIS/beis-report-official-development-assistance
Bumps auth0 from 4.15.0 to 4.16.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/auth0/ruby-auth0/releases">auth0's releases</a>.</em></p> <blockquote> <h2>v4.16.0</h2> <h2><a href="https://github.com/auth0/ruby-auth0/tree/v4.16.0">v4.16.0</a> (2020-10-02)</h2> <p><a href="https://github.com/auth0/ruby-auth0/compare/v4.15.0...v4.16.0">Full Changelog</a></p> <p><strong>Added</strong></p> <ul> <li>New Email Verification Fields <a href="https://github-redirect.dependabot.com/auth0/ruby-auth0/pull/237">#237</a> (<a href="https://github.com/davidpatrick">davidpatrick</a>)</li> </ul> <p><strong>Security</strong></p> <ul> <li>Bump actionview from 6.0.3.2 to 6.0.3.3 <a href="https://github-redirect.dependabot.com/auth0/ruby-auth0/pull/236">#236</a> (<a href="https://github.com/apps/dependabot">dependabot[bot]</a>)</li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/auth0/ruby-auth0/blob/master/CHANGELOG.md">auth0's changelog</a>.</em></p> <blockquote> <h2><a href="https://github.com/auth0/ruby-auth0/tree/v4.16.0">v4.16.0</a> (2020-10-02)</h2> <p><a href="https://github.com/auth0/ruby-auth0/compare/v4.15.0...v4.16.0">Full Changelog</a></p> <p><strong>Added</strong></p> <ul> <li>New Email Verification Fields <a href="https://github-redirect.dependabot.com/auth0/ruby-auth0/pull/237">#237</a> (<a href="https://github.com/davidpatrick">davidpatrick</a>)</li> </ul> <p><strong>Security</strong></p> <ul> <li>Bump actionview from 6.0.3.2 to 6.0.3.3 <a href="https://github-redirect.dependabot.com/auth0/ruby-auth0/pull/236">#236</a> (<a href="https://github.com/apps/dependabot">dependabot[bot]</a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/auth0/ruby-auth0/commit/54c8d67b323397ff9176076e88b7b9ac814c0dee"><code>54c8d67</code></a> Bump to 4.16.0</li> <li><a href="https://github.com/auth0/ruby-auth0/commit/392acd280afd2e572cce65d23086ff3d9033144c"><code>392acd2</code></a> Bump actionview from 6.0.3.2 to 6.0.3.3 (<a href="https://github-redirect.dependabot.com/auth0/ruby-auth0/issues/236">#236</a>)</li> <li><a href="https://github.com/auth0/ruby-auth0/commit/dc1a6b52f6c679db7dfdad3e84d74f46dbe721d7"><code>dc1a6b5</code></a> New Email Verification Fields (<a href="https://github-redirect.dependabot.com/auth0/ruby-auth0/issues/237">#237</a>)</li> <li><a href="https://github.com/auth0/ruby-auth0/commit/a7ffeb50a095041bc3263aeb0de87cc02594d85a"><code>a7ffeb5</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/auth0/ruby-auth0/issues/235">#235</a> from fossabot/add-license-scan-badge</li> <li><a href="https://github.com/auth0/ruby-auth0/commit/79b83f33aaeef83622fd63eaa30952963bc8665d"><code>79b83f3</code></a> Add license scan report and status</li> <li><a href="https://github.com/auth0/ruby-auth0/commit/94efce954ee914938294f866791f96011cc2a96b"><code>94efce9</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/auth0/ruby-auth0/issues/234">#234</a> from auth0/release-4.15.0</li> <li><a href="https://github.com/auth0/ruby-auth0/commit/8ee30e04dfed003c790d034d0827d9a49afae30f"><code>8ee30e0</code></a> Update CHANGELOG.md</li> <li><a href="https://github.com/auth0/ruby-auth0/commit/9ed9dd3b6b386907ef131a77b4120d17933ca8bb"><code>9ed9dd3</code></a> Update gems</li> <li>See full diff in <a href="https://github.com/auth0/ruby-auth0/compare/v4.15.0...v4.16.0">compare view</a></li> </ul> </details> <br />
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
<details> <summary>Dependabot commands and options</summary> <br />
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot mergewill merge this PR after your CI passes on it@dependabot squash and mergewill squash and merge this PR after your CI passes on it@dependabot cancel mergewill cancel a previously requested merge and block automerging@dependabot reopenwill reopen this PR if it is closed@dependabot closewill close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually@dependabot ignore this major versionwill close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this minor versionwill close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this dependencywill close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)@dependabot use these labelswill set the current labels as the default for future PRs for this repo and language@dependabot use these reviewerswill set the current reviewers as the default for future PRs for this repo and language@dependabot use these assigneeswill set the current assignees as the default for future PRs for this repo and language@dependabot use this milestonewill set the current milestone as the default for future PRs for this repo and language@dependabot badge mewill comment on this PR with code to add a "Dependabot enabled" badge to your readme
Additionally, you can set the following in your Dependabot dashboard:
- Update frequency (including time of day and day of week)
- Pull request limits (per update run and/or open at any time)
- Automerge options (never/patch/minor, and dev/runtime dependencies)
- Out-of-range updates (receive only lockfile updates, if desired)
- Security updates (receive only security updates, if desired)
</details>
pr closed time in 12 days
pull request commentUKGovernmentBEIS/beis-report-official-development-assistance
Bump auth0 from 4.15.0 to 4.16.0
@dependabot rebase
comment created time in 12 days
push eventUKGovernmentBEIS/beis-report-official-development-assistance
commit sha 26cc9e771b4429d321712c850b59993176a68f4f
Stop modifying the Bullet whitelist during tests A few of our tests have this configuration that seems to designed to make Bullet temporarily ignore certain optimisations: before :all do Bullet.add_whitelist(type: :unused_eager_loading, class_name: "Activity", association: :parent) end after :all do Bullet.delete_whitelist(type: :unused_eager_loading, class_name: "Activity", association: :parent) end This particular setup makes Bullet ignore unused eager loads of the `Activity#parent` association. The problem is that, if this assoication was already ignored before these tests ran, then the call to `delete_whitelist` will cause it _not_ to be ignored in any tests that run later. This is because of how Bullet stores its whitelist. It indexes associations by optimisation type and class name, so if we have this in the application init code: Bullet.add_whitelist(type: :unused_eager_loading, class_name: "Activity", association: :parent) Bullet.add_whitelist(type: :unused_eager_loading, class_name: "Activity", association: :organisation) Then `Bullet.whitelist` will contain: { :unused_eager_loading => { "Activity" => [:parent, :organisation] } } The association names are stored in an Array, not a Set, so it's possible for them to appear twice. When we run the `before` block above, `:parent` is added again to this list: { :unused_eager_loading => { "Activity" => [:parent, :organisation, :parent] } } When we run the `after` block, `delete_whitelist` removes _all_ occurrences of the given association from the whitelist, so the state ends up containing: { :unused_eager_loading => { "Activity" => [:organisation] } } The `:parent` association is no longer ignored and so all subsequent tests may trigger warnings about it. Because RSpec randomises test order, this means we'll see inconsistent results with Bullet causing different tests to fail each time. It doesn't look like Bullet provides a safe API for this use case of temporarily ignoring a certain warning type. `delete_whitelist` itself is undocumented so it's unclear we should even be using it, or what it's intended to do, but it doesn't provide the inverse of `add_whitelist` as it removes _all_ occurrences of the given association from the list. `add_whitelist` doesn't provide any indication that the association we've just added was already in the whitelist, and therefore we shouldn't remove it after the test. The safest thing to do is probably to only use `Bullet.add_whitelist` in the application config, and not in individual tests.
push time in 13 days
push eventUKGovernmentBEIS/beis-report-official-development-assistance
commit sha e6ea93dcf0b1ee2bdef03dc8b9db9a52b2cb382b
Downgrade better_errors to 2.8.1 There is a bug in better_errors 2.8.2, where it uses its VERSION constant without having loaded it first [1]. This is preventing our app from booting so I'm reverting this version bump until that issue is fixed. This reverts commit a6e2172a42abefaf4e2cf5f50762d0d52999def2, reversing changes made to ce4f953ada4ef1b01db0289f9a161bfe6b20706b. [1]: https://github.com/BetterErrors/better_errors/issues/483
commit sha 01448e08874b979f985c8a1257be487d4027ee1f
Merge pull request #685 from UKGovernmentBEIS/fix/downgrade-better-errors Downgrade better_errors to 2.8.1
push time in 18 days
delete branch UKGovernmentBEIS/beis-report-official-development-assistance
delete branch : fix/downgrade-better-errors
delete time in 18 days
PR merged UKGovernmentBEIS/beis-report-official-development-assistance
There is a bug in better_errors 2.8.2, where it uses its VERSION constant without having loaded it first 1. This is preventing our app from booting so I'm reverting this version bump until that issue is fixed.
This reverts commit a6e2172a42abefaf4e2cf5f50762d0d52999def2, reversing changes made to ce4f953ada4ef1b01db0289f9a161bfe6b20706b.
pr closed time in 18 days
issue commentBetterErrors/better_errors
uninitialized constant BetterErrors::Middleware::VERSION (NameError)
I believe this change is the source of the problem: referring to VERSION without having explicitly loaded that constant. It's stopping our Rails app from booting.
https://github.com/BetterErrors/better_errors/compare/v2.8.1..master#diff-5e3c1d57ea649185a86b730f951f5a18R43
comment created time in 18 days
issue commentBetterErrors/better_errors
I believe this change is the source of the problem: referring to VERSION without having explicitly loaded that constant. It's stopping our Rails app from booting.
https://github.com/BetterErrors/better_errors/compare/v2.8.1..master#diff-5e3c1d57ea649185a86b730f951f5a18R43
comment created time in 18 days
PR opened UKGovernmentBEIS/beis-report-official-development-assistance
There is a bug in better_errors 2.8.2, where it uses its VERSION constant without having loaded it first 1. This is preventing our app from booting so I'm reverting this version bump until that issue is fixed.
This reverts commit a6e2172a42abefaf4e2cf5f50762d0d52999def2, reversing changes made to ce4f953ada4ef1b01db0289f9a161bfe6b20706b.
pr created time in 18 days
create barnchUKGovernmentBEIS/beis-report-official-development-assistance
branch : fix/downgrade-better-errors
created branch time in 18 days
pull request commentUKGovernmentBEIS/beis-report-official-development-assistance
As mentioned in #655, I'd like to hold off on this until Puma 5.0 has been out a little longer.
comment created time in 18 days
delete branch UKGovernmentBEIS/beis-report-official-development-assistance
delete branch : dependabot/bundler/better_errors-2.8.2
delete time in 18 days
push eventUKGovernmentBEIS/beis-report-official-development-assistance
commit sha 7cd47bdd70d4a961f8ea1d9bebdd14ee97969ca2
Bump better_errors from 2.8.1 to 2.8.2 Bumps [better_errors](https://github.com/BetterErrors/better_errors) from 2.8.1 to 2.8.2. - [Release notes](https://github.com/BetterErrors/better_errors/releases) - [Commits](https://github.com/BetterErrors/better_errors/commits) Signed-off-by: dependabot-preview[bot] <[email protected]>
commit sha a6e2172a42abefaf4e2cf5f50762d0d52999def2
Merge pull request #682 from UKGovernmentBEIS/dependabot/bundler/better_errors-2.8.2 Bump better_errors from 2.8.1 to 2.8.2
push time in 18 days
PR merged UKGovernmentBEIS/beis-report-official-development-assistance
Bumps better_errors from 2.8.1 to 2.8.2. <details> <summary>Commits</summary> <ul> <li>See full diff in <a href="https://github.com/BetterErrors/better_errors/commits">compare view</a></li> </ul> </details> <br />
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
<details> <summary>Dependabot commands and options</summary> <br />
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot mergewill merge this PR after your CI passes on it@dependabot squash and mergewill squash and merge this PR after your CI passes on it@dependabot cancel mergewill cancel a previously requested merge and block automerging@dependabot reopenwill reopen this PR if it is closed@dependabot closewill close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually@dependabot ignore this major versionwill close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this minor versionwill close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this dependencywill close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)@dependabot use these labelswill set the current labels as the default for future PRs for this repo and language@dependabot use these reviewerswill set the current reviewers as the default for future PRs for this repo and language@dependabot use these assigneeswill set the current assignees as the default for future PRs for this repo and language@dependabot use this milestonewill set the current milestone as the default for future PRs for this repo and language@dependabot badge mewill comment on this PR with code to add a "Dependabot enabled" badge to your readme
Additionally, you can set the following in your Dependabot dashboard:
- Update frequency (including time of day and day of week)
- Pull request limits (per update run and/or open at any time)
- Automerge options (never/patch/minor, and dev/runtime dependencies)
- Out-of-range updates (receive only lockfile updates, if desired)
- Security updates (receive only security updates, if desired)
</details>
pr closed time in 18 days
pull request commentUKGovernmentBEIS/beis-report-official-development-assistance
Bump better_errors from 2.8.1 to 2.8.2
@dependabot rebase
comment created time in 18 days
push eventUKGovernmentBEIS/beis-report-official-development-assistance
commit sha 2eab06329564cced5275e7dfb57ea645eb903a5a
Bump brakeman from 4.9.1 to 4.10.0 Bumps [brakeman](https://github.com/presidentbeef/brakeman) from 4.9.1 to 4.10.0. - [Release notes](https://github.com/presidentbeef/brakeman/releases) - [Changelog](https://github.com/presidentbeef/brakeman/blob/main/CHANGES.md) - [Commits](https://github.com/presidentbeef/brakeman/compare/v4.9.1...v4.10.0) Signed-off-by: dependabot-preview[bot] <[email protected]>
commit sha ce4f953ada4ef1b01db0289f9a161bfe6b20706b
Merge pull request #677 from UKGovernmentBEIS/dependabot/bundler/brakeman-4.10.0 Bump brakeman from 4.9.1 to 4.10.0
push time in 18 days
delete branch UKGovernmentBEIS/beis-report-official-development-assistance
delete branch : dependabot/bundler/brakeman-4.10.0
delete time in 18 days
PR merged UKGovernmentBEIS/beis-report-official-development-assistance
Bumps brakeman from 4.9.1 to 4.10.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/presidentbeef/brakeman/releases">brakeman's releases</a>.</em></p> <blockquote> <h2>4.10.0</h2> <ul> <li>Add SARIF report format (<a href="https://github.com/swinton">Steve Winton</a>)</li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/presidentbeef/brakeman/blob/main/CHANGES.md">brakeman's changelog</a>.</em></p> <blockquote> <h1>4.10.0 - 2020-09-28</h1> <ul> <li>Add SARIF report format (Steve Winton)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/presidentbeef/brakeman/commit/8f696e3568523563f83e8af4512a30f3bda70866"><code>8f696e3</code></a> Bump to 4.10.0</li> <li><a href="https://github.com/presidentbeef/brakeman/commit/2beaac0ff9d9cfdd58d59e81b4b38f7720049587"><code>2beaac0</code></a> Update CHANGES</li> <li><a href="https://github.com/presidentbeef/brakeman/commit/5daa392b570f2aa5588ad1b983ca21cb5ebbbfbd"><code>5daa392</code></a> Add SARIF output format</li> <li><a href="https://github.com/presidentbeef/brakeman/commit/4cb7ad25839a30a28ac991f020c8353bca529310"><code>4cb7ad2</code></a> Fix permissions during gem build</li> <li>See full diff in <a href="https://github.com/presidentbeef/brakeman/compare/v4.9.1...v4.10.0">compare view</a></li> </ul> </details> <br />
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
<details> <summary>Dependabot commands and options</summary> <br />
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot mergewill merge this PR after your CI passes on it@dependabot squash and mergewill squash and merge this PR after your CI passes on it@dependabot cancel mergewill cancel a previously requested merge and block automerging@dependabot reopenwill reopen this PR if it is closed@dependabot closewill close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually@dependabot ignore this major versionwill close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this minor versionwill close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this dependencywill close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)@dependabot use these labelswill set the current labels as the default for future PRs for this repo and language@dependabot use these reviewerswill set the current reviewers as the default for future PRs for this repo and language@dependabot use these assigneeswill set the current assignees as the default for future PRs for this repo and language@dependabot use this milestonewill set the current milestone as the default for future PRs for this repo and language@dependabot badge mewill comment on this PR with code to add a "Dependabot enabled" badge to your readme
Additionally, you can set the following in your Dependabot dashboard:
- Update frequency (including time of day and day of week)
- Pull request limits (per update run and/or open at any time)
- Automerge options (never/patch/minor, and dev/runtime dependencies)
- Out-of-range updates (receive only lockfile updates, if desired)
- Security updates (receive only security updates, if desired)
</details>
pr closed time in 18 days
Pull request review commentUKGovernmentBEIS/beis-report-official-development-assistance
(1036) BEIS Users can download Programme (Level B) xml
def call(eager_load_parent: true) .includes(eager_load_associations) .order("created_at ASC") - programmes = if organisation.service_owner- programmes.all- else- programmes.where(extending_organisation_id: organisation.id)- end- programmes+ return programmes if organisation.service_owner++ programmes.where(extending_organisation_id: organisation.id).where(fund_query_condition(fund_id))
None of these snippets execute a query -- where does not trigger a database request on its own. The query is only executed when you attempted to read the result, e.g. by calling each or first. So you can build a query by incrementally adding where calls to a chain, and it effectively merges their conditions into a single query.
comment created time in a month
Pull request review commentUKGovernmentBEIS/beis-report-official-development-assistance
(1036) BEIS Users can download Programme (Level B) xml
def show @organisation_presenter = OrganisationPresenter.new(organisation) - @project_activities = FindProjectActivities.new(organisation: organisation, user: current_user).call(eager_load_parent: false).publishable_to_iati- @third_party_project_activities = FindThirdPartyProjectActivities.new(organisation: organisation, user: current_user).call(eager_load_parent: false).publishable_to_iati+ fund_ids_for_organisation_programmes = Activity.where(+ level: :programme,+ extending_organisation_id: organisation.id+ ).pluck(:parent_id)+ @funds = Activity.find(fund_ids_for_organisation_programmes)++ @project_activities = FindProjectActivities.new(+ organisation: organisation,+ user: current_user+ ).call(eager_load_parent: false).publishable_to_iati++ @third_party_project_activities = FindThirdPartyProjectActivities.new(+ organisation: organisation,+ user: current_user+ ).call(eager_load_parent: false).publishable_to_iati
These variables are well-named, and it's probably worth pulling these queries out into methods so the show method becomes a high-level description of what's being displayed, rather than the implementation details of fetching that data.
comment created time in a month
push eventUKGovernmentBEIS/beis-report-official-development-assistance
commit sha c4309d7e6929b48c79ec0a4c0ee55005c0e8dcc8
Show an error message if the user does not select a recipient country Previously we were not showing an error message if the user did not select a recipient country. If no country was selected, we were attempting to find the associated recipient region despite the country being blank. This was causing an error. How, if the country is blank, we assign it to the activity, thereby forcing a validation error to be shown. We do not try to associate the blank country value to a region.
commit sha 89e5fcce0ac5e8a0c70d795c00c63b4505518f45
Merge pull request #670 from UKGovernmentBEIS/fix/872-bug-no-error-message-shown-when-you-dont-select-a-country Show an error message if the user does not select a recipient country
push time in a month
delete branch UKGovernmentBEIS/beis-report-official-development-assistance
delete branch : fix/872-bug-no-error-message-shown-when-you-dont-select-a-country
delete time in a month
PR merged UKGovernmentBEIS/beis-report-official-development-assistance
This is a repeat of #667, which I removed from develop after messing up the history during a dependabot merge.
pr closed time in a month
Pull request review commentUKGovernmentBEIS/beis-report-official-development-assistance
(1036) BEIS Users can download Programme (Level B) xml
def call(eager_load_parent: true) .includes(eager_load_associations) .order("created_at ASC") - programmes = if organisation.service_owner- programmes.all- else- programmes.where(extending_organisation_id: organisation.id)- end- programmes+ return programmes if organisation.service_owner++ programmes.where(extending_organisation_id: organisation.id).where(fund_query_condition(fund_id))
If you're worried about this being too magical, another option would be to make the whole .where call conditional on having a fund_id, i.e.:
programmes = programmes.where(extending_organisation_id: organisation.id)
programmes = programmes.where(parent_id: fund_id) if fund_id.present?
programmes
Or, dynamically build the conditions, as in:
conditions = { extending_organisation_id: organisation.id }
conditions[:parent_id] = fund_id if fund_id.present?
programmes.where(conditions)
comment created time in a month
PR opened UKGovernmentBEIS/beis-report-official-development-assistance
This is a repeat of #667, which I removed from develop after messing up the history during a dependabot merge.
pr created time in a month
create barnchUKGovernmentBEIS/beis-report-official-development-assistance
branch : fix/872-bug-no-error-message-shown-when-you-dont-select-a-country
created branch time in a month
push eventUKGovernmentBEIS/beis-report-official-development-assistance
push time in a month
push eventUKGovernmentBEIS/beis-report-official-development-assistance
commit sha 6d77febc1f2cfc286f785a2012403e44dd32d1ab
Bump omniauth-auth0 from 2.3.1 to 2.4.0 Bumps [omniauth-auth0](https://github.com/auth0/omniauth-auth0) from 2.3.1 to 2.4.0. - [Release notes](https://github.com/auth0/omniauth-auth0/releases) - [Changelog](https://github.com/auth0/omniauth-auth0/blob/master/CHANGELOG.md) - [Commits](auth0/[email protected]) Signed-off-by: dependabot-preview[bot] <[email protected]>
commit sha a8b8fa88d463fc2ba6f2c4cf3b494974aafaee1f
Merge pull request #668 from UKGovernmentBEIS/dependabot/bundler/omniauth-auth0-2.4.0 Bump omniauth-auth0 from 2.3.1 to 2.4.0
push time in a month
delete branch UKGovernmentBEIS/beis-report-official-development-assistance
delete branch : dependabot/bundler/omniauth-auth0-2.4.0
delete time in a month
PR merged UKGovernmentBEIS/beis-report-official-development-assistance
Bumps omniauth-auth0 from 2.3.1 to 2.4.0.
pr closed time in a month
PR opened UKGovernmentBEIS/beis-report-official-development-assistance
Bumps omniauth-auth0 from 2.3.1 to 2.4.0.
pr created time in a month
create barnchUKGovernmentBEIS/beis-report-official-development-assistance
branch : dependabot/bundler/omniauth-auth0-2.4.0
created branch time in a month
push eventUKGovernmentBEIS/beis-report-official-development-assistance
push time in a month
push eventUKGovernmentBEIS/beis-report-official-development-assistance
commit sha 97ec43b642041a4c743e838c94b350a8eceb674e
Bump omniauth-auth0 from 2.3.1 to 2.4.0 Bumps [omniauth-auth0](https://github.com/auth0/omniauth-auth0) from 2.3.1 to 2.4.0. - [Release notes](https://github.com/auth0/omniauth-auth0/releases) - [Changelog](https://github.com/auth0/omniauth-auth0/blob/master/CHANGELOG.md) - [Commits](https://github.com/auth0/omniauth-auth0/compare/v2.3.1...v2.4.0) Signed-off-by: dependabot-preview[bot] <[email protected]>
commit sha a923ae9a1e824594c017447c17bc774ada6a9f0c
Merge branch 'develop' into dependabot/bundler/omniauth-auth0-2.4.0
commit sha 61128885d8eb139461431f0089ca3db53207bb8e
Merge pull request #663 from UKGovernmentBEIS/dependabot/bundler/omniauth-auth0-2.4.0 Bump omniauth-auth0 from 2.3.1 to 2.4.0
push time in a month
delete branch UKGovernmentBEIS/beis-report-official-development-assistance
delete branch : dependabot/bundler/omniauth-auth0-2.4.0
delete time in a month
PR merged UKGovernmentBEIS/beis-report-official-development-assistance
Bumps omniauth-auth0 from 2.3.1 to 2.4.0. <details> <summary>Release notes</summary> <p><em>Sourced from <a href="https://github.com/auth0/omniauth-auth0/releases">omniauth-auth0's releases</a>.</em></p> <blockquote> <h2>v2.4.0</h2> <h2><a href="https://github.com/auth0/omniauth-auth0/tree/v2.4.0">v2.4.0</a> (2020-09-22)</h2> <p><a href="https://github.com/auth0/omniauth-auth0/compare/v2.3.1...v2.4.0">Full Changelog</a></p> <p><strong>Security</strong></p> <ul> <li>Bump rack from 2.2.2 to 2.2.3 <a href="https://github-redirect.dependabot.com/auth0/omniauth-auth0/pull/107">#107</a> (<a href="https://github.com/dependabot">dependabot</a>)</li> <li>Update dependencies <a href="https://github-redirect.dependabot.com/auth0/omniauth-auth0/pull/100">#100</a> (<a href="https://github.com/Albalmaceda">Albalmaceda</a>)</li> </ul> <p><strong>Added</strong></p> <ul> <li>Add support for screen_hint=signup param <a href="https://github-redirect.dependabot.com/auth0/omniauth-auth0/pull/103">#103</a> (<a href="https://github.com/bbean86">bbean86</a>)</li> <li>Add support for <code>connection_scope</code> in params <a href="https://github-redirect.dependabot.com/auth0/omniauth-auth0/pull/99">#99</a> (<a href="https://github.com/felixclack">felixclack</a>)</li> </ul> </blockquote> </details> <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/auth0/omniauth-auth0/blob/master/CHANGELOG.md">omniauth-auth0's changelog</a>.</em></p> <blockquote> <h2><a href="https://github.com/auth0/omniauth-auth0/tree/v2.4.0">v2.4.0</a> (2020-09-22)</h2> <p><a href="https://github.com/auth0/omniauth-auth0/compare/v2.3.1...v2.4.0">Full Changelog</a></p> <p><strong>Security</strong></p> <ul> <li>Bump rack from 2.2.2 to 2.2.3 <a href="https://github-redirect.dependabot.com/auth0/omniauth-auth0/pull/107">#107</a> (<a href="https://github.com/dependabot">dependabot</a>)</li> <li>Update dependencies <a href="https://github-redirect.dependabot.com/auth0/omniauth-auth0/pull/100">#100</a> (<a href="https://github.com/Albalmaceda">Albalmaceda</a>)</li> </ul> <p><strong>Added</strong></p> <ul> <li>Add support for screen_hint=signup param <a href="https://github-redirect.dependabot.com/auth0/omniauth-auth0/pull/103">#103</a> (<a href="https://github.com/bbean86">bbean86</a>)</li> <li>Add support for <code>connection_scope</code> in params <a href="https://github-redirect.dependabot.com/auth0/omniauth-auth0/pull/99">#99</a> (<a href="https://github.com/felixclack">felixclack</a>)</li> </ul> </blockquote> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/auth0/omniauth-auth0/commit/860bd451eeb8390ba33a7fcf8fcb969e214a7236"><code>860bd45</code></a> Bump omniauth-auth0 to 2.4.0 (<a href="https://github-redirect.dependabot.com/auth0/omniauth-auth0/issues/108">#108</a>)</li> <li><a href="https://github.com/auth0/omniauth-auth0/commit/7cfc4faa7b4aed9126082ecf6b7e3d8b0fb238f0"><code>7cfc4fa</code></a> Bump rack from 2.2.2 to 2.2.3 (<a href="https://github-redirect.dependabot.com/auth0/omniauth-auth0/issues/107">#107</a>)</li> <li><a href="https://github.com/auth0/omniauth-auth0/commit/fb70eacc6bfa211b138781330a9b818871f167e7"><code>fb70eac</code></a> Add support for screen_hint=signup param (<a href="https://github-redirect.dependabot.com/auth0/omniauth-auth0/issues/103">#103</a>)</li> <li><a href="https://github.com/auth0/omniauth-auth0/commit/7f415614e4219ed157a2587a5dab2cc3b4ef6086"><code>7f41561</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/auth0/omniauth-auth0/issues/106">#106</a> from fossabot/add-license-scan-badge</li> <li><a href="https://github.com/auth0/omniauth-auth0/commit/c3eb541a3d9f7e2140952628919d0182a6e085f1"><code>c3eb541</code></a> Add license scan report and status</li> <li><a href="https://github.com/auth0/omniauth-auth0/commit/9ed2ce50652b97459c0f97bf911bd170912e081c"><code>9ed2ce5</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/auth0/omniauth-auth0/issues/104">#104</a> from auth0/davidpatrick-patch-1</li> <li><a href="https://github.com/auth0/omniauth-auth0/commit/55a4dc09df60b5a3e0f48d840af81f935fecef5c"><code>55a4dc0</code></a> Update README.md</li> <li><a href="https://github.com/auth0/omniauth-auth0/commit/fafaba32fbddc1daf9c833883851b17de5f27baf"><code>fafaba3</code></a> Add security disclaimer for using library</li> <li><a href="https://github.com/auth0/omniauth-auth0/commit/ed9f4e4905d351c54a653b27804be08068ed7f6c"><code>ed9f4e4</code></a> Merge pull request <a href="https://github-redirect.dependabot.com/auth0/omniauth-auth0/issues/99">#99</a> from felixclack/connection-scope</li> <li><a href="https://github.com/auth0/omniauth-auth0/commit/1cf2b95c7d15e448c3d84d05196843170f2f47d4"><code>1cf2b95</code></a> Merge branch 'master' into connection-scope</li> <li>Additional commits viewable in <a href="https://github.com/auth0/omniauth-auth0/compare/v2.3.1...v2.4.0">compare view</a></li> </ul> </details> <br />
Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.
<details> <summary>Dependabot commands and options</summary> <br />
You can trigger Dependabot actions by commenting on this PR:
@dependabot rebasewill rebase this PR@dependabot recreatewill recreate this PR, overwriting any edits that have been made to it@dependabot mergewill merge this PR after your CI passes on it@dependabot squash and mergewill squash and merge this PR after your CI passes on it@dependabot cancel mergewill cancel a previously requested merge and block automerging@dependabot reopenwill reopen this PR if it is closed@dependabot closewill close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually@dependabot ignore this major versionwill close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this minor versionwill close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)@dependabot ignore this dependencywill close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)@dependabot use these labelswill set the current labels as the default for future PRs for this repo and language@dependabot use these reviewerswill set the current reviewers as the default for future PRs for this repo and language@dependabot use these assigneeswill set the current assignees as the default for future PRs for this repo and language@dependabot use this milestonewill set the current milestone as the default for future PRs for this repo and language@dependabot badge mewill comment on this PR with code to add a "Dependabot enabled" badge to your readme
Additionally, you can set the following in your Dependabot dashboard:
- Update frequency (including time of day and day of week)
- Pull request limits (per update run and/or open at any time)
- Automerge options (never/patch/minor, and dev/runtime dependencies)
- Out-of-range updates (receive only lockfile updates, if desired)
- Security updates (receive only security updates, if desired)
</details>
pr closed time in a month
push eventUKGovernmentBEIS/beis-report-official-development-assistance
commit sha f73aba0b189569d287b3455ccbe42db1dec0d746
Extract updating model into save_changes! method This will be re-used for saving transactions as well as activities.
commit sha 8df3b563a36f761a34476a191f91913edbf57fa1
Ingest transactions when CSV has 'transaction_type' If the input CSV contains a `transaction_type` column, we assume that we're ingesting transactions only, so the Transaction associated with an Activity is modified, rather than the Activity itself. We assume that the combination of delivery_partner_identifier and description is unique enough to allow existing transactions to be overwritten with new data. When creating a transaction we attempt to associate it with the correct report. Skipping the validation check for report means we can still save the transaction on the occasions one doesn't exist. Since we only know the providing organisation side of these transactions, we also skip the validation check on receiving_organisation_name and receiving_organisation_type so we can save the record.
commit sha 6aea9f3835510ed3ce6a98ddc2831adc23679bb9
IngestCsv processes transaction columns - transaction_type - date - currency - value - providing_organisation_type NB: description, providing_organisation_name and providing_organisation_reference are ingested as-is.
commit sha ddbe3fb4e6778d46b6b1d0776226142db83fa7f7
IngestCsv ignores some additional validations Ignore validation errors on the following attributes: - date (so we can ingest future dated transactions) - collaboration_type (existing activities may not have this set)
commit sha 18d2eeb3a104de0578c2737e350a017fbc63793e
Add special cases for Sector and Country lookups There were a small number of ocassions where the CSV contained sectors or countries that were not exact matches for those specified in the IATI approved codelists. Therefore, some additional mappings were manually made to allow ingest to complete successfully. NB: Some forthcoming work will ensure that the codelists are updated in RODA so that activities can be assigned correctly.
commit sha df98c32a6f09ebbd499d08de9e884906bcf3ed4f
Merge pull request #653 from UKGovernmentBEIS/feature/986-ingest-financials-from-csv (986) Ingest financials from CSV
commit sha 970ecef727dceb48cf131f2dd50e2b8f6f10a674
Swap order of Actuals & Forecast so Forecast comes first in the CSV columns
commit sha 486f280dd85cf99487163960c638fc2a273a6c27
Merge pull request #664 from UKGovernmentBEIS/fix/swap-order-of-fields-in-report-csv Swap order of Actuals & Forecast so Forecast comes first in the CSV columns
commit sha 45bfba4496f9ea50d3e730e2c54037bdcf3d56a4
Reject non-numeric values in bulk transaction import Monetary values entered in the `Value` column in bulk upload are passed through the `CreateTransaction` service, which uses `Monetize.parse` to interpret them. This is much more permissive than we want; it allows various currency symbols and many different kinds of formatting, whereas we want to specifically allow this format: - an optional `£` symbol, followed by - an optional minus (`-`) symbol, followed by - at least 1 digit, with comma (`,`) allowed for readability - an optional decimal point followed by 2 digits We also cannot just remove the Monetize call, as passing the string directly through to ActiveRecord results in truncation; a string like "3a45" is truncated to the value 3.0. We would like any non-numeric characters in the input treated as an error, as they may represent a mistake on the user's part. This is especially important in bulk upload, where a user is adding a lot of data at once and not necessarily checking all of it afterwards. The ConvertFinancialValue service implements the desired format, and ImportTransactions can then report any parsing errors back to the user.
commit sha 6fa5b18d7fb400e79091e3b6c218308547ad69fc
Merge pull request #662 from UKGovernmentBEIS/feature/check-financial-values Reject non-numeric values in bulk transaction import
commit sha e5b9cabe3ce476c9454c6fd5d92b5bdeed14d83b
Modify ConvertFinancialValue to handle a single decimal place The existing tests for CreateTransaction expect a number with a single decimal place to be handled, so amend ConvertFinancialValue to respect single decimal places (but reject three or more decimal places)
commit sha 190ee3da0ac2550399f7b5ce2dc881414a627f96
Do not automatically strip letters from a monetary value Previously, letters accidentally entered in the `value` fields for budgets, transactions and planned disbursements were being stripped and the numbers accepted. We would prefer these entries to be rejected as an error instead of silently stripped and passed through. Use the `ConvertFinancialValue` to sanitize the values before they are added to their parent entities, and show a relevant error message if any letters are present.
commit sha 1e5c006a3e4a3a82d6376d7651a9bbba6797e4b2
Merge pull request #665 from UKGovernmentBEIS/fix/1014-when-a-monetary-value-is-supplied-with-letters-in-it-do-not-strip-lettters (1014) Do not strip alphabetical characters from values, reject the value instead
commit sha a923ae9a1e824594c017447c17bc774ada6a9f0c
Merge branch 'develop' into dependabot/bundler/omniauth-auth0-2.4.0
push time in a month
Pull request review commentUKGovernmentBEIS/beis-report-official-development-assistance
(1014) Do not strip alphabetical characters from values, reject the value instead
def call(attributes: {}) result end - private def editable_report_for_activity(activity:)+ private++ def editable_report_for_activity(activity:) Report.find_by(organisation: activity.organisation, fund: activity.associated_fund, state: Report::EDITABLE_STATES) end - private def sanitize_monetary_string(value:)- Monetize.parse(value)+ def convert_and_assign_value(budget, value)+ budget.value = ConvertFinancialValue.new.convert(value.to_s)+ budget
Cool cool :shipit:
comment created time in a month
pull request commentUKGovernmentBEIS/beis-report-official-development-assistance
(861) Sign out link not active on Users page
@mec That's fair enough, I think if this was a behaviour bug e.g. people not being able to log out, a test would be necessary but it's more of a judgement call for presentational concerns. Do go ahead and merge :)
comment created time in a month
Pull request review commentUKGovernmentBEIS/beis-report-official-development-assistance
(1014) Do not strip alphabetical characters from values, reject the value instead
def call(attributes: {}) result end - private def editable_report_for_activity(activity:)+ private++ def editable_report_for_activity(activity:) Report.find_by(organisation: activity.organisation, fund: activity.associated_fund, state: Report::EDITABLE_STATES) end - private def sanitize_monetary_string(value:)- Monetize.parse(value)+ def convert_and_assign_value(budget, value)+ budget.value = ConvertFinancialValue.new.convert(value.to_s)+ budget
What's the reason for returning the record in each of these methods? The result of convert_and_assign_value does not appear to be used.
comment created time in a month
Pull request review commentUKGovernmentBEIS/beis-report-official-development-assistance
(1014) Do not strip alphabetical characters from values, reject the value instead
class ConvertFinancialValue- VALUE_FORMAT = /^(?:£ *)?((?:- *)?[0-9,]+(?:\.[0-9]{2})?)$/+ VALUE_FORMAT = /^(?:£ *)?((?:- *)?[0-9,]+(?:\.[0-9]{1,2})?)$/
Apologies, I overlooked this as a requirement. Thanks for correcting it!
comment created time in a month
push eventUKGovernmentBEIS/beis-report-official-development-assistance
commit sha 45bfba4496f9ea50d3e730e2c54037bdcf3d56a4
Reject non-numeric values in bulk transaction import Monetary values entered in the `Value` column in bulk upload are passed through the `CreateTransaction` service, which uses `Monetize.parse` to interpret them. This is much more permissive than we want; it allows various currency symbols and many different kinds of formatting, whereas we want to specifically allow this format: - an optional `£` symbol, followed by - an optional minus (`-`) symbol, followed by - at least 1 digit, with comma (`,`) allowed for readability - an optional decimal point followed by 2 digits We also cannot just remove the Monetize call, as passing the string directly through to ActiveRecord results in truncation; a string like "3a45" is truncated to the value 3.0. We would like any non-numeric characters in the input treated as an error, as they may represent a mistake on the user's part. This is especially important in bulk upload, where a user is adding a lot of data at once and not necessarily checking all of it afterwards. The ConvertFinancialValue service implements the desired format, and ImportTransactions can then report any parsing errors back to the user.
commit sha 6fa5b18d7fb400e79091e3b6c218308547ad69fc
Merge pull request #662 from UKGovernmentBEIS/feature/check-financial-values Reject non-numeric values in bulk transaction import
push time in a month
delete branch UKGovernmentBEIS/beis-report-official-development-assistance
delete branch : feature/check-financial-values
delete time in a month
PR merged UKGovernmentBEIS/beis-report-official-development-assistance
Monetary values entered in the Value column in bulk upload are passed
through the CreateTransaction service, which uses Monetize.parse to
interpret them. This is much more permissive than we want; it allows
various currency symbols and many different kinds of formatting, whereas
we want to specifically allow this format:
- an optional
£symbol, followed by - an optional minus (
-) symbol, followed by - at least 1 digit, with comma (
,) allowed for readability - an optional decimal point followed by 2 digits
We also cannot just remove the Monetize call, as passing the string directly through to ActiveRecord results in truncation; a string like "3a45" is truncated to the value 3.0. We would like any non-numeric characters in the input treated as an error, as they may represent a mistake on the user's part. This is especially important in bulk upload, where a user is adding a lot of data at once and not necessarily checking all of it afterwards.
The ConvertFinancialValue service implements the desired format, and ImportTransactions can then report any parsing errors back to the user.
Changes in this PR
Screenshots of UI changes
Before
After
Next steps
- [ ] Is an ADR required? An ADR should be added if this PR introduces a change to the architecture.
- [x] Is a changelog entry required? An entry should always be made in
CHANGELOG.md, unless this PR is a small tweak which has no impact outside the development team. - [ ] Do any environment variables need amending or adding?
- [ ] Have any changes to the XML been checked with the IATI validator? See XML Validation
pr closed time in a month
push eventUKGovernmentBEIS/beis-report-official-development-assistance
commit sha f73aba0b189569d287b3455ccbe42db1dec0d746
Extract updating model into save_changes! method This will be re-used for saving transactions as well as activities.
commit sha 8df3b563a36f761a34476a191f91913edbf57fa1
Ingest transactions when CSV has 'transaction_type' If the input CSV contains a `transaction_type` column, we assume that we're ingesting transactions only, so the Transaction associated with an Activity is modified, rather than the Activity itself. We assume that the combination of delivery_partner_identifier and description is unique enough to allow existing transactions to be overwritten with new data. When creating a transaction we attempt to associate it with the correct report. Skipping the validation check for report means we can still save the transaction on the occasions one doesn't exist. Since we only know the providing organisation side of these transactions, we also skip the validation check on receiving_organisation_name and receiving_organisation_type so we can save the record.
commit sha 6aea9f3835510ed3ce6a98ddc2831adc23679bb9
IngestCsv processes transaction columns - transaction_type - date - currency - value - providing_organisation_type NB: description, providing_organisation_name and providing_organisation_reference are ingested as-is.
commit sha ddbe3fb4e6778d46b6b1d0776226142db83fa7f7
IngestCsv ignores some additional validations Ignore validation errors on the following attributes: - date (so we can ingest future dated transactions) - collaboration_type (existing activities may not have this set)
commit sha 18d2eeb3a104de0578c2737e350a017fbc63793e
Add special cases for Sector and Country lookups There were a small number of ocassions where the CSV contained sectors or countries that were not exact matches for those specified in the IATI approved codelists. Therefore, some additional mappings were manually made to allow ingest to complete successfully. NB: Some forthcoming work will ensure that the codelists are updated in RODA so that activities can be assigned correctly.
commit sha df98c32a6f09ebbd499d08de9e884906bcf3ed4f
Merge pull request #653 from UKGovernmentBEIS/feature/986-ingest-financials-from-csv (986) Ingest financials from CSV
commit sha 970ecef727dceb48cf131f2dd50e2b8f6f10a674
Swap order of Actuals & Forecast so Forecast comes first in the CSV columns
commit sha 486f280dd85cf99487163960c638fc2a273a6c27
Merge pull request #664 from UKGovernmentBEIS/fix/swap-order-of-fields-in-report-csv Swap order of Actuals & Forecast so Forecast comes first in the CSV columns
commit sha 45bfba4496f9ea50d3e730e2c54037bdcf3d56a4
Reject non-numeric values in bulk transaction import Monetary values entered in the `Value` column in bulk upload are passed through the `CreateTransaction` service, which uses `Monetize.parse` to interpret them. This is much more permissive than we want; it allows various currency symbols and many different kinds of formatting, whereas we want to specifically allow this format: - an optional `£` symbol, followed by - an optional minus (`-`) symbol, followed by - at least 1 digit, with comma (`,`) allowed for readability - an optional decimal point followed by 2 digits We also cannot just remove the Monetize call, as passing the string directly through to ActiveRecord results in truncation; a string like "3a45" is truncated to the value 3.0. We would like any non-numeric characters in the input treated as an error, as they may represent a mistake on the user's part. This is especially important in bulk upload, where a user is adding a lot of data at once and not necessarily checking all of it afterwards. The ConvertFinancialValue service implements the desired format, and ImportTransactions can then report any parsing errors back to the user.
push time in a month
push eventUKGovernmentBEIS/beis-report-official-development-assistance
commit sha c7623f332fb7132218e3321b32afdab652285b1c
Reject non-numeric values in bulk transaction import Monetary values entered in the `Value` column in bulk upload are passed through the `CreateTransaction` service, which uses `Monetize.parse` to interpret them. This is much more permissive than we want; it allows various currency symbols and many different kinds of formatting, whereas we want to specifically allow this format: - an optional `£` symbol, followed by - an optional minus (`-`) symbol, followed by - at least 1 digit, with comma (`,`) allowed for readability - an optional decimal point followed by 2 digits We also cannot just remove the Monetize call, as passing the string directly through to ActiveRecord results in truncation; a string like "3a45" is truncated to the value 3.0. We would like any non-numeric characters in the input treated as an error, as they may represent a mistake on the user's part. This is especially important in bulk upload, where a user is adding a lot of data at once and not necessarily checking all of it afterwards. The ConvertFinancialValue service implements the desired format, and ImportTransactions can then report any parsing errors back to the user.
push time in a month
PR opened UKGovernmentBEIS/beis-report-official-development-assistance
Monetary values entered in the Value column in bulk upload are passed
through the CreateTransaction service, which uses Monetize.parse to
interpret them. This is much more permissive than we want; it allows
various currency symbols and many different kinds of formatting, whereas
we want to specifically allow this format:
- an optional
£symbol, followed by - an optional minus (
-) symbol, followed by - at least 1 digit, with comma (
,) allowed for readability - an optional decimal point followed by 2 digits
We also cannot just remove the Monetize call, as passing the string directly through to ActiveRecord results in truncation; a string like "3a45" is truncated to the value 3.0. We would like any non-numeric characters in the input treated as an error, as they may represent a mistake on the user's part. This is especially important in bulk upload, where a user is adding a lot of data at once and not necessarily checking all of it afterwards.
The ConvertFinancialValue service implements the desired format, and ImportTransactions can then report any parsing errors back to the user.
Changes in this PR
Screenshots of UI changes
Before
After
Next steps
- [ ] Is an ADR required? An ADR should be added if this PR introduces a change to the architecture.
- [ ] Is a changelog entry required? An entry should always be made in
CHANGELOG.md, unless this PR is a small tweak which has no impact outside the development team. - [ ] Do any environment variables need amending or adding?
- [ ] Have any changes to the XML been checked with the IATI validator? See XML Validation
pr created time in a month
create barnchUKGovernmentBEIS/beis-report-official-development-assistance
branch : feature/check-financial-values
created branch time in a month
pull request commentUKGovernmentBEIS/beis-report-official-development-assistance
(861) Sign out link not active on Users page
If this was a bug, is it possible to add a test that confirms the expected behaviour?
comment created time in a month
push eventUKGovernmentBEIS/beis-report-official-development-assistance
commit sha 651f7542ad763f7d442a4caead023260ced6285e
Add Activity#roda_identifier This method lets you get the RODA identifier for an activity while hiding the implementation details of the `roda_identifier_fragment` and `roda_identifier_compound` fields. It returns the "compound" identifier, i.e. the cached ID formed by concatenating the fragments of all the ancestor activities.
commit sha f0b4aedb0fbaa3a61b893e3e0c1a47c534a63d4f
Use Activity#roda_identifier where applicable Now that `Activity#roda_identifier` exists to return the RODA identifier, we should use this everywhere we were previously using the `roda_identifier_compound` field directly, to clarify the code and signal how devs _should_ get the RODA identifier for an activity.
commit sha 8ffc8ab558d9390324f948a304a5db7923cbb941
Merge pull request #659 from UKGovernmentBEIS/chore/roda-identifier-alias Add Activity#roda_identifier
push time in a month
delete branch UKGovernmentBEIS/beis-report-official-development-assistance
delete branch : chore/roda-identifier-alias
delete time in a month
PR merged UKGovernmentBEIS/beis-report-official-development-assistance
Changes in this PR
This is refactoring triggered by a conversation I had with @lozette, to make it clearer how one should get the RODA identifier for an activity.
Next steps
- [ ] Is an ADR required? An ADR should be added if this PR introduces a change to the architecture.
- [ ] Is a changelog entry required? An entry should always be made in
CHANGELOG.md, unless this PR is a small tweak which has no impact outside the development team. - [ ] Do any environment variables need amending or adding?
- [ ] Have any changes to the XML been checked with the IATI validator? See XML Validation
pr closed time in a month
PR opened UKGovernmentBEIS/beis-report-official-development-assistance
Changes in this PR
This is refactoring triggered by a conversation I had with @lozette, to make it clearer how one should get the RODA identifier for an activity.
Next steps
- [ ] Is an ADR required? An ADR should be added if this PR introduces a change to the architecture.
- [ ] Is a changelog entry required? An entry should always be made in
CHANGELOG.md, unless this PR is a small tweak which has no impact outside the development team. - [ ] Do any environment variables need amending or adding?
- [ ] Have any changes to the XML been checked with the IATI validator? See XML Validation
pr created time in a month
create barnchUKGovernmentBEIS/beis-report-official-development-assistance
branch : chore/roda-identifier-alias
created branch time in a month
issue commentfaye/websocket-driver-ruby
Unhandled server-side disconnects
@tjad Just to confirm: websocket-driver should always send a pong message if it receives a ping. It will only send a ping message if explicitly instructed.
comment created time in a month
pull request commentUKGovernmentBEIS/beis-report-official-development-assistance
We discussed this and agreed we should perform this upgrade when we have more time to thoroughly test it on staging. The release notes are worth reading -- it sounds like this is mostly opt-in experiemental performance improvements, but as it's a major version change I'd like to thoroughly test it before going to production.
comment created time in a month
push eventUKGovernmentBEIS/beis-report-official-development-assistance
commit sha 794678187dd4ae0f397a613c265ef9a4ee983f11
Display the parent activity's RODA Identifier When entering a RODA Identifier, there's nothing to tell the user where they are, and the data they enter here is final, so it's both easy and very costly to make a mistake and enter the wrong thing. As a quick improvement, we can add the parent activity's RODA identifier to the page, if the activity has a parent. You can only enter a RODA ID if the parent activity has one completed, so this is guaranteed to always have a value to display.
commit sha 33a94698ff928bcf35cbfd8bf284565eef92a371
Merge pull request #650 from UKGovernmentBEIS/feature/998-show-roda-identifier-prefixes (998) Display the parent activity's RODA Identifier
push time in a month
delete branch UKGovernmentBEIS/beis-report-official-development-assistance
delete branch : feature/998-show-roda-identifier-prefixes
delete time in a month
PR merged UKGovernmentBEIS/beis-report-official-development-assistance
When entering a RODA Identifier, there's nothing to tell the user where they are, and the data they enter here is final, so it's both easy and very costly to make a mistake and enter the wrong thing.
As a quick improvement, we can add the parent activity's RODA identifier to the page, if the activity has a parent. You can only enter a RODA ID if the parent activity has one completed, so this is guaranteed to always have a value to display.
Screenshots of UI changes

Next steps
- [ ] Is an ADR required? An ADR should be added if this PR introduces a change to the architecture.
- [x] Is a changelog entry required? An entry should always be made in
CHANGELOG.md, unless this PR is a small tweak which has no impact outside the development team. - [ ] Do any environment variables need amending or adding?
- [ ] Have any changes to the XML been checked with the IATI validator? See XML Validation
pr closed time in a month
push eventUKGovernmentBEIS/beis-report-official-development-assistance
commit sha 421ed74e742c4f60d9ac20c834026db9d93a6d80
Bump better_errors from 2.8.0 to 2.8.1 Bumps [better_errors](https://github.com/BetterErrors/better_errors) from 2.8.0 to 2.8.1. - [Release notes](https://github.com/BetterErrors/better_errors/releases) - [Commits](https://github.com/BetterErrors/better_errors/compare/v2.8.0...v2.8.1) Signed-off-by: dependabot-preview[bot] <[email protected]>
commit sha 1fd7fb05eb192217c04d3078a295df06a3336170
Merge pull request #654 from UKGovernmentBEIS/dependabot/bundler/better_errors-2.8.1 Bump better_errors from 2.8.0 to 2.8.1
commit sha 794678187dd4ae0f397a613c265ef9a4ee983f11
Display the parent activity's RODA Identifier When entering a RODA Identifier, there's nothing to tell the user where they are, and the data they enter here is final, so it's both easy and very costly to make a mistake and enter the wrong thing. As a quick improvement, we can add the parent activity's RODA identifier to the page, if the activity has a parent. You can only enter a RODA ID if the parent activity has one completed, so this is guaranteed to always have a value to display.
push time in a month
push eventUKGovernmentBEIS/beis-report-official-development-assistance
commit sha e648afc8007da20676b43556761d7655200bb5a1
Import a single transaction from a CSV This begins the work of importing transactions in bulk from a CSV input. Most of the machinery will like in the ImportTransactions service, which takes an array of hashes rather than a CSV file directly, to make it easier to test. The initial implementation maps all the bulk import columns to their places in the database, but does not do any additional validation or conversion beyond that performed by the model. It does provide some defaults for mandatory fields: - The currency is GBP - The transaction type is 'Disbursement' - The 'providing organisation' information is inherited from the Activity
commit sha 5d771ec8eaae97569c176863ffc982d391954670
Validate the activity as part of a bulk transaction upload This is the first piece of validation that ImportTransactions does, so this also sets up a bunch of machinery to support error reporting. Errors report the row offset, column name, cell value, and error message, so errors can be easily located in a spreadsheet. Because this class doesn't assume CSV input per se but takes an array of hashes, the row offset is zero-based. It can be adjusted if the caller knows more about the input format. This first validation checks that the `Activity RODA Identifier` field refers to an existing activity, otherwise an error is produced and the transaction is not imported.
commit sha 3ff7786d8e2840d528ddf593ee65610fe076fc71
Validate dates on bulk-uploaded transactions Here we introduce support for reporting errors that result from conversions from the data format supported by the bulk upload CSV service. The conversion methods can raise exceptions, which will be caught and stored on the ImportTransactions instance. The first such validation is that Date must parse as ISO-8601, according to the Ruby implementation of this format. The value reported in the error is the original value from the imported row, not the value after any conversion is applied.
commit sha 896a4237f5121939df084fbb685362a7ca41dc12
Refactor error handling in ImportTransactions The `import_row` method is getting fairly complex, so here we break it into smaller pieces, in particular not mixing error handling in with the rest of the logic quite so much. For each row, errors are collected in a hash, which can be updated incrementally or import all the errors from another object, as in errors.update(converter.errors) At the end of `import_row` we convert the hash's contents into `Error` objects with row numbers attached.
commit sha acb2ee4e8416419aed8f74dc9a73a4b0c1cf2930
Validate dates at the model layer in ImportTransactions If the `Date` field of an imported row is empty, we return `nil` from the conversion function rather than an empty string. This triggers a more meaningful validation in the model. Our imported should only validate the _formatting_ of the date, not its content -- any semantic validation should be done by the model, this class is merely a parser/translator. To effect this, the result of `CreateTransaction` is extracted and we copy any ActiveRecord validation errors into our own error set, using the attribute name to fetch the raw value from the input.
commit sha b56dcf9ba11ddca9436ac6e129ec47554e8bb0f5
Test validation of values during transaction bulk upload Having integrated model-layer validation into ImportTransactions in the previous commit, we add tests that the `Value` field is correctly interpreted. The logic for converting these values is already handled by the model.
commit sha 5605cbb903b615b4b47b4ec6d5db3ca53450c36e
Validate the receiving organisation during bulk transaction upload The transaction bulk upload CSV schema contains three fields about the receiving organisation: - `Receiving Organisation Name` - `Receiving Organisation Type` - `Receiving Organisation IATI Reference` The former two are required, and the Type must be one of the codes in the IATI Organisation Type codelist: https://iatistandard.org/en/iati-standard/203/codelists/organisationtype/ We validate this using our existing codelist file.
commit sha 5ea9fa486db5ee7f03347ff6e1cacad34b5a987d
Test that IATI references are saved during bulk transaction upload Here we're just adding a test for the `Receiving Organisation IATI Reference` field's behaviour, after dealing with the other Receiving Organisation fields in the previous commit. This field is not validated, we just want to check that it gets saved.
commit sha 5cdb600433efec79f0c1a681b0921d22b751a7f4
Validate the disbursement type during bulk transaction upload Much like the `Receiving Organisation Type` field in the CSV upload, `Disbursement Type` must be a valid code from the following IATI codelist: https://iatistandard.org/en/iati-standard/203/codelists/disbursementchannel/ Again, we can use our existing copies of these codelists to validate the provided value.
commit sha 27087c6e040b39ee806881c7362a972a946a7b09
Refactor codelist validations in ImportTransactions::Converter We know have two very similar methods in this class that take a value and check it against an IATI codelist. By factoring our the name of the codelist we can reuse more of the logic and remove a little duplication here.
commit sha b9d3af4fbb62c8ef9bdb4c5ee8ccbce99c23b555
Refactor CreatePlannedDisbursement#report and CreateTransaction#report These two classes both infer the currently active report that a planned disbursement or transaction should be added to. During bulk transaction upload, this report will be provided explicitly, because we'll launch the upload process from the page for a specific report. When ImportTransactions is calling CreateTransaction, I would rather pass the Report in, rather than have CreateTransaction infer the current Report and possibly disagree with the caller. Therefore I've factored the logic shared by these two classes into `Report#editable_for_activity`, and made both classes call this if they're not given a Report explicitly. The one difference introduced here is that the previous implementation looked only for `active` reports. We actually want to find any report the delivery partner can currently edit, which means it may be `active` or `awaiting_changes`.
commit sha 70fb4fb95a6034f9b403c0cb681fdd702fcdf47f
Pass Report explicitly from ImportTransactions to CreateTransaction The plan is that the bulk transaction upload process will be initiated from a report's page, and so we'll pass the report in question into the ImportTransactions class so it knows which report to add transactions to. We do this by passing a Report into ImportTransactions, and forwarding this on to CreateTransaction following the refactor in the previous commit. This is not just a correctness improvement (making sure all transactions are added to the same report without needing to lock anything), it's a performance improvement too. Without this, each transaction added via CreateTransaction would trigger a new query to look the Report for its Activity up. All of these should result in the same Report record, so doing this for every row is wasteful.
commit sha 1b11710e69cdd8ad1df72c350a7acb22a6b93611
Generate a default description for bulk-uploaded transactions In bulk upload, we want the `Description` field to be optional, whereas in the data model it's mandatory. If `Description` is blank in the CSV, then we'll generate a default description for the transaction based on the name of the activity and the description of the report.
commit sha 6dd411cea771edfe6557990ba3531940fb0c50aa
Authorise the user for each bulk upload transaction In normal controller actions we only need to authorise the user for a single resource/action. In bulk upload, the uploaded CSV might name any activity, so we need to make sure the current user is allowed to add data for that activity, and that the activity is under the scope of the current report. To check the uploader is authorised to create this transaction, I'm checking ActivityPolicy#create?, which is the same thing that Staff::TransactionsController#create checks. To check the activity is part of the current report, I'm checking that their organisation and fund match, which is what would cause `Report.editable_for_activity` to link the two objects.
commit sha 526a833c5472a3ca0ad30d3e479ff6cf0b872034
Refactor `import_row` to a RowImporter object We had built up a lot of logic in the methods of ImportTransactions that were passing the same objects between each other. This suggested pulling the whole process of importing a row into a distinct object, so that the various objects involved can live in its instance variables and not have to be passed around explicitly. This looks like a large change but it's not meant to change any logic. It just moves a set of logic into a new class while replacing function parameters with instance variables.
commit sha cc6ec2d5d35ffc2656a99586571c7be5207ef9b6
Import multiple transactions in ImportTransactions All the commits leading up to this have focussed on the logic for importing a single row: translating CSV columns to fields in the database, parsing and validating inputs, and so on. We now implement support for importing multiple rows at the same time. For ease of use, we want a CSV upload to be atomic: either all rows are imported, or none are. Partial success would mean the user being told which rows succeeded or failed, then retrying with a new spreadsheet of just the failed rows. If one failure leads to nothing being imported, the user just needs to amend their existing CSV and try the whole thing again. We acheive this by wrapping the import loop in a database transaction, and rolling it back if any of the rows fail to validate or save.
commit sha bd455d9bd5fa909ec150e481e3fd92c7c3b48a56
Refactor Activity.projects_and_third_party_projects_for_report The implementation of this method would be problematic if used for a bulk operation involving lots of activities under the same report. It loads all the level C/D Activity records for a report's Organisation, and then further filters them by checking whether the Activity's `associated_fund` matches that of the Report. `Activity#associated_fund` requires 2 or 3 further queries to look up the fund under which the level C/D activity sites. So if the initial query returned 500 activities, this would add an additional 1,000-1,500 queries to load the fund for each result, which will often be the same object. Instead, we invert this as follows. Given the fund ID from the Report, we can get the IDs of all programmes (level B) under this fund. From these we can get the IDs of all projects (level C) under those programmes. From these two lists of IDs, we can load all the level C and D activities that have one of them as a parent. The query ends up looking like this? SELECT * FROM activities WHERE organisation_id = ? AND (level = 'project' AND parent_id IN (<programme_ids>)) OR (level = 'third_party_project' AND parent_id IN (<project_ids>)) This reduces the lookup for all relevant activities to three queries.
commit sha 941349e1a987e427533311e4f5247840ff493e4e
Include organisations when loading report variances While writing tests for bulk transaction uploads, accessing this page cause a test to fail because of Bullet complaining about an unoptimised N+1 query. This was its suggestion for fixing the problem.
commit sha 50c1d4755b326d5a87c77cc04ff021f9172cda14
Download a CSV template to begin bulk transaction upload To initiate the bulk upload process, a user downloads a CSV that contains the column headers we require in ImportTransactions. It lists out the RODA Identifier for each activity that should appear in the current report, meaning all the level C and D activities for the report's organisation and fund, that have a RODA Identifier. (The Delivery Partner Identifier and Name are also included in the download for informational purposes and are ignored on upload.) The tests include a third project that belongs to the same organisation, but to a different fund from the other two, to make sure it's not included in the download for this report.
commit sha ca47d7c5a00b2a321f96a267c7cc2b7f60c0bc53
Upload transactions in bulk as CSV This exposes the functionality of ImportTransactions via a file upload that accepts CSV. The CSV is parsed in the controller and the resulting rows fed into the importer, which itself is thoroughly unit-tested. If any errors occur, then no transactions are imported, and the errors are displayed to the user. If all the rows are valid, then the transactions are saved and a success message is displayed.
push time in a month
push eventUKGovernmentBEIS/beis-report-official-development-assistance
commit sha e648afc8007da20676b43556761d7655200bb5a1
Import a single transaction from a CSV This begins the work of importing transactions in bulk from a CSV input. Most of the machinery will like in the ImportTransactions service, which takes an array of hashes rather than a CSV file directly, to make it easier to test. The initial implementation maps all the bulk import columns to their places in the database, but does not do any additional validation or conversion beyond that performed by the model. It does provide some defaults for mandatory fields: - The currency is GBP - The transaction type is 'Disbursement' - The 'providing organisation' information is inherited from the Activity
commit sha 5d771ec8eaae97569c176863ffc982d391954670
Validate the activity as part of a bulk transaction upload This is the first piece of validation that ImportTransactions does, so this also sets up a bunch of machinery to support error reporting. Errors report the row offset, column name, cell value, and error message, so errors can be easily located in a spreadsheet. Because this class doesn't assume CSV input per se but takes an array of hashes, the row offset is zero-based. It can be adjusted if the caller knows more about the input format. This first validation checks that the `Activity RODA Identifier` field refers to an existing activity, otherwise an error is produced and the transaction is not imported.
commit sha 3ff7786d8e2840d528ddf593ee65610fe076fc71
Validate dates on bulk-uploaded transactions Here we introduce support for reporting errors that result from conversions from the data format supported by the bulk upload CSV service. The conversion methods can raise exceptions, which will be caught and stored on the ImportTransactions instance. The first such validation is that Date must parse as ISO-8601, according to the Ruby implementation of this format. The value reported in the error is the original value from the imported row, not the value after any conversion is applied.
commit sha 896a4237f5121939df084fbb685362a7ca41dc12
Refactor error handling in ImportTransactions The `import_row` method is getting fairly complex, so here we break it into smaller pieces, in particular not mixing error handling in with the rest of the logic quite so much. For each row, errors are collected in a hash, which can be updated incrementally or import all the errors from another object, as in errors.update(converter.errors) At the end of `import_row` we convert the hash's contents into `Error` objects with row numbers attached.
commit sha acb2ee4e8416419aed8f74dc9a73a4b0c1cf2930
Validate dates at the model layer in ImportTransactions If the `Date` field of an imported row is empty, we return `nil` from the conversion function rather than an empty string. This triggers a more meaningful validation in the model. Our imported should only validate the _formatting_ of the date, not its content -- any semantic validation should be done by the model, this class is merely a parser/translator. To effect this, the result of `CreateTransaction` is extracted and we copy any ActiveRecord validation errors into our own error set, using the attribute name to fetch the raw value from the input.
commit sha b56dcf9ba11ddca9436ac6e129ec47554e8bb0f5
Test validation of values during transaction bulk upload Having integrated model-layer validation into ImportTransactions in the previous commit, we add tests that the `Value` field is correctly interpreted. The logic for converting these values is already handled by the model.
commit sha 5605cbb903b615b4b47b4ec6d5db3ca53450c36e
Validate the receiving organisation during bulk transaction upload The transaction bulk upload CSV schema contains three fields about the receiving organisation: - `Receiving Organisation Name` - `Receiving Organisation Type` - `Receiving Organisation IATI Reference` The former two are required, and the Type must be one of the codes in the IATI Organisation Type codelist: https://iatistandard.org/en/iati-standard/203/codelists/organisationtype/ We validate this using our existing codelist file.
commit sha 5ea9fa486db5ee7f03347ff6e1cacad34b5a987d
Test that IATI references are saved during bulk transaction upload Here we're just adding a test for the `Receiving Organisation IATI Reference` field's behaviour, after dealing with the other Receiving Organisation fields in the previous commit. This field is not validated, we just want to check that it gets saved.
commit sha 5cdb600433efec79f0c1a681b0921d22b751a7f4
Validate the disbursement type during bulk transaction upload Much like the `Receiving Organisation Type` field in the CSV upload, `Disbursement Type` must be a valid code from the following IATI codelist: https://iatistandard.org/en/iati-standard/203/codelists/disbursementchannel/ Again, we can use our existing copies of these codelists to validate the provided value.
commit sha 27087c6e040b39ee806881c7362a972a946a7b09
Refactor codelist validations in ImportTransactions::Converter We know have two very similar methods in this class that take a value and check it against an IATI codelist. By factoring our the name of the codelist we can reuse more of the logic and remove a little duplication here.
commit sha b9d3af4fbb62c8ef9bdb4c5ee8ccbce99c23b555
Refactor CreatePlannedDisbursement#report and CreateTransaction#report These two classes both infer the currently active report that a planned disbursement or transaction should be added to. During bulk transaction upload, this report will be provided explicitly, because we'll launch the upload process from the page for a specific report. When ImportTransactions is calling CreateTransaction, I would rather pass the Report in, rather than have CreateTransaction infer the current Report and possibly disagree with the caller. Therefore I've factored the logic shared by these two classes into `Report#editable_for_activity`, and made both classes call this if they're not given a Report explicitly. The one difference introduced here is that the previous implementation looked only for `active` reports. We actually want to find any report the delivery partner can currently edit, which means it may be `active` or `awaiting_changes`.
commit sha 70fb4fb95a6034f9b403c0cb681fdd702fcdf47f
Pass Report explicitly from ImportTransactions to CreateTransaction The plan is that the bulk transaction upload process will be initiated from a report's page, and so we'll pass the report in question into the ImportTransactions class so it knows which report to add transactions to. We do this by passing a Report into ImportTransactions, and forwarding this on to CreateTransaction following the refactor in the previous commit. This is not just a correctness improvement (making sure all transactions are added to the same report without needing to lock anything), it's a performance improvement too. Without this, each transaction added via CreateTransaction would trigger a new query to look the Report for its Activity up. All of these should result in the same Report record, so doing this for every row is wasteful.
commit sha 1b11710e69cdd8ad1df72c350a7acb22a6b93611
Generate a default description for bulk-uploaded transactions In bulk upload, we want the `Description` field to be optional, whereas in the data model it's mandatory. If `Description` is blank in the CSV, then we'll generate a default description for the transaction based on the name of the activity and the description of the report.
commit sha 6dd411cea771edfe6557990ba3531940fb0c50aa
Authorise the user for each bulk upload transaction In normal controller actions we only need to authorise the user for a single resource/action. In bulk upload, the uploaded CSV might name any activity, so we need to make sure the current user is allowed to add data for that activity, and that the activity is under the scope of the current report. To check the uploader is authorised to create this transaction, I'm checking ActivityPolicy#create?, which is the same thing that Staff::TransactionsController#create checks. To check the activity is part of the current report, I'm checking that their organisation and fund match, which is what would cause `Report.editable_for_activity` to link the two objects.
commit sha 526a833c5472a3ca0ad30d3e479ff6cf0b872034
Refactor `import_row` to a RowImporter object We had built up a lot of logic in the methods of ImportTransactions that were passing the same objects between each other. This suggested pulling the whole process of importing a row into a distinct object, so that the various objects involved can live in its instance variables and not have to be passed around explicitly. This looks like a large change but it's not meant to change any logic. It just moves a set of logic into a new class while replacing function parameters with instance variables.
commit sha cc6ec2d5d35ffc2656a99586571c7be5207ef9b6
Import multiple transactions in ImportTransactions All the commits leading up to this have focussed on the logic for importing a single row: translating CSV columns to fields in the database, parsing and validating inputs, and so on. We now implement support for importing multiple rows at the same time. For ease of use, we want a CSV upload to be atomic: either all rows are imported, or none are. Partial success would mean the user being told which rows succeeded or failed, then retrying with a new spreadsheet of just the failed rows. If one failure leads to nothing being imported, the user just needs to amend their existing CSV and try the whole thing again. We acheive this by wrapping the import loop in a database transaction, and rolling it back if any of the rows fail to validate or save.
commit sha bd455d9bd5fa909ec150e481e3fd92c7c3b48a56
Refactor Activity.projects_and_third_party_projects_for_report The implementation of this method would be problematic if used for a bulk operation involving lots of activities under the same report. It loads all the level C/D Activity records for a report's Organisation, and then further filters them by checking whether the Activity's `associated_fund` matches that of the Report. `Activity#associated_fund` requires 2 or 3 further queries to look up the fund under which the level C/D activity sites. So if the initial query returned 500 activities, this would add an additional 1,000-1,500 queries to load the fund for each result, which will often be the same object. Instead, we invert this as follows. Given the fund ID from the Report, we can get the IDs of all programmes (level B) under this fund. From these we can get the IDs of all projects (level C) under those programmes. From these two lists of IDs, we can load all the level C and D activities that have one of them as a parent. The query ends up looking like this? SELECT * FROM activities WHERE organisation_id = ? AND (level = 'project' AND parent_id IN (<programme_ids>)) OR (level = 'third_party_project' AND parent_id IN (<project_ids>)) This reduces the lookup for all relevant activities to three queries.
commit sha 941349e1a987e427533311e4f5247840ff493e4e
Include organisations when loading report variances While writing tests for bulk transaction uploads, accessing this page cause a test to fail because of Bullet complaining about an unoptimised N+1 query. This was its suggestion for fixing the problem.
commit sha 50c1d4755b326d5a87c77cc04ff021f9172cda14
Download a CSV template to begin bulk transaction upload To initiate the bulk upload process, a user downloads a CSV that contains the column headers we require in ImportTransactions. It lists out the RODA Identifier for each activity that should appear in the current report, meaning all the level C and D activities for the report's organisation and fund, that have a RODA Identifier. (The Delivery Partner Identifier and Name are also included in the download for informational purposes and are ignored on upload.) The tests include a third project that belongs to the same organisation, but to a different fund from the other two, to make sure it's not included in the download for this report.
commit sha ca47d7c5a00b2a321f96a267c7cc2b7f60c0bc53
Upload transactions in bulk as CSV This exposes the functionality of ImportTransactions via a file upload that accepts CSV. The CSV is parsed in the controller and the resulting rows fed into the importer, which itself is thoroughly unit-tested. If any errors occur, then no transactions are imported, and the errors are displayed to the user. If all the rows are valid, then the transactions are saved and a success message is displayed.
push time in a month
delete branch UKGovernmentBEIS/beis-report-official-development-assistance
delete branch : feature/876-upload-transactions-as-csv
delete time in a month
PR merged UKGovernmentBEIS/beis-report-official-development-assistance
Changes in this PR
This allows delivery partner users to upload transactions for a report by uploading a CSV. They work by filling out a template we give them, which contains the required column headings, and the IDs of all the activities they need to report on. The RODA ID is used to identify activities on upload, while the Delivery Partner ID and Name are there for the user's information and are ignored on upload.
This is quite a large PR but the code is mostly fairly boring validation and translation of CSV fields onto database columns, plus a little glue at the end to expose the functionality via the service. I'd recommend reading each commit rather than all the changes at once.
I'd like to check that all the logic for validation and translation the fields is correct, that the right associations between transactions, activities and reports are created, and that any permissions-related checks are correct.
Screenshots of UI changes
When viewing a report, a new button is available in the menu to "Upload actuals":

This links to a page where the user can download a CSV template, and upload their finished spreadsheet:

The download gives them a CSV with the required column headings, and the IDs of each activity they need to provide actuals for in this report.

The user fills this spreadsheet out and re-exports as CSV. If what they upload contains any errors, then the transactions are not created, and the errors are displayed.

If they don't upload a file at all, an error message is shown:

If all the data is valid and the transactions are saved, a success message is shown.

Next steps
- [ ] Is an ADR required? An ADR should be added if this PR introduces a change to the architecture.
- [x] Is a changelog entry required? An entry should always be made in
CHANGELOG.md, unless this PR is a small tweak which has no impact outside the development team. - [ ] Do any environment variables need amending or adding?
- [ ] Have any changes to the XML been checked with the IATI validator? See XML Validation
pr closed time in a month
pull request commentUKGovernmentBEIS/beis-report-official-development-assistance
(998) Display the parent activity's RODA Identifier
jcoglancomment created time in a month
PR opened UKGovernmentBEIS/beis-report-official-development-assistance
When entering a RODA Identifier, there's nothing to tell the user where they are, and the data they enter here is final, so it's both easy and very costly to make a mistake and enter the wrong thing.
As a quick improvement, we can add the parent activity's RODA identifier to the page, if the activity has a parent. You can only enter a RODA ID if the parent activity has one completed, so this is guaranteed to always have a value to display.
Screenshots of UI changes

Next steps
- [ ] Is an ADR required? An ADR should be added if this PR introduces a change to the architecture.
- [ ] Is a changelog entry required? An entry should always be made in
CHANGELOG.md, unless this PR is a small tweak which has no impact outside the development team. - [ ] Do any environment variables need amending or adding?
- [ ] Have any changes to the XML been checked with the IATI validator? See XML Validation
pr created time in a month
create barnchUKGovernmentBEIS/beis-report-official-development-assistance
branch : feature/998-show-roda-identifier-prefixes
created branch time in a month
pull request commentUKGovernmentBEIS/beis-report-official-development-assistance
(876) Upload transactions as CSV
@mec don't feel bad :) You've already caught some things I missed, any feedback is useful!
Also, if any of the team don't understand something I've done, I'd rather make it clearer than just leave something unmaintainable for others, so do shout if you have questions.
comment created time in a month
Pull request review commentUKGovernmentBEIS/beis-report-official-development-assistance
(876) Upload transactions as CSV
+# frozen_string_literal: true++require "csv"++class Staff::TransactionUploadsController < Staff::BaseController+ include Secured++ before_action :authorize_report++ def new+ @report_presenter = ReportPresenter.new(@report)+ end++ def show+ response.headers["Content-Type"] = "text/csv"+ response.headers["Content-Disposition"] = "attachment; filename=transactions.csv"++ csv = CSV.generate { |table|
Cool, I've amended that, thank you :)
comment created time in a month
push eventUKGovernmentBEIS/beis-report-official-development-assistance
commit sha 50c1d4755b326d5a87c77cc04ff021f9172cda14
Download a CSV template to begin bulk transaction upload To initiate the bulk upload process, a user downloads a CSV that contains the column headers we require in ImportTransactions. It lists out the RODA Identifier for each activity that should appear in the current report, meaning all the level C and D activities for the report's organisation and fund, that have a RODA Identifier. (The Delivery Partner Identifier and Name are also included in the download for informational purposes and are ignored on upload.) The tests include a third project that belongs to the same organisation, but to a different fund from the other two, to make sure it's not included in the download for this report.
commit sha ca47d7c5a00b2a321f96a267c7cc2b7f60c0bc53
Upload transactions in bulk as CSV This exposes the functionality of ImportTransactions via a file upload that accepts CSV. The CSV is parsed in the controller and the resulting rows fed into the importer, which itself is thoroughly unit-tested. If any errors occur, then no transactions are imported, and the errors are displayed to the user. If all the rows are valid, then the transactions are saved and a success message is displayed.
commit sha 62a06d775838b657784562251b6a9b82e388ff11
Changelog: Upload transaction data in bulk as CSV
push time in a month