Has anyone had any success trying to setup parallel tests on Heroku CI? Currently having issues with setting up the databases because Heroku ignores your database.yml

Although this approach works perfectly, it doesn't make use of multiple CPUs in performance dynos and runs only a single process in each dyno. To be able to run multiple processes I took a different approach.

As others have mentioned, Heroku builds a default database and sets DATABASE_URL, which overrides your database.yml configuration. This causes errors in rake parallel:setup. So instead, I unset the DATABASE_URL and clone the default database in a bash script. Then I split the tests in according to number of CPUs and the number of dynos.

Here is my configuration:

// app.json
{
  ...
  "environments": {
    "test": {
      "formation": {
        "test": {
          "quantity": 4,
          "size": "performance-l"
        }
      },
      "env": {  "POSTGRESQL_VERSION": "11"  },
      "addons": ["heroku-postgresql:in-dyno", "heroku-redis:in-dyno"],
      "scripts": {
        "test": "bash ./script/test"
      }
    }
  }
}

# test/script

#!/bin/bash

concurrency=$( nproc --all )
echo "-----> Found ${concurrency} CPUs"

spec_node_count=$(( $CI_NODE_TOTAL ))
process_count=$(( $spec_node_count * $concurrency ))

# Heroku buildpack creates a default database and sets $DATABASE_URL pointing to that database
# That means tests running on Heroku CI will ignore database.yml settings
# Also running rake parallel:setup somehow can not create the databases and causes errors
#
# The script below simply clones the default database provided by Heroku
# and then splits the tests into groups so that we can have as many processes as the number of CPUs

regex="^postgres:\/\/([a-zA-Z0-9]+):([a-zA-Z0-9]+)@([a-zA-Z0-9]+):([0-9]+)/([a-zA-Z0-9_]+)"

if [[ $DATABASE_URL =~ $regex ]]
then
  export DB_USERNAME="${BASH_REMATCH[1]}" # random value created by the buildpack
  export DB_PASSWORD="${BASH_REMATCH[2]}" # random value created by the buildpack
  export DB_HOST="${BASH_REMATCH[3]}" # most probably localhost
  export DB_PORT="${BASH_REMATCH[4]}" # most probably 5432

  buildpack_db_name="${BASH_REMATCH[5]}" # most probably postgres_buildpack_db
  export DB_NAME_PREFIX=$buildpack_db_name

  unset DATABASE_URL # we unset this, so that Rails uses database.yml settings

  # Create new databases using the default database as the template. One database per process
  # This is similar to rake parallel:setup
  # It should create databases such as postgres_buildpack_db2, postgres_buildpack_db3, ...
  echo "-----> Creating test databases for paralel testing"

  for (( idx=2; idx<=$concurrency; idx++ ))
  do
    createdb -O $DB_USERNAME -T $buildpack_db_name ${DB_NAME_PREFIX}${idx}
    echo "       Database ${DB_NAME_PREFIX}${idx} created"
  done

  # Find out which groups will be run on this machine
  # e.g. With 2 performance-l dynos,
  # dyno 1 should run groups 0,1,2,3,4,5,6,7
  # dyno 2 should run groups 8,9,10,11,12,13,14,15
  process_index_start=$(( $concurrency * $CI_NODE_INDEX ))
  process_index_end=$(( process_index_start + concurrency ))

  groups=$process_index_start
  for (( idx=$process_index_start + 1; idx<$process_index_end; idx++ ))
  do
    groups+=",${idx}"
  done

  echo "-----> Running groups ${groups} from ${process_count} processes"
  bundle exec parallel_rspec spec/ -n $process_count --only-group $groups
else
    echo "Unexpected DATABASE_URL format"
fi

# database.yml

test:
  <<: *default
  database: <%= ENV['DB_NAME_PREFIX'] || 'parasut_test' %><%= ENV['TEST_ENV_NUMBER'] %>
  pool: 5
  host: <%= ENV['DB_HOST'] || '127.0.0.1' %>
  port: <%= ENV['DB_PORT'] || 5432 %>
  username: <%= ENV['DB_USERNAME'] %>
  password: <%= ENV['DB_PASSWORD'] %>

As a result, our test suite, which took around 15 minutes on 4 performance-m dynos, now finishes in around 3 minutes with 4 performance-l dynos.