Last updated August 10, 2022

This article describes the sequence of events performed by Heroku CI during a test run. The details of this sequence can be helpful if you are debugging a failing test run, or if you are adding Heroku CI support to a custom development tool or Heroku buildpack.

Step 1: A CI-specific copy of your app is provisioned

• The app copy is provisioned on a Performance-M dyno (see Dyno Types for details).
• Note that the CI app does not appear as a separate app in your pipeline’s details in the Heroku Dashboard.

Step 2: Required add-ons are provisioned for the CI app

• You specify the add-ons your app requires in your app’s app.json file, as shown in Provisioning add-ons.

Step 3: Applicable buildpacks are invoked

• If you specify custom buildpacks for your app in its app.json file, those buildpacks are invoked in the listed order.
• If you don’t specify custom buildpacks, Heroku CI automatically determines which officially supported buildpack to invoke for your app.
• If a particular buildpack includes a bin/test-compile script, Heroku CI runs it. Otherwise, Heroku CI runs bin/compile.
• If any buildpack exits with a non-zero exit code, the test run is flagged as errored.

Step 4: The test-setup command is executed

• If your app’s app.json file specifies a scripts/test-setup script (as shown in Specifying custom test commands), Heroku CI executes it from the $HOME directory.
• If the test-setup script exits with a non-zero exit code, the test run is flagged as errored.
• If the test-setup script completes successfully, the setup artifacts will be cached and re-used for the next run.

Step 5: The test run is executed

• If your app’s app.json file specifies a scripts/test script (as shown in Specifying custom test commands), it is run in the $HOME directory.
• If you don’t specify a scripts/test script, Heroku CI instead runs each associated buildpack’s bin/test script.
• If any executed script exits with a non-zero status, the test run is flagged as a failure.
• If all test scripts exit with a zero status, the test run is flagged as a success.
• If the executed test scripts emit TAP-formatted output to STDOUT, individual passes and failures are counted and displayed in the test run’s UI.
• If the executed test scripts emit TAP-formatted output to STDOUT, failures in the tests take precedence over the exit code for the test run status.

Step 6: Test run results are reported

• Your app’s buildpacks stream output from compilation, test-setup scripts, and the test runs themselves. This output is presented in the Tests tab of your pipeline’s Heroku Dashboard page.
• If a buildpack or test script does not support TAP format, the result of the entire test run is presented as either red (failure) or green (success).
• Results from every test run are stored at a permanent URL in the Heroku Dashboard.

Step 7: Cleanup

• Test runs that are still executing after one hour are terminated automatically.
• The CI app is deprovisioned and deleted, along with any associated add-ons.