• Craig Anderson

Testing your full Heroku stack (or: how to move fast without breaking things)

Modern software is complicated. Any software of a reasonable size has an array of dependencies, each of which have their own dependencies, and so on. An update to any of these dependencies can potentially break things in weird, unpredictable ways.



Unit tests ensure single components of this tower of dependencies work in isolation, and integration tests can help ensure sets of these components work together, but often we don’t bother testing the entire stack.


At My Data Chameleon, we think we've got a pretty robust solution to integration testing which exercises the entire stack. It involves:

  • Creating a copy of our production environment. Other than the data in the database and the sizes of attached services, this environment is identical to production. Heroku makes this simple, though it should be possible with any reproducible setup.

  • Run tests against that site. We test the important user journeys to ensure everything is operating as expected. We use Cypress to run these tests, but any tool which lets you to automate a web browser should work here.

This process has been really helpful, letting us deploy changes with a high degree of confidence, and we’re excited to share this approach with the world.


Our environment


Our solution makes use of:

However, this approach could be adapted for many other tools. We'd love to hear if you make use of a similar approach with a different setup.


But why? (Or: that time an update broke email)

Quite early on in the development of My Data Chameleon, we updated py-amqp (one of our dependencies’ dependencies’ dependencies) from version 5.0.2 to 5.0.3. Patch releases like this normally include security updates and bug fixes, so it’s a good idea to apply them when they’re available. This update made its way through our CI/CD pipeline, passing integration tests, and making its way onto our production environment.


It quickly became apparent the update wasn’t good. Anything which sent email needed to write a task to our task queue, and any attempt to write a task caused errors thanks to a bug introduced in that update.


We quickly rolled back that change and everything went back to normal, but it highlighted the need for better integration test coverage if we wanted to be able to apply security fixes with confidence.


This is the approach we came up with.


Our approach


I've broken our approach down into eight steps:


Also, you can check out the finished Cypress test and GitHub Actions workflow in our demo app's GitHub repository.

Step 1: Write some tests

I’m not going to go into how to write a Cypress test here—check out Cypress’ documentation for that—but here’s an example to show what's happening. This tests the functionality of our demo app which demonstrates how to use Django and HTMX together:



describe("Order a meal", () => {
  it("Successfully order a large vegetarian pizza", () => {
    cy.visit("/order-meal/");
    cy.get("[data-cy-meal]")
      .select("Large Vegetarian Pizza (serves 3)")
      .blurAndWaitForHTMX();
    cy.get("[data-cy-num-people]").clear();
    cy.get("[data-cy-num-people]")
      .type("6")
      .blurAndWaitForHTMX();
    cy.get("[data-cy-vegetarian]").click().blurAndWaitForHTMX();
    cy.get("[data-cy-order-button]").click();
    cy.contains("Order successfully placed!");
  });
});

You can see this test in our demo app's GitHub repository.

Step 2: Create the app


Next, we need to automate the process of creating our app. Once our app is set up for it, we can do this pretty easily using Heroku’s app setup API. To use this API, you need two important things:

  • A Heroku API key. Run heroku authorizations:create locally to create this. We store this in GitHub’s secret store as HEROKU_API_KEY.

  • A tarball of the code you want to deploy, somewhere on the internet. If you’re deploying from a public GitHub repository, a URL of the form https://api.github.com/repos/<username>/<repo name>/tarball/<commit hash>/ is perfect for this.

In GitHub Actions, creating a full stack looks like this:



# Call the app setup API to deploy the app.
TARBALL_URL="https://api.github.com/repos/${{ github.repository }}/tarball/${{ github.sha }}"
RESPONSE=`curl \
  --request POST https://api.heroku.com/app-setups \
  --header \
    "Authorization: Bearer ${{ secrets.HEROKU_API_KEY }}" \
  --header "Content-Type: application/json" \
  --header "Accept: application/vnd.heroku+json; version=3" \
  --data "{\"source_blob\": {\"url\": \"$TARBALL_URL\"}}" \
  --silent \
  --fail`

# Get the app setup ID and name from the response, then write
# them to environment variables.
HEROKU_SETUP_ID=`echo "$RESPONSE" | jq ".id" -r`
echo "HEROKU_SETUP_ID=$HEROKU_SETUP_ID" >> $GITHUB_ENV
HEROKU_APP=`echo "$RESPONSE" | jq ".app.name" -r`
echo "HEROKU_APP=$HEROKU_APP" >> $GITHUB_ENV


In this step we could also set environment variables, create the app in an organization, and more. Check out the app setup API documentation for details.


Note that at the end we get the setup ID and app name generated by Heroku and put them into environment variables. These will be important in the the next steps!

Step 3: Set up external services and configure the app (optional)


We’ve triggered an app setup in Heroku, but it won't be available for testing yet. This is a good opportunity to set up any external services and make any configuration changes which aren't possible via the app setup API.


In our example, we take this opportunity to enable Heroku's runtime dyno metadata feature.



heroku labs:enable runtime-dyno-metadata


To make use of the Heroku CLI tool as we do here, you'll need to set HEROKU_API_KEY and HEROKU_APP environment variables.

Step 4: Wait for the app

Depending on how much setup and configuration we needed to do in the previous step, we might still be waiting for our app to be ready. To check the app’s status, we call the app setup API again.

The following script polls the app setup API, waiting for the setup to move beyond "pending", then flagging an error If the status is anything other than "success".



STATUS="pending"
until [ "$STATUS" != "pending" ]
do
  sleep 1
  RESPONSE=`curl \
    https://api.heroku.com/app-setups/$HEROKU_SETUP_ID \
      --header \
        "Authorization: Bearer ${{ secrets.HEROKU_API_KEY }}" \
      --header "Content-Type: application/json" \
      --header \
        "Accept: application/vnd.heroku+json; version=3"  \
      --silent \
      --fail`
  STATUS=`echo "$RESPONSE" | jq ".status" -r`
done
echo "Heroku app setup complete."
echo "$RESPONSE" | jq
if [ "$STATUS" != "succeeded" ]
then
  exit 1
fi
  

Step 5: Run tests against the app


Once your stack has moved on from pending, it’s time to run your tests! Cypress offer guides on running tests in a number of CI providers , most of which are pretty straightforward to integrate into your existing setup.


In GitHub Actions, we run our tests using the following step:



- name: Run tests
  uses: cypress-io/github-action@v2
  with:
    wait-on: https://${{ env.HEROKU_APP }}.herokuapp.com
    config: baseUrl=https://${{ env.HEROKU_APP }}.herokuapp.com


It’s important to remember to pass in baseUrl here! This is the URL Cypress will run tests against. We’ve also told Cypress to wait for the home page of the site to be available—even though our Heroku app setup is no longer pending, it can take a little longer for the web server to be ready to accept requests.

Step 6: Report failures

At some point these tests will fail and we’ll want to know why.

Cypress produces videos and screenshots of failing test runs which can be super-useful when diagnosing problems. We can store these and make them available for download in GitHub Actions by saving them as artifacts:



- name: Save Cypress videos
  if: ${{ failure() }}
  uses: actions/upload-artifact@v2
  with:
    name: cypress-videos
    path: "cypress/videos/"

- name: Save Cypress screenshots
  if: ${{ failure() }}
  uses: actions/upload-artifact@v2
  with:
    name: cypress-screenshots
    path: "cypress/screenshots/"


It’s also often worth getting logs from Heroku to get an in-depth view of what’s happening behind the scenes:



- name: Get Logs from Heroku
  if: ${{ always() }}
  run: heroku logs --num 1500 > heroku-logs.txt

- name: Save Heroku Logs
  if: ${{ always() }}
  uses: actions/upload-artifact@v2
  with:
    name: heroku-logs
    path: "heroku-logs.txt"


Unfortunately this command will only retrieve the latest 1500 log entries. For a simple app this is more than enough, but for a more complicated application you’ll probably want to put something more robust in place.

Step 7: Clean up

Finally, we need to clean up and delete our app.



- name: Stop Heroku dynos
  if: ${{ always() }}
  run: heroku ps:scale web=0

- name: Destroy Heroku app
  if: ${{ always() }}
  run: heroku apps:destroy $HEROKU_APP --confirm $HEROKU_APP


In my experience, if it’s a good idea to stop your dynos before destroying your app. Destroying the app makes no guarantees about the order services are terminated in. If your app includes worker dynos which are regularly polling a message queue, this can trigger errors when the queue happens to be stopped before the worker dyno.

Step 8: Deploy with impunity!


With all of this is in place, and assuming your tests exercise all of your important user journeys, we can now be confident that the entire stack is being tested.


You can see a complete version of this GitHub Action workflow in our demo app's GitHub repository.

Next steps

This has been a pretty trivial example, but hopefully it's demonstrated how useful this approach can be.


There’s a lot more to the full stack tests we have built over time for My Data Chameleon. These tests now:

  • Run a daily check that we’re able to restore a database backup and recover from a disaster.

  • Run automated accessibility checks across most of the site.

  • Run snapshot tests across the site to ensure pages are rendered correctly.

  • Ensure our error handlers are configured correctly.

  • Run in parallel using Cypress dashboard.

If you have any questions, or are interested in hearing more about any of these next steps, please get in touch! You can reach Sharper Informatics Solutions on @SharperInfo, or get in touch with the author directly at @_craiga.