Integration testing across multiple repos with Codefresh

·

9 min read

Let's say you're like me and you happen to have a project that consists of PHP (🤮)... or an Angular app... or Golang... with some embedded react components... where those react components communicate with backend services.

Your goal is to make sure that changes to the backend don't break the front-end... that "fixing" a line in PHP code doesn't break the embedded applications (angular, react, or whatever) and you don't want to have six different pipelines running the same integration tests!

If you're like me... you also don't enjoy maintaining multiple integration test pipelines. And if you're still like me... you use Codefresh and Cypress already. So let's make the Codefresh pipelines a little more maintainable.

winning

You're like me

At a high level... we should have the following containers:

  • a bundled react app (built and served by nginx)
  • a PHP app
  • a backend proxy service
  • a backend internal service
  • (probably a database... auth... could be a lot more stuff...)
  • cypress integration tests

The PHP app pulls the react bundle in... the react app communicates with the proxy service... which communicates with the internal service.

And let's say that if you make a change to one codebase you want to run a Cypress test that does full end-to-end testing across all of them.

planning

Integration Test Planning...

Tech Used

Codefresh and Cypress are 🔥... go check out their sites... Their docs are well-written and I can't do them justice. I've been using Cypress for a while (this article isn't really about that) and I've been using Codefresh for ~3 weeks.

Codefresh is a CI/CD tool that is very easy to use with Docker/docker-compose (among other things).

Cypress is an end-to-end testing tool that is super-fast and super-easy to use.

Pipeline architecture

The goal is to have each repo build its own image, do its own unit testing and then that "primary" pipeline (the repo with the code change) should then kick off a separate integration testing pipeline that pulls in the other images.

If we were lazy... The easy thing to do would be to have the integration testing pipeline checkout all of the repos... build all the images... and run the tests... and copy-paste the ~100 lines of yaml in X different repos embedding it in the same unit test pipeline. Ultimately you'd have one less pipeline and 100s of lines of yaml duplicated throughout your codebase. It would be insanely slow.

The trick here is to make sure you're not rebuilding every image each time you run your integration tests... and the answer to that is to use the codefresh run command to pass in environment variables that include the tag the integration tests should be using.

Pipeline for the project being tested

Here's a simplified pipeline for the hypothetical react app that has some code changes.

Alt Text

... and the yaml ...

version: '1.0'
steps:
  # clone step...
  main_clone:
    stage: build
    git: github
    title: Cloning main repository...
    type: git-clone
    repo: '${{CF_REPO_OWNER}}/${{CF_REPO_NAME}}'
    revision: '${{CF_REVISION}}'

  # Dockerfile bundles the react app
  react_image:
    title: Building React app
    stage: build
    type: build
    image_name: '${{CF_REPO_OWNER}}/${{CF_REPO_NAME}}'
    dockerfile: Dockerfile
    tag: '${{CF_BRANCH_TAG_NORMALIZED_LOWER}}-${{CF_SHORT_REVISION}}'

  # e2e_test uses the codefresh cli to launch a separate pipeline
  e2e_test:
    title: E2E Tests
    stage: test
    image: 'codefresh/cli:latest'

    # the -v pass in separate environment variables to the next pipeline
    command: >-
      codefresh run codefresh_project_name/integration_pipeline -v
      CYPRESS_TAG=latest -v
      REACT_TAG=${{CF_BRANCH_TAG_NORMALIZED_LOWER}}-${{CF_SHORT_REVISION}} -v
      PHP_TAG=latest -v PROXY_TAG=latest -v INTERNAL_TAG=latest

  # PROBABLY some unit tests / other tests specific to this specific
  # project in here... probably done in parallel...

  # once the test passes while in master, tag this image with latest
  publish_stable:
    type: push
    stage: publish
    title: Tag Stable Project
    candidate: '${{react_image}}'
    tags:
      - STABLE
    image_name: martzcodes/project
    when:
      branch:
        only:
          - master
      steps:
        - name: e2e_test
          'on':
            - success
stages:
  - build
  - test
  - publish

The key here are the environment parameters being passed in to the integration pipeline. The image being tested is tagged with the branch name and short hash... everything else is using the 'latest' tag. See codefresh run docs for more information.

Integration pipeline

The integration pipeline uses those environment variables to pull the tag of the right images. The images that haven't been change use the 'latest' stable code, while the image under test uses the branch-short_hash tagging convention.

version: '1.0'
stages:
  - test
steps:
  e2e_composition:
    title: The Actual End-to-End tests
    stage: test
    type: composition
    description: Validate that one projects changes didn't break things
    fail_fast: true
    working_directory: ./
    composition_candidates:
        e2e_tests:
            image: path/to/cypress/tests:${{CYPRESS_TAG}}
            # sleep 30 is to wait for services to start up...
            # ...could use something like wait-for-it instead
            entrypoint: bash -c "sleep 30 && cypress run"
            # these are helpful for debugging, but optional
            environment:
              - 'DEBUG=cypress:electron'
              - ELECTRON_ENABLE_LOGGING=true
            depends_on:
              - react-app
              - php-app
              - proxy-service
              - internal-service
            networks:
              martzcodes: null

    # NOTE: A COMPOSITION IS BASICALLY A DOCKER-COMPOSE FILE
    composition:
      version: '2'
      services:
        react-app:
          image: path/to/react/app/image:${{REACT_TAG}}
          container_name: react-app
          networks:
            martzcodes:
        php-app:
            image: path/to/php/app/image:${{PHP_TAG}}
            container_name: php-app
            networks:
                martzcodes:
        proxy-service:
            image: path/to/proxy/service/image:${{PROXY_TAG}}
            container_name: proxy-service
            networks:
                martzcodes:
        internal-service:
            image: path/to/internal/service/image:${{INTERNAL_TAG}}
            container_name: internal-service
            networks:
                martzcodes:
      networks:
        martzcodes:

This is just a single step, but if you had other types of tests that you need to do across the entire app (performance testing, whatever)... you could do them in here as well.

One neat thing to note: that composition entry in the integration yaml file... that is the same structure as a docker-compose file. You can even store it separately (with a similar setup I can just do docker-compose -f codefresh-composition.yml up -d and be good to go... the only thing to be aware of is Codefresh and Docker use different string interpolations

Summary

The concept of a separate integration testing pipeline is very easy to extend... if you can logically separate your app (for example... several separate react apps inside a php app)... each could have their own integration chain... but if something in the php app were to change you'd want to test all of them.

A completely random example would be something like...

Alt Text

Surely this never happens in real life

This example would still be a lot of pipelines, but at least they'd be logically broken up and easier to re-use.

But there's a problem...

This breaks down if you're working on a story / doing multiple Pull Requests that go across several repos that depend on each other... I don't really have a good way around that.

thinking

Thinking...

If you have any suggestions for cases like that... let me know!