ci(editor): Run e2e tests in parallel and improve build caching (#5445)
* WIP: Cypress parallel CI run test * Trigger action on branch push * Change build artifacts path * Make sure to checkout the repo for testing job * Use Cypress action for installing * Lock cypress action userd version * Skip node install step since we're using cypress node16 container * Let Cypress handle pnpm install * Use setup-node action for caching pnpm * Set CYPRESS_CACHE_FOLDER * Set CYPRESS_CACHE_FOLDER * Manually cache pnpm store * Dont fix pnpm version * Use caching action also in testing job * Zip packages dist before uploading the artifacts and change caching key * Use absolute build paths for zipping job * Use zip command in action * Use tar for zipping packages * Debuggin directory ls * Debugging caching of modules * Attempt to fix permissions issue * Porivde Cypress executable via `CYPRESS_RUN_BINARY` * Cache /github/home * Adjust caching keys * Debug: search for cypress exec * Debugging: List dirs * Use pnpm install action to install node_modules * Do not log /home/runner * Use node_modules/.bin Cypress binary * Use absolute path to nodue modules * Run Cypress via custom command * Try with patched cypress action * Revert logging * Manually specify cypress config file * Use absolute paths * Fix cypress config name * Debug print cypress config * Remove debugging, increase to 4 containers * Increase amount of containers * Add env-version matrix * Replace node14 with node18 in testing matrix * Remove debugging and add node 14 * Use just node14 * Use cypress:base and remove browser req * Give more general timeouts * Try with node16 * Change cache directive position * Replace zip artifact upload with cache * Cache full packages not just dist * Test with variable inputs * Add commit info message * Remove wrongly commited code * Allow WF API dispatch * Try Chrome browser again for comparison * Include Monaco in the build * Make e2e workflow re-usable * Comment out invalid reusable workflow args * Use electron and add node 14 run * Fix env arg * Provide custom ci-build-id * Refactor remaining e2e workflow to use reusable action * Remove single matrix directive * Refactor ci-pull-req * Make lint job dependant on test jobs * Disable debugging job * Make containers dynamic * Cleanup & install git for linting action * Use regular buntu image for PR linting * Debugging failing tests * Remove fixed spec name * Debug e2e env var * Do not use realkeypress which crashes electron runner * Debugging * chore: remove console * chore: remove console * test: remove node 14 tests * test: replace test branch with master * test: use tests in current branch * test: use relative path * chore: clean up * test: only trigger on approval * ci: update test PR * ci: use curr branch * ci: only run 14 on schedule, not for slack command * ci: only run test on approval * ci: clean up branch, rename step * ci: rename steps * ci: clean up cancel * ci: clean up env var * ci: set var * ci: use chromef * ci: use electron * chore: add console log * chore: add console log * ci: update to string * ci: set all env options * test: build * ci: fix step issue * Fix failing tests & upgrade to Cypress 12 * Allow WF dispatch of e2e reusable * Fix wrong naming in e2e-tests workflow * Redeploy * Fix tests * Fix NDV tests and remove skipping of webhooks execution tests * Fix clipboard read command * Fix execution failing tests * Reset before each 15 and 3 * Fix flaky tests * Cleanup and log envs * Test fixes * Default owner spec fixes * Get rid of CYPRESS_RUN_ENV * Increase amount of containers, cleanup and add mock for credentials test call * Cleanup & fix PR tests unit tests * Wait for WF to loade in sharing spec * Do linting and unit tests first * Use frozen lockfile * Revert back ci pull request jobs order * Refine credential input selector and move cy.waitForLoad to correct position in 15-scheduler spec * test: build * Wait for WF execution instead of arbitraty timeout in WF execution spec, change order of jobs for ci pull request * Fix flaky 3-default owner spec and wait for execution list to load in 20-workflow-executions * Use setup node action * Remove caching for lint/unit tests * Experiment with parallel test & lint on ci * Provide cache key dynamically * Run e2e in parallel on pr * Only run node14 e2e on daily schedule * Make sure to generate generate new ci-build-id on re-runs * Remove debugging prints * Address PR comments * Rename custom onBeforeUnload handler * Make sure 19-execution spec waits for wf to load properly before import fixtures --------- Co-authored-by: Mutasem <mutdmour@gmail.com>
This commit is contained in:
OlegIvaniv
GitHub
105
.github/workflows/ci-pull-requests.yml
vendored
105
.github/workflows/ci-pull-requests.yml
vendored
@@ -1,27 +1,24 @@
|
||||
name: Test Pull Requests
|
||||
name: Build, unit/smoke test and lint branch
|
||||
|
||||
on: [pull_request]
|
||||
|
||||
jobs:
|
||||
build:
|
||||
install:
|
||||
name: Install & Build
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
timeout-minutes: 30
|
||||
|
||||
strategy:
|
||||
matrix:
|
||||
node-version: [14.x, 16.x]
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
with:
|
||||
repository: n8n-io/n8n
|
||||
ref: ${{ inputs.branch }}
|
||||
|
||||
- uses: pnpm/action-setup@v2.2.4
|
||||
|
||||
- name: Use Node.js ${{ matrix.node-version }}
|
||||
- name: Use Node.js 16
|
||||
uses: actions/setup-node@v3
|
||||
with:
|
||||
node-version: ${{ matrix.node-version }}
|
||||
cache: 'pnpm'
|
||||
node-version: 16.x
|
||||
cache: pnpm
|
||||
|
||||
- name: Install dependencies
|
||||
run: pnpm install --frozen-lockfile
|
||||
@@ -29,13 +26,74 @@ jobs:
|
||||
- name: Build
|
||||
run: pnpm build
|
||||
|
||||
- name: Cache build artifacts
|
||||
uses: actions/cache@v3
|
||||
with:
|
||||
path: |
|
||||
/github/home/.cache
|
||||
/github/home/.pnpm-store
|
||||
./node_modules
|
||||
./packages
|
||||
key: ${{ github.sha }}-base:16.18.1-test-lint
|
||||
|
||||
unit-test:
|
||||
name: Unit tests
|
||||
runs-on: ubuntu-latest
|
||||
needs: install
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
with:
|
||||
repository: n8n-io/n8n
|
||||
ref: ${{ inputs.branch }}
|
||||
|
||||
- name: Restore cached build artifacts
|
||||
uses: actions/cache@v3
|
||||
with:
|
||||
path: |
|
||||
/github/home/.cache
|
||||
/github/home/.pnpm-store
|
||||
./node_modules
|
||||
./packages
|
||||
key: ${{ github.sha }}-base:16.18.1-test-lint
|
||||
|
||||
- uses: pnpm/action-setup@v2.2.4
|
||||
|
||||
- name: Use Node.js 16
|
||||
uses: actions/setup-node@v3
|
||||
with:
|
||||
node-version: 16.x
|
||||
cache: pnpm
|
||||
|
||||
- name: Test
|
||||
run: pnpm test
|
||||
|
||||
- name: Test E2E
|
||||
run: |
|
||||
pnpm cypress:install
|
||||
pnpm test:e2e:smoke
|
||||
lint:
|
||||
name: Lint changes
|
||||
runs-on: ubuntu-latest
|
||||
needs: install
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
with:
|
||||
repository: n8n-io/n8n
|
||||
ref: ${{ inputs.branch }}
|
||||
|
||||
- name: Restore cached build artifacts
|
||||
uses: actions/cache@v3
|
||||
with:
|
||||
path: |
|
||||
/github/home/.cache
|
||||
/github/home/.pnpm-store
|
||||
./node_modules
|
||||
./packages
|
||||
key: ${{ github.sha }}-base:16.18.1-test-lint
|
||||
|
||||
- uses: pnpm/action-setup@v2.2.4
|
||||
|
||||
- name: Use Node.js 16
|
||||
uses: actions/setup-node@v3
|
||||
with:
|
||||
node-version: 16.x
|
||||
cache: pnpm
|
||||
|
||||
- name: Fetch base branch for `git diff`
|
||||
run: git fetch origin ${{ github.event.pull_request.base.ref }}:${{ github.event.pull_request.base.ref }}
|
||||
@@ -44,3 +102,18 @@ jobs:
|
||||
env:
|
||||
ESLINT_PLUGIN_DIFF_COMMIT: ${{ github.event.pull_request.base.ref }}
|
||||
run: pnpm lint
|
||||
|
||||
smoke-test:
|
||||
name: E2E [Electron/Node 16]
|
||||
uses: ./.github/workflows/e2e-reusable.yml
|
||||
if: ${{ !contains(github.event.pull_request.labels.*.name, 'skip-e2e') }}
|
||||
with:
|
||||
branch: ${{ github.event.pull_request.base.ref }}
|
||||
user: ${{ github.event.inputs.user || 'PR User' }}
|
||||
spec: ${{ github.event.inputs.spec || 'e2e/0-smoke.cy.ts' }}
|
||||
run-env: base:16.18.1
|
||||
record: false
|
||||
parallel: false
|
||||
containers: '[1]'
|
||||
secrets:
|
||||
CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}
|
||||
|
||||
150
.github/workflows/e2e-reusable.yml
vendored
Normal file
150
.github/workflows/e2e-reusable.yml
vendored
Normal file
@@ -0,0 +1,150 @@
|
||||
name: Reusable e2e workflow
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
inputs:
|
||||
branch:
|
||||
description: 'GitHub branch to test.'
|
||||
required: false
|
||||
type: string
|
||||
default: 'master'
|
||||
user:
|
||||
description: 'User who kicked this off.'
|
||||
required: false
|
||||
type: string
|
||||
default: 'schedule'
|
||||
spec:
|
||||
description: 'Specify specs.'
|
||||
required: false
|
||||
default: 'e2e/*'
|
||||
type: string
|
||||
run-env:
|
||||
description: 'Node env version to run tests with.'
|
||||
required: false
|
||||
default: 'browsers:node16.18.0-chrome90-ff88'
|
||||
type: string
|
||||
cache-key:
|
||||
description: 'Cache key for modules and build artifacts.'
|
||||
required: false
|
||||
default: ${{ github.sha }}-${{ inputs.run-env }}-e2e-modules
|
||||
type: string
|
||||
record:
|
||||
description: 'Record test run.'
|
||||
required: false
|
||||
default: true
|
||||
type: boolean
|
||||
parallel:
|
||||
description: 'Run tests in parallel.'
|
||||
required: false
|
||||
default: true
|
||||
type: boolean
|
||||
containers:
|
||||
description: 'Number of containers to run tests in.'
|
||||
required: false
|
||||
default: '[1, 2, 3, 4, 5, 6, 7, 8]'
|
||||
type: string
|
||||
secrets:
|
||||
CYPRESS_RECORD_KEY:
|
||||
description: 'Cypress record key.'
|
||||
required: true
|
||||
|
||||
jobs:
|
||||
# single job that generates and outputs a common id
|
||||
prepare:
|
||||
runs-on: ubuntu-latest
|
||||
outputs:
|
||||
uuid: ${{ steps.uuid.outputs.value }}
|
||||
steps:
|
||||
- name: Generate unique ID 💎
|
||||
id: uuid
|
||||
# take the current commit + timestamp together
|
||||
# the typical value would be something like
|
||||
# "sha-5d3fe...35d3-time-1620841214"
|
||||
run: echo "value=sha-$GITHUB_SHA-time-$(date +"%s")" >> $GITHUB_OUTPUT
|
||||
|
||||
install:
|
||||
runs-on: ubuntu-latest
|
||||
needs: ['prepare']
|
||||
container:
|
||||
image: cypress/${{ inputs.run-env }}
|
||||
options: --user 1001
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
with:
|
||||
repository: n8n-io/n8n
|
||||
ref: ${{ inputs.branch }}
|
||||
|
||||
- name: Setup pnpm
|
||||
uses: pnpm/action-setup@v2.2.4
|
||||
with:
|
||||
run_install: true
|
||||
|
||||
- name: Cache pnpm modules
|
||||
uses: actions/cache@v3
|
||||
with:
|
||||
path: |
|
||||
/github/home/.cache
|
||||
/github/home/.pnpm-store
|
||||
./node_modules
|
||||
./packages
|
||||
key: ${{ inputs.cache-key }}
|
||||
|
||||
- name: Cypress build
|
||||
uses: cypress-io/github-action@v5
|
||||
with:
|
||||
# Disable running of tests within install job
|
||||
runTests: false
|
||||
install: false
|
||||
build: pnpm build
|
||||
|
||||
- name: Cypress install
|
||||
run: pnpm cypress:install
|
||||
|
||||
testing:
|
||||
runs-on: ubuntu-latest
|
||||
container:
|
||||
image: cypress/${{ inputs.run-env }}
|
||||
options: --user 1001
|
||||
needs: ['prepare', 'install']
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
containers: ${{ fromJSON(inputs.containers) }}
|
||||
steps:
|
||||
- uses: actions/checkout@v3
|
||||
with:
|
||||
repository: n8n-io/n8n
|
||||
ref: ${{ inputs.branch }}
|
||||
|
||||
- name: Setup pnpm
|
||||
uses: pnpm/action-setup@v2.2.4
|
||||
|
||||
- name: Restore cached pnpm modules
|
||||
uses: actions/cache@v3
|
||||
with:
|
||||
path: |
|
||||
/github/home/.cache
|
||||
/github/home/.pnpm-store
|
||||
./node_modules
|
||||
./packages
|
||||
key: ${{ inputs.cache-key }}
|
||||
|
||||
- name: Cypress run
|
||||
uses: cypress-io/github-action@v5
|
||||
with:
|
||||
install: false
|
||||
start: pnpm start
|
||||
wait-on: 'http://localhost:5678'
|
||||
wait-on-timeout: 120 #
|
||||
record: ${{ inputs.record }}
|
||||
parallel: ${{ inputs.parallel }}
|
||||
# We have to provide custom ci-build-id key to make sure that this workflow could be run multiple times
|
||||
# in the same parent workflow
|
||||
ci-build-id: ${{ needs.prepare.outputs.uuid }}
|
||||
spec: "/__w/n8n/n8n/cypress/${{ inputs.spec }}"
|
||||
config-file: /__w/n8n/n8n/cypress.config.js
|
||||
env:
|
||||
CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
E2E_TESTS: true
|
||||
COMMIT_INFO_MESSAGE: 🌳 ${{ inputs.branch }} 🖥️ ${{ inputs.run-env }} 🤖 ${{ inputs.user }} 🗃️ ${{ inputs.spec }}
|
||||
71
.github/workflows/e2e-tests-pr.yml
vendored
71
.github/workflows/e2e-tests-pr.yml
vendored
@@ -7,53 +7,28 @@ on:
|
||||
- 'master'
|
||||
|
||||
jobs:
|
||||
runTests:
|
||||
name: 'Run E2E tests'
|
||||
runs-on: ubuntu-latest
|
||||
if: ${{ !contains(github.event.pull_request.labels.*.name, 'skip-e2e') }}
|
||||
# We disable this for now because cancelling runs makes the Cypress Cloud tests to hang.
|
||||
# cancel-previous-runs:
|
||||
# runs-on: ubuntu-latest
|
||||
# name: 'Cancel previous e2e test runs'
|
||||
# strategy:
|
||||
# matrix:
|
||||
# node-version: [16.x]
|
||||
|
||||
timeout-minutes: 30
|
||||
# steps:
|
||||
# - name: 'Cancel previous runs'
|
||||
# uses: styfle/cancel-workflow-action@0.9.0
|
||||
# with:
|
||||
# access_token: ${{ github.token }}
|
||||
|
||||
strategy:
|
||||
matrix:
|
||||
node-version: [16.x]
|
||||
|
||||
steps:
|
||||
- name: 'Cancel previous runs'
|
||||
uses: styfle/cancel-workflow-action@0.9.0
|
||||
with:
|
||||
access_token: ${{ github.token }}
|
||||
|
||||
- name: 'Cancel if PR is not approved'
|
||||
if: github.event.review.state != 'approved'
|
||||
uses: andymckay/cancel-action@0.2
|
||||
|
||||
- uses: actions/checkout@v3
|
||||
|
||||
- uses: pnpm/action-setup@v2.2.4
|
||||
|
||||
- name: Use Node.js ${{ matrix.node-version }}
|
||||
uses: actions/setup-node@v3
|
||||
with:
|
||||
node-version: ${{ matrix.node-version }}
|
||||
cache: 'pnpm'
|
||||
|
||||
- name: Install dependencies
|
||||
run: pnpm install --frozen-lockfile
|
||||
|
||||
- name: Build
|
||||
run: pnpm build
|
||||
|
||||
- name: Test E2E
|
||||
run: |
|
||||
pnpm cypress:install
|
||||
pnpm test:e2e:all
|
||||
|
||||
- name: Upload artifacts
|
||||
uses: actions/upload-artifact@v3
|
||||
if: failure()
|
||||
with:
|
||||
name: cypress-screenshots
|
||||
path: |
|
||||
cypress/screenshots
|
||||
retention-days: 1
|
||||
run-e2e-tests:
|
||||
name: E2E [Electron/Node 16]
|
||||
uses: ./.github/workflows/e2e-reusable.yml
|
||||
if: ${{ github.event.review.state == 'approved' && !contains(github.event.pull_request.labels.*.name, 'skip-e2e') }}
|
||||
with:
|
||||
branch: ${{ github.event.pull_request.head.ref }}
|
||||
user: ${{ github.event.pull_request.user.login || 'PR User' }}
|
||||
spec: 'e2e/*'
|
||||
run-env: base:16.18.1
|
||||
secrets:
|
||||
CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}
|
||||
|
||||
72
.github/workflows/e2e-tests.yml
vendored
72
.github/workflows/e2e-tests.yml
vendored
@@ -24,54 +24,46 @@ on:
|
||||
default: ''
|
||||
|
||||
jobs:
|
||||
build:
|
||||
calls-start-url:
|
||||
name: Calls start URL
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
timeout-minutes: 30
|
||||
|
||||
strategy:
|
||||
matrix:
|
||||
node-version: [14.x, 16.x]
|
||||
|
||||
if: ${{ github.event.inputs.start-url != '' }}
|
||||
steps:
|
||||
- name: Call Start URL - optionally
|
||||
- name: Calls start URL
|
||||
run: |
|
||||
[[ "${{github.event.inputs.start-url}}" != "" ]] && curl -v -X POST -d 'url=${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}' ${{github.event.inputs.start-url}} || echo ""
|
||||
shell: bash
|
||||
|
||||
- uses: actions/checkout@v3
|
||||
with:
|
||||
repository: ${{ github.event.inputs.repository || 'n8n-io/n8n' }}
|
||||
ref: ${{ github.event.inputs.branch || 'master' }}
|
||||
run-e2e-tests:
|
||||
name: E2E [Electron/Node 16]
|
||||
uses: ./.github/workflows/e2e-reusable.yml
|
||||
if: ${{ !contains(github.event.pull_request.labels.*.name, 'skip-e2e') }}
|
||||
with:
|
||||
branch: ${{ github.event.inputs.branch || 'master' }}
|
||||
user: ${{ github.event.inputs.user || 'PR User' }}
|
||||
spec: ${{ github.event.inputs.spec || 'e2e/*' }}
|
||||
run-env: base:16.18.1
|
||||
secrets:
|
||||
CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}
|
||||
|
||||
- uses: pnpm/action-setup@v2.2.4
|
||||
|
||||
- name: Use Node.js ${{ matrix.node-version }}
|
||||
uses: actions/setup-node@v3
|
||||
with:
|
||||
node-version: ${{ matrix.node-version }}
|
||||
cache: 'pnpm'
|
||||
|
||||
- name: Install dependencies
|
||||
run: pnpm install --frozen-lockfile
|
||||
|
||||
- name: Build
|
||||
run: pnpm build
|
||||
|
||||
- name: Test E2E
|
||||
run: |
|
||||
pnpm cypress:install
|
||||
pnpm test:e2e:all
|
||||
|
||||
- name: Upload artifacts
|
||||
uses: actions/upload-artifact@v3
|
||||
if: failure()
|
||||
with:
|
||||
name: cypress-screenshots
|
||||
path: |
|
||||
cypress/screenshots
|
||||
retention-days: 1
|
||||
run-e2e-tests-node-14:
|
||||
name: E2E [Electron/Node 14]
|
||||
uses: ./.github/workflows/e2e-reusable.yml
|
||||
if: ${{ github.event_name == 'schedule' }}
|
||||
with:
|
||||
branch: ${{ github.event.inputs.branch || 'master' }}
|
||||
user: ${{ github.event.inputs.user || 'schedule' }}
|
||||
spec: ${{ github.event.inputs.spec || 'e2e/*' }}
|
||||
run-env: base:14.21.1
|
||||
secrets:
|
||||
CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}
|
||||
|
||||
calls-success-url-notify:
|
||||
name: Calls success URL and notifies
|
||||
runs-on: ubuntu-latest
|
||||
needs: [run-e2e-tests, run-e2e-tests-node-14]
|
||||
if: ${{ github.event.inputs.success-url != '' }}
|
||||
steps:
|
||||
- name: Notify Slack on failure
|
||||
uses: act10ns/slack@v2.0.0
|
||||
if: failure()
|
||||
|
||||
Reference in New Issue
Block a user