This is related to an issue with how Bull handles stalled jobs, see
https://github.com/OptimalBits/bull/issues/1415 for reference.
CPU intensive workflows can in certain cases take a long while to finish
up, thereby blocking the thread and causing Bull queue to think the job
has stalled, even though it finished successfully. In these cases the
error handling could then overwrite the successful execution data with
the error message.
In a rare edge case an undefined queue could be returned - this should
not happen and now an error is thrown.
Also using the opportunity to remove a cyclic dependency from the Queue.
Based on #7065 | Story: https://linear.app/n8n/issue/PAY-771
n8n on filesystem mode marks binary data to delete on manual execution
deletion, on unsaved execution completion, and on every execution
pruning cycle. We later prune binary data in a separate cycle via these
marker files, based on the configured TTL. In the context of introducing
an S3 client to manage binary data, the filesystem mode's mark-and-prune
setup is too tightly coupled to the general binary data management
client interface.
This PR...
- Ensures the deletion of an execution causes the deletion of any binary
data associated to it. This does away with the need for binary data TTL
and simplifies the filesystem mode's mark-and-prune setup.
- Refactors all execution deletions (including pruning) to cause soft
deletions, hard-deletes soft-deleted executions based on the existing
pruning config, and adjusts execution endpoints to filter out
soft-deleted executions. This reduces DB load, and keeps binary data
around long enough for users to access it when building workflows with
unsaved executions.
- Moves all execution pruning work from an execution lifecycle hook to
`execution.repository.ts`. This keeps related logic in a single place.
- Removes all marking logic from the binary data manager. This
simplifies the interface that the S3 client will meet.
- Adds basic sanity-check tests to pruning logic and execution deletion.
Out of scope:
- Improving existing pruning logic.
- Improving existing execution repository logic.
- Adjusting dir structure for filesystem mode.
---------
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
# Motivation
In Queue mode, finished executions would cause the main instance to
always pull all execution data from the database, unflatten it and then
use it to send out event log events and telemetry events, as well as
required returns to Respond to Webhook nodes etc.
This could cause OOM errors when the data was large, since it had to be
fully unpacked and transformed on the main instance’s side, using up a
lot of memory (and time).
This PR attempts to limit this behaviour to only happen in those
required cases where the data has to be forwarded to some waiting
webhook, for example.
# Changes
Execution data is only required in cases, where the active execution has
a `postExecutePromise` attached to it. These usually forward the data to
some other endpoint (e.g. a listening webhook connection).
By adding a helper `getPostExecutePromiseCount()`, we can decide that in
cases where there is nothing listening at all, there is no reason to
pull the data on the main instance.
Previously, there would always be postExecutePromises because the
telemetry events were called. Now, these have been moved into the
workers, which have been given the various InternalHooks calls to their
hook function arrays, so they themselves issue these telemetry and event
calls.
This results in all event log messages to now be logged on the worker’s
event log, as well as the worker’s eventbus being the one to send out
the events to destinations. The main event log does…pretty much nothing.
We are not logging executions on the main event log any more, because
this would require all events to be replicated 1:1 from the workers to
the main instance(s) (this IS possible and implemented, see the worker’s
`replicateToRedisEventLogFunction` - but it is not enabled to reduce the
amount of traffic over redis).
Partial events in the main log could confuse the recovery process and
would result in, ironically, the recovery corrupting the execution data
by considering them crashed.
# Refactor
I have also used the opportunity to reduce duplicate code and move some
of the hook functionality into
`packages/cli/src/executionLifecycleHooks/shared/sharedHookFunctions.ts`
in preparation for a future full refactor of the hooks
* first commit for postgres migration
* (not working)
* sqlite migration
* quicksave
* fix tests
* fix pg test
* fix postgres
* fix variables import
* fix execution saving
* add user settings fix
* change migration to single lines
* patch preferences endpoint
* cleanup
* improve variable import
* cleanup unusued code
* Update packages/cli/src/PublicApi/v1/handlers/workflows/workflows.handler.ts
Co-authored-by: Omar Ajoue <krynble@gmail.com>
* address review notes
* fix var update/import
* refactor: Separate execution data to its own table (#6323)
* wip: Temporary migration process
* refactor: Create boilerplate repository methods for executions
* fix: Lint issues
* refactor: Added search endpoint to repository
* refactor: Make the execution list work again
* wip: Updating how we create and update executions everywhere
* fix: Lint issues and remove most of the direct access to execution model
* refactor: Remove includeWorkflowData flag and fix more tests
* fix: Lint issues
* fix: Fixed ordering of executions for FE, removed transaction when saving execution and removed unnecessary update
* refactor: Add comment about missing feature
* refactor: Refactor counting executions
* refactor: Add migration for other dbms and fix issues found
* refactor: Fix lint issues
* refactor: Remove unnecessary comment and auto inject repo to internal hooks
* refactor: remove type assertion
* fix: Fix broken tests
* fix: Remove unnecessary import
* Remove unnecessary toString() call
Co-authored-by: Iván Ovejero <ivov.src@gmail.com>
* fix: Address comments after review
* refactor: Remove unused import
* fix: Lint issues
* fix: Add correct migration files
---------
Co-authored-by: Iván Ovejero <ivov.src@gmail.com>
* remove null values from credential export
* fix: Fix an issue with queue mode where all running execution would be returned
* fix: Update n8n node to allow for workflow ids with letters
* set upstream on set branch
* remove typo
* add nodeAccess to credentials
* fix unsaved run check for undefined id
* fix(core): Rename version control feature to source control (#6480)
* rename versionControl to sourceControl
* fix source control tooltip wording
---------
Co-authored-by: Romain Minaud <romain.minaud@gmail.com>
* fix(editor): Pay 548 hide the set up version control button (#6485)
* feat(DebugHelper Node): Fix and include in main app (#6406)
* improve node a bit
* fixing continueOnFail() ton contain error in json
* improve pairedItem
* fix random data returning object results
* fix nanoId length typo
* update pnpm-lock file
---------
Co-authored-by: Marcus <marcus@n8n.io>
* fix(editor): Remove setup source control CTA button
* fix(editor): Remove setup source control CTA button
---------
Co-authored-by: Michael Auerswald <michael.auerswald@gmail.com>
Co-authored-by: Marcus <marcus@n8n.io>
* fix(editor): Update source control docs links (#6488)
* feat(DebugHelper Node): Fix and include in main app (#6406)
* improve node a bit
* fixing continueOnFail() ton contain error in json
* improve pairedItem
* fix random data returning object results
* fix nanoId length typo
* update pnpm-lock file
---------
Co-authored-by: Marcus <marcus@n8n.io>
* feat(editor): Replace root events with event bus events (no-changelog) (#6454)
* feat: replace root events with event bus events
* fix: prevent cypress from replacing global with globalThis in import path
* feat: remove emitter mixin
* fix: replace component events with event bus
* fix: fix linting issue
* fix: fix breaking expression switch
* chore: prettify ndv e2e suite code
* fix(editor): Update source control docs links
---------
Co-authored-by: Michael Auerswald <michael.auerswald@gmail.com>
Co-authored-by: Marcus <marcus@n8n.io>
Co-authored-by: Alex Grozav <alex@grozav.com>
* fix tag endpoint regex
---------
Co-authored-by: Omar Ajoue <krynble@gmail.com>
Co-authored-by: Iván Ovejero <ivov.src@gmail.com>
Co-authored-by: Romain Minaud <romain.minaud@gmail.com>
Co-authored-by: Csaba Tuncsik <csaba@n8n.io>
Co-authored-by: Marcus <marcus@n8n.io>
Co-authored-by: Alex Grozav <alex@grozav.com>
* use typedi for UserManagementMailer
* use typedi for SamlService
* fix typos
* use typedi for Queue
* use typedi for License
* convert some more code to use typedi
* add typedi
* convert ActiveWorkflowRunner into an injectable service
* convert ExternalHooks into an injectable service
* convert InternalHooks into an injectable service
* convert LoadNodesAndCredentials into an injectable service
* convert NodeTypes and CredentialTypes into an injectable service
* convert ActiveExecutions into an injectable service
* convert WaitTracker into an injectable service
* convert Push into an injectable service
* convert ActiveWebhooks and TestWebhooks into an injectable services
* handle circular references, and log errors when a circular dependency is found
* adds ExecutionEvents view modal to ExecutionList
* fix time rendering and remove wf column
* checks for unfinished executions and fails them
* prevent re-setting stoppedAt for execution
* some cleanup / manually create rundata after crash
* quicksave
* remove Threads lib, log worker rewrite
* cleanup comment
* fix sentry destination return value
* test for tests...
* run tests with single worker
* fix tests
* remove console log
* add endpoint for execution data recovery
* lint cleanup and some refactoring
* fix accidental recursion
* remove cyclic imports
* add rundata recovery to Workflowrunner
* remove comments
* cleanup and refactor
* adds a status field to executions
* setExecutionStatus on queued worker
* fix onWorkflowPostExecute
* set waiting from worker
* get crashed status into frontend
* remove comment
* merge fix
* cleanup
* catch empty rundata in recovery
* refactor IExecutionsSummary and inject nodeExecution Errors
* reduce default event log size to 10mb from 100mb
* add per node execution status
* lint fix
* merge and lint fix
* phrasing change
* improve preview rendering and messaging
* remove debug
* Improve partial rundata recovery
* fix labels
* fix line through
* send manual rundata to ui at crash
* some type and msg push fixes
* improve recovered item rendering in preview
* update workflowStatistics on recover
* merge fix
* review fixes
* merge fix
* notify eventbus when ui is back up
* add a small timeout to make sure the UI is back up
* increase reconnect timeout to 30s
* adjust recover timeout and ui connection lost msg
* do not stop execution in editor after x reconnects
* add executionRecovered push event
* fix recovered connection not green
* remove reconnect toast and merge existing rundata
* merge editor and recovered data for own mode
* feat(cli): Setup error tracking using Sentry
* make error reporting available in the workflows package
* address some of the PR comments
* create a ErrorReporterProxy like LoggerProxy
* remove the `captureError` helper. use ErrorReporterProxy directly
* fix linting issues
* remove ErrorReporterProxy warnings in tests
* check for NODE_ENV === 'production' instead
* IErrorReporter -> ErrorReporter
* ErrorReporterProxy.getInstance() -> ErrorReporter
* allow capturing stacks in warnings as well
* make n8n debugging consistent with `npm start`
* IReportingOptions -> ReportingOptions
* use consistent signature for `error` and `warn`
* use Logger instead of console.log
* 🎉 starting feature development
* ✨ sharing/unsharing a credential (#3601)
* 🎉 initial design
* ✨ sharing/unsharing of credentials
* ✅ add tests for EE credentials controller
* 💪 implement review comments
* 🛠 refactor agent creation and credential role locking
* 👕 linting adjustments (#3691)
* 👕 Adjust rule `naming-convention`
* 👕 Fix `naming-convention` config value
* 👕 Disregard casing for EE-prefixed vars
Co-authored-by: Iván Ovejero <ivov.src@gmail.com>
* 🛠 refactor authAgents in tests (#3725)
* 🛠 refactor authAgent
* 🛠 refactor authAgent
* 🛠 refactor authAgent
* 🛠 refactor authAgent
* 👕 fix ts issue
* 🐘 add migration for mysql and postgres + add AuthAgent type
Co-authored-by: Iván Ovejero <ivov.src@gmail.com>
* ⚡️ refactor existing credentials routes (#3672)
* 🎉 initial design
* ✨ sharing/unsharing of credentials
* ♻️ split credential update route into controller and service
* 🔥 remove credentials test that is no longer applicable
* ♻️ split credential creation route into controller and service
* ♻️ split single credential get
* ♻️ split delete credentials route
* ♻️ split get all credentials route
* 🔥 remove unused imports in credentials contoller
* 🔥 remove console.log
* :refactor: changes to credentials controller and service from review
- removed credentials from service function names
- made relations list optional
- put allowGlobalOwner in options objects
- check length of relations array so join doesn't happen if empty
- update some comments to further explain rationale
- remove unneeded `Object.assign`
- remove non-null assertion from test
* ♻️ move filtered credentials selected fields to variable
* ♻️ remove unneeded merges in credentials service
Co-authored-by: Ben Hesseldieck <b.hesseldieck@gmail.com>
Co-authored-by: Ben Hesseldieck <1849459+BHesseldieck@users.noreply.github.com>
* ✅ fix test
* 🐛 fix imports
* 👕 fix lint issue
* User Management: switch over to decorators to define routes (#3827)
* Add permissions details to credentials for User Management (#3863)
* ⚡ Open `GET /users`
* ⚡ Add permissions to cred service
* 🚚 Rename method
* ⚡ Refactor cred controller
* 🧪 Adjust test
* ✏️ Improve comment
* ✏️ Improve another comment
* ⚡ Account for multiple sharings
* 🐛 Fix access when user is editor
* 📘 Expand interface
* 📘 Relocate types
* 📘 Exempt cred entity with service-injected fields
* 📘 Adjust interface
* ♻️ Add permissions only in `GET /credentials`
* 🧪 Add expectations for `ownedBy`
* 🧪 Add sharing details test
* 🧪 Make `ownedBy` checks more granular
* 📘 Adjust interface
* 🚚 Rename cred getter
* ♻️ Refactor cred getter
* 🧪 Expand tests
* ♻️ Refactor to use guard
* 👕 Remove unneeded lint exception
* 🔥 Remove unneeded relation
* 🚚 Move relation to `GET /credentials/:id`
* 📘 Consolidate typings
* 🎨 Add multiline for readability
* 🔥 Remove unneeded type
* ✏️ Clarity comment
* ✏️ Make comments consistent
* 👕 Add exception to fix build
* 👕 Add more lint exceptions to fix build
* 🐛 Check for non-owner
* 📘 Improve typings
* 🧪 Temporarily skip tests
* 🔥 Remove `@ts-ignore`
* 👕 Move lint exceptions
* ♻️ Refactor cred service and controller
* ⚡ Simplify check
* ✏️ adjust naming to experimental
* ⚡️ add credentialsSharing flag to settings
* 🛠 add helper to check if UM is also enabled as dependency for CredentialsSharing
* 👕 fix lint error
* 🐘 change name of credential role
* 🚧 WIP batch sharing
* 🚧 WIP use put for sharing
* ✅ add tests for batch sharing, 🛠 implement review suggestions
* ✅ expand credential sharing tests for User Management (#3931)
* 🧪 Expand cred sharing tests
* ⚡ Add recently added flags
* ✅ fix and adjust tests for /credentials
Co-authored-by: Ben Hesseldieck <b.hesseldieck@gmail.com>
* ✨ User management v2 Front End (#3795)
* feat: Added responsive generic page view layout.
* feat: Added empty state.
* feat: Added credentials view empty state.
* test: Added unit tests for N8nActionBox
* feat: Added credentials list initial design.
* feat: Added credential actions. Started working on filters.
* feat: Updated InfoTip markup, added tests and changed stories to typescript.
* feat: Added credentials filtering by type. Added support for apply/reset filters.
* feat: Added credential sharing user select and user list. Added paywall component.
* feat: Updated credentials view permissions.
* feat: Added support for temporary sharing config for unsaved credentials.
* test: Fixed broken snapshots.
* feat: Added overflow styles to page-view-layout list.
* feat: Handled sharee specific views.
* feat: Integration between FE and BE to support real-world credential sharing scenario.
* feat: Added front end permissions table.
* feat: Refactored credential sharing flow. Updated design elements.
* feat: Added margin and padding auto spacer utilities.
* feat: Rehauled permissions to support instanceOwner role and action inheritance.
* feat: Updated credentials view to apply filters automatically.
* feat: Removed apply filters button and added active button state.
* test: Updated component snapshots.
* refactor: Renamed ResourceSharee to ResourceReader.
* feat: Credential sharing error handling, permissions improvement.
* feat: Updated permissions and error handling.
* chore: Removed console.log.
* 🛠 refactor enabling of credentialsSharing
* feat: Removed owner menu selector from credentials when sharing is disabled.
* refactor: Moved EE features into ee store module file.
* 🛠 add sharing info to GET credentials/:id
* fix: Fixed initial credential data loading for sharing.
* chore: Removed console.log.
* 🐛 owner can fetch any credential
* 🛠 refactor users test
* 👕 fix build type issue
* fix: Removed owner tag when credential sharing is disabled. Fixed small reactivity issue.
* chore: Removed console.log.
* 🚧 separate fetching credentials between EE and open
* fix: Fixed empty dropdown in users list.
* fix: Fixed error message and initialization when credential gets unshared.
* ✅ add tests for fetching single credential
* Revert decorators based controllers
* ⚡️ adjust credentials test route to also allow testing for sharees (#3999)
* ⚡️ pull data if user is sharee
* fix: Removed sharedWith and ownedBy from credentialData on testing credentials.
Co-authored-by: Alex Grozav <alex@grozav.com>
* 📈 add BE analytics
* 💪 improve credential test
* ⚡️ adjust tracking properties
* ⚡️ removed roles from tracking
* 🐛 fix build by removing imports
* 🐛 fix missed merge conflict
* feat: User management P2 Front End bug bash and improvements (#4014)
* fix: Fixed type select size after reopening dropdown.
* fix: Fixed template cards.
* fix: Fixed card content size and copy input.
* fix: Fixed horizontal overflow.
* fix: Hiding el-tags scrollbar in select.
* fix: Added fallback credential icon. Added oAuth credential owner check.
* feat: Added disabled state to user select.
* feat: Added fallback scenario for non-existent credential types.
* feat: Adjusted credentials empty state to show that there are shared credentials.
* fix: Fixed time title.
* feat: Added actionable empty state when shared credentials are present.
* fix: Made action box x padding smaller
* feat: Repositioned owner tag for credential card.
* feat: Updated message box styling to use n8n css variables.
* feat: Added confirmation for deleting sharee.
* fix: Fixed deleted credential types. Fixed select in dropdown bug.
* fix: Various code improvements. Addressed PR review comments.
* fix: Fixed credential deletion errors.
* fix: Various code quality improvements.
* feat: N8N-4531 update cloud coming soon features (#4025)
* feat: Showing different upcoming feature messages and format for cloud.
* fix: Changed url format.
* fix: Updated how cloud deployment is determined.
* feat: N8N-4527 implementing credential sharing FE telemetry (#4023)
* feat: Added credential sharing telemetry.
* chore: Renamed computed function for consistency.
* refactor: Simplified subview telemetry sending.
* fix: Changed to callDebounced() helper.
* 📧 update email text
* fix: Adjusted feature coming soon margin.
* chore: Fixed type and line height for delete sharee confirmation modal.
* refactor(editor-ui): Update telemetry (#4040)
* 🔥 Remove `identify` from BE
* ⚡ Add `versionCli`
* ⚡ Add node creator ignore input
* ⚡ Move obfuscators to editor-ui
* ⚡ Refactor `ph-no-capture`
* ⚡ Pass `user_id` to manual exec props
* 🚚 Relocate class in `SettingsApiView`
* ⚡ Add `userId` to BE PH `identify` call
* ⏪ Revert "⚡ Add `userId` to BE PH `identify` call"
This reverts commit 895aaa45e51506d5dbdcbdabe249a2c743d8e468.
* Revert "⏪ Revert "⚡ Add `userId` to BE PH `identify` call""
This reverts commit b86a098c202155742c927c88c04c971a5d34dce5.
* 🐛 Fix `Promise` handling in `track()` call
* ⏪ Restore `Db.collections` call
* ⚡ Set up PH payload to mirror RS
* 🔥 Remove excess `userId`
* 📘 Remove `userId` from interface
* 🔥 Remove unused ref and method
* fix: Fixed bug causing instanceOwner to become credential owner on update. (#4079)
* 🐛 fix test for credential shared with member
* 👕 fix lint issues
* delete conflicting migration. this data is already seeded in CreateUserManagement
* feat: Expand obfuscation to User Management credential sharing (#4070)
⚡ Expand obfuscation
* feat: Added credential sharing infotip for instance owner.
* bring back the migration. add a check to avoid conflicts on inserts
* fix(cli): use a non-env config flag to detect of enterprise features are enabled (#4105)
* chore: Changed ampersand to and in translation.
* refactor(telemetry): Obfuscate code and JSON editors (#4118)
⚡ Obfuscate code and JSON editors
* feat(editor): improve design and functionality of coming soon features (#4116)
* feat: Improved coming soon feature design and functionality.
* style: Removed empty line.
* chore: Removed unused translation.
* fix: fix telemetry for credential creates and updates (#4125)
fix telemetry for credential creates and updates
* feat: Display errors due to missing credentials in the correct node (#4124)
feat: Display errors due to invalid credentials in the correct node when missing permissions
* fix: remove duplicate header for coming soon features in cloud deployment
* telemetry: fix the payload for `User viewed credential tab`
* telemetry: add credential_id to 'User selected credential from node modal'
* feat: update empty states for coming soon features
* Update ActionBox.spec.ts.snap
* replace UserSharingsDetails with a subset of User properties
* rename the CreateCredentialsEditorRole to CreateCredentialsUserRole
* move IUser to the workflow package
* use IUser in the frontend as well
Co-authored-by: Iván Ovejero <ivov.src@gmail.com>
Co-authored-by: Valya <68596159+valya@users.noreply.github.com>
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <netroy@users.noreply.github.com>
Co-authored-by: Alex Grozav <alex@grozav.com>
Co-authored-by: कारतोफ्फेलस्क्रिप्ट™ <aditya@netroy.in>
Co-authored-by: Omar Ajoue <krynble@gmail.com>
* Improve code health
Fix TS typos in local variables
Fix CSS typos in local styles
Fix typos in comments
Fix typos in strings
* Fix order of n8n setup sections in CONTRIBUTING.md
