-
Notifications
You must be signed in to change notification settings - Fork 101
Document Tentacle script abandonment #3175
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from 2 commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -0,0 +1,98 @@ | ||||||
| --- | ||||||
| layout: src/layouts/Default.astro | ||||||
| pubDate: 2026-05-27 | ||||||
| modDate: 2026-05-27 | ||||||
| title: Tentacle script abandonment | ||||||
| description: How Octopus Tentacle abandons a deployment script when it can't run normally on the target, what you'll see when it happens, and what to do about the underlying cause. | ||||||
| navOrder: 58 | ||||||
| --- | ||||||
|
|
||||||
| Octopus Tentacle can abandon a deployment script when the script can't run normally on the target. Abandonment releases the Tentacle's per-target mutex so the next deployment in the queue can start, even though the script's underlying process may still be running on the target. | ||||||
|
|
||||||
| This page covers when abandonment fires, what you'll see when it does, why these failures happen, and what to do about the underlying cause. | ||||||
|
|
||||||
| ## How abandonment works | ||||||
|
|
||||||
| When Tentacle abandons a script: | ||||||
|
|
||||||
| - The Tentacle's per-target mutex is released. The next deployment in the queue for that target can start immediately. | ||||||
|
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not in the queue, the tentacle will begin executing the next script it has for that target |
||||||
| - The Tentacle-side runtime locks holding state for the script are dropped. | ||||||
|
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. How is this different to the mutex? |
||||||
| - The abandonment is logged in the server-side task log and in the Tentacle log. | ||||||
|
|
||||||
| What abandonment does **not** do: | ||||||
|
|
||||||
| - It does not kill the script's underlying process on the target. If your script was performing an operation that could leave the target in an inconsistent state (a database migration, a file system change, and so on), inspect the target and clean up manually. | ||||||
| - It does not introduce a new task status. Depending on which trigger fired, the task is marked as `Failed` (PowerShell startup detection) or `Cancelled` (cancellation timeout). Check the task log to know which path your task took. | ||||||
|
|
||||||
| The Tentacle itself stays healthy after abandoning a script. It doesn't need to be restarted. | ||||||
|
|
||||||
| ## When abandonment fires | ||||||
|
|
||||||
| Tentacle abandons a script in response to one of two triggers. | ||||||
|
|
||||||
| ### PowerShell startup detection | ||||||
|
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @LukeButters did we really do this for powershell only? And if so was that just because thats the only thing we'd observed the behaviour for?
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes only for powershell. Afaik we have never seen this issue affect bash (on linux). |
||||||
|
|
||||||
| Scope: Windows Tentacles running `powershell.exe`. Linux `pwsh` is not currently supported. | ||||||
|
|
||||||
| When Tentacle launches `powershell.exe` to run your script, the PowerShell process can sometimes start but never actually begin executing the script body. This typically happens when antivirus or endpoint-protection software hooks into PowerShell startup and the script content never reaches the runtime. | ||||||
|
|
||||||
| If `powershell.exe` doesn't reach the first instruction of your script in 5 minutes, Tentacle marks the task as `Failed` with exit code `-47` and prevents the script body from running, even if PowerShell wakes up later. Tentacle records a log line like: | ||||||
|
|
||||||
| ``` | ||||||
|
Check failure on line 41 in src/pages/docs/infrastructure/deployment-targets/tentacle/tentacle-script-abandonment.md
|
||||||
|
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. specify this code block as whatever the most generic thing is. Its a block that contains sample logs |
||||||
| PowerShell startup detection: PowerShell did not start within 5 minutes for task <task ID> | ||||||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Do we need to disambiguate which script or step didn't start, and on which machine? Or is that inferred from the surrounding logs? Could we make it clearer with: |
||||||
| ``` | ||||||
|
|
||||||
| Version requirements: | ||||||
|
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Is this something we do standard. If so lets keep it as is. If not lets make this less table format and more an explination |
||||||
|
|
||||||
| - Octopus Server `2026.2.5952` or later | ||||||
| - Tentacle `9.1.3801` or later | ||||||
|
|
||||||
| ### Cancellation timeout | ||||||
|
|
||||||
| Scope: any script on Tentacle. Both Windows and Linux Tentacles. SSH targets and the Kubernetes agent are not in scope. | ||||||
|
|
||||||
| If you cancel a deployment from the Octopus Web Portal and the cancellation can't take effect on the Tentacle in 2 minutes, Octopus tells the Tentacle to abandon the script. The task is marked as `Cancelled`. | ||||||
|
|
||||||
| The server-side task log records: | ||||||
|
|
||||||
| ``` | ||||||
|
Check failure on line 58 in src/pages/docs/infrastructure/deployment-targets/tentacle/tentacle-script-abandonment.md
|
||||||
|
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Again give this a language, use a generic one
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Add this to all code blocks |
||||||
| Cancellation hasn't taken effect on Tentacle after 2 minutes. Abandoning the script so this target can accept new deployments. | ||||||
| Tentacle abandoned the script. | ||||||
| ``` | ||||||
|
|
||||||
| If the script had already completed by the time abandonment was attempted, the second line reads: | ||||||
|
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Can you confirm we have unit tests on these log lines that include a comment ensuring people know there is documentation dependant on this? If not ask me to go add them |
||||||
|
|
||||||
| ``` | ||||||
|
Check failure on line 65 in src/pages/docs/infrastructure/deployment-targets/tentacle/tentacle-script-abandonment.md
|
||||||
| Script had already completed before abandon was needed. | ||||||
| ``` | ||||||
|
|
||||||
| Tentacle's own log also records: | ||||||
|
|
||||||
| ``` | ||||||
|
Check failure on line 71 in src/pages/docs/infrastructure/deployment-targets/tentacle/tentacle-script-abandonment.md
|
||||||
| Tentacle has abandoned this script. The underlying script process may still be running on this host. | ||||||
| ``` | ||||||
|
|
||||||
| If your cancellation succeeded cleanly, no abandonment runs and the task is marked `Cancelled` without these messages. Check your task log to know which path your cancellation took. | ||||||
|
|
||||||
| Version requirements: | ||||||
|
|
||||||
| - Octopus Server version supporting cancellation timeout abandonment (to be confirmed when the work ships) | ||||||
| - Tentacle version that publishes `AbandonScript` on `IScriptServiceV2` (to be confirmed when the work ships) | ||||||
|
|
||||||
| ## Why these failures happen | ||||||
|
|
||||||
| The conditions that lead to abandonment are usually on the target machine, not in Octopus. | ||||||
|
|
||||||
| Antivirus and endpoint-protection software (CrowdStrike, Rapid7, and similar) can hook into `powershell.exe` at process startup. When two agents race for the same kernel locks, the process can fail to begin executing the script body. The same agents can hold file locks on the Tentacle's working directories (`Output.log`, `stdout.txt`), blocking the script from making progress or a cancellation from being processed. | ||||||
|
|
||||||
| For a worked example with stack traces and a detailed analysis of a CrowdStrike + Rapid7 deadlock on a customer's target, see [OctopusTentacle issue #1208](https://github.com/OctopusDeploy/OctopusTentacle/issues/1208). | ||||||
|
|
||||||
| Multiple security agents installed on the same host are the most common pattern. The fix is on the target machine. | ||||||
|
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The fix is on the target machine is an AIism. Make it sound like me |
||||||
|
|
||||||
| ## What to do about it | ||||||
|
|
||||||
| Both abandonment triggers are mitigation, not a fix. The underlying problem is on the target machine, and you're best placed to fix it. Three steps, in order: | ||||||
|
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
|
||||||
| 1. **Configure your antivirus or endpoint-protection software to exclude Tentacle's working directories.** Specifically `<Tentacle Home>\Tools` and `<Tentacle Home>\Work`. The full exclusion list and additional directories you can include if you're still seeing issues are documented in [Troubleshooting failed or hanging tasks: Antivirus software](/docs/support/troubleshooting-failed-or-hanging-tasks#anti-virus-software). | ||||||
| 2. **Keep target-side security tooling updated.** Known interactions between specific CrowdStrike and Rapid7 versions cause the deadlock; vendor updates have addressed similar issues before. | ||||||
|
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Where is the example where vendors have addressed similar issues before |
||||||
| 3. **If abandonment fires on the same target more than occasionally,** contact support and include a process dump from the target during the next occurrence. This helps support identify which agent is interfering. You can identify how often abandonment is firing on a specific target by searching your task logs for the messages above across recent deployments to that target. | ||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,7 +1,7 @@ | ||
| --- | ||
| layout: src/layouts/Default.astro | ||
| pubDate: 2023-01-01 | ||
| modDate: 2025-01-13 | ||
| modDate: 2026-05-26 | ||
| title: Troubleshooting failed or hanging tasks | ||
| description: A guide for troubleshooting tasks that fail unexpectedly or are unresponsive | ||
| navOrder: 8 | ||
|
|
@@ -63,6 +63,10 @@ In some instances, Octopus will automatically trigger the failure of a task that | |
|
|
||
| This is generally indicative of an internal error in Octopus. In Octopus Cloud we actively monitor for these issues, but please reach out to support for further assistance, especially if the problem persists. | ||
|
|
||
| ### Tentacle script abandonment | ||
|
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This sounds like this would cause a hanging task. When really its the solution to many of these problems |
||
|
|
||
| If your task ends in `Cancelled` or `Failed` and your Tentacle has logged that the script was abandoned, see [Tentacle script abandonment](/docs/infrastructure/deployment-targets/tentacle/tentacle-script-abandonment) for the full explanation and what to do next. | ||
|
|
||
| ### Antivirus software {#anti-virus-software} | ||
|
|
||
| If the task appears to hang after a log message output by the Octopus Server or Tentacle, then in most cases the cause is antivirus or anti-malware software interfering with the task. The first step is to determine if your antivirus software is actually affecting your Tasks, and this can easily be done by removing your antivirus protection and confirming whether the tasks continue to be unresponsive. | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we add a link to documentation about the per target mutex?