Ansible module to restart ec2 instances by AmitPhulera · Pull Request #6905 · dimagi/commcare-cloud

AmitPhulera · 2026-06-02T20:02:56Z

https://dimagi.atlassian.net/browse/SAAS-19382

A redo of #6858, the changes were so much that it made no sense continuing on the existing branch.

This is part of the groundwork required to automate the machine restarts.

I have tested the command locally on staging web13 machine and the results were as expected, following are the output for each command -

describe

$ cat > /tmp/args.json <<'EOF'
{"ANSIBLE_MODULE_ARGS": {"instance_ids": ["i-0fb86252d8c3490e9"], "command": "describe"}}
EOF
python src/commcare_cloud/ansible/library/ec2_instance_state.py /tmp/args.json

{"changed": false, "command": "describe", "instances": [{"instance_id": "i-0fb86252d8c3490e9", "previous_state": "running", "current_state": "running", "name": "web13-staging", "instance_type": "t3a.xlarge", "availability_zone": "us-east-1a", "private_ip": "10.201.10.244", "public_ip": null, "tags": {"Name": "web13-staging", "Environment": "staging", "Group": "webworkers"}, "launch_time": "2025-08-30T17:34:24+00:00"}], "unchanged_instance_ids": [], "diff": {"before": {"states": {"i-0fb86252d8c3490e9": "running"}}, "after": {"states": {"i-0fb86252d8c3490e9": "running"}}}, "invocation": {"module_args": {"instance_ids": ["i-0fb86252d8c3490e9"], "command": "describe", "wait": true, "region": null}}}

start

$ cat > /tmp/args.json <<'EOF'
{"ANSIBLE_MODULE_ARGS": {"instance_ids": ["i-0fb86252d8c3490e9"], "command": "start"}}
EOF
python src/commcare_cloud/ansible/library/ec2_instance_state.py /tmp/args.json

{"changed": true, "command": "start", "instances": [{"instance_id": "i-0fb86252d8c3490e9", "previous_state": "stopped", "current_state": "running", "name": "web13-staging", "instance_type": "t3a.xlarge", "availability_zone": "us-east-1a", "private_ip": "10.201.10.244", "public_ip": null, "tags": {"Name": "web13-staging", "Environment": "staging", "Group": "webworkers"}, "launch_time": "2026-06-05T10:14:44+00:00"}], "unchanged_instance_ids": [], "diff": {"before": {"states": {"i-0fb86252d8c3490e9": "stopped"}}, "after": {"states": {"i-0fb86252d8c3490e9": "running"}}}, "invocation": {"module_args": {"instance_ids": ["i-0fb86252d8c3490e9"], "command": "start", "wait": true, "region": null}}}

stop

$ cat > /tmp/args.json <<'EOF'
{"ANSIBLE_MODULE_ARGS": {"instance_ids": ["i-0fb86252d8c3490e9"], "command": "stop"}}          
EOF
python src/commcare_cloud/ansible/library/ec2_instance_state.py /tmp/args.json

{"changed": true, "command": "stop", "instances": [{"instance_id": "i-0fb86252d8c3490e9", "previous_state": "running", "current_state": "stopped", "name": "web13-staging", "instance_type": "t3a.xlarge", "availability_zone": "us-east-1a", "private_ip": "10.201.10.244", "public_ip": null, "tags": {"Name": "web13-staging", "Environment": "staging", "Group": "webworkers"}, "launch_time": "2026-06-05T10:12:22+00:00"}], "unchanged_instance_ids": [], "diff": {"before": {"states": {"i-0fb86252d8c3490e9": "running"}}, "after": {"states": {"i-0fb86252d8c3490e9": "stopped"}}}, "invocation": {"module_args": {"instance_ids": ["i-0fb86252d8c3490e9"], "command": "stop", "wait": true, "region": null}}}

stop_start

$ cat > /tmp/args.json <<'EOF'
{"ANSIBLE_MODULE_ARGS": {"instance_ids": ["i-0fb86252d8c3490e9"], "command": "stop_and_start"}}
EOF
python src/commcare_cloud/ansible/library/ec2_instance_state.py /tmp/args.json

{"changed": true, "command": "stop_and_start", "instances": [{"instance_id": "i-0fb86252d8c3490e9", "previous_state": "running", "current_state": "running", "name": "web13-staging", "instance_type": "t3a.xlarge", "availability_zone": "us-east-1a", "private_ip": "10.201.10.244", "public_ip": null, "tags": {"Name": "web13-staging", "Environment": "staging", "Group": "webworkers"}, "launch_time": "2026-06-05T10:12:22+00:00"}], "unchanged_instance_ids": [], "diff": {"before": {"states": {"i-0fb86252d8c3490e9": "running"}}, "after": {"states": {"i-0fb86252d8c3490e9": "running"}}}, "invocation": {"module_args": {"instance_ids": ["i-0fb86252d8c3490e9"], "command": "stop_and_start", "wait": true, "region": null}}}

Environments Affected

All

contains documentation on what this module is going to do and some basic initialization, all the functionalities will be implemented in later commits

…so that they don't have to tossed around in every helper

gherceg · 2026-06-04T02:30:21Z

+    - Manages the running state of EC2 instances given an explicit list of
+      instance IDs. Supports four commands - describe, start, stop, stop_and_start,
+      and is idempotent (no API call is made if the instance is already in the requested state).
+    - Designed to run with delegate_to localhost. AWS credentials and the target region are picked up from the standard boto3 credential chain; in the commcare-cloud workflow the AWS_PROFILE and AWS_REGION environment variables are exported automatically before ansible runs.


I'm just not too familiar with this. Do you mind elaborating on the "delete_to localhost" workflow?

It is delegate_to localhost. This is the command that we want to run locally on the host from where the command was run instead of running it on the target machine.
For this case the AWS credentials will be on our local system so we want to run the module from our local systems.

gherceg · 2026-06-04T02:36:37Z

+    bad = [i for i in instance_ids if not INSTANCE_ID_RE.match(i)]
+    if bad:


nit

Suggested change

bad = [i for i in instance_ids if not INSTANCE_ID_RE.match(i)]

if bad:

bad_ids = [i for i in instance_ids if not INSTANCE_ID_RE.match(i)]

if bad_ids:

gherceg · 2026-06-04T02:40:47Z

+    except ImportError:
+        raise RuntimeError(
+            "boto3 is required by ec2_instance_state but is not installed."
+        )


Why not call module.fail_json directly here, similar to what _get_region does, and then that way main doesn't need to wrap the call to _get_ec2_client in a try/except.

Thats a good point. Will update it.

gherceg · 2026-06-04T02:44:01Z

+    """Per-run context shared by the flow helpers.
+
+    Bundles the EC2 client, the AnsibleModule, so these don't have to be
+    passed as arguments to every helper.
+    """


Suggested change

"""Per-run context shared by the flow helpers.

Bundles the EC2 client, the AnsibleModule, so these don't have to be

passed as arguments to every helper.

"""

"""

Bundles the EC2 client and Ansible module for convenience

"""

46189a2#diff-c6913cbc417347f1f21fdf62868aba7c3a544d1561404dc8c4ff1c5581760b6bL409-L414

gherceg · 2026-06-04T02:44:50Z

        )
    return boto3.client('ec2', region_name=region)

+class _Ctx:


Not my favorite name, but need to continue reviewing to offer suggestions.

All of the functions that accept this type (like _do_start(ctx, ...)) look like instance methods. What do you think of renaming this to something like StopStarter and moving those functions to methods?

class StopStarter: def __init__(self, client, module): self.client = client self.module = module def describe(self, instance_ids): ... def start(self, instance_ids, wait): ... def stop(self, instance_ids, wait): ... ...

Feel free to pick a different name, that was just the first thing that come to mind.

Thanks @millerdev. Used EC2InstanceManager class name 616ad06
This looks much better.

gherceg · 2026-06-04T17:06:03Z

+def _wait_for(ctx, waiter_name, wait_instances):
+    if not wait_instances or ctx.module.check_mode:
+        return
+    waiter = ctx.client.get_waiter(waiter_name)
+    try:
+        waiter.wait(InstanceIds=[i.instance_id for i in wait_instances])
+    except Exception as e:  # noqa: BLE001 - surface any waiter failure as module failure
+        ctx.module.fail_json(
+            msg=f"Waiter {waiter_name!r} failed for {_labels(wait_instances)}: {e}")
+        return


nit: wait_instances -> instances

46189a2#diff-c6913cbc417347f1f21fdf62868aba7c3a544d1561404dc8c4ff1c5581760b6bL310-L316

gherceg · 2026-06-04T17:11:25Z

+    return ', '.join(i.label for i in instances)
+
+
+def _check_not_terminated(ctx, instances, action):


nit: my preference would be to not pass action into this method since when I first saw _check_not_terminated(ctx, instances, InstanceCommand.START), my first thought was "why would the action impact a check for terminated instances?". I guess the alternatives are generalize the error message or leave it up to the caller to call module.fail_json.

While moving stuff to EC2InstanceManager, this was addressed.
616ad06

gherceg · 2026-06-04T17:24:53Z

+            ctx.module.fail_json(msg=f"StopInstances failed for {labels}: {e}")
+            return
+
+    wait_for_stopped = list(targets) + list(already_stopping)


targets and already_stopping are added together on line 336 as well. Could put this variable up with where targets and already_stopping are already defined. I also don't think these need to be wrapped in list(...) right?

Good call. 02e4d10

gherceg · 2026-06-04T17:33:37Z

+    # Honor the user's `wait` choice for the final running wait.
+    start_payload = _do_start(ctx, instance_ids, wait=wait)
+
+    # Combine: previous_state = state before the stop; current_state = after start.


nit: remove

gherceg · 2026-06-04T17:38:23Z

+    # unchanged = ids that were no-ops in BOTH phases.
+    # Highly unlikely to happen in practice, but we sort to make the result deterministic.


How would this be possible?

This would only happen when the instance state would be terminated but given the fact we are checking for it already so it should never happen. But I kept it for to keep the output same.

…are by helper

… on _do_start and _do_stop to build payload and the output uses the output from _do_stop

AmitPhulera added 13 commits June 3, 2026 00:43

add basic setup for the ansible module

ba1b46a

contains documentation on what this module is going to do and some basic initialization, all the functionalities will be implemented in later commits

get region and init ec2 client

a432ff3

have a lightweight context class to bundle ansible and boto clients, …

1cc0e41

…so that they don't have to tossed around in every helper

add basic structure for all the commands that module will support

2be5cf9

allow test ansible module to capture warnings as well

8c3e9b1

add basic tests for module's basic calling checks

6e7ea8f

have an Instance class to represent the instance state

29991c2

add FakeEC2Instance class to be used in the tests

1002e41

describe instances

d2f78e7

start instances

5b26467

implement stop

ea6ed5f

implement stop_and_start

036229a

lint

974a90e

AmitPhulera requested review from gherceg and millerdev June 4, 2026 13:11

AmitPhulera mentioned this pull request Jun 4, 2026

Ansible module to start/stop/restart ec2 instances #6858

Closed

gherceg reviewed Jun 4, 2026

View reviewed changes

AmitPhulera added 11 commits June 5, 2026 15:49

don't update library path to call ansible module, it is being taken c…

813a7d5

…are by helper

remove redundant list casting

6d3d240

fail with module.fail_json if client is not available

69749cb

rename property name to be same as variable name

88b69ed

remove docstrings that are less helpful

10e8703

nits from review

46189a2

fix tests

f65f251

refactor: from review

02e4d10

use f strings everywhere in tests

9e12be6

move module functions to InstanceManager to remove _Ctx class

616ad06

fix failing test - this failed because the start_and_stop was relying…

111910c

… on _do_start and _do_stop to build payload and the output uses the output from _do_stop

		bad = [i for i in instance_ids if not INSTANCE_ID_RE.match(i)]
		if bad:

-    bad = [i for i in instance_ids if not INSTANCE_ID_RE.match(i)]
-    if bad:
+    bad_ids = [i for i in instance_ids if not INSTANCE_ID_RE.match(i)]
+    if bad_ids:

		return ', '.join(i.label for i in instances)


		def _check_not_terminated(ctx, instances, action):

		# unchanged = ids that were no-ops in BOTH phases.
		# Highly unlikely to happen in practice, but we sort to make the result deterministic.

Conversation

AmitPhulera commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Environments Affected

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

AmitPhulera commented Jun 2, 2026 •

edited

Loading