feat: add --screencap and --control options for powerful CLI automation#6749
Draft
dapangyu-fish wants to merge 32 commits into
Draft
feat: add --screencap and --control options for powerful CLI automation#6749dapangyu-fish wants to merge 32 commits into
dapangyu-fish wants to merge 32 commits into
Conversation
Author
|
Note: This PR mainly modifies the client-side code. I have only done basic testing and have not fully tested all features yet. I'm not actively seeking a merge at this time. If anyone needs this automation capability, feel free to build it separately from my branch. Thank you! |
Collaborator
|
Please mention when the code is AI generated. |
Author
Thank you for the reminder. |
|
👍 |
Resolve conflicts with upstream scrcpy 4.0 (SDL3, FFmpeg 8): - meson.build: keep libswscale (needed by screencap) alongside sdl3 - cli.c/events.h/options.h: keep both feature options (--screencap, --control) and new upstream options; drop the deprecated --rotation entry removed upstream - screencap.c: adapt packet sink open() to the new signature taking a struct sc_stream_session parameter Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Avoid any conflict with an official scrcpy installation on the same machine or device: - client executable: scrcpy -> scrcpy-auto - server binary: scrcpy-server -> scrcpy-auto-server, installed in share/scrcpy-auto/ and pushed to the device as /data/local/tmp/scrcpy-auto-server.jar - renamed installed data accordingly (manpage, icons, desktop entries, bash/zsh completions) and the SDL app name Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Feasibility analysis and implementation specification for --daemon-port/--client-port: a long-running per-device daemon that keeps the device session alive and serves screenshot/control requests from thin client invocations over localhost TCP, with reconnection supervision, an on-disk registry for discovery, and full backward compatibility with the one-shot CLI. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add --daemon-port: run a persistent per-device daemon that keeps the
device session alive (server + video demuxer + decoder + controller),
retains the latest decoded frame, and serves client requests over a
length-prefixed JSON protocol on 127.0.0.1, with automatic reconnection
(1s..30s backoff, --daemon-reconnect policy) and an advisory on-disk
registry.
Add --client-port: thin client executing --control, --screencap,
--daemon-status and --daemon-stop against a running daemon in
milliseconds, without starting a new device session.
- app/src/control_exec.{c,h}: --control executors moved out of
scrcpy.c, parameterized by touch pointer-id base so concurrent
daemon requests use disjoint ranges
- app/src/screencap.{c,h}: split PNG encoding into sc_frame_to_png()
- app/src/daemon/frame_keeper.{c,h}: re-readable latest-frame sink;
screencap requests wait up to 2s for a frame, optional max_age_ms
freshness via RESET_VIDEO
- app/src/daemon/protocol.{c,h}: 4-byte BE length framing + minimal
JSON (vendored jsmn, MIT) with escape/unescape
- app/src/daemon/registry.{c,h}: <runtime dir>/scrcpy-auto/<port>.json
- app/src/daemon/daemon.c: listener, per-connection threads (16 max),
request dispatch, session supervisor state machine
- app/src/daemon/client.c: hello/protocol check, ops in order
control -> screencap -> status -> shutdown, exit code 2 when the
daemon is unreachable
- cli.c: new options + validation matrix (daemon requires --no-window;
client mode excludes session options)
All sources compile warning-free (clang 18 via zig cc) against SDL3
3.2.0 and FFmpeg 8.0 headers; protocol framing/JSON verified by a
standalone smoke test. Not yet built with meson or tested against a
real device (no toolchain on this machine).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- cli.c: client mode no longer rejects the default window=true; it forces --no-window semantics itself, so the documented "scrcpy-auto --client-port PORT --screencap out.png" works - daemon.c: enlarge send_error() header buffer (malformed JSON for request ids >= 4 digits); retry on transient accept() failures instead of permanently exiting the accept loop; derive touch pointer-id ranges from the connection slot (the wrapping allocator could hand out overlapping ranges); clamp screencap max_age_ms (int64 overflow / negative min_tick bypassed the first-frame wait); record stop requests while draining in-flight requests; add the missing audio_codec_options/audio_encoder server params - frame_keeper: add sc_frame_keeper_reset(), called on session stop, so a screencap after reconnect cannot serve a stale frame from the previous session - control_exec.c: fix use-after-free (text logged after ownership moved to the controller queue, which may free it concurrently); deduplicate the quoted/unquoted branches; cap step durations at 60 s so one command cannot block daemon teardown for hours All 77 sources still compile warning-free (clang 18 via zig cc, -Wall -Wextra) against SDL3 3.2.0 and FFmpeg 8.0 headers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add doc/fork.md: authoritative inventory of every difference from upstream scrcpy (rename, --screencap/--control, daemon mode), the full modified-file list, known conflict hotspots with resolution rules, semantic coupling points that break without textual conflicts, and a post-merge verification checklist. Add CLAUDE.md so AI-assisted sessions automatically load the merge invariants and are pointed at doc/fork.md before resolving conflicts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Touch events were silently ignored by the device when video capture is
enabled: the server-side PositionMapper drops any event whose
screen_size does not match the video size, and the executor hardcoded a
{UINT16_MAX, UINT16_MAX} placeholder (which only works when video is
disabled, where the device falls back to raw coordinates -- the one-shot
--control path).
- sc_control_exec_run() now takes the reference screen_size; the
daemon passes the actual video size (waiting for the first frame if
needed), so click/swipe coordinates are interpreted in video-frame
space -- the same pixel space as the screenshots returned by the
daemon. --no-video daemons and one-shot mode keep raw coordinates.
- one-shot control mode now explicitly disables video, and rejects the
--control + --screencap combination (it exited before the screenshot
completed and would have hit the same position-mapping issue).
- --control now takes a required argument, so the space-separated form
(--control "click 500 800") works, not only --control=...;
--control help prints the command reference.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
"click 540 1500 540 5000" silently parsed as a click at (540,1500) with a 540 ms duration, ignoring the trailing token -- the user intended a long press and got a short one with no error. Fail with the usage message when a touch command has trailing arguments. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rejected commands (e.g. "click 540 1500 540 5000", one argument too many) previously surfaced only as a generic "control commands failed" on the client, with the usage message hidden in the daemon log. Add sc_control_exec_check() and validate the request up front in the daemon, so the client receives E_BAD_REQUEST with the expected syntax. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A bare DOWN followed by a late UP is less reliable for holds than a gesture with a continuous event stream: some gesture detectors drop or mis-handle a pointer that stays silent. Send MOVE events at the click position every 10ms while held (identical to "swipe x y x y ms"), and raise the default click duration from 100ms to 300ms. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Expose the interfaces a web control tool (scrcpy-auto-web) needs: - subscribe_video: turns an IPC connection into a one-way push stream of the ENCODED device video (Annex-B, forwarded verbatim, no re-encode) so a browser can decode it with WebCodecs. Implemented as a second packet sink on the video demuxer (app/src/daemon/broadcaster.c) alongside the existing decoder-for-screenshots; sends codec metadata + config (SPS/PPS) + frames, re-announced to subscribers on reconnect/resolution change. - inject_touch / inject_key / inject_text / inject_scroll: single, non-blocking realtime input events (unlike the batch "control" op), pushing one sc_control_msg each. Touch/scroll coordinates are in video-pixel space, stamped with the current video size. Additive protocol changes only (no version bump). Documented in doc/daemon.md §8.3/§8.7. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Force RESET_VIDEO when a client subscribes so a mid-stream subscriber starts decoding at the next frame instead of waiting for the device's periodic I-frame interval (seconds of blank screen otherwise). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add --auto-test-report=DIR (daemon) and --action=TEXT (client) for the
scrcpy-auto test-report platform (scrcpy-auto-web/DESIGN-test-report.md):
- app/src/daemon/report.{c,h}: creates DIR, records the screen to
DIR/recording.mp4 (sc_recorder as a third packet sink on the video
demuxer), and appends every deliberate client operation to
DIR/events.jsonl with a frame-accurate, drift-free timestamp; writes
and finalizes DIR/manifest.json.
- Frame-accurate clock: frame_keeper tracks first/last video PTS (us)
and exposes sc_frame_keeper_video_time_ms() (decision: eliminate
drift by deriving op times from the video PTS clock, not wall time).
- Logging: control and screencap ops are logged (control carries the
raw cmds + video size so the web renderer can rebuild finger paths;
screencap logs timestamp + dimensions, no image copy). Realtime
inject_* (web live-control) is not logged (decision: manual web
gestures are excluded from the report).
- --action threads through the client control/screencap requests.
- Report mode records one session and ends on device disconnect
(finalizes the mp4). MP4 format; orientation locked implicitly.
- CLI: --auto-test-report requires --daemon-port and video.
All 79 sources compile warning-free (clang via zig cc). Report JSON
output verified by a unit harness (manifest + events.jsonl parse
cleanly, escaping correct). Not yet run against a real device.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add a client --note="title: description" option and a daemon "note" op
for report annotations that are not tied to a control command — marking
test-case boundaries, assertions, reasoning, and milestones.
The daemon splits the note on the first ':' into a structured
{title, text} and logs a "note" event with the video-correlated
timestamp. --note is a standalone client operation (no --control needed).
Compiles warning-free; note title/text parsing unit-tested. Documented
in doc/daemon.md §8.3.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Let external tools provide custom --<command> options without touching
the C code — intended for AI/Python workflows that reuse scrcpy-auto for
screenshots and input (doc/addons.md).
- Daemon --add-on=PATH (repeatable): app/src/daemon/addon.c queries each
entrypoint (SC_ADDON_MODE=register) for its command name, then runs it
(SC_ADDON_MODE=run) with the command value as $1 and runtime info in
the environment (SC_DAEMON_PORT, SC_DEVICE_SERIAL, SC_REPORT_DIR,
SC_VIDEO_WIDTH/HEIGHT, SC_ADDON_NAME, SCRCPY_AUTO). New "plugin" op.
- Client: any unrecognized --name=value is pre-scanned out of argv and
forwarded as {op:"plugin",name,args}; the daemon runs the registered
add-on (unknown -> error). Requires --client-port.
- Report: the daemon auto-logs a "plugin" start event; the add-on's own
screenshot/note/click callbacks are logged in real time (no batching),
giving the timeline: plugin start -> screenshot -> note -> click.
- Lifecycle: client call blocks until the script exits; a running add-on
child is terminated on daemon shutdown. Unix/macOS; localhost trust.
All 80 sources compile warning-free (clang via zig cc). Add-on
register/run mechanics verified with a shell entrypoint (name query,
$1 + env passing, exit code). Documented in doc/addons.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fix tap/action markers landing seconds too early (and the real action showing up as the later note): the report timestamped each op with the last decoded frame's PTS, but Android's encoder emits frames only when the screen changes, so during a static period (e.g. a ~5s AI add-on think) the PTS clock freezes. An op logged then got a stale, too-early time, while an op logged just after the screen changed got the correct later time. Timestamp by wall-clock elapsed since the first recorded frame (frame_keeper first_frame_tick) instead — that matches the mp4's real-time playback timeline, which holds static frames for their true duration. The web-side rescale to video.duration still corrects any residual linear rate difference. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Extend the add-on framework (doc/addons.md) into one protocol every plugin
shares, so new AI capabilities never add bespoke CLI flags or wire ops:
- Register metadata gains result=/env=/meta= and arg types path/pathlist
(in addition to name=/arg=); advertised in the daemon hello as keyed
strings, discovered by both the C client and the web UI.
- --json (global): the client prints only the plugin's result object, or a
JSON error object on failure; banner/logs are suppressed so stdout stays
pure JSON.
- --daemon-host (default 127.0.0.1) + adaptive reference-image transport: a
loopback daemon reads path/pathlist args directly (no copy); a remote
daemon receives the bytes via a new "upload" op (per-connection temp
file). Plugins are locality-oblivious.
- Result channel: SC_RESULT_FILE (the plugin writes {result,reason,
thinking,...}); the daemon reads it back into the plugin response.
Required env-var validation. Protocol bumped to v3.
The AI plugins themselves (exec_prompt/assert_prompt) are intentionally not
committed; .gitignore excludes plugin implementations and their example
assets.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
After an add-on writes SC_RESULT_FILE, log its structured result
({result,reason,thinking,...}) as a "result" event in events.jsonl (only
when --auto-test-report is active), so the web report can display the full
plugin result JSON on the timeline.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
When a report is recording and a plugin is invoked with path/pathlist arguments (e.g. --ref-images), copy those files into <report>/assets/ and record their relative paths on the plugin event, so the report (and its offline export) can render the reference images. Best-effort; works for local paths and daemon-side uploaded temp files alike. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two add-ons declaring the same argument name (e.g. both --exec_prompt and --assert_prompt accepting --ref-images) previously collided: the client attributed the bare flag to whichever plugin declared it first, so only that one could ever receive it. The client now resolves each raw flag as (1) a command, (2) a qualified --<command>.<arg>, or (3) a bare arg routed to the *invoked* declaring command. A bare name is unambiguous whenever a single declaring plugin is invoked; when several are, it is rejected in favour of the qualified form. The <command>. prefix is stripped before the call is sent, so the daemon and entrypoint scripts still see the plain SC_ARG_<NAME> and need no changes. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
New client operation: --clip-start/--clip-end (seconds, decimals allowed)
and --output save that segment of the daemon's session as a standalone MP4,
while the session and its test-report recording continue untouched.
The daemon gains a clip buffer (app/src/daemon/clip_buffer.c), a fourth,
fully independent packet sink on the video demuxer: every encoded packet is
appended to an unlinked spool file and indexed in memory {pts, offset, size,
keyframe}. The new "clip" op selects [keyframe <= start, last packet <= end],
muxes the slice into an in-memory MP4 outside the buffer lock (timestamps
rebased to 0, extradata from the config packet like the recorder) and returns
the bytes as the response payload; the CLIENT writes the output file, so
remote clients work and the daemon needs no write access to the path.
Contract: times are relative to the first video packet (the report timeline
origin); the start snaps back to the nearest keyframe and the response
reports the actual bounds; an end beyond the last recorded packet fails with
E_RANGE. Spool write errors disable clips but never fail the pipeline.
SC_PACKET_SOURCE_MAX_SINKS is bumped to 4 (decoder, web broadcaster, clip
buffer, report recorder); fork invariants updated accordingly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
handle_status now also emits the loaded add-on schemas (the same array the hello advertises), a "report" object (enabled/dir/recording/video path), and a "config" object with the referenced capture parameters (port, control, video, and when video is on codec/bit_rate/max_size/max_fps/encoder), so --daemon-status is a one-stop readout of how the daemon was launched. Also fixes a latent bug in the new code: sc_strbuf_append_staticstr() uses sizeof(S)-1, so passing a ternary of two string literals (a const char*, not an array) computed the pointer size as the length — harmless under assertions but an out-of-bounds read under NDEBUG. Booleans now use sc_strbuf_append_str. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add-ons now receive the daemon's capture parameters as SC_* environment variables at launch, so a plugin reads them from its environment instead of shelling back into the client to introspect: SC_VIDEO_CODEC, SC_VIDEO_BIT_RATE, SC_VIDEO_MAX_FPS, SC_VIDEO_MAX_SIZE, SC_VIDEO_ENCODER and SC_CONTROL, next to the existing SC_DEVICE_SERIAL / SC_REPORT_DIR / SC_VIDEO_WIDTH/HEIGHT. Empty string / 0 mirrors "unset / server default", as with SC_REPORT_DIR. doc/addons.md lists the new variables. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
sc_report_init did a single-level mkdir, so a nested report target such as "./v/test1" failed with "could not create directory" when "./v" did not exist yet. make_dir now creates every missing parent component before the leaf (EEXIST treated as success, leading '/' or '.' preserved, empty path handled without reading past the buffer). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Install the client with `brew install <raw formula URL>` (or `--HEAD`, or by tapping the repo). The formula builds scrcpy-auto from source against Homebrew SDL3/FFmpeg(+libswscale)/libusb, and downloads the matching upstream scrcpy-server-v4.0 release artifact (sha256-pinned) as the prebuilt server so no Android SDK is needed — meson installs it to share/scrcpy-auto/scrcpy-auto-server, where the client resolves it. adb comes from the android-platform-tools cask. README documents install, the server-handling rationale, device setup, and how to keep the source revision and server resource in lock-step across releases. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…n by name
A management layer over the add-on system (doc/plugins.md):
- `scrcpy-auto plugins-install <git-url> --add-on-name=<name>|ALL` and
`plugins-upgrade` shallow-clone the repo and copy the plugin directory(ies)
into ~/.scrcpy-auto/plugins/<name>/. ALL installs every plugin dir; upgrade
ALL refreshes only the ones already installed. (app/src/plugins.{c,h})
- `--add-on=<name>` now accepts a bare plugin name, resolved to
~/.scrcpy-auto/plugins/<name>/entrypoint.sh; a value that looks like a path
is still used as-is. (daemon/addon.c: sc_addons_load)
- main.c dispatches the two subcommands (Unix only, like add-ons); the plugin
code is SDL/log-free and #ifndef _WIN32 guarded (Windows no-op).
Docs: doc/plugins.md (new) + pointer in doc/addons.md + doc/fork.md inventory.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The caller's cwd may be a deleted directory (e.g. after brew reinstall), which made git abort with 'cannot read current working directory'. Plugin paths are absolute, so move to HOME before spawning git/cp/rm.
access(spec) matched a same-named directory in the current dir (e.g. running from ~/.scrcpy-auto/plugins), so bare names were treated as local paths and never resolved. Now: '/'-containing specs are used as-is; a bare name resolves to $HOME/.scrcpy-auto/plugins/<name>/entrypoint.sh only if it exists.
The hicolor PNG icons are useless on macOS and are a common `brew link` conflict: a prior manual `sudo ninja install` leaves share/icons/hicolor/.../apps root-owned, which Homebrew (correctly refusing to run as root) cannot overwrite. Remove them from the keg after meson install; the man page and shell completions are unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
scrcpy is an excellent mirroring tool. However, many users (including myself) need to perform automated operations on the device directly from the command line — for UI automation, testing, scripting, bot development, etc.
I understand that the scrcpy maintainers may intentionally want to keep scrcpy as a pure display/mirroring tool. Implementing full control features might go beyond the original vision. That said, these features are not very complex to implement and are extremely useful for many automation users.
Therefore, I have implemented a complete CLI control system.
PR Description
What does this PR do?
This PR adds two new powerful options for CLI automation:
--screencap=FILE- Take a screenshot and save it locally (can be combined with--no-playbackand-mresolution options)--control=COMMAND- Execute control commands (click, swipe, input text, etc.)Multiple
--controlflags can be used simultaneously for multi-touch gestures.Supported control commands
input "text": Send text input (full Unicode/emoji support)click x y [duration_ms]: Click or long-pressswipe x1 y1 x2 y2 duration_ms: Swipe gesturesleep ms: Delay between actionsCommands can be chained with
&&. Multi touch can use multiple --control=COMMAND.Examples
Attached is a demo video of the 10 point touch limit test.
https://github.com/user-attachments/assets/acd72a86-8b3c-4a3b-a852-528e209efa61
Notes
How to build & test
https://github.com/fish-dapangyu-dev/scrcpy.git cd scrcpy rm -rf build meson setup build --buildtype=release ninja -C build app/scrcpy rm /opt/homebrew/bin/scrcpy cp app/scrcpy /opt/homebrew/bin/