feat(cli): add mastra run command for headless agent execution#15348
Closed
cmphilpot wants to merge 5 commits intomastra-ai:mainfrom
Closed
feat(cli): add mastra run command for headless agent execution#15348cmphilpot wants to merge 5 commits intomastra-ai:mainfrom
cmphilpot wants to merge 5 commits intomastra-ai:mainfrom
Conversation
Contributor
|
Thank you for your contribution! Please ensure that your PR fixes an existing issue and that you have linked it in the description (e.g. with We use CodeRabbit for automated code reviews. Please address all feedback from CodeRabbit by either making changes to your PR or leaving a comment explaining why you disagree with the feedback. Since CodeRabbit is an AI, it may occasionally provide incorrect feedback. Addressing CodeRabbit's feedback will greatly increase the chances of your PR being merged. We appreciate your understanding and cooperation in helping us maintain high code quality standards. Comment |
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
WalkthroughAdds a headless Changes
Estimated code review effort🎯 4 (Complex) | ⏱️ ~55 minutes 🚥 Pre-merge checks | ✅ 2 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (2)
packages/cli/src/commands/run/run.ts (1)
1-10: Minor inconsistency:FileServiceimport path differs fromRunBundler.ts.
RunBundler.tsimports from'@mastra/deployer/build'while this file imports from'@mastra/deployer'. Both may resolve correctly, but consider using consistent import paths across the codebase for clarity.♻️ Suggested change for consistency
-import { FileService } from '@mastra/deployer'; +import { FileService } from '@mastra/deployer/build';🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/cli/src/commands/run/run.ts` around lines 1 - 10, The import for FileService in run.ts is inconsistent with RunBundler.ts (one uses '@mastra/deployer' while the other uses '@mastra/deployer/build'); update the FileService import in packages/cli/src/commands/run/run.ts to match the same module path used in RunBundler (use the '@mastra/deployer/build' path if that's what's used in RunBundler.js) so imports are consistent across the codebase; locate the import statement referencing FileService and adjust it to the same module string as in RunBundler to avoid ambiguity.packages/cli/src/commands/run/RunBundler.test.ts (1)
147-163: Consider adding test coverage forMASTRA_SKIP_DOTENV='1'.The
shouldSkipDotenvLoading()utility (frompackages/cli/src/commands/utils.ts) accepts both'true'and'1'values, but only'true'is tested here.🧪 Optional: Add test for '1' value
it('should return empty array when MASTRA_SKIP_DOTENV is set', async () => { process.env.MASTRA_SKIP_DOTENV = 'true'; const bundler = new RunBundler(defaultOptions); const envFiles = await bundler.getEnvFiles(); expect(envFiles).toEqual([]); }); + + it('should return empty array when MASTRA_SKIP_DOTENV is "1"', async () => { + process.env.MASTRA_SKIP_DOTENV = '1'; + + const bundler = new RunBundler(defaultOptions); + const envFiles = await bundler.getEnvFiles(); + + expect(envFiles).toEqual([]); + }); });🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/cli/src/commands/run/RunBundler.test.ts` around lines 147 - 163, Add a test case in RunBundler.test.ts that mirrors the existing MASTRA_SKIP_DOTENV='true' test but sets process.env.MASTRA_SKIP_DOTENV = '1' to assert getEnvFiles() returns an empty array; this ensures the shouldSkipDotenvLoading() behavior (used by RunBundler.getEnvFiles) correctly handles the '1' value. Reuse the same setup as the existing test (instantiate new RunBundler(defaultOptions), call await bundler.getEnvFiles(), and expect(envFiles).toEqual([])) and clean up/reset the environment variable after the test.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.changeset/thirty-regions-rush.md:
- Around line 5-19: The changeset entry titled "Added `mastra run` command for
headless one-shot agent invocation." is too long and reads like CLI docs;
replace the current long example block and flag-by-flag breakdown with a brief
1–2 sentence release-note summarizing the change (e.g., "Add `mastra run`
command to invoke agents headlessly with JSON output support."), and move the
detailed usage example and flags out of .changeset/thirty-regions-rush.md into
the CLI help or project docs; ensure the changeset only contains the concise
summary sentence(s) and nothing else.
In `@packages/cli/src/commands/actions/run-agent.ts`:
- Around line 15-17: The analytics call using analytics.trackCommandExecution
currently sends the raw args object (variable args) which contains sensitive
fields like the free-form prompt, JSON schema, and path/env overrides; update
the call so you redact those fields before sending: build a sanitized payload
from args that omits or replaces prompt, schema, and any path/env override keys
with redacted flags, and instead include only non-sensitive metadata such as
output format and boolean presence flags (e.g., hasPrompt, hasSchema,
hasLocalOverrides); update the call site where analytics.trackCommandExecution
is invoked to pass this sanitized payload rather than the original args.
In `@packages/core/src/harness/run-headless.ts`:
- Around line 82-105: The code currently relies on RunHeadlessIO.exit (a void
callback) to stop execution after SIGINT; instead add an explicit aborted
state/cancellation flow: modify registerSigint to set a shared aborted flag or
produce an AbortSignal/AbortController (and call io.exit as before), then
propagate that signal into the async formatting calls (formatText, formatJson,
formatStreamJson) or at minimum check the shared aborted flag immediately after
each await (e.g., after the chosen format* call) before inspecting fullOutput,
writing warnings or calling io.exit; ensure any subsequent branches (the strict
warnings branch and the normal exit path) short-circuit when aborted to avoid
emitting further output or calling io.exit again.
- Around line 65-80: The harness must reject a provided jsonSchema when
outputFormat is 'text' before calling the agent: in runHeadless (where
jsonSchema and structuredOutput are handled) add a guard that if jsonSchema is
set and outputFormat === 'text' then write an error to io.stderr and call
io.exit(EXIT_CONFIG_ERROR) (same code path used for invalid JSON parsing) so we
fail fast; only build structuredOutput and call agent.stream(...) when
outputFormat permits structured outputs, and keep references to jsonSchema,
structuredOutput, outputFormat, agent.stream and formatText to locate the
change.
---
Nitpick comments:
In `@packages/cli/src/commands/run/run.ts`:
- Around line 1-10: The import for FileService in run.ts is inconsistent with
RunBundler.ts (one uses '@mastra/deployer' while the other uses
'@mastra/deployer/build'); update the FileService import in
packages/cli/src/commands/run/run.ts to match the same module path used in
RunBundler (use the '@mastra/deployer/build' path if that's what's used in
RunBundler.js) so imports are consistent across the codebase; locate the import
statement referencing FileService and adjust it to the same module string as in
RunBundler to avoid ambiguity.
In `@packages/cli/src/commands/run/RunBundler.test.ts`:
- Around line 147-163: Add a test case in RunBundler.test.ts that mirrors the
existing MASTRA_SKIP_DOTENV='true' test but sets process.env.MASTRA_SKIP_DOTENV
= '1' to assert getEnvFiles() returns an empty array; this ensures the
shouldSkipDotenvLoading() behavior (used by RunBundler.getEnvFiles) correctly
handles the '1' value. Reuse the same setup as the existing test (instantiate
new RunBundler(defaultOptions), call await bundler.getEnvFiles(), and
expect(envFiles).toEqual([])) and clean up/reset the environment variable after
the test.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 9dfab51a-5b3a-499b-95a3-aafea2c1f7bd
📒 Files selected for processing (13)
.changeset/five-guests-play.md.changeset/thirty-regions-rush.mdpackages/cli/src/commands/actions/run-agent.tspackages/cli/src/commands/run/RunBundler.test.tspackages/cli/src/commands/run/RunBundler.tspackages/cli/src/commands/run/run.test.tspackages/cli/src/commands/run/run.tspackages/cli/src/index.tspackages/core/src/harness/index.tspackages/core/src/harness/output-formatter.test.tspackages/core/src/harness/output-formatter.tspackages/core/src/harness/run-headless.test.tspackages/core/src/harness/run-headless.ts
🦋 Changeset detectedLatest commit: 15bdd2e The changes in this PR will be included in the next version bump. This PR includes changesets to release 21 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (1)
packages/core/src/harness/run-headless.ts (1)
81-99:⚠️ Potential issue | 🟠 MajorSIGINT still doesn't unblock a hung formatter.
The new
abortedchecks only run afterformatText()/formatJson()/formatStreamJson()resolve. Fortextmode,packages/core/src/harness/output-formatter.ts:114-143waits onreader.read()until the stream closes, so a stalled stream still leavesrunHeadless()pending forever after Ctrl-C whenio.exit()is mocked or embedded instead of terminating the process.Possible fix
- if (outputFormat === 'text') { - fullOutput = await formatText(streamOutput, io.stdout, io.stderr); - } else if (outputFormat === 'json') { - fullOutput = await formatJson(streamOutput, io.stdout, startTime); - } else { - fullOutput = await formatStreamJson(streamOutput, io.stdout, io.stderr); - } + if (outputFormat === 'text') { + fullOutput = await formatText(streamOutput, io.stdout, io.stderr, abortController.signal); + } else if (outputFormat === 'json') { + fullOutput = await formatJson(streamOutput, io.stdout, startTime, abortController.signal); + } else { + fullOutput = await formatStreamJson(streamOutput, io.stdout, io.stderr, abortController.signal); + }// packages/core/src/harness/output-formatter.ts export async function formatText<OUTPUT>( streamOutput: MastraModelOutput<OUTPUT>, stdout: Writable, stderr: Writable, signal?: AbortSignal, ): Promise<FullOutput<OUTPUT>> { const reader = streamOutput.fullStream.getReader(); const onAbort = () => { void reader.cancel('interrupted'); }; signal?.addEventListener('abort', onAbort, { once: true }); try { while (true) { const { done, value } = await reader.read(); if (done) break; // ... } } finally { signal?.removeEventListener('abort', onAbort); reader.releaseLock(); } return await streamOutput.getFullOutput(); }Also applies to: 116-134
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/core/src/harness/run-headless.ts` around lines 81 - 99, runHeadless registers an AbortController but only checks aborted after formatText/formatJson/formatStreamJson return, so a hung formatter blocks SIGINT; modify the formatters (formatText, formatJson, formatStreamJson) to accept an optional AbortSignal parameter and inside obtain the reader from streamOutput.fullStream.getReader(), add an onAbort handler that calls reader.cancel('interrupted') and register it via signal.addEventListener(..., { once: true }), ensure the handler is removed in finally and releaseLock on the reader; then pass abortController.signal from runHeadless (where registerSigint is already wired) into the chosen formatter so Ctrl-C cancels the reader and unblocks runHeadless.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@packages/cli/src/commands/run/run.ts`:
- Around line 41-47: mastraDir is being resolved against process.cwd() instead
of using the computed rootDir (so --root is ignored); update the mastraDir
computation to base relative paths on rootDir (use rootDir in the join and in
the startsWith/resolve logic) so that the default and provided relative args.dir
are resolved from rootDir rather than process.cwd(); adjust the code that sets
mastraDir (the variable named mastraDir, referencing args.dir and join) to
reference rootDir for both the relative branch and the default branch.
---
Duplicate comments:
In `@packages/core/src/harness/run-headless.ts`:
- Around line 81-99: runHeadless registers an AbortController but only checks
aborted after formatText/formatJson/formatStreamJson return, so a hung formatter
blocks SIGINT; modify the formatters (formatText, formatJson, formatStreamJson)
to accept an optional AbortSignal parameter and inside obtain the reader from
streamOutput.fullStream.getReader(), add an onAbort handler that calls
reader.cancel('interrupted') and register it via signal.addEventListener(..., {
once: true }), ensure the handler is removed in finally and releaseLock on the
reader; then pass abortController.signal from runHeadless (where registerSigint
is already wired) into the chosen formatter so Ctrl-C cancels the reader and
unblocks runHeadless.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 0623c1c4-8b26-4ea0-a099-ac37a8e191bf
📒 Files selected for processing (5)
.changeset/thirty-regions-rush.mdpackages/cli/src/commands/actions/run-agent.tspackages/cli/src/commands/run/run.tspackages/core/src/harness/run-headless.test.tspackages/core/src/harness/run-headless.ts
✅ Files skipped from review due to trivial changes (1)
- .changeset/thirty-regions-rush.md
🚧 Files skipped from review as they are similar to previous changes (1)
- packages/cli/src/commands/actions/run-agent.ts
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (1)
packages/core/src/harness/run-headless.test.ts (1)
10-62: Consider extracting shared test helpers.The
createMockFullOutput,createMockStreamOutput, andcollectStreamfunctions are duplicated fromoutput-formatter.test.ts. These could be extracted to a shared test utilities file (e.g.,packages/core/src/harness/__tests__/helpers.tsorpackages/core/src/harness/test-utils.ts) to reduce duplication.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/core/src/harness/run-headless.test.ts` around lines 10 - 62, Tests duplicate helper functions createMockFullOutput, createMockStreamOutput, and collectStream; extract these into a shared test utilities module (e.g., harness/test-utils.ts) that exports those functions, then replace the local definitions in run-headless.test.ts and output-formatter.test.ts with imports from the shared module and update any types (FullOutput, ChunkType) imports as needed so both tests reference the single implementation of createMockFullOutput, createMockStreamOutput, and collectStream.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Nitpick comments:
In `@packages/core/src/harness/run-headless.test.ts`:
- Around line 10-62: Tests duplicate helper functions createMockFullOutput,
createMockStreamOutput, and collectStream; extract these into a shared test
utilities module (e.g., harness/test-utils.ts) that exports those functions,
then replace the local definitions in run-headless.test.ts and
output-formatter.test.ts with imports from the shared module and update any
types (FullOutput, ChunkType) imports as needed so both tests reference the
single implementation of createMockFullOutput, createMockStreamOutput, and
collectStream.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 1e1c85ea-e248-4267-b8f0-dc355f7b4130
📒 Files selected for processing (6)
.changeset/five-guests-play.mdpackages/core/src/harness/index.tspackages/core/src/harness/output-formatter.test.tspackages/core/src/harness/output-formatter.tspackages/core/src/harness/run-headless.test.tspackages/core/src/harness/run-headless.ts
✅ Files skipped from review due to trivial changes (2)
- .changeset/five-guests-play.md
- packages/core/src/harness/output-formatter.ts
🚧 Files skipped from review as they are similar to previous changes (2)
- packages/core/src/harness/index.ts
- packages/core/src/harness/run-headless.ts
Contributor
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
packages/core/src/harness/test-utils.ts (1)
37-50: Keep the shared stream-output helper strongly typed.
as anyhere removes the compile-time contract for a helper that multiple harness suites now depend on. If the real stream-output shape changes, these tests will keep compiling and only fail at runtime. Please return the concrete stream-output type instead of erasing it.As per coding guidelines,
**/*.{ts,tsx}: All packages use TypeScript with strict type checking.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@packages/core/src/harness/test-utils.ts` around lines 37 - 50, The helper createMockStreamOutput currently returns an any which erases the compile-time contract; change its return type to the concrete stream-output interface used by consumers (e.g., the type that exposes fullStream: ReadableStream<ChunkType<any>> and getFullOutput: () => Promise<FullOutput<any>>), update the function signature to return that exact type instead of `any`, and ensure imports/use of ChunkType and FullOutput in the signature match the existing types so callers get strong typing for fullStream and getFullOutput.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@packages/core/src/harness/run-headless.test.ts`:
- Around line 307-328: The test should actually await the headless run and make
the mocked agent/stream respond to abort so cancellation is exercised: change
the test to capture the promise returned by runHeadless(mastra, ..., h.io)
instead of discarding it, update the mock streamOutput
(fullStream/getFullOutput) to resolve/finish when the harness SIGINT handler is
invoked (e.g., have getFullOutput return a promise that can be rejected/resolved
by calling a stored abort callback), then after calling h.sigintHandlers[0]()
await the captured runHeadless promise before asserting exits and parsing
stdout; update other similar tests that call runHeadless (the ones referenced
around the other ranges) the same way.
---
Nitpick comments:
In `@packages/core/src/harness/test-utils.ts`:
- Around line 37-50: The helper createMockStreamOutput currently returns an any
which erases the compile-time contract; change its return type to the concrete
stream-output interface used by consumers (e.g., the type that exposes
fullStream: ReadableStream<ChunkType<any>> and getFullOutput: () =>
Promise<FullOutput<any>>), update the function signature to return that exact
type instead of `any`, and ensure imports/use of ChunkType and FullOutput in the
signature match the existing types so callers get strong typing for fullStream
and getFullOutput.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 3972660a-1df3-4fbd-813d-ead6bc0cf28a
📒 Files selected for processing (3)
packages/core/src/harness/output-formatter.test.tspackages/core/src/harness/run-headless.test.tspackages/core/src/harness/test-utils.ts
🚧 Files skipped from review as they are similar to previous changes (1)
- packages/core/src/harness/output-formatter.test.ts
Comment on lines
+307
to
+328
| it('json sigint handler should emit abort chunk and exit 1', async () => { | ||
| const h = createHarness(); | ||
| // Use a stream that doesn't resolve immediately | ||
| const never = new ReadableStream({ start() {} }); | ||
| const streamOutput = { fullStream: never, getFullOutput: () => new Promise(() => {}) } as any; | ||
| const agent = { stream: vi.fn().mockResolvedValue(streamOutput) }; | ||
| const mastra = createMockMastra(() => agent); | ||
|
|
||
| // Don't await — kick it off, then fire sigint | ||
| void runHeadless(mastra, { ...baseOptions, outputFormat: 'json' }, h.io); | ||
| await new Promise(r => setImmediate(r)); | ||
|
|
||
| expect(h.sigintHandlers).toHaveLength(1); | ||
| h.sigintHandlers[0]!(); | ||
| h.stdout.end(); | ||
| h.stderr.end(); | ||
|
|
||
| expect(h.exits).toEqual([1]); | ||
| const chunk = JSON.parse((await h.stdoutPromise).trim()); | ||
| expect(chunk.type).toBe('abort'); | ||
| expect(chunk.payload.reason).toBe('interrupted'); | ||
| }); |
Contributor
There was a problem hiding this comment.
Await the interrupted run in these SIGINT cases.
These tests currently only assert the handler’s immediate stdout/exit side effects. Because the mocked fullStream/getFullOutput never react to abort and the kicked-off runHeadless() promise is discarded, a regression where Ctrl-C leaves the headless run hung would still pass here. Please make the mock settle on abort and await the run promise so cancellation actually gets exercised.
Also applies to: 330-366, 406-424
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@packages/core/src/harness/run-headless.test.ts` around lines 307 - 328, The
test should actually await the headless run and make the mocked agent/stream
respond to abort so cancellation is exercised: change the test to capture the
promise returned by runHeadless(mastra, ..., h.io) instead of discarding it,
update the mock streamOutput (fullStream/getFullOutput) to resolve/finish when
the harness SIGINT handler is invoked (e.g., have getFullOutput return a promise
that can be rejected/resolved by calling a stored abort callback), then after
calling h.sigintHandlers[0]() await the captured runHeadless promise before
asserting exits and parsing stdout; update other similar tests that call
runHeadless (the ones referenced around the other ranges) the same way.
Contributor
Author
|
Closing — on reflection this work belonged in mastracode/src/headless.ts, not a new framework-CLI command. Will open a narrower PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Related Issue(s)
Type of Change
Checklist
ELI5 Summary
This PR adds a new "mastra run" command so you can run a Mastra agent from the terminal without running a server: provide an agent ID and a prompt (or pipe input), choose an output format (text, JSON, or streaming JSON), and it runs the agent and prints the results. It also adds a headless runner and output-formatting helpers to support that command.
Overview
Adds a headless, one‑shot CLI workflow to execute Mastra agents locally with selectable output formats, optional structured output via JSON Schema, SIGINT handling, and clear exit-code semantics. Exposes a runHeadless orchestrator and output-formatting utilities from @mastra/core/harness and wires a top-level mastra run command in the CLI.
Key Changes
CLI
Core headless orchestrator & exports
Tests
Releases / metadata
Output & Signal Behavior (summary)
Validation & Errors