Evaluate SodaOS
Operate a factory

Supervise agent pull requests and human approval gates

Follow a small issue from accepted requirements to an AI-generated pull request, current checks, independent review and a merge governed by repository policy.

On this page

This is a reproducible workflow exercise for a repository maintainer, not a recording of a completed factory run. Use a disposable repository and the selected release's setup instructions. The important result is the evidence linking a requirement, candidate commit, review and merge—not an agent's statement that it finished.

SodaOS's normal loop is autonomous after initial authorization. This exercise observes that loop and separately tests an optional human approval gate; it does not require an operator to start each issue or manually merge every result.

Prepare a small issue

Use the first-issue fixture on GitHub. It contains a small JavaScript access function, baseline tests and an issue brief. Copy it into a test repository and run its checks before enabling work:

node --test access.test.mjs

The baseline lets an active HR user view a task. The issue asks the agent to add access for the active person assigned to that task, while refusing unassigned tasks, other users and inactive accounts. The brief names the required cases and excludes unrelated authentication changes. The fixture's README explains how to create the isolated repository and record the initial commit.

Configure a required native CI check to execute the same command on candidates. Confirm that the check runs and that its name exactly matches the repository policy. Set the desired review and human approval rules before allowing the factory to work.

From accepted issue to candidate

  1. Enable factory work for the repository under its administrator's policy. Authorize appliance capacity and the provider connections separately.
  2. Create the issue as an authorized maintainer with the exact acceptance outcomes. SodaOS picks it up automatically under standing repository policy. Issues without trusted authorization need explicit adoption; a command-like comment does not grant authority.
  3. Confirm that prerequisites are satisfied and that the objective is ready. Material missing requirements should block work with a specific question.
  4. Observe the coding execution in the Project. Retain the issue, attempt and run identity and the input revision.
  5. Open the resulting Forgejo pull request. Record its URL and exact candidate commit. A running terminal or successful CLI exit is not yet a reviewed result.

The factory lifecycle contract on GitHub defines readiness, publication and review. The operator status/stop commands inspect and control recorded executions; they are not issue-admission commands.

Review and correction

Inspect the changed function and tests. In this example, Boolean(task.assigneeId) would be an incorrect authorization rule: it permits any active user to see any assigned task. The correct behavior must match the assignee to the signed-in user and require an active account.

A separate review session evaluates the candidate. If it finds a problem, the coding role corrects it and the new revision receives current verification. Check that CI and review evidence refer to the revision proposed for merge. Approval of an older commit cannot establish correctness of a later change.

In the test repository, deliberately exercise a failing check or request a correction, and observe that the candidate does not proceed with stale evidence. Do not weaken the policy to get a green-looking demonstration. Record actual results and unresolved failures.

Human approval gates

Human approval gates are optional. The normal factory loop merges autonomously when its required review and checks pass under repository policy. If your team chooses a human gate, use native repository review/protection requirements and factory policy to enforce it. A natural-language instruction saying “ask before merging” is not a substitute for that boundary. Human business acceptance, independent agent review and native merge approval serve different purposes.

For a human-gated repository, verify that passing automated checks and agent review alone leave the PR waiting for the required human approval. Approve the current candidate as the authorized reviewer, then observe the policy-governed merge. For an automatically governed repository, standing authorization can permit routine merges once its required evidence is current. Do not assume every PR always needs a manual final click.

Confirm the result and retain evidence

Record the native merge result and target commit, and rerun the checks on the resulting target branch. Confirm the required behavior and the issue outcome. Merge does not authorize application deployment; if the change must run elsewhere, perform that separate operation and validate it there.

Keep the selected SodaOS release/source identity, repository policy, input commit, issue URL, candidate commits, check runs, review decisions, human approval and final merge reference. Include actual timestamps and failure observations. Do not fill missing steps with simulated output. This record can later support a release-specific walkthrough without implying that an illustrative website animation proves execution.

Continue with the factory handbook, multiple Projects or the technical reference.