Skip to content

feat: Slack HITL agent actions 2.6.6 - #23

Open
Himanshu040604 wants to merge 29 commits into
mainfrom
feat/slack-hitl-coda
Open

Himanshu040604 wants to merge 29 commits into
mainfrom
feat/slack-hitl-coda

Conversation

@Himanshu040604

@Himanshu040604 Himanshu040604 commented May 16, 2026 •

Copy link
Copy Markdown

Summary

Pins agno[os] to >=2.6.6,<2.7 and adds Slack human-in-the-loop (HITL) approval gating: an agent's destructive GitHub actions pause for human approval via an interactive Slack card before they execute.

When a gated agent decides to act, agno pauses the run and posts an approval card to the Slack thread — one row per action, each with Approve / Deny. Only approved rows execute on GitHub.

What's gated

  • Triager — label_issue, comment_on_issue, close_issue, create_issue, reopen_issue, assign_issue, edit_issue
  • Planner — create_issue (one multi-row card when proposing several)
  • Explorer — comment_on_issue, create_pull_request_comment
  • Cron triage — the daily /triage-issues job posts a header to TRIAGE_CHANNEL, threads the HITL card under it, and skips if a prior run is still unresolved

Changes

  • approvals_table="coda_approvals" added to PostgresDb for HITL pause/resume state
  • requires_confirmation_tools wired onto the Triager / Planner / Explorer GitHub toolkits
  • tasks/review_issues.py reworked for HITL-gated cron triage (header + threaded card + pending-run dedup)
  • coda/slack_patches.py — caps agno's HITL confirmation-card body under Slack's 201-char block limit; without it, long tool args silently fail to render the card (candidate for an upstream agno fix)
  • Planner instructions hardened — create_issue calls made mandatory; denied issues are not retried
  • pytest infrastructure + 17 tests

Deliberately not included

  • Coder HITL — Coder has run_shell (arbitrary shell, needed for autonomous testing) which bypasses structured-tool gates. Gating a shell-enabled agent needs an allowlist approach — deferred to a follow-up PR rather than shipping non-enforceable gates.
  • Severity picker (UserFeedbackTools) — ask_user is LLM-discretionary and fired unreliably; the requires_confirmation gate on label_issue already covers priority labels.

Test plan

  • 17 unit tests pass (pytest tests/), ruff clean
  • Manual e2e in a live Slack workspace against a test GitHub repo:
    • Triager — multi-row card; approve fires the action, deny blocks it; multi-pause chain
    • Planner — create_issue multi-row; approve creates, deny does not retry
    • Explorer — comment_on_issue gate on PR review
    • Cron — curl /triage-issues posts header + threaded card

Himanshu040604 and others added 17 commits May 15, 2026 19:06
- Bumps agno 2.5.14 -> 2.6.6 to enable Slack HITL multi-row approvals (agno PR #7574)
- Updates 41 transitive deps via ./scripts/generate_requirements.sh upgrade
- Adds approvals_table="coda_approvals" to AgentOS PostgresDb so create_all
  provisions the HITL state table on boot
- Add pytest + pytest-asyncio to dev deps in pyproject.toml
- Configure pytest with asyncio_mode=auto and testpaths=tests
- Lift agno's HITL test fixtures (make_signed_request, build_app,
  make_agent_mock, make_slack_mock, make_async_client_mock,
  make_stream_mock, make_requirement, wait_for_call) verbatim from
  agno/libs/agno/tests/unit/os/routers/conftest.py
- Add a smoke test verifying agno 2.6.x is installed and
  approvals_table is wired on the AgentOS PostgresDb

No source changes — pure test scaffolding so commits 3-15 can land
with their own per-feature tests.
Adds Slack HITL multi-row approval gating to the Triager specialist.
When the LLM emits N parallel tool calls (e.g. close 12 SLOP issues +
label 3), agno renders a single Slack card with N rows for per-action
approval before any GitHub write fires.

Gated ops (all 7 public-write GithubTools methods):
- close_issue, comment_on_issue, label_issue, create_issue,
  reopen_issue, assign_issue, edit_issue

Read ops (list_issues, get_issue, search_*, etc.) stay un-gated.

Tests:
- tests/test_triager_hitl.py — 2 unit tests asserting the kwargs are
  set on the right toolkit and read ops are not over-gated
Adds the second HITL pause type (user_feedback) to the Triager
specialist. When triaging a bug where severity isn't obvious from
the issue text, Triager now calls ask_user with structured questions
(severity P0-P3, component) and uses the human's response to apply
the right priority/component labels.

Changes:
- Import UserFeedbackTools from agno.tools.user_feedback
- Add UserFeedbackTools() to triager.tools list
- Add "Asking for Severity (HITL)" instruction section telling the
  LLM when to call ask_user (only for ambiguous bug severities, not
  for clear cases like SLOP, DUPLICATE, QUESTION)

Pause flow: ask_user (user_feedback card) -> human picks options ->
label_issue (confirmation card from commit 3) -> approved -> applies.

Tests:
- tests/test_triager_hitl.py adds test_triager_has_user_feedback_tools
GitTools.__init__ currently has a fixed signature (base_dir, read_only)
and calls super().__init__(name=..., tools=...) without forwarding
**kwargs. As a result, any Toolkit-level HITL kwargs passed to
GitTools (requires_confirmation_tools, external_execution_required_tools)
are silently dropped.

This commit adds **kwargs to the signature and forwards them to the
base Toolkit constructor, enabling commit 6 to wire git_push behind
HITL confirmation:

    GitTools(read_only=False, requires_confirmation_tools=["git_push"])

Backward compatible: existing callers in triager/explorer/planner
(GitTools(base_dir=..., read_only=True)) are unaffected.

Tests:
- tests/test_git_tools_hitl.py asserts kwarg forwarding works and
  default construction has no false-positive HITL flags.
Adds HITL confirmation gates to Coder's two highest-blast-radius
external writes: pushing a branch and opening a PR. Before either
fires, agno posts a single Approve/Deny card to Slack.

Changes:
- GitTools(requires_confirmation_tools=["git_push"]) — gates the push
- GithubTools(requires_confirmation_tools=["create_pull_request"]) —
  gates the PR

Intentionally NOT gated:
- create_worktree, remove_worktree (local sandbox)
- edit_file, write_file, run_shell (Coder needs free hands inside
  the worktree — gating would make it useless)
- create_issue, comment_on_issue (low-volume from Coder; can extend
  later if needed)

Existing safety still applies on top of HITL:
- git_push branch allowlist (coda/* only) at coda/tools/git.py:381
- Worktree path-traversal guard at coda/tools/git.py:43-54

Tests:
- tests/test_coder_hitl.py: 3 unit tests asserting both gates fire
  and local-only ops stay un-gated.
Adds the second multi-row HITL showcase to the PR. When Planner
emits N parallel create_issue calls in a single response (typical
for a "plan how to add X" request that produces 3-7 issues), agno
collects them into one Slack card with N rows for per-issue approval.

Why only create_issue: Planner also has label_issue but calls it
per-issue right after creation. Gating both would create N+M card
noise. Stick with multi-row create_issue for a clean showcase.

Tests:
- tests/test_planner_hitl.py: 2 unit tests asserting create_issue
  is gated and read ops are not over-gated.
Adds HITL confirmation gates to Explorer's two public-write ops:
- create_pull_request_comment (inline PR review comments)
- comment_on_issue (issue comments)

Explorer is read-mostly by spirit, but these two methods can write
to public discussions. Confirmation makes "read-only by spirit"
enforced by code.

When Explorer wants to leave a single comment, a single Approve/Deny
card fires. When it leaves multiple comments in one PR review pass,
agno renders a multi-row card.

Tests:
- tests/test_explorer_hitl.py: 2 unit tests asserting both writes
  are gated and the 10 read-only ops are not over-gated.
Reworks tasks/review_issues.py so the daily triage job can render a
Slack HITL approval card for paused Triager runs and resume via the
existing /interactions route.

Key pieces:
- triage_repo() runs Triager with session_id=f"{triager.id}:{thread_ts}"
  matching the format agno's interactions router reconstructs from the
  Slack click payload — enabling resume without modifying agno.
- _post_header() posts a header message to TRIAGE_CHANNEL and returns
  its ts so any HITL card threads under it.
- _post_hitl_card() delegates to agno's post_pause_card helper for
  the actual Block Kit rendering and posting.
- _has_pending_triage() prevents stacked cards when a previous run's
  approval is still unresolved.
- Falls back to dashboard-only mode when SLACK_TOKEN/TRIAGE_CHANNEL
  aren't set; pauses persist in /approvals for browser approval.

The schedule + endpoint are still commented out in app/main.py;
commit 10 re-enables them.

Tests:
- tests/test_cron_hitl.py: 7 unit tests covering dedup behavior,
  header posting, HITL card delegation, and session_id format.
Activates the daily-issue-triage cron job that was disabled in
e8e9309. The HITL plumbing landed in the previous commit, so the
re-enabled job now posts a multi-row approval card to TRIAGE_CHANNEL
instead of taking destructive actions autonomously.

Changes (all in app/main.py):
- Uncomment the run_daily_triage import
- Uncomment the schedule registration (cron 0 4 * * * UTC, gated on
  TRIAGE_CHANNEL + SLACK_TOKEN being set)
- Uncomment the POST /triage-issues endpoint that the scheduler hits

Manual trigger for testing:
    curl -X POST http://localhost:8000/triage-issues
- Pin agno[os]>=2.6.6,<2.7 in pyproject.toml so smoke test stays
  valid against future minor releases.
- Add .memory/, codatest.md, and uv.lock to .gitignore so local
  artifacts and the manual test playbook never get accidentally
  committed.
- Add "Approvals (Slack HITL)" section to README explaining the
  approval flow across all specialists + cron behavior.
- Add comment to example.env clarifying TRIAGE_CHANNEL's role in
  the cron HITL flow.
Long tool args made agno's pause card exceed Slack's block-body limit, silently failing to render the approval card. Patch agno's _build_confirmation_card to truncate; wired in at app startup.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Planner produced text plans instead of calling create_issue. Make the tool call mandatory and add a Handling Denials section so denied issues are not retried.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
ask_user is LLM-discretionary; gpt-5.4 answered in plain text instead of firing the Slack form. The requires_confirmation gate on label_issue already covers priority-label decisions.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Coder's run_shell (needed for autonomous testing) bypasses the structured git_push/create_pull_request gates. Gating a shell-enabled agent needs an allowlist approach - deferred to a follow-up PR.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Resolves ruff E402 - install() need not precede imports, only run before a HITL pause card is built.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings May 16, 2026 15:00

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds Slack human-in-the-loop (HITL) approval gating for destructive GitHub actions across the Triager, Planner, and Explorer agents, and reworks the daily cron triage to post a header + threaded approval card with cross-run dedup. Pins agno[os] to >=2.6.6,<2.7 and bumps the locked requirements, adds a Slack patch to truncate confirmation-card body text under Slack's block limit, and introduces a pytest harness with 17 tests covering the wiring.

Changes:

  • Wire requires_confirmation_tools onto Triager / Planner / Explorer GithubTools and harden Planner instructions; allow GitTools to forward HITL kwargs.
  • Rework tasks/review_issues.py for HITL-gated cron triage (pending-run dedup, header + threaded card) and re-enable the /triage-issues endpoint and 04:00 schedule.
  • Add coda/slack_patches.py (cap confirmation-card body) and configure approvals_table="coda_approvals" on the Postgres DB; bring up pytest infrastructure and tests.

Reviewed changes

Copilot reviewed 18 out of 21 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
README.md Documents the new HITL approval workflow.
app/main.py Installs Slack patch, re-enables /triage-issues endpoint and cron schedule.
coda/agents/explorer.py Gates comment_on_issue and create_pull_request_comment.
coda/agents/planner.py Gates create_issue and updates instructions to mandate calls and handle denials.
coda/agents/triager.py Gates all destructive issue ops.
coda/slack_patches.py Monkey-patches agno confirmation-card builder to truncate body text.
coda/tools/git.py Forwards **kwargs (e.g. requires_confirmation_tools) to Toolkit base.
db/session.py Configures approvals_table="coda_approvals".
example.env Notes new HITL behavior for TRIAGE_CHANNEL.
pyproject.toml Pins agno to 2.6.x; adds pytest dev deps and pytest config.
requirements.txt Regenerated lock for the agno bump.
tasks/review_issues.py Reworked for HITL with header, threaded card, pending-run dedup.
tests/conftest.py New pytest fixtures (mocks, signed Slack requests).
tests/test_*.py New unit tests for HITL wiring, kwargs forwarding, and cron plumbing.
.gitignore Adds logs/, uv.lock, .memory/, codatest.md.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tasks/review_issues.py Outdated
try:
with db.engine.connect() as conn:
result = conn.execute(
text(f"SELECT count(*) FROM {table} WHERE status = 'pending'")
Comment thread tasks/review_issues.py Outdated
Comment on lines +250 to +284
header_ts: Optional[str] = None
if slack_client:
header_ts = _post_header(slack_client, channel, name)

session_id = (
f"{triager.id}:{header_ts}"
if header_ts
else f"cron-triage-{name}-{int(time.time())}"
)

try:
summary = triage_issues(issues, owner_repo)
response = triage_repo(owner_repo, issues, session_id)
except Exception:
log.exception("Triager agent failed for %s — skipping", name)
continue

post_triage_to_slack(summary, name)
status = getattr(response, "status", None)

if status == "PAUSED" and slack_client and header_ts:
try:
asyncio.run(_post_hitl_card(token, response, channel, header_ts))
log.info("HITL card posted for %s — awaiting approval", name)
except Exception:
log.exception("Failed to post HITL card for %s — fall back to dashboard", name)
elif status == "PAUSED":
log.info("Triager paused for %s — approve via agno dashboard /approvals", name)
else:
if slack_client and header_ts:
summary = getattr(response, "content", "") or "Triage complete."
try:
slack_client.chat_postMessage(
channel=channel, text=summary, mrkdwn=True, thread_ts=header_ts,
)
except SlackApiError as exc:
log.error("Slack summary post failed for %s: %s", name, exc.response.get("error"))
Comment thread coda/slack_patches.py Outdated
Comment on lines +30 to +32

def install() -> None:
"""Idempotently install the body-text cap on agno's confirmation card builder."""
Comment thread tests/test_cron_hitl.py Outdated
Comment on lines +79 to +103
def test_session_id_format_when_header_ts_present():
"""session_id format must match agno's contract: f'{triager.id}:{header_ts}'."""
from coda.agents.triager import triager

header_ts = "1234567890.123456"
session_id = f"{triager.id}:{header_ts}"

# Format check: exactly two colon-separated parts, second part is the ts
parts = session_id.split(":")
assert len(parts) == 2
assert parts[0] == triager.id
assert parts[1] == header_ts


def test_session_id_falls_back_to_synthetic_when_no_header():
"""Without a Slack header_ts, session_id uses a synthetic cron-triage prefix."""
import time as time_module

repo_name = "agno"
fake_ts = 1700000000
expected = f"cron-triage-{repo_name}-{fake_ts}"

with patch("time.time", return_value=fake_ts):
synthetic = f"cron-triage-{repo_name}-{int(time_module.time())}"
assert synthetic == expected
@Himanshu040604 Himanshu040604 changed the title feat: Slack HITL approval gating for destructive agent actions feat: Slack HITL approval gating for agent actions May 16, 2026
@Himanshu040604 Himanshu040604 changed the title feat: Slack HITL approval gating for agent actions feat: Slack HITL agent actions 2.6.6 May 16, 2026
- Header-post failure: skip the repo + log a warning when _post_header
  returns None, instead of running Triager with no Slack delivery path.
- slack_patches.install(): sentinel guard so repeat/reload installs
  don't re-wrap the confirmation-card builder.
- Cron session_id: extract _build_session_id() helper; tests exercise
  it instead of reconstructing the string inline.
- Remove _has_pending_triage() dedup: it queried coda_approvals, which
  requires_confirmation HITL never populates (that table is agno's
  separate Approvals feature) - the dedup never worked. A correct
  dedup is a follow-up.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@Himanshu040604

Copy link
Copy Markdown
Author

I addressed all the issues by copilot, also removed a few things for future work in the v1

CI's validate job failed at the format step, masking 16 latent mypy
errors (the lint and type-check steps never ran).

- ruff format: reformat review_issues.py + 3 test files to ruff style.
- mypy: add type: ignore comments for 16 errors across conftest.py and
  the HITL test files (broad agno union types, **kwargs construction,
  Mock.await_args indexing, a function-attribute sentinel). Type-only
  changes - no runtime behavior change.

Verified locally with CI's exact commands (ruff format --check, ruff
check, mypy) - all pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mustafa-Esoofally added a commit to agno-agi/agno that referenced this pull request May 23, 2026
## Summary

Slack Block Kit rejects **Card block** body text longer than 200
characters with `invalid_blocks` error, causing HITL approval cards to
silently fail to render. When this happens, agents pause indefinitely
waiting for approval that can never come.

This fix truncates the card body text using the existing `truncate()`
helper (already in `types.py`) before building the card.

## Changes

- Import `truncate` from `agno.os.interfaces.slack.types`
- Apply `truncate(body_text, 200)` in `_build_confirmation_card()` 
- Apply `truncate(body_text, 200)` in `build_confirmation_toggle_card()`
- Add 2 tests verifying truncation behavior

## Root cause

Tool calls like `comment_on_issue` with 250+ character arguments cause
the body text to exceed [Slack's Card block body limit of 200
chars](https://docs.slack.dev/reference/block-kit/blocks/card-block/).
The card post request returns `ok: false` with `invalid_blocks` error,
but this was not being detected — the agent just hangs waiting for
approval.

Note: Section blocks have a 3000 char limit, but agno's HITL cards use
the newer Card block type (2024) which has stricter limits.

## How it was discovered

agno-agi/coda#23 implemented a monkey-patch workaround
(`coda/slack_patches.py`) to fix this. This PR moves the fix upstream
where it belongs.

## Test plan

- [x] Ruff passes
- [x] 46 unit tests pass (`pytest
libs/agno/tests/unit/os/routers/test_slack_blocks.py`)
- [x] Added `test_long_tool_args_truncated_to_200_chars` for
`_build_confirmation_card`
- [x] Added `test_long_body_truncated_to_200_chars` for
`build_confirmation_toggle_card`
- Remove verbose "MUST actually call create_issue" phrasing — testing
  proved the model calls tools naturally without emphatic instructions
- Revert description changes inside commented-out triage code
- Gate git_push, create_pull_request, create_issue, comment_on_issue
- Fix Interactivity URL path (/slack/interactions not /interactions)
- Remove disabled daily triage claim from Approvals section
Cherry-picked from PR #21, added requires_confirmation_tools for:
- git_push
- create_pull_request

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants