Add outcome-aware preflight contracts for v0.9 (#6)

Add VF010-VF013, structural outcome contracts, adversarial fixtures, documentation, and Friday release materials.
This commit is contained in:
Felipe Domingues 2026-07-22 13:58:46 -03:00 committed by GitHub
parent b312f18251
commit f62a78faaf
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
23 changed files with 1107 additions and 95 deletions

View file

@ -24,6 +24,18 @@ Rules have a stable ID, default severity, description, and remediation. Trusted
Graph policies use the validated n8n `main` connection shape, not node order; AI resource wiring is not control flow. VF005 requires a high-confidence atomic ledger gate that emits no item for duplicate claims and dominates inbound paths to side effects. VF006 rejects any AI entry path that bypasses a positive IF check against a direct agent-status reference. VF007 requires a reachable external handoff action.
VF010-VF013 introduce outcome contracts. High-confidence classification inspects a side-effect node's name, type, operation, URL, and parameters for money, customer communication, privileged access, and destructive-data semantics. Explicit `outcomeContracts` cover domain actions the built-in vocabulary does not recognize.
For a contracted action, the analyzer verifies graph evidence rather than labels alone:
- an atomic idempotency claim dominates the action;
- durable audit storage dominates the action;
- approval, amount, and counterparty guards are IF/Switch nodes with both an allowed route and a route that cannot reach the action;
- the declared failure notification is reachable from the action's error output;
- compensation and rollback nodes are downstream from the action; replay nodes must exist and carry recovery semantics.
Money and privileged outcomes default to blocking VF010. Customer communications and destructive writes default to advisory VF011. Missing recovery and silent error paths are separate findings so teams can triage them independently.
`--locked` rejects configuration that weakens built-in severity, changes safety vocabulary, or removes a default banned node. The GitHub Action always enables locked mode so a pull request cannot silence its own findings by editing `.vibeflow.json`.
## Trust boundaries
@ -35,6 +47,8 @@ Output paths are selected by the caller. The GitHub Action passes inputs as quot
## Known limitations
- Static structure cannot prove that a condition, SQL claim, or handoff works at runtime.
- Outcome contracts cannot prove caller authorization, amount calculation, counterparty identity, audit durability, or successful compensation at runtime.
- Automatic outcome classification is intentionally conservative and can miss domain-specific actions; declare them explicitly in configuration.
- Secret detection can miss unusual key names and can produce false positives.
- Community nodes unknown to the side-effect catalog need explicit policy additions.
- Locked mode protects policy content, but repository owners must still review changes to the CI workflow itself.

View file

@ -12,7 +12,7 @@ Prepared: 2026-07-22
Submit in **English**, even through the PT-BR form. OpenAI publishes no language requirement and no evidence that language changes selection odds. English is recommended only to reduce translation friction for a global technical review.
Submit now. Stars, forks, downloads, repository usage, ecosystem importance, and active maintenance are evaluation signals, not published minimum thresholds. Vibeflow's strongest evidence is its Codex-native engineering and its first real 92-node audit, not early popularity metrics.
Submit immediately after the v0.9.0 release on 2026-07-24. Do not wait for arbitrary star, fork, or PR targets. Vibeflow's strongest evidence is its Codex-native engineering, first real 92-node audit, and a public maintenance loop that turned community feedback into tested policy features.
## Copy-and-paste form
@ -42,9 +42,9 @@ Select: **Principal maintainer**.
### Why is this repository eligible?
Character count: **420/500**.
Recount before submission; keep under 500 characters.
> Vibeflow is an MIT-licensed safety and contract gate for AI-generated n8n workflows, rebuilt end-to-end with Codex. Its v0.8.0 plugin audited a real 92-node workflow in read-only mode and produced provenance-backed results: 3 blocking AI-safety findings, 57 prioritized warnings, and concrete revalidation gates. It fills a growing n8n need: deterministic pre-production checks for workflows created by people or agents.
> Vibeflow is an MIT-licensed safety gate for AI-generated n8n workflows, rebuilt end-to-end with Codex. It audited a real 92-node workflow in read-only mode, finding 3 blocking risks and 57 warnings. Community feedback then shaped v0.9: outcome-aware checks for refunds, customer actions, silent failures, audit, idempotency, approval, limits, and recovery.
### Interests
@ -67,9 +67,9 @@ Character count: **390/500**.
### Anything else we should know?
Character count: **431/500**.
Recount before submission; keep under 500 characters.
> Vibeflow is a public case study in Codex-native OSS development. Codex helped reposition an old prototype, implement the CLI, run repeated Red Team reviews, fix regressions, package a GitHub Action and Codex plugin, open and merge PRs, and publish v0.8.0. Its first production-scale audit exposed workflow risks and analyzer limitations, creating a concrete maintenance backlog. Adoption is early; the engineering evidence is real.
> Vibeflow is a public case study in Codex-native OSS development. Codex drove the project from product repositioning through implementation, adversarial review, remediation, CI, plugin packaging, releases, a real-workflow audit, and the v0.9 response to user feedback. Adoption is early, but the engineering, maintenance history, and real-world evidence are public and reproducible.
## PT-BR review translation
@ -77,7 +77,7 @@ These translations are for review only. Paste the English versions above into th
### Repository eligibility
> Vibeflow é um gate MIT de segurança e contratos para workflows n8n gerados por IA, reconstruído de ponta a ponta com Codex. O plugin v0.8.0 auditou em modo somente leitura um workflow real de 92 nós e produziu resultados com proveniência: 3 bloqueios de segurança de IA, 57 avisos priorizados e gates concretos de revalidação. Ele atende a uma necessidade crescente do ecossistema n8n: checks determinísticos antes da produção para workflows criados por pessoas ou agentes.
> Vibeflow é um gate MIT de segurança para workflows n8n gerados por IA, reconstruído de ponta a ponta com Codex. Ele auditou em modo somente leitura um workflow real de 92 nós, encontrando 3 riscos bloqueantes e 57 avisos. O feedback da comunidade então moldou a v0.9: checks de resultados reais para reembolsos, ações com clientes, falhas silenciosas, auditoria, idempotência, aprovação, limites e recuperação.
### API-credit use
@ -85,14 +85,14 @@ These translations are for review only. Paste the English versions above into th
### Additional context
> Vibeflow é um estudo de caso público de desenvolvimento open source nativo em Codex. O Codex ajudou a reposicionar um protótipo antigo, implementar a CLI, executar Red Teams repetidos, corrigir regressões, empacotar uma GitHub Action e um plugin Codex, mesclar PRs e publicar a v0.8.0. A primeira auditoria em escala de produção revelou riscos do workflow e limitações do analisador, criando um backlog concreto. A adoção ainda é inicial; a evidência de engenharia é real.
> Vibeflow é um estudo de caso público de desenvolvimento open source nativo em Codex. O Codex conduziu o projeto desde o reposicionamento e implementação até revisão adversarial, correções, CI, plugin, releases, auditoria de workflow real e a resposta da v0.9 ao feedback de usuários. A adoção ainda é inicial, mas a engenharia, o histórico de manutenção e a evidência real são públicos e reproduzíveis.
## Evidence map
Use these links only if OpenAI requests verification; the form has no dedicated evidence field.
- Public MIT repository: <https://github.com/domfelipe/vibeflow-n8n>
- Executable release: <https://github.com/domfelipe/vibeflow-n8n/releases/tag/v0.8.0>
- Executable release after Friday launch: <https://github.com/domfelipe/vibeflow-n8n/releases/tag/v0.9.0>
- Reproducible demo: <https://github.com/domfelipe/vibeflow-n8n/blob/main/docs/demo.md>
- Release and Red Team audit: <https://github.com/domfelipe/vibeflow-n8n/blob/main/docs/release-audit.md>
- CI history: <https://github.com/domfelipe/vibeflow-n8n/actions/workflows/ci.yml>
@ -101,16 +101,16 @@ Use these links only if OpenAI requests verification; the form has no dedicated
## Evidence snapshot
Captured on 2026-07-22:
Refresh this section on 2026-07-24 immediately before submission. Current candidate evidence:
- public repository with MIT license;
- `v0.8.0` release;
- 26 adversarial tests on Node.js 20, 22, and 24;
- `v0.9.0` release candidate, with v0.8.0 already public;
- 36 adversarial tests passing locally and in remote Node.js 20, 22, and 24 CI;
- dependency-free CLI, GitHub Action, and installable Codex plugin;
- 3 maintainer PRs merged with green CI;
- 3 prior maintainer PRs merged with green CI; add the v0.9 PR after merge;
- 2 stars, 0 forks, and no verified external contributor yet;
- first real audit: anonymized 92-node workflow, 3 blocking findings, 57 warnings, no workflow mutation;
- Codex used across product repositioning, implementation, review, Red Team, remediation, packaging, CI, release, and real-workflow audit.
- Codex used across product repositioning, implementation, review, Red Team, remediation, packaging, CI, release, real-workflow audit, and the community-feedback-driven v0.9 cycle.
Do not describe maintainer PRs, the maintainer's own workflow, clones, or unattributed stars as external adoption.
@ -144,7 +144,7 @@ The approved public description is: **“a real, production-scale 92-node conver
Continue collecting organic evidence without delaying the application:
1. publish the r/n8n post;
1. publish the v0.9 follow-up in r/n8n;
2. invite users to report anonymized false positives and missed unsafe cases;
3. respond to every issue or discussion with reproducible evidence;
4. keep releases, CI, and the public maintenance trail current;

View file

@ -16,6 +16,23 @@ node bin/vibeflow.mjs check examples/unsafe-support-agent.workflow.json --fail-o
Expected result: VF001-VF009 findings covering secrets, dangerous nodes, webhook authentication, error handling, idempotency, AI safety, timeouts, and retries.
## Dangerous outcome
```bash
node bin/vibeflow.mjs check examples/unsafe-refund.workflow.json --fail-on never
```
Expected result: the ordinary HTTP refund node triggers blocking VF010 plus VF012 and VF013, even though its node type is not inherently dangerous.
## Contracted outcome
```bash
node bin/vibeflow.mjs check examples/safe-refund.workflow.json \
--config examples/outcome-contracts.vibeflow.json
```
Expected result: exit `0`, zero findings. The graph contains a dominating atomic claim and durable audit, structural approval/amount/counterparty gates, error notification, and compensation evidence.
## Automation output
```bash

View file

@ -1,22 +1,28 @@
# Launch checklist
# v0.9.0 launch checklist
Target: Friday, 2026-07-24
## Release gate
- [x] `npm run verify` passes on Node.js 20, 22, and 24 in CI.
- [x] Official skill and plugin validators pass.
- [x] Safe fixture exits 0; unsafe fixture exits 1.
- [x] SARIF is valid JSON and uploaded by CI.
- [x] Repository description and topics match the new product.
- [x] `v0.8.0` release notes match `CHANGELOG.md`.
- [x] Remote CLI and pinned Codex marketplace install successfully.
- [x] Private vulnerability reporting and Discussions are enabled.
- [x] VF010-VF013 implementation and configuration schema are complete.
- [x] Safe and unsafe refund fixtures are reproducible.
- [x] Local `npm run verify` and `npm audit --omit=dev` pass.
- [x] QA, adversarial Red Team, and Guardião reviews are documented.
- [x] Pull request CI passes on Node.js 20, 22, and 24.
- [ ] Release commit is merged and tagged `v0.9.0`.
- [ ] Released CLI and pinned Codex marketplace install successfully.
- [ ] GitHub release is published on Friday.
Release: <https://github.com/domfelipe/vibeflow-n8n/releases/tag/v0.8.0>
## Positioning
Launch discussion: <https://github.com/domfelipe/vibeflow-n8n/discussions/3>
The launch message is: **dangerous outcomes are not limited to dangerous nodes**.
A normal HTTP, database, or messaging node can refund money, contact a customer, change access, or destroy data. Vibeflow v0.9 adds a preflight contract for the controls that should surround those outcomes: atomic idempotency, approval, amount and counterparty limits, durable audit, operator-visible failure paths, and recovery.
## Announcement
> Vibeflow is now an executable safety gate for AI-generated n8n workflows. It checks exported JSON for secrets, exposed webhooks, missing kill switches and handoffs, idempotency, failure paths, timeouts, and unsafe retries. It is dependency-free, runs locally or in GitHub Actions, and includes a Codex plugin.
> Vibeflow v0.9 asks a more useful preflight question for generated n8n workflows: not only “is this valid JSON?” or “does it use a dangerous node?”, but “what can this workflow do in the real world if the input is messy or the model is wrong?”
>
> The new outcome contracts detect money, customer, privileged, and destructive-data actions — including ordinary HTTP nodes — and verify structural evidence for idempotency, approval, limits, durable audit, failure notification, and recovery. It is local, dependency-free, CI-friendly, and explicit about what still needs runtime enforcement.
Link to the repository and the safe/unsafe demo. Ask users for anonymized false-positive cases and real workflow fixtures, not stars alone.
Link to the repository and the safe/unsafe refund demo. Ask users for anonymized workflows, classifier false positives/negatives, and missing domain actions.

View file

@ -6,7 +6,7 @@ Vibeflow is an open-source safety and contract gate for exported n8n workflows,
## Problem
Natural-language workflow generation is now common. The remaining failure is operational: a structurally valid workflow can still leak a credential, answer while disabled, duplicate a side effect, lack a human fallback, retry unsafely, or run forever.
Natural-language workflow generation is now common. The remaining failure is operational: a structurally valid workflow can still leak a credential, answer while disabled, duplicate a side effect, lack a human fallback, retry unsafely, run forever, issue a refund, notify a customer, or destroy data without the required controls.
Existing builders and MCP servers should keep building. Vibeflow checks the result before production.
@ -21,12 +21,14 @@ Existing builders and MCP servers should keep building. Vibeflow checks the resu
Given an exported workflow, produce a reproducible pass/fail report with concrete remediation and no network access.
## Version 0.8 scope
## Version 0.9 scope
- dependency-free Node.js CLI;
- text, JSON, and SARIF output;
- configurable VF000-VF009 policies;
- safe and unsafe fixtures;
- configurable VF000-VF013 policies;
- outcome contracts for money, customer, privileged, and destructive-data actions;
- structural checks for approval, durable audit, idempotency, limits, failure notification, and recovery;
- safe and unsafe support and refund fixtures;
- GitHub Action;
- Codex skill and plugin package.
@ -40,7 +42,7 @@ Given an exported workflow, produce a reproducible pass/fail report with concret
## Differentiation
Vibeflow starts with policies learned from real conversational-agent operations: an off switch before inference, human handoff after low-confidence output, duplicate-event protection, explicit error paths, bounded retries, and trust-boundary hygiene.
Vibeflow starts with policies learned from real operations: an off switch before inference, human handoff after low-confidence output, duplicate-event protection, explicit error paths, bounded retries, trust-boundary hygiene, and declared controls around real-world outcomes. It asks not only whether the workflow is valid JSON or uses a suspicious node, but what it can do when input is messy or a model is wrong.
## Evidence gate

View file

@ -1,54 +1,70 @@
# Release audit — v0.8.0
# Release audit — v0.9.0 candidate
Date: 2026-07-22
Target release: 2026-07-24
## Decision
**APTO COM RESSALVAS** for the first public executable release.
**APTO COM RESSALVAS** for the Friday release, pending merge, tag, and released-install checks listed below.
No blocking defect remains in the reviewed static-analysis boundary. The remaining caveats require runtime or node-specific knowledge that an exported JSON gate cannot prove.
No blocking defect remains in the reviewed static-analysis boundary. Vibeflow can verify structural evidence in an exported workflow, but it cannot enforce authorization, amount limits, counterparty identity, audit durability, or recovery behavior in the executing systems.
## QA evidence
- 26 automated tests pass, including one safe and one intentionally unsafe workflow.
- The safe fixture produces zero findings; the unsafe fixture produces VF001-VF009.
- Text, JSON, and SARIF output paths are exercised.
- Node 20, 22, and 24 are required in CI; local verification used the available Node runtime and the remote matrix is the release gate.
- JSON/YAML parsing, package dry-run, official skill/plugin validators, `npm audit`, and `git diff --check` are part of the final gate.
- 36 automated tests pass, including safe and unsafe support workflows and a fully contracted refund workflow.
- The unsafe refund fixture detects an ordinary HTTP node as a money outcome and produces VF010, VF012, and VF013.
- The contracted refund fixture exits 0 in normal and `--locked` modes.
- Text, JSON, SARIF, configuration validation, package packing, and CLI exit behavior are exercised.
- `npm run verify`, `npm audit --omit=dev`, JSON parsing, and `git diff --check` pass locally.
- The package remains dependency-free and targets Node.js 20+.
## Red Team
The first implementation was rejected. Regression tests now cover the reproduced bypasses:
The first v0.9 implementation was hardened after adversarial review. Regression tests now cover:
- declared webhook authentication without the matching credential reference;
- literal secrets in raw headers, URLs, expressions, pinned data, and static data;
- disconnected, nominal, or non-gating idempotency controls;
- parallel AI paths that bypass a kill switch, inverted or constant conditions, and decorative Code/NoOp nodes;
- resource connections misread as control flow and fabricated connection shapes;
- disconnected or cyclic error handling;
- read-only HTTP requests mislabeled as handoff;
- excessive or non-numeric timeouts and unsafe retry settings;
- policy weakening from an untrusted checkout;
- terminal-control injection, deep JSON, excessive nodes/edges/files/config terms, finding amplification, and quadratic traversal.
1. ordinary HTTP refunds that would evade a dangerous-node-only policy;
2. read-only refund queries, preventing an obvious classifier false positive;
3. named or disconnected approval, amount, counterparty, audit, and idempotency controls;
4. constant approval conditions and decorative amount values;
5. read-only or fail-open audit nodes, including an audit error branch that still reaches the action;
6. fail-open idempotency nodes and idempotency error branches that still reach the side effect;
7. prototype-like contract keys, self-referential notification/recovery evidence, and audit-like labels used to hide a money action;
8. connected error paths that terminate without a recognized operator notification.
At the 5,000-node limit, the corrected linear traversal completed the synthetic chain in tens of milliseconds on the development machine. Finding truncation always adds blocking VF000.
The graph checks use actual `main` edges, require dominating controls, and reject evidence that reaches the action after the control itself fails.
## Supply chain
## Guardião security review
- GitHub-owned actions are pinned to full verified commit SHAs.
- Checkout credentials are not persisted; workflow permissions are `contents: read`.
- Jobs have a ten-minute timeout and package installation ignores scripts.
- The package has no runtime dependencies and uses a publish allowlist.
- The bundled action always enables `--locked`.
### In scope
The first merge SHA, `4998605ed7dc12b9b867d69d7005d25778c7e109`, pins the CLI, GitHub Action, and Codex marketplace examples before the release tag is created.
- local CLI parsing of untrusted workflow/configuration JSON;
- graph and parameter analysis;
- text, JSON, and SARIF output;
- npm package contents, GitHub Action boundary, and Codex plugin instructions.
## Residual limitations
### Controls confirmed
- Static analysis cannot prove a referenced credential exists or works.
- SQL and IF checks are high-confidence structural evidence, not runtime execution proofs.
- An HTTP POST labeled as a ticket or handoff may still fail or target the wrong service.
- Unknown community nodes may require a new side-effect adapter and regression fixture.
- Repository owners must still review changes to the CI workflow itself.
- no runtime dependencies, telemetry, hosted service, or live n8n mutation;
- no workflow expressions or embedded code are evaluated;
- no URLs found in a workflow are contacted;
- file, node, edge, contract, vocabulary, and finding budgets are bounded;
- terminal text is sanitized and prototype-like contract keys are handled as data;
- the GitHub Action uses locked policy mode so a pull request cannot disable its own built-in checks;
- examples contain credential references and reserved invalid domains, not live secrets.
The official Codex Security workspace was opened, but its setup was never submitted through the app interface; no result from that scanner is claimed here. The release decision is based on the direct QA, independent Red Team, supply-chain review, and regression evidence above.
### Residual limitations
- an exported graph cannot prove a referenced credential, approval identity, SQL policy, external API limit, notification, or compensation works at runtime;
- custom/community nodes may need explicit impact declarations or new regression-backed adapters;
- repository owners may intentionally weaken policy outside `--locked` mode;
- remote Node 20/22/24 CI passed; released `npx`/plugin installation can only be confirmed after the candidate is tagged.
## Release blockers
- [x] Pull request CI passes on Node.js 20, 22, and 24.
- [ ] Release commit is merged without unrelated changes.
- [ ] `v0.9.0` tag and GitHub release are published on 2026-07-24.
- [ ] Released CLI and Codex plugin install paths are smoke-tested.
## Final gate
There are zero known critical or high security findings in the candidate. The release remains **APTO COM RESSALVAS** until the remaining remote gates above are complete; a failed gate blocks publication or requires an immediate corrective release.

View file

@ -0,0 +1,24 @@
# Vibeflow v0.9.0 — Outcome-aware preflight for n8n
Generated workflows need more than valid JSON. A normal HTTP node can still issue a refund, notify a customer, change access, or delete production data.
Vibeflow v0.9 introduces outcome contracts and four policies:
- **VF010** blocks uncontracted money and privileged actions.
- **VF011** warns on uncontracted customer communications and destructive writes.
- **VF012** detects connected error paths that notify nobody.
- **VF013** requires a compensation, rollback, or replay story.
For contracted actions, Vibeflow checks whether atomic idempotency and durable audit dominate the action, approval and limit nodes have real allow/deny branches, failure notification is connected to the error output, and recovery is represented in the graph.
Try the unsafe case:
```bash
npx --yes github:domfelipe/vibeflow-n8n#v0.9.0 check examples/unsafe-refund.workflow.json --fail-on never
```
Then inspect the passing contract in `examples/outcome-contracts.vibeflow.json` and `examples/safe-refund.workflow.json`.
This is a static preflight, not a runtime policy engine. Authorization, amount/counterparty enforcement, audit durability, and tested recovery still belong in the services that execute the action.
Full changelog: <https://github.com/domfelipe/vibeflow-n8n/blob/v0.9.0/CHANGELOG.md>

View file

@ -4,17 +4,23 @@
Ship the executable reset: CLI, nine configurable policies, fixtures, tests, SARIF, GitHub Action, and Codex plugin.
## 0.9.0 — target 2026-07-24
Separate dangerous nodes from dangerous outcomes. Add VF010-VF013, explicit outcome contracts, graph evidence for policy gates, safe/unsafe refund fixtures, and clear runtime boundaries.
## Next release gate
Do not add another integration by default. Prioritize evidence from real workflows:
1. Measure false positives by rule.
1. Measure false positives by rule and impact classifier.
2. Accept anonymized fixtures from external users.
3. Add a node type or policy only with a failing fixture.
4. Improve GitHub annotations if SARIF users request it.
5. Package for npm only when GitHub installation creates material friction.
6. Add node-specific handoff and side-effect adapters only with adversarial safe/unsafe fixtures.
7. Model additional atomic idempotency gates only when their duplicate path is proven to stop downstream items.
8. Add impact categories and adapters only with a real anonymized workflow and an adversarial fixture.
9. Explore optional runtime attestations only after users demonstrate that static contracts are insufficient.
## Explicitly deferred