Verdi Researcher Guide — institutional recovery study

This guide is written for researchers who want to run and compare the fixed vs revisable institutional experiment using only config files and documented commands — no Python editing required. Related docs: the scenario gallery and the framework contract and limitations.

What this framework studies (plain language)

Verdi simulates a small village of AI agents that share resources, work under a common constitution, and periodically meet to propose and vote on rule changes. At a scheduled moment an external shock hits (food and credits disappear from the commons), and the simulation then measures whether the village can rebuild through production and cooperation. The experiment runs the same configured village twice: once where the constitution is fixed (agents can still deliberate and vote, but no amendment can ever be enacted) and once where it is revisable (valid amendments can take effect). Comparing the two tells us whether the ability to change the rules itself helps a society recover from disruption.

What it does NOT prove yet (honest limitations)

Adapted from INSTITUTIONAL_RECOVERY.md:

• No validated causal finding exists yet. Treat all results as provisional until they hold up over repeated runs.
• Seeding does not make LLM outputs deterministic. Seeds make the simulator reproducible (scheduling, order, world state), but model generations are still stochastic. Paired repetitions across seeds — not a single seed pair — are how you get confidence in a difference.
• Enacted narrative amendments remain non-binding. Only typed, machine- enforceable policy keys (max_transfer_units, max_withdraw_units, min_commons_reserve, max_produce_units_per_tick) change real behavior. Prose changes about fairness or rights stay descriptive metadata.
• Value-check self-reports are exploratory, not evidence that actions complied with a value. Primary outcomes should come from committed world state (delivery objectives), not agent claims.
• Future-dated version bookkeeping for multiple overlapping amendments needs a full audit before being relied on.
• Sanction/punishment rules are independent of policy-limit violations; a violation is logged (policy_limit) but not automatically punished.

Quickstart

Prerequisites

• Python 3.10+
• A language model server that speaks the OpenAI API (for example vLLM or Ollama)

pip install verdi-agents
verdi-agents init my-study && cd my-study

Configure the endpoint

verdi-agents init created a .env file. Put your key in it (never share or commit this file). The model name and server address go in the scenario's model: block.

# edit .env: set MODEL_API_KEY
# edit examples/institutional_recovery_constrained.yaml: set model.base_url and model.name

Run the paired experiment — ONE command

verdi-agents pair examples/paired_institutional_recovery.yaml

That's the whole workflow. The runner:

1. Loads the base scenario once.
2. Builds the same scenario for each arm (fixed, revisable), changing only institutional_condition — it asserts programmatically that nothing else differs, so the comparison cannot be silently biased.
3. Runs every (arm, seed) pair sequentially, writing a frozen scenario_resolved.yaml into each run dir.
4. Writes pair_manifest.json atomically after every arm. If a run was interrupted or one arm failed, re-running the same command resumes: completed pairs are skipped, failed ones re-run.

Where outputs land

runs/paired-institutional-recovery/
├── pair_manifest.json          # status of every (seed, arm) pair
├── fixed-seed42/
│   ├── report.html             # open this first
│   ├── scenario_resolved.yaml  # the exact frozen scenario used
│   ├── events.jsonl            # event log: every action, in order
│   └── ...                     # metrics.json, timeline.md, etc.
└── revisable-seed42/
    ├── report.html
    └── ...

Open report.html per arm in a browser to inspect the run. The manifest's per-pair fields (status, final_state_hash, replay_matches, error) tell you which pairs are trustworthy.

Optional CLI overrides (no config edit needed):

verdi-agents pair examples/paired_institutional_recovery.yaml \
  --seeds 42,7,99 \
  --output-root runs/my-experiment \
  --concurrency 4

Changing the experiment WITHOUT touching Python

Everything is driven by two YAML files:

1. The paired config — examples/paired_institutional_recovery.yaml
2. The base scenario it points at — examples/institutional_recovery_constrained.yaml

Editable fields in the paired config

Unknown keys are rejected with a readable error listing allowed keys.

Editable fields in the base scenario

"What if" recipes

All of these are pure config edits + re-run of the single command.

More seeds for uncertainty (the most important knob — one seed pair is a pilot, not a result):

# in the paired config
seeds: [42, 7, 99, 123, 555]     # five matched pairs

A stronger shock (make recovery harder):

# in the base scenario
shocks:
  - {tick: 14, resource: food, delta: -50.0}
  - {tick: 14, resource: credits, delta: -18.0}   # burn more of the credit reserve

A different production cap (tighten or loosen what the village can make):

economy:
  version: 1
  production_limit_per_tick:
    food: 1.0        # was 2.0 — scarcer production
  production_cost:
    food: 1.0
  consumption_per_tick:
    food: 1.0

Or let the society itself change the cap mid-run via an amendment: max_produce_units_per_tick is a typed, enforceable constitutional key, so a revisable-arm assembly can enact it and the validator will apply it from the effective tick onward (in the fixed arm the identical proposal is rejected with reason institution_fixed_rules).

Add an assembly meeting (more chances to revise rules — in BOTH arms, so meeting opportunities stay matched):

assemblies: [16, 32]      # was [] in the smoke scenario

Note: adding assemblies also gives the fixed arm more deliberation time, so interpret differences carefully — the intervention is enactment, not deliberation.

Reading the results

After the paired run completes, generate the comparison report and machine-readable metrics with:

verdi-agents compare runs/paired-institutional-recovery

This writes comparison.html, metrics.json, and metrics.csv at the paired output root. The comparison includes objective completion, committed and attempted resource flows, exclusion episodes, institutional mechanisms, and recovery measurements where there is enough recorded evidence. Inspect the individual events.jsonl event logs to see where a number came from.

Interpretation guardrails:

• A pair with status != complete in the manifest must be excluded or re-run; partial failure is flagged, never silent.
• One seed pair is directional at best. Report the number of matched seeds and their variance; do not claim an effect from a single pair.
• Compare committed outcomes (delivery success from world state), not agent claims.

Glossary

• Fixed condition — the constitution can be debated and voted on, but no amendment is ever enacted (rejections carry reason institution_fixed_rules).
• Revisable condition — valid amendments (typed, quorum + threshold met, temporally valid) are enacted and affect action validation from their effective tick.
• Amendment — a proposed change to the constitution. Only recognized, correctly typed policy keys are executable; prose stays metadata.
• Quorum — minimum fraction of eligible members who must participate for a vote to count (e.g. 0.5).
• Threshold — fraction of participating votes an amendment needs to pass (e.g. 0.67).
• Shock — a deterministic, externally scripted resource change at a given tick; identical in both arms.
• Delivery objective — an objectively checkable task: deliver X units of a resource to a pool by a due tick, verified from committed world state only.
• Recovery time — ticks needed to regain a preregistered target fraction of pre-shock successful-task rate; runs that never recover are censored, not dropped.
• Censored — a run that did not reach the recovery target within the horizon; kept in analysis as a censored observation rather than removed.
• Ledger / replay — the immutable event log (events.jsonl) plus the recorded model calls; replaying them must reproduce the final state hash (replay_matches in the manifest). This is the audit backbone.
• Seed pairing — running the same seed in both arms so schedules and initial states line up. Honest caveat: seeds do NOT make LLM outputs deterministic — model sampling is still stochastic. Paired repetitions across multiple seeds are how you get confidence, not one lucky pair.

Troubleshooting

General rule: if something looks wrong, the manifest is the source of truth about which pairs are usable, and each run dir's scenario_resolved.yaml is the source of truth about what actually ran.

Edit this page on GitHub