Enclave SOP

Opplet Operations: Standard Operating Procedures

Version 1.3 · DRAFT · Tier 2 · part of Charter Release 2026.2 · effective 2026-06-10

Purpose and Scope

This document holds the operational cadences, thresholds, and runbook procedures that the Constitution mandates but does not specify. The Constitution declares what must happen; this SOP defines how often, at what threshold, and through which mechanism.

These procedures may be tuned by the Custodian (Constitution §13B) without Constitutional amendment as operational experience accumulates. Changes are recorded in this document’s changelog and version-bumped accordingly.

1. The Backup Bridge (implementing Constitution §5A Exception 3)

1A. Backup Frequency and Schedule

The Annex pushes state to Proxmox Backup Server (PBS) on the Manor every four hours at fixed UTC times:

• 00:00 UTC
• 04:00 UTC
• 08:00 UTC
• 12:00 UTC
• 16:00 UTC
• 20:00 UTC

1B. Canary Verification

n8n-Alpha runs a scheduled workflow every four hours, offset by 30 minutes from the backup window (i.e., 00:30, 04:30, 08:30, etc.), that checks the latest PBS backup timestamp.

• Trigger condition: If the most recent backup timestamp is older than 5 hours, n8n-Alpha fires a high-priority Pushover alert to the Custodian.
• Alert level: Critical (Pushover priority +2).

1C. GitLab Backup Integrity Test

Once per week (Sunday 03:00 UTC), n8n-Alpha triggers a test restore of the latest GitLab backup archive into a throwaway container on the Manor.

• Trigger condition: If the integrity check fails (archive corrupt, restore errors out, or post-restore database query fails), a critical alert is dispatched immediately.
• Cleanup: The throwaway container is destroyed within 30 minutes of test completion regardless of pass/fail outcome.

2. The External Pulse (implementing Constitution §10B observability mandate)

2A. Uptime Monitoring

A micro-VPS running Uptime Kuma performs continuous reachability checks against enclave endpoints.

• Public IP pings: Every 60 seconds against the Annex, Manor, and Den public IPs.
• Service-level health checks: Every 120 seconds against:
  • GitLab health endpoint
  • Moodle login page
  • Traefik dashboard
  • Authentik-Business OIDC discovery URL
  • Den Gateway SMTP port (25)
  • Den Gateway IMAP port (993)
  • Den Gateway SIP registration

2B. Dead Man’s Switch

If the Manor goes dark (three consecutive failed pings), Uptime Kuma dispatches a high-priority Pushover notification to the Custodian via 5G/LTE — independent of any in-enclave alerting that would also be affected by the Manor’s failure.

3. Active Alerting Baseline (implementing Constitution §10B)

The following alert rules are mandatory across all zones and must be configured in Wazuh and/or Grafana. These thresholds may be tuned based on operational experience without Constitutional amendment.

Tuning policy: Thresholds should be reviewed quarterly during the RAM Headroom Audit window (see §6) and adjusted based on observed false-positive and false-negative rates. Adjustments are documented in this SOP’s changelog.

4. OPNsense Resilience Procedures (implementing Constitution §5E)

4A. Configuration Export

n8n-Alpha exports the OPNsense configuration backup (XML) to BookStack-Alpha (The Grimoire) every 24 hours at 02:00 UTC.

• Storage location: BookStack-Alpha shelf “OPNsense Configurations” — page named with ISO date prefix (e.g., 2026-06-02-opnsense-config).
• Retention: Last 30 days kept in BookStack-Alpha; older exports archived to PBS as part of the standard Manor backup.

4B. Recovery Target

With a current config export and a Proxmox template, OPNsense must be rebuildable from scratch within 30 minutes. This target is verified during quarterly tabletop exercises (see §7).

4C. HA Priority

The OPNsense VM holds the highest HA restart priority on the Manor cluster. Any change to HA priorities must verify OPNsense remains at maximum before applying.

5. Disaster Recovery Procedures (implementing Constitution §9)

The Constitution defines RTO/RPO commitments and the rebuild priority order. This section defines the procedural implementation.

5A. Backup Implementations Per Node

The Manor:

• Local ZFS replication with 15-minute snapshot interval (achieves RPO of 15 minutes).
• Snapshots retained per standard ZFS schedule: hourly for 24h, daily for 30d, weekly for 12w, monthly indefinitely.

The Annex:

• State pushed to Manor PBS every 4 hours per §1A (achieves RPO of 4 hours).
• GitLab artifacts and Moodle database included in every push.

The Outpost:

• VM templates maintained on the Outpost local ZFS.
• No PBS push — targets are rebuildable from templates.
• Wazuh telemetry continuously forwarded to Watchtower (not a backup; non-repudiation only).

The Den Gateway:

• HestiaCP and FreePBX configuration files exported daily to the Den Engine VPS via Tailscale (automated by n8n-Den).
• A copy stored in Vaultwarden (Den instance) as an encrypted attachment.

The Den Engine:

• Hetzner Cloud automated snapshots run daily.
• Docker Compose files and environment configs version-controlled in a private Git repository on the Engine itself.
• Mirror push to a separate Hetzner Storage Box daily.
• Seafile data on the mounted Hetzner Storage Box has its own independent snapshot schedule.

5B. DR Runbook Location

The complete step-by-step disaster recovery runbook, including credential bootstrap procedures, is stored in BookStack-Alpha (The Grimoire) at the shelf “Disaster Recovery / Runbook.” A printed hard copy of the credential bootstrap section is maintained in the Custodian’s physical safe.

5C. Multi-Node Failure Drill

When multiple nodes are affected simultaneously, follow the Rebuild Priority Order from Constitution §9 strictly. Do not skip ahead even if a lower-priority node appears easier to restore — identity must come up before identity-dependent services, edge routing must come up before any Hetzner service.

6. RAM Headroom Audit (implementing Constitution §10B principle, operationalized)

Memory contention is a silent availability risk. The following governance applies:

• Quarterly Review: Actual memory utilization per container/VM is reviewed against allocated RAM every quarter. Results logged in BookStack-Alpha shelf “RAM Audits.”
• 75% Ceiling: No node may sustain average memory utilization above 75% of total physical RAM over a rolling 7-day window. Breaching this threshold triggers a mandatory rebalancing review within 14 days.
• Den Engine Ceiling: The Engine VPS must maintain at least 1 GB of free RAM at all times to absorb container startup spikes.

Schedule: Audits occur in the first week of January, April, July, and October. Same window used for alert threshold tuning review (§3) and tabletop exercise (§7).

7. Tabletop Exercise Cadence

A tabletop DR walkthrough must be conducted quarterly. The exercise simulates the total loss of one node and walks through the rebuild sequence from bare metal to service restoration.

• Schedule: Same quarterly window as the RAM Audit and alert tuning review.
• Scope: One node simulated as totally lost per exercise. Rotate through nodes across the year (Manor → Annex → Outpost → Den Gateway → Den Engine → Manor → …).
• Documentation: Findings recorded in BookStack-Alpha shelf “DR Exercises.”
• Gap resolution: Any identified gaps must be resolved within 14 days of the exercise.

8. Documentation Structure (implementing Constitution §6)

The Constitution mandates the documentation split; this SOP specifies the operational details.

8A. GitLab (The Annex – Kitchen) — Technical Source of Truth

• Infrastructure as Code (Terraform, Ansible playbooks)
• CI/CD pipeline definitions
• Opplet platform source code
• KenyaX, WiseNxt, CNMCyber static site source
• Public-facing technical documentation (mkdocs sources, etc.)

8B. BookStack-Alpha (The Manor – Basement) — The Grimoire

• This SOP and its revisions
• Participant Doctrine canonical copy
• DR runbook (§5B)
• OPNsense config exports (§4A)
• Quarterly audit logs (§6, §7)
• Custodian’s private notes and decision records
• Hardware Manifest current state
• Software Stack Manifest current state
• Den Migration Project status (until migration completes)
• Callsign suffix wordlist (per Participant Doctrine §7)

8C. BookStack-Beta (The Annex – Lounge) — The Common Library

Public shelves (anonymous read; LDAP-Beta write):

• Community onboarding guides
• Participant Doctrine participant-facing mirror
• Public FAQ
• Public-facing track descriptions

Member shelves (LDAP-Beta read and write):

• Internal community discussions
• Draft documents
• L1 work products (Scribe drafts, Herald slogan drafts, etc.)
• Gate 2 endorsement records (Developer-space vote summaries and curation notes, per Participant Doctrine §11)

8D. Den-Local Documentation

Docker Compose files and service configs version-controlled locally on the Engine VPS. The Den has no dependency on GitLab or BookStack for its own operation.

9. Routine Operational Checks

Cadences for procedures not covered above:

Every recurring review above touches the Basement (Zone 0) — observability config, n8n-Alpha, the Grimoire — which only the Custodian can access, so all of them, along with critical alerts and the credential-bootstrap safe (§5B), sit with the Custodian. The Tech Board’s role in operations is to fund and commission contracted work (Constitution §16; its mechanics are §10 below), not to execute Basement procedures it cannot reach.

10. Recruitment and Disbursement (implementing Constitution §16)

Constitution §16 establishes the Tech Board, its authority to award paid contracts at Gate 3, and its disbursement of Economic Group funds. This section specifies the mechanics by which those are executed — the ERPNext recruitment workflow and the disbursement procedure. It sets no authority: who awards, the board’s composition, and the disbursement threshold are fixed in Constitution §16. These mechanics are tunable as the process matures; the authority above them is not.

10A. The Two Award Paths

Paid work reaches a candidate by one of two paths, matching the two contract types (Constitution §11 Gate 3):

• Project contract (one-time deliverable). A WiseNxt Track Lead scopes the work and recommends a specific Associate; there is no open posting. The Tech Board reviews the recommendation and awards or declines.
• Operation contract (term-based running of a standing system). The Tech Board defines the role and opens an internal posting; eligible members apply; the board screens, selects, and awards.

Both paths end in a Tech Board award, which is the Gate 3 crossing. Internal sourcing is absolute (Constitution §12 Rule 3): postings are visible only inside the directory, and only Gate-1 alumni may apply or be recommended.

10B. The ERPNext Recruitment Workflow

Requisitions, applications, and awards are recorded in the ERPNext recruitment module:

1. Requisition. The board — or, for a project contract, the recommending Track Lead — opens a requisition scoping the deliverable or term, the fee, and the least-privilege GitLab roles the work requires.
2. Posting (operation contracts only). The requisition is posted internally, visible to the LDAP-Alpha workforce. Project contracts skip this step; they arrive as a named recommendation.
3. Selection. Applicants, or the recommended Associate, are screened for Gate-1-alumni eligibility and fit; the board selects.
4. Award. The board records the award in ERPNext. This is the Gate 3 crossing: the Associate becomes a Contractor (Constitution §11).
5. Onboarding. No directory change occurs — the new Contractor already holds an LDAP-Alpha account as an Associate (Constitution §11). Onboarding records the contract scope and grants only the least-privilege GitLab roles the contract names; the callsign is unchanged.

10C. Disbursement

• Envelope and threshold. The Economic Group sets a budget envelope and a per-award approval threshold (Constitution §16). The board allocates within the envelope.
• Within threshold. Awards at or below the threshold are the board’s to make and disburse.
• Above threshold. Awards above the threshold require Economic Group approval before any funds move — the owner keeps large spend.
• Payment. A disbursement is recorded against the awarding contract in ERPNext finance and is traceable to it. Project contracts pay on delivery or milestone; operation contracts pay on the contract’s term schedule.

10D. Records and Access

Every requisition, award, and disbursement lives in ERPNext, an application in the Office zone; board decisions and Economic-Group threshold approvals are recorded there for audit (Pillar 4, “Observation is Truth”). The board administers ERPNext at the application layer through LDAP-Alpha — no root and no Basement access is required to run recruitment or disbursement. The infrastructure beneath ERPNext remains the Custodian’s to operate (§9). That is the separation in practice: the board runs its process, the Custodian runs the platform under it.

11. Cutting a Charter Release

The Opplet Project Charter is the complex of governed documents defined in Constitution §13; a Charter Release is a certified-coherent snapshot of it. Cutting one is the Custodian’s stewardship charge (Constitution §17): the Custodian reconciles the set, certifies that it coheres, and freezes the snapshot. The Tech Board has no part. This section is the mechanics; the authority sits in the Constitution.

The site reports drift automatically — each member’s live doc_version against the version pinned in the active release — but it cannot judge coherence. That judgment is the reconcile, and it is the human step.

11A. Reconcile (the certification)

For every document the drift report flags, confirm it coheres with the rest: cross-references resolve, shared models agree (the three powers, the veto, the ratification path), shared terms are used the same way throughout. Glance at the un-drifted documents too — the report cannot flag a document that should have changed but did not. A release may certify a constitutional amendment only after that amendment has been ratified under Constitution §17; an unratified DRAFT cannot be pinned as certified. Proceed only once the set coheres and any constitutional content is ratified.

11B. Cut and publish

1. Cut a new lockfile. Copy the current data/releases/charter-YYYY-N.yaml to the next sequence (charter-YYYY-N+1.yaml), rolling the year at the boundary. Do not edit the old file — releases are immutable, the historical record. In the new file, set name and date, set each pinned entry to that member’s current doc_version, add a key for any new governed document, and drop the key for any retired one.
2. Quote every version, in the lockfile and in each document’s doc_version front matter. An unquoted 1.0 parses as the float 1 and breaks the comparison.
3. Move the current-release pointer. The site resolves “the current release” from a single source — the charterRelease parameter in hugo.toml. Set it to the new key. (If any shortcode still hardcodes the release key instead of reading the parameter, repoint it as well: charter-drift and charter-matrix must point at the same release, or one will silently compare against the old snapshot.)
4. Rebuild and verify. The drift box should read clean against the new release. A document missing from the matrix entirely has a doc_id that matches no pinned key — see below.

11C. Rules that bite

• doc_id is a stable primary key, not a display name. The lookup is by doc_id; rename it and the document is silently skipped, and the frozen lockfiles cannot be repaired to match. Leave ids alone even when a title changes (wisenxt-doctrine keeps its id though its title is “Participant Doctrine”). A genuine id migration is a coordinated change at a release boundary.
• Releases are immutable. Correct a mistake by cutting the next release, never by editing a cut one.

Changelog

v1.3 (2026-06-10)

• Added §11 Cutting a Charter Release. The Custodian’s runbook for reconciling the governed set and freezing a certified-coherent snapshot: reconcile, cut the immutable lockfile, move the current-release pointer, rebuild and verify, plus the doc_id-stability and immutability rules. Authority sits in Constitution §17 and §13; this section is the mechanics. Procedure relocated here from the Charter index page.
• Status remains DRAFT pending ratification of the Tech Board amendment.

v1.2 (2026-06-10)

• Added §10 Recruitment and Disbursement, implementing Constitution §16 (The Tech Board Amendment): the two award paths, the ERPNext recruitment workflow, the disbursement procedure (envelope, threshold, payment), and the application-layer access model. It sets no authority — composition, award power, and the threshold remain in Constitution §16; this section is the tunable mechanics.
• Operations Charter references removed. The §9 note and the v1.1 changelog pointed to the abandoned Operations Charter; both now point to Constitution §16 and §10.
• Status remains DRAFT pending ratification of the Tech Board amendment.

v1.1 (2026-06-10)

• Aligned to Constitution v11.0 (the Custodian & Ladder Amendment). The role formerly called the Sovereign is split per v11 into the Custodian (operational root and exclusive Basement access — runs and stewards the infrastructure) and the Tech Board (the purse and contracts). No threshold, cadence, or procedure value was changed.
• Role terminology updated. “Sovereign” as a role → Custodian. Operational on-call alerts (§1B, §2B) and the credential-bootstrap safe (§5B) → Custodian; the Grimoire’s private notes (§8B) → Custodian’s.
• Recurring operational duties confirmed with the Custodian (§9): RAM headroom audit, alert-threshold tuning review, and tabletop DR exercise all require Basement (Zone 0) access, which is Custodian-only, so they remain the Custodian’s. The Tech Board’s operational role is funding and commissioning work (Constitution §16), not execution.
• SOP tuning authority sits with the Custodian (Constitution §13B), who runs this infrastructure — succeeding the Sovereign’s tuning role.
• Abolished body removed. The “CNMCyber Community Board endorsement records” shelf (§8C) is replaced by Gate 2 endorsement records (Developer-space votes plus curation); the Community Board no longer exists (Constitution §11, §15C).
• Document references updated. “WiseNxt Doctrine” → “Participant Doctrine” (§8B, §8C).
• Status set to DRAFT pending ratification of the Custodian/Tech Board responsibility split.

v1.0 (2026-06-02)

• Initial document, extracted from Constitution v9.3 sections §6D, §8C, §8D, §9, §10A–E, §11, §12 during the Charter Split refactor.
• No substantive changes to any threshold, cadence, or procedure. All values carried forward from Constitution v9.3.

END OF DOCUMENT

• Opplet Master Architecture: Enclave Constitution, URL Strategy
• Opplet Operations: Enclave SOP (this document), Hardware Manifest, Software Stack
• Opplet Development Plan: Den Migration