Compare commits
11 commits
c5349e30bd
...
6b8dc23b2c
| Author | SHA1 | Date | |
|---|---|---|---|
| 6b8dc23b2c | |||
| 9ee3a265bb | |||
| 7965a7295f | |||
| b14d8268fe | |||
| e2bec62727 | |||
| c545d08f39 | |||
| d6cc369392 | |||
| 4e7f04d0b7 | |||
| 07d75fabdd | |||
| fdde22d33d | |||
| ce0be1efad |
18 changed files with 736 additions and 1 deletions
23
.repo-classification.yaml
Normal file
23
.repo-classification.yaml
Normal file
|
|
@ -0,0 +1,23 @@
|
|||
repo_classification:
|
||||
standard: Repo Classification Standard
|
||||
version: "1.0"
|
||||
classified_at: "2026-08-14"
|
||||
classified_by: agent
|
||||
category: project
|
||||
domain: financials
|
||||
secondary_domains:
|
||||
- infotech
|
||||
capability_tags:
|
||||
- platform
|
||||
- operations
|
||||
- storage
|
||||
- governance
|
||||
business_stake:
|
||||
- technology
|
||||
- operations
|
||||
business_mechanics:
|
||||
- coordination
|
||||
- operation
|
||||
repo_family: reef
|
||||
ownership_repo: railiance-platform
|
||||
substrate_kind: object-storage
|
||||
17
AGENTS.md
Normal file
17
AGENTS.md
Normal file
|
|
@ -0,0 +1,17 @@
|
|||
# reef-storage — Agent Instructions
|
||||
|
||||
## Repo Identity
|
||||
|
||||
**Purpose:** Provider-delegated S3-compatible storage substrate. Owns
|
||||
non-secret operating attributes. Does not host a rail.
|
||||
|
||||
**Domain:** financials
|
||||
**Repo slug:** reef-storage
|
||||
**Workplan prefix:** `REEF-STORAGE-WP-`
|
||||
|
||||
## Standing rules
|
||||
|
||||
- No rail. Scaleway (or a successor provider) operates S3.
|
||||
- No secrets in this repo. Cite `secret:railiance-platform/<id>`.
|
||||
- Attribute files are the source for `reef:storage/…#key` references.
|
||||
- Do not fold this substrate into `reef-railiance`.
|
||||
31
INTENT.md
Normal file
31
INTENT.md
Normal file
|
|
@ -0,0 +1,31 @@
|
|||
# INTENT
|
||||
|
||||
## Why This Repo Exists
|
||||
|
||||
`reef-storage` is the substrate boundary for **provider-delegated
|
||||
S3-compatible object storage**. The first instance is Scaleway, used for
|
||||
postgres backup and other large, slow-tolerant volumes.
|
||||
|
||||
We consume the S3 capability. We do **not** host a rail and we do **not**
|
||||
operate how Scaleway provides or runs object storage. This reef names the
|
||||
boundary, holds the committed non-secret attributes, and is the place
|
||||
operators look for “where is our object store?”
|
||||
|
||||
It is a different reef from `reef-railiance`. Backup is not bolted onto
|
||||
the home compute substrate.
|
||||
|
||||
## What This Repo Must Protect
|
||||
|
||||
- the split: procure/cost in `resource-control`, operate-by-reference here,
|
||||
secrets in `railiance-platform`, consume in `rapp-*`
|
||||
- no rail invented to satisfy a compute-centric schema
|
||||
- no secret material in git
|
||||
- independent failure domain from Host Europe / `reef-railiance`
|
||||
|
||||
## What This Repo Is Not
|
||||
|
||||
- a Scaleway account manager or billing ledger
|
||||
- a Barman/CNPG operator (`rapp-postgres`)
|
||||
- a secret store
|
||||
- a generic S3 client library
|
||||
- an extension of `reef-railiance`
|
||||
11
README.md
11
README.md
|
|
@ -1,3 +1,12 @@
|
|||
# reef-storage
|
||||
|
||||
Low cost data secure storage. Runtime independent S3 compatible storage facility for backup and large volume storage that can tolerate rather slow access.
|
||||
Low-cost, data-secure, runtime-independent **S3-compatible** storage
|
||||
substrate for backup and large volumes that can tolerate slow access.
|
||||
|
||||
Scaleway provides the S3 capability. This reef does **not** host a rail
|
||||
and does not operate the object store. It names the boundary and holds
|
||||
non-secret attributes.
|
||||
|
||||
- Declaration: `declarations/reef.yaml`
|
||||
- Attributes: `substrate/object-stores/`
|
||||
- Join: `reef:storage/<path>#<key>` from `resource-control`
|
||||
29
SCOPE.md
Normal file
29
SCOPE.md
Normal file
|
|
@ -0,0 +1,29 @@
|
|||
# SCOPE
|
||||
|
||||
## One-liner
|
||||
|
||||
Provider-delegated S3-compatible storage reef: identity and non-secret
|
||||
attributes, not a rail, not a secret store.
|
||||
|
||||
## In Scope
|
||||
|
||||
- Reef declaration (`substrate_kind: object-storage`, no rail)
|
||||
- Committed non-secret object-store attributes (endpoint, region, bucket,
|
||||
prefix, versioning/lifecycle flags, provider project ref)
|
||||
- Runbooks for “how we use the delegated store,” not how Scaleway runs it
|
||||
- Binding notes for consumers (`rapp-postgres` first)
|
||||
|
||||
## Out of Scope
|
||||
|
||||
- Purchase approval, forecasts, settlement (`resource-control`)
|
||||
- Access keys and OpenBao delivery (`railiance-platform`)
|
||||
- WAL archive, backup schedule, restore (`rapp-postgres`)
|
||||
- Hosting Kubernetes or any rail
|
||||
- Implementing S3
|
||||
|
||||
## Getting Oriented
|
||||
|
||||
- Declaration: `declarations/reef.yaml`
|
||||
- Attribute contract: `schemas/object-store-attributes.schema.json`
|
||||
- Attribute files: `substrate/object-stores/`
|
||||
- Join convention: `resource-control/docs/operational-reference-convention.md`
|
||||
19
bindings/consumers.yaml
Normal file
19
bindings/consumers.yaml
Normal file
|
|
@ -0,0 +1,19 @@
|
|||
reef_id: reef-storage
|
||||
consumers_potential:
|
||||
- consumer_id: rapp-postgres
|
||||
resource_id: resource:platform:audit-storage
|
||||
purpose: WAL archive and physical base backups for platform-pg
|
||||
readiness_state: declared
|
||||
evidence:
|
||||
- repo: resource-control
|
||||
path: workplans/RESOURCE-WP-0002-procure-postgres-backup-storage.md
|
||||
consumers_actual:
|
||||
- consumer_id: rapp-postgres
|
||||
resource_id: resource:platform:audit-storage
|
||||
readiness_state: wal-archiving
|
||||
evidence:
|
||||
- repo: resource-control
|
||||
path: docs/evidence/RESOURCE-WP-0002-T05-restore-2026-08-14.md
|
||||
notes:
|
||||
- Actual consumer is recorded only after WAL flows (RESOURCE-WP-0002-T05).
|
||||
- rapp-postgres consumes a reviewed destination; it does not procure the bucket.
|
||||
21
declarations/reef.yaml
Normal file
21
declarations/reef.yaml
Normal file
|
|
@ -0,0 +1,21 @@
|
|||
kind: substrate-reef
|
||||
reef_id: reef-storage
|
||||
repo: reef-storage
|
||||
ownership_repo: railiance-platform
|
||||
substrate_kind: object-storage
|
||||
lifecycle_state: planned
|
||||
criticality: high
|
||||
current_members:
|
||||
- railiance-platform-pg-backup
|
||||
source_documents:
|
||||
- repo: railiance-master
|
||||
path: docs/reef-substrate-model.md
|
||||
- repo: railiance-master
|
||||
path: workplans/RMASTER-WP-0022-establish-reef-storage.md
|
||||
- repo: resource-control
|
||||
path: docs/operational-reference-convention.md
|
||||
compatibility_notes:
|
||||
- "S3-compatible capability is delegated to Scaleway. This reef does not host a rail and does not operate the object store internals."
|
||||
- "Non-secret attributes (endpoint, bucket, region, prefix, lifecycle) live under substrate/object-stores/."
|
||||
- "Credentials live in railiance-platform OpenBao. resource-control cites reef: and secret: references only."
|
||||
- "This substrate is independent of reef-railiance. Backup is not provided on the home compute reef."
|
||||
25
docs/post-purchase-seed.md
Normal file
25
docs/post-purchase-seed.md
Normal file
|
|
@ -0,0 +1,25 @@
|
|||
# Fill reef-storage after Scaleway purchase
|
||||
|
||||
Run only after human financial authority approves
|
||||
`resource-control/docs/evidence/RESOURCE-WP-0002-primary-selection-2026-08-14.md`.
|
||||
|
||||
1. Create a Scaleway project on the Railiance (or GmbH) paying account.
|
||||
2. Create a **private** bucket in `nl-ams`, Standard Multi-AZ.
|
||||
3. Enable versioning. Set lifecycle to the 30-day recovery window.
|
||||
4. Create a key that can only list/get/put/delete on that bucket/prefix.
|
||||
5. Turn on a project cost alert.
|
||||
6. Write the non-secret facts into
|
||||
`substrate/object-stores/platform-audit-storage.yaml`:
|
||||
`endpoint`, `bucket`, `prefix`, `provider_project_ref`, `versioning`,
|
||||
`lifecycle`. Status → `active`.
|
||||
7. Append the bucket/project to `declarations/reef.yaml` `current_members`
|
||||
and `substrate/topology.yaml` `members`. Reef `lifecycle_state` may stay
|
||||
`planned` until the first restore (T05), or become `active` when the
|
||||
bucket exists — prefer `active` only after T05.
|
||||
8. Hand `secret:railiance-platform/backup` to OpenBao (T04).
|
||||
Do not commit the key.
|
||||
9. Tell resource-control: set inventory `decision.status: approved`,
|
||||
`status: ordered`, `ordered_on`, `provider.provider_resource_id`.
|
||||
|
||||
Do not paste endpoint or keys into chat as the source of truth. The YAML
|
||||
file is the source for `reef:storage/…` references.
|
||||
37
docs/put-scaleway-bootstrap.md
Normal file
37
docs/put-scaleway-bootstrap.md
Normal file
|
|
@ -0,0 +1,37 @@
|
|||
# Put the Scaleway bootstrap key (founder, local only)
|
||||
|
||||
Do this on a trusted terminal. **Do not paste ACCESS_KEY or SECRET_KEY
|
||||
into chat, Git, or State Hub.**
|
||||
|
||||
CCR: `railiance-platform/credential-change-requests/CCR-2026-0011-scaleway-object-storage-bootstrap.yaml`
|
||||
Path: `platform/workloads/railiance/scaleway/bootstrap`
|
||||
|
||||
1. In [console.scaleway.com](https://console.scaleway.com) create or reuse
|
||||
the Railiance (or GmbH) project. IAM → API keys: create a key that can
|
||||
create Object Storage buckets in that project. Copy org id and project id
|
||||
from the project dashboard (those two are not as sensitive as the secret
|
||||
key, but still keep them out of git).
|
||||
2. On this host, with a token that can write the `platform` mount:
|
||||
|
||||
```bash
|
||||
bao kv put platform/workloads/railiance/scaleway/bootstrap \
|
||||
ACCESS_KEY='SCWxxxxxxxx' \
|
||||
SECRET_KEY='xxxxxxxx' \
|
||||
DEFAULT_ORGANIZATION_ID='xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' \
|
||||
DEFAULT_PROJECT_ID='xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
|
||||
```
|
||||
|
||||
3. Confirm **without printing values**:
|
||||
|
||||
```bash
|
||||
bao kv metadata get platform/workloads/railiance/scaleway/bootstrap
|
||||
```
|
||||
|
||||
You should see a current version. Then tell the agent “bootstrap is in
|
||||
OpenBao.” They will run `tools/create-platform-audit-bucket.sh`, which
|
||||
creates the private Multi-AZ bucket, 30-day lifecycle, and writes only
|
||||
endpoint/bucket/region into `substrate/object-stores/platform-audit-storage.yaml`.
|
||||
|
||||
The scoped backup key is a **different** path
|
||||
(`platform/workloads/railiance/backup/object-storage`) and is T04.
|
||||
After the scoped key works, delete or lock down this bootstrap key.
|
||||
24
schemas/object-store-attributes.schema.json
Normal file
24
schemas/object-store-attributes.schema.json
Normal file
|
|
@ -0,0 +1,24 @@
|
|||
{
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"$id": "https://railiance.dev/reef-storage/object-store-attributes.schema.json",
|
||||
"title": "Non-secret object-store attributes",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["schema_version", "resource_id", "provider", "status"],
|
||||
"properties": {
|
||||
"schema_version": {"const": "0.1"},
|
||||
"resource_id": {"type": "string", "pattern": "^resource:"},
|
||||
"provider": {"type": "string", "minLength": 1},
|
||||
"product": {"type": ["string", "null"]},
|
||||
"status": {"enum": ["planned", "active", "retired"]},
|
||||
"endpoint": {"type": ["string", "null"]},
|
||||
"region": {"type": ["string", "null"]},
|
||||
"bucket": {"type": ["string", "null"]},
|
||||
"prefix": {"type": ["string", "null"]},
|
||||
"versioning": {"type": ["boolean", "null"]},
|
||||
"lifecycle": {"type": ["string", "null"]},
|
||||
"provider_project_ref": {"type": ["string", "null"]},
|
||||
"intended": {"type": ["object", "null"]},
|
||||
"notes": {"type": "array", "items": {"type": "string"}}
|
||||
}
|
||||
}
|
||||
15
substrate/identity.yaml
Normal file
15
substrate/identity.yaml
Normal file
|
|
@ -0,0 +1,15 @@
|
|||
reef_id: reef-storage
|
||||
display_name: Railiance delegated object storage
|
||||
substrate_kind: object-storage
|
||||
ownership_repo: railiance-platform
|
||||
lifecycle_state: planned
|
||||
criticality: high
|
||||
primary_rail: null
|
||||
current_members:
|
||||
- railiance-platform-pg-backup
|
||||
current_role: Provider-delegated S3-compatible store. Scaleway operates the capability; this reef names the boundary and non-secret attributes.
|
||||
source_of_truth:
|
||||
repo: reef-storage
|
||||
paths:
|
||||
- declarations/reef.yaml
|
||||
- substrate/object-stores/platform-audit-storage.yaml
|
||||
25
substrate/object-stores/platform-audit-storage.yaml
Normal file
25
substrate/object-stores/platform-audit-storage.yaml
Normal file
|
|
@ -0,0 +1,25 @@
|
|||
# Live attributes for RESOURCE-WP-0002. No secrets in this file.
|
||||
schema_version: "0.1"
|
||||
resource_id: resource:platform:audit-storage
|
||||
provider: Scaleway
|
||||
product: standard-multi-az
|
||||
status: active
|
||||
endpoint: https://s3.nl-ams.scw.cloud
|
||||
region: nl-ams
|
||||
bucket: railiance-platform-pg-backup
|
||||
prefix: platform-pg/
|
||||
versioning: true
|
||||
lifecycle: 30-day current and noncurrent expiration
|
||||
provider_project_ref: e1a0dd0e-04b8-4ea9-8b30-d53f8c35c688
|
||||
intended:
|
||||
public_access: disabled
|
||||
versioning: true
|
||||
lifecycle: 30-day recovery window matching data/demand/platform-audit-storage.json
|
||||
encryption_at_rest: provider-managed
|
||||
identity: bucket-and-prefix scoped key only
|
||||
notes:
|
||||
- Destination for rapp-postgres WAL and base backups once procured.
|
||||
- resource-control cites reef:storage/substrate/object-stores/platform-audit-storage.yaml#<key>
|
||||
- "Decision record: resource-control/docs/evidence/RESOURCE-WP-0002-primary-selection-2026-08-14.md"
|
||||
- Created 2026-08-14. Versioning enabled. ACL owner-only FULL_CONTROL.
|
||||
- Cost backstop is a founder email billing alert. Scaleway has no euro hard-stop.
|
||||
21
substrate/topology.yaml
Normal file
21
substrate/topology.yaml
Normal file
|
|
@ -0,0 +1,21 @@
|
|||
reef_id: reef-storage
|
||||
membership_policy: Add a member when a purchased object-store (bucket/project) is live. Members are provider resources, not hosts.
|
||||
members:
|
||||
- member_id: platform-audit-storage
|
||||
role: primary-backup-object-store
|
||||
provider: Scaleway
|
||||
product: standard-multi-az
|
||||
region: nl-ams
|
||||
bucket: railiance-platform-pg-backup
|
||||
resource_id: resource:platform:audit-storage
|
||||
status: live
|
||||
evidence:
|
||||
- repo: resource-control
|
||||
path: docs/evidence/RESOURCE-WP-0002-primary-selection-2026-08-14.md
|
||||
- path: substrate/object-stores/platform-audit-storage.yaml
|
||||
intended_members: []
|
||||
access_surfaces:
|
||||
- type: s3-https
|
||||
note: Endpoint and bucket live in substrate/object-stores/platform-audit-storage.yaml. No rail.
|
||||
unresolved_questions:
|
||||
- Whether Host Europe Backup Storage or Nextcloud is the T06 second copy
|
||||
165
tools/apply-backup-bucket-policy.sh
Executable file
165
tools/apply-backup-bucket-policy.sh
Executable file
|
|
@ -0,0 +1,165 @@
|
|||
#!/usr/bin/env bash
|
||||
# Apply a prefix-scoped bucket policy for the resource-control backup key.
|
||||
# Reads APPLICATION_ID from OpenBao. Never prints secret values.
|
||||
# Uses the bootstrap key to PutBucketPolicy (org owner can always edit policies).
|
||||
set -euo pipefail
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
BAO_ADDR="${BAO_ADDR:-${VAULT_ADDR:-https://bao.coulomb.social}}"
|
||||
REGION="nl-ams"
|
||||
BUCKET="${BUCKET:-railiance-platform-pg-backup}"
|
||||
PREFIX="${PREFIX:-platform-pg/}"
|
||||
|
||||
need() { command -v "$1" >/dev/null || { echo "missing $1" >&2; exit 2; }; }
|
||||
export PATH="${HOME}/.local/bin:${PATH}"
|
||||
need python3
|
||||
need curl
|
||||
|
||||
TOKEN="${OPENBAO_TOKEN:-${VAULT_TOKEN:-}}"
|
||||
if [[ -z "$TOKEN" && -f "$HOME/.vault-token" ]]; then
|
||||
TOKEN="$(cat "$HOME/.vault-token")"
|
||||
fi
|
||||
[[ -n "$TOKEN" ]] || { echo "no OpenBao token" >&2; exit 2; }
|
||||
|
||||
read_kv() {
|
||||
local path="$1"
|
||||
curl -fsS -H "X-Vault-Token: $TOKEN" \
|
||||
"$BAO_ADDR/v1/platform/data/${path#platform/}" \
|
||||
| python3 -c 'import json,sys; d=json.load(sys.stdin); print(json.dumps(d["data"]["data"]))'
|
||||
}
|
||||
|
||||
BACKUP_JSON="$(read_kv workloads/railiance/backup/object-storage)"
|
||||
BOOTSTRAP_JSON="$(read_kv workloads/railiance/scaleway/bootstrap)"
|
||||
|
||||
EXPORTS="$(
|
||||
BACKUP_JSON="$BACKUP_JSON" BOOTSTRAP_JSON="$BOOTSTRAP_JSON" python3 - <<'PY'
|
||||
import json, os, sys, re
|
||||
backup = json.loads(os.environ["BACKUP_JSON"])
|
||||
boot = json.loads(os.environ["BOOTSTRAP_JSON"])
|
||||
app = (backup.get("APPLICATION_ID") or "").strip()
|
||||
if not app or app in {"xxx", "redacted", "changeme"}:
|
||||
sys.stderr.write(
|
||||
"APPLICATION_ID missing on platform/workloads/railiance/backup/object-storage\n"
|
||||
"Founder: add that field in the OpenBao UI (IAM application resource-control UUID).\n"
|
||||
"It is not a secret. Then re-run this script.\n"
|
||||
)
|
||||
sys.exit(6)
|
||||
if not re.fullmatch(r"[0-9a-fA-F-]{36}", app):
|
||||
sys.stderr.write("APPLICATION_ID is not a 36-char UUID\n")
|
||||
sys.exit(7)
|
||||
def g(data, *ks):
|
||||
for k in ks:
|
||||
if data.get(k):
|
||||
return data[k]
|
||||
return ""
|
||||
print("export APPLICATION_ID=" + json.dumps(app))
|
||||
print("export SCW_ACCESS_KEY=" + json.dumps(g(boot, "ACCESS_KEY")))
|
||||
print("export SCW_SECRET_KEY=" + json.dumps(g(boot, "SECRET_KEY")))
|
||||
print("export BACKUP_ACCESS_KEY=" + json.dumps(g(backup, "ACCESS_KEY", "ACCESS_KEY_ID")))
|
||||
print("export BACKUP_SECRET_KEY=" + json.dumps(g(backup, "SECRET_KEY", "ACCESS_SECRET_KEY")))
|
||||
PY
|
||||
)"
|
||||
eval "$EXPORTS"
|
||||
|
||||
export SCW_DEFAULT_REGION="$REGION"
|
||||
export AWS_ACCESS_KEY_ID="$SCW_ACCESS_KEY"
|
||||
export AWS_SECRET_ACCESS_KEY="$SCW_SECRET_KEY"
|
||||
|
||||
VENV="${TMPDIR:-/tmp}/reef-storage-boto3"
|
||||
if [[ ! -x "$VENV/bin/python" ]]; then
|
||||
python3 -m venv "$VENV"
|
||||
"$VENV/bin/pip" -q install boto3
|
||||
fi
|
||||
|
||||
echo "putting prefix-scoped bucket policy on $BUCKET (application id not printed)"
|
||||
BUCKET="$BUCKET" PREFIX="$PREFIX" APPLICATION_ID="$APPLICATION_ID" \
|
||||
"$VENV/bin/python" - <<'PY'
|
||||
import json, os
|
||||
import boto3
|
||||
bucket = os.environ["BUCKET"]
|
||||
prefix = os.environ["PREFIX"]
|
||||
app = os.environ["APPLICATION_ID"]
|
||||
policy = {
|
||||
"Version": "2023-04-17",
|
||||
"Id": "resource-control-scoped-backup",
|
||||
"Statement": [
|
||||
{
|
||||
"Sid": "BackupListPrefix",
|
||||
"Effect": "Allow",
|
||||
"Principal": {"SCW": f"application_id:{app}"},
|
||||
"Action": [
|
||||
"s3:ListBucket",
|
||||
"s3:ListBucketMultipartUploads",
|
||||
"s3:ListBucketVersions",
|
||||
"s3:GetBucketLocation",
|
||||
"s3:GetBucketVersioning",
|
||||
"s3:GetLifecycleConfiguration",
|
||||
],
|
||||
"Resource": [bucket],
|
||||
"Condition": {"StringLike": {"s3:prefix": [prefix, prefix + "*"]}},
|
||||
},
|
||||
{
|
||||
"Sid": "BackupObjectsUnderPrefix",
|
||||
"Effect": "Allow",
|
||||
"Principal": {"SCW": f"application_id:{app}"},
|
||||
"Action": [
|
||||
"s3:GetObject",
|
||||
"s3:GetObjectVersion",
|
||||
"s3:PutObject",
|
||||
"s3:DeleteObject",
|
||||
"s3:DeleteObjectVersion",
|
||||
"s3:AbortMultipartUpload",
|
||||
"s3:ListMultipartUploadParts",
|
||||
],
|
||||
"Resource": [f"{bucket}/{prefix}*"],
|
||||
},
|
||||
],
|
||||
}
|
||||
client = boto3.client(
|
||||
"s3",
|
||||
region_name="nl-ams",
|
||||
endpoint_url="https://s3.nl-ams.scw.cloud",
|
||||
aws_access_key_id=os.environ["AWS_ACCESS_KEY_ID"],
|
||||
aws_secret_access_key=os.environ["AWS_SECRET_ACCESS_KEY"],
|
||||
)
|
||||
client.put_bucket_policy(Bucket=bucket, Policy=json.dumps(policy))
|
||||
print("bucket_policy=put")
|
||||
got = client.get_bucket_policy(Bucket=bucket)
|
||||
body = json.loads(got["Policy"])
|
||||
print("statements", [s.get("Sid") for s in body.get("Statement", [])])
|
||||
PY
|
||||
|
||||
echo "verifying backup key: prefix ok, sibling denied"
|
||||
BUCKET="$BUCKET" PREFIX="$PREFIX" \
|
||||
AWS_ACCESS_KEY_ID="$BACKUP_ACCESS_KEY" AWS_SECRET_ACCESS_KEY="$BACKUP_SECRET_KEY" \
|
||||
"$VENV/bin/python" - <<'PY'
|
||||
import os
|
||||
from botocore.exceptions import ClientError
|
||||
import boto3
|
||||
c = boto3.client(
|
||||
"s3",
|
||||
region_name="nl-ams",
|
||||
endpoint_url="https://s3.nl-ams.scw.cloud",
|
||||
aws_access_key_id=os.environ["AWS_ACCESS_KEY_ID"],
|
||||
aws_secret_access_key=os.environ["AWS_SECRET_ACCESS_KEY"],
|
||||
)
|
||||
bucket = os.environ["BUCKET"]
|
||||
prefix = os.environ["PREFIX"]
|
||||
ok = prefix + ".policy-probe"
|
||||
sib = "other-consumer/.policy-probe"
|
||||
try:
|
||||
c.put_object(Bucket=bucket, Key=ok, Body=b"ok")
|
||||
c.delete_object(Bucket=bucket, Key=ok)
|
||||
print("prefix_write=ok")
|
||||
except ClientError as e:
|
||||
print("prefix_write=fail", e.response.get("Error", {}).get("Code"))
|
||||
try:
|
||||
c.put_object(Bucket=bucket, Key=sib, Body=b"no")
|
||||
print("sibling_write=UNEXPECTED_OK")
|
||||
c.delete_object(Bucket=bucket, Key=sib)
|
||||
except ClientError as e:
|
||||
print("sibling_write=denied", e.response.get("Error", {}).get("Code"))
|
||||
PY
|
||||
|
||||
echo "done"
|
||||
unset SCW_SECRET_KEY SCW_ACCESS_KEY BACKUP_SECRET_KEY BACKUP_ACCESS_KEY AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY APPLICATION_ID
|
||||
139
tools/create-platform-audit-bucket.sh
Executable file
139
tools/create-platform-audit-bucket.sh
Executable file
|
|
@ -0,0 +1,139 @@
|
|||
#!/usr/bin/env bash
|
||||
# Create the planned Scaleway backup bucket. Reads bootstrap creds from
|
||||
# OpenBao. Never prints secret values. Writes only non-secret attributes
|
||||
# into substrate/object-stores/platform-audit-storage.yaml.
|
||||
set -euo pipefail
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
ATTR="$ROOT/substrate/object-stores/platform-audit-storage.yaml"
|
||||
BAO_ADDR="${BAO_ADDR:-${VAULT_ADDR:-https://bao.coulomb.social}}"
|
||||
BOOTSTRAP_PATH="platform/workloads/railiance/scaleway/bootstrap"
|
||||
SCOPED_PATH="platform/workloads/railiance/backup/object-storage"
|
||||
REGION="nl-ams"
|
||||
BUCKET="${BUCKET:-railiance-platform-pg-backup}"
|
||||
PREFIX="${PREFIX:-platform-pg/}"
|
||||
ENDPOINT="https://s3.nl-ams.scw.cloud"
|
||||
|
||||
need() { command -v "$1" >/dev/null || { echo "missing $1" >&2; exit 2; }; }
|
||||
export PATH="${HOME}/.local/bin:${PATH}"
|
||||
need python3
|
||||
need curl
|
||||
need scw
|
||||
|
||||
TOKEN="${OPENBAO_TOKEN:-${VAULT_TOKEN:-}}"
|
||||
if [[ -z "$TOKEN" && -f "$HOME/.vault-token" ]]; then
|
||||
TOKEN="$(cat "$HOME/.vault-token")"
|
||||
fi
|
||||
[[ -n "$TOKEN" ]] || { echo "no OpenBao token" >&2; exit 2; }
|
||||
|
||||
read_kv() {
|
||||
local path="$1"
|
||||
curl -fsS -H "X-Vault-Token: $TOKEN" \
|
||||
"$BAO_ADDR/v1/platform/data/${path#platform/}" \
|
||||
| python3 -c 'import json,sys; d=json.load(sys.stdin); print(json.dumps(d["data"]["data"]))'
|
||||
}
|
||||
|
||||
if ! BOOTSTRAP_JSON="$(read_kv workloads/railiance/scaleway/bootstrap 2>/dev/null)"; then
|
||||
echo "Bootstrap secret missing at $BOOTSTRAP_PATH" >&2
|
||||
echo "Founder: put ACCESS_KEY SECRET_KEY DEFAULT_ORGANIZATION_ID DEFAULT_PROJECT_ID locally (not in chat)." >&2
|
||||
echo "See docs/put-scaleway-bootstrap.md" >&2
|
||||
exit 3
|
||||
fi
|
||||
|
||||
eval "$(BOOTSTRAP_JSON="$BOOTSTRAP_JSON" python3 - <<'PY'
|
||||
import json, os, sys
|
||||
data = json.loads(os.environ["BOOTSTRAP_JSON"])
|
||||
needed = ("ACCESS_KEY", "SECRET_KEY", "DEFAULT_ORGANIZATION_ID", "DEFAULT_PROJECT_ID")
|
||||
missing = [k for k in needed if not (data.get(k) or data.get(k.lower()))]
|
||||
if missing:
|
||||
sys.stderr.write("bootstrap fields missing: " + ",".join(missing) + "\n")
|
||||
sys.exit(4)
|
||||
placeholders = [k for k in needed if (data.get(k) or data.get(k.lower()) or "") in {"xxx", "redacted", "changeme"}]
|
||||
if placeholders:
|
||||
sys.stderr.write("bootstrap still has example placeholders; replace them in the OpenBao UI first\n")
|
||||
sys.exit(5)
|
||||
def g(k):
|
||||
return data.get(k) or data.get(k.lower())
|
||||
print("export SCW_ACCESS_KEY=" + json.dumps(g("ACCESS_KEY")))
|
||||
print("export SCW_SECRET_KEY=" + json.dumps(g("SECRET_KEY")))
|
||||
print("export SCW_DEFAULT_ORGANIZATION_ID=" + json.dumps(g("DEFAULT_ORGANIZATION_ID")))
|
||||
print("export SCW_DEFAULT_PROJECT_ID=" + json.dumps(g("DEFAULT_PROJECT_ID")))
|
||||
PY
|
||||
)"
|
||||
|
||||
export SCW_DEFAULT_REGION="$REGION"
|
||||
|
||||
if scw object bucket get "$BUCKET" region="$REGION" -o json >/dev/null 2>&1; then
|
||||
echo "bucket $BUCKET already exists in $REGION; skipping create"
|
||||
else
|
||||
echo "creating private bucket $BUCKET in $REGION (versioning on)"
|
||||
scw object bucket create "$BUCKET" region="$REGION" acl=private enable-versioning=true -o json
|
||||
fi
|
||||
|
||||
echo "applying 30-day lifecycle (current + noncurrent versions)"
|
||||
VENV="${TMPDIR:-/tmp}/reef-storage-boto3"
|
||||
if [[ ! -x "$VENV/bin/python" ]]; then
|
||||
python3 -m venv "$VENV"
|
||||
"$VENV/bin/pip" -q install boto3
|
||||
fi
|
||||
BUCKET="$BUCKET" AWS_ACCESS_KEY_ID="$SCW_ACCESS_KEY" AWS_SECRET_ACCESS_KEY="$SCW_SECRET_KEY" \
|
||||
"$VENV/bin/python" - <<'PY'
|
||||
import os
|
||||
import boto3
|
||||
client = boto3.client(
|
||||
"s3",
|
||||
region_name="nl-ams",
|
||||
endpoint_url="https://s3.nl-ams.scw.cloud",
|
||||
aws_access_key_id=os.environ["AWS_ACCESS_KEY_ID"],
|
||||
aws_secret_access_key=os.environ["AWS_SECRET_ACCESS_KEY"],
|
||||
)
|
||||
client.put_bucket_lifecycle_configuration(
|
||||
Bucket=os.environ["BUCKET"],
|
||||
LifecycleConfiguration={
|
||||
"Rules": [
|
||||
{
|
||||
"ID": "retain-30-days",
|
||||
"Status": "Enabled",
|
||||
"Filter": {"Prefix": ""},
|
||||
"Expiration": {"Days": 30},
|
||||
"NoncurrentVersionExpiration": {"NoncurrentDays": 30},
|
||||
}
|
||||
]
|
||||
},
|
||||
)
|
||||
print("lifecycle applied")
|
||||
PY
|
||||
|
||||
echo "creating a 20 EUR monthly budget alert if none exists"
|
||||
scw billing budget create consumption-limit=20 enabled=true -o json | python3 -c '
|
||||
import json,sys
|
||||
b=json.load(sys.stdin)
|
||||
print("budget_id", b.get("id") or b.get("budget",{}).get("id"))
|
||||
print("limit", b.get("consumption_limit") or b.get("budget",{}).get("consumption_limit"))
|
||||
' || echo "budget create skipped (permissions or API shape)"
|
||||
|
||||
echo "writing non-secret attributes (no keys)"
|
||||
python3 - <<PY
|
||||
from pathlib import Path
|
||||
p = Path("$ATTR")
|
||||
text = p.read_text()
|
||||
repl = {
|
||||
"status: planned": "status: active",
|
||||
"endpoint: null": "endpoint: $ENDPOINT",
|
||||
"region: nl-ams": "region: $REGION",
|
||||
"bucket: null": "bucket: $BUCKET",
|
||||
"prefix: null": "prefix: $PREFIX",
|
||||
"versioning: null": "versioning: true",
|
||||
"lifecycle: null": "lifecycle: 30-day current and noncurrent expiration",
|
||||
"provider_project_ref: null": "provider_project_ref: ${SCW_DEFAULT_PROJECT_ID}",
|
||||
}
|
||||
for a,b in repl.items():
|
||||
text = text.replace(a, b, 1)
|
||||
p.write_text(text)
|
||||
print(f"updated {p}")
|
||||
print(f"endpoint={ENDPOINT} bucket={BUCKET} region={REGION}")
|
||||
PY
|
||||
|
||||
echo "Cost alert: set a project budget in the Scaleway console if scw billing is unavailable."
|
||||
echo "Scoped backup key is T04: $SCOPED_PATH (not written here)."
|
||||
echo "done. Commit the YAML; do not commit any key."
|
||||
54
tools/ingest-scaleway-tfvars.sh
Executable file
54
tools/ingest-scaleway-tfvars.sh
Executable file
|
|
@ -0,0 +1,54 @@
|
|||
#!/usr/bin/env bash
|
||||
# Founder-only. Reads a local scaleway.auto.tfvars (or equivalent) and
|
||||
# writes the four fields to OpenBao. Never prints values. Never commit
|
||||
# the tfvars file.
|
||||
set -euo pipefail
|
||||
|
||||
TFVARS="${1:-}"
|
||||
[[ -n "$TFVARS" && -f "$TFVARS" ]] || {
|
||||
echo "usage: $0 /path/to/scaleway.auto.tfvars" >&2
|
||||
exit 2
|
||||
}
|
||||
command -v bao >/dev/null || { echo "missing bao" >&2; exit 2; }
|
||||
|
||||
mapfile -t EXPORTS < <(python3 - "$TFVARS" <<'PY'
|
||||
import re, json, sys
|
||||
text = open(sys.argv[1], encoding="utf-8").read()
|
||||
# HCL-ish: key = "value" (ignore comments and the main.tf provider block)
|
||||
found = {}
|
||||
for name, dest in (
|
||||
("access_key", "ACCESS_KEY"),
|
||||
("secret_key", "SECRET_KEY"),
|
||||
("organization_id", "DEFAULT_ORGANIZATION_ID"),
|
||||
("project_id", "DEFAULT_PROJECT_ID"),
|
||||
):
|
||||
m = re.search(rf'(?m)^\s*{name}\s*=\s*"([^"]*)"\s*$', text)
|
||||
if not m or not m.group(1) or m.group(1) in {"xxx", "redacted"}:
|
||||
sys.stderr.write(f"missing or placeholder field: {name}\n")
|
||||
sys.exit(4)
|
||||
found[dest] = m.group(1)
|
||||
for dest, val in found.items():
|
||||
print(f"{dest}={json.dumps(val)}")
|
||||
PY
|
||||
)
|
||||
|
||||
# Write via env to bao without putting values on argv.
|
||||
# bao kv put supports @file; we use a mode-0600 temp dir.
|
||||
umask 077
|
||||
tmp="$(mktemp -d)"
|
||||
trap 'rm -rf "$tmp"' EXIT
|
||||
for line in "${EXPORTS[@]}"; do
|
||||
field="${line%%=*}"
|
||||
python3 -c 'import json,os,sys; open(sys.argv[2],"w").write(json.loads(sys.argv[1]))' \
|
||||
"${line#*=}" "$tmp/$field"
|
||||
done
|
||||
|
||||
bao kv put platform/workloads/railiance/scaleway/bootstrap \
|
||||
ACCESS_KEY=@"$tmp/ACCESS_KEY" \
|
||||
SECRET_KEY=@"$tmp/SECRET_KEY" \
|
||||
DEFAULT_ORGANIZATION_ID=@"$tmp/DEFAULT_ORGANIZATION_ID" \
|
||||
DEFAULT_PROJECT_ID=@"$tmp/DEFAULT_PROJECT_ID"
|
||||
|
||||
echo "wrote four fields to platform/workloads/railiance/scaleway/bootstrap"
|
||||
bao kv metadata get platform/workloads/railiance/scaleway/bootstrap
|
||||
echo "shred or keep your tfvars; do not commit it"
|
||||
44
workplans/REEF-STORAGE-WP-0001-bootstrap.md
Normal file
44
workplans/REEF-STORAGE-WP-0001-bootstrap.md
Normal file
|
|
@ -0,0 +1,44 @@
|
|||
---
|
||||
id: REEF-STORAGE-WP-0001
|
||||
type: workplan
|
||||
title: "Bootstrap reef-storage"
|
||||
domain: financials
|
||||
repo: reef-storage
|
||||
status: active
|
||||
owner: grok
|
||||
topic_slug: railiance
|
||||
created: "2026-08-14"
|
||||
updated: "2026-08-14"
|
||||
related:
|
||||
- RMASTER-WP-0022
|
||||
- RESOURCE-WP-0002
|
||||
---
|
||||
|
||||
# REEF-STORAGE-WP-0001 — bootstrap
|
||||
|
||||
## Goal
|
||||
|
||||
Stand up the planned storage reef: declaration without a rail, attribute
|
||||
file layout, and the contract for non-secret object-store facts.
|
||||
|
||||
## T01 — Declare the reef
|
||||
|
||||
```task
|
||||
id: REEF-STORAGE-WP-0001-T01
|
||||
status: done
|
||||
priority: high
|
||||
```
|
||||
|
||||
`declarations/reef.yaml`: `substrate_kind: object-storage`, no
|
||||
`primary_rail`, no `hosted_rails`, empty `current_members`, planned.
|
||||
|
||||
## T02 — Attribute contract
|
||||
|
||||
```task
|
||||
id: REEF-STORAGE-WP-0001-T02
|
||||
status: done
|
||||
priority: high
|
||||
```
|
||||
|
||||
`schemas/object-store-attributes.schema.json` and a planned
|
||||
`platform-audit-storage.yaml` with null operating fields until purchase.
|
||||
37
workplans/REEF-STORAGE-WP-0002-fill-after-purchase.md
Normal file
37
workplans/REEF-STORAGE-WP-0002-fill-after-purchase.md
Normal file
|
|
@ -0,0 +1,37 @@
|
|||
---
|
||||
id: REEF-STORAGE-WP-0002
|
||||
type: workplan
|
||||
title: "Fill Scaleway attributes after purchase"
|
||||
domain: financials
|
||||
repo: reef-storage
|
||||
status: active
|
||||
owner: grok
|
||||
topic_slug: railiance
|
||||
created: "2026-08-14"
|
||||
updated: "2026-08-14"
|
||||
related:
|
||||
- RESOURCE-WP-0002
|
||||
---
|
||||
|
||||
# REEF-STORAGE-WP-0002 — fill after purchase
|
||||
|
||||
Purchase happened 2026-08-14. Follow `docs/post-purchase-seed.md`.
|
||||
|
||||
## T01 — Write live non-secret attributes
|
||||
|
||||
```task
|
||||
id: REEF-STORAGE-WP-0002-T01
|
||||
status: done
|
||||
priority: high
|
||||
```
|
||||
|
||||
Fill `substrate/object-stores/platform-audit-storage.yaml` endpoint,
|
||||
bucket, prefix, project ref, versioning, lifecycle. Promote topology
|
||||
members. No secrets.
|
||||
|
||||
Done 2026-08-14: bucket `railiance-platform-pg-backup` in `nl-ams` is
|
||||
live (versioning on, 30-day current and noncurrent expiration, owner-only
|
||||
ACL). Attributes, topology member, and declaration `current_members`
|
||||
written. Reef `lifecycle_state` stays `planned` until RESOURCE-WP-0002-T05.
|
||||
Cost backstop is a founder email billing alert. Scaleway has no
|
||||
euro hard-stop; that gap is accepted.
|
||||
Loading…
Add table
Add a link
Reference in a new issue