Compare commits
11 commits
c5349e30bd
...
6b8dc23b2c
| Author | SHA1 | Date | |
|---|---|---|---|
| 6b8dc23b2c | |||
| 9ee3a265bb | |||
| 7965a7295f | |||
| b14d8268fe | |||
| e2bec62727 | |||
| c545d08f39 | |||
| d6cc369392 | |||
| 4e7f04d0b7 | |||
| 07d75fabdd | |||
| fdde22d33d | |||
| ce0be1efad |
18 changed files with 736 additions and 1 deletions
23
.repo-classification.yaml
Normal file
23
.repo-classification.yaml
Normal file
|
|
@ -0,0 +1,23 @@
|
||||||
|
repo_classification:
|
||||||
|
standard: Repo Classification Standard
|
||||||
|
version: "1.0"
|
||||||
|
classified_at: "2026-08-14"
|
||||||
|
classified_by: agent
|
||||||
|
category: project
|
||||||
|
domain: financials
|
||||||
|
secondary_domains:
|
||||||
|
- infotech
|
||||||
|
capability_tags:
|
||||||
|
- platform
|
||||||
|
- operations
|
||||||
|
- storage
|
||||||
|
- governance
|
||||||
|
business_stake:
|
||||||
|
- technology
|
||||||
|
- operations
|
||||||
|
business_mechanics:
|
||||||
|
- coordination
|
||||||
|
- operation
|
||||||
|
repo_family: reef
|
||||||
|
ownership_repo: railiance-platform
|
||||||
|
substrate_kind: object-storage
|
||||||
17
AGENTS.md
Normal file
17
AGENTS.md
Normal file
|
|
@ -0,0 +1,17 @@
|
||||||
|
# reef-storage — Agent Instructions
|
||||||
|
|
||||||
|
## Repo Identity
|
||||||
|
|
||||||
|
**Purpose:** Provider-delegated S3-compatible storage substrate. Owns
|
||||||
|
non-secret operating attributes. Does not host a rail.
|
||||||
|
|
||||||
|
**Domain:** financials
|
||||||
|
**Repo slug:** reef-storage
|
||||||
|
**Workplan prefix:** `REEF-STORAGE-WP-`
|
||||||
|
|
||||||
|
## Standing rules
|
||||||
|
|
||||||
|
- No rail. Scaleway (or a successor provider) operates S3.
|
||||||
|
- No secrets in this repo. Cite `secret:railiance-platform/<id>`.
|
||||||
|
- Attribute files are the source for `reef:storage/…#key` references.
|
||||||
|
- Do not fold this substrate into `reef-railiance`.
|
||||||
31
INTENT.md
Normal file
31
INTENT.md
Normal file
|
|
@ -0,0 +1,31 @@
|
||||||
|
# INTENT
|
||||||
|
|
||||||
|
## Why This Repo Exists
|
||||||
|
|
||||||
|
`reef-storage` is the substrate boundary for **provider-delegated
|
||||||
|
S3-compatible object storage**. The first instance is Scaleway, used for
|
||||||
|
postgres backup and other large, slow-tolerant volumes.
|
||||||
|
|
||||||
|
We consume the S3 capability. We do **not** host a rail and we do **not**
|
||||||
|
operate how Scaleway provides or runs object storage. This reef names the
|
||||||
|
boundary, holds the committed non-secret attributes, and is the place
|
||||||
|
operators look for “where is our object store?”
|
||||||
|
|
||||||
|
It is a different reef from `reef-railiance`. Backup is not bolted onto
|
||||||
|
the home compute substrate.
|
||||||
|
|
||||||
|
## What This Repo Must Protect
|
||||||
|
|
||||||
|
- the split: procure/cost in `resource-control`, operate-by-reference here,
|
||||||
|
secrets in `railiance-platform`, consume in `rapp-*`
|
||||||
|
- no rail invented to satisfy a compute-centric schema
|
||||||
|
- no secret material in git
|
||||||
|
- independent failure domain from Host Europe / `reef-railiance`
|
||||||
|
|
||||||
|
## What This Repo Is Not
|
||||||
|
|
||||||
|
- a Scaleway account manager or billing ledger
|
||||||
|
- a Barman/CNPG operator (`rapp-postgres`)
|
||||||
|
- a secret store
|
||||||
|
- a generic S3 client library
|
||||||
|
- an extension of `reef-railiance`
|
||||||
11
README.md
11
README.md
|
|
@ -1,3 +1,12 @@
|
||||||
# reef-storage
|
# reef-storage
|
||||||
|
|
||||||
Low cost data secure storage. Runtime independent S3 compatible storage facility for backup and large volume storage that can tolerate rather slow access.
|
Low-cost, data-secure, runtime-independent **S3-compatible** storage
|
||||||
|
substrate for backup and large volumes that can tolerate slow access.
|
||||||
|
|
||||||
|
Scaleway provides the S3 capability. This reef does **not** host a rail
|
||||||
|
and does not operate the object store. It names the boundary and holds
|
||||||
|
non-secret attributes.
|
||||||
|
|
||||||
|
- Declaration: `declarations/reef.yaml`
|
||||||
|
- Attributes: `substrate/object-stores/`
|
||||||
|
- Join: `reef:storage/<path>#<key>` from `resource-control`
|
||||||
29
SCOPE.md
Normal file
29
SCOPE.md
Normal file
|
|
@ -0,0 +1,29 @@
|
||||||
|
# SCOPE
|
||||||
|
|
||||||
|
## One-liner
|
||||||
|
|
||||||
|
Provider-delegated S3-compatible storage reef: identity and non-secret
|
||||||
|
attributes, not a rail, not a secret store.
|
||||||
|
|
||||||
|
## In Scope
|
||||||
|
|
||||||
|
- Reef declaration (`substrate_kind: object-storage`, no rail)
|
||||||
|
- Committed non-secret object-store attributes (endpoint, region, bucket,
|
||||||
|
prefix, versioning/lifecycle flags, provider project ref)
|
||||||
|
- Runbooks for “how we use the delegated store,” not how Scaleway runs it
|
||||||
|
- Binding notes for consumers (`rapp-postgres` first)
|
||||||
|
|
||||||
|
## Out of Scope
|
||||||
|
|
||||||
|
- Purchase approval, forecasts, settlement (`resource-control`)
|
||||||
|
- Access keys and OpenBao delivery (`railiance-platform`)
|
||||||
|
- WAL archive, backup schedule, restore (`rapp-postgres`)
|
||||||
|
- Hosting Kubernetes or any rail
|
||||||
|
- Implementing S3
|
||||||
|
|
||||||
|
## Getting Oriented
|
||||||
|
|
||||||
|
- Declaration: `declarations/reef.yaml`
|
||||||
|
- Attribute contract: `schemas/object-store-attributes.schema.json`
|
||||||
|
- Attribute files: `substrate/object-stores/`
|
||||||
|
- Join convention: `resource-control/docs/operational-reference-convention.md`
|
||||||
19
bindings/consumers.yaml
Normal file
19
bindings/consumers.yaml
Normal file
|
|
@ -0,0 +1,19 @@
|
||||||
|
reef_id: reef-storage
|
||||||
|
consumers_potential:
|
||||||
|
- consumer_id: rapp-postgres
|
||||||
|
resource_id: resource:platform:audit-storage
|
||||||
|
purpose: WAL archive and physical base backups for platform-pg
|
||||||
|
readiness_state: declared
|
||||||
|
evidence:
|
||||||
|
- repo: resource-control
|
||||||
|
path: workplans/RESOURCE-WP-0002-procure-postgres-backup-storage.md
|
||||||
|
consumers_actual:
|
||||||
|
- consumer_id: rapp-postgres
|
||||||
|
resource_id: resource:platform:audit-storage
|
||||||
|
readiness_state: wal-archiving
|
||||||
|
evidence:
|
||||||
|
- repo: resource-control
|
||||||
|
path: docs/evidence/RESOURCE-WP-0002-T05-restore-2026-08-14.md
|
||||||
|
notes:
|
||||||
|
- Actual consumer is recorded only after WAL flows (RESOURCE-WP-0002-T05).
|
||||||
|
- rapp-postgres consumes a reviewed destination; it does not procure the bucket.
|
||||||
21
declarations/reef.yaml
Normal file
21
declarations/reef.yaml
Normal file
|
|
@ -0,0 +1,21 @@
|
||||||
|
kind: substrate-reef
|
||||||
|
reef_id: reef-storage
|
||||||
|
repo: reef-storage
|
||||||
|
ownership_repo: railiance-platform
|
||||||
|
substrate_kind: object-storage
|
||||||
|
lifecycle_state: planned
|
||||||
|
criticality: high
|
||||||
|
current_members:
|
||||||
|
- railiance-platform-pg-backup
|
||||||
|
source_documents:
|
||||||
|
- repo: railiance-master
|
||||||
|
path: docs/reef-substrate-model.md
|
||||||
|
- repo: railiance-master
|
||||||
|
path: workplans/RMASTER-WP-0022-establish-reef-storage.md
|
||||||
|
- repo: resource-control
|
||||||
|
path: docs/operational-reference-convention.md
|
||||||
|
compatibility_notes:
|
||||||
|
- "S3-compatible capability is delegated to Scaleway. This reef does not host a rail and does not operate the object store internals."
|
||||||
|
- "Non-secret attributes (endpoint, bucket, region, prefix, lifecycle) live under substrate/object-stores/."
|
||||||
|
- "Credentials live in railiance-platform OpenBao. resource-control cites reef: and secret: references only."
|
||||||
|
- "This substrate is independent of reef-railiance. Backup is not provided on the home compute reef."
|
||||||
25
docs/post-purchase-seed.md
Normal file
25
docs/post-purchase-seed.md
Normal file
|
|
@ -0,0 +1,25 @@
|
||||||
|
# Fill reef-storage after Scaleway purchase
|
||||||
|
|
||||||
|
Run only after human financial authority approves
|
||||||
|
`resource-control/docs/evidence/RESOURCE-WP-0002-primary-selection-2026-08-14.md`.
|
||||||
|
|
||||||
|
1. Create a Scaleway project on the Railiance (or GmbH) paying account.
|
||||||
|
2. Create a **private** bucket in `nl-ams`, Standard Multi-AZ.
|
||||||
|
3. Enable versioning. Set lifecycle to the 30-day recovery window.
|
||||||
|
4. Create a key that can only list/get/put/delete on that bucket/prefix.
|
||||||
|
5. Turn on a project cost alert.
|
||||||
|
6. Write the non-secret facts into
|
||||||
|
`substrate/object-stores/platform-audit-storage.yaml`:
|
||||||
|
`endpoint`, `bucket`, `prefix`, `provider_project_ref`, `versioning`,
|
||||||
|
`lifecycle`. Status → `active`.
|
||||||
|
7. Append the bucket/project to `declarations/reef.yaml` `current_members`
|
||||||
|
and `substrate/topology.yaml` `members`. Reef `lifecycle_state` may stay
|
||||||
|
`planned` until the first restore (T05), or become `active` when the
|
||||||
|
bucket exists — prefer `active` only after T05.
|
||||||
|
8. Hand `secret:railiance-platform/backup` to OpenBao (T04).
|
||||||
|
Do not commit the key.
|
||||||
|
9. Tell resource-control: set inventory `decision.status: approved`,
|
||||||
|
`status: ordered`, `ordered_on`, `provider.provider_resource_id`.
|
||||||
|
|
||||||
|
Do not paste endpoint or keys into chat as the source of truth. The YAML
|
||||||
|
file is the source for `reef:storage/…` references.
|
||||||
37
docs/put-scaleway-bootstrap.md
Normal file
37
docs/put-scaleway-bootstrap.md
Normal file
|
|
@ -0,0 +1,37 @@
|
||||||
|
# Put the Scaleway bootstrap key (founder, local only)
|
||||||
|
|
||||||
|
Do this on a trusted terminal. **Do not paste ACCESS_KEY or SECRET_KEY
|
||||||
|
into chat, Git, or State Hub.**
|
||||||
|
|
||||||
|
CCR: `railiance-platform/credential-change-requests/CCR-2026-0011-scaleway-object-storage-bootstrap.yaml`
|
||||||
|
Path: `platform/workloads/railiance/scaleway/bootstrap`
|
||||||
|
|
||||||
|
1. In [console.scaleway.com](https://console.scaleway.com) create or reuse
|
||||||
|
the Railiance (or GmbH) project. IAM → API keys: create a key that can
|
||||||
|
create Object Storage buckets in that project. Copy org id and project id
|
||||||
|
from the project dashboard (those two are not as sensitive as the secret
|
||||||
|
key, but still keep them out of git).
|
||||||
|
2. On this host, with a token that can write the `platform` mount:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bao kv put platform/workloads/railiance/scaleway/bootstrap \
|
||||||
|
ACCESS_KEY='SCWxxxxxxxx' \
|
||||||
|
SECRET_KEY='xxxxxxxx' \
|
||||||
|
DEFAULT_ORGANIZATION_ID='xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' \
|
||||||
|
DEFAULT_PROJECT_ID='xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
|
||||||
|
```
|
||||||
|
|
||||||
|
3. Confirm **without printing values**:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bao kv metadata get platform/workloads/railiance/scaleway/bootstrap
|
||||||
|
```
|
||||||
|
|
||||||
|
You should see a current version. Then tell the agent “bootstrap is in
|
||||||
|
OpenBao.” They will run `tools/create-platform-audit-bucket.sh`, which
|
||||||
|
creates the private Multi-AZ bucket, 30-day lifecycle, and writes only
|
||||||
|
endpoint/bucket/region into `substrate/object-stores/platform-audit-storage.yaml`.
|
||||||
|
|
||||||
|
The scoped backup key is a **different** path
|
||||||
|
(`platform/workloads/railiance/backup/object-storage`) and is T04.
|
||||||
|
After the scoped key works, delete or lock down this bootstrap key.
|
||||||
24
schemas/object-store-attributes.schema.json
Normal file
24
schemas/object-store-attributes.schema.json
Normal file
|
|
@ -0,0 +1,24 @@
|
||||||
|
{
|
||||||
|
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||||
|
"$id": "https://railiance.dev/reef-storage/object-store-attributes.schema.json",
|
||||||
|
"title": "Non-secret object-store attributes",
|
||||||
|
"type": "object",
|
||||||
|
"additionalProperties": false,
|
||||||
|
"required": ["schema_version", "resource_id", "provider", "status"],
|
||||||
|
"properties": {
|
||||||
|
"schema_version": {"const": "0.1"},
|
||||||
|
"resource_id": {"type": "string", "pattern": "^resource:"},
|
||||||
|
"provider": {"type": "string", "minLength": 1},
|
||||||
|
"product": {"type": ["string", "null"]},
|
||||||
|
"status": {"enum": ["planned", "active", "retired"]},
|
||||||
|
"endpoint": {"type": ["string", "null"]},
|
||||||
|
"region": {"type": ["string", "null"]},
|
||||||
|
"bucket": {"type": ["string", "null"]},
|
||||||
|
"prefix": {"type": ["string", "null"]},
|
||||||
|
"versioning": {"type": ["boolean", "null"]},
|
||||||
|
"lifecycle": {"type": ["string", "null"]},
|
||||||
|
"provider_project_ref": {"type": ["string", "null"]},
|
||||||
|
"intended": {"type": ["object", "null"]},
|
||||||
|
"notes": {"type": "array", "items": {"type": "string"}}
|
||||||
|
}
|
||||||
|
}
|
||||||
15
substrate/identity.yaml
Normal file
15
substrate/identity.yaml
Normal file
|
|
@ -0,0 +1,15 @@
|
||||||
|
reef_id: reef-storage
|
||||||
|
display_name: Railiance delegated object storage
|
||||||
|
substrate_kind: object-storage
|
||||||
|
ownership_repo: railiance-platform
|
||||||
|
lifecycle_state: planned
|
||||||
|
criticality: high
|
||||||
|
primary_rail: null
|
||||||
|
current_members:
|
||||||
|
- railiance-platform-pg-backup
|
||||||
|
current_role: Provider-delegated S3-compatible store. Scaleway operates the capability; this reef names the boundary and non-secret attributes.
|
||||||
|
source_of_truth:
|
||||||
|
repo: reef-storage
|
||||||
|
paths:
|
||||||
|
- declarations/reef.yaml
|
||||||
|
- substrate/object-stores/platform-audit-storage.yaml
|
||||||
25
substrate/object-stores/platform-audit-storage.yaml
Normal file
25
substrate/object-stores/platform-audit-storage.yaml
Normal file
|
|
@ -0,0 +1,25 @@
|
||||||
|
# Live attributes for RESOURCE-WP-0002. No secrets in this file.
|
||||||
|
schema_version: "0.1"
|
||||||
|
resource_id: resource:platform:audit-storage
|
||||||
|
provider: Scaleway
|
||||||
|
product: standard-multi-az
|
||||||
|
status: active
|
||||||
|
endpoint: https://s3.nl-ams.scw.cloud
|
||||||
|
region: nl-ams
|
||||||
|
bucket: railiance-platform-pg-backup
|
||||||
|
prefix: platform-pg/
|
||||||
|
versioning: true
|
||||||
|
lifecycle: 30-day current and noncurrent expiration
|
||||||
|
provider_project_ref: e1a0dd0e-04b8-4ea9-8b30-d53f8c35c688
|
||||||
|
intended:
|
||||||
|
public_access: disabled
|
||||||
|
versioning: true
|
||||||
|
lifecycle: 30-day recovery window matching data/demand/platform-audit-storage.json
|
||||||
|
encryption_at_rest: provider-managed
|
||||||
|
identity: bucket-and-prefix scoped key only
|
||||||
|
notes:
|
||||||
|
- Destination for rapp-postgres WAL and base backups once procured.
|
||||||
|
- resource-control cites reef:storage/substrate/object-stores/platform-audit-storage.yaml#<key>
|
||||||
|
- "Decision record: resource-control/docs/evidence/RESOURCE-WP-0002-primary-selection-2026-08-14.md"
|
||||||
|
- Created 2026-08-14. Versioning enabled. ACL owner-only FULL_CONTROL.
|
||||||
|
- Cost backstop is a founder email billing alert. Scaleway has no euro hard-stop.
|
||||||
21
substrate/topology.yaml
Normal file
21
substrate/topology.yaml
Normal file
|
|
@ -0,0 +1,21 @@
|
||||||
|
reef_id: reef-storage
|
||||||
|
membership_policy: Add a member when a purchased object-store (bucket/project) is live. Members are provider resources, not hosts.
|
||||||
|
members:
|
||||||
|
- member_id: platform-audit-storage
|
||||||
|
role: primary-backup-object-store
|
||||||
|
provider: Scaleway
|
||||||
|
product: standard-multi-az
|
||||||
|
region: nl-ams
|
||||||
|
bucket: railiance-platform-pg-backup
|
||||||
|
resource_id: resource:platform:audit-storage
|
||||||
|
status: live
|
||||||
|
evidence:
|
||||||
|
- repo: resource-control
|
||||||
|
path: docs/evidence/RESOURCE-WP-0002-primary-selection-2026-08-14.md
|
||||||
|
- path: substrate/object-stores/platform-audit-storage.yaml
|
||||||
|
intended_members: []
|
||||||
|
access_surfaces:
|
||||||
|
- type: s3-https
|
||||||
|
note: Endpoint and bucket live in substrate/object-stores/platform-audit-storage.yaml. No rail.
|
||||||
|
unresolved_questions:
|
||||||
|
- Whether Host Europe Backup Storage or Nextcloud is the T06 second copy
|
||||||
165
tools/apply-backup-bucket-policy.sh
Executable file
165
tools/apply-backup-bucket-policy.sh
Executable file
|
|
@ -0,0 +1,165 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# Apply a prefix-scoped bucket policy for the resource-control backup key.
|
||||||
|
# Reads APPLICATION_ID from OpenBao. Never prints secret values.
|
||||||
|
# Uses the bootstrap key to PutBucketPolicy (org owner can always edit policies).
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||||
|
BAO_ADDR="${BAO_ADDR:-${VAULT_ADDR:-https://bao.coulomb.social}}"
|
||||||
|
REGION="nl-ams"
|
||||||
|
BUCKET="${BUCKET:-railiance-platform-pg-backup}"
|
||||||
|
PREFIX="${PREFIX:-platform-pg/}"
|
||||||
|
|
||||||
|
need() { command -v "$1" >/dev/null || { echo "missing $1" >&2; exit 2; }; }
|
||||||
|
export PATH="${HOME}/.local/bin:${PATH}"
|
||||||
|
need python3
|
||||||
|
need curl
|
||||||
|
|
||||||
|
TOKEN="${OPENBAO_TOKEN:-${VAULT_TOKEN:-}}"
|
||||||
|
if [[ -z "$TOKEN" && -f "$HOME/.vault-token" ]]; then
|
||||||
|
TOKEN="$(cat "$HOME/.vault-token")"
|
||||||
|
fi
|
||||||
|
[[ -n "$TOKEN" ]] || { echo "no OpenBao token" >&2; exit 2; }
|
||||||
|
|
||||||
|
read_kv() {
|
||||||
|
local path="$1"
|
||||||
|
curl -fsS -H "X-Vault-Token: $TOKEN" \
|
||||||
|
"$BAO_ADDR/v1/platform/data/${path#platform/}" \
|
||||||
|
| python3 -c 'import json,sys; d=json.load(sys.stdin); print(json.dumps(d["data"]["data"]))'
|
||||||
|
}
|
||||||
|
|
||||||
|
BACKUP_JSON="$(read_kv workloads/railiance/backup/object-storage)"
|
||||||
|
BOOTSTRAP_JSON="$(read_kv workloads/railiance/scaleway/bootstrap)"
|
||||||
|
|
||||||
|
EXPORTS="$(
|
||||||
|
BACKUP_JSON="$BACKUP_JSON" BOOTSTRAP_JSON="$BOOTSTRAP_JSON" python3 - <<'PY'
|
||||||
|
import json, os, sys, re
|
||||||
|
backup = json.loads(os.environ["BACKUP_JSON"])
|
||||||
|
boot = json.loads(os.environ["BOOTSTRAP_JSON"])
|
||||||
|
app = (backup.get("APPLICATION_ID") or "").strip()
|
||||||
|
if not app or app in {"xxx", "redacted", "changeme"}:
|
||||||
|
sys.stderr.write(
|
||||||
|
"APPLICATION_ID missing on platform/workloads/railiance/backup/object-storage\n"
|
||||||
|
"Founder: add that field in the OpenBao UI (IAM application resource-control UUID).\n"
|
||||||
|
"It is not a secret. Then re-run this script.\n"
|
||||||
|
)
|
||||||
|
sys.exit(6)
|
||||||
|
if not re.fullmatch(r"[0-9a-fA-F-]{36}", app):
|
||||||
|
sys.stderr.write("APPLICATION_ID is not a 36-char UUID\n")
|
||||||
|
sys.exit(7)
|
||||||
|
def g(data, *ks):
|
||||||
|
for k in ks:
|
||||||
|
if data.get(k):
|
||||||
|
return data[k]
|
||||||
|
return ""
|
||||||
|
print("export APPLICATION_ID=" + json.dumps(app))
|
||||||
|
print("export SCW_ACCESS_KEY=" + json.dumps(g(boot, "ACCESS_KEY")))
|
||||||
|
print("export SCW_SECRET_KEY=" + json.dumps(g(boot, "SECRET_KEY")))
|
||||||
|
print("export BACKUP_ACCESS_KEY=" + json.dumps(g(backup, "ACCESS_KEY", "ACCESS_KEY_ID")))
|
||||||
|
print("export BACKUP_SECRET_KEY=" + json.dumps(g(backup, "SECRET_KEY", "ACCESS_SECRET_KEY")))
|
||||||
|
PY
|
||||||
|
)"
|
||||||
|
eval "$EXPORTS"
|
||||||
|
|
||||||
|
export SCW_DEFAULT_REGION="$REGION"
|
||||||
|
export AWS_ACCESS_KEY_ID="$SCW_ACCESS_KEY"
|
||||||
|
export AWS_SECRET_ACCESS_KEY="$SCW_SECRET_KEY"
|
||||||
|
|
||||||
|
VENV="${TMPDIR:-/tmp}/reef-storage-boto3"
|
||||||
|
if [[ ! -x "$VENV/bin/python" ]]; then
|
||||||
|
python3 -m venv "$VENV"
|
||||||
|
"$VENV/bin/pip" -q install boto3
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "putting prefix-scoped bucket policy on $BUCKET (application id not printed)"
|
||||||
|
BUCKET="$BUCKET" PREFIX="$PREFIX" APPLICATION_ID="$APPLICATION_ID" \
|
||||||
|
"$VENV/bin/python" - <<'PY'
|
||||||
|
import json, os
|
||||||
|
import boto3
|
||||||
|
bucket = os.environ["BUCKET"]
|
||||||
|
prefix = os.environ["PREFIX"]
|
||||||
|
app = os.environ["APPLICATION_ID"]
|
||||||
|
policy = {
|
||||||
|
"Version": "2023-04-17",
|
||||||
|
"Id": "resource-control-scoped-backup",
|
||||||
|
"Statement": [
|
||||||
|
{
|
||||||
|
"Sid": "BackupListPrefix",
|
||||||
|
"Effect": "Allow",
|
||||||
|
"Principal": {"SCW": f"application_id:{app}"},
|
||||||
|
"Action": [
|
||||||
|
"s3:ListBucket",
|
||||||
|
"s3:ListBucketMultipartUploads",
|
||||||
|
"s3:ListBucketVersions",
|
||||||
|
"s3:GetBucketLocation",
|
||||||
|
"s3:GetBucketVersioning",
|
||||||
|
"s3:GetLifecycleConfiguration",
|
||||||
|
],
|
||||||
|
"Resource": [bucket],
|
||||||
|
"Condition": {"StringLike": {"s3:prefix": [prefix, prefix + "*"]}},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"Sid": "BackupObjectsUnderPrefix",
|
||||||
|
"Effect": "Allow",
|
||||||
|
"Principal": {"SCW": f"application_id:{app}"},
|
||||||
|
"Action": [
|
||||||
|
"s3:GetObject",
|
||||||
|
"s3:GetObjectVersion",
|
||||||
|
"s3:PutObject",
|
||||||
|
"s3:DeleteObject",
|
||||||
|
"s3:DeleteObjectVersion",
|
||||||
|
"s3:AbortMultipartUpload",
|
||||||
|
"s3:ListMultipartUploadParts",
|
||||||
|
],
|
||||||
|
"Resource": [f"{bucket}/{prefix}*"],
|
||||||
|
},
|
||||||
|
],
|
||||||
|
}
|
||||||
|
client = boto3.client(
|
||||||
|
"s3",
|
||||||
|
region_name="nl-ams",
|
||||||
|
endpoint_url="https://s3.nl-ams.scw.cloud",
|
||||||
|
aws_access_key_id=os.environ["AWS_ACCESS_KEY_ID"],
|
||||||
|
aws_secret_access_key=os.environ["AWS_SECRET_ACCESS_KEY"],
|
||||||
|
)
|
||||||
|
client.put_bucket_policy(Bucket=bucket, Policy=json.dumps(policy))
|
||||||
|
print("bucket_policy=put")
|
||||||
|
got = client.get_bucket_policy(Bucket=bucket)
|
||||||
|
body = json.loads(got["Policy"])
|
||||||
|
print("statements", [s.get("Sid") for s in body.get("Statement", [])])
|
||||||
|
PY
|
||||||
|
|
||||||
|
echo "verifying backup key: prefix ok, sibling denied"
|
||||||
|
BUCKET="$BUCKET" PREFIX="$PREFIX" \
|
||||||
|
AWS_ACCESS_KEY_ID="$BACKUP_ACCESS_KEY" AWS_SECRET_ACCESS_KEY="$BACKUP_SECRET_KEY" \
|
||||||
|
"$VENV/bin/python" - <<'PY'
|
||||||
|
import os
|
||||||
|
from botocore.exceptions import ClientError
|
||||||
|
import boto3
|
||||||
|
c = boto3.client(
|
||||||
|
"s3",
|
||||||
|
region_name="nl-ams",
|
||||||
|
endpoint_url="https://s3.nl-ams.scw.cloud",
|
||||||
|
aws_access_key_id=os.environ["AWS_ACCESS_KEY_ID"],
|
||||||
|
aws_secret_access_key=os.environ["AWS_SECRET_ACCESS_KEY"],
|
||||||
|
)
|
||||||
|
bucket = os.environ["BUCKET"]
|
||||||
|
prefix = os.environ["PREFIX"]
|
||||||
|
ok = prefix + ".policy-probe"
|
||||||
|
sib = "other-consumer/.policy-probe"
|
||||||
|
try:
|
||||||
|
c.put_object(Bucket=bucket, Key=ok, Body=b"ok")
|
||||||
|
c.delete_object(Bucket=bucket, Key=ok)
|
||||||
|
print("prefix_write=ok")
|
||||||
|
except ClientError as e:
|
||||||
|
print("prefix_write=fail", e.response.get("Error", {}).get("Code"))
|
||||||
|
try:
|
||||||
|
c.put_object(Bucket=bucket, Key=sib, Body=b"no")
|
||||||
|
print("sibling_write=UNEXPECTED_OK")
|
||||||
|
c.delete_object(Bucket=bucket, Key=sib)
|
||||||
|
except ClientError as e:
|
||||||
|
print("sibling_write=denied", e.response.get("Error", {}).get("Code"))
|
||||||
|
PY
|
||||||
|
|
||||||
|
echo "done"
|
||||||
|
unset SCW_SECRET_KEY SCW_ACCESS_KEY BACKUP_SECRET_KEY BACKUP_ACCESS_KEY AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY APPLICATION_ID
|
||||||
139
tools/create-platform-audit-bucket.sh
Executable file
139
tools/create-platform-audit-bucket.sh
Executable file
|
|
@ -0,0 +1,139 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# Create the planned Scaleway backup bucket. Reads bootstrap creds from
|
||||||
|
# OpenBao. Never prints secret values. Writes only non-secret attributes
|
||||||
|
# into substrate/object-stores/platform-audit-storage.yaml.
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||||
|
ATTR="$ROOT/substrate/object-stores/platform-audit-storage.yaml"
|
||||||
|
BAO_ADDR="${BAO_ADDR:-${VAULT_ADDR:-https://bao.coulomb.social}}"
|
||||||
|
BOOTSTRAP_PATH="platform/workloads/railiance/scaleway/bootstrap"
|
||||||
|
SCOPED_PATH="platform/workloads/railiance/backup/object-storage"
|
||||||
|
REGION="nl-ams"
|
||||||
|
BUCKET="${BUCKET:-railiance-platform-pg-backup}"
|
||||||
|
PREFIX="${PREFIX:-platform-pg/}"
|
||||||
|
ENDPOINT="https://s3.nl-ams.scw.cloud"
|
||||||
|
|
||||||
|
need() { command -v "$1" >/dev/null || { echo "missing $1" >&2; exit 2; }; }
|
||||||
|
export PATH="${HOME}/.local/bin:${PATH}"
|
||||||
|
need python3
|
||||||
|
need curl
|
||||||
|
need scw
|
||||||
|
|
||||||
|
TOKEN="${OPENBAO_TOKEN:-${VAULT_TOKEN:-}}"
|
||||||
|
if [[ -z "$TOKEN" && -f "$HOME/.vault-token" ]]; then
|
||||||
|
TOKEN="$(cat "$HOME/.vault-token")"
|
||||||
|
fi
|
||||||
|
[[ -n "$TOKEN" ]] || { echo "no OpenBao token" >&2; exit 2; }
|
||||||
|
|
||||||
|
read_kv() {
|
||||||
|
local path="$1"
|
||||||
|
curl -fsS -H "X-Vault-Token: $TOKEN" \
|
||||||
|
"$BAO_ADDR/v1/platform/data/${path#platform/}" \
|
||||||
|
| python3 -c 'import json,sys; d=json.load(sys.stdin); print(json.dumps(d["data"]["data"]))'
|
||||||
|
}
|
||||||
|
|
||||||
|
if ! BOOTSTRAP_JSON="$(read_kv workloads/railiance/scaleway/bootstrap 2>/dev/null)"; then
|
||||||
|
echo "Bootstrap secret missing at $BOOTSTRAP_PATH" >&2
|
||||||
|
echo "Founder: put ACCESS_KEY SECRET_KEY DEFAULT_ORGANIZATION_ID DEFAULT_PROJECT_ID locally (not in chat)." >&2
|
||||||
|
echo "See docs/put-scaleway-bootstrap.md" >&2
|
||||||
|
exit 3
|
||||||
|
fi
|
||||||
|
|
||||||
|
eval "$(BOOTSTRAP_JSON="$BOOTSTRAP_JSON" python3 - <<'PY'
|
||||||
|
import json, os, sys
|
||||||
|
data = json.loads(os.environ["BOOTSTRAP_JSON"])
|
||||||
|
needed = ("ACCESS_KEY", "SECRET_KEY", "DEFAULT_ORGANIZATION_ID", "DEFAULT_PROJECT_ID")
|
||||||
|
missing = [k for k in needed if not (data.get(k) or data.get(k.lower()))]
|
||||||
|
if missing:
|
||||||
|
sys.stderr.write("bootstrap fields missing: " + ",".join(missing) + "\n")
|
||||||
|
sys.exit(4)
|
||||||
|
placeholders = [k for k in needed if (data.get(k) or data.get(k.lower()) or "") in {"xxx", "redacted", "changeme"}]
|
||||||
|
if placeholders:
|
||||||
|
sys.stderr.write("bootstrap still has example placeholders; replace them in the OpenBao UI first\n")
|
||||||
|
sys.exit(5)
|
||||||
|
def g(k):
|
||||||
|
return data.get(k) or data.get(k.lower())
|
||||||
|
print("export SCW_ACCESS_KEY=" + json.dumps(g("ACCESS_KEY")))
|
||||||
|
print("export SCW_SECRET_KEY=" + json.dumps(g("SECRET_KEY")))
|
||||||
|
print("export SCW_DEFAULT_ORGANIZATION_ID=" + json.dumps(g("DEFAULT_ORGANIZATION_ID")))
|
||||||
|
print("export SCW_DEFAULT_PROJECT_ID=" + json.dumps(g("DEFAULT_PROJECT_ID")))
|
||||||
|
PY
|
||||||
|
)"
|
||||||
|
|
||||||
|
export SCW_DEFAULT_REGION="$REGION"
|
||||||
|
|
||||||
|
if scw object bucket get "$BUCKET" region="$REGION" -o json >/dev/null 2>&1; then
|
||||||
|
echo "bucket $BUCKET already exists in $REGION; skipping create"
|
||||||
|
else
|
||||||
|
echo "creating private bucket $BUCKET in $REGION (versioning on)"
|
||||||
|
scw object bucket create "$BUCKET" region="$REGION" acl=private enable-versioning=true -o json
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "applying 30-day lifecycle (current + noncurrent versions)"
|
||||||
|
VENV="${TMPDIR:-/tmp}/reef-storage-boto3"
|
||||||
|
if [[ ! -x "$VENV/bin/python" ]]; then
|
||||||
|
python3 -m venv "$VENV"
|
||||||
|
"$VENV/bin/pip" -q install boto3
|
||||||
|
fi
|
||||||
|
BUCKET="$BUCKET" AWS_ACCESS_KEY_ID="$SCW_ACCESS_KEY" AWS_SECRET_ACCESS_KEY="$SCW_SECRET_KEY" \
|
||||||
|
"$VENV/bin/python" - <<'PY'
|
||||||
|
import os
|
||||||
|
import boto3
|
||||||
|
client = boto3.client(
|
||||||
|
"s3",
|
||||||
|
region_name="nl-ams",
|
||||||
|
endpoint_url="https://s3.nl-ams.scw.cloud",
|
||||||
|
aws_access_key_id=os.environ["AWS_ACCESS_KEY_ID"],
|
||||||
|
aws_secret_access_key=os.environ["AWS_SECRET_ACCESS_KEY"],
|
||||||
|
)
|
||||||
|
client.put_bucket_lifecycle_configuration(
|
||||||
|
Bucket=os.environ["BUCKET"],
|
||||||
|
LifecycleConfiguration={
|
||||||
|
"Rules": [
|
||||||
|
{
|
||||||
|
"ID": "retain-30-days",
|
||||||
|
"Status": "Enabled",
|
||||||
|
"Filter": {"Prefix": ""},
|
||||||
|
"Expiration": {"Days": 30},
|
||||||
|
"NoncurrentVersionExpiration": {"NoncurrentDays": 30},
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
)
|
||||||
|
print("lifecycle applied")
|
||||||
|
PY
|
||||||
|
|
||||||
|
echo "creating a 20 EUR monthly budget alert if none exists"
|
||||||
|
scw billing budget create consumption-limit=20 enabled=true -o json | python3 -c '
|
||||||
|
import json,sys
|
||||||
|
b=json.load(sys.stdin)
|
||||||
|
print("budget_id", b.get("id") or b.get("budget",{}).get("id"))
|
||||||
|
print("limit", b.get("consumption_limit") or b.get("budget",{}).get("consumption_limit"))
|
||||||
|
' || echo "budget create skipped (permissions or API shape)"
|
||||||
|
|
||||||
|
echo "writing non-secret attributes (no keys)"
|
||||||
|
python3 - <<PY
|
||||||
|
from pathlib import Path
|
||||||
|
p = Path("$ATTR")
|
||||||
|
text = p.read_text()
|
||||||
|
repl = {
|
||||||
|
"status: planned": "status: active",
|
||||||
|
"endpoint: null": "endpoint: $ENDPOINT",
|
||||||
|
"region: nl-ams": "region: $REGION",
|
||||||
|
"bucket: null": "bucket: $BUCKET",
|
||||||
|
"prefix: null": "prefix: $PREFIX",
|
||||||
|
"versioning: null": "versioning: true",
|
||||||
|
"lifecycle: null": "lifecycle: 30-day current and noncurrent expiration",
|
||||||
|
"provider_project_ref: null": "provider_project_ref: ${SCW_DEFAULT_PROJECT_ID}",
|
||||||
|
}
|
||||||
|
for a,b in repl.items():
|
||||||
|
text = text.replace(a, b, 1)
|
||||||
|
p.write_text(text)
|
||||||
|
print(f"updated {p}")
|
||||||
|
print(f"endpoint={ENDPOINT} bucket={BUCKET} region={REGION}")
|
||||||
|
PY
|
||||||
|
|
||||||
|
echo "Cost alert: set a project budget in the Scaleway console if scw billing is unavailable."
|
||||||
|
echo "Scoped backup key is T04: $SCOPED_PATH (not written here)."
|
||||||
|
echo "done. Commit the YAML; do not commit any key."
|
||||||
54
tools/ingest-scaleway-tfvars.sh
Executable file
54
tools/ingest-scaleway-tfvars.sh
Executable file
|
|
@ -0,0 +1,54 @@
|
||||||
|
#!/usr/bin/env bash
|
||||||
|
# Founder-only. Reads a local scaleway.auto.tfvars (or equivalent) and
|
||||||
|
# writes the four fields to OpenBao. Never prints values. Never commit
|
||||||
|
# the tfvars file.
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
TFVARS="${1:-}"
|
||||||
|
[[ -n "$TFVARS" && -f "$TFVARS" ]] || {
|
||||||
|
echo "usage: $0 /path/to/scaleway.auto.tfvars" >&2
|
||||||
|
exit 2
|
||||||
|
}
|
||||||
|
command -v bao >/dev/null || { echo "missing bao" >&2; exit 2; }
|
||||||
|
|
||||||
|
mapfile -t EXPORTS < <(python3 - "$TFVARS" <<'PY'
|
||||||
|
import re, json, sys
|
||||||
|
text = open(sys.argv[1], encoding="utf-8").read()
|
||||||
|
# HCL-ish: key = "value" (ignore comments and the main.tf provider block)
|
||||||
|
found = {}
|
||||||
|
for name, dest in (
|
||||||
|
("access_key", "ACCESS_KEY"),
|
||||||
|
("secret_key", "SECRET_KEY"),
|
||||||
|
("organization_id", "DEFAULT_ORGANIZATION_ID"),
|
||||||
|
("project_id", "DEFAULT_PROJECT_ID"),
|
||||||
|
):
|
||||||
|
m = re.search(rf'(?m)^\s*{name}\s*=\s*"([^"]*)"\s*$', text)
|
||||||
|
if not m or not m.group(1) or m.group(1) in {"xxx", "redacted"}:
|
||||||
|
sys.stderr.write(f"missing or placeholder field: {name}\n")
|
||||||
|
sys.exit(4)
|
||||||
|
found[dest] = m.group(1)
|
||||||
|
for dest, val in found.items():
|
||||||
|
print(f"{dest}={json.dumps(val)}")
|
||||||
|
PY
|
||||||
|
)
|
||||||
|
|
||||||
|
# Write via env to bao without putting values on argv.
|
||||||
|
# bao kv put supports @file; we use a mode-0600 temp dir.
|
||||||
|
umask 077
|
||||||
|
tmp="$(mktemp -d)"
|
||||||
|
trap 'rm -rf "$tmp"' EXIT
|
||||||
|
for line in "${EXPORTS[@]}"; do
|
||||||
|
field="${line%%=*}"
|
||||||
|
python3 -c 'import json,os,sys; open(sys.argv[2],"w").write(json.loads(sys.argv[1]))' \
|
||||||
|
"${line#*=}" "$tmp/$field"
|
||||||
|
done
|
||||||
|
|
||||||
|
bao kv put platform/workloads/railiance/scaleway/bootstrap \
|
||||||
|
ACCESS_KEY=@"$tmp/ACCESS_KEY" \
|
||||||
|
SECRET_KEY=@"$tmp/SECRET_KEY" \
|
||||||
|
DEFAULT_ORGANIZATION_ID=@"$tmp/DEFAULT_ORGANIZATION_ID" \
|
||||||
|
DEFAULT_PROJECT_ID=@"$tmp/DEFAULT_PROJECT_ID"
|
||||||
|
|
||||||
|
echo "wrote four fields to platform/workloads/railiance/scaleway/bootstrap"
|
||||||
|
bao kv metadata get platform/workloads/railiance/scaleway/bootstrap
|
||||||
|
echo "shred or keep your tfvars; do not commit it"
|
||||||
44
workplans/REEF-STORAGE-WP-0001-bootstrap.md
Normal file
44
workplans/REEF-STORAGE-WP-0001-bootstrap.md
Normal file
|
|
@ -0,0 +1,44 @@
|
||||||
|
---
|
||||||
|
id: REEF-STORAGE-WP-0001
|
||||||
|
type: workplan
|
||||||
|
title: "Bootstrap reef-storage"
|
||||||
|
domain: financials
|
||||||
|
repo: reef-storage
|
||||||
|
status: active
|
||||||
|
owner: grok
|
||||||
|
topic_slug: railiance
|
||||||
|
created: "2026-08-14"
|
||||||
|
updated: "2026-08-14"
|
||||||
|
related:
|
||||||
|
- RMASTER-WP-0022
|
||||||
|
- RESOURCE-WP-0002
|
||||||
|
---
|
||||||
|
|
||||||
|
# REEF-STORAGE-WP-0001 — bootstrap
|
||||||
|
|
||||||
|
## Goal
|
||||||
|
|
||||||
|
Stand up the planned storage reef: declaration without a rail, attribute
|
||||||
|
file layout, and the contract for non-secret object-store facts.
|
||||||
|
|
||||||
|
## T01 — Declare the reef
|
||||||
|
|
||||||
|
```task
|
||||||
|
id: REEF-STORAGE-WP-0001-T01
|
||||||
|
status: done
|
||||||
|
priority: high
|
||||||
|
```
|
||||||
|
|
||||||
|
`declarations/reef.yaml`: `substrate_kind: object-storage`, no
|
||||||
|
`primary_rail`, no `hosted_rails`, empty `current_members`, planned.
|
||||||
|
|
||||||
|
## T02 — Attribute contract
|
||||||
|
|
||||||
|
```task
|
||||||
|
id: REEF-STORAGE-WP-0001-T02
|
||||||
|
status: done
|
||||||
|
priority: high
|
||||||
|
```
|
||||||
|
|
||||||
|
`schemas/object-store-attributes.schema.json` and a planned
|
||||||
|
`platform-audit-storage.yaml` with null operating fields until purchase.
|
||||||
37
workplans/REEF-STORAGE-WP-0002-fill-after-purchase.md
Normal file
37
workplans/REEF-STORAGE-WP-0002-fill-after-purchase.md
Normal file
|
|
@ -0,0 +1,37 @@
|
||||||
|
---
|
||||||
|
id: REEF-STORAGE-WP-0002
|
||||||
|
type: workplan
|
||||||
|
title: "Fill Scaleway attributes after purchase"
|
||||||
|
domain: financials
|
||||||
|
repo: reef-storage
|
||||||
|
status: active
|
||||||
|
owner: grok
|
||||||
|
topic_slug: railiance
|
||||||
|
created: "2026-08-14"
|
||||||
|
updated: "2026-08-14"
|
||||||
|
related:
|
||||||
|
- RESOURCE-WP-0002
|
||||||
|
---
|
||||||
|
|
||||||
|
# REEF-STORAGE-WP-0002 — fill after purchase
|
||||||
|
|
||||||
|
Purchase happened 2026-08-14. Follow `docs/post-purchase-seed.md`.
|
||||||
|
|
||||||
|
## T01 — Write live non-secret attributes
|
||||||
|
|
||||||
|
```task
|
||||||
|
id: REEF-STORAGE-WP-0002-T01
|
||||||
|
status: done
|
||||||
|
priority: high
|
||||||
|
```
|
||||||
|
|
||||||
|
Fill `substrate/object-stores/platform-audit-storage.yaml` endpoint,
|
||||||
|
bucket, prefix, project ref, versioning, lifecycle. Promote topology
|
||||||
|
members. No secrets.
|
||||||
|
|
||||||
|
Done 2026-08-14: bucket `railiance-platform-pg-backup` in `nl-ams` is
|
||||||
|
live (versioning on, 30-day current and noncurrent expiration, owner-only
|
||||||
|
ACL). Attributes, topology member, and declaration `current_members`
|
||||||
|
written. Reef `lifecycle_state` stays `planned` until RESOURCE-WP-0002-T05.
|
||||||
|
Cost backstop is a founder email billing alert. Scaleway has no
|
||||||
|
euro hard-stop; that gap is accepted.
|
||||||
Loading…
Add table
Add a link
Reference in a new issue