Compare commits
No commits in common. "6b8dc23b2c6adc3c5880639c159008456d06aa86" and "c5349e30bdd0912474bde63f78a3d44ce72b2b44" have entirely different histories.
6b8dc23b2c
...
c5349e30bd
18 changed files with 1 additions and 736 deletions
|
|
@ -1,23 +0,0 @@
|
||||||
repo_classification:
|
|
||||||
standard: Repo Classification Standard
|
|
||||||
version: "1.0"
|
|
||||||
classified_at: "2026-08-14"
|
|
||||||
classified_by: agent
|
|
||||||
category: project
|
|
||||||
domain: financials
|
|
||||||
secondary_domains:
|
|
||||||
- infotech
|
|
||||||
capability_tags:
|
|
||||||
- platform
|
|
||||||
- operations
|
|
||||||
- storage
|
|
||||||
- governance
|
|
||||||
business_stake:
|
|
||||||
- technology
|
|
||||||
- operations
|
|
||||||
business_mechanics:
|
|
||||||
- coordination
|
|
||||||
- operation
|
|
||||||
repo_family: reef
|
|
||||||
ownership_repo: railiance-platform
|
|
||||||
substrate_kind: object-storage
|
|
||||||
17
AGENTS.md
17
AGENTS.md
|
|
@ -1,17 +0,0 @@
|
||||||
# reef-storage — Agent Instructions
|
|
||||||
|
|
||||||
## Repo Identity
|
|
||||||
|
|
||||||
**Purpose:** Provider-delegated S3-compatible storage substrate. Owns
|
|
||||||
non-secret operating attributes. Does not host a rail.
|
|
||||||
|
|
||||||
**Domain:** financials
|
|
||||||
**Repo slug:** reef-storage
|
|
||||||
**Workplan prefix:** `REEF-STORAGE-WP-`
|
|
||||||
|
|
||||||
## Standing rules
|
|
||||||
|
|
||||||
- No rail. Scaleway (or a successor provider) operates S3.
|
|
||||||
- No secrets in this repo. Cite `secret:railiance-platform/<id>`.
|
|
||||||
- Attribute files are the source for `reef:storage/…#key` references.
|
|
||||||
- Do not fold this substrate into `reef-railiance`.
|
|
||||||
31
INTENT.md
31
INTENT.md
|
|
@ -1,31 +0,0 @@
|
||||||
# INTENT
|
|
||||||
|
|
||||||
## Why This Repo Exists
|
|
||||||
|
|
||||||
`reef-storage` is the substrate boundary for **provider-delegated
|
|
||||||
S3-compatible object storage**. The first instance is Scaleway, used for
|
|
||||||
postgres backup and other large, slow-tolerant volumes.
|
|
||||||
|
|
||||||
We consume the S3 capability. We do **not** host a rail and we do **not**
|
|
||||||
operate how Scaleway provides or runs object storage. This reef names the
|
|
||||||
boundary, holds the committed non-secret attributes, and is the place
|
|
||||||
operators look for “where is our object store?”
|
|
||||||
|
|
||||||
It is a different reef from `reef-railiance`. Backup is not bolted onto
|
|
||||||
the home compute substrate.
|
|
||||||
|
|
||||||
## What This Repo Must Protect
|
|
||||||
|
|
||||||
- the split: procure/cost in `resource-control`, operate-by-reference here,
|
|
||||||
secrets in `railiance-platform`, consume in `rapp-*`
|
|
||||||
- no rail invented to satisfy a compute-centric schema
|
|
||||||
- no secret material in git
|
|
||||||
- independent failure domain from Host Europe / `reef-railiance`
|
|
||||||
|
|
||||||
## What This Repo Is Not
|
|
||||||
|
|
||||||
- a Scaleway account manager or billing ledger
|
|
||||||
- a Barman/CNPG operator (`rapp-postgres`)
|
|
||||||
- a secret store
|
|
||||||
- a generic S3 client library
|
|
||||||
- an extension of `reef-railiance`
|
|
||||||
11
README.md
11
README.md
|
|
@ -1,12 +1,3 @@
|
||||||
# reef-storage
|
# reef-storage
|
||||||
|
|
||||||
Low-cost, data-secure, runtime-independent **S3-compatible** storage
|
Low cost data secure storage. Runtime independent S3 compatible storage facility for backup and large volume storage that can tolerate rather slow access.
|
||||||
substrate for backup and large volumes that can tolerate slow access.
|
|
||||||
|
|
||||||
Scaleway provides the S3 capability. This reef does **not** host a rail
|
|
||||||
and does not operate the object store. It names the boundary and holds
|
|
||||||
non-secret attributes.
|
|
||||||
|
|
||||||
- Declaration: `declarations/reef.yaml`
|
|
||||||
- Attributes: `substrate/object-stores/`
|
|
||||||
- Join: `reef:storage/<path>#<key>` from `resource-control`
|
|
||||||
29
SCOPE.md
29
SCOPE.md
|
|
@ -1,29 +0,0 @@
|
||||||
# SCOPE
|
|
||||||
|
|
||||||
## One-liner
|
|
||||||
|
|
||||||
Provider-delegated S3-compatible storage reef: identity and non-secret
|
|
||||||
attributes, not a rail, not a secret store.
|
|
||||||
|
|
||||||
## In Scope
|
|
||||||
|
|
||||||
- Reef declaration (`substrate_kind: object-storage`, no rail)
|
|
||||||
- Committed non-secret object-store attributes (endpoint, region, bucket,
|
|
||||||
prefix, versioning/lifecycle flags, provider project ref)
|
|
||||||
- Runbooks for “how we use the delegated store,” not how Scaleway runs it
|
|
||||||
- Binding notes for consumers (`rapp-postgres` first)
|
|
||||||
|
|
||||||
## Out of Scope
|
|
||||||
|
|
||||||
- Purchase approval, forecasts, settlement (`resource-control`)
|
|
||||||
- Access keys and OpenBao delivery (`railiance-platform`)
|
|
||||||
- WAL archive, backup schedule, restore (`rapp-postgres`)
|
|
||||||
- Hosting Kubernetes or any rail
|
|
||||||
- Implementing S3
|
|
||||||
|
|
||||||
## Getting Oriented
|
|
||||||
|
|
||||||
- Declaration: `declarations/reef.yaml`
|
|
||||||
- Attribute contract: `schemas/object-store-attributes.schema.json`
|
|
||||||
- Attribute files: `substrate/object-stores/`
|
|
||||||
- Join convention: `resource-control/docs/operational-reference-convention.md`
|
|
||||||
|
|
@ -1,19 +0,0 @@
|
||||||
reef_id: reef-storage
|
|
||||||
consumers_potential:
|
|
||||||
- consumer_id: rapp-postgres
|
|
||||||
resource_id: resource:platform:audit-storage
|
|
||||||
purpose: WAL archive and physical base backups for platform-pg
|
|
||||||
readiness_state: declared
|
|
||||||
evidence:
|
|
||||||
- repo: resource-control
|
|
||||||
path: workplans/RESOURCE-WP-0002-procure-postgres-backup-storage.md
|
|
||||||
consumers_actual:
|
|
||||||
- consumer_id: rapp-postgres
|
|
||||||
resource_id: resource:platform:audit-storage
|
|
||||||
readiness_state: wal-archiving
|
|
||||||
evidence:
|
|
||||||
- repo: resource-control
|
|
||||||
path: docs/evidence/RESOURCE-WP-0002-T05-restore-2026-08-14.md
|
|
||||||
notes:
|
|
||||||
- Actual consumer is recorded only after WAL flows (RESOURCE-WP-0002-T05).
|
|
||||||
- rapp-postgres consumes a reviewed destination; it does not procure the bucket.
|
|
||||||
|
|
@ -1,21 +0,0 @@
|
||||||
kind: substrate-reef
|
|
||||||
reef_id: reef-storage
|
|
||||||
repo: reef-storage
|
|
||||||
ownership_repo: railiance-platform
|
|
||||||
substrate_kind: object-storage
|
|
||||||
lifecycle_state: planned
|
|
||||||
criticality: high
|
|
||||||
current_members:
|
|
||||||
- railiance-platform-pg-backup
|
|
||||||
source_documents:
|
|
||||||
- repo: railiance-master
|
|
||||||
path: docs/reef-substrate-model.md
|
|
||||||
- repo: railiance-master
|
|
||||||
path: workplans/RMASTER-WP-0022-establish-reef-storage.md
|
|
||||||
- repo: resource-control
|
|
||||||
path: docs/operational-reference-convention.md
|
|
||||||
compatibility_notes:
|
|
||||||
- "S3-compatible capability is delegated to Scaleway. This reef does not host a rail and does not operate the object store internals."
|
|
||||||
- "Non-secret attributes (endpoint, bucket, region, prefix, lifecycle) live under substrate/object-stores/."
|
|
||||||
- "Credentials live in railiance-platform OpenBao. resource-control cites reef: and secret: references only."
|
|
||||||
- "This substrate is independent of reef-railiance. Backup is not provided on the home compute reef."
|
|
||||||
|
|
@ -1,25 +0,0 @@
|
||||||
# Fill reef-storage after Scaleway purchase
|
|
||||||
|
|
||||||
Run only after human financial authority approves
|
|
||||||
`resource-control/docs/evidence/RESOURCE-WP-0002-primary-selection-2026-08-14.md`.
|
|
||||||
|
|
||||||
1. Create a Scaleway project on the Railiance (or GmbH) paying account.
|
|
||||||
2. Create a **private** bucket in `nl-ams`, Standard Multi-AZ.
|
|
||||||
3. Enable versioning. Set lifecycle to the 30-day recovery window.
|
|
||||||
4. Create a key that can only list/get/put/delete on that bucket/prefix.
|
|
||||||
5. Turn on a project cost alert.
|
|
||||||
6. Write the non-secret facts into
|
|
||||||
`substrate/object-stores/platform-audit-storage.yaml`:
|
|
||||||
`endpoint`, `bucket`, `prefix`, `provider_project_ref`, `versioning`,
|
|
||||||
`lifecycle`. Status → `active`.
|
|
||||||
7. Append the bucket/project to `declarations/reef.yaml` `current_members`
|
|
||||||
and `substrate/topology.yaml` `members`. Reef `lifecycle_state` may stay
|
|
||||||
`planned` until the first restore (T05), or become `active` when the
|
|
||||||
bucket exists — prefer `active` only after T05.
|
|
||||||
8. Hand `secret:railiance-platform/backup` to OpenBao (T04).
|
|
||||||
Do not commit the key.
|
|
||||||
9. Tell resource-control: set inventory `decision.status: approved`,
|
|
||||||
`status: ordered`, `ordered_on`, `provider.provider_resource_id`.
|
|
||||||
|
|
||||||
Do not paste endpoint or keys into chat as the source of truth. The YAML
|
|
||||||
file is the source for `reef:storage/…` references.
|
|
||||||
|
|
@ -1,37 +0,0 @@
|
||||||
# Put the Scaleway bootstrap key (founder, local only)
|
|
||||||
|
|
||||||
Do this on a trusted terminal. **Do not paste ACCESS_KEY or SECRET_KEY
|
|
||||||
into chat, Git, or State Hub.**
|
|
||||||
|
|
||||||
CCR: `railiance-platform/credential-change-requests/CCR-2026-0011-scaleway-object-storage-bootstrap.yaml`
|
|
||||||
Path: `platform/workloads/railiance/scaleway/bootstrap`
|
|
||||||
|
|
||||||
1. In [console.scaleway.com](https://console.scaleway.com) create or reuse
|
|
||||||
the Railiance (or GmbH) project. IAM → API keys: create a key that can
|
|
||||||
create Object Storage buckets in that project. Copy org id and project id
|
|
||||||
from the project dashboard (those two are not as sensitive as the secret
|
|
||||||
key, but still keep them out of git).
|
|
||||||
2. On this host, with a token that can write the `platform` mount:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
bao kv put platform/workloads/railiance/scaleway/bootstrap \
|
|
||||||
ACCESS_KEY='SCWxxxxxxxx' \
|
|
||||||
SECRET_KEY='xxxxxxxx' \
|
|
||||||
DEFAULT_ORGANIZATION_ID='xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx' \
|
|
||||||
DEFAULT_PROJECT_ID='xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
|
|
||||||
```
|
|
||||||
|
|
||||||
3. Confirm **without printing values**:
|
|
||||||
|
|
||||||
```bash
|
|
||||||
bao kv metadata get platform/workloads/railiance/scaleway/bootstrap
|
|
||||||
```
|
|
||||||
|
|
||||||
You should see a current version. Then tell the agent “bootstrap is in
|
|
||||||
OpenBao.” They will run `tools/create-platform-audit-bucket.sh`, which
|
|
||||||
creates the private Multi-AZ bucket, 30-day lifecycle, and writes only
|
|
||||||
endpoint/bucket/region into `substrate/object-stores/platform-audit-storage.yaml`.
|
|
||||||
|
|
||||||
The scoped backup key is a **different** path
|
|
||||||
(`platform/workloads/railiance/backup/object-storage`) and is T04.
|
|
||||||
After the scoped key works, delete or lock down this bootstrap key.
|
|
||||||
|
|
@ -1,24 +0,0 @@
|
||||||
{
|
|
||||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
|
||||||
"$id": "https://railiance.dev/reef-storage/object-store-attributes.schema.json",
|
|
||||||
"title": "Non-secret object-store attributes",
|
|
||||||
"type": "object",
|
|
||||||
"additionalProperties": false,
|
|
||||||
"required": ["schema_version", "resource_id", "provider", "status"],
|
|
||||||
"properties": {
|
|
||||||
"schema_version": {"const": "0.1"},
|
|
||||||
"resource_id": {"type": "string", "pattern": "^resource:"},
|
|
||||||
"provider": {"type": "string", "minLength": 1},
|
|
||||||
"product": {"type": ["string", "null"]},
|
|
||||||
"status": {"enum": ["planned", "active", "retired"]},
|
|
||||||
"endpoint": {"type": ["string", "null"]},
|
|
||||||
"region": {"type": ["string", "null"]},
|
|
||||||
"bucket": {"type": ["string", "null"]},
|
|
||||||
"prefix": {"type": ["string", "null"]},
|
|
||||||
"versioning": {"type": ["boolean", "null"]},
|
|
||||||
"lifecycle": {"type": ["string", "null"]},
|
|
||||||
"provider_project_ref": {"type": ["string", "null"]},
|
|
||||||
"intended": {"type": ["object", "null"]},
|
|
||||||
"notes": {"type": "array", "items": {"type": "string"}}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
@ -1,15 +0,0 @@
|
||||||
reef_id: reef-storage
|
|
||||||
display_name: Railiance delegated object storage
|
|
||||||
substrate_kind: object-storage
|
|
||||||
ownership_repo: railiance-platform
|
|
||||||
lifecycle_state: planned
|
|
||||||
criticality: high
|
|
||||||
primary_rail: null
|
|
||||||
current_members:
|
|
||||||
- railiance-platform-pg-backup
|
|
||||||
current_role: Provider-delegated S3-compatible store. Scaleway operates the capability; this reef names the boundary and non-secret attributes.
|
|
||||||
source_of_truth:
|
|
||||||
repo: reef-storage
|
|
||||||
paths:
|
|
||||||
- declarations/reef.yaml
|
|
||||||
- substrate/object-stores/platform-audit-storage.yaml
|
|
||||||
|
|
@ -1,25 +0,0 @@
|
||||||
# Live attributes for RESOURCE-WP-0002. No secrets in this file.
|
|
||||||
schema_version: "0.1"
|
|
||||||
resource_id: resource:platform:audit-storage
|
|
||||||
provider: Scaleway
|
|
||||||
product: standard-multi-az
|
|
||||||
status: active
|
|
||||||
endpoint: https://s3.nl-ams.scw.cloud
|
|
||||||
region: nl-ams
|
|
||||||
bucket: railiance-platform-pg-backup
|
|
||||||
prefix: platform-pg/
|
|
||||||
versioning: true
|
|
||||||
lifecycle: 30-day current and noncurrent expiration
|
|
||||||
provider_project_ref: e1a0dd0e-04b8-4ea9-8b30-d53f8c35c688
|
|
||||||
intended:
|
|
||||||
public_access: disabled
|
|
||||||
versioning: true
|
|
||||||
lifecycle: 30-day recovery window matching data/demand/platform-audit-storage.json
|
|
||||||
encryption_at_rest: provider-managed
|
|
||||||
identity: bucket-and-prefix scoped key only
|
|
||||||
notes:
|
|
||||||
- Destination for rapp-postgres WAL and base backups once procured.
|
|
||||||
- resource-control cites reef:storage/substrate/object-stores/platform-audit-storage.yaml#<key>
|
|
||||||
- "Decision record: resource-control/docs/evidence/RESOURCE-WP-0002-primary-selection-2026-08-14.md"
|
|
||||||
- Created 2026-08-14. Versioning enabled. ACL owner-only FULL_CONTROL.
|
|
||||||
- Cost backstop is a founder email billing alert. Scaleway has no euro hard-stop.
|
|
||||||
|
|
@ -1,21 +0,0 @@
|
||||||
reef_id: reef-storage
|
|
||||||
membership_policy: Add a member when a purchased object-store (bucket/project) is live. Members are provider resources, not hosts.
|
|
||||||
members:
|
|
||||||
- member_id: platform-audit-storage
|
|
||||||
role: primary-backup-object-store
|
|
||||||
provider: Scaleway
|
|
||||||
product: standard-multi-az
|
|
||||||
region: nl-ams
|
|
||||||
bucket: railiance-platform-pg-backup
|
|
||||||
resource_id: resource:platform:audit-storage
|
|
||||||
status: live
|
|
||||||
evidence:
|
|
||||||
- repo: resource-control
|
|
||||||
path: docs/evidence/RESOURCE-WP-0002-primary-selection-2026-08-14.md
|
|
||||||
- path: substrate/object-stores/platform-audit-storage.yaml
|
|
||||||
intended_members: []
|
|
||||||
access_surfaces:
|
|
||||||
- type: s3-https
|
|
||||||
note: Endpoint and bucket live in substrate/object-stores/platform-audit-storage.yaml. No rail.
|
|
||||||
unresolved_questions:
|
|
||||||
- Whether Host Europe Backup Storage or Nextcloud is the T06 second copy
|
|
||||||
|
|
@ -1,165 +0,0 @@
|
||||||
#!/usr/bin/env bash
|
|
||||||
# Apply a prefix-scoped bucket policy for the resource-control backup key.
|
|
||||||
# Reads APPLICATION_ID from OpenBao. Never prints secret values.
|
|
||||||
# Uses the bootstrap key to PutBucketPolicy (org owner can always edit policies).
|
|
||||||
set -euo pipefail
|
|
||||||
|
|
||||||
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
|
||||||
BAO_ADDR="${BAO_ADDR:-${VAULT_ADDR:-https://bao.coulomb.social}}"
|
|
||||||
REGION="nl-ams"
|
|
||||||
BUCKET="${BUCKET:-railiance-platform-pg-backup}"
|
|
||||||
PREFIX="${PREFIX:-platform-pg/}"
|
|
||||||
|
|
||||||
need() { command -v "$1" >/dev/null || { echo "missing $1" >&2; exit 2; }; }
|
|
||||||
export PATH="${HOME}/.local/bin:${PATH}"
|
|
||||||
need python3
|
|
||||||
need curl
|
|
||||||
|
|
||||||
TOKEN="${OPENBAO_TOKEN:-${VAULT_TOKEN:-}}"
|
|
||||||
if [[ -z "$TOKEN" && -f "$HOME/.vault-token" ]]; then
|
|
||||||
TOKEN="$(cat "$HOME/.vault-token")"
|
|
||||||
fi
|
|
||||||
[[ -n "$TOKEN" ]] || { echo "no OpenBao token" >&2; exit 2; }
|
|
||||||
|
|
||||||
read_kv() {
|
|
||||||
local path="$1"
|
|
||||||
curl -fsS -H "X-Vault-Token: $TOKEN" \
|
|
||||||
"$BAO_ADDR/v1/platform/data/${path#platform/}" \
|
|
||||||
| python3 -c 'import json,sys; d=json.load(sys.stdin); print(json.dumps(d["data"]["data"]))'
|
|
||||||
}
|
|
||||||
|
|
||||||
BACKUP_JSON="$(read_kv workloads/railiance/backup/object-storage)"
|
|
||||||
BOOTSTRAP_JSON="$(read_kv workloads/railiance/scaleway/bootstrap)"
|
|
||||||
|
|
||||||
EXPORTS="$(
|
|
||||||
BACKUP_JSON="$BACKUP_JSON" BOOTSTRAP_JSON="$BOOTSTRAP_JSON" python3 - <<'PY'
|
|
||||||
import json, os, sys, re
|
|
||||||
backup = json.loads(os.environ["BACKUP_JSON"])
|
|
||||||
boot = json.loads(os.environ["BOOTSTRAP_JSON"])
|
|
||||||
app = (backup.get("APPLICATION_ID") or "").strip()
|
|
||||||
if not app or app in {"xxx", "redacted", "changeme"}:
|
|
||||||
sys.stderr.write(
|
|
||||||
"APPLICATION_ID missing on platform/workloads/railiance/backup/object-storage\n"
|
|
||||||
"Founder: add that field in the OpenBao UI (IAM application resource-control UUID).\n"
|
|
||||||
"It is not a secret. Then re-run this script.\n"
|
|
||||||
)
|
|
||||||
sys.exit(6)
|
|
||||||
if not re.fullmatch(r"[0-9a-fA-F-]{36}", app):
|
|
||||||
sys.stderr.write("APPLICATION_ID is not a 36-char UUID\n")
|
|
||||||
sys.exit(7)
|
|
||||||
def g(data, *ks):
|
|
||||||
for k in ks:
|
|
||||||
if data.get(k):
|
|
||||||
return data[k]
|
|
||||||
return ""
|
|
||||||
print("export APPLICATION_ID=" + json.dumps(app))
|
|
||||||
print("export SCW_ACCESS_KEY=" + json.dumps(g(boot, "ACCESS_KEY")))
|
|
||||||
print("export SCW_SECRET_KEY=" + json.dumps(g(boot, "SECRET_KEY")))
|
|
||||||
print("export BACKUP_ACCESS_KEY=" + json.dumps(g(backup, "ACCESS_KEY", "ACCESS_KEY_ID")))
|
|
||||||
print("export BACKUP_SECRET_KEY=" + json.dumps(g(backup, "SECRET_KEY", "ACCESS_SECRET_KEY")))
|
|
||||||
PY
|
|
||||||
)"
|
|
||||||
eval "$EXPORTS"
|
|
||||||
|
|
||||||
export SCW_DEFAULT_REGION="$REGION"
|
|
||||||
export AWS_ACCESS_KEY_ID="$SCW_ACCESS_KEY"
|
|
||||||
export AWS_SECRET_ACCESS_KEY="$SCW_SECRET_KEY"
|
|
||||||
|
|
||||||
VENV="${TMPDIR:-/tmp}/reef-storage-boto3"
|
|
||||||
if [[ ! -x "$VENV/bin/python" ]]; then
|
|
||||||
python3 -m venv "$VENV"
|
|
||||||
"$VENV/bin/pip" -q install boto3
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo "putting prefix-scoped bucket policy on $BUCKET (application id not printed)"
|
|
||||||
BUCKET="$BUCKET" PREFIX="$PREFIX" APPLICATION_ID="$APPLICATION_ID" \
|
|
||||||
"$VENV/bin/python" - <<'PY'
|
|
||||||
import json, os
|
|
||||||
import boto3
|
|
||||||
bucket = os.environ["BUCKET"]
|
|
||||||
prefix = os.environ["PREFIX"]
|
|
||||||
app = os.environ["APPLICATION_ID"]
|
|
||||||
policy = {
|
|
||||||
"Version": "2023-04-17",
|
|
||||||
"Id": "resource-control-scoped-backup",
|
|
||||||
"Statement": [
|
|
||||||
{
|
|
||||||
"Sid": "BackupListPrefix",
|
|
||||||
"Effect": "Allow",
|
|
||||||
"Principal": {"SCW": f"application_id:{app}"},
|
|
||||||
"Action": [
|
|
||||||
"s3:ListBucket",
|
|
||||||
"s3:ListBucketMultipartUploads",
|
|
||||||
"s3:ListBucketVersions",
|
|
||||||
"s3:GetBucketLocation",
|
|
||||||
"s3:GetBucketVersioning",
|
|
||||||
"s3:GetLifecycleConfiguration",
|
|
||||||
],
|
|
||||||
"Resource": [bucket],
|
|
||||||
"Condition": {"StringLike": {"s3:prefix": [prefix, prefix + "*"]}},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"Sid": "BackupObjectsUnderPrefix",
|
|
||||||
"Effect": "Allow",
|
|
||||||
"Principal": {"SCW": f"application_id:{app}"},
|
|
||||||
"Action": [
|
|
||||||
"s3:GetObject",
|
|
||||||
"s3:GetObjectVersion",
|
|
||||||
"s3:PutObject",
|
|
||||||
"s3:DeleteObject",
|
|
||||||
"s3:DeleteObjectVersion",
|
|
||||||
"s3:AbortMultipartUpload",
|
|
||||||
"s3:ListMultipartUploadParts",
|
|
||||||
],
|
|
||||||
"Resource": [f"{bucket}/{prefix}*"],
|
|
||||||
},
|
|
||||||
],
|
|
||||||
}
|
|
||||||
client = boto3.client(
|
|
||||||
"s3",
|
|
||||||
region_name="nl-ams",
|
|
||||||
endpoint_url="https://s3.nl-ams.scw.cloud",
|
|
||||||
aws_access_key_id=os.environ["AWS_ACCESS_KEY_ID"],
|
|
||||||
aws_secret_access_key=os.environ["AWS_SECRET_ACCESS_KEY"],
|
|
||||||
)
|
|
||||||
client.put_bucket_policy(Bucket=bucket, Policy=json.dumps(policy))
|
|
||||||
print("bucket_policy=put")
|
|
||||||
got = client.get_bucket_policy(Bucket=bucket)
|
|
||||||
body = json.loads(got["Policy"])
|
|
||||||
print("statements", [s.get("Sid") for s in body.get("Statement", [])])
|
|
||||||
PY
|
|
||||||
|
|
||||||
echo "verifying backup key: prefix ok, sibling denied"
|
|
||||||
BUCKET="$BUCKET" PREFIX="$PREFIX" \
|
|
||||||
AWS_ACCESS_KEY_ID="$BACKUP_ACCESS_KEY" AWS_SECRET_ACCESS_KEY="$BACKUP_SECRET_KEY" \
|
|
||||||
"$VENV/bin/python" - <<'PY'
|
|
||||||
import os
|
|
||||||
from botocore.exceptions import ClientError
|
|
||||||
import boto3
|
|
||||||
c = boto3.client(
|
|
||||||
"s3",
|
|
||||||
region_name="nl-ams",
|
|
||||||
endpoint_url="https://s3.nl-ams.scw.cloud",
|
|
||||||
aws_access_key_id=os.environ["AWS_ACCESS_KEY_ID"],
|
|
||||||
aws_secret_access_key=os.environ["AWS_SECRET_ACCESS_KEY"],
|
|
||||||
)
|
|
||||||
bucket = os.environ["BUCKET"]
|
|
||||||
prefix = os.environ["PREFIX"]
|
|
||||||
ok = prefix + ".policy-probe"
|
|
||||||
sib = "other-consumer/.policy-probe"
|
|
||||||
try:
|
|
||||||
c.put_object(Bucket=bucket, Key=ok, Body=b"ok")
|
|
||||||
c.delete_object(Bucket=bucket, Key=ok)
|
|
||||||
print("prefix_write=ok")
|
|
||||||
except ClientError as e:
|
|
||||||
print("prefix_write=fail", e.response.get("Error", {}).get("Code"))
|
|
||||||
try:
|
|
||||||
c.put_object(Bucket=bucket, Key=sib, Body=b"no")
|
|
||||||
print("sibling_write=UNEXPECTED_OK")
|
|
||||||
c.delete_object(Bucket=bucket, Key=sib)
|
|
||||||
except ClientError as e:
|
|
||||||
print("sibling_write=denied", e.response.get("Error", {}).get("Code"))
|
|
||||||
PY
|
|
||||||
|
|
||||||
echo "done"
|
|
||||||
unset SCW_SECRET_KEY SCW_ACCESS_KEY BACKUP_SECRET_KEY BACKUP_ACCESS_KEY AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY APPLICATION_ID
|
|
||||||
|
|
@ -1,139 +0,0 @@
|
||||||
#!/usr/bin/env bash
|
|
||||||
# Create the planned Scaleway backup bucket. Reads bootstrap creds from
|
|
||||||
# OpenBao. Never prints secret values. Writes only non-secret attributes
|
|
||||||
# into substrate/object-stores/platform-audit-storage.yaml.
|
|
||||||
set -euo pipefail
|
|
||||||
|
|
||||||
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
|
||||||
ATTR="$ROOT/substrate/object-stores/platform-audit-storage.yaml"
|
|
||||||
BAO_ADDR="${BAO_ADDR:-${VAULT_ADDR:-https://bao.coulomb.social}}"
|
|
||||||
BOOTSTRAP_PATH="platform/workloads/railiance/scaleway/bootstrap"
|
|
||||||
SCOPED_PATH="platform/workloads/railiance/backup/object-storage"
|
|
||||||
REGION="nl-ams"
|
|
||||||
BUCKET="${BUCKET:-railiance-platform-pg-backup}"
|
|
||||||
PREFIX="${PREFIX:-platform-pg/}"
|
|
||||||
ENDPOINT="https://s3.nl-ams.scw.cloud"
|
|
||||||
|
|
||||||
need() { command -v "$1" >/dev/null || { echo "missing $1" >&2; exit 2; }; }
|
|
||||||
export PATH="${HOME}/.local/bin:${PATH}"
|
|
||||||
need python3
|
|
||||||
need curl
|
|
||||||
need scw
|
|
||||||
|
|
||||||
TOKEN="${OPENBAO_TOKEN:-${VAULT_TOKEN:-}}"
|
|
||||||
if [[ -z "$TOKEN" && -f "$HOME/.vault-token" ]]; then
|
|
||||||
TOKEN="$(cat "$HOME/.vault-token")"
|
|
||||||
fi
|
|
||||||
[[ -n "$TOKEN" ]] || { echo "no OpenBao token" >&2; exit 2; }
|
|
||||||
|
|
||||||
read_kv() {
|
|
||||||
local path="$1"
|
|
||||||
curl -fsS -H "X-Vault-Token: $TOKEN" \
|
|
||||||
"$BAO_ADDR/v1/platform/data/${path#platform/}" \
|
|
||||||
| python3 -c 'import json,sys; d=json.load(sys.stdin); print(json.dumps(d["data"]["data"]))'
|
|
||||||
}
|
|
||||||
|
|
||||||
if ! BOOTSTRAP_JSON="$(read_kv workloads/railiance/scaleway/bootstrap 2>/dev/null)"; then
|
|
||||||
echo "Bootstrap secret missing at $BOOTSTRAP_PATH" >&2
|
|
||||||
echo "Founder: put ACCESS_KEY SECRET_KEY DEFAULT_ORGANIZATION_ID DEFAULT_PROJECT_ID locally (not in chat)." >&2
|
|
||||||
echo "See docs/put-scaleway-bootstrap.md" >&2
|
|
||||||
exit 3
|
|
||||||
fi
|
|
||||||
|
|
||||||
eval "$(BOOTSTRAP_JSON="$BOOTSTRAP_JSON" python3 - <<'PY'
|
|
||||||
import json, os, sys
|
|
||||||
data = json.loads(os.environ["BOOTSTRAP_JSON"])
|
|
||||||
needed = ("ACCESS_KEY", "SECRET_KEY", "DEFAULT_ORGANIZATION_ID", "DEFAULT_PROJECT_ID")
|
|
||||||
missing = [k for k in needed if not (data.get(k) or data.get(k.lower()))]
|
|
||||||
if missing:
|
|
||||||
sys.stderr.write("bootstrap fields missing: " + ",".join(missing) + "\n")
|
|
||||||
sys.exit(4)
|
|
||||||
placeholders = [k for k in needed if (data.get(k) or data.get(k.lower()) or "") in {"xxx", "redacted", "changeme"}]
|
|
||||||
if placeholders:
|
|
||||||
sys.stderr.write("bootstrap still has example placeholders; replace them in the OpenBao UI first\n")
|
|
||||||
sys.exit(5)
|
|
||||||
def g(k):
|
|
||||||
return data.get(k) or data.get(k.lower())
|
|
||||||
print("export SCW_ACCESS_KEY=" + json.dumps(g("ACCESS_KEY")))
|
|
||||||
print("export SCW_SECRET_KEY=" + json.dumps(g("SECRET_KEY")))
|
|
||||||
print("export SCW_DEFAULT_ORGANIZATION_ID=" + json.dumps(g("DEFAULT_ORGANIZATION_ID")))
|
|
||||||
print("export SCW_DEFAULT_PROJECT_ID=" + json.dumps(g("DEFAULT_PROJECT_ID")))
|
|
||||||
PY
|
|
||||||
)"
|
|
||||||
|
|
||||||
export SCW_DEFAULT_REGION="$REGION"
|
|
||||||
|
|
||||||
if scw object bucket get "$BUCKET" region="$REGION" -o json >/dev/null 2>&1; then
|
|
||||||
echo "bucket $BUCKET already exists in $REGION; skipping create"
|
|
||||||
else
|
|
||||||
echo "creating private bucket $BUCKET in $REGION (versioning on)"
|
|
||||||
scw object bucket create "$BUCKET" region="$REGION" acl=private enable-versioning=true -o json
|
|
||||||
fi
|
|
||||||
|
|
||||||
echo "applying 30-day lifecycle (current + noncurrent versions)"
|
|
||||||
VENV="${TMPDIR:-/tmp}/reef-storage-boto3"
|
|
||||||
if [[ ! -x "$VENV/bin/python" ]]; then
|
|
||||||
python3 -m venv "$VENV"
|
|
||||||
"$VENV/bin/pip" -q install boto3
|
|
||||||
fi
|
|
||||||
BUCKET="$BUCKET" AWS_ACCESS_KEY_ID="$SCW_ACCESS_KEY" AWS_SECRET_ACCESS_KEY="$SCW_SECRET_KEY" \
|
|
||||||
"$VENV/bin/python" - <<'PY'
|
|
||||||
import os
|
|
||||||
import boto3
|
|
||||||
client = boto3.client(
|
|
||||||
"s3",
|
|
||||||
region_name="nl-ams",
|
|
||||||
endpoint_url="https://s3.nl-ams.scw.cloud",
|
|
||||||
aws_access_key_id=os.environ["AWS_ACCESS_KEY_ID"],
|
|
||||||
aws_secret_access_key=os.environ["AWS_SECRET_ACCESS_KEY"],
|
|
||||||
)
|
|
||||||
client.put_bucket_lifecycle_configuration(
|
|
||||||
Bucket=os.environ["BUCKET"],
|
|
||||||
LifecycleConfiguration={
|
|
||||||
"Rules": [
|
|
||||||
{
|
|
||||||
"ID": "retain-30-days",
|
|
||||||
"Status": "Enabled",
|
|
||||||
"Filter": {"Prefix": ""},
|
|
||||||
"Expiration": {"Days": 30},
|
|
||||||
"NoncurrentVersionExpiration": {"NoncurrentDays": 30},
|
|
||||||
}
|
|
||||||
]
|
|
||||||
},
|
|
||||||
)
|
|
||||||
print("lifecycle applied")
|
|
||||||
PY
|
|
||||||
|
|
||||||
echo "creating a 20 EUR monthly budget alert if none exists"
|
|
||||||
scw billing budget create consumption-limit=20 enabled=true -o json | python3 -c '
|
|
||||||
import json,sys
|
|
||||||
b=json.load(sys.stdin)
|
|
||||||
print("budget_id", b.get("id") or b.get("budget",{}).get("id"))
|
|
||||||
print("limit", b.get("consumption_limit") or b.get("budget",{}).get("consumption_limit"))
|
|
||||||
' || echo "budget create skipped (permissions or API shape)"
|
|
||||||
|
|
||||||
echo "writing non-secret attributes (no keys)"
|
|
||||||
python3 - <<PY
|
|
||||||
from pathlib import Path
|
|
||||||
p = Path("$ATTR")
|
|
||||||
text = p.read_text()
|
|
||||||
repl = {
|
|
||||||
"status: planned": "status: active",
|
|
||||||
"endpoint: null": "endpoint: $ENDPOINT",
|
|
||||||
"region: nl-ams": "region: $REGION",
|
|
||||||
"bucket: null": "bucket: $BUCKET",
|
|
||||||
"prefix: null": "prefix: $PREFIX",
|
|
||||||
"versioning: null": "versioning: true",
|
|
||||||
"lifecycle: null": "lifecycle: 30-day current and noncurrent expiration",
|
|
||||||
"provider_project_ref: null": "provider_project_ref: ${SCW_DEFAULT_PROJECT_ID}",
|
|
||||||
}
|
|
||||||
for a,b in repl.items():
|
|
||||||
text = text.replace(a, b, 1)
|
|
||||||
p.write_text(text)
|
|
||||||
print(f"updated {p}")
|
|
||||||
print(f"endpoint={ENDPOINT} bucket={BUCKET} region={REGION}")
|
|
||||||
PY
|
|
||||||
|
|
||||||
echo "Cost alert: set a project budget in the Scaleway console if scw billing is unavailable."
|
|
||||||
echo "Scoped backup key is T04: $SCOPED_PATH (not written here)."
|
|
||||||
echo "done. Commit the YAML; do not commit any key."
|
|
||||||
|
|
@ -1,54 +0,0 @@
|
||||||
#!/usr/bin/env bash
|
|
||||||
# Founder-only. Reads a local scaleway.auto.tfvars (or equivalent) and
|
|
||||||
# writes the four fields to OpenBao. Never prints values. Never commit
|
|
||||||
# the tfvars file.
|
|
||||||
set -euo pipefail
|
|
||||||
|
|
||||||
TFVARS="${1:-}"
|
|
||||||
[[ -n "$TFVARS" && -f "$TFVARS" ]] || {
|
|
||||||
echo "usage: $0 /path/to/scaleway.auto.tfvars" >&2
|
|
||||||
exit 2
|
|
||||||
}
|
|
||||||
command -v bao >/dev/null || { echo "missing bao" >&2; exit 2; }
|
|
||||||
|
|
||||||
mapfile -t EXPORTS < <(python3 - "$TFVARS" <<'PY'
|
|
||||||
import re, json, sys
|
|
||||||
text = open(sys.argv[1], encoding="utf-8").read()
|
|
||||||
# HCL-ish: key = "value" (ignore comments and the main.tf provider block)
|
|
||||||
found = {}
|
|
||||||
for name, dest in (
|
|
||||||
("access_key", "ACCESS_KEY"),
|
|
||||||
("secret_key", "SECRET_KEY"),
|
|
||||||
("organization_id", "DEFAULT_ORGANIZATION_ID"),
|
|
||||||
("project_id", "DEFAULT_PROJECT_ID"),
|
|
||||||
):
|
|
||||||
m = re.search(rf'(?m)^\s*{name}\s*=\s*"([^"]*)"\s*$', text)
|
|
||||||
if not m or not m.group(1) or m.group(1) in {"xxx", "redacted"}:
|
|
||||||
sys.stderr.write(f"missing or placeholder field: {name}\n")
|
|
||||||
sys.exit(4)
|
|
||||||
found[dest] = m.group(1)
|
|
||||||
for dest, val in found.items():
|
|
||||||
print(f"{dest}={json.dumps(val)}")
|
|
||||||
PY
|
|
||||||
)
|
|
||||||
|
|
||||||
# Write via env to bao without putting values on argv.
|
|
||||||
# bao kv put supports @file; we use a mode-0600 temp dir.
|
|
||||||
umask 077
|
|
||||||
tmp="$(mktemp -d)"
|
|
||||||
trap 'rm -rf "$tmp"' EXIT
|
|
||||||
for line in "${EXPORTS[@]}"; do
|
|
||||||
field="${line%%=*}"
|
|
||||||
python3 -c 'import json,os,sys; open(sys.argv[2],"w").write(json.loads(sys.argv[1]))' \
|
|
||||||
"${line#*=}" "$tmp/$field"
|
|
||||||
done
|
|
||||||
|
|
||||||
bao kv put platform/workloads/railiance/scaleway/bootstrap \
|
|
||||||
ACCESS_KEY=@"$tmp/ACCESS_KEY" \
|
|
||||||
SECRET_KEY=@"$tmp/SECRET_KEY" \
|
|
||||||
DEFAULT_ORGANIZATION_ID=@"$tmp/DEFAULT_ORGANIZATION_ID" \
|
|
||||||
DEFAULT_PROJECT_ID=@"$tmp/DEFAULT_PROJECT_ID"
|
|
||||||
|
|
||||||
echo "wrote four fields to platform/workloads/railiance/scaleway/bootstrap"
|
|
||||||
bao kv metadata get platform/workloads/railiance/scaleway/bootstrap
|
|
||||||
echo "shred or keep your tfvars; do not commit it"
|
|
||||||
|
|
@ -1,44 +0,0 @@
|
||||||
---
|
|
||||||
id: REEF-STORAGE-WP-0001
|
|
||||||
type: workplan
|
|
||||||
title: "Bootstrap reef-storage"
|
|
||||||
domain: financials
|
|
||||||
repo: reef-storage
|
|
||||||
status: active
|
|
||||||
owner: grok
|
|
||||||
topic_slug: railiance
|
|
||||||
created: "2026-08-14"
|
|
||||||
updated: "2026-08-14"
|
|
||||||
related:
|
|
||||||
- RMASTER-WP-0022
|
|
||||||
- RESOURCE-WP-0002
|
|
||||||
---
|
|
||||||
|
|
||||||
# REEF-STORAGE-WP-0001 — bootstrap
|
|
||||||
|
|
||||||
## Goal
|
|
||||||
|
|
||||||
Stand up the planned storage reef: declaration without a rail, attribute
|
|
||||||
file layout, and the contract for non-secret object-store facts.
|
|
||||||
|
|
||||||
## T01 — Declare the reef
|
|
||||||
|
|
||||||
```task
|
|
||||||
id: REEF-STORAGE-WP-0001-T01
|
|
||||||
status: done
|
|
||||||
priority: high
|
|
||||||
```
|
|
||||||
|
|
||||||
`declarations/reef.yaml`: `substrate_kind: object-storage`, no
|
|
||||||
`primary_rail`, no `hosted_rails`, empty `current_members`, planned.
|
|
||||||
|
|
||||||
## T02 — Attribute contract
|
|
||||||
|
|
||||||
```task
|
|
||||||
id: REEF-STORAGE-WP-0001-T02
|
|
||||||
status: done
|
|
||||||
priority: high
|
|
||||||
```
|
|
||||||
|
|
||||||
`schemas/object-store-attributes.schema.json` and a planned
|
|
||||||
`platform-audit-storage.yaml` with null operating fields until purchase.
|
|
||||||
|
|
@ -1,37 +0,0 @@
|
||||||
---
|
|
||||||
id: REEF-STORAGE-WP-0002
|
|
||||||
type: workplan
|
|
||||||
title: "Fill Scaleway attributes after purchase"
|
|
||||||
domain: financials
|
|
||||||
repo: reef-storage
|
|
||||||
status: active
|
|
||||||
owner: grok
|
|
||||||
topic_slug: railiance
|
|
||||||
created: "2026-08-14"
|
|
||||||
updated: "2026-08-14"
|
|
||||||
related:
|
|
||||||
- RESOURCE-WP-0002
|
|
||||||
---
|
|
||||||
|
|
||||||
# REEF-STORAGE-WP-0002 — fill after purchase
|
|
||||||
|
|
||||||
Purchase happened 2026-08-14. Follow `docs/post-purchase-seed.md`.
|
|
||||||
|
|
||||||
## T01 — Write live non-secret attributes
|
|
||||||
|
|
||||||
```task
|
|
||||||
id: REEF-STORAGE-WP-0002-T01
|
|
||||||
status: done
|
|
||||||
priority: high
|
|
||||||
```
|
|
||||||
|
|
||||||
Fill `substrate/object-stores/platform-audit-storage.yaml` endpoint,
|
|
||||||
bucket, prefix, project ref, versioning, lifecycle. Promote topology
|
|
||||||
members. No secrets.
|
|
||||||
|
|
||||||
Done 2026-08-14: bucket `railiance-platform-pg-backup` in `nl-ams` is
|
|
||||||
live (versioning on, 30-day current and noncurrent expiration, owner-only
|
|
||||||
ACL). Attributes, topology member, and declaration `current_members`
|
|
||||||
written. Reef `lifecycle_state` stays `planned` until RESOURCE-WP-0002-T05.
|
|
||||||
Cost backstop is a founder email billing alert. Scaleway has no
|
|
||||||
euro hard-stop; that gap is accepted.
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue