railiance-master/history/260724-InitialExplorationOfOperationModels.md

Exploration of railiance modes to operate managed workloads

--worsch, 24.7.26 r1

Intro

We need to consistently run workloads and there are tradeoffs depending on which operations architecture aka "rail" one uses for a specific workload. This is an exploration how we can do that, with knative as the first example.

This is about how and which rails might be relevant.

Knative in one sentence

Knative adds a serverless application platform on top of Kubernetes. Instead of developers managing Deployment, Service, autoscaling, ingress, revisions, and traffic splitting separately, they deploy a Knative service or function and let Knative construct and operate those resources.

Knative is now a CNCF Graduated project and consists of three related parts:

• Knative Serving — request-driven container deployment, revisions, traffic routing, autoscaling, and scale-to-zero.
• Knative Eventing — CloudEvents-based routing from sources through brokers and triggers to event consumers.
• Knative Functions — a developer-oriented function framework built on Serving and Eventing. (CNCF)

Conceptual architecture

Developer / GitOps / CI
          │
          ▼
   Knative Service
          │
     Configuration
          │
   immutable Revisions
          │
        Route
          │
   Gateway / Ingress
          │
      ┌───┴─────────────────┐
      │                     │
Activator               Running revision
cold-start path              │
      │                  queue-proxy
      └───────────────────────│
                              ▼
                       application container


Event producer
      │
    Source
      │
 Broker / Channel
      │
   Trigger filter
      │
      ▼
 Sink: Knative Service, Kubernetes Service, etc.

Each change creates an immutable Revision, and traffic can be distributed between revisions for canary, blue-green, or rollback scenarios. When a service is at zero replicas, the Activator can hold incoming requests while the autoscaler starts a revision. Requests reaching an active revision normally pass through Knative’s queue-proxy sidecar before reaching the application container. (knative.dev)

---

What Knative changes in your infrastructure

1. Kubernetes becomes an application platform

Without Knative, your application platform typically exposes:

Deployment + Service + Gateway/Ingress + HPA + rollout tooling

With Knative, the primary developer abstraction becomes:

Knative Service + optional Eventing resources

This is a meaningful architectural improvement when you want HelixForge-style repositories to publish applications through a consistent platform contract. It is less attractive when infrastructure teams want applications to remain completely explicit and close to native Kubernetes resources.

The benefit is platform standardization. The cost is that Knative becomes another reconciliation layer whose behaviour must be understood during incidents.

2. The network architecture changes

Knative Serving needs a compatible networking layer. Current installation options include Kourier, Contour, Istio, and a Gateway API integration. The Knative project currently tests Gateway API implementations based on Istio, Contour, and Envoy Gateway. (knative.dev)

This has practical consequences:

• Your existing ingress controller might not be reusable.
• Knative may introduce a second gateway or service-mesh data plane.
• DNS and wildcard domains become part of the platform contract.
• External TLS, internal routing, Activator routing, and ordinary Kubernetes service routing need to be observed separately.

For a k3s installation using its typical Traefik setup, for example, you should expect either an additional supported Knative gateway or a deliberate migration to a supported Gateway API implementation. That conclusion follows from the supported and tested networking options rather than from a general inability of Traefik to route HTTP traffic.

3. Autoscaling becomes request-aware

Knative’s default KPA autoscaler can scale based on request concurrency and supports scale-to-zero. The ordinary HPA option supports CPU-based scaling but does not provide Knative’s scale-to-zero behaviour. (knative.dev)

That works particularly well for:

• Bursty APIs.
• Webhook receivers.
• Infrequently used tenant-specific services.
• Preview environments.
• Internal tools with long idle periods.
• Lightweight event handlers.

It is less suitable for services requiring:

• Predictable sub-second latency at all times.
• Persistent in-memory state.
• Sticky sessions.
• Long-lived connections that cannot tolerate revision replacement.
• Large runtimes or models with expensive initialization.

Scale-to-zero reduces idle consumption, but converts part of your capacity problem into a cold-start engineering problem. Container pull time, application initialization, secrets retrieval, database connection setup, readiness checks, and node capacity all become part of request latency.

4. Every application pod gains platform behaviour

Knative normally adds a queue-proxy sidecar to revision pods. The Activator may additionally enter the request path during scale-from-zero or capacity transitions. (knative.dev)

This affects:

• Pod resource requests and limits.
• Service-mesh sidecar combinations.
• Network policy.
• Request tracing.
• Graceful termination.
• Per-request metrics.
• Failure analysis.
• Capacity calculations.

For a large number of small services, platform sidecars and control-plane objects can become a significant portion of total resource consumption.

5. The control plane becomes larger

A basic Serving installation introduces CRDs, controllers, admission webhooks, an autoscaler, Activator components, networking integration, and configuration objects. Eventing adds its own controllers, brokers, dispatchers, sources, channels, and potentially Kafka infrastructure.

The official single-node installation guidance currently begins at 6 CPUs, 6 GB RAM, and 30 GB disk, although real production sizing depends heavily on the number of revisions, services, requests, and Eventing resources. (knative.dev)

Knative also follows a relatively fast Kubernetes compatibility schedule. As of July 24, 2026, the supported Knative release lines are 1.21 and 1.22, requiring Kubernetes 1.33 and 1.34 respectively; Knative 1.23 is scheduled for July 28, 2026. (GitHub)

That means Knative can influence when your k3s or Kubernetes platform must be upgraded.

6. Observability becomes more important

Scale-to-zero means pods and their local logs disappear regularly. Knative explicitly recommends centralized log collection because Serving deletes pods as they are no longer needed. Its control plane and request path expose metrics and support OpenTelemetry-based integration. (knative.dev)

Your OpsCatalog should distinguish at least:

Gateway → Knative Route → Activator → queue-proxy → application

Otherwise, an apparent application outage may really be:

• A gateway routing problem.
• A Knative Route or Revision condition.
• Failed scale-from-zero.
• Unschedulable pods.
• Image pull delay.
• A queue-proxy readiness issue.
• Application initialization failure.

7. Knative is not a hard multitenancy boundary

Knative’s own threat model describes a Namespace-as-a-Service model for teams within a common organization sharing a cluster, control plane, and nodes. (knative.dev)

That aligns well with trusted internal platform teams. It does not by itself establish strong isolation between mutually hostile customers.

For your multi-vendor and government/corporate security ambitions, Knative should therefore sit inside a broader tenancy architecture involving:

• RBAC and admission policy.
• Resource quotas and limit ranges.
• Default-deny network policy.
• Workload identity.
• Secret isolation.
• Separate node pools where appropriate.
• Separate clusters or control planes for high-assurance tenant boundaries.

Kubernetes itself distinguishes trusted multi-team sharing from multi-customer tenancy and requires additional controls around resource and security isolation. (Kubernetes)

There is also an important current security qualification: Knative documents its cluster-local and system-internal TLS features as experimental, and states that not all control-plane traffic is encrypted. (knative.dev)

---

SWOT assessment

	Positive	Negative
Internal	Strengths: Strong Kubernetes-native serverless abstraction; container rather than language lock-in; scale-to-zero; immutable revisions; built-in traffic splitting; CloudEvents integration; pluggable networking; compatible with GitOps; mature CNCF governance.	Weaknesses: Significant CRD and controller footprint; additional network data path; cold-start latency; queue-proxy overhead; more difficult troubleshooting; assumes largely stateless and fungible instances; Kubernetes-version coupling; Eventing can add another messaging abstraction.
External	Opportunities: A standard application-runtime surface for HelixForge; tenant- or project-specific services that consume no resources while idle; disposable environments; webhook and adapter execution; KServe-based AI inference; consistent rollout and rollback; a common event vocabulary across projects.	Threats: Duplication with Gateway API, service mesh, Argo Rollouts, KEDA, Dapr, Temporal, and existing brokers; platform lock-in at the Kubernetes-CRD level; accidental use for stateful or latency-critical workloads; insufficient isolation for hostile tenants; uncontrolled creation of revisions and services; upgrade burden becoming larger than saved application effort.

Overall SWOT verdict

Knative’s greatest strength is not merely scale-to-zero. It is the creation of a coherent application execution contract.

Its greatest weakness is also that contract: once adopted broadly, networking, rollout, autoscaling, observability, security, and deployment semantics become tied to Knative.

---

Alternative open-source approaches

There is no universally “more advanced” alternative. Several tools are more advanced for a narrower problem.

KEDA plus ordinary Kubernetes workloads

KEDA is the strongest alternative when you primarily want event-driven scaling rather than a serverless application platform.

It works alongside the Kubernetes HPA, can scale deployments and other resources to zero based on queues, databases, APIs, or other external metrics, and can create Kubernetes Jobs for event-driven batch processing. KEDA is CNCF Graduated. (KEDA)

Prefer KEDA when:

• You want to retain standard Deployment, StatefulSet, and Job objects.
• Workers pull from Kafka, RabbitMQ, NATS, Redis, or another queue.
• You do not need revision-aware HTTP routing.
• You want incremental, low-impact adoption.
• Different workloads require different scaling models.

KEDA’s HTTP add-on remains beta, so Knative is currently the more established choice for transparent HTTP scale-from-zero routing. (KEDA)

For your infrastructure, KEDA is likely the better default autoscaling component, with Knative reserved for services that specifically benefit from request-driven scale-to-zero and revision routing.

Fission

Fission is a function-oriented serverless platform. Developers can submit source code against language environments without constructing container images themselves. It maintains warm runtime pools and emphasizes fast function startup and developer productivity. The project remains Apache-2.0 licensed and released version 1.27.0 on June 22, 2026. (Fission)

Prefer Fission when:

• The primary product is a FaaS developer experience.
• Developers should submit small Python, JavaScript, Go, or similar functions.
• Building and managing OCI images should be hidden.
• Warm runtime pools are acceptable.
• Functions are the primary unit, rather than arbitrary containerized applications.

Fission is more specialized and potentially more convenient for function developers, but Knative is the broader and more composable application platform.

Nuclio

Nuclio concentrates on high-performance event and data processing, including CPU- and GPU-intensive execution and integration with streaming and data-science environments. It is Apache-2.0 licensed and released version 1.17.1 on July 7, 2026. (GitHub)

Prefer Nuclio when:

• Functions consume high-volume streams.
• Processing is data-, I/O-, or compute-intensive.
• GPU execution matters.
• Jupyter, Kubeflow, or data-science integration is central.
• Function processor performance is more important than a generic application-platform abstraction.

Nuclio may be “more advanced” for high-performance data functions, but not for general microservice platform governance.

Dapr

Dapr is not a Knative replacement at the deployment level. It is a distributed application runtime exposing building blocks for service invocation, pub/sub, state, actors, jobs, secrets, configuration, and workflows. It normally adds a Dapr runtime process or sidecar to each participating service and is CNCF Graduated. (Dapr Docs)

Prefer or add Dapr when:

• Applications need portable infrastructure APIs.
• You want to swap brokers or state stores without rewriting application integration.
• Service-to-service resiliency is more important than scale-to-zero.
• Actors, pub/sub, or application-level workflows are required.

Knative answers “how is this container deployed and activated?”

Dapr answers “how does this distributed application use infrastructure capabilities?”

They can be combined, but combining their sidecars and control planes should be justified by concrete use cases.

Temporal

Temporal is paramount when events initiate durable, stateful, multi-step coordination.

A Knative event handler can receive an event and execute code, but it does not replace durable workflow state, replay, long-lived waiting, activity retries, compensation, or workflow versioning. Temporal provides those capabilities and supports self-hosted production operation. (Temporal Docs)

Given your existing Temporal-centered event-backbone direction, I would preserve this separation:

Knative Serving  → bursty request-driven execution
KEDA             → queue- and metric-driven worker scaling
Temporal         → durable coordination and control loops
Kafka/NATS/etc.  → durable event transport

Knative Eventing can then act as a CloudEvents routing and adaptation surface rather than becoming the authoritative workflow or event-history system.

SpinKube or wasmCloud

SpinKube and wasmCloud represent a more radical WebAssembly-based architecture.

SpinKube runs Spin-based Wasm applications through Kubernetes and is currently a CNCF Sandbox project. wasmCloud provides a distributed Wasm component platform spanning Kubernetes, cloud, datacenter, and edge, and is a CNCF Incubating project. (SpinKube)

Prefer evaluating Wasm when:

• Very high workload density is important.
• Edge or intermittently connected environments matter.
• Applications are small, portable components.
• Fast startup and low runtime overhead outweigh ecosystem maturity.
• You want stronger capability-oriented sandboxing for extension code.
• Polyglot components must run consistently across Kubernetes and non-Kubernetes environments.

This is potentially more forward-looking than Knative, but it imposes a more substantial programming and runtime model change. I would currently treat it as an experimental runtime class, not the default foundation for all services.

KServe

For model inference, KServe is more appropriate than building a generic Knative function platform yourself. KServe’s default deployment mode currently uses Knative for request-driven serverless inference; its Standard mode can optionally use KEDA, although Standard mode does not currently support HTTP scale-from-zero. (kserve.github.io)

Thus, Knative may become an underlying dependency of an AI inference platform, rather than the user-facing AI platform itself.

OpenFaaS caveat

OpenFaaS remains technically capable, but it is no longer a straightforward fully open-source production alternative for a business platform. Its Community Edition is limited to personal exploration and short commercial proofs of concept; production features and commercial use require a paid edition, and scale-from-zero is not included in CE. (openfaas.com)

That licensing model appears poorly aligned with your preference for sovereign, commercially usable open-source infrastructure.

---

When a different architecture should take precedence

Dominant requirement	Prefer
Stateless HTTP services with scale-to-zero and traffic splitting	Knative Serving
Queue workers and event-driven batch jobs	KEDA + Deployments/Jobs
Source-code-oriented FaaS developer experience	Fission
High-performance stream, data, CPU, or GPU functions	Nuclio
Portable distributed-application APIs	Dapr
Durable processes, retries, waiting, compensation, control loops	Temporal
ML model serving	KServe, possibly backed by Knative
Edge and high-density component execution	wasmCloud or SpinKube
Stateful, latency-critical, always-on services	Standard Kubernetes Deployments/StatefulSets
Mutually untrusted customers or high-assurance isolation	Separate clusters/control planes, possibly with Knative inside each boundary

---

Recommendation for your platform

I would introduce Knative as an optional execution capability, not as the universal deployment model.

A sensible initial architecture would be:

                     Platform deployment classes
                               │
        ┌──────────────────────┼──────────────────────┐
        │                      │                      │
Standard Kubernetes      Knative Serving          KEDA workers
long-lived/stateful      bursty HTTP/events       queues and jobs
        │                      │                      │
        └──────────────────────┴──────────────────────┘
                               │
                       Temporal coordination
                               │
                     Kafka/NATS/RabbitMQ events

Start with Knative Serving only. Reuse a supported Gateway API implementation where possible, keep Eventing out of the first deployment, and evaluate three representative workloads:

1. A bursty webhook or adapter service.
2. An infrequently used internal API.
3. A tenant- or project-specific disposable service.

Measure:

• Idle control-plane and sidecar consumption.
• Cold-start median and tail latency.
• Time from request to ready revision.
• Behaviour under node exhaustion.
• Revision rollout and rollback.
• Log and trace completeness.
• Network-policy compatibility.
• Upgrade effort.
• Failure isolation between namespaces.

Add Knative Eventing later only where its Source–Broker–Trigger model demonstrably simplifies CloudEvents routing. For your architecture, Temporal should remain the durable coordination layer and KEDA the default event-driven scaler. Knative then becomes a valuable, bounded runtime for stateless, bursty services rather than an additional universal control plane competing with the systems you already intend to establish.