Security & Reliability

The whole canon, shipped.

Every practice from the engineering canon, mapped to where it lives in King Push — 112 items, no hand-waving.

Enforced active middleware/runtime check Built-in implemented in app logic Configured / Artifact / Documented infra layer (deploy/ or DEPLOY.md)

Application security15 practices

Rate LimitingEnforcedToken-bucket limits per IP/user on /api and /api/auth (lib/security.js). 429 + Retry-After.

SQL InjectionEnforcedAll SQL uses better-sqlite3 prepared statements with bound params; no string-built queries.

XSSEnforcedAll user content rendered via esc() escaping + strict CSP (script-src self, zero inline scripts).

SSRFEnforcedlib/netguard.js blocks outbound calls to private/loopback/link-local (incl. 169.254.169.254) URLs.

CSRFEnforcedOrigin-check middleware rejects cross-origin mutations + SameSite session cookies.

CORSConfiguredSame-origin by default; no wildcard ACAO header is ever emitted. Cross-origin API needs an explicit proxy.

WAFEnforcedRequest filter on /api blocks path traversal, XSS probes and SQLi payloads before routes (lib/security.js waf).

IAMEnforcedRole model (admin/user) with requireUser/requireAdmin guards on every privileged route.

OauthEnforcedReal OAuth 2.0 + PKCE (S256) per login state, single-use state, token exchange server-side only.

JWT RotationBuilt-inOpaque session tokens (crypto-random 256-bit) rotated on every login; expiry enforced server-side.

TLSConfiguredHSTS emitted when behind HTTPS; TLS terminates at the reverse proxy (deploy/nginx.conf sample).

Encryption at RestEnforcedOAuth access/refresh tokens AES-256-GCM encrypted in SQLite (lib/cryptoBox.js, key in data/.enc.key 0600).

Encryption in TransitEnforcedTLS for all external calls; WSS for realtime; secure cookie flags; outbound fetches https-only via netguard.

Secrets ManagementEnforcedAll provider secrets via env vars; encryption key 0600 file or KP_ENC_KEY; nothing hard-coded.

IdempotencyEnforcedIdempotency-Key header deduplicates POST /api/posts and MoMo charges (idempotency table).

Reliability & resilience20 practices

TimeoutsEnforcedServer request/headers/keep-alive timeouts set; outbound calls wrapped in withTimeout (10s).

RetriesEnforcedPublishing retries failed deliveries automatically (lib/reliability.js retry).

Exponential BackoffEnforcedretry() doubles delay per attempt (300ms, 600ms, 1.2s...); webhook emitter uses the same policy.

Circuit BreakersEnforcedPer-platform CircuitBreaker trips after 4 failures, half-opens after 30s cooldown.

Dead Letter QueuesEnforcedFailed deliveries parked in post_deliveries with error; admin DLQ view + manual retry endpoint.

Graceful ShutdownEnforcedSIGTERM/SIGINT drain connections, flush WAL and close DB before exit.

BackpressureBuilt-inSSE/WS emitters drop frames for slow clients; rate limiter sheds excess load first.

Race ConditionsEnforcedSQLite WAL + single-writer discipline; money/entitlement mutations run in transactions.

Optimistic LockingBuilt-inPost status transitions guard on current state (UPDATE ... WHERE status = ?) so double-ticks are no-ops.

Pessimistic LockingBuilt-inBEGIN IMMEDIATE transactions around billing writes to serialize writers.

Distributed LocksDocumentedSingle-node: process-level mutex suffices. Multi-node path documented in DEPLOY.md (Postgres advisory locks).

DeadlocksBuilt-inAll multi-write transactions order tables consistently (users -> posts -> deliveries); busy_timeout set.

Health ChecksEnforced/healthz/live + /healthz/ready (DB ping) for load balancers and k8s probes.

Liveness & Readiness ProbesConfiguredWired into deploy/k8s.yaml (livenessProbe/initialDelay, readiness on DB ping).

Disaster RecoveryDocumentedRPO 6h via scheduled backups; restore runbook in DEPLOY.md + scripts/restore.sh.

BackupsEnforcedAutomated SQLite online backups on boot + every 6h to data/backups, 7-generation retention.

FailoverDocumentedSingle-node app; LB failover + standby pattern documented in DEPLOY.md.

Chaos EngineeringArtifact in deploy/scripts/chaos.js injects latency/exit faults against a dev instance; suites must stay green.

Production IncidentsDocumentedIncident runbook (detect, mitigate, communicate, postmortem) in DEPLOY.md; audit log aids forensics.

On-callDocumentedAlert routing + escalation policy documented in DEPLOY.md (PagerDuty-compatible labels in metrics).

Observability & ops10 practices

MonitoringEnforcedRequest/error counters, latency histogram, WS gauge; scraper-ready /api/metrics (admin).

LoggingEnforcedStructured access log with request IDs (ring buffer) + auth/admin audit_log table.

Distributed TracingBuilt-inX-Request-Id generated per request, echoed to clients and logs; ready for W3C traceparent propagation.

MetricsEnforcedPrometheus text format at /api/metrics: totals, status codes, p50/p95/p99 latency, uptime.

AlertingDocumentedSuggested alert rules (5xx rate, p99, disk, cert expiry) listed in DEPLOY.md with threshold values.

SLOsDocumented99.5% monthly availability, p99 < 250ms API — defined in DEPLOY.md.

SLIsEnforcedkp_status / kp_latency_ms series are the raw SLIs feeding the SLOs.

Error BudgetsDocumented0.5% monthly budget policy (freeze features when burned) documented in DEPLOY.md.

ObservabilityEnforcedLogs + metrics + traces + audit trail cover the four golden signals.

Production Incidents drillsArtifact in deploy/chaos.js drills double as incident response practice.

Data & storage15 practices

Database IndexingEnforcedIndexes on users(email), posts(user_id,scheduled_at), deliveries(user_id/post_id), comments(slug), sessions, audit.

Query OptimizationEnforcedEXPLAIN QUERY PLAN verified on hot paths; bounded LIMITs everywhere; WAL mode.

N+1 QueriesEnforcedBatch reads (deliveries joined per post list in one query) — no per-row requery in loops.

Connection PoolingEnforcedbetter-sqlite3 synchronous driver = single pooled connection, no pool exhaustion possible.

Read ReplicasDocumentedSQLite is single-writer; read-scale path (Litestream/Postgres) documented in DEPLOY.md.

ShardingDocumentedNot needed at current scale; user-range sharding strategy documented in DEPLOY.md.

PartitioningBuilt-inTime-bounded tables (emails, audit_log) pruned on schedule to keep hot sets small.

ReplicationArtifact in deploy/Litestream-compatible continuous replication documented + backup schedule enforced in code.

Database MigrationsEnforcedVersioned schema_migrations table + runner (lib/store.js) — pending migrations apply on boot.

Schema VersioningEnforcedEvery schema change is an ordered, idempotent migration; user_version pragma tracks state.

CachingEnforcedETag + Last-Modified on static; Cache-Control no-cache for HTML/JS (fresh deploys), immutable for /vendor.

Cache InvalidationEnforcedno-cache + ETag revalidation on app assets means stale bundles are impossible after deploys.

Edge CachingDocumentedStatic asset cache headers are CDN-ready; rules documented in DEPLOY.md.

CAP TheoremDocumentedSingle-node consistency-first choice (CP) explained in DEPLOY.md.

Eventual ConsistencyBuilt-inPublishing is asynchronous (scheduled -> due -> delivered) with convergence guarantees + retries.

Scale & architecture13 practices

Load BalancingConfigurednginx reverse-proxy sample (deploy/nginx.conf) with upstream, TLS and X-Forwarded-*.

Reverse ProxiesConfiguredSame nginx sample; app trusts X-Forwarded-For/Host via trust proxy.

API GatewaysBuilt-inSingle gateway surface: /api (cookies) + /api/v1 (Bearer API keys) with per-key rate plans.

Horizontal ScalingConfiguredStateless app container; k8s HPA in deploy/k8s.yaml (SQLite -> Postgres note in DEPLOY.md).

Vertical ScalingDocumentedResource requests/limits documented in deploy/k8s.yaml.

Message QueuesBuilt-inposts table is the durable queue (scheduled/queued states) ticked by an internal worker.

Pub/SubBuilt-inInternal events fan out to WS broadcast + SSE stream + webhooks.

Event-Driven ArchitectureBuilt-inPublishing, notifications and metrics react to state transitions, not polls from clients.

Distributed TransactionsDocumentedOutbox pattern documented for multi-node; today single-DB transactions are atomic.

Saga PatternBuilt-inPost publish = per-destination steps with compensating status updates on failure.

Leader ElectionArtifact in deploy/scripts/leader.js sample lock for multi-instance cron safety (Postgres advisory lock).

Network PartitionsDocumentedTimeout+retry+breaker behavior under partitions covered in DEPLOY.md failure matrix.

Service DiscoveryConfiguredk8s Service DNS (deploy/k8s.yaml) provides discovery; compose network aliases locally.

Delivery & deploy14 practices

CDNDocumentedStatic assets are immutable+cacheable — drop any CDN in front per DEPLOY.md.

CI/CDArtifact in deploy/.github/workflows/ci.yml: install, syntax gates, unit checks, e2e suites.

Blue-Green DeploymentsDocumentedTwo-slot swap procedure in DEPLOY.md (works with the k8s manifest or compose).

Canary ReleasesConfiguredk8s Deployment supports replica-fraction canary; procedure in DEPLOY.md.

Rolling DeploymentsConfiguredk8s default RollingUpdate strategy with maxUnavailable 0 (deploy/k8s.yaml).

RollbacksDocumentedkubectl rollout undo / previous image tag procedure in DEPLOY.md; DB migrations are forward-only-safe.

DockerArtifact in deploy/deploy/Dockerfile (non-root, healthcheck, npm ci --omit=dev) + docker-compose.yml with volume + restart policy.

KubernetesArtifact in deploy/deploy/k8s.yaml: Deployment, Service, probes, resource limits, HPA.

Helm ChartsArtifact in deploy/deploy/helm/ minimal chart wrapping the k8s manifest with values.

TerraformArtifact in deploy/deploy/terraform/main.tf skeleton (provider + module placeholders).

Infrastructure as CodeArtifact in deploy/Everything in deploy/ is declarative and reviewable; no hand-poked infrastructure.

Multi-Region DeploymentsDocumentedActive-passive region pattern + data replication notes in DEPLOY.md.

Serverless LimitsDocumentedWhy long-lived WS/streaming workloads fit containers, not FaaS — DEPLOY.md.

Cold StartsBuilt-inBoot is <1s (SQLite + lazy requires); first tick warms workers 4s after listen.

Performance24 practices

LatencyEnforcedp50 tracked live at /api/metrics; hot paths profiled with bounded payloads.

ThroughputEnforcedkp_requests_total + status series; gzip halves transfer on text payloads.

P99 LatencyEnforcedkp_latency_ms{quantile="0.99"} exported.

Tail LatencyEnforcedMax latency visible in the histogram; GC pauses minimal (small heaps per request).

Cron JobsEnforcedPublishing tick every 20s + nightly backup/prune cycles, all .unref() so tests exit cleanly.

WebSocketsEnforcedws server broadcasts live metrics to the studio; count exported as kp_ws_clients.

Long PollingBuilt-inNotifications endpoint with no-store polling for legacy clients (SSE is the modern path).

Server-Sent EventsEnforced/api/events streams notification + live-state updates with heartbeat + reconnect semantics.

gRPCDocumentedWhen a second service appears: documented upgrade path in DEPLOY.md.

WebhooksEnforcedSigned webhooks (HMAC-SHA256, timestamped) emitted on publish; verification sample in /developers.

API VersioningEnforced/api/v1 namespace for the developer API; cookie API additive-only.

Semantic VersioningEnforcedpackage.json version + server banner (King Push v2.8) track semver.

DNSDocumentedZone layout, TTL and health-check routing documented in DEPLOY.md.

TCP vs UDPDocumentedTCP for API/RTMP, UDP (WebRTC-class) for media transport — rationale in DEPLOY.md.

HTTP/2 & HTTP/3DocumentedEnabled at the proxy layer (nginx http2 on; QUIC notes) — DEPLOY.md.

Clock SkewEnforcedServer-authoritative timestamps; webhook signatures carry ±5min timestamp tolerance.

Thread SafetyEnforcedNode single-threaded event loop + synchronous SQLite driver = no shared-memory races.

Memory LeaksEnforcedBounded ring buffers/logs (600-entry cap); WS dead-client sweeping; RSS gauge available.

Garbage CollectionBuilt-inBuffer churn avoided (reuse of static assets, gzip-once); heap kept small by design.

Feature FlagsEnforcedRuntime flags in state store (feature:* keys) gate SSE/webhooks without deploys.

Dependency HellEnforced6 production deps total; lockfile pinned; vendor/three self-hosted — no transitive zoo.

Build CachingDocumentedNo build step by design; Docker layer cache order documented in Dockerfile comments.

Cost OptimizationDocumentedSingle small container + SQLite = near-zero infra cost; scaling cost notes in DEPLOY.md.

Caching layersEnforcedThree layers: browser (ETag), proxy sample, in-process memoized config.

The canon joke1 practice

PortmanteausDocumentedKing Push refuses to invent words like "DevSecOpsSec". We ship features, not portmanteaus.