[{"data":1,"prerenderedAt":457},["ShallowReactive",2],{"blog-post-detail-building-sluice-qos-aware-capacity-governance-for-self-hosted-llm-inference-flbu":3,"blogs-all-posts-detail-suggestions-en":28,"blog-comments-building-sluice-qos-aware-capacity-governance-for-self-hosted-llm-inference-flbu-en":393},{"status":4,"source":5,"data":6},"success","markdown-file",{"id":7,"title":8,"slug":9,"lang":10,"category":11,"categorySlug":12,"summary":13,"excerpt":13,"author":14,"date":15,"readTime":16,"image":17,"tags":18,"publishedAt":22,"createdAt":22,"updatedAt":22,"filePath":23,"sourceUrl":24,"content":25,"seoTitle":26,"seoDescription":13,"canonicalUrl":27},"cron-1786955346614","Building Sluice: QoS-Aware Capacity Governance for Self-Hosted LLM Inference","building-sluice-qos-aware-capacity-governance-for-self-hosted-llm-inference-flbu","en","Kubernetes","kubernetes","📦 Project: https:\u002F\u002Fgithub.com\u002FVampiricCyborg\u002Fsluice           1. The Problem: When Capacity Becomes...","Madhav M S","8\u002F17\u002F2026","6 phút","https:\u002F\u002Fmedia2.dev.to\u002Fdynamic\u002Fimage\u002Fwidth=1200,height=627,fit=cover,gravity=auto,format=auto\u002Fhttps%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fh3vuie9oakcn3bqzxtbh.png",[11,19,20,12,21],"distributedsystems","llm","systemdesign","2026-08-17T08:29:06.613Z","\u002FUsers\u002Fnguyenanhtuan\u002FCode\u002FNuxtjs\u002Fnextpress\u002Fintlight\u002Fpublic\u002Fcontents\u002Fkubernetes\u002Fbuilding-sluice-qos-aware-capacity-governance-for-self-hosted-llm-inference-flbu.md","https:\u002F\u002Fdev.to\u002Fvampiriccyborg\u002Fbuilding-sluice-qos-aware-capacity-governance-for-self-hosted-llm-inference-13ja","\n📦 Project: https:\u002F\u002Fgithub.com\u002FVampiricCyborg\u002Fsluice\n\n\n## 1. The Problem: When Capacity Becomes the Bottleneck\n\nA self-hosted vLLM deployment runs on a GPU pool of fixed size. That's the fact that changes everything about how you have to think about load.\n\nOnce that pool's KV-cache capacity comes under pressure, requests don't naturally understand business tiers. They don't know that one tenant is an enterprise customer with a contract, another is an internal team, and a third is a batch job that can wait. Without something external saying otherwise, the system's behavior under contention is driven by arrival order and backend scheduling — and every tenant, regardless of what they were promised, experiences the same degraded latency, queueing, or failure.\n\nThere's no built-in mechanism that says:\n\n> Preserve Guaranteed traffic, reduce the work done by Standard traffic, and shed Best-Effort traffic first.\n\nSluice exists to make that decision — before the request ever reaches vLLM.\n\nConcretely, it evaluates three live signals per decision cycle:\n\n1. **pressure** — vLLM GPU KV-cache usage, read from Prometheus.\n2. **queue_depth** — vLLM's waiting-request depth, also from Prometheus.\n3. **sla_violation_rate** — computed from completed requests in the PostgreSQL decision ledger.\n\nTenants are mapped to tiers in `config\u002Ftenants.yaml`:\n\n```yaml\ntenants:\n  support-bot: Guaranteed\n  coding-assistant: Standard\n  batch-analytics: Best-Effort\n```\n\nand each tier has an explicit SLA target, hard-coded in `sluice_proxy\u002Fapp.py`:\n\n```python\nsla_targets = {\n    \"Guaranteed\": 2000,\n    \"Standard\": 1000,\n    \"Best-Effort\": 500,\n}\n```\n\nThe important thing to understand about Sluice's scope from the start: it is not trying to make vLLM's own execution more efficient. It governs *which* requests reach the inference engine, *how much work* they're allowed to request, and *which backend pool* they land on. Everything downstream of that decision is still vLLM's job.\n\n## 2. Why Existing Layers Don't Solve It\n\nIt's worth being precise about why this problem doesn't already have a home in the stack, because Sluice's design only makes sense in contrast to the layers around it.\n\n**Kubernetes** can place workloads, restart failed replicas, and add capacity if an autoscaler is configured. None of that helps when GPU capacity is fixed or slow to provision — which, for self-hosted inference, is closer to the default case than the exception. Kubernetes has no concept of a \"tenant\" or a \"tier\" at the request level; it schedules pods, not chat completions.\n\n**vLLM's own scheduler** is the right layer for token-level execution, batching, and KV-cache management inside a worker — and it's *better* at that than anything sitting in front of it could be. Sluice deliberately stays outside that loop. It has no interest in improving token scheduling; it adds tenant identity, tier semantics, and an auditable reason for each request-level action, then gets out of the way.\n\n**A generic API gateway** (Envoy, Kong) gives you authentication, routing, retries, and rate limiting — genuinely useful, and Sluice doesn't try to replace any of it. But rate limits alone can't express \"degrade this tier's `max_tokens` before rejecting that tier's requests, based on live GPU pressure.\" That's a specialized policy signal and action set that a generic gateway has no vocabulary for.\n\nThe actual boundary in code is narrow. The original Phase 1 implementation of `sluice_proxy\u002Fapp.py` did five things: identify the tenant from request headers, look up their tier, read cache pressure, apply a policy decision, and optionally rewrite `max_tokens` before forwarding. It never touched token scheduling, batching, or KV-cache allocation — those stayed vLLM's problem, from the very first commit.\n\nThe `is_shapeable()` function makes this boundary explicit today. Only `POST`, `PUT`, `PATCH` requests to `\u002Fv1\u002Fchat\u002Fcompletions` or `\u002Fv1\u002Fcompletions` are eligible for shaping at all. Everything else passes through untouched. And even for shapeable requests, Sluice only ever changes two fields: `max_tokens` (via `apply_max_tokens()`) and `model` (via `apply_model()`, for fallback routing). That's the entire surface area of what Sluice is willing to touch in a request body.\n\nThe same discipline shows up in cluster routing. `ClusterRoutingStage` in `sluice_proxy\u002Fpolicy.py` carries this as an explicit design intent in its own docstring:\n\n> QoS route between pools, never utilization\u002Fload balance.\n\nRouting is a service-policy decision — Guaranteed always gets on-demand, Standard gets on-demand, Best-Effort prefers spot unless spot pressure or a health failure forces eviction — not a round-robin balancing act. And the Kubernetes deployment in `k8s\u002Fsluice-ops.yaml` is intentionally just a Deployment, a Service, a ConfigMap, and probes. No operator, no autoscaler, no scheduler extension.\n\n## 3. The Constraint: What Sluice Is Actually Responsible For\n\nBefore getting into design, it's worth stating plainly what Sluice owns, because everything downstream follows from this boundary. The clearest way to see it is through its actual interfaces.\n\nThe request path is a single catch-all:\n\n```plaintext\n\u002Fv1\u002F{path:path}\n```\n\nFor every request that hits it, Sluice does exactly seven things: resolve tenant and tier, read current telemetry, evaluate policy, apply shaping\u002Fqueueing\u002Ffallback\u002Frejection\u002Frouting, forward to the selected backend, record the result in PostgreSQL, and emit a structured log.\n\nOperational visibility is a separate, small surface: `\u002Fhealthz`, `\u002Flivez`, `\u002Freadyz`, `\u002Fmetrics`, `\u002Fstatus`, `\u002Fdecisions\u002Frecent`. And for local development, a narrow set of simulation endpoints — `PUT`\u002F`DELETE` on `\u002Fadmin\u002Fpressure`, `\u002Fadmin\u002Fqueue-depth`, `\u002Fadmin\u002Fsla-violation-rate` — let you force a signal value without waiting on real telemetry. These are intentionally minimal development controls, not a control-plane API, and they stay unauthenticated even when API-key auth is otherwise enabled on `\u002Fv1\u002F*` — a decision that's convenient for local testing and a real risk if ever exposed (more on that in Section 11).\n\nThe external dependencies are equally explicit: Prometheus for live inference telemetry, PostgreSQL for durable decisions and SLA history, Redis for distributed queue\u002Fadmission state, and any number of vLLM-compatible HTTP backends. Sluice doesn't implement any of these things — it coordinates between them.\n\n## Architecture\n\n                     REQUEST\n                        │\n                        ▼\n                ┌──────────────┐\n                │    Sluice    │\n                └──────┬───────┘\n                       │\n          ┌────────────▼────────────┐\n          │   Pressure Evaluation   │\n          │ KV Cache │ Queue │ SLA  │\n          └────────────┬────────────┘\n                       │\n                       ▼\n               ┌───────────────┐\n               │ Tenant Policy │\n               └───────┬───────┘\n                       │\n                       ▼\n               ┌───────────────┐\n               │ Action Select │\n               │ Pass\u002FDegrade  │\n               │ Queue\u002FFallback│\n               │ Reject        │\n               └───────┬───────┘\n                       │\n                       ▼\n               ┌───────────────┐\n               │ Cluster Route │\n               └───────┬───────┘\n                       │\n              ┌────────┴────────┐\n              ▼                 ▼\n        On-Demand GPU       Spot GPU\n\n## 4. Design: From Pressure to Routing\n\nSluice's decision logic didn't start as a pipeline. It started as one function.\n\nIn the first commit of the policy engine (`c101afb`), `CapacityPolicy` had a single `decide()` method with nested tier conditionals:\n\n```python\nif tier == \"Best-Effort\":\n    if pressure >= 60:\n        reject\nelif tier == \"Standard\":\n    if pressure >= 90:\n        reject\n    if pressure >= 70:\n        degrade\nelif tier == \"Guaranteed\":\n    if pressure >= 100:\n        reject\n    if pressure >= 95:\n        degrade\n```\n\nThat was fine when there was exactly one signal. It stopped being fine the moment a second one showed up. Commit `91c00fa` extended `decide()` to accept `pressure`, `queue_depth`, and `sla_violation_rate` together, and the shape it took was still recognizably one function — checking whether *any* signal crossed a reject threshold, then whether any crossed a degrade threshold. It worked, but the conditional surface was growing in a direction that wasn't going to scale to fallback routing, queueing, or multi-cluster decisions without becoming unreadable.\n\nCommit `ca80787` is where the real refactor happened: four explicit stages, each owning one concern.\n\n```python\nPressureEvaluationStage.evaluate(tier, signals) -> PressureEvaluation\nTenantPolicyStage.apply(evaluation, *, shapeable, fallback_active, fallback_model) -> PressureEvaluation\nActionSelectionStage.select(evaluation, *, shapeable, fallback_active, fallback_model) -> PolicyDecision\nClusterRoutingStage.route(tier, decision, clusters, health) -> PolicyDecision\n```\n\norchestrated by a single `PolicyPipeline.run()`. Pressure Evaluation turns raw signals into a named state (Normal, Elevated, Critical). Tenant Policy is the extension point for tier-specific semantics. Action Selection picks the actual intervention. Cluster Routing decides which pool handles it.\n\nI want to be honest about one detail here rather than present this as a clean four-stage design from the start: `TenantPolicyStage.apply()` is currently a no-op — it returns the evaluation unchanged. The refactor established the stage *boundary*, but not every piece of tier-specific behavior actually lives there yet. Some of it is still inside `CapacityPolicy._select_action()` and `_thresholds_for()`. That's a real, current gap between the architecture's intent and its full implementation — not a mistake, just unfinished separation. The pipeline shape was worth building before every piece of logic was moved into it, because it made everything that came after (fallback, queueing, multi-cluster routing) addable without another rewrite.\n\n## 5. Making Pressure Stable: State, Hysteresis, and Cooldowns\n\nThree named states drive everything downstream:\n\n```python\nPressureState.NORMAL\nPressureState.ELEVATED\nPressureState.CRITICAL\n```\n\nThe combination rule across three signals is deliberately conservative — a max-severity OR, not an average: any available signal crossing a reject threshold produces `CRITICAL`; failing that, any signal crossing a degrade threshold produces `ELEVATED`; otherwise `NORMAL`. If every signal is unavailable, the policy fails open rather than guessing.\n\nThe stabilization defaults, set in `CapacityPolicy.__init__()`:\n\n```python\nSLUICE_STATE_C = 2\nSLUICE_STATE_HYSTERESIS = 5\n```\n\nI'll say up front that these are operational defaults, not values fitted to production traces — there's no evidence in the repository of them being derived from real traffic, and that's worth being upfront about rather than implying more rigor than exists.\n\nWhat they buy you is protection against a specific, well-understood failure mode: a signal hovering right at a threshold causing the system to flip actions on every poll. `PressureStabilityModel.observe()` enforces three conditions before a transition is allowed to happen: the new state must persist for two consecutive samples, it must respect the cooldown window, and moving *down* a severity level requires crossing a lower recovery threshold than the one that triggered the escalation — not just dipping back under the original line.\n\n`tests\u002Ftest_fallback_queue.py` shows this concretely. Starting from Normal:\n\n```plaintext\n75 at t=1 -> no transition (first sample, waiting for confirmation)\n75 at t=2 -> Elevated (confirmed)\n\n69 at t=3 -> remains Elevated (below the up-threshold of 70, but still above recovery threshold of 65)\n64 at t=4 -> no transition (waiting for confirmation)\n64 at t=5 -> Normal (confirmed)\n```\n\nThe interesting line is `69 at t=3`. Without an asymmetric recovery threshold, a value that dips just under the escalation threshold would immediately flip the state back — and if pressure is oscillating around 70, that's a state flip on every poll. With the wider recovery band, 69 simply isn't low enough to count as recovery yet.\n\nI should be precise about what evidence backs this: there's no log excerpt in the repository from a real pre-hysteresis deployment flapping in production — the evidence is test-driven, not observed. The synthetic benchmark's Phase 2 run does record `action_flapping_count: 0`, but that demonstrates the *stabilized* simulator didn't flap under a controlled pressure timeline; it doesn't prove a live, unstabilized system would have.\n\nFallback gets its own tracker with its own defaults, since \"sustained\" pressure for a fallback decision is a meaningfully different bar than a degrade decision:\n\n```python\nSLUICE_FALLBACK_PRESSURE_AT = 85\nSLUICE_FALLBACK_RECOVER_BELOW = 75\nSLUICE_FALLBACK_SUSTAINED_INTERVALS = 3\nSLUICE_FALLBACK_COOLDOWN_SEC 6. The Intervention Ladder: Degrade, Queue, Fallback, Reject\n\nSluice has five possible outcomes for a request, escalating roughly with tier and pressure.\n\n**Pass** is the default — signals under threshold, a non-shapeable endpoint, or telemetry unavailable and failing open. The reason is always recorded explicitly (`below_tier_pressure_threshold`, `cache_pressure_unavailable_fail_open`, `endpoint_not_shapeable`, `signals_unavailable_fail_open`), because a \"pass\" with no explanation is just as much a policy decision as a rejection and deserves the same audit trail.\n\n**Degrade** only applies to shapeable completion endpoints, and only lowers `max_tokens` — never touches anything else in the payload:\n\n```python\nSTANDARD_MAX_TOKENS = 256\nGUARANTEED_MAX_TOKENS = 512\n```\n\n`apply_max_tokens()` is conservative by design: if `max_tokens` is missing, set it to the cap; if it's not an integer, set it to the cap; if it exceeds the cap, replace it; if it's already under the cap, leave it alone. Best-Effort has no degrade step at all — it moves straight toward rejection, since there's no meaningful \"smaller\" version of a request that's already lowest priority.\n\n**Queue** is Standard-tier only, and only fires in the Elevated state — Guaranteed never queues, and Best-Effort doesn't get access to the Standard queue. The bounds are tight on purpose:\n\n```python\nSLUICE_QUEUE_MAX_SIZE = 100\nSLUICE_QUEUE_MAX_WAIT_SEC = 0.05\n```\n\nThe local, single-replica implementation is a `BoundedRequestQueue`: acquire a lock, check size against the bound, increment, poll for release, stop at the deadline, decrement in a `finally` block so accounting can't leak. A full queue produces `queue_dropped` and a `429`; a timed-out wait produces `queue_timed_out` and a `429`. The distributed version (Section 7) replaces this with a Redis sorted set keyed by deadline, but the lifecycle events — `queue_queued`, `queue_released`, `queue_timed_out`, `queue_dropped` — stay identical either way, so the ledger doesn't care which implementation handled a given request.\n\n**Fallback** is driven by a separate `SustainedPressureTracker`. With the defaults above, pressure has to sit at or above 85% for three consecutive polling intervals before fallback activates — a single spike doesn't trigger it, because routing an entire tier to a smaller model is a heavier intervention than a temporary queue wait, and it should require more evidence. When it does trigger and a fallback model is configured, the decision comes back as:\n\n```python\nPolicyDecision(\n    Decision.FALLBACK,\n    \"sustained_pressure_routed_to_quantized_model\",\n    route_model=fallback_model,\n)\n```\n\nand the proxy rewrites the request body's `model` field before forwarding — the one other field, besides `max_tokens`, that Sluice will ever touch. Recovery requires pressure back below 75% for three intervals, mirroring the asymmetric up\u002Fdown logic from Section 5. The fallback target itself is plain config:\n\n```yaml\nfallbacks:\n  - name: quantized\n    url: http:\u002F\u002Flocalhost:8000\n    model: quantized-model\n```\n\n**Reject** returns a `429` and never contacts vLLM at all:\n\n```json\n{\n  \"error\": {\n    \"message\": \"Request rejected by Sluice due to capacity signals\",\n    \"type\": \"sluice_capacity\",\n    \"code\": \"capacity_rejected\",\n    \"reason\": \"standard_rejected_at_signal_threshold\",\n    \"tier\": \"Standard\",\n    \"pressure\": 95,\n    \"queue_depth\": 0,\n    \"sla_violation_rate\": 0\n  }\n}\n```\n\nWorth distinguishing: a *policy* rejection (capacity-driven, `sluice_capacity`) is a different thing from a *backend* failure. If no healthy backend is available at all, the response is a `503` with `type: sluice_backend, code: backend_unavailable` — the caller shouldn't have to guess whether they were shaped by policy or failed by infrastructure.\n\n## 7. Distributed Correctness: Making Admission Safe Across Replicas\n\nThe original queue was process-local, full stop. `_size` lived inside one process's memory, membership and ordering were represented by waiting coroutines in that same process, and FIFO ordering — along with any future in-flight counter — simply didn't cross replica boundaries. That's documented plainly in `DISTRIBUTED_STATE_DESIGN.md` as the actual state inventory that needed to move.\n\nThe race this creates with two replicas is the textbook check-then-act problem: replica A reads the current in-flight count and sees room below capacity; replica B reads the same count and sees the same room; both decide independently that they can admit; both increment; the shared capacity is exceeded by however many replicas raced past the check simultaneously.\n\nThere's no production incident log proving this happened — but it's exactly the race that any naive multi-replica admission scheme has, and it's the one the distributed implementation exists to prevent. The proof lives in `tests\u002Ftest_distributed_queue.py`, using an asyncio lock to model Redis's script atomicity: `test_atomic_inflight_admission_allows_only_one_replica()` spins up two coordinators with `max_inflight=1` and calls `acquire_inflight(\"Standard\")` on both concurrently. The assertion is `sorted(results) == [False, True]`. A naive read-then-write implementation would be vulnerable to both returning `True`.\n\nThe coordination primitive is Redis, but specifically **atomic Lua scripts**, not a distributed lock:\n\n- **Enqueue** — removes expired entries, checks for a duplicate request ID, checks `ZCARD` against the queue bound, adds the request to a sorted set scored by deadline.\n- **FIFO claim** — reads the first sorted-set member, verifies it matches the calling request, removes it, increments the in-flight counter.\n- **Admission** — reads the current counter, compares to capacity, increments only if there's room.\n- **Release** — decrements the counter, clamped at zero.\n\nThe reasoning for Lua over a lock: the critical section here is short and entirely key-local, and Redis executes a script atomically without introducing lock ownership, expiry semantics, or the possibility of deadlock. A lock is the right tool when you need to hold a critical section across multiple round trips; this doesn't need that.\n\nThe distributed test suite covers more than the single admission race — shared queue ordering across replicas, queue bounds enforced across replicas, timeout cleanup, and fail-open behavior when Redis itself is unreachable. That last one is a deliberate, documented tradeoff: **Redis outages fail open, which preserves request availability at the cost of strictly enforced distributed bounds.** It's not a hidden gap — it's a choice, and it's the right one for a system whose job is protecting SLA compliance, not the one whose job is refusing to ever slightly over-admit.\n\n## 8. Proving the System: Benchmark Results\n\nBefore the numbers, one correction to make against the project's own README: the narrative describes four separate workload profiles. What `benchmarks\u002Fload_test.py` actually runs is three modes — `direct`, `phase1`, `phase2` — against a single generated 600-request pressure timeline: 20% pressure at the start, a climb to 75%, a spike to 98%, then back down to 20%. It's a synthetic, deterministic simulation, and calling it four workload profiles overstates what the harness currently does. Worth fixing in the docs; worth being honest about here.\n\n**Direct mode** (no Sluice at all) is the baseline:\n\n| Tier | P50 | P95 | P99 | SLA violations |\n|---|---|---|---|---|\n| Guaranteed | 1542.9 ms | 2134.3 ms | 2134.3 ms | 30.0% |\n| Standard | 1200.0 ms | 1660.0 ms | 1660.0 ms | 60.0% |\n| Best-Effort | 771.4 ms | 1067.1 ms | 1067.1 ms | 60.0% |\n\nNo proxy overhead, and no protection — every tier bears the same overload equally.\n\n**Phase 1** (pressure-only policy) already shows the mechanism working:\n\n| Tier | P50 | SLA violations | Reject | Degrade | Overhead |\n|---|---|---|---|---|---|\n| Guaranteed | 1237.9 ms | 0.0% | 0% | 30.0% | 0.0022 ms |\n| Standard | 700.0 ms | 0.0% | 30.0% | 30.0% | 0.0022 ms |\n| Best-Effort | 450.0 ms | 0.0% | 60.0% | 0% | 0.0022 ms |\n\n**Phase 2** (full multi-signal pipeline) is the headline:\n\n| Tier | P50 | P99 | SLA violations | Reject | Fallback | Queue wait | Overhead |\n|---|---|---|---|---|---|---|---|\n| Guaranteed | 1024.5 ms | 2134.3 ms | **2.0%** | 0% | 30.0% | 0 ms | 0.0058 ms |\n| Standard | 796.8 ms | 1401.2 ms | 32.14% | 2.0% | 30.0% | 40 ms | 0.0059 ms |\n| Best-Effort | 450.0 ms | 771.4 ms | 41.61% | 31.5% | 30.0% | 0 ms | 0.0056 ms |\n\nThe headline number: **Guaranteed-tier SLA violations fell from 30.0% direct to 2.0% under Phase 2**, at a median proxy overhead of roughly 0.006 ms — negligible next to inference latency measured in seconds.\n\nThe less flattering, equally important number: Standard and Best-Effort are *not* fully protected in this profile — 32% and 42% SLA violations respectively, even under active policy. That's not a hidden failure; it's what the tradeoff actually looks like. Sluice reduces overload impact for lower tiers by design, it doesn't eliminate it, because eliminating it would mean taking protection away from the tier that's supposed to have it. The Phase 2 run also recorded `recovery_time_ms: 18000`, `state_transition_count: 9`, and `action_flapping_count: 0` — the last one being the direct evidence that the hysteresis work from Section 5 held up under this load pattern, at least in simulation.\n\n## 9. Breaking the System: Chaos and HA Evaluation\n\nIt matters to be precise about what this evaluation actually is: `evaluations\u002Fphase4_2` is a deterministic, deployment-shaped *simulator* — it models shared tier admission, replica failure, load-balancer detection delay, spot backend failure and health detection, tier routing, and tenant scale. It is not a live Kubernetes kill test against a running cluster. The results below describe how the current implementation *should* behave under these conditions, modeled faithfully against the real policy code — not a production incident record.\n\n**Replica failure.** With simulated capacities of 8\u002F12\u002F6 (Guaranteed\u002FStandard\u002FBest-Effort), a failure at 1000ms, detection at 1200ms, and measurement through 2600ms, the maximum observed in-flight counts were 7, 11, and 5 respectively — the configured capacities were never exceeded, even through the failure window.\n\n| Window | Error rate | P95 latency | SLA violations |\n|---|---|---|---|\n| Before | 0.00% | 620 ms | 0.00% |\n| During | 6.25% | 620 ms | 0.00% |\n| After | 0.00% | 620 ms | 0.00% |\n\nThe client-visible errors during the failure window came specifically from requests sent to the failed replica in the 200ms gap before detection — after detection, the surviving replica handled traffic without ever exceeding the shared capacity counters. Modeled recovery time was 1600ms from the start of the failure.\n\n**Spot-pool failure**, with the same routing rules from Section 2 (Guaranteed and Standard to on-demand, Best-Effort preferring spot):\n\n| Window | Error rate | P95 latency |\n|---|---|---|\n| Before | 0.00% | 620 ms |\n| During | 5.71% | 620 ms |\n| After | 0.00% | 620 ms |\n\nGuaranteed traffic was unaffected throughout, by construction of the routing policy. Best-Effort traffic split 36 requests to spot against 20 to on-demand across the run, with eviction becoming active 200ms after the failure and recovery converging 1200ms after detection.\n\n**Tenant scale**, at 10, 50, and 100 simulated tenants cycled across the three tiers:\n\n| Tenants | Requests | Mapping errors | Median overhead | P95 overhead |\n|---|---|---|---|---|\n| 10 | 200 | 0 | 0.0045 ms | 0.0063 ms |\n| 50 | 1000 | 0 | 0.0043 ms | 0.0050 ms |\n| 100 | 2000 | 0 | 0.0053 ms | 0.0083 ms |\n\nZero tenant-to-tier mapping errors and sub-0.01ms P95 policy overhead at 100 tenants. But it's worth being clear about what this does and doesn't test: Sluice's policy is *tier*-scoped, not *tenant*-quota-scoped — the scale test verifies that mapping and per-request overhead hold up as tenant count grows, not that individual tenants within a tier get fair treatment against each other. That's a real distinction, not a technicality.\n\n## 10. What Sluice Deliberately Isn't\n\nThe README states these as non-goals: AI-generated policies, Kubernetes operator behavior, infrastructure autoscaling, billing or tenant management, implementing an inference engine, multi-region orchestration, replacing backend schedulers, becoming an agent framework.\n\nWhat's worth adding here is that the git history backs this up as genuine restraint, not retrospective framing. There's no commit history showing an autoscaler, an operator, or a billing subsystem that was attempted and later ripped out — the boundaries were never crossed in the first place. The Kubernetes deployment stayed plain manifests from the start; there's no abandoned Helm chart or half-built operator sitting in history.\n\nThe one place that comes closest to a partial implementation is the `TenantPolicyStage` no-op from Section 4. It's worth naming honestly: that's not a disguised tenancy or billing system waiting to be finished — it's an incomplete internal separation within the policy pipeline, a structural boundary that was drawn before every piece of logic was moved across it. Different category of gap than the non-goals above, and it's called out again in the next section.\n\n## 11. Known Limitations and Engineering Judgment\n\nSome of these are already documented; several are rough edges surfaced by going through the code directly rather than the top-level docs.\n\n**The HA proof is simulated, not live.** `evaluations\u002Fphase4_2\u002Frun.py` defines its own `SharedAdmission` class — it doesn't connect to the real Redis coordinator or exercise an actual Kubernetes deployment. The report is honest about this in its own text, but it bears repeating here: this should never be presented as a production chaos-engineering result.\n\n**The Kubernetes example doesn't configure real in-flight limits.** The manifest enables Redis and queueing but leaves `SLUICE_TIER_INFLIGHT_CAPACITY` at its default of `0`, which the implementation treats as *unlimited* admission. The Section 9 simulator uses explicit capacities of 8\u002F12\u002F6 — those aren't the defaults of the actual deployed manifest. That's a meaningful gap between what's demonstrated and what's deployed by default.\n\n**The Redis queue claim accounting has a real edge case.** `CLAIM_SCRIPT` increments the Redis in-flight key whenever a queued request is claimed — but `release_inflight()` returns immediately when `max_inflight \u003C= 0`, meaning a claim can increment a counter that then never gets decremented. Since enforcement only happens when `max_inflight > 0`, this doesn't cause incorrect rejections today, but it's stale state accumulating quietly, and it needs cleanup before this could be called hardened.\n\n**Graceful shutdown is approximate, not exact.** The shutdown path sets `app.state.draining = True` and waits on active-request and queue-size counts. But the middleware's active-request count ends when the endpoint returns a `StreamingResponse` — not when the stream body actually finishes sending, which happens later in `stream_body()`. Shutdown drain accounting and true streaming completion aren't perfectly aligned. Queued requests, separately, aren't explicitly cancelled during shutdown; they ride out their normal timeout or release conditions, bounded by the shutdown grace period.\n\n**Readiness is intentionally basic.** `\u002Freadyz` checks the primary backend, PostgreSQL, and Redis — it doesn't require every fallback backend and every cluster pool to be healthy. Reasonable for failover behavior, but \"ready\" doesn't mean \"every configured dependency is healthy.\"\n\n**Metrics are minimal by design, but that has a real cost.** The dependency-free Prometheus exporter in `sluice_proxy\u002Ftelemetry.py` covers decision counters, pressure and pressure-state gauges, queue depth, and the latest proxy-overhead latency — but that latency metric is a current-value gauge, not a proper histogram with buckets or quantiles. Metrics also live in process memory, scraped per-replica, with no aggregation layer.\n\n**Admin endpoints remain unauthenticated**, even with API-key enforcement on `\u002Fv1\u002F*`. This is called out in the README, but it's worth restating plainly: it's a real deployment risk if those paths are ever exposed outside a local environment. Related: the Kubernetes API keys themselves currently live in a ConfigMap, not a Secret — a demo-appropriate shortcut, not a production one.\n\n**The ledger schema evolves via startup DDL**, not a migration system — `DecisionLedger.connect()` runs `ALTER TABLE ... ADD COLUMN IF NOT EXISTS` statements at connection time. Fine for a demo; not a substitute for a real migration lifecycle.\n\n**Unknown startup health is treated as available.** `BackendHealthRegistry` initializes every target to `None`, and routing treats health as available unless it's been explicitly marked `False`. That preserves fail-open behavior at startup, but it means a backend that hasn't been checked yet is briefly treated as usable regardless of whether it actually is.\n\n**And the benchmark narrative overstates its own workload matrix** — four profiles claimed, one deterministic timeline across three modes actually implemented, as covered in Section 8. Small, but worth fixing, and worth naming here rather than letting the discrepancy stand unaddressed.\n\n## 12. Conclusion: What the Experiment Demonstrated\n\nThe hardest problem here was never the first policy threshold — that part was genuinely simple. The difficulty was keeping behavior coherent as the system accumulated more signals, more actions, more backend pools, and eventually multiple replicas, all interacting: telemetry freshness, per-tier thresholds, hysteresis, fallback state, queue state, backend health, cluster routing, Redis admission, ledger durability, and streaming request cleanup all had to stay consistent with each other at once.\n\nThe single most consequential decision was moving admission state out of process memory and into Redis-backed coordination — preserving FIFO queue behavior and admission bounds across replicas without turning Redis into a hard availability dependency. The tradeoff landed explicitly on the side of availability: Redis outages fail open, which means request availability is preserved at the cost of strict distributed bounds during an outage. That's documented, and it's tested, not just asserted.\n\nThe second was preserving backward compatibility while the policy model grew — Phase 2's implementation still falls back to the original pressure-only decision path when no other signals are present, which is precisely what made a fair Phase 1 vs. Phase 2 benchmark comparison possible in the first place, rather than comparing two systems that had quietly become incomparable.\n\nIf this were being started over, the changes worth making are concrete, not aspirational: split `sluice_proxy\u002Fapp.py` — at 675 lines it currently owns application assembly, lifecycle, middleware, routing, policy integration, queue integration, admin endpoints, health, and metrics all at once. Make `TenantPolicyStage` a real stage instead of a no-op. Configure tier in-flight capacities from the start rather than shipping the reference deployment at unlimited. Build the benchmark harness around one canonical event model that can run both deterministic simulation and live HTTP\u002FKubernetes tests, instead of the two diverging as they have. Add integration tests against real Redis and real PostgreSQL, not just their local doubles. Replace the latest-value latency gauge with real histograms. Move API keys into Secrets and protect admin routes independently from inference auth. And close the Redis queue\u002Fin-flight accounting edge case before calling the distributed admission path hardened.\n\nThere's no evidence in the git history of major dead ends or discarded architectures — the evolution reads as almost entirely additive: a single pressure-threshold function, extended to multi-signal policy, refactored into explicit pipeline stages, extended again with fallback, queueing, and cluster routing, backed by a reliability evaluation harness, made distributed with Redis, wrapped in operational deployment tooling, and finally stress-tested against simulated failure and scale.\n\nThat progression is what makes the most honest description of what this project actually is:\n\n> Sluice is a deliberately narrow, explainable capacity-governance control plane whose strongest contribution is the integration of tier policy, live inference telemetry, request shaping, fallback routing, distributed admission, and durable decision evidence — not a new inference scheduler, not an autoscaler, and not a hardened production platform. What the benchmarks and chaos evaluation demonstrate is that the mechanism works as designed and holds up under simulated contention and failure. What they don't demonstrate — and what the limitations section says plainly — is that it's been proven against real production traffic yet. That's the next honest claim to go earn, not the one to imply already having.\n\n---\n\n> 🔗 **Nguồn bài viết gốc**: [Madhav M S](https:\u002F\u002Fdev.to\u002Fvampiriccyborg\u002Fbuilding-sluice-qos-aware-capacity-governance-for-self-hosted-llm-inference-13ja)\n","Building Sluice: QoS-Aware Capacity Governance for Self-Hosted LLM Inference - Intlight Insights","https:\u002F\u002Fintlighttech.com\u002Fblogs\u002Fbuilding-sluice-qos-aware-capacity-governance-for-self-hosted-llm-inference-flbu",{"items":29,"total":356,"page":357,"totalPages":357,"limit":358,"lang":10,"categories":359,"popularTags":373},[30,48,64,80,94,108,120,133,147,149,163,175,188,200,214,227,241,253,267,280,294,307,319,332,344],{"id":31,"title":32,"slug":33,"lang":10,"category":34,"categorySlug":35,"summary":36,"excerpt":36,"author":37,"date":15,"readTime":38,"image":39,"tags":40,"publishedAt":45,"createdAt":45,"updatedAt":45,"filePath":46,"sourceUrl":47},"cron-1786956218725-en","I Thought I'd Lost the Plot. I Was Writing It.","i-thought-id-lost-the-plot-i-was-writing-it-wjvz","AI Agents","ai-agents","I Thought I'd Lost the Plot. I Was Writing It.            I set out to build autonomous...","Joe Black","6 min read","https:\u002F\u002Fmedia2.dev.to\u002Fdynamic\u002Fimage\u002Fwidth=1200,height=627,fit=cover,gravity=auto,format=auto\u002Fhttps%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fa3l9h76kr920p8bm2mjp.png",[34,41,42,43,44],"claudecode","aiagents","developmenttools","autonomousagents","2026-08-17T08:43:38.725Z","\u002FUsers\u002Fnguyenanhtuan\u002FCode\u002FNuxtjs\u002Fnextpress\u002Fintlight\u002Fpublic\u002Fcontents\u002Fai-agents\u002Fi-thought-id-lost-the-plot-i-was-writing-it-wjvz.md","https:\u002F\u002Fdev.to\u002Fjoeblackwaslike\u002Fi-thought-id-lost-the-plot-i-was-writing-it-5fil",{"id":49,"title":50,"slug":51,"lang":10,"category":52,"categorySlug":53,"summary":54,"excerpt":54,"author":55,"date":15,"readTime":38,"image":56,"tags":57,"publishedAt":61,"createdAt":61,"updatedAt":61,"filePath":62,"sourceUrl":63},"cron-1786956216746-en","What Is the Circuit Breaker Pattern? A Practical Guide","what-is-the-circuit-breaker-pattern-a-practical-guide-4j68","Microservices","microservices","What Is the Circuit Breaker Pattern? A Practical Guide for Developers   Imagine your...","Avijit Bera","https:\u002F\u002Fmedia2.dev.to\u002Fdynamic\u002Fimage\u002Fwidth=1000,height=420,fit=cover,gravity=auto,format=auto\u002Fhttps%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fi6cy1uyqxjd14ikkveev.png",[52,53,58,59,60],"backend","api","Trending","2026-08-17T08:43:36.746Z","\u002FUsers\u002Fnguyenanhtuan\u002FCode\u002FNuxtjs\u002Fnextpress\u002Fintlight\u002Fpublic\u002Fcontents\u002Fmicroservices\u002Fwhat-is-the-circuit-breaker-pattern-a-practical-guide-4j68.md","https:\u002F\u002Fdev.to\u002Favijitbera\u002Fwhat-is-the-circuit-breaker-pattern-a-practical-guide-20i4",{"id":65,"title":66,"slug":67,"lang":10,"category":68,"categorySlug":69,"summary":70,"excerpt":70,"author":71,"date":15,"readTime":38,"image":72,"tags":73,"publishedAt":77,"createdAt":77,"updatedAt":77,"filePath":78,"sourceUrl":79},"cron-1786956214008-en","I attacked my own npm package before launching it. It let the proposer approve their own writes","i-attacked-my-own-npm-package-before-launching-it-it-let-the-proposer-approve-their-own-writes-oi6y","Security","security","My library exists so a human approves an LLM's UPDATE before it runs. It never checked that the approver was somebody other than the proposer — and wrote \\\"approved\\\" into the audit trail anyway.","hyuga","https:\u002F\u002Fmedia2.dev.to\u002Fdynamic\u002Fimage\u002Fwidth=1200,height=627,fit=cover,gravity=auto,format=auto\u002Fhttps%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F65yv5n4qvrgn3t6ot4gy.png",[68,74,75,69,76],"opensource","ai","database","2026-08-17T08:43:34.007Z","\u002FUsers\u002Fnguyenanhtuan\u002FCode\u002FNuxtjs\u002Fnextpress\u002Fintlight\u002Fpublic\u002Fcontents\u002Fsecurity\u002Fi-attacked-my-own-npm-package-before-launching-it-it-let-the-proposer-approve-their-own-writes-oi6y.md","https:\u002F\u002Fdev.to\u002Fhyuga611\u002Fi-attacked-my-own-npm-package-before-launching-it-it-let-the-proposer-approve-their-own-writes-4mki",{"id":81,"title":82,"slug":83,"lang":10,"category":11,"categorySlug":12,"summary":84,"excerpt":84,"author":85,"date":15,"readTime":38,"image":86,"tags":87,"publishedAt":91,"createdAt":91,"updatedAt":91,"filePath":92,"sourceUrl":93},"cron-1786956210950-en","Build an MCP Server in Go (Part 1): Designing a diagnostic-grade Kubernetes client","build-an-mcp-server-in-go-part-1-designing-a-diagnostic-grade-kubernetes-client-p1d4","This post designs the Kubernetes client. The next post wraps it as an MCP server and wires it to an...","Fer Rios","https:\u002F\u002Fmedia2.dev.to\u002Fdynamic\u002Fimage\u002Fwidth=1000,height=420,fit=cover,gravity=auto,format=auto\u002Fhttps%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fpw4b055hh8hxl5unrmvv.png",[11,12,88,89,90],"go","devops","mcp","2026-08-17T08:43:30.949Z","\u002FUsers\u002Fnguyenanhtuan\u002FCode\u002FNuxtjs\u002Fnextpress\u002Fintlight\u002Fpublic\u002Fcontents\u002Fkubernetes\u002Fbuild-an-mcp-server-in-go-part-1-designing-a-diagnostic-grade-kubernetes-client-p1d4.md","https:\u002F\u002Fdev.to\u002Fferztyle\u002Fbuild-an-mcp-server-in-go-part-1-designing-a-diagnostic-grade-kubernetes-client-49a2",{"id":95,"title":96,"slug":97,"lang":10,"category":34,"categorySlug":35,"summary":98,"excerpt":98,"author":99,"date":15,"readTime":38,"image":100,"tags":101,"publishedAt":105,"createdAt":105,"updatedAt":105,"filePath":106,"sourceUrl":107},"cron-1786956117904-en","The Write Policy Is the Hard Part: Promotion Pipelines for Agent Memory","the-write-policy-is-the-hard-part-promotion-pipelines-for-agent-memory-4qv9","Storing agent memory is easy. Deciding what earns a permanent write, and keeping the write-path alive through RBAC and network policy, is the real work.","Guatu","https:\u002F\u002Fmedia2.dev.to\u002Fdynamic\u002Fimage\u002Fwidth=1000,height=420,fit=cover,gravity=auto,format=auto\u002Fhttps%3A%2F%2Fguatulabs.dev%2Fog%2Fthe-write-policy-is-the-hard-part-promotion-pipelines-for-agent-memory.png",[34,42,102,103,104],"agentmemory","rbac","networkpolicies","2026-08-17T08:41:57.903Z","\u002FUsers\u002Fnguyenanhtuan\u002FCode\u002FNuxtjs\u002Fnextpress\u002Fintlight\u002Fpublic\u002Fcontents\u002Fai-agents\u002Fthe-write-policy-is-the-hard-part-promotion-pipelines-for-agent-memory-4qv9.md","https:\u002F\u002Fdev.to\u002Ffuthgar\u002Fthe-write-policy-is-the-hard-part-promotion-pipelines-for-agent-memory-5mc",{"id":109,"title":110,"slug":111,"lang":10,"category":34,"categorySlug":35,"summary":112,"excerpt":112,"author":113,"date":15,"readTime":16,"image":114,"tags":115,"publishedAt":117,"createdAt":117,"updatedAt":117,"filePath":118,"sourceUrl":119},"cron-1786955347611","I Changed How I Think About AI Memory","i-changed-how-i-think-about-ai-memory-fnpm","I Changed How I Think About AI Memory   When I first built Lean AI Memory, I focused too...","Phúc Phùng","https:\u002F\u002Fmedia2.dev.to\u002Fdynamic\u002Fimage\u002Fwidth=1200,height=627,fit=cover,gravity=auto,format=auto\u002Fhttps%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0oqod5zshdan74573bau.png",[34,75,42,74,116],"git","2026-08-17T08:29:07.610Z","\u002FUsers\u002Fnguyenanhtuan\u002FCode\u002FNuxtjs\u002Fnextpress\u002Fintlight\u002Fpublic\u002Fcontents\u002Fai-agents\u002Fi-changed-how-i-think-about-ai-memory-fnpm.md","https:\u002F\u002Fdev.to\u002Fphucphungbk\u002Fi-changed-how-i-think-about-ai-memory-4mkd",{"id":121,"title":122,"slug":123,"lang":10,"category":52,"categorySlug":53,"summary":124,"excerpt":124,"author":125,"date":15,"readTime":16,"image":126,"tags":127,"publishedAt":130,"createdAt":130,"updatedAt":130,"filePath":131,"sourceUrl":132},"cron-1786955347443","Real-Life Refactoring Example: ~3x Less Code to Read","real-life-refactoring-example-3x-less-code-to-read-dccm","There is a popular idea that refactoring is making code shorter. It is not entirely wrong....","Valentine Shi","https:\u002F\u002Fmedia2.dev.to\u002Fdynamic\u002Fimage\u002Fwidth=1200,height=627,fit=cover,gravity=auto,format=auto\u002Fhttps%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Foubl9dlj23byhsak2mqy.png",[52,128,129,58,53],"node","software","2026-08-17T08:29:07.443Z","\u002FUsers\u002Fnguyenanhtuan\u002FCode\u002FNuxtjs\u002Fnextpress\u002Fintlight\u002Fpublic\u002Fcontents\u002Fmicroservices\u002Freal-life-refactoring-example-3x-less-code-to-read-dccm.md","https:\u002F\u002Fdev.to\u002Fvalentineshi-dev\u002Freal-life-refactoring-example-3x-less-code-to-read-3mdl",{"id":134,"title":135,"slug":136,"lang":10,"category":68,"categorySlug":69,"summary":137,"excerpt":137,"author":138,"date":15,"readTime":16,"image":139,"tags":140,"publishedAt":144,"createdAt":144,"updatedAt":144,"filePath":145,"sourceUrl":146},"cron-1786955347034","The Tragedy of the Clean-Handed Auditor","the-tragedy-of-the-clean-handed-auditor-rgoz","\\\"I could save them if they'd only listen...\\\"  Hey, you. Yeah, you: the compliance or governance...","Ben Link","https:\u002F\u002Fmedia2.dev.to\u002Fdynamic\u002Fimage\u002Fwidth=1000,height=420,fit=cover,gravity=auto,format=auto\u002Fhttps%3A%2F%2Fdev-to-uploads.s3.amazonaws.com%2Fuploads%2Farticles%2F5z426h3tt6e2b3sthd2f.png",[68,69,141,142,143],"compliance","developers","careerdevelopment","2026-08-17T08:29:07.034Z","\u002FUsers\u002Fnguyenanhtuan\u002FCode\u002FNuxtjs\u002Fnextpress\u002Fintlight\u002Fpublic\u002Fcontents\u002Fsecurity\u002Fthe-tragedy-of-the-clean-handed-auditor-rgoz.md","https:\u002F\u002Fdev.to\u002Flinkbenjamin\u002Fthe-tragedy-of-the-clean-handed-auditor-1253",{"id":7,"title":8,"slug":9,"lang":10,"category":11,"categorySlug":12,"summary":13,"excerpt":13,"author":14,"date":15,"readTime":16,"image":17,"tags":148,"publishedAt":22,"createdAt":22,"updatedAt":22,"filePath":23,"sourceUrl":24},[11,19,20,12,21],{"id":150,"title":151,"slug":152,"lang":10,"category":34,"categorySlug":35,"summary":153,"excerpt":153,"author":154,"date":15,"readTime":16,"image":155,"tags":156,"publishedAt":160,"createdAt":160,"updatedAt":160,"filePath":161,"sourceUrl":162},"cron-1786954879160","Test What Your AI Agents Must Not Do","test-what-your-ai-agents-must-not-do-fj6y","A Guardrail Without A Negative Test Is Still An Assumption   Most AI agent governance starts...","Bobai Kato","https:\u002F\u002Fmedia2.dev.to\u002Fdynamic\u002Fimage\u002Fwidth=1000,height=420,fit=cover,gravity=auto,format=auto\u002Fhttps%3A%2F%2Fres.cloudinary.com%2Fota-run%2Fimage%2Fupload%2Fq_auto%2Ftest-what-your-ai-agents-must-not-do.png",[34,42,157,158,159],"agentsafety","negativetesting","executiongovernance","2026-08-17T08:21:19.160Z","\u002FUsers\u002Fnguyenanhtuan\u002FCode\u002FNuxtjs\u002Fnextpress\u002Fintlight\u002Fpublic\u002Fcontents\u002Fai-agents\u002Ftest-what-your-ai-agents-must-not-do-fj6y.md","https:\u002F\u002Fdev.to\u002Fotaready\u002Ftest-what-your-ai-agents-must-not-do-3e1a",{"id":164,"title":165,"slug":166,"lang":10,"category":52,"categorySlug":53,"summary":167,"excerpt":167,"author":168,"date":15,"readTime":16,"image":169,"tags":170,"publishedAt":172,"createdAt":172,"updatedAt":172,"filePath":173,"sourceUrl":174},"cron-1786954878754","Protecting Microservices: Implementing End-to-End Encryption Across REST APIs","protecting-microservices-implementing-end-to-end-encryption-across-rest-apis-gtfz","End-to-end encryption across REST APIs is the difference between a microservices architecture that...","Fu'ad Husnan","https:\u002F\u002Fmedia2.dev.to\u002Fdynamic\u002Fimage\u002Fwidth=1000,height=420,fit=cover,gravity=auto,format=auto\u002Fhttps%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fba3onadsxu08gb14brer.png",[52,59,53,60,171],"2026","2026-08-17T08:21:18.754Z","\u002FUsers\u002Fnguyenanhtuan\u002FCode\u002FNuxtjs\u002Fnextpress\u002Fintlight\u002Fpublic\u002Fcontents\u002Fmicroservices\u002Fprotecting-microservices-implementing-end-to-end-encryption-across-rest-apis-gtfz.md","https:\u002F\u002Fdev.to\u002Ffuadhusnan_f44f3e13\u002Fprotecting-microservices-implementing-end-to-end-encryption-across-rest-apis-26hb",{"id":176,"title":177,"slug":178,"lang":10,"category":68,"categorySlug":69,"summary":179,"excerpt":179,"author":180,"date":15,"readTime":16,"image":181,"tags":182,"publishedAt":185,"createdAt":185,"updatedAt":185,"filePath":186,"sourceUrl":187},"cron-1786954878272","I Gave My Agent One Signed Permission It Couldn’t Mint Itself","i-gave-my-agent-one-signed-permission-it-couldnt-mint-itself-nm1o","Evidence status. The supervised operator run completed on 2026-08-09. An operator-signed job...","Self-Correcting Systems","https:\u002F\u002Fmedia2.dev.to\u002Fdynamic\u002Fimage\u002Fwidth=1000,height=420,fit=cover,gravity=auto,format=auto\u002Fhttps%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fg1gjbzx4muvma5fznpjl.png",[68,183,89,69,184],"machinelearning","agents","2026-08-17T08:21:18.271Z","\u002FUsers\u002Fnguyenanhtuan\u002FCode\u002FNuxtjs\u002Fnextpress\u002Fintlight\u002Fpublic\u002Fcontents\u002Fsecurity\u002Fi-gave-my-agent-one-signed-permission-it-couldnt-mint-itself-nm1o.md","https:\u002F\u002Fdev.to\u002Fkenielzep97\u002Fi-gave-my-agent-one-signed-permission-it-couldnt-mint-itself-2lpc",{"id":189,"title":190,"slug":191,"lang":10,"category":11,"categorySlug":12,"summary":192,"excerpt":192,"author":193,"date":15,"readTime":16,"image":194,"tags":195,"publishedAt":197,"createdAt":197,"updatedAt":197,"filePath":198,"sourceUrl":199},"cron-1786954877848","One GPU, four ways to share it: ten scenarios, and the headline finding I had to retract","one-gpu-four-ways-to-share-it-ten-scenarios-and-the-headline-finding-i-had-to-retract-3y1v","I measured every GPU sharing mode across ten scenarios, published a headline finding that duplicate models are nearly free on unified memory, then failed to replicate it and retracted it. Here is what the controlled replication showed and how the original measurement fooled me.","Christopher Maher","https:\u002F\u002Fmedia2.dev.to\u002Fdynamic\u002Fimage\u002Fwidth=1000,height=420,fit=cover,gravity=auto,format=auto\u002Fhttps%3A%2F%2Fllmkube.com%2Fog-gpu-sharing-four-ways-devto.png",[11,12,196,75,89],"gpu","2026-08-17T08:21:17.847Z","\u002FUsers\u002Fnguyenanhtuan\u002FCode\u002FNuxtjs\u002Fnextpress\u002Fintlight\u002Fpublic\u002Fcontents\u002Fkubernetes\u002Fone-gpu-four-ways-to-share-it-ten-scenarios-and-the-headline-finding-i-had-to-retract-3y1v.md","https:\u002F\u002Fdev.to\u002Fdefilan\u002Fone-gpu-four-ways-to-share-it-ten-scenarios-and-the-one-number-that-inverts-on-your-hardware-1bih",{"id":201,"title":202,"slug":203,"lang":10,"category":34,"categorySlug":35,"summary":204,"excerpt":204,"author":99,"date":205,"readTime":16,"image":206,"tags":207,"publishedAt":211,"createdAt":211,"updatedAt":211,"filePath":212,"sourceUrl":213},"cron-1786954563310","FastMCP Agent Mail: RBAC Tokens vs Anonymous Access, and the 403 Errors in Between","fastmcp-agent-mail-rbac-tokens-vs-anonymous-access-and-the-403-errors-in-between-ef0p","Why a FastMCP agent mail server that works anonymously in dev returns 403 behind TLS ingress, and how to wire bearer tokens without leaking them.","17\u002F8\u002F2026","https:\u002F\u002Fmedia2.dev.to\u002Fdynamic\u002Fimage\u002Fwidth=1000,height=420,fit=cover,gravity=auto,format=auto\u002Fhttps%3A%2F%2Fguatulabs.dev%2Fog%2Ffastmcp-agent-mail-rbac-token-vs-anonymous-lessons-from-403-errors.png",[34,208,209,42,210],"fastmcp","mcpservers","authentication","2026-08-17T08:16:03.310Z","\u002FUsers\u002Fnguyenanhtuan\u002FCode\u002FNuxtjs\u002Fnextpress\u002Fintlight\u002Fpublic\u002Fcontents\u002Fai-agents\u002Ffastmcp-agent-mail-rbac-tokens-vs-anonymous-access-and-the-403-errors-in-between-ef0p.md","https:\u002F\u002Fdev.to\u002Ffuthgar\u002Ffastmcp-agent-mail-rbac-tokens-vs-anonymous-access-and-the-403-errors-in-between-22pk",{"id":215,"title":216,"slug":217,"lang":10,"category":52,"categorySlug":53,"summary":218,"excerpt":218,"author":219,"date":205,"readTime":16,"image":220,"tags":221,"publishedAt":224,"createdAt":224,"updatedAt":224,"filePath":225,"sourceUrl":226},"cron-1786954562962","eBPF-Powered Request Tracing in Go Microservices Without Instrumentation Tax","ebpf-powered-request-tracing-in-go-microservices-without-instrumentation-tax-k53p","How eBPF uprobes and ring buffers replace manual trace propagation in Go services—mechanics, tradeoffs, and failure modes.","Neeraj Singhi","https:\u002F\u002Fmedia2.dev.to\u002Fdynamic\u002Fimage\u002Fwidth=1200,height=627,fit=cover,gravity=auto,format=auto\u002Fhttps%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fws515jitp8rvalxr8d63.png",[52,222,88,53,223],"architecture","performance","2026-08-17T08:16:02.962Z","\u002FUsers\u002Fnguyenanhtuan\u002FCode\u002FNuxtjs\u002Fnextpress\u002Fintlight\u002Fpublic\u002Fcontents\u002Fmicroservices\u002Febpf-powered-request-tracing-in-go-microservices-without-instrumentation-tax-k53p.md","https:\u002F\u002Fdev.to\u002Fneeraj_singhi_golang\u002Febpf-powered-request-tracing-in-go-microservices-without-instrumentation-tax-34kf",{"id":228,"title":229,"slug":230,"lang":10,"category":68,"categorySlug":69,"summary":231,"excerpt":231,"author":232,"date":205,"readTime":16,"image":233,"tags":234,"publishedAt":238,"createdAt":238,"updatedAt":238,"filePath":239,"sourceUrl":240},"cron-1786954562825","How Dopamine Works: The Architecture of a Modern iOS Jailbreak","how-dopamine-works-the-architecture-of-a-modern-ios-jailbreak-woxq","Most developers will never jailbreak a phone. That is fine. This article is not a how-to, and there...","ArshTechPro","https:\u002F\u002Fmedia2.dev.to\u002Fdynamic\u002Fimage\u002Fwidth=1000,height=420,fit=cover,gravity=auto,format=auto\u002Fhttps%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F0gnfjvl3fzvb1r1rpvci.png",[68,235,236,69,237],"ios","mobile","programming","2026-08-17T08:16:02.825Z","\u002FUsers\u002Fnguyenanhtuan\u002FCode\u002FNuxtjs\u002Fnextpress\u002Fintlight\u002Fpublic\u002Fcontents\u002Fsecurity\u002Fhow-dopamine-works-the-architecture-of-a-modern-ios-jailbreak-woxq.md","https:\u002F\u002Fdev.to\u002Farshtechpro\u002Fhow-dopamine-works-the-architecture-of-a-modern-ios-jailbreak-2hj3",{"id":242,"title":243,"slug":244,"lang":10,"category":11,"categorySlug":12,"summary":245,"excerpt":245,"author":246,"date":205,"readTime":16,"image":247,"tags":248,"publishedAt":250,"createdAt":250,"updatedAt":250,"filePath":251,"sourceUrl":252},"cron-1786954562468","I got tired of SSHing into 10 VMs a day, so I built a live map of my whole infrastructure","i-got-tired-of-sshing-into-10-vms-a-day-so-i-built-a-live-map-of-my-whole-infrastructure-iqz9","Every day at work looked the same. Something breaks, or I need to push a new image, and I'm SSHing...","ByteStrix","https:\u002F\u002Fmedia2.dev.to\u002Fdynamic\u002Fimage\u002Fwidth=1200,height=627,fit=cover,gravity=auto,format=auto\u002Fhttps%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4xdxpc7fsu7nv8usri7a.png",[11,249,89,74,12],"productivity","2026-08-17T08:16:02.468Z","\u002FUsers\u002Fnguyenanhtuan\u002FCode\u002FNuxtjs\u002Fnextpress\u002Fintlight\u002Fpublic\u002Fcontents\u002Fkubernetes\u002Fi-got-tired-of-sshing-into-10-vms-a-day-so-i-built-a-live-map-of-my-whole-infrastructure-iqz9.md","https:\u002F\u002Fdev.to\u002Fbytestrix\u002Fi-got-tired-of-sshing-into-10-vms-a-day-so-i-built-a-live-map-of-my-whole-infrastructure-2iil",{"id":254,"title":255,"slug":256,"lang":10,"category":34,"categorySlug":35,"summary":257,"excerpt":257,"author":258,"date":205,"readTime":16,"image":259,"tags":260,"publishedAt":264,"createdAt":264,"updatedAt":264,"filePath":265,"sourceUrl":266},"cron-1786954558132","The Coordinated Rename Is the Agent's Most Dangerous Refactor","the-coordinated-rename-is-the-agents-most-dangerous-refactor-iazu","Multi-agent rename tooling rewrites two hundred files in ten seconds because it noticed the drift. Half the time the drift was a load-bearing distinction the team encoded on purpose. Vocabulary curation is a real-time review surface now, and senior includes refusing changes that would be technically more consistent because the domain has two concepts the agent has no way to see.","Travis Frisinger","https:\u002F\u002Fmedia2.dev.to\u002Fdynamic\u002Fimage\u002Fwidth=1000,height=420,fit=cover,gravity=auto,format=auto\u002Fhttps%3A%2F%2Fwww.tddbuddy.com%2Fimages%2Fcovers%2Fthe-coordinated-rename-is-the-dangerous-refactor.png",[34,261,42,262,263],"vocabulary","domainmodeling","codereview","2026-08-17T08:15:58.132Z","\u002FUsers\u002Fnguyenanhtuan\u002FCode\u002FNuxtjs\u002Fnextpress\u002Fintlight\u002Fpublic\u002Fcontents\u002Fai-agents\u002Fthe-coordinated-rename-is-the-agents-most-dangerous-refactor-iazu.md","https:\u002F\u002Fdev.to\u002Ftmfrisinger\u002Fthe-coordinated-rename-is-the-agents-most-dangerous-refactor-6a",{"id":268,"title":269,"slug":270,"lang":10,"category":52,"categorySlug":53,"summary":271,"excerpt":271,"author":272,"date":205,"readTime":16,"image":273,"tags":274,"publishedAt":277,"createdAt":277,"updatedAt":277,"filePath":278,"sourceUrl":279},"cron-1786954557526","Microservices: Building Applications as Independent, Communicating Services","microservices-building-applications-as-independent-communicating-services-c45k","Microservices: Building Applications as Independent, Communicating Services   A practical,...","Rhuturaj Takle","https:\u002F\u002Fmedia2.dev.to\u002Fdynamic\u002Fimage\u002Fwidth=1000,height=420,fit=cover,gravity=auto,format=auto\u002Fhttps%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F8puio1c7flrtyivr04hl.png",[52,53,275,237,276],"dotnet","learning","2026-08-17T08:15:57.526Z","\u002FUsers\u002Fnguyenanhtuan\u002FCode\u002FNuxtjs\u002Fnextpress\u002Fintlight\u002Fpublic\u002Fcontents\u002Fmicroservices\u002Fmicroservices-building-applications-as-independent-communicating-services-c45k.md","https:\u002F\u002Fdev.to\u002Frhuturaj_takle\u002Fmicroservices-building-applications-as-independent-communicating-services-2eo8",{"id":281,"title":282,"slug":283,"lang":10,"category":68,"categorySlug":69,"summary":284,"excerpt":284,"author":285,"date":205,"readTime":16,"image":286,"tags":287,"publishedAt":291,"createdAt":291,"updatedAt":291,"filePath":292,"sourceUrl":293},"cron-1786954556056","From Arduino To Automotive: How I Escaped The IDE And Owned The Bus","from-arduino-to-automotive-how-i-escaped-the-ide-and-owned-the-bus-hj7g","Arduino taught me how to build. Bare metal taught me how the build actually works.  I have a lot of...","v. Splicer","https:\u002F\u002Fmedia2.dev.to\u002Fdynamic\u002Fimage\u002Fwidth=1000,height=420,fit=cover,gravity=auto,format=auto\u002Fhttps%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fabn3138rnp9vm0nyk54b.jpg",[68,288,289,290,69],"esp32","arduino","canbus","2026-08-17T08:15:56.056Z","\u002FUsers\u002Fnguyenanhtuan\u002FCode\u002FNuxtjs\u002Fnextpress\u002Fintlight\u002Fpublic\u002Fcontents\u002Fsecurity\u002Ffrom-arduino-to-automotive-how-i-escaped-the-ide-and-owned-the-bus-hj7g.md","https:\u002F\u002Fdev.to\u002Fnumbpill3d\u002Ffrom-arduino-to-automotive-how-i-escaped-the-ide-and-owned-the-bus-f8f",{"id":295,"title":296,"slug":297,"lang":10,"category":11,"categorySlug":12,"summary":298,"excerpt":298,"author":299,"date":205,"readTime":16,"image":300,"tags":301,"publishedAt":304,"createdAt":304,"updatedAt":304,"filePath":305,"sourceUrl":306},"cron-1786954555663","The Backup Awakens: A Star Wars Story","the-backup-awakens-a-star-wars-story-jzpq","The Quest Begins (The \\\"Why\\\")   Honestly, I used to think backups were the boring chores you...","Timevolt","https:\u002F\u002Fmedia2.dev.to\u002Fdynamic\u002Fimage\u002Fwidth=1200,height=627,fit=cover,gravity=auto,format=auto\u002Fhttps%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F4ynsfxiz14nn4b9ylhn6.png",[11,89,302,12,303],"docker","cicd","2026-08-17T08:15:55.662Z","\u002FUsers\u002Fnguyenanhtuan\u002FCode\u002FNuxtjs\u002Fnextpress\u002Fintlight\u002Fpublic\u002Fcontents\u002Fkubernetes\u002Fthe-backup-awakens-a-star-wars-story-jzpq.md","https:\u002F\u002Fdev.to\u002Ftimevolt\u002Fthe-backup-awakens-a-star-wars-story-1616",{"id":308,"title":309,"slug":310,"lang":10,"category":34,"categorySlug":35,"summary":311,"excerpt":311,"author":258,"date":205,"readTime":16,"image":312,"tags":313,"publishedAt":316,"createdAt":316,"updatedAt":316,"filePath":317,"sourceUrl":318},"cron-1786954321408","Test Deletion Is a Privileged Operation","test-deletion-is-a-privileged-operation-2pfz","The cheapest way for an agent to make a failing test pass is to delete it. That is logical for the agent and catastrophic for the codebase. Tests are append-only by default. Deletion needs a human author, a separate commit, and a separate review.","https:\u002F\u002Fmedia2.dev.to\u002Fdynamic\u002Fimage\u002Fwidth=1000,height=420,fit=cover,gravity=auto,format=auto\u002Fhttps%3A%2F%2Fwww.tddbuddy.com%2Fimages%2Fcovers%2Ftest-deletion-is-a-privileged-operation.png",[34,314,42,315,263],"tdd","testdesign","2026-08-17T08:12:01.408Z","\u002FUsers\u002Fnguyenanhtuan\u002FCode\u002FNuxtjs\u002Fnextpress\u002Fintlight\u002Fpublic\u002Fcontents\u002Fai-agents\u002Ftest-deletion-is-a-privileged-operation-2pfz.md","https:\u002F\u002Fdev.to\u002Ftmfrisinger\u002Ftest-deletion-is-a-privileged-operation-264a",{"id":320,"title":321,"slug":322,"lang":10,"category":52,"categorySlug":53,"summary":323,"excerpt":323,"author":324,"date":205,"readTime":16,"image":325,"tags":326,"publishedAt":329,"createdAt":329,"updatedAt":329,"filePath":330,"sourceUrl":331},"cron-1786954321239","You Don't Always Need a Workflow Engine to Roll Back a Failed Checkout","you-dont-always-need-a-workflow-engine-to-roll-back-a-failed-checkout-1iwf","Here's a sequence that shows up in almost every Laravel app that talks to the outside world:   Charge...","Sient","https:\u002F\u002Fmedia2.dev.to\u002Fdynamic\u002Fimage\u002Fwidth=1000,height=420,fit=cover,gravity=auto,format=auto\u002Fhttps%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fu6g3j4z4af9tbuozbz2v.png",[52,327,328,222,53],"laravel","php","2026-08-17T08:12:01.239Z","\u002FUsers\u002Fnguyenanhtuan\u002FCode\u002FNuxtjs\u002Fnextpress\u002Fintlight\u002Fpublic\u002Fcontents\u002Fmicroservices\u002Fyou-dont-always-need-a-workflow-engine-to-roll-back-a-failed-checkout-1iwf.md","https:\u002F\u002Fdev.to\u002Fsient\u002Fyou-dont-always-need-a-workflow-engine-to-roll-back-a-failed-checkout-5gop",{"id":333,"title":334,"slug":335,"lang":10,"category":68,"categorySlug":69,"summary":336,"excerpt":336,"author":337,"date":205,"readTime":16,"image":338,"tags":339,"publishedAt":341,"createdAt":341,"updatedAt":341,"filePath":342,"sourceUrl":343},"cron-1786954321092","I Stopped Trusting AI Agents With Tools. So I Built a Gatekeeper.","i-stopped-trusting-ai-agents-with-tools-so-i-built-a-gatekeeper-9i2m","Update 08\u002F15 0.2.0 Released   github.com\u002Fdeghosal-2026\u002Fagent-tooltrust · pip install agent-tooltrust...","Debashish Ghosal","https:\u002F\u002Fmedia2.dev.to\u002Fdynamic\u002Fimage\u002Fwidth=1000,height=420,fit=cover,gravity=auto,format=auto\u002Fhttps%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fr97gsrqar0qk7ejjibih.png",[68,75,184,69,340],"gatekeeper","2026-08-17T08:12:01.092Z","\u002FUsers\u002Fnguyenanhtuan\u002FCode\u002FNuxtjs\u002Fnextpress\u002Fintlight\u002Fpublic\u002Fcontents\u002Fsecurity\u002Fi-stopped-trusting-ai-agents-with-tools-so-i-built-a-gatekeeper-9i2m.md","https:\u002F\u002Fdev.to\u002Fdebashish_ghosal\u002Fi-stopped-trusting-ai-agents-with-tools-so-i-built-a-gatekeeper-26fb",{"id":345,"title":346,"slug":347,"lang":10,"category":11,"categorySlug":12,"summary":348,"excerpt":348,"author":349,"date":205,"readTime":16,"image":350,"tags":351,"publishedAt":353,"createdAt":353,"updatedAt":353,"filePath":354,"sourceUrl":355},"cron-1786954320913","I Automated My Entire GitOps Security Stack. The First Thing It Blocked Was My Own Salary.","i-automated-my-entire-gitops-security-stack-the-first-thing-it-blocked-was-my-own-salary-pko8","I Automated My Entire GitOps Security Stack. The First Thing It Blocked Was My Own...","Le Beltagy","https:\u002F\u002Fmedia2.dev.to\u002Fdynamic\u002Fimage\u002Fwidth=1000,height=420,fit=cover,gravity=auto,format=auto\u002Fhttps%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Frvjfi7ee0tmi1xp1mem5.png",[11,12,69,352,89],"gitops","2026-08-17T08:12:00.911Z","\u002FUsers\u002Fnguyenanhtuan\u002FCode\u002FNuxtjs\u002Fnextpress\u002Fintlight\u002Fpublic\u002Fcontents\u002Fkubernetes\u002Fi-automated-my-entire-gitops-security-stack-the-first-thing-it-blocked-was-my-own-salary-pko8.md","https:\u002F\u002Fdev.to\u002Fle_beltagy\u002Fi-automated-my-entire-gitops-security-stack-the-first-thing-it-blocked-was-my-own-salary-227e",25,1,50,[360,363,367,369,370,371],{"name":361,"slug":362,"count":356},"All","all",{"name":364,"slug":365,"count":366},"Nuxt 4","nuxt-4",0,{"name":11,"slug":12,"count":368},6,{"name":68,"slug":69,"count":368},{"name":52,"slug":53,"count":368},{"name":34,"slug":35,"count":372},7,[374,375,376,377,378,379,380,381,382,383,385,387,389,390,392],{"name":34,"slug":35,"count":372},{"name":42,"slug":42,"count":372},{"name":69,"slug":69,"count":372},{"name":52,"slug":53,"count":368},{"name":53,"slug":53,"count":368},{"name":68,"slug":69,"count":368},{"name":11,"slug":12,"count":368},{"name":12,"slug":12,"count":368},{"name":89,"slug":89,"count":368},{"name":75,"slug":75,"count":384},4,{"name":74,"slug":74,"count":386},3,{"name":58,"slug":58,"count":388},2,{"name":59,"slug":59,"count":388},{"name":60,"slug":391,"count":388},"trending",{"name":88,"slug":88,"count":388},{"success":394,"slug":9,"lang":10,"items":395,"total":455,"page":357,"limit":456,"hasMore":394,"remaining":456},true,[396,407,417,427,436],{"author":397,"avatar":398,"role":399,"date":400,"createdAt":401,"content":402,"likes":403,"isLiked":404,"replies":405,"id":406},"Hannah Schmidt","H","DevOps & CI\u002FCD Lead","1 day ago","2026-08-16T10:09:00.577Z","Can confirm: automated canary deployments with Argo Rollouts and Prometheus metrics analysis prevented several outages for our payment gateways.",31,false,[],"c-building-sluice-qos-aware-capacity-governance-for-self-hosted-llm-inference-flbu-en-1",{"author":408,"avatar":409,"role":410,"date":411,"createdAt":412,"content":413,"likes":414,"isLiked":404,"replies":415,"id":416},"Elena Rostova","E","Lead SRE & Platform Architect","45 mins ago","2026-08-17T09:24:00.576Z","The KEDA autoscaling setup with custom Prometheus metrics is production-grade. We observed a 60% compute cost reduction after switching to event-driven pod scaling.",34,[],"c-building-sluice-qos-aware-capacity-governance-for-self-hosted-llm-inference-flbu-en-2",{"author":418,"avatar":419,"role":420,"date":421,"createdAt":422,"content":423,"likes":424,"isLiked":404,"replies":425,"id":426},"Liam O'Connor","L","Frontend Performance Specialist","3 hours ago","2026-08-17T07:09:00.576Z","Nuxt 4 with selective hydration and zero-JS interactive islands delivers mind-blowing speed. Sub-100ms INP and 99+ Core Web Vitals out of the box.",21,[],"c-building-sluice-qos-aware-capacity-governance-for-self-hosted-llm-inference-flbu-en-3",{"author":428,"avatar":419,"role":429,"date":430,"createdAt":431,"content":432,"likes":433,"isLiked":404,"replies":434,"id":435},"Lucas Moreau","Cloud Native Developer","12 hours ago","2026-08-16T22:09:00.577Z","Kafka event streaming with schema registry ensures backward compatibility even as payload models evolve across microservice boundaries.",18,[],"c-building-sluice-qos-aware-capacity-governance-for-self-hosted-llm-inference-flbu-en-4",{"author":437,"avatar":438,"role":439,"date":440,"createdAt":441,"content":442,"likes":443,"isLiked":404,"replies":444,"id":454},"Alexander Wright","A","Principal Systems Architect @ Stripe","20 mins ago","2026-08-17T09:49:00.576Z","Superb architectural breakdown! The hybrid L1 in-memory + L2 distributed Redis cache pattern is crucial for mitigating high-concurrency thundering herd issues.",29,[445],{"author":446,"avatar":447,"role":448,"date":449,"createdAt":450,"content":451,"likes":452,"isLiked":404,"id":453},"David Chen","D","Staff Infrastructure Engineer","12 mins ago","2026-08-17T09:57:00.576Z","Totally agree, Alexander. Pairing that with singleflight request deduplication on the backend virtually eliminates DB spikes.",15,"r-building-sluice-qos-aware-capacity-governance-for-self-hosted-llm-inference-flbu-en-5-1","c-building-sluice-qos-aware-capacity-governance-for-self-hosted-llm-inference-flbu-en-5",12,5,1786961340902]