Engineering, operations & design history
One-world scale — Stack recommendation and capacity assessment
Full repository source: docs/world-scale-architecture.md
Technical documentation includes historical decisions, proposals and verification reports. For playing instructions and practical rules, open Gameplay.
No matching entry. Try a shorter phrase or clear the search.
One-world scale — Stack recommendation and capacity assessment
Status: proposed architecture, September 22, 2026. This assessment addresses thousands of concurrent human players and hundreds of AI factions, including continent-scale empires. It does not implement services, migrate storage, change Railway settings or certify capacity. The living-world design records the recommendation; neutral houses and major factions define the gameplay it must support.
Assessment
The recommended technologies are suitable candidates for this strategy simulation. The current single-process living-world implementation is not a sufficient production architecture for the stated target. Player connection count is only one dimension: persistent settlements, production chains, moving groups, knowledge records, hot battles, durable writes and history dominate many workloads. A large faction can cost more than thousands of idle human accounts.
One shared world can span multiple simulation processes while retaining one geography, economy and history. These internal partitions must not become isolated player worlds. The historical thousand-client movement baseline and current movement load harness do not demonstrate thousands of fully active communities or autonomous faction empires. No new capacity test was run for this assessment; the missing faction systems prevent an honest end-to-end measurement today.
Recommended stack
| Layer | Recommendation | Reason and adoption boundary |
|---|---|---|
| Browser | Retain the existing JavaScript/Canvas client and binary movement protocol | Server scaling does not require a frontend rewrite; separately benchmark crowded rendering and resynchronization |
| New server modules | TypeScript compiled for Node.js 24, sharing existing pure JavaScript rules | Typed commands, events, snapshots and ownership messages; runtime validation remains mandatory; migrate incrementally |
| Connections | Node.js with the existing ws transport, independently deployed gateways | Authentication, rate limits, subscriptions and bounded outbound traffic; no durable world authority in gateways |
| Simulation | Node.js regional worker processes; pooled worker threads for measured CPU-heavy tasks | Exclusive authority per region or explicitly owned aggregate; keep pathfinding and planning off gateway event loops |
| Faction automation | Engine scripts, deterministic utility scoring, bounded hierarchical plans and explicit state machines | All routine strategy, administration, quests and succession run without language-model inference; no model agent per faction |
| Durable state | PostgreSQL with explicit SQL transactions, versioned migrations and bounded connection pools | Aggregate revisions, inventory/title custody, due jobs, command receipts, decision state and recovery |
| Messaging at distributed stage | NATS JetStream plus a PostgreSQL transactional outbox/inbox | Deliver committed work between processes with retry and application deduplication; database remains authoritative |
| Static assets and archives | CDN-backed object storage | Keep map art and historical exports off gameplay servers; choose a provider when deploying this layer |
| Hosting | Docker services on Railway, initially in one physical region | Separate gateway, simulation and planner workloads; keep simulation services awake |
| Operations and load tests | Prometheus/Grafana, structured logs and OpenTelemetry traces; protocol-aware headless bots plus targeted browser checks | Measure command durability, queue age, per-zone load, networking, recovery and client behavior |
Redis is not required in the initial target stack; add a disposable cache only if a measured read pattern needs one. Kubernetes, Kafka, GPU inference and a wholesale language rewrite are not prerequisites. Consider a Rust or other native implementation only for a profiled numerical/pathfinding kernel that still misses its budget after algorithmic improvements. It must retain deterministic behavior and contract tests.
Node worker threads execute JavaScript in parallel and suit CPU-heavy tasks; use a pool rather than one thread per faction or request. This supports the proposed execution model, not a promise about its capacity. Node.js worker documentation
Engine-first automation and model spending
User constraint: avoid large-scale model-agent use and keep costs low. Throughout this design, a faction controller or planner means ordinary game code, not an LLM agent. The complete world must remain playable and autonomous with no model provider, API key or inference service. Routine faction operation has a zero model-call budget. This covers local/self-hosted inference as well as paid APIs; running a model locally would still consume compute and is not the default alternative.
| Responsibility | Execution method |
|---|---|
| Food, staffing, production, construction and upkeep | Engine rules, stock thresholds and scheduled jobs |
| Trade offers, reserves, partner selection and relationship memory | Parameterized policies, bounded utility scoring and factual event records |
| Campaign goals, force allocation, route selection and defense | Authored plan templates, feasibility checks, conventional pathfinding and deterministic combat |
| Mage quest chains, permissions and border responses | Authored quest graphs, evidence validation and state machines |
| Undead raising, command relays and soul anchors | Explicit resource and lifecycle rules |
| Orc rulers, heirs, clan support and succession | Persisted dynasty records, seeded traits and political rules |
| Dialogue, refusal reasons and return briefings | Authored templates populated from permitted facts |
Personality is data: risk tolerance, reserve targets, partner preferences and objective weights. Variety comes from different traits, geography, resource constraints and remembered events. Neither every citizen nor every faction receives a continuously running agent. Regional planners share bounded execution pools and retain useful plans instead of repeating expensive searches. CPU planning quotas, database load and network traffic remain separate costs to measure; zero inference spend is not zero hosting cost.
Optional model use is confined initially to development-time content assistance and non-authoritative presentation. Examples are drafting dialogue variants for review, summarizing a selected set of public historical events, and read-only analysis of anonymized balancing reports. Approved development content is versioned and shipped as static data. Runtime narrative, if enabled later, consumes only a permitted fact projection and can change wording but never facts, offers, quest completion, resources, permissions, AI strategy or command execution. Templates remain the complete default and immediate fallback. Model-derived strategic orders are outside this cost-constrained design.
There is no provider or paid model enabled by this proposal. Optional runtime generation defaults to disabled with a world-wide budget of zero. Enabling it later requires an explicitly configured nonzero spending ceiling and the following controls:
- One global daily and monthly money limit across all factions, users, replicas and jobs; per-feature quotas are subordinate, not extra allowances. Reserve each call's worst-case cost atomically before dispatch, using an allowlisted model and bounded input/output. Refuse calls if the cost bound or budget cannot be established.
- Record actual usage and reconcile reservations. Keep uncertain charges reserved until resolved. Retries, timeouts and failed calls count toward the same ceilings; no automatic switch to a more expensive model and no open-ended tool/retry loop.
- Use finite jobs, token caps, timeouts, low concurrency and deduplication. Reuse output only for the same fact/version/audience key; never share private narrative across unauthorized viewers. Player chat, population growth and new factions cannot bypass quotas.
- Cap breaches, provider errors, disabled features and invalid responses immediately select the template. Generation never blocks simulation, delays command acknowledgement or changes a faction's decision cadence. Operational alerts go to maintainers; players keep normal factual output.
Acceptance requires the complete faction workload to run without provider credentials and make zero inference calls by default. Enabling optional prose must leave canonical world state, AI decisions, economy, quests and combat identical for the same seed and inputs. Budget tests include concurrent workers, duplicate jobs, uncertain timeouts and daily/monthly boundaries. These controls are design requirements, not implemented integration or an authorization to spend.
Current bottlenecks found in the repository
| Current code | Scale consequence | Required change |
|---|---|---|
gateway.js serializes living-world work through one promise chain | Independent communities share one mutation bottleneck; slow work delays later work | Separate gateway from simulation and establish bounded ownership domains |
The 250 ms loop snapshots the living world before advancing it; ZoneEngine.execute clones the full state | Allocation and copying grow with the entire persistent world, even for local work | Transaction scope limited to touched aggregates; bounded snapshots/checkpoints |
commitLivingWorld includes the complete state in each logged commit | Disk volume, serialization, restore time and memory grow with world size times commit rate | Incremental durable updates/events plus periodic regional checkpoints |
advance repeatedly scans/sorts jobs and scans settlement due times | Event selection becomes expensive before the simulation logic itself runs | Indexed durable due work and an in-memory priority queue per owner |
| Living-world broadcasts iterate clients and calculate overviews broadly | A local change can trigger work for unrelated players | Index subscribers by affected domain and coalesce authorized projections |
| Movement ownership is a hash into threads inside one host | Adding hosts does not extend durable world ownership safely | Versioned ownership directory, leases/fencing and explicit transfers |
scripts/load-test.js defaults to 100 clients and 10 seconds, sends movement inputs and measures snapshots | Useful transport smoke test; no sustained economic/faction/campaign proof | Seeded persistent gameplay workloads, hotspot tests and recovery tests |
These observations are code review findings, not measured maximum throughput. A faster server may postpone each problem, but does not remove full-world copying, broad fan-out or exclusive ownership requirements.
One world, several owners
Gateways route authenticated human commands to the current regional owner. Faction planners submit proposals through the same validated command boundary. Regional owners load their active aggregates, advance scheduled work, commit changes and publish authorized deltas. A browser keeps its gateway connection when a character crosses regions; the gateway changes the internal destination and subscription set.
Geographic partitioning follows interaction locality rather than political borders. A single empire spans many regional workers. One strategic controller chooses the empire's objectives using compact, permitted summaries; regional planners handle local needs within reserved budgets. The strategic controller never clones or searches every worker and warehouse for each decision. Its budget ledger has explicit ownership and finite allocations so regions cannot spend the same allowance.
Zone ownership has a database-checked epoch/fencing token. Reassignment increments it; every authoritative write must reject an obsolete epoch. A crashed or partitioned former owner must not resume writing alongside its successor. Merely acquiring a process-local lock or launching another replica does not establish this guarantee.
Commit affected state, resource transfers, command receipt and outbox messages atomically. Use stable IDs and inbox records to deduplicate deliveries. JetStream can redeliver messages, so transport acknowledgement is not proof that gameplay happened exactly once. NATS JetStream documentation
Cross-region travel uses a durable custody transfer: mark the group or cargo in transit, transfer it under a unique identity, and activate it at the destination once. An uncertain acknowledgement must not cause a second activation or refund. Small transfers may use a single PostgreSQL transaction; long operations use a persisted multi-step protocol with explicit recovery. Lock affected records in consistent order and keep transactions short.
Process causally related events in recorded order, with deterministic tie-breaking within the owning domain. Cross-domain outcomes depend on recorded handoffs, not wall-clock network arrival or camera presence. Do not impose a global barrier on every tick; coordinate only interacting domains. Catch-up cannot let one region use future goods before the source transfer is committed.
Durable job claiming may use PostgreSQL queue-style row locks, but claiming must preserve per-domain sequence and ownership. SKIP LOCKED is appropriate for independent queue items, not permission to reorder an army's travel or a warehouse's consumption. PostgreSQL SELECT documentation
Large populations without individual thinking loops
Represent ordinary residents as cohorts and armies as formations containing counts, composition, supplies, condition and commanders. A 100,000-soldier army need not create 100,000 independent pathfinders or network entities. Its formation count and simultaneous interactions determine work. Individually simulate named characters only where gameplay needs their identity; retain explicit casualty and inventory accounting.
The broad map remains generated geography with sparse mutable state. Owning thousands of tiles does not require ticking every tile. Production runs on due events, journeys on route checkpoints, and strategy on scheduled decisions and bounded event wake-ups. Hierarchical routing and revision-keyed route caches avoid repeated continent-wide searches; invalidate routes when relevant passage, hazards or terrain change.
A cohort formula may skip empty time only if it preserves meaningful intervening events, thresholds, stockouts, spoilage and deaths. Visual detail can depend on viewers; authoritative results cannot. Battle outcomes remain formation-based and asynchronous as already designed. Thousands of individually controlled units in one real-time tactical fight would be a different, substantially harder requirement.
Proposed validation envelope
For assessment, interpret “thousands of human players” as concurrent connections, not merely registered accounts. These are deliberately concrete load fixtures for testing, not new world-generation values or promises of supported capacity.
| Dimension | Initial target fixture |
|---|---|
| Concurrent humans | 5,000, with a later 10,000-player stress run |
| Persistent human identities | 25,000, including disconnected players whose communities still operate |
| AI factions | 300: 270 with 10 settlements, 27 with 100, and the three major powers with 1,000 each |
| AI settlements | 8,400 across the above distribution; sizes are test inputs, not faction lore |
| Total settlements | 30,000 including human communities; physical map fit must be checked separately |
| Active journeys, projects and other jobs | 100,000, plus separate due-event burst and history-growth tests |
| Commands | 5,000 humans averaging one durable order per 30 seconds = about 167/s; a 10x coordinated burst, plus autonomous simulation work |
| Concentration | Separate tests with 1,000 viewers in one city or battle region; other players remain active elsewhere |
Settlement counts alone are inadequate: publish workers/buildings/cohorts per settlement, active formations, trade routes, event cadence, history size and knowledge density with each result. Populate factions through representative development fixtures and let them produce actual decisions; hundreds of empty faction records prove little.
Illustrative capacity arithmetic, not measured timings:
- If 300 strategic evaluations each cost 50 ms of CPU every 30 real seconds, that is 0.5 CPU-core equivalent on average. At 500 ms per evaluation it is five cores. Regional administration, simulation, garbage collection, persistence and bursts are additional. This is a sensitivity example, not a change to village-day gameplay cadence.
- For 30,000 settlements with 100 effective economic events per village day, the average event rate is
3,000,000 / realSecondsPerVillageDay. Simultaneous daily boundaries may be much more expensive than that average. Test the actual clock and preserve original due times when spreading computation. - At an illustrative 5–15 kB/s outbound per connected player, 5,000 players require 25–75 MB/s, or about 2.16–6.48 TB/day at constant occupancy, before art and protocol overhead. That is a planning range, not a measured traffic forecast. Interest filtering, deltas and CDN delivery are material architecture choices.
A continent-sized faction can dominate the workload even with few leaders. Conversely, many idle characters may be inexpensive. Capacity should be advertised as a tested workload envelope, not a single player count.
Railway deployment assessment
Railway is a candidate host for gateway and worker services. Begin the distributed test deployment in one physical region near the database, with explicit CPU/memory limits, bounded database connection pools and simulation sleeping disabled. Start with a small pool and select replica/worker counts from profiling; this assessment gives no guaranteed machine count or monthly price.
Move durable state out of the application container before replicating gateways. Railway's current volume documentation says services with volumes cannot use replicas and mounted-volume deployments have a short redeployment interruption. Thus the existing local-volume server cannot become a shared MMO merely by raising the replica count. Railway volume limitations
Persistent PostgreSQL and JetStream require separately designed storage, backup, restore and failover. A single database container is still a single failure point. For production persistence, use a PostgreSQL arrangement with verified recovery/PITR and appropriate standby/failover support, on Railway if it meets the requirement or a nearby managed service otherwise. Do not assume an ordinary template deployment provides high availability. JetStream persistence/replication also needs explicit configuration; app replicas do not provision a broker cluster.
Scale connection gateways independently of simulation. Adding a simulation process requires safe assignment and recovery of owned regions. Long-lived sockets require reconnect/resync during deployments, and replay must not repeat orders. Retain portable Docker services so compute or storage can move if latency, sustained CPU, network costs or availability fail the measured targets. Globally distributed authoritative writers are not the first milestone.
Required evidence before claiming scale
Progress through 100, 1,000 and 5,000 concurrent clients while increasing real economy and faction activity. Run a 24-hour target-load soak and a multi-day history-growth/recovery run, followed by a 10,000-connection stress test. Generate load on separate machines and monitor the generators themselves. Include representative browsers; successful headless sockets do not prove map rendering performance.
Proposed release gates, to measure under the declared fixture and hardware:
- Durable economic-command latency within the hosting region: p95 below 250 ms and p99 below one second, excluding the player's internet path. Measure acknowledgement after commit, not queue admission.
- Active 20 Hz movement simulation: p99 processing below 25 ms of the 50 ms tick budget. Observe event-loop delay and gateway delivery independently.
- Routine interactive world events: p99 scheduling lag below two seconds; strategic evaluations have separately declared deadlines. No continually growing queue, unbounded timers, or silently dropped simulation work.
- Sustain target load with at least 30% measured capacity headroom on limiting resources, including the hottest ownership domain. Explicit backpressure and temporary rejection handle overload without changing outcomes.
- Kill a worker, duplicate/reorder messages, interrupt transfers and perform rolling deploys: zero duplicated goods, lost titles or repeated quest rewards; no acknowledged durable-command loss for an application-worker crash with the database healthy. Aim to recover an affected region within 60 seconds. Database/region disaster RPO and RTO need their own tested infrastructure contract.
- Test a crowded market, mass invasion, coordinated quest completion, expiry bursts, disconnections, reconnect storms and catch-up after a long outage. Thousands of viewers are not the same as thousands of combatants interacting pairwise.
- Restore backups and compare inventories, ownership, quest progress and historical receipts. Measure database WAL/write rate, locks, checkpoints, replay time, broker lag, memory growth and network egress.
No gate has been run or passed by this document. The current movement harness must be extended; listing a future performance budget does not establish it.
Implementation order
- Measure the current complete gameplay loop; remove broad state copying, repeated global scans and unrelated broadcasts. Define versioned ownership/command/event boundaries.
- Introduce PostgreSQL durability for affected aggregates, receipts, scheduled work and transactional publication. Migrate a development copy with conservation/restart checks and rollback preparation; keep live migration a separate reviewed deployment.
- Separate gateway and simulation, implement ownership fencing and cross-region handoffs, then add JetStream when work crosses processes. Preserve a simple local development topology.
- Add bounded AI planners and three neutral houses. Use realistic synthetic large-realm fixtures to exercise the architecture before all major-faction content is complete.
- Add faction-specific systems and prove the combined 300-faction, large-realm and human load. Address hotspot/database limits from profiles before increasing capacity claims.
This refines earlier “wait for measured pressure” guidance: defer unnecessary distributed deployment during the small prototype, but do not present the single-writer/file-snapshot prototype as the architecture for a thousands-player public world. Define the migration boundary now and require measured evidence before public scale-up.