feat: implement input event system with durable command handling
- Introduced InputEvent model with status tracking (PENDING, PROCESSING, SUCCEEDED, FAILED) and unique request IDs. - Added DatabaseTurnDaemonTransport for sending commands and handling idempotency. - Implemented executeInputEvent function to manage input event lifecycle and error handling. - Created DatabaseTurnDaemonCommandQueue for managing command processing and lease recovery. - Enhanced turn daemon lifecycle to support atomic command execution and error recovery. - Added tests for input event atomicity, command queuing, and error handling scenarios.
This commit is contained in:
@@ -82,32 +82,37 @@ Gateway runs a lightweight cron loop (setInterval) that:
|
||||
## Current Implementation Status
|
||||
|
||||
- Turn daemon lifecycle + in-memory state live in `app/game-engine` with DB flush hooks.
|
||||
- Redis transport for daemon control is implemented and wired into the daemon.
|
||||
- API server already exposes turn-daemon commands (run/pause/resume/status) via tRPC, communicating via Redis Streams.
|
||||
- API server writes reserved turns and messages directly to the DB; daemon focuses on world state/logs.
|
||||
- 모든 tRPC mutation은 PostgreSQL `input_event` 경계에서 request ID와 처리
|
||||
상태를 기록한다.
|
||||
- 월드 mutation과 daemon control은 PostgreSQL inbox로 전달된다. Redis는
|
||||
더 이상 이 명령들의 영속 큐가 아니며 realtime/battle-sim 전송에만 남아
|
||||
있다.
|
||||
- API가 직접 변경하는 예약 턴·메시지 등의 DB 쓰기는 API input event 완료와
|
||||
같은 transaction에서 commit된다.
|
||||
- 엔진 명령은 도메인 DB 쓰기, in-memory world flush, 예약 턴, 로그, 결과
|
||||
저장과 inbox 완료를 하나의 transaction으로 commit한다.
|
||||
|
||||
## Turn Daemon and API Server Behavior (Outline)
|
||||
|
||||
- Turn daemon responsibilities: scheduling, turn resolution, state persistence
|
||||
- API server responsibilities: query/command intake, validation, response shaping
|
||||
- Concurrency model between daemon and API server
|
||||
- Communication channel: Redis Stream or Redis pub/sub
|
||||
- Durable command channel: PostgreSQL `input_event`
|
||||
- Client updates: SSE between API server and frontend where appropriate
|
||||
|
||||
## Redis Communication Recommendation (Draft)
|
||||
## Durable Input Event Split
|
||||
|
||||
Use Redis Streams for daemon control and mutation requests, and Redis pub/sub
|
||||
for transient fan-out events. Streams provide durability, backpressure, and
|
||||
replay while pub/sub keeps live updates simple and low-latency.
|
||||
PostgreSQL is the source of truth for accepted input. Redis pub/sub remains a
|
||||
best-effort notification/fan-out path and must not decide whether a gameplay
|
||||
mutation was committed.
|
||||
|
||||
### Recommended Split
|
||||
|
||||
- Redis Streams:
|
||||
- API server -> daemon: mutation requests, turn-run commands.
|
||||
- Daemon -> API server: run status events, job results, error reports.
|
||||
- Use consumer groups for daemon workers and API server listeners.
|
||||
- Require `requestId` for correlation and idempotency.
|
||||
- Ack on success; move failed items to a dead-letter stream after retry.
|
||||
- PostgreSQL `input_event`:
|
||||
- API mutation acceptance, idempotency, attempts and audit state.
|
||||
- API server -> daemon mutation/control payloads and durable results.
|
||||
- `FOR UPDATE SKIP LOCKED` claim plus worker lease for concurrent consumers.
|
||||
- engine result and gameplay state commit in the same transaction.
|
||||
- Redis pub/sub:
|
||||
- Daemon -> API server: low-stakes live update signals (run started/ended).
|
||||
- API server -> frontend: SSE fan-out triggered by pub/sub updates.
|
||||
@@ -115,11 +120,11 @@ replay while pub/sub keeps live updates simple and low-latency.
|
||||
|
||||
### Operational Notes
|
||||
|
||||
- Stream keys should be namespaced per server+scenario profile.
|
||||
- Use bounded stream length (`MAXLEN`) to cap storage.
|
||||
- API server should guard against duplicate processing by `requestId`.
|
||||
- When the daemon is busy, API queues new mutations to stream and responds
|
||||
with an accepted status to clients.
|
||||
- 각 profile은 별도 game DB schema/connection을 사용한다.
|
||||
- HTTP `Idempotency-Key`(없으면 Fastify request ID)와 tRPC path를 합친 값이
|
||||
API input event key다.
|
||||
- 처리 중 lease가 만료된 engine event만 `PENDING`으로 회수한다.
|
||||
- 완료 event 보존/정리 기간과 `FAILED` 재처리 운영 정책은 별도로 정해야 한다.
|
||||
|
||||
Detailed lifecycle and control flow are defined in
|
||||
`docs/architecture/turn-daemon-lifecycle.md`.
|
||||
@@ -167,13 +172,14 @@ also supports local ID/password login for users who cannot use Kakao.
|
||||
immediately even if requests remain queued.
|
||||
- While the daemon is resolving a turn, the API server queues incoming requests.
|
||||
|
||||
Note: the current implementation does not yet process API mutation requests
|
||||
between turns; only control commands are handled by the in-process queue.
|
||||
현재 구현은 control 명령과 registry의 모든 world mutation을 같은 DB queue로
|
||||
읽어 턴 사이에 처리한다.
|
||||
|
||||
### Daemon Control Contract (Draft)
|
||||
|
||||
API server commands are delivered to the daemon over the control channel
|
||||
(Redis Stream or in-process). The daemon replies with status and run events.
|
||||
API server commands are inserted into PostgreSQL `input_event`; the daemon
|
||||
stores status/results on the same row. An in-process queue remains available
|
||||
for tests and internal signals.
|
||||
|
||||
```ts
|
||||
export type RunReason = 'schedule' | 'manual' | 'poke';
|
||||
@@ -193,19 +199,17 @@ export type DaemonEvent =
|
||||
|
||||
### API Server Flow
|
||||
|
||||
- The API server validates queries/commands and writes them to Redis Streams
|
||||
or Redis pub/sub.
|
||||
- The API server validates commands and records every tRPC mutation in
|
||||
PostgreSQL.
|
||||
- After a request is processed, the API server returns the result to clients.
|
||||
- Read-only queries may access the DBMS directly.
|
||||
- The API server may use SSE to stream live updates to the frontend.
|
||||
|
||||
### Queue and Rate Limits
|
||||
|
||||
- API server requests are delivered to the daemon via Redis Streams or
|
||||
Redis pub/sub.
|
||||
- Redis Stream mutation requests are rate-limited per user.
|
||||
- Each user can have up to 30 pending mutation requests.
|
||||
- Additional requests are rejected once the limit is exceeded.
|
||||
- Engine-owned requests are delivered through the PostgreSQL inbox.
|
||||
- Per-user pending limits and event retention remain operational follow-up
|
||||
work; idempotency and exclusive claim are implemented.
|
||||
|
||||
### In-Memory and DBMS Flush
|
||||
|
||||
|
||||
@@ -8,14 +8,14 @@ Move items into the main docs once they are finalized.
|
||||
- [AI suggestion] Make profile turn execution a durable single-writer contract:
|
||||
add a DB-backed lease with fencing epoch/world version, and reject stale
|
||||
daemon commits after a restart or split-brain.
|
||||
- [AI suggestion] Replace the current destructive dirty-state drain and
|
||||
multi-query flush with bounded atomic batches. Persist entity deltas, logs,
|
||||
reserved turns, checkpoint, and outbox events in one transaction, then clear
|
||||
dirty state only after commit.
|
||||
- [AI suggestion] Introduce durable command inbox/outbox records with unique
|
||||
request IDs, idempotent results, and per-profile ordering. Keep Redis as a
|
||||
wake-up/fan-out layer, or add consumer groups, ACK/pending recovery, DLQ, and
|
||||
stream trimming if Redis Streams remain the command source.
|
||||
- [AI suggestion] Bound very large world flush batches and add a transactional
|
||||
realtime outbox. Entity deltas, logs, reserved turns, checkpoint and input
|
||||
results are atomic now, but realtime publication remains post-commit
|
||||
best-effort.
|
||||
- [AI suggestion] Define `input_event` retention, failed-event inspection/DLQ,
|
||||
manual replay authorization and per-user pending limits. Durable request IDs,
|
||||
idempotent results, ordering, exclusive claim and expired-lease recovery are
|
||||
implemented.
|
||||
- [AI suggestion] Add ownership authorization to all turn reservation
|
||||
procedures (`auth user -> owned general -> current officer role -> nation`)
|
||||
and add revision/optimistic concurrency to reservation queue updates.
|
||||
|
||||
@@ -8,14 +8,15 @@ from the API server.
|
||||
|
||||
- One daemon process per server+scenario profile.
|
||||
- API server is the only ingress for user/admin requests.
|
||||
- The daemon owns world-state mutations; the API server mutates reserved turns and messages.
|
||||
- The daemon owns world-state mutations; the API server mutates reserved turns
|
||||
and messages inside the common API input-event transaction.
|
||||
|
||||
## Current Implementation Status
|
||||
|
||||
- The daemon loop handles scheduled runs plus Redis-based control queue commands
|
||||
- The daemon loop handles scheduled runs plus PostgreSQL inbox control commands
|
||||
(run/pause/resume/shutdown/getStatus).
|
||||
- API mutation requests (troopJoin, vacation, etc.) are drained and handled by the daemon loop between turns.
|
||||
- `getStatus` is fully supported and responds via Redis.
|
||||
- `getStatus` and command results are stored durably on their input-event row.
|
||||
- API server currently writes reserved turns and messages directly to the DB.
|
||||
|
||||
## Responsibilities
|
||||
@@ -47,8 +48,7 @@ Idle -> Running -> Flushing -> Idle
|
||||
The daemon interleaves API request handling between scheduled turn executions.
|
||||
API requests are drained until the next turn time is reached; once the turn
|
||||
starts, incoming requests are queued and processed after the run.
|
||||
Current implementation does not yet drain API requests between turns; this
|
||||
section is the intended target behavior.
|
||||
The current implementation follows this interleaving model.
|
||||
|
||||
```ts
|
||||
while (!stopping) {
|
||||
@@ -101,7 +101,7 @@ Minimum checkpoint data:
|
||||
|
||||
## API Server Interaction
|
||||
|
||||
- API server enqueues mutations and pokes the daemon.
|
||||
- API server inserts mutations and pokes into the PostgreSQL inbox.
|
||||
- Read-only queries can read from DBMS or a read model.
|
||||
- During `running`, incoming requests are queued and processed after the run.
|
||||
|
||||
@@ -171,7 +171,8 @@ Admin controls should toggle `paused`, trigger manual run, and request catch-up.
|
||||
|
||||
### Daemon Control Contract (Draft)
|
||||
|
||||
API server to daemon control messages (Redis Stream or in-process channel):
|
||||
API server to daemon control payloads (PostgreSQL inbox or in-process test
|
||||
channel):
|
||||
|
||||
```ts
|
||||
export type RunReason = 'schedule' | 'manual' | 'poke';
|
||||
@@ -192,9 +193,33 @@ export type DaemonEvent =
|
||||
## Recovery and Shutdown
|
||||
|
||||
- On startup, load `game_env.turntime` and catch up to `now`.
|
||||
- Claim pending input events with `FOR UPDATE SKIP LOCKED`. A worker ID and
|
||||
expiry timestamp prevent another live consumer from claiming the same event;
|
||||
only expired processing leases return to `PENDING`.
|
||||
- A handler exception or transaction failure pauses the daemon without
|
||||
acknowledging the event. Restart/recovery reloads world state before retry.
|
||||
- On shutdown, stop accepting new triggers, finish the current run, flush, then
|
||||
exit cleanly.
|
||||
|
||||
## Input Event Commit Boundary
|
||||
|
||||
For engine-owned mutations the lifecycle opens one database-owned unit of work:
|
||||
|
||||
```text
|
||||
claim input_event
|
||||
-> execute command against in-memory world
|
||||
-> specialized domain DB writes (auction/tournament)
|
||||
-> persist world/reserved turns/logs
|
||||
-> store command result and mark input_event SUCCEEDED
|
||||
-> commit
|
||||
-> acknowledge in-memory dirty sets
|
||||
-> respond
|
||||
```
|
||||
|
||||
If any step before commit fails, PostgreSQL rolls back and dirty sets are not
|
||||
cleared. Realtime publication happens after commit and cannot roll gameplay
|
||||
state back.
|
||||
|
||||
## Testing with a Controlled Clock
|
||||
|
||||
- Turn daemon tests should use a controllable `Clock` implementation (예: `ManualClock`) to
|
||||
|
||||
Reference in New Issue
Block a user