test(load): 1분 페이지 동시 부하 계측을 추가한다

국가 권한 사용자 10명과 800 NPC fixture를 만들고 Chromium 화면 순환, SSE, wall-clock 턴을 같은 시각에 측정한다.
This commit is contained in:
2026-08-21 23:17:24 +00:00
parent 7066d13c5b
commit e67a40e51d
8 changed files with 601 additions and 15 deletions
+40 -4
View File
@@ -62,7 +62,9 @@ pnpm --filter @sammo-ts/load-tests activate-coverage \
``` ```
`seed`는 해당 `load_` schema에 migration을 적용하고 scenario 2601을 고정 seed/time으로 설치한 뒤 정확히 `seed`는 해당 `load_` schema에 migration을 적용하고 scenario 2601을 고정 seed/time으로 설치한 뒤 정확히
900 NPC + 300 synthetic 사용자 장수로 재구성한다. 각 사용자의 24시간 access token은 Redis 전용 DB와 900 NPC + 300 synthetic 사용자 장수로 재구성한다. synthetic 사용자는 같은 fixture 국가와 도시에 배치하고
관직 5 이상을 순환 배정하여 암행부·사령부·감찰부·내무부를 실제 권한으로 읽을 수 있게 한다. 시나리오에
비중립 국가가 없으면 전용 fixture 안에만 측정 국가 하나를 만든다. 각 사용자의 24시간 access token은 Redis 전용 DB와
`0600` JSON에만 저장한다. stdout에는 token, user/general ID, DB/Redis URL을 내보내지 않고 count와 `0600` JSON에만 저장한다. stdout에는 token, user/general ID, DB/Redis URL을 내보내지 않고 count와
비밀값을 제외한 fixture SHA-256만 기록한다. token 파일이 이미 있으면 DB 작업 전에 실패한다. 비밀값을 제외한 fixture SHA-256만 기록한다. token 파일이 이미 있으면 DB 작업 전에 실패한다.
@@ -83,6 +85,7 @@ pnpm --filter @sammo-ts/game-api build
systemd-run --user --unit=sammo-capacity-api --collect \ systemd-run --user --unit=sammo-capacity-api --collect \
--property=MemoryMax=8G \ --property=MemoryMax=8G \
--setenv=POSTGRES_POOL_MAX=4 \
--working-directory="$(pwd)" \ --working-directory="$(pwd)" \
"$(pwd)/tools/load-tests/scripts/run-capacity-api.sh" "$(pwd)/tools/load-tests/scripts/run-capacity-api.sh"
``` ```
@@ -181,7 +184,40 @@ schema 간 row-lock 격리와 공유 CPU/I/O/connection 경합을 확인한다.
- `tools/load-tests/config/nya-10-users-800-npcs-1m.json` - `tools/load-tests/config/nya-10-users-800-npcs-1m.json`
- `tools/load-tests/config/pya-10-users-800-npcs-1m.json` - `tools/load-tests/config/pya-10-users-800-npcs-1m.json`
### 5. 명시적 cleanup ### 5. 1분 턴과 10-user 실제 화면 동시 측정
`measure-turn-cycle`은 fixture의 권위 `turn_time` 분포대로 장수 턴을 wall clock 1분에 pacing하고 월 경계를
한 번 처리한다. `measure-page-navigation`은 실제 Chromium context 10개가 암행부, 사령부, 현재 도시,
감찰부, 내무부를 순환하며 페이지와 tRPC 지연을 기록한다. 화면 이동 중에도 사용자마다 별도 SSE 한 개를
계속 유지한다. 이는 제품 UI가 메인 화면 밖에서 dashboard SSE를 닫는 동작과 “각 사용자가 실시간 알림을
계속 받는다”는 부하 조건을 분리해 재현하기 위한 구성이다.
두 명령에 같은 미래 `LOAD_TEST_START_AT_EPOCH_MS`를 주면 턴과 브라우저 측정 구간을 맞출 수 있다.
`LOAD_TEST_FRONTEND_URL`은 private/loopback URL과 정확한 profile suffix여야 하며,
`LOAD_TEST_API_PID`는 같은 workspace에서 실행 중인 game-api PID여야 한다. 결과에는 token, 응답 본문과
사용자 식별자를 넣지 않는다.
```sh
export LOAD_TEST_START_AT_EPOCH_MS=REPLACE_WITH_NEAR_FUTURE_EPOCH_MS
export LOAD_TEST_FRONTEND_URL=http://127.0.0.1:15000/pya/
export LOAD_TEST_API_PID=REPLACE_WITH_LOCAL_GAME_API_PID
pnpm --filter @sammo-ts/load-tests measure-page-navigation \
--config tools/load-tests/config/pya-10-users-800-npcs-1m.json \
--tokens tools/load-tests/secrets/game-tokens.json \
--output tools/load-tests/results/page-navigation.json
pnpm --filter @sammo-ts/load-tests measure-turn-cycle \
--config tools/load-tests/config/pya-10-users-800-npcs-1m.json \
--confirm load_capacity_pya_10_800_1m \
--output tools/load-tests/results/turn-cycle.json
```
페이지 latency는 production frontend의 lazy chunk, 인증 초기화와 Playwright `networkidle` 500ms를 모두
포함하므로 API procedure latency와 함께 해석한다. API CPU/RSS는 지정 PID만, PostgreSQL 지표는 같은 DB의
observer를 포함한 database-wide delta만 나타낸다. 호스트의 다른 profile, Gateway와 worker 부하는 포함하지 않는다.
### 6. 명시적 cleanup
token 파일은 별도로 안전하게 삭제하고, fixture schema/Redis token은 schema명을 그대로 확인 인자로 주어 token 파일은 별도로 안전하게 삭제하고, fixture schema/Redis token은 schema명을 그대로 확인 인자로 주어
정리한다. named volume은 보존한다. 데이터 폐기가 필요하지 않으면 이 명령을 실행하지 않는다. 정리한다. named volume은 보존한다. 데이터 폐기가 필요하지 않으면 이 명령을 실행하지 않는다.
@@ -195,8 +231,8 @@ docker compose -f tools/load-tests/compose.capacity.yml down
## 아직 남은 측정 경계 ## 아직 남은 측정 경계
- `measure-turn-flush`는 한 달을 wall-clock보다 빠르게 replay하는 처리량 시험이다. 실제 schedule lag, - `measure-turn-flush`는 한 달을 wall-clock보다 빠르게 replay하는 처리량 시험이다. 실제 1분 schedule lag
장시간 pool wait와 autovacuum/checkpoint 영향을 보려면 profile별 속도로 pacing한 soak가 별도로 필요하다. `measure-turn-cycle`로 확인할 수 있지만 장시간 pool wait와 autovacuum/checkpoint는 더 긴 soak가 필요하다.
- 월 경계 latency는 실행당 표본이 하나다. scenario 진행 시점과 월별 event 차이를 포괄하지 않는다. - 월 경계 latency는 실행당 표본이 하나다. scenario 진행 시점과 월별 event 차이를 포괄하지 않는다.
- 두 1분 profile 동시 실행은 최악 turn-rate 조합의 국소 증거이며, 여섯 profile의 전체 PM2 RSS, - 두 1분 profile 동시 실행은 최악 turn-rate 조합의 국소 증거이며, 여섯 profile의 전체 PM2 RSS,
Gateway·worker connection과 운영 container cgroup을 재현하지 않는다. Gateway·worker connection과 운영 container cgroup을 재현하지 않는다.
+2
View File
@@ -11,6 +11,8 @@
"seed": "pnpm -w exec tsx tools/load-tests/src/cli.ts seed", "seed": "pnpm -w exec tsx tools/load-tests/src/cli.ts seed",
"verify-fixture": "pnpm -w exec tsx tools/load-tests/src/cli.ts verify-fixture", "verify-fixture": "pnpm -w exec tsx tools/load-tests/src/cli.ts verify-fixture",
"activate-coverage": "pnpm -w exec tsx tools/load-tests/src/cli.ts activate-coverage", "activate-coverage": "pnpm -w exec tsx tools/load-tests/src/cli.ts activate-coverage",
"measure-page-navigation": "pnpm -w exec tsx tools/load-tests/src/cli.ts measure-page-navigation",
"measure-turn-cycle": "pnpm -w exec tsx tools/load-tests/src/cli.ts measure-turn-cycle",
"measure-turn-flush": "pnpm -w exec tsx tools/load-tests/src/cli.ts measure-turn-flush", "measure-turn-flush": "pnpm -w exec tsx tools/load-tests/src/cli.ts measure-turn-flush",
"materialize-calibration": "pnpm -w exec tsx tools/load-tests/src/cli.ts materialize-calibration", "materialize-calibration": "pnpm -w exec tsx tools/load-tests/src/cli.ts materialize-calibration",
"cleanup": "pnpm -w exec tsx tools/load-tests/src/cli.ts cleanup", "cleanup": "pnpm -w exec tsx tools/load-tests/src/cli.ts cleanup",
+33 -4
View File
@@ -10,6 +10,7 @@ import {
seedCapacityFixture, seedCapacityFixture,
verifyCapacityFixture, verifyCapacityFixture,
} from './fixture.js'; } from './fixture.js';
import { measurePageNavigation } from './pageNavigation.js';
import { describeDryRun, runLoadTest } from './runner.js'; import { describeDryRun, runLoadTest } from './runner.js';
import { measureTurnFlush } from './turnFlush.js'; import { measureTurnFlush } from './turnFlush.js';
@@ -21,13 +22,15 @@ type Command =
| 'seed' | 'seed'
| 'verify-fixture' | 'verify-fixture'
| 'activate-coverage' | 'activate-coverage'
| 'measure-page-navigation'
| 'measure-turn-cycle'
| 'measure-turn-flush' | 'measure-turn-flush'
| 'materialize-calibration' | 'materialize-calibration'
| 'cleanup'; | 'cleanup';
const usage = (): never => { const usage = (): never => {
process.stderr.write( process.stderr.write(
'usage: cli.ts <validate|dry-run|run|prepare|seed|verify-fixture|activate-coverage|measure-turn-flush|materialize-calibration|cleanup> --config <file> [--tokens <0600-gitignored-file>] [--output <new-json-file>] [--confirm <load_schema>]\n' 'usage: cli.ts <validate|dry-run|run|prepare|seed|verify-fixture|activate-coverage|measure-page-navigation|measure-turn-cycle|measure-turn-flush|materialize-calibration|cleanup> --config <file> [--tokens <0600-gitignored-file>] [--output <new-json-file>] [--confirm <load_schema>]\n'
); );
process.exit(64); process.exit(64);
}; };
@@ -45,6 +48,8 @@ const parseArguments = (
'seed', 'seed',
'verify-fixture', 'verify-fixture',
'activate-coverage', 'activate-coverage',
'measure-page-navigation',
'measure-turn-cycle',
'measure-turn-flush', 'measure-turn-flush',
'materialize-calibration', 'materialize-calibration',
'cleanup', 'cleanup',
@@ -73,7 +78,12 @@ const parseArguments = (
) )
usage(); usage();
if ( if (
command === 'measure-turn-flush' && command === 'measure-page-navigation' &&
(!values.get('--tokens') || !values.get('--output') || values.has('--confirm'))
)
usage();
if (
(command === 'measure-turn-flush' || command === 'measure-turn-cycle') &&
(!values.get('--confirm') || !values.get('--output') || values.has('--tokens')) (!values.get('--confirm') || !values.get('--output') || values.has('--tokens'))
) )
usage(); usage();
@@ -124,10 +134,14 @@ const main = async (): Promise<void> => {
process.stdout.write(`${JSON.stringify(await activateCapacityCoverage(config, args.confirm!))}\n`); process.stdout.write(`${JSON.stringify(await activateCapacityCoverage(config, args.confirm!))}\n`);
return; return;
} }
if (args.command === 'measure-turn-flush') { if (args.command === 'measure-turn-flush' || args.command === 'measure-turn-cycle') {
const output = path.resolve(args.output!); const output = path.resolve(args.output!);
await mkdir(path.dirname(output), { recursive: true }); await mkdir(path.dirname(output), { recursive: true });
const result = await measureTurnFlush({ config, confirmation: args.confirm! }); const result = await measureTurnFlush({
config,
confirmation: args.confirm!,
paced: args.command === 'measure-turn-cycle',
});
await writeFile(output, `${JSON.stringify(result, null, 2)}\n`, { await writeFile(output, `${JSON.stringify(result, null, 2)}\n`, {
encoding: 'utf8', encoding: 'utf8',
flag: 'wx', flag: 'wx',
@@ -138,6 +152,21 @@ const main = async (): Promise<void> => {
); );
return; return;
} }
if (args.command === 'measure-page-navigation') {
const tokens = await loadTokens(args.tokens!, workspaceRoot, config.capacity.authenticatedViewers);
const output = path.resolve(args.output!);
await mkdir(path.dirname(output), { recursive: true });
const result = await measurePageNavigation({ config, tokens, workspaceRoot });
await writeFile(output, `${JSON.stringify(result, null, 2)}\n`, {
encoding: 'utf8',
flag: 'wx',
mode: 0o600,
});
process.stdout.write(
`${JSON.stringify({ completed: true, viewers: result.fixture.viewers, outputWritten: true })}\n`
);
return;
}
if (args.command === 'materialize-calibration') { if (args.command === 'materialize-calibration') {
process.stdout.write( process.stdout.write(
`${JSON.stringify( `${JSON.stringify(
+61 -2
View File
@@ -182,9 +182,24 @@ const migrateDedicatedSchema = async (workspaceRoot: string, databaseUrl: string
type GeneralRow = Awaited<ReturnType<GamePrismaClient['general']['findMany']>>[number]; type GeneralRow = Awaited<ReturnType<GamePrismaClient['general']['findMany']>>[number];
const PRIVILEGED_VIEWER_LEVELS = [12, 10, 8, 6, 11, 9, 7, 5] as const;
export const privilegedViewerPlacement = (index: number, nationId: number, cityId: number) => ({
nationId,
cityId,
officerLevel: PRIVILEGED_VIEWER_LEVELS[index % PRIVILEGED_VIEWER_LEVELS.length]!,
});
const cloneGeneral = ( const cloneGeneral = (
source: GeneralRow, source: GeneralRow,
input: { id: number; userId: string | null; npcState: number } input: {
id: number;
userId: string | null;
npcState: number;
nationId?: number;
cityId?: number;
officerLevel?: number;
}
): GamePrisma.GeneralCreateManyInput => ): GamePrisma.GeneralCreateManyInput =>
({ ({
...source, ...source,
@@ -192,6 +207,9 @@ const cloneGeneral = (
name: `${source.name}#L${input.id}`, name: `${source.name}#L${input.id}`,
userId: input.userId, userId: input.userId,
npcState: input.npcState, npcState: input.npcState,
nationId: input.nationId ?? source.nationId,
cityId: input.cityId ?? source.cityId,
officerLevel: input.officerLevel ?? source.officerLevel,
turnTime: new Date(source.turnTime), turnTime: new Date(source.turnTime),
recentWarTime: source.recentWarTime ? new Date(source.recentWarTime) : null, recentWarTime: source.recentWarTime ? new Date(source.recentWarTime) : null,
createdAt: FIXED_NOW, createdAt: FIXED_NOW,
@@ -237,8 +255,17 @@ const projectFixtureState = async (db: GamePrismaClient) => {
}; };
const resizeSeededGenerals = async (db: GamePrismaClient, config: LoadConfig): Promise<void> => { const resizeSeededGenerals = async (db: GamePrismaClient, config: LoadConfig): Promise<void> => {
const source = await db.general.findMany({ orderBy: { id: 'asc' } }); const [source, nations, cities] = await Promise.all([
db.general.findMany({ orderBy: { id: 'asc' } }),
db.nation.findMany({ orderBy: { id: 'asc' } }),
db.city.findMany({ orderBy: { id: 'asc' } }),
]);
if (source.length === 0) throw new Error('scenario seed produced no generals'); if (source.length === 0) throw new Error('scenario seed produced no generals');
const existingViewerNation = nations.find((nation) => nation.id > 0);
const viewerCity =
(existingViewerNation ? cities.find((city) => city.nationId === existingViewerNation.id) : null) ?? cities[0];
if (!viewerCity) throw new Error('scenario seed produced no city for privileged page load');
const viewerNationId = existingViewerNation?.id ?? Math.max(...nations.map((nation) => nation.id), 0) + 1;
const expectedNpc = config.capacity.npcGenerals; const expectedNpc = config.capacity.npcGenerals;
const expectedHuman = config.capacity.humanGenerals; const expectedHuman = config.capacity.humanGenerals;
if (expectedHuman !== config.capacity.authenticatedViewers) { if (expectedHuman !== config.capacity.authenticatedViewers) {
@@ -246,20 +273,43 @@ const resizeSeededGenerals = async (db: GamePrismaClient, config: LoadConfig): P
} }
await db.$transaction(async (transaction) => { await db.$transaction(async (transaction) => {
await transaction.general.deleteMany(); await transaction.general.deleteMany();
if (!existingViewerNation) {
await transaction.nation.create({
data: {
id: viewerNationId,
name: '부하측정국',
color: '#334466',
capitalCityId: viewerCity.id,
gold: 100_000,
rice: 100_000,
tech: 1_000,
level: 1,
typeCode: 'che_중립',
meta: { cityIds: [viewerCity.id], infoText: null, secretlimit: 3 },
},
});
}
await transaction.city.update({ where: { id: viewerCity.id }, data: { nationId: viewerNationId } });
const rows: GamePrisma.GeneralCreateManyInput[] = []; const rows: GamePrisma.GeneralCreateManyInput[] = [];
for (let index = 0; index < expectedNpc; index += 1) { for (let index = 0; index < expectedNpc; index += 1) {
rows.push(cloneGeneral(source[index % source.length]!, { id: index + 1, userId: null, npcState: 2 })); rows.push(cloneGeneral(source[index % source.length]!, { id: index + 1, userId: null, npcState: 2 }));
} }
for (let index = 0; index < expectedHuman; index += 1) { for (let index = 0; index < expectedHuman; index += 1) {
const placement = privilegedViewerPlacement(index, viewerNationId, viewerCity.id);
rows.push( rows.push(
cloneGeneral(source[(expectedNpc + index) % source.length]!, { cloneGeneral(source[(expectedNpc + index) % source.length]!, {
id: expectedNpc + index + 1, id: expectedNpc + index + 1,
userId: `load-user-${String(index + 1).padStart(4, '0')}`, userId: `load-user-${String(index + 1).padStart(4, '0')}`,
npcState: 0, npcState: 0,
...placement,
}) })
); );
} }
await transaction.general.createMany({ data: rows }); await transaction.general.createMany({ data: rows });
await transaction.nation.update({
where: { id: viewerNationId },
data: { chiefGeneralId: expectedNpc + 1 },
});
const world = await transaction.worldState.findFirstOrThrow({ select: { id: true, meta: true, config: true } }); const world = await transaction.worldState.findFirstOrThrow({ select: { id: true, meta: true, config: true } });
await transaction.worldState.update({ await transaction.worldState.update({
where: { id: world.id }, where: { id: world.id },
@@ -435,10 +485,17 @@ export const verifyCapacityFixture = async (config: LoadConfig, env: NodeJS.Proc
const redisVersion = /^redis_version:(.+)$/mu.exec(redisInfo)?.[1]?.trim() ?? 'unknown'; const redisVersion = /^redis_version:(.+)$/mu.exec(redisInfo)?.[1]?.trim() ?? 'unknown';
const npcGenerals = state.generals.filter((general) => general.npcState >= 2).length; const npcGenerals = state.generals.filter((general) => general.npcState >= 2).length;
const humanGenerals = state.generals.filter((general) => general.npcState === 0 && general.userId).length; const humanGenerals = state.generals.filter((general) => general.npcState === 0 && general.userId).length;
const privilegedHumanGenerals = state.generals.filter(
(general) =>
general.npcState === 0 && general.userId && general.nationId > 0 && general.officerLevel >= 5
);
const viewerNationIds = [...new Set(privilegedHumanGenerals.map((general) => general.nationId))];
const valid = const valid =
state.generals.length === config.capacity.npcGenerals + config.capacity.humanGenerals && state.generals.length === config.capacity.npcGenerals + config.capacity.humanGenerals &&
npcGenerals === config.capacity.npcGenerals && npcGenerals === config.capacity.npcGenerals &&
humanGenerals === config.capacity.humanGenerals && humanGenerals === config.capacity.humanGenerals &&
privilegedHumanGenerals.length === config.capacity.humanGenerals &&
viewerNationIds.length === 1 &&
accessTokens === config.capacity.authenticatedViewers && accessTokens === config.capacity.authenticatedViewers &&
manifestFixtureSha256 === fixtureSha256; manifestFixtureSha256 === fixtureSha256;
return { return {
@@ -447,6 +504,8 @@ export const verifyCapacityFixture = async (config: LoadConfig, env: NodeJS.Proc
generals: state.generals.length, generals: state.generals.length,
npcGenerals, npcGenerals,
humanGenerals, humanGenerals,
privilegedHumanGenerals: privilegedHumanGenerals.length,
viewerNations: viewerNationIds.length,
accessTokens, accessTokens,
redisManifestPresent: rawManifest !== null, redisManifestPresent: rawManifest !== null,
redisManifestMatches: manifestFixtureSha256 === fixtureSha256, redisManifestMatches: manifestFixtureSha256 === fixtureSha256,
+380
View File
@@ -0,0 +1,380 @@
import { execFile } from 'node:child_process';
import { readFile, realpath } from 'node:fs/promises';
import { promisify } from 'node:util';
import { chromium, type Page } from '@playwright/test';
import { isPrivateTargetHost, type LoadConfig } from './config.js';
import { summarizeDistribution } from './metrics.js';
import { PhaseMetrics } from './metrics.js';
import { runSseConnection } from './sse.js';
const execFileAsync = promisify(execFile);
const PAGE_ROUTES = [
{ name: 'nation-secret', path: 'nation/secret' },
{ name: 'chief-center', path: 'chief-center' },
{ name: 'current-city', path: 'current-city' },
{ name: 'battle-center', path: 'battle-center' },
{ name: 'nation-finance', path: 'nation/finance' },
] as const;
const wait = (milliseconds: number): Promise<void> => new Promise((resolve) => setTimeout(resolve, milliseconds));
const increment = (target: Map<string, number>, key: string): void => {
target.set(key, (target.get(key) ?? 0) + 1);
};
const mapToObject = (target: ReadonlyMap<string, number>): Record<string, number> =>
Object.fromEntries([...target.entries()].sort(([left], [right]) => left.localeCompare(right)));
export const parseTrpcProcedures = (requestUrl: string, trpcPath: string): string[] => {
const pathname = new URL(requestUrl).pathname;
const prefix = `${trpcPath.replace(/\/$/u, '')}/`;
if (!pathname.startsWith(prefix)) return [];
return decodeURIComponent(pathname.slice(prefix.length))
.split(',')
.filter((procedure) => /^[A-Za-z][A-Za-z0-9_.]+$/u.test(procedure));
};
type ProcessStat = { cpuTicks: number; rssPages: number; startTicks: number };
const readProcessStat = async (pid: number): Promise<ProcessStat> => {
const raw = await readFile(`/proc/${pid}/stat`, 'utf8');
const commandEnd = raw.lastIndexOf(')');
if (commandEnd < 0) throw new Error('LOAD_TEST_API_PID has an invalid /proc stat record');
const fields = raw.slice(commandEnd + 2).trim().split(/\s+/u);
const cpuTicks = Number(fields[11]) + Number(fields[12]);
const startTicks = Number(fields[19]);
const rssPages = Number(fields[21]);
if (![cpuTicks, startTicks, rssPages].every(Number.isFinite)) {
throw new Error('LOAD_TEST_API_PID has an incomplete /proc stat record');
}
return { cpuTicks, startTicks, rssPages };
};
const startApiProcessSampler = async (pid: number, workspaceRoot: string) => {
const [commandLine, processCwd] = await Promise.all([
readFile(`/proc/${pid}/cmdline`, 'utf8').then((value) => value.replaceAll('\0', ' ')),
realpath(`/proc/${pid}/cwd`),
]);
if (!commandLine.includes('game-api') || processCwd !== workspaceRoot) {
throw new Error('LOAD_TEST_API_PID must be the game-api process from this workspace');
}
const [{ stdout: clockText }, { stdout: pageText }, initial] = await Promise.all([
execFileAsync('getconf', ['CLK_TCK']),
execFileAsync('getconf', ['PAGESIZE']),
readProcessStat(pid),
]);
const clockTicks = Number(clockText.trim());
const pageSize = Number(pageText.trim());
if (!Number.isFinite(clockTicks) || !Number.isFinite(pageSize)) {
throw new Error('could not determine process accounting units');
}
let sampling = true;
let maxRssPages = initial.rssPages;
let samples = 1;
const startedNs = process.hrtime.bigint();
const loop = (async () => {
while (sampling) {
await wait(250);
if (!sampling) break;
const sample = await readProcessStat(pid);
if (sample.startTicks !== initial.startTicks) throw new Error('game-api process changed during measurement');
maxRssPages = Math.max(maxRssPages, sample.rssPages);
samples += 1;
}
})();
return async () => {
sampling = false;
await loop;
const final = await readProcessStat(pid);
if (final.startTicks !== initial.startTicks) throw new Error('game-api process changed during measurement');
const elapsedSeconds = Number(process.hrtime.bigint() - startedNs) / 1_000_000_000;
const cpuSeconds = (final.cpuTicks - initial.cpuTicks) / clockTicks;
return {
pid,
samples,
elapsedMs: Math.round(elapsedSeconds * 1000),
cpuSeconds: Math.round(cpuSeconds * 1000) / 1000,
cpuPercentOfOneCore: Math.round((cpuSeconds / Math.max(elapsedSeconds, 0.001)) * 1000) / 10,
rssBytes: {
initial: initial.rssPages * pageSize,
final: final.rssPages * pageSize,
max: maxRssPages * pageSize,
},
};
};
};
const readHealth = async (baseUrl: string) => {
const response = await fetch(new URL('/healthz', baseUrl));
if (!response.ok) throw new Error(`game-api health returned HTTP ${response.status}`);
const body = (await response.json()) as Record<string, unknown>;
const pool = body.postgresPool;
return {
ok: body.ok === true,
postgresPool:
typeof pool === 'object' && pool !== null
? Object.fromEntries(
['max', 'total', 'active', 'idle', 'waiting']
.map((key) => [key, (pool as Record<string, unknown>)[key]])
.filter(([, value]) => typeof value === 'number')
)
: {},
};
};
const assertFrontendUrl = (value: string, config: LoadConfig): URL => {
const url = new URL(value);
if (!['http:', 'https:'].includes(url.protocol) || !isPrivateTargetHost(url.hostname)) {
throw new Error('LOAD_TEST_FRONTEND_URL must use HTTP(S) on a private or loopback host');
}
if (url.username || url.password || url.search || url.hash) {
throw new Error('LOAD_TEST_FRONTEND_URL must not contain credentials, query, or fragment');
}
const expectedSuffix = `/${config.isolation.postgresSchema.split('_')[2] ?? ''}/`;
if (!url.pathname.endsWith(expectedSuffix)) {
throw new Error(`LOAD_TEST_FRONTEND_URL must end with the configured profile path ${expectedSuffix}`);
}
return url;
};
export const measurePageNavigation = async (options: {
config: LoadConfig;
tokens: readonly string[];
workspaceRoot: string;
env?: NodeJS.ProcessEnv;
}) => {
const env = options.env ?? process.env;
const frontendUrl = assertFrontendUrl(env.LOAD_TEST_FRONTEND_URL ?? '', options.config);
const apiPid = Number(env.LOAD_TEST_API_PID);
if (!Number.isSafeInteger(apiPid) || apiPid <= 1) throw new Error('LOAD_TEST_API_PID must be a positive process id');
if (options.tokens.length !== options.config.capacity.authenticatedViewers) {
throw new Error('page navigation requires exactly one token per authenticated viewer');
}
const dwellMs = Number(env.LOAD_TEST_PAGE_DWELL_MS ?? '1000');
if (!Number.isSafeInteger(dwellMs) || dwellMs < 250 || dwellMs > 10_000) {
throw new Error('LOAD_TEST_PAGE_DWELL_MS must be an integer from 250 to 10000');
}
const routeLatencyMs = new Map<string, number[]>();
const routeSuccess = new Map<string, number>();
const routeErrors = new Map<string, number>();
const procedureRequests = new Map<string, number>();
const procedureErrors = new Map<string, number>();
const procedureLatencyMs = new Map<string, number[]>();
let pageErrors = 0;
let permissionErrors = 0;
const pendingResponses = new Set<Promise<void>>();
const browser = await chromium.launch({ headless: true });
const viewers: Array<{ page: Page; close: () => Promise<void> }> = [];
const activeRouteByPage = new Map<Page, string>();
let activeSseController: AbortController | null = null;
let activeSseTasks: Promise<void>[] = [];
let activeSseSampleTimer: NodeJS.Timeout | null = null;
const profile = options.config.isolation.postgresSchema.split('_')[2] ?? '';
try {
for (const token of options.tokens) {
const context = await browser.newContext({ viewport: { width: 1280, height: 900 } });
await context.addInitScript(
({ accessToken, profileName }) => {
const storage = (
globalThis as unknown as { localStorage: { setItem: (key: string, value: string) => void } }
).localStorage;
storage.setItem('sammo-game-token', accessToken);
storage.setItem('sammo-game-profile', profileName);
},
{ accessToken: token, profileName: profile }
);
const page = await context.newPage();
page.setDefaultTimeout(20_000);
page.setDefaultNavigationTimeout(20_000);
await page.goto(new URL('current-city', frontendUrl).toString(), { waitUntil: 'networkidle' });
const warmupText = await page.locator('body').innerText();
if (warmupText.includes('권한이 부족합니다')) {
throw new Error('privileged viewer fixture failed the current-city warmup');
}
viewers.push({ page, close: () => context.close() });
}
const sseMetrics = new PhaseMetrics();
const sseController = new AbortController();
activeSseController = sseController;
let activeSse = 0;
sseMetrics.sseActiveConnections.push(0);
const sseUrl = new URL(options.config.target.ssePath, options.config.target.baseUrl).toString();
const sseTasks = options.tokens.map((token) =>
runSseConnection({
url: sseUrl,
token,
signal: sseController.signal,
metrics: sseMetrics,
onActiveChange: (delta) => {
activeSse += delta;
sseMetrics.sseActiveConnections.push(activeSse);
},
})
);
activeSseTasks = sseTasks;
const connectionDeadline = Date.now() + 10_000;
while (activeSse !== options.tokens.length && Date.now() < connectionDeadline) await wait(50);
if (activeSse !== options.tokens.length) {
sseController.abort();
await Promise.allSettled(sseTasks);
throw new Error(`only ${activeSse} of ${options.tokens.length} SSE connections became active`);
}
const healthBefore = await readHealth(options.config.target.baseUrl);
let scheduledStartAtEpochMs: number | null = null;
if (env.LOAD_TEST_START_AT_EPOCH_MS) {
scheduledStartAtEpochMs = Number(env.LOAD_TEST_START_AT_EPOCH_MS);
if (!Number.isSafeInteger(scheduledStartAtEpochMs)) {
throw new Error('LOAD_TEST_START_AT_EPOCH_MS must be an integer epoch in milliseconds');
}
const waitMs = scheduledStartAtEpochMs - Date.now();
if (waitMs > 120_000) throw new Error('LOAD_TEST_START_AT_EPOCH_MS must be within the next two minutes');
if (waitMs > 0) await wait(waitMs);
}
const stopApiSampler = await startApiProcessSampler(apiPid, options.workspaceRoot);
const startedAt = new Date().toISOString();
const started = performance.now();
const deadline = started + options.config.capacity.turnIntervalMs;
const activeSseDuringMeasurement = [activeSse];
activeSseSampleTimer = setInterval(() => activeSseDuringMeasurement.push(activeSse), 250);
for (const [viewerIndex, viewer] of viewers.entries()) {
activeRouteByPage.set(viewer.page, 'warmup');
viewer.page.on('pageerror', () => {
pageErrors += 1;
});
viewer.page.on('response', (response) => {
const procedures = parseTrpcProcedures(response.url(), options.config.target.trpcPath);
if (procedures.length === 0) return;
const task = (async () => {
await response.finished().catch(() => null);
const timing = response.request().timing();
const latencyMs = timing.responseEnd;
for (const procedure of procedures) {
const key = `${activeRouteByPage.get(viewer.page) ?? 'unknown'}:${procedure}`;
increment(procedureRequests, key);
if (response.status() >= 400) increment(procedureErrors, `${key}:http-${response.status()}`);
if (latencyMs >= 0) {
const values = procedureLatencyMs.get(key) ?? [];
values.push(latencyMs);
procedureLatencyMs.set(key, values);
}
}
})().finally(() => pendingResponses.delete(task));
pendingResponses.add(task);
});
void viewerIndex;
}
await Promise.all(
viewers.map(async ({ page }, viewerIndex) => {
let iteration = 0;
while (performance.now() < deadline) {
const route = PAGE_ROUTES[(viewerIndex + iteration) % PAGE_ROUTES.length]!;
activeRouteByPage.set(page, route.name);
const routeStarted = performance.now();
try {
await page.goto(new URL(route.path, frontendUrl).toString(), { waitUntil: 'networkidle' });
const bodyText = await page.locator('body').innerText();
if (bodyText.includes('권한이 부족합니다')) {
permissionErrors += 1;
increment(routeErrors, `${route.name}:permission`);
} else {
increment(routeSuccess, route.name);
}
} catch {
increment(routeErrors, `${route.name}:navigation`);
} finally {
const values = routeLatencyMs.get(route.name) ?? [];
values.push(performance.now() - routeStarted);
routeLatencyMs.set(route.name, values);
}
iteration += 1;
if (performance.now() < deadline) await wait(dwellMs);
}
})
);
clearInterval(activeSseSampleTimer);
activeSseSampleTimer = null;
activeSseDuringMeasurement.push(activeSse);
await Promise.allSettled([...pendingResponses]);
const apiProcess = await stopApiSampler();
const elapsedMs = Math.round(performance.now() - started);
const healthAfter = await readHealth(options.config.target.baseUrl);
sseController.abort();
await Promise.allSettled(sseTasks);
sseMetrics.sseActiveConnections.push(activeSse);
return {
formatVersion: 1,
startedAt,
finishedAt: new Date().toISOString(),
configuredDurationMs: options.config.capacity.turnIntervalMs,
elapsedMs,
scheduling: {
scheduledStartAtEpochMs,
actualStartAtEpochMs: Date.parse(startedAt),
startDelayMs:
scheduledStartAtEpochMs === null ? null : Math.max(0, Date.parse(startedAt) - scheduledStartAtEpochMs),
},
fixture: {
name: options.config.name,
viewers: options.tokens.length,
npcGenerals: options.config.capacity.npcGenerals,
pageDwellMs: dwellMs,
},
browser: {
name: 'chromium',
contexts: viewers.length,
viewport: { width: 1280, height: 900 },
routes: Object.fromEntries(
PAGE_ROUTES.map((route) => [
route.name,
{
success: routeSuccess.get(route.name) ?? 0,
latencyMs: summarizeDistribution(routeLatencyMs.get(route.name) ?? []),
},
])
),
routeErrors: mapToObject(routeErrors),
pageErrors,
permissionErrors,
},
trpc: {
requests: mapToObject(procedureRequests),
errors: mapToObject(procedureErrors),
latencyMs: Object.fromEntries(
[...procedureLatencyMs.entries()]
.sort(([left], [right]) => left.localeCompare(right))
.map(([key, values]) => [key, summarizeDistribution(values)])
),
},
sse: {
attempts: sseMetrics.sseAttempts,
opened: sseMetrics.sseOpened,
closed: sseMetrics.sseClosed,
reconnects: sseMetrics.sseReconnects,
failures: sseMetrics.sseFailures,
privacyViolations: sseMetrics.ssePrivacyViolations,
events: mapToObject(sseMetrics.sseEvents),
activeConnections: summarizeDistribution(activeSseDuringMeasurement),
},
server: {
healthBefore,
healthAfter,
apiProcess,
},
};
} finally {
if (activeSseSampleTimer) clearInterval(activeSseSampleTimer);
activeSseController?.abort();
await Promise.allSettled(activeSseTasks);
await Promise.allSettled(viewers.map((viewer) => viewer.close()));
await browser.close();
}
};
+53 -4
View File
@@ -105,6 +105,7 @@ const includeSubMillisecondGameTick = (turnTime: Date): Date =>
export const measureTurnFlush = async (options: { export const measureTurnFlush = async (options: {
config: LoadConfig; config: LoadConfig;
confirmation: string; confirmation: string;
paced?: boolean;
env?: NodeJS.ProcessEnv; env?: NodeJS.ProcessEnv;
}) => { }) => {
const env = options.env ?? process.env; const env = options.env ?? process.env;
@@ -153,9 +154,8 @@ export const measureTurnFlush = async (options: {
const activityPromise = sampleActivity(); const activityPromise = sampleActivity();
const histogram = monitorEventLoopDelay({ resolution: 20 }); const histogram = monitorEventLoopDelay({ resolution: 20 });
histogram.enable(); let cpuStart: NodeJS.CpuUsage | null = null;
const cpuStart = process.cpuUsage(); let wallStartNs: bigint | null = null;
const wallStartNs = process.hrtime.bigint();
let maxRssBytes = process.memoryUsage().rss; let maxRssBytes = process.memoryUsage().rss;
const generalTransactionMs: number[] = []; const generalTransactionMs: number[] = [];
const monthlyTransactionMs: number[] = []; const monthlyTransactionMs: number[] = [];
@@ -167,6 +167,10 @@ export const measureTurnFlush = async (options: {
let endYearMonth: string | null = null; let endYearMonth: string | null = null;
let initialGeneralCount: number | null = null; let initialGeneralCount: number | null = null;
let finalGeneralCount: number | null = null; let finalGeneralCount: number | null = null;
let scheduledStartAtEpochMs: number | null = null;
let actualStartAtEpochMs: number | null = null;
const scheduleLagMs: number[] = [];
let backdatedGeneralTurns = 0;
let runError: unknown; let runError: unknown;
try { try {
@@ -189,6 +193,21 @@ export const measureTurnFlush = async (options: {
const boundary = getNextTickTime(initialState.lastTurnTime, tickMinutes); const boundary = getNextTickTime(initialState.lastTurnTime, tickMinutes);
let checkpoint: TurnCheckpoint | undefined = await runtime.stateStore.loadCheckpoint(); let checkpoint: TurnCheckpoint | undefined = await runtime.stateStore.loadCheckpoint();
if (env.LOAD_TEST_START_AT_EPOCH_MS) {
scheduledStartAtEpochMs = Number(env.LOAD_TEST_START_AT_EPOCH_MS);
if (!Number.isSafeInteger(scheduledStartAtEpochMs)) {
throw new Error('LOAD_TEST_START_AT_EPOCH_MS must be an integer epoch in milliseconds');
}
const waitMs = scheduledStartAtEpochMs - Date.now();
if (waitMs > 120_000) throw new Error('LOAD_TEST_START_AT_EPOCH_MS must be within the next two minutes');
if (waitMs > 0) await new Promise((resolve) => setTimeout(resolve, waitMs));
}
actualStartAtEpochMs = Date.now();
const pacingStarted = performance.now();
histogram.enable();
cpuStart = process.cpuUsage();
wallStartNs = process.hrtime.bigint();
const execute = async (target: Date, maxGenerals: number): Promise<TurnRunResult> => { const execute = async (target: Date, maxGenerals: number): Promise<TurnRunResult> => {
const started = performance.now(); const started = performance.now();
const result = await runtime!.stateManager.transaction(async () => { const result = await runtime!.stateManager.transaction(async () => {
@@ -219,6 +238,16 @@ export const measureTurnFlush = async (options: {
while (true) { while (true) {
const nextGeneral = await runtime.stateStore.loadNextGeneralTurnTime(); const nextGeneral = await runtime.stateStore.loadNextGeneralTurnTime();
if (!nextGeneral || nextGeneral.getTime() >= boundary.getTime()) break; if (!nextGeneral || nextGeneral.getTime() >= boundary.getTime()) break;
if (options.paced) {
const targetElapsedMs = nextGeneral.getTime() - initialState.lastTurnTime.getTime();
if (targetElapsedMs < 0) {
backdatedGeneralTurns += 1;
} else {
const remainingMs = targetElapsedMs - (performance.now() - pacingStarted);
if (remainingMs > 0) await new Promise((resolve) => setTimeout(resolve, remainingMs));
scheduleLagMs.push(Math.max(0, performance.now() - pacingStarted - targetElapsedMs));
}
}
const result = await execute(includeSubMillisecondGameTick(nextGeneral), 1); const result = await execute(includeSubMillisecondGameTick(nextGeneral), 1);
if (result.processedGenerals !== 1 || result.processedTurns !== 0) { if (result.processedGenerals !== 1 || result.processedTurns !== 0) {
throw new Error( throw new Error(
@@ -227,6 +256,11 @@ export const measureTurnFlush = async (options: {
} }
} }
if (options.paced) {
const remainingMs =
options.config.capacity.turnIntervalMs - (performance.now() - pacingStarted);
if (remainingMs > 0) await new Promise((resolve) => setTimeout(resolve, remainingMs));
}
const monthly = await execute(boundary, 200); const monthly = await execute(boundary, 200);
if (monthly.processedTurns !== 1) { if (monthly.processedTurns !== 1) {
throw new Error('turn-flush measurement did not cross exactly one monthly boundary'); throw new Error('turn-flush measurement did not cross exactly one monthly boundary');
@@ -275,6 +309,9 @@ export const measureTurnFlush = async (options: {
} }
await observer.disconnect(); await observer.disconnect();
histogram.disable(); histogram.disable();
if (cpuStart === null || wallStartNs === null || actualStartAtEpochMs === null) {
throw new Error('turn-flush measurement did not start its measured interval');
}
const elapsedMs = Number(process.hrtime.bigint() - wallStartNs) / 1_000_000; const elapsedMs = Number(process.hrtime.bigint() - wallStartNs) / 1_000_000;
const cpu = process.cpuUsage(cpuStart); const cpu = process.cpuUsage(cpuStart);
const cpuMs = (cpu.user + cpu.system) / 1_000; const cpuMs = (cpu.user + cpu.system) / 1_000;
@@ -287,7 +324,9 @@ export const measureTurnFlush = async (options: {
fixtureSha256: fixture.fixtureSha256, fixtureSha256: fixture.fixtureSha256,
capacity: options.config.capacity, capacity: options.config.capacity,
}, },
mode: 'chronological-one-general-per-transaction-plus-month-boundary', mode: options.paced
? 'wall-clock-paced-one-general-per-transaction-plus-month-boundary'
: 'chronological-one-general-per-transaction-plus-month-boundary',
startYearMonth, startYearMonth,
endYearMonth, endYearMonth,
elapsedMs: round(elapsedMs), elapsedMs: round(elapsedMs),
@@ -306,6 +345,16 @@ export const measureTurnFlush = async (options: {
monthlyTransaction: summarizeDistribution(monthlyTransactionMs), monthlyTransaction: summarizeDistribution(monthlyTransactionMs),
redisPublication: summarizeDistribution(publicationMs), redisPublication: summarizeDistribution(publicationMs),
}, },
scheduling: {
paced: options.paced === true,
configuredTurnIntervalMs: options.config.capacity.turnIntervalMs,
scheduledStartAtEpochMs,
actualStartAtEpochMs,
startDelayMs:
scheduledStartAtEpochMs === null ? null : Math.max(0, actualStartAtEpochMs - scheduledStartAtEpochMs),
backdatedGeneralTurns,
generalScheduleLagMs: summarizeDistribution(scheduleLagMs),
},
postgres: { postgres: {
statsScope: 'database-wide-including-observer-sampler', statsScope: 'database-wide-including-observer-sampler',
statsDelta: subtractDatabaseStats(beforeStats, afterStats), statsDelta: subtractDatabaseStats(beforeStats, afterStats),
+13 -1
View File
@@ -5,10 +5,22 @@ import path from 'node:path';
import test from 'node:test'; import test from 'node:test';
import { validateLoadConfig } from '../src/config.js'; import { validateLoadConfig } from '../src/config.js';
import { activateCapacityCoverage, assertFixtureIsolation, prepareCapacitySecrets } from '../src/fixture.js'; import {
activateCapacityCoverage,
assertFixtureIsolation,
prepareCapacitySecrets,
privilegedViewerPlacement,
} from '../src/fixture.js';
const samplePath = new URL('../config/300-users-900-npcs-5m.json', import.meta.url); const samplePath = new URL('../config/300-users-900-npcs-5m.json', import.meta.url);
void test('viewer placement stays in one nation and covers every privileged chief level deterministically', () => {
assert.deepEqual(
Array.from({ length: 10 }, (_, index) => privilegedViewerPlacement(index, 3, 7)),
[12, 10, 8, 6, 11, 9, 7, 5, 12, 10].map((officerLevel) => ({ nationId: 3, cityId: 7, officerLevel }))
);
});
void test('fixture accepts only the configured private schema and dedicated Redis database', async () => { void test('fixture accepts only the configured private schema and dedicated Redis database', async () => {
const config = validateLoadConfig(JSON.parse(await readFile(samplePath, 'utf8'))); const config = validateLoadConfig(JSON.parse(await readFile(samplePath, 'utf8')));
assert.doesNotThrow(() => assert.doesNotThrow(() =>
@@ -0,0 +1,19 @@
import assert from 'node:assert/strict';
import test from 'node:test';
import { parseTrpcProcedures } from '../src/pageNavigation.js';
void test('tRPC page-load requests are attributed for single and batched procedures', () => {
assert.deepEqual(
parseTrpcProcedures('http://127.0.0.1:15001/api/trpc/auth.status?batch=1&input=%7B%7D', '/api/trpc'),
['auth.status']
);
assert.deepEqual(
parseTrpcProcedures(
'http://127.0.0.1:15001/api/trpc/world.getMap%2Cworld.getMapLayout?batch=1',
'/api/trpc'
),
['world.getMap', 'world.getMapLayout']
);
assert.deepEqual(parseTrpcProcedures('http://127.0.0.1:15001/healthz', '/api/trpc'), []);
});