Skip to content

Parse protobuf battle journals and keep the JSON journal shape - #392

Open
blevouschurverantunt wants to merge 1 commit into
STFC-Mod:devfrom
blevouschurverantunt:fix/journal-protobuf
Open

blevouschurverantunt wants to merge 1 commit into
STFC-Mod:devfrom
blevouschurverantunt:fix/journal-protobuf

Conversation

@blevouschurverantunt

Copy link
Copy Markdown
Contributor

What changed and why

Since 2026-11-19 the game server answers POST /journals/get with protobuf instead of JSON. The body is a
ServiceResponse whose entity group 221 (TYPE_BATTLEREPORT) holds a JournalResponse. FetchBattle still parsed
the body as JSON, so every journal failed and battle log sync stopped for all targets. Fixes #391.

The mod now converts the JournalResponse back into the journal object the JSON server sent. It keeps the same key
names, nesting and JSON types, so sync consumers and the battlelog record stay unchanged. Bodies whose first
non-whitespace byte is { still take the JSON path.

  • src/mod/sync/journal.h, journal.cc (new): IsJsonBody() and JournalFromProtobuf(). They depend only on the
    generated protobuf code, nlohmann::json and the standard library. Conversion rules:
    • int64 ids stay JSON integers, all digits.
    • float32 values go out in shortest form (1.1, not 1.100000023841858), through to_chars and the JSON parser,
      as ShortNumber does.
    • Map keys become decimal strings, and battle_time is YYYY-MM-DDTHH:MM:SS UTC.
    • battle_log entries keep their number text, so 5.0 stays a float.
    • bridge_officers is the 10-slot crew array with nulls, {} for an NPC side and null for an armada.
    • Malformed, truncated or oversized bodies are refused without throwing.
  • src/mod/sync/battlelogs.cc: FetchBattle picks the path from the first byte and logs one error with the battle
    id, body size and reason when a protobuf journal does not convert. As before, it then retries. Participants now
    reads deployed_fleets as an object keyed by fleet id, as journals have always sent it. The old is_array()
    check skipped it, so only initiator_id/target_id were resolved. Empty uids are skipped.
  • src/mod/game/proto/Digit.PrimeServer.Models.proto: 10 new messages (JournalResponse, JournalFleetData,
    JournalFleetOfficerEntry, BridgeCrewOfficer, CargoProto, JournalChestDrop, JournalContextData,
    JournalContextDataInner, JournalMultitargetArmadaInfo, JournalGalacticAnomalyData). They were recovered from
    the client's message descriptors and checked against live responses. These are pure additions: existing messages,
    including DeployedFleet and JournalHeaderProto, are untouched.
  • zig/deps/protobuf_roots.txt: JournalResponse and ServiceResponse.
  • tests/sync_logic.cc, zig/tests.zig: a protobuf journals test in test-sync.
  • CONTRIBUTING.md: sync/journal.cc in the sync ownership table.

Converted journals carry "journal_src":"pb1" inside the journal object, so a consumer can tell them apart.

Keys the server no longer sends are left out, never set to 0 or invented. They return automatically if the server
sends the backing field again:

  • Journal: num_references.
  • Chest drop: loot_roll_key, loot_roll_type.
  • Chests gained, params: chest_name, items, chest_item_<id>_amount/_type, source, id.
  • Fleet data (both sides): faction_id, active_officer_slots; and fleet_id on an armada side, whose old value is
    not derivable.
  • Deployed fleet:
    • battle_start_time, state, is_active and the other is_* flags;
    • node_id, course_id, latest_course_vector_x/_y;
    • internal_stats, mining_amount_bonus_stats, node_address;
    • active_buffs[].attributes.

xp_gained and warp_data stay null, as in every JSON journal. Values that were server doubles are now float32
on the wire. Integral values are exact; fractional stats and ratings above 2^24 can differ in the last digits.

Validation

  • Run locally on Windows with Zig 0.16: zig build mods build-tests -Dtarget=x86_64-windows-msvc,
    zig build test-sync test-mod test-codegen, the full ReleaseFast DLL build, and zig build fmt.
  • The new protobuf journals test builds a synthetic player-vs-NPC ServiceResponse with the protobuf API. It
    compares the result with an exact expected journal, number kinds included. It also checks that a JSON body is
    detected, and that empty, garbage, truncated, wrong-group and id-less bodies are refused.
  • Parity: the converter's output for 20 captured journals (battle types 2 and 5) matches a separate reference decoder,
    integers digit for digit and all keys and types. The only differences are float32 values that sit exactly halfway
    between two shortest forms, where to_chars rounds half to even. A value-level self-check of 1,430 assertions
    against the decoded wire passed.
  • In game (the same converter, ported to the 1.1.10 release layout): one session of about four minutes sent 84 battle logs to a private sync endpoint, all hostile fights (battle type 5), 83 wins and one loss. All 84 were accepted and parsed by the consumer with the same columns as journals from before the server change, apart from what was derived from the omitted keys. The mod log had no errors, where sessions before the fix logged a parse error for every battle.
    The dev build is compile-, unit- and parity-tested only, not run in game.
  • CI: build-win currently fails on the runner's newer MSVC toolset, with STL1000 in every unit including
    dependencies. This is unrelated to this change.

Settings and platforms

  • No new or changed settings. Battle fetching stays opt-in (battlelogs).
  • Untested battle types: armada (8), PvP, station and base raids, hazards and wave defense. They follow the same
    rules. An armada keeps deployed_fleets keyed by fleet id, bridge_officers is null and the side's fleet_id is
    left out. A side without a fleet leaves out deployed_fleet.
  • For sync-site maintainers: the record shape is unchanged apart from the omitted keys above and the journal_src
    marker. Match armada participants by hull or ship id, not by array position.
  • Not built or run on macOS locally. The converter avoids float from_chars, which Apple's libc++ lacks
    before macOS 26.

🤖 Generated with Claude Code

Since 2026-11-19 the game server answers POST /journals/get with protobuf
instead of JSON: a ServiceResponse whose entity group 221 (BattleReport)
holds a JournalResponse. The JSON parse failed, so no battle journal
reached any sync target.

The mod now converts the JournalResponse into the journal object the JSON
server sent: same keys, nesting and JSON types, so sync consumers keep
working unchanged. Keys the server no longer sends are left out rather
than invented, and converted journals carry "journal_src":"pb1". Bodies
that start with '{' still take the JSON path.

The battle report messages are added to the models proto as new messages
only; existing messages are untouched. Participant lookup now reads
deployed_fleets as the object it is in journals.

Validated with a synthetic unit test and against 20 captured journals of
single-fleet battles. Armada, PvP and station battles are untested.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant