Network Tracing¶

Audience

This document is written to be readable by both human integrators and coding-agent LLMs (e.g. Claude Code). The recipe sections follow a fixed shape — file, function, insertion point, literal macro line, why — so an agent can apply each site without ambiguity. Human readers can safely skim subsections that don't apply to their engine configuration.

Engine version

This recipe was authored from a UE 5.7 codebase but is expected to apply across UE 5.x releases. Function names referenced below have been stable across recent 5.x versions, but line numbers and surrounding code will differ in your fork — anchor on the function name and landmark statement, not on line numbers. The rest of the page refers to the target as "UE 5".

For macro signatures, parameters, and semantics, see Instrumentation API → Network Tracing.

1. Overview¶

Micromegas net tracing captures per-connection replication traffic at the UE engine layer and lands it in the lakehouse as a structured event stream. Unlike generic spans or logs, net events carry a bit-size attribution model: each root object or RPC records how many bits of content it contributed to a connection, letting you answer "which actors/properties dominate per-connection bandwidth?" directly in SQL.

Net tracing is separate from spans/logs/metrics because:

It needs tight coupling to replication internals (bit stream positions, bunch writers, content blocks) that generic spans can't observe.
It has its own verbosity ladder (packets / root objects / objects / properties) independent of span sampling.
It produces a nested event hierarchy — Connection → Object → Subobject → Property or Connection → RPC → Property — that mirrors the wire format rather than call stacks.

Event hierarchy. Net events stream into a dedicated net tagged stream. The wire shape depends on whether the project uses classic replication or Iris.

Classic incoming example (ProcessBunch path):

NetConnectionBeginEvent       connection="127.0.0.1:7777"  is_outgoing=false
  NetObjectBeginEvent         name="PlayerPawn_C_0"
  NetPropertyEvent            name="Health"            bit_size=16
  NetPropertyEvent            name="ReplicatedMovement" bit_size=112
  NetObjectEndEvent           bit_size=512
  NetObjectBeginEvent         name="InventoryComponent"   # peer, not nested
  NetPropertyEvent            name="GoldCount"         bit_size=24
  NetObjectEndEvent           bit_size=88
  NetRPCBeginEvent            name="ClientNotifyHit"
  NetPropertyEvent            name="HitLocation"       bit_size=96
  NetRPCEndEvent              bit_size=160
NetConnectionEndEvent         bit_size=760             # sum of root Object/RPC bits

Iris outgoing example (WriteObjectAndSubObjects path):

NetConnectionBeginEvent       connection="127.0.0.1:7777"  is_outgoing=true
  NetObjectBeginEvent         name="PlayerPawn_C_0"
    NetPropertyEvent          name="Health"            bit_size=16
    NetObjectBeginEvent       name="WeaponComponent"   # nested inside parent
      NetPropertyEvent        name="AmmoCount"         bit_size=24
    NetObjectEndEvent         bit_size=88
  NetObjectEndEvent           bit_size=520
NetConnectionEndEvent         bit_size=520

The classic shape emits subobjects as peers at depth 0; Iris emits them nested at depth 1+. Both are correct — they mirror the actual replication traversal order in each system. See Architecture → Classic vs Iris hierarchy.

2. Architecture¶

This section is load-bearing: the recipe in §3 makes judgment calls (which macro form, which bit source, when to suspend) that only make sense once the architecture is understood. When the implementer faces a novel call site or a renamed function, this section is the fallback.

Data flow¶

flowchart LR
    A[Engine code<br/>replication path] -->|MICROMEGAS_NET_* macros| B[Dispatch]
    B --> C[NetTraceWriter<br/>depth counters,<br/>verbosity gate]
    C --> D[NetStream<br/>tagged 'net']
    D --> E[HttpEventSink]
    E --> F[Telemetry ingestion<br/>→ object store]
    F --> G[Lakehouse<br/>parse_block, views]

Verbosity model¶

Runtime verbosity is a 0–4 enum:

Level	Name	Emits
0	`Off`	Nothing
1	`Packets`	Connection scopes only
2	`RootObjects`	+ root object scopes (depth 0)
3	`Objects`	+ nested object scopes (depth 1+)
4	`Properties`	+ per-property leaf events, + RPC scopes (root and nested)

Depth-based gating rule. Inside NetTraceWriter:

Object scopes: depth 0 = at least level RootObjects; depth 1+ = at least level Objects
Property leaves = level Properties
RPC scopes (Begin/End events) = level Properties

Default: level 2 (RootObjects) — production setting, cheap enough to run continuously.

Note that root RPC bit_size (ObjectDepth == 0 at EndRPC) still accumulates into NetConnectionEndEvent.bit_size at every verbosity ≥ Packets, regardless of whether the NetRPC* events themselves are emitted. Only the per-RPC event records require level 4.

Snapshot invariant (Decision 6). The writer captures EffectiveVerbosity at the outermost BeginConnection and uses that snapshot for every gating decision in the scope. CVar-driven changes take effect at the next outer connection scope, never mid-scope. This eliminates orphaned Begin/End pairs and partially-gated subtrees.

Content-attribution vs wire-bit semantics¶

Net tracing emits two distinct families of data. Pick the right one for your question:

Question	Use
Which actors/properties dominate per-connection bandwidth?	`net_spans` rows where `kind = 'object'` and `name` matches an actor / `kind = 'rpc'` / `kind = 'property'`
Which framing class (header, padding, exports) dominates?	`net_spans` rows where `kind = 'object'` and `name` is one of `PacketHeader`, `BunchHeader`, `BitPadding`, `PacketHandler`, `NetGUIDExport`, `Retransmit`
What's the connection's true bandwidth on the wire?	`sum(net.packet_sent_bits)` filtered by `connection_name`
What fraction is unattributed (control bunches, voice, etc.)?	`1 − sum(net_spans.bit_size where depth = 1) / sum(net.packet_*_bits)`

bit_size on the NetConnectionEndEvent is the sum of all root-level (depth 1) Object and RPC bits inside the connection scope — content scopes (§3.7 / §3.10 / §3.11) and the named wire-framing events from §3.14. With both kinds instrumented, that sum closes to within a few percent of the wire metric. Pre-§3.14, it only contained content and was deliberately a lower bound — leaving the residual to identify framing overhead.

The depth-based separation is what makes this clean: actor/RPC events at depth 1 sit alongside framing events at depth 1, both peers to each other. Nested scopes (Iris subobjects, classic fast-array) sit at depth 2+ and roll into their parent's bit_size rather than into the connection total — preventing double-count.

The ObjectDepth == 0 gate in EndObject / EndRPC prevents nested-scope double-counting: a receive RPC inside a subobject scope measures the same Reader delta as the outer object, so only the outer contribution is accumulated. Don't try to defeat this — see Pitfalls → RPC bits double-count.

Classic vs Iris hierarchy¶

Classic — UActorChannel::ReplicateActor opens a root-actor scope, replicates the actor, closes the scope, then calls DoSubObjectReplication, which opens a peer scope at depth 0 for each subobject. The two scopes are siblings in the event stream.

Why: classic's actor/subobject split is serialized into separate scopes because they run sequentially in different bunches.

Iris — FReplicationWriter::WriteObjectAndSubObjects is recursive: the parent scope stays open while child scopes run inside it, producing a nested tree in the event stream.

Why: Iris serializes an object and all its subobjects into a single batch, so the scope structure follows the recursion.

Consequence for queries. Queries that walk hierarchy must handle both shapes — or filter on a per-process flag that identifies Iris vs classic processes.

Non-nesting invariant (Decision 6)¶

Connection scopes do not nest. Only the outermost BeginConnection emits and resets writer state; nested ones are classified at Begin time and recorded on a per-scope EScopeKind stack so the matching End knows exactly what bookkeeping it owns. The classification:

Inner vs outer	Classification	Behavior
`SuspendDepth > 0` at Begin	`OuterSuspendedOut`	Begin/End are both no-ops (same as a top-level scope under suspend).
Same name, same direction	`Absorbed` (silent)	Inner is a structural duplicate (e.g. RPC-forced ReplicateActor inside §3.3). No log.
Different name, same direction	`Absorbed` (logged at `VeryVerbose`)	Inner bits roll into outer's `AccumulatedBits` if they close at root depth, or are dropped if they close inside an outer object scope.
Direction mismatch	`Suspended` (logged at `VeryVerbose`)	The writer auto-suspends for the inner scope's lifetime so its `OBJECT_EVENT`s do not emit nested under a wrong-direction parent.

The direction-mismatch auto-suspend matters because direction changes the bit source — children measured against an outgoing writer would be reported under an incoming-reader parent, and sum(children) > parent would surface in queries. Auto-suspending eliminates the bad data outright instead of trying to reconcile it downstream.

Practical guidance:

Instrument so scopes don't overlap naturally (one per replication entry point) — this remains the goal.
For paths that process packets/bunches but shouldn't contribute to attribution (demo, replay), use MICROMEGAS_NET_SUSPEND_SCOPE() at the entry point.
A repeating LogMicromegasNet with the same (inner, outer) pair points to a real re-entry path you may want to investigate.

Suspend mechanism¶

MICROMEGAS_NET_SUSPEND_SCOPE() zeroes out every MICROMEGAS_NET_* call inside its lifetime without touching depth counters. Safe to nest under an active live scope — the live scope resumes unchanged when the suspend scope exits.

Use it for:

Demo recording / replay scrubbing
Server-side simulation that re-runs replication
Any synchronous engine callback that can fire from inside a live packet/replication scope and shouldn't be attributed

RAII everywhere¶

Every MICROMEGAS_NET_* macro is RAII. There are no public BEGIN_* / END_* macros. Do not call Dispatch::NetEndConnection / NetEndObject / NetEndRPC directly — only the scope guard destructors are authorized callers.

Early returns are safe: the destructors close scopes automatically when the enclosing block exits.

3. Engine instrumentation recipe¶

Every site below follows the same shape:

File — UE 5 path (Engine/Source/... or Engine/Plugins/...).
Function — qualified signature.
Insertion point — a named landmark statement in the existing code.
Insert — the literal macro line (copy-pasteable C++).
Why — one sentence on what breaks or is misattributed if skipped.

Line numbers will differ in your fork. Anchor on the function name and landmark statement.

Connection name strategy. Every connection-scope site reads a cached FName MmDisplayName member added to UNetConnection. Caching matters because (a) the resolution chain is non-trivial — PlayerName|PlayerId → PlayerId → addr:port → GetFName() — and (b) the same name appears in dozens of trace events per tick on a busy server. Recomputing on every site would defeat the verbosity-2 "always on" budget.

Refresh MmDisplayName at the lifecycle points where its inputs become valid or change:

UNetConnection::HandleClientPlayer / OnHandshakeComplete — once the connection is associated with a player controller
APlayerController::OnRep_PlayerState and APlayerState::OnRep_PlayerName / OnRep_UniqueId — when player identity arrives or changes
Any custom hook your project uses for "player identity finalized"

The refresh function picks the first non-empty value from the priority chain above and assigns it to MmDisplayName. Until the first refresh fires, the field falls back to Connection->GetFName() so trace events still carry a usable identifier.

Sites then read it directly:

MICROMEGAS_NET_CONNECTION_SCOPE(Connection->MmDisplayName, /*bIsOutgoing=*/ false);

Adjust Connection-> if the local handle is named differently (e.g. Params.Connection, NetConnection).

3.1 Incoming packet scope¶

File: Engine/Source/Runtime/Engine/Private/NetConnection.cpp
Function: UNetConnection::ReceivedPacket
Insertion point: after the if (PacketNotify.ReadHeader(Reader)) success branch, before InTraceCollector setup.

Insert:

// micromegas net trace
MICROMEGAS_NET_CONNECTION_SCOPE(MmDisplayName, /*bIsOutgoing=*/ false);

Why: natural function-scoped boundary for one received UDP packet; RAII closes the scope on every early-return path. MmDisplayName is the cached connection identifier described above.

Then, near the existing UE_NET_TRACE_PACKET_RECV(...) line at the end of the function, add the wire-bit metric:

Insert:

// micromegas net trace — physical wire bits
MICROMEGAS_IMETRIC("net", MicromegasTracing::Verbosity::Med,
                   TEXT("net.packet_received_bits"), TEXT("bits"),
                   Reader.GetNumBits());

3.2 Outgoing packet metric¶

File: Engine/Source/Runtime/Engine/Private/NetConnection.cpp
Function: UNetConnection::FlushNet
Insertion point: immediately after the UE_NET_TRACE_PACKET_SEND(...) line, before the LowLevelSend call.

Insert:

// micromegas net trace — physical wire bits
MICROMEGAS_IMETRIC("net", MicromegasTracing::Verbosity::Med,
                   TEXT("net.packet_sent_bits"), TEXT("bits"),
                   SendBuffer.GetNumBits());

Why: physical wire-bit counter. Do not open a MICROMEGAS_NET_CONNECTION_SCOPE here — outgoing scopes live at higher-level replication entry points (§3.3, §3.4, §3.5).

3.3 Classic server replication scope¶

File: Engine/Source/Runtime/Engine/Private/NetDriver.cpp
Function: UNetDriver::ServerReplicateActors_ForConnection
Insertion point: first executable line of the function body, inside the #if WITH_SERVER_CODE guard.

Insert:

// micromegas net trace
MICROMEGAS_NET_CONNECTION_SCOPE(Params.Connection->MmDisplayName, /*bIsOutgoing=*/ true);

Why: wraps the per-connection actor-replication walk on a vanilla server (no RepGraph). Skipped entirely when UReplicationDriver is set — that path needs §3.4 instead.

3.4 RepGraph server replication scope¶

File: Engine/Plugins/Runtime/ReplicationGraph/Source/Private/ReplicationGraph.cpp
Function: UReplicationGraph::ServerReplicateActors
Insertion point: inside the for (UNetReplicationGraphConnection* ConnectionManager : Connections) loop, after the early-continue checks (saturation, invalid connection) and before the gather phase.

Insert:

// micromegas net trace
MICROMEGAS_NET_CONNECTION_SCOPE(NetConnection->MmDisplayName, /*bIsOutgoing=*/ true);

Why: RepGraph short-circuits UNetDriver::ServerReplicateActors, so the §3.3 scope never opens for RepGraph traffic. Symptom of missing this site: object/property events appear outside any BeginConnection event — especially ~120 actors per tick with bit_size:0 from the relevancy walk.

3.5 Iris outgoing scope¶

File: Engine/Source/Runtime/Engine/Private/Net/Experimental/Iris/DataStreamChannel.cpp
Function: UDataStreamChannel::WriteData
Insertion point: after the if (Result == EWriteResult::NoData) return; early return, before the main write loop.

Insert:

// micromegas net trace
MICROMEGAS_NET_CONNECTION_SCOPE(Connection->MmDisplayName, /*bIsOutgoing=*/ true);

Why: opens only when there's actual work — empty ticks don't produce empty scopes. Iris bypasses both classic scopes (§3.3, §3.4).

3.6 Queued-bunch deferred-flush scope¶

File: Engine/Source/Runtime/Engine/Private/DataChannel.cpp
Function: UActorChannel::ProcessQueuedBunches
Insertion point: inside the bHasTimeToProcess && PendingGuidResolves.Num() == 0 branch, immediately before the while ((BunchIndex < QueuedBunches.Num()) && ...) drain loop.

Insert:

// micromegas net trace
MICROMEGAS_NET_CONNECTION_SCOPE(Connection->MmDisplayName, /*bIsOutgoing=*/ false);

Why: queued bunches (parked waiting for NetGUID resolution) are flushed from UActorChannel::Tick(), outside the §3.1 ReceivedPacket scope. Without this, multi-kilobit InventoryManagerComponent and similar large initial-state objects orphan ~1 s after the packet arrives.

3.7 Classic object scopes¶

Four sites.

3.7.1 `UActorChannel::ReplicateActor` (root actor, send)¶

File: Engine/Source/Runtime/Engine/Private/DataChannel.cpp
Insertion point: inside the // The Actor block, immediately after the existing UE_NET_TRACE_OBJECT_SCOPE(ActorReplicator->ObjectNetGUID, Bunch, ...) line.

Insert:

// micromegas net trace
MICROMEGAS_NET_OBJECT_SCOPE(Actor->GetFName(), Bunch.GetNumBits());

Why: root actor body. Bit source is the bunch writer because that's the wire stream the actor's bits land in.

3.7.2 `UActorChannel::WriteSubObjectInBunch` (subobject, send)¶

File: Engine/Source/Runtime/Engine/Private/DataChannel.cpp
Insertion point: immediately after the existing UE_NET_TRACE_OBJECT_SCOPE(ObjectReplicator->ObjectNetGUID, Bunch, ...) line, before the bWroteSomething = ObjectReplicator.Get().ReplicateProperties(Bunch, ObjRepFlags); call.

Insert:

// micromegas net trace
MICROMEGAS_NET_OBJECT_SCOPE(Obj->GetFName(), Bunch.GetNumBits());

Why: subobject body. In classic, this opens at depth 0 (peer to the root actor's scope, not nested) — the §3.7.1 scope already closed by the time WriteSubObjectInBunch is called from DoSubObjectReplication.

3.7.3 `UActorChannel::ProcessBunch` content-block receive loop¶

File: Engine/Source/Runtime/Engine/Private/DataChannel.cpp
Insertion point: immediately after TSharedRef<FObjectReplicator>& Replicator = FindOrCreateReplicator(RepObj);, before bool bHasUnmapped = false;.

Insert:

// micromegas net trace
MICROMEGAS_NET_OBJECT_SCOPE(RepObj->GetFName(), Reader.GetPosBits());

Why: covers both root actors and subobjects on receive — ProcessBunch's loop body handles both. Bit source is Reader (the local FNetBitReader containing the content-block payload), not Bunch (the outer packet reader — using Bunch here would measure the wrong delta).

3.7.4 `FObjectReplicator::ReplicateCustomDeltaProperties` (fast arrays / custom delta)¶

File: Engine/Source/Runtime/Engine/Private/DataReplication.cpp
Insertion point: inside the per-property loop, after the trace-collection reset block, before the TSharedPtr<INetDeltaBaseState>& OldState = ... assignment.

Insert:

// micromegas net trace
MICROMEGAS_NET_OBJECT_SCOPE(Property->GetFName(), TempBitWriter.GetNumBits());

Why: outer container for fast-array / custom-delta properties. Opens at depth 1 (nested inside the root-actor scope, since ReplicateCustomDeltaProperties is called from ReplicateProperties_r inside ReplicateActor's scope) — the one classic exception to the peer-not-nested rule.

3.8 Classic property scopes¶

Five sites in Engine/Source/Runtime/Engine/Private/RepLayout.cpp. The macro form (flat vs scope) is dictated by what's already available; pick wrong and you create unnecessary divergence.

3.8.1 `SendProperties_r`, shared-property branch¶

Insertion point: immediately after NETWORK_PROFILER(GNetworkProfiler.TrackReplicateProperty(ParentCmd.Property, SharedPropInfo->PropBitLength, nullptr));.

Insert:

// micromegas net trace
MICROMEGAS_NET_PROPERTY(Cmd.Property->GetFName(), SharedPropInfo->PropBitLength);

Why flat: SharedPropInfo->PropBitLength is pre-computed, no delta needed.

3.8.2 `SendProperties_r`, regular-property branch¶

Insertion point: immediately after NETWORK_PROFILER(GNetworkProfiler.TrackReplicateProperty(ParentCmd.Property, NumEndBits - NumStartBits, nullptr));.

Insert:

// micromegas net trace
MICROMEGAS_NET_PROPERTY(Cmd.Property->GetFName(), NumEndBits - NumStartBits);

Why flat: NumStartBits / NumEndBits are pre-existing engine locals captured for NETWORK_PROFILER — reuse them in a single line. Switching to the scope form would require wrapping NetSerializeItem in a new brace-bounded block to avoid measuring the trailing ENABLE_PROPERTY_CHECKSUMS code, increasing divergence.

3.8.3 `SendProperties_BackwardsCompatible_r`¶

Insertion point: immediately after NETWORK_PROFILER(GNetworkProfiler.TrackReplicateProperty(Cmd.Property, NumEndBits - NumStartBits, nullptr));.

Insert:

// micromegas net trace
MICROMEGAS_NET_PROPERTY(Cmd.Property->GetFName(), NumEndBits - NumStartBits);

Why flat: same reasoning as §3.8.2.

3.8.4 `ReceiveProperties_r`¶

Insertion point: inside the per-property block, immediately after the existing UE_NET_TRACE_DYNAMIC_NAME_SCOPE(Cmd.Property->GetFName(), Params.Bunch, ...) line, before the if (ReceivePropertyHelper(...)) call.

Insert:

// micromegas net trace
MICROMEGAS_NET_PROPERTY_SCOPE(Cmd.Property->GetFName(), Params.Bunch.GetPosBits());

Why scope: there's no pre-existing bit capture here, and the scope form absorbs what would otherwise be a dedicated MmStartBits capture — one line instead of three.

3.8.5 `ReceiveProperties_BackwardsCompatible_r`¶

Insertion point: after the TempReader.ResetData(Reader, NumBits) and error-check block, before if (NetFieldExportGroup->NetFieldExports[NetFieldExportHandle].bIncompatible) continue;.

Insert:

// micromegas net trace
MICROMEGAS_NET_PROPERTY(NetFieldExportGroup->NetFieldExports[NetFieldExportHandle].ExportName, NumBits);

Why flat: NumBits is read directly from the stream as a packed int — no delta needed.

3.9 Classic RPCs¶

Three sites.

3.9.1 `ProcessRemoteFunctionForChannelPrivate`, send-side¶

File: Engine/Source/Runtime/Engine/Private/NetDriver.cpp
Function: UNetDriver::ProcessRemoteFunctionForChannelPrivate
Insertion point: after TSharedPtr<FRepLayout> RepLayout = GetFunctionRepLayout(Function);, immediately before RepLayout->SendPropertiesForRPC(Function, Ch, TempWriter, Parms);.

Insert:

// micromegas net trace
MICROMEGAS_NET_RPC_SCOPE(Function->GetFName(), TempWriter.GetNumBits());

Why: TempWriter is fresh per-RPC, so GetNumBits() gives the full payload size via the destructor delta after SendPropertiesForRPC returns.

3.9.2 `ProcessRemoteFunctionForChannelPrivate`, RPC-forced ReplicateActor block¶

File: Engine/Source/Runtime/Engine/Private/NetDriver.cpp
Function: UNetDriver::ProcessRemoteFunctionForChannelPrivate
Insertion point: earlier in the same function — the Ch->SetForcedSerializeFromRPC(true); Ch->ReplicateActor(); Ch->SetForcedSerializeFromRPC(false); triple. Wrap it in a brace-bounded block with a connection scope:

Insert (replaces the triple):

// micromegas net trace
{
    MICROMEGAS_NET_CONNECTION_SCOPE(Connection->MmDisplayName, /*bIsOutgoing=*/ true);

    Ch->SetForcedSerializeFromRPC(true);
    Ch->ReplicateActor();
    Ch->SetForcedSerializeFromRPC(false);
}

Why: when an RPC fires for an actor whose channel isn't up to date, ReplicateActor is forced — emitting a full actor + subobject + fast-array walk that needs connection attribution. The brace bounds the RAII lifetime so EndConnection fires immediately after SetForcedSerializeFromRPC(false). If reached while another scope is open (e.g. inside §3.3), Decision-6 absorbs the nested Begin/End as no-ops.

3.9.3 `FObjectReplicator::ReceivedRPC`¶

File: Engine/Source/Runtime/Engine/Private/DataReplication.cpp
Insertion point: immediately before FuncRepLayout->ReceivePropertiesForRPC(Object, LayoutFunction, OwningChannel, Reader, Parms, UnmappedGuids);.

Insert:

// micromegas net trace — FunctionName is already in scope from earlier FieldCache->Field.GetFName()
MICROMEGAS_NET_RPC_SCOPE(FunctionName, Reader.GetPosBits());

Why: receive-side RPC body. Reader is the bit reader the RPC parameters are deserialized from.

3.10 Iris properties¶

Eight sites in Engine/Source/Runtime/Net/Iris/Private/Iris/ReplicationSystem/ReplicationOperations.cpp — one per Serialize / Deserialize variant.

Pattern (identical at every site): inside the for (uint32 MemberIt = 0; MemberIt < MemberCount; ++MemberIt) loop, immediately before the MemberSerializerDescriptor.Serializer->{Serialize,Deserialize,SerializeDelta,DeserializeDelta}(Context, Args); call. The macro line is:

// micromegas net trace — Serialize variants
MICROMEGAS_NET_PROPERTY_SCOPE(MemberDebugDescriptors[MemberIt].DebugName->Name,
                              Context.GetBitStreamWriter()->GetPosBits());

with GetBitStreamWriter() for Serialize* variants, GetBitStreamReader() for Deserialize* variants.

The eight functions:

Function	Bit stream	Serializer call
`Serialize`	`GetBitStreamWriter()`	`Serializer->Serialize`
`Deserialize`	`GetBitStreamReader()`	`Serializer->Deserialize`
`SerializeDelta`	`GetBitStreamWriter()`	`Serializer->SerializeDelta`
`DeserializeDelta`	`GetBitStreamReader()`	`Serializer->DeserializeDelta`
`SerializeWithMask`	`GetBitStreamWriter()`	`Serializer->Serialize`
`DeserializeWithMask`	`GetBitStreamReader()`	`Serializer->Deserialize`
`SerializeDeltaWithMask`	`GetBitStreamWriter()`	`Serializer->SerializeDelta`
`DeserializeDeltaWithMask`	`GetBitStreamReader()`	`Serializer->DeserializeDelta`

For the four WithMask variants, the macro lives inside the if (ChangeMask.IsAnyBitSet(...)) (or equivalent changemask-check) block so only dirty properties trace.

Why: Iris has no NETWORK_PROFILER capture to reuse, and the bit position is a GetPosBits delta across the serializer call — the scope form is the right tool. DebugName->Name is const TCHAR* (process-lifetime stable, allocated by CreatePersistentNetDebugName) — pass directly; the macro wraps it in StaticStringRef internally.

3.11 Iris objects¶

Two sites.

3.11.1 `FReplicationWriter::SerializeObjectStateDelta`¶

File: Engine/Source/Runtime/Net/Iris/Private/Iris/ReplicationSystem/ReplicationWriter.cpp
Insertion point: first executable line of the function body.

Insert:

// micromegas net trace
MICROMEGAS_NET_OBJECT_SCOPE(
    (ObjectData.Protocol && ObjectData.Protocol->DebugName) ? ObjectData.Protocol->DebugName->Name : TEXT("Unknown"),
    Context.GetBitStreamWriter()->GetPosBits());

Why: per-object Iris send entry point, called for both root objects and subobjects within a batch. Null-guard on Protocol / DebugName falls back to the literal TEXT("Unknown") (also process-lifetime stable). Depth-based gating in NetTraceWriter decides which level emits.

3.11.2 `FReplicationReader::DeserializeObjectStateDelta`¶

File: Engine/Source/Runtime/Net/Iris/Private/Iris/ReplicationSystem/ReplicationReader.cpp
Insertion point: first executable line of the function body.

Insert:

// micromegas net trace
MICROMEGAS_NET_OBJECT_SCOPE(
    (ObjectData.Protocol && ObjectData.Protocol->DebugName) ? ObjectData.Protocol->DebugName->Name : TEXT("Unknown"),
    Context.GetBitStreamReader()->GetPosBits());

Why: symmetric receive path with the same null-guard pattern.

3.12 Iris RPCs¶

Four sites in Engine/Source/Runtime/Net/Iris/Private/Iris/ReplicationSystem/NetBlob/NetRPC.cpp.

3.12.1 `FNetRPC::SerializeWithObject`¶

Insertion point: after FNetBitStreamWriter& Writer = *Context.GetBitStreamWriter();, before const uint32 HeaderPos = Writer.GetPosBits();.

Insert:

// micromegas net trace
MICROMEGAS_NET_RPC_SCOPE(BlobDescriptor->DebugName->Name, Writer.GetPosBits());

Why: BlobDescriptor is a member field, valid at function entry — open the scope as early as possible to capture the full RPC including header.

3.12.2 `FNetRPC::Serialize`¶

Insertion point: same shape — after the FNetBitStreamWriter& Writer = ... line, before the HeaderPos capture.

Insert:

// micromegas net trace
MICROMEGAS_NET_RPC_SCOPE(BlobDescriptor->DebugName->Name, Writer.GetPosBits());

3.12.3 `FNetRPC::DeserializeWithObject`¶

Insertion point: after ResolveFunctionAndObject(...) succeeds and UE_NET_TRACE_SET_SCOPE_NAME(TraceScope, BlobDescriptor->DebugName); runs, immediately before InternalDeserializeBlob(Context);.

Insert:

// micromegas net trace
MICROMEGAS_NET_RPC_SCOPE(BlobDescriptor->DebugName->Name, Context.GetBitStreamReader()->GetPosBits());

Why: the RPC name isn't valid until after ResolveFunctionAndObject — opening the scope earlier would dereference an invalid BlobDescriptor.

3.12.4 `FNetRPC::Deserialize`¶

Insertion point: same post-resolve placement as §3.12.3.

Insert:

// micromegas net trace
MICROMEGAS_NET_RPC_SCOPE(BlobDescriptor->DebugName->Name, Context.GetBitStreamReader()->GetPosBits());

3.13 Demo recording suspend¶

Sites in Engine/Source/Runtime/Engine/Private/DemoNetDriver.cpp, all using MICROMEGAS_NET_SUSPEND_SCOPE(); (no arguments) immediately before the demo operation.

The demo connection is, mechanically, a UNetConnection — its packet flush goes through the same LowLevelSend / FlushNet paths that emit net.packet_sent_bits and that wire-framing OBJECT_EVENTs (§3.14) attach to. Without suspending, all of that traffic gets attributed against the demo connection's name and inflates totals that are supposed to represent real network bandwidth. Cover both the per-frame replication entry points and any path that drives the demo connection through the standard packet pipeline.

3.13.1 `UDemoNetDriver::ProcessRemoteFunction`¶

Insertion point: immediately before InternalProcessRemoteFunction(Actor, SubObject, ClientConnections[0], Function, Parameters, OutParms, Stack, IsServer());.

Insert:

// micromegas net trace
MICROMEGAS_NET_SUSPEND_SCOPE();

Why: a multicast RPC processed by DemoNetDriver during live-game ReceivedPacket would otherwise call BeginConnection / EndConnection and clobber the live packet's depth counters and AccumulatedBits.

3.13.2 `UDemoNetDriver::TickDemoRecordFrame`¶

Insertion point: immediately before the demo actor-replication block (around the STAT_ReplayReplicateActors cycle counter / Replay actor replication time scope).

Insert:

// micromegas net trace
MICROMEGAS_NET_SUSPEND_SCOPE();

Why: demo recording isn't real network bandwidth — including its 1-bit "no changes" fast-array markers and 0-dirty-bits objects would be misleading.

3.13.3 `UDemoNetDriver::NotifyActorTornOff`¶

Insertion point: immediately before ReplayHelper.ReplicateActor(Actor, ClientConnections[0], true);.

Insert:

// micromegas net trace
MICROMEGAS_NET_SUSPEND_SCOPE();

Why: event-driven, same re-entrancy risk as ProcessRemoteFunction.

3.13.4 Demo connection packet pipeline (TickFlush / TickDispatch)¶

Insertion point: the per-tick entry that drives the demo UNetConnection through its packet flush — typically UDemoNetDriver::TickFlush (recording) and UDemoNetDriver::TickDispatch (playback). Wrap each in a brace-bounded suspend so the underlying FlushNet / ReceivedPacket calls inherit it.

Insert (per site):

// micromegas net trace
MICROMEGAS_NET_SUSPEND_SCOPE();

Why: the §3.1 ReceivedPacket and §3.2 net.packet_sent_bits instrumentation are unconditional — they fire on every UNetConnection, including the demo connection. Without suspending here, the demo connection's flush emits packet metrics, wire-framing object events, and (during playback) a full incoming-packet attribution stream against the demo connection's display name. Suspending at the tick boundary silences all of that in one place.

3.14 Wire-framing instrumentation¶

The §3.7 / §3.10 / §3.11 object scopes only attribute the content of replicated objects and RPCs. The per-packet wire bits (net.packet_*_bits) include a substantial fraction of framing overhead that those scopes never see — handshake/encryption components, byte-alignment padding, the packet header and ack record, per-bunch headers, NetGUID export bunches, and bunches resent in response to NAKs. Without instrumenting these, sum(net_spans content) / sum(net.packet_sent_bits) settles around 0.5–0.7 even on a clean integration, leaving a large unattributed mass when investigators try to explain bandwidth.

This section adds named MICROMEGAS_NET_OBJECT_EVENT (and MICROMEGAS_NET_OBJECT_SIZE_SCOPE for retransmits) emissions that label each framing class so it appears in the same net_spans view as content. Each event opens at depth 0 inside the active connection scope (peer to actor scopes), so a query like SELECT name, sum(bit_size) FROM net_spans GROUP BY name partitions the connection's wire bits into content and named-framing buckets.

All sites assume an active MICROMEGAS_NET_CONNECTION_SCOPE from §3.1–§3.5; outside one, the events are dropped by the writer's depth gate. The macro forms:

MICROMEGAS_NET_OBJECT_EVENT(Name, Bits) — fire-and-forget. Use when the bit count is known up front and the wrapped code does not mutate it. Zero-bit calls are suppressed.
MICROMEGAS_NET_OBJECT_SIZE_SCOPE(Name, GetSizeExpr) — RAII scope that reads the size at destruction. Use for retransmits where SendRawBunch does not mutate the bunch, so a GetNumBits-diff scope would read 0 and the writer's elision path would drop it.

3.14.1 PacketHandler chain bits (outgoing)¶

File: Engine/Plugins/Online/OnlineSubsystemUtils/Source/OnlineSubsystemUtils/Private/IpConnection.cpp (or your LowLevelSend override).
Function: UIpConnection::LowLevelSend.
Insertion point: immediately after the Handler->Outgoing(...) call, where Traits.NumberOfBits (or your equivalent) reflects the post-handler bit count. Compute the delta against the pre-handler bit count.

Insert:

// micromegas net trace — handler-added bits (handshake, encryption, checksum)
MICROMEGAS_NET_OBJECT_EVENT(TEXT("PacketHandler"), PostHandlerBits - PreHandlerBits);

Why: the PacketHandler chain — StatelessConnectHandler, encryption, checksums, etc. — inflates every outgoing packet between FlushNet writing the buffer and LowLevelSend putting it on the socket. The delta is per-packet but typically small (tens of bits); aggregated, it explains a meaningful slice of the wire/content gap.

3.14.2 BitPadding (outgoing byte alignment)¶

File: Engine/Source/Runtime/Engine/Private/NetConnection.cpp
Function: UNetConnection::FlushNet
Insertion point: at the byte-alignment step that pads SendBuffer up to a byte boundary before LowLevelSend. Capture the pre-pad bit count, run the existing alignment code, then emit the difference.

Insert:

// micromegas net trace — trailing pad bits
MICROMEGAS_NET_OBJECT_EVENT(TEXT("BitPadding"), SendBuffer.GetNumBits() - PrePadBits);

Why: every packet pads up to 0–7 bits at the end. Per packet it's nothing; multiplied by hundreds of packets per second it's a few percent of bandwidth that would otherwise vanish.

3.14.3 PacketHeader / Ack¶

File: Engine/Source/Runtime/Engine/Private/NetConnection.cpp
Function: UNetConnection::FlushNet
Insertion point: around the PacketNotify.WriteHeader(...) call (and any WriteFinalPacketInfo your fork has). Capture pre/post bit positions on SendBuffer.

Insert:

// micromegas net trace — packet ID + ack record
MICROMEGAS_NET_OBJECT_EVENT(TEXT("PacketHeader"), PostHeaderBits - PreHeaderBits);

Why: the packet header carries packet sequence ID and the ack history bitfield. It is fixed-cost per packet and the largest single framing line item on most connections.

3.14.4 BunchHeader¶

File: Engine/Source/Runtime/Engine/Private/DataChannel.cpp
Function: UChannel::SendRawBunch (or wherever your fork serializes the bunch header onto the outgoing buffer — look for the SerializeBits/WriteBit sequence that emits bReliable, ChIndex, bClose, BunchDataBits, etc.).
Insertion point: capture pre-serialize bit position, run the existing header-serialize block, then emit the difference. (The existing Bunch.GetNumBits() body bits are attributed by the §3.7 object scopes, so this only needs to cover the header itself.)

Insert:

// micromegas net trace — per-bunch header
MICROMEGAS_NET_OBJECT_EVENT(TEXT("BunchHeader"), PostHeaderBits - PreHeaderBits);

Why: every bunch carries a header (channel index, reliability flag, sequence, payload length). Bunch headers add up fast on a busy server — typically the second-largest framing class after PacketHeader.

3.14.5 Retransmits (NAK resend)¶

File: Engine/Source/Runtime/Engine/Private/NetConnection.cpp
Function: the NAK-handling path that walks OutBunch chains and re-queues bunches via SendRawBunch (typically UNetConnection::ReceivedNak / WriteBitsToSendBuffer follow-up).
Insertion point: wrap the resend call so the bunch's existing bit count is read on scope exit.

Insert:

// micromegas net trace — bunch resent in response to NAK
MICROMEGAS_NET_OBJECT_SIZE_SCOPE(TEXT("Retransmit"), Bunch->GetNumBits());

Why: retransmits are real bandwidth that no §3.7 scope sees — SendRawBunch consumes a pre-built bunch without re-running the actor write path, so a GetNumBits()-diff scope would observe 0 and the writer's elision path would drop the event. The SIZE_SCOPE form reads the size directly.

3.14.6 NetGUID exports¶

File: Engine/Source/Runtime/Engine/Private/DataChannel.cpp (or wherever your fork emits export bunches — WritePackageMapAck / WriteContentBlockHeader export-bunch path).
Insertion point: when the export bunch is built and known to be sent, before the SendRawBunch.

Insert:

// micromegas net trace — NetGUID export bunch
MICROMEGAS_NET_OBJECT_EVENT(TEXT("NetGUIDExport"), ExportBunch.GetNumBits());

Why: NetGUID exports are sent in their own bunches (separate from the actor body that referenced them) so they don't show up under any actor's content attribution. Initial-state-heavy connections (join spikes, level streaming) lean heavily on exports; without this, those spikes look like unattributable bandwidth.

3.14.7 Picking sites in your fork¶

The exact landmark in each function above moves between UE versions. Anchor on the existing wire-bit telemetry:

For each site, find where UE_NET_TRACE_* already brackets the same code — those macros already chose the right pre/post boundary.
Reuse the same captured StartPos / EndPos locals when present; add a private capture only when the existing trace points don't expose one.
Run parse_block over a recorded session and confirm the new event names appear at depth 0 with non-trivial bit_size. If a wire-framing event consistently emits 0 bits, the boundaries are wrong — recheck which buffer you're measuring against.

4. Pitfalls¶

Each entry below gives a symptom you'd see in the lakehouse or the log, the underlying cause, and the structural fix. Together these encode the rules an implementer needs to extrapolate to call sites the recipe doesn't explicitly cover.

Object events orphan outside any connection scope (server traces)¶

Symptom: root NetObjectBeginEvent events appear in the stream with no preceding NetConnectionBeginEvent, often ~120 actors per tick with bit_size:0.
Cause: the project sets a UReplicationDriver (typically UReplicationGraph), which makes UNetDriver::ServerReplicateActors short-circuit into the driver — the §3.3 scope never opens.
Fix: add the §3.4 RepGraph scope.
Generalization: any custom replication driver needs its own per-connection scope at its top-level per-connection loop.

Multi-kilobit objects appear ~1 s after their packet (client traces, initial state)¶

Symptom: large initial-state subobjects (e.g. InventoryManagerComponent) orphan outside a ReceivedPacket scope, ~1 s late.
Cause: bunches blocked on unresolved NetGUIDs are parked in UActorChannel::QueuedBunches and flushed later from Tick(), outside ReceivedPacket.
Fix: add the §3.6 ProcessQueuedBunches scope.
Generalization: any code path that calls ProcessBunch from outside ReceivedPacket needs its own incoming connection scope.

`LogMicromegasNet` fires repeatedly with the same `(inner, outer)` pair¶

Symptom: a diagnostic log line repeats at runtime with the same inner/outer connection name+direction (visible at VeryVerbose).
Cause: a real re-entry path — the inner scope opens while the outer is still active. Same name + same direction is silently absorbed (no log); the log fires only when the names differ or the directions disagree.
Fix: identify the call path. If the inner work shouldn't be attributed (demo, replay, server-side simulation), add MICROMEGAS_NET_SUSPEND_SCOPE() at the inner call site instead of a connection scope. If it should be attributed and shares the outer's direction, accept the Decision-6 absorption. If the directions disagree (reply RPC, NAK echo), the writer auto-suspends the inner scope so its OBJECT_EVENTs don't emit nested under a wrong-direction parent — see "Children measure on a different bit stream than parent" below.
Generalization: any synchronous engine callback that can fire from inside a packet/replication scope (e.g. PostLogin, OnRep_* calling into game code, RPC dispatch from inside FlushNet, NAK handling from inside ReceivedRawPacket) is a re-entry candidate.

Children measure on a different bit stream than parent (reply RPC / NAK)¶

Symptom (pre-auto-suspend): sum(child bit_size) > parent bit_size for a connection scope where an outgoing reply RPC fires inside an incoming packet handler (or vice versa for a NAK echo). Children would be measuring bytes against an outgoing writer while the parent measured against an incoming reader.
Cause: the outer connection scope's direction and the inner scope's direction disagree, so each measures a different bit source. The writer cannot meaningfully aggregate children into a parent that's tracking the wrong stream.
Fix: automatic — the EScopeKind::Suspended classification at Begin time auto-suspends the inner scope so its OBJECT_EVENTs and OBJECT_SCOPEs don't emit. Aggregate sums recover the sum(children) <= parent invariant. No call-site change required; documented here so the silent suppression doesn't surprise investigators looking for the missing inner spans.

`NetConnectionEndEvent.bit_size` much smaller than `sum(net.packet_*_bits)`¶

Symptom: content bits look way off from wire bits for the same connection and window.
Cause: NetConnectionEndEvent.bit_size is the content sum — root NetObject* and root NetRPC* bit_size totals only. The wire metric also includes packet headers, byte-alignment padding, per-bunch headers, NetGUID exports, retransmits, the PacketHandler chain, control bunches, and voice.
Fix: with §3.14 wire-framing instrumentation in place, the named framing classes (PacketHeader, BunchHeader, BitPadding, PacketHandler, NetGUIDExport, Retransmit) appear in net_spans alongside content, and sum(net_spans bit_size) / sum(net.packet_*_bits) should land within a few percent of 1.0 over a steady window. A persistent residual is normal — it represents control bunches, voice, and other sources not yet labeled.
Generalization: if a residual grows without explanation, group net_spans by name at depth = 1 (root-level peers) and look for an unfamiliar dominant name — or for content names that have inflated past their wire share (likely a missing SUSPEND_SCOPE).

Demo recording bits show up as live network bandwidth¶

Symptom: unexpected attribution under the demo connection's MmDisplayName, or net.packet_sent_bits totals significantly higher than the sum of live connections' wire bits.
Cause: missing SUSPEND_SCOPE somewhere in §3.13 — most often the demo connection's tick/flush pipeline (§3.13.4), which is easy to forget because §3.13.1–.3 only cover replication entry points, not the packet-flush path that emits net.packet_*_bits and §3.14 framing events.
Fix: add the missing suspend. Verify with SELECT connection_name, sum(bit_size) FROM net_spans GROUP BY 1 — the demo connection should report 0 (or be absent) once all sites are suspended.
Generalization: any system that drives a UNetConnection while the live game is also doing so (replay, server-side simulation, ghost recording, late-join cinematic scrubbers) needs to suspend at every entry point that walks its connection through FlushNet / ReceivedPacket / ReplicateActor.

`EndConnection` events with no preceding `BeginConnection`¶

Symptom: unbalanced Begin/End in the event stream.
Cause: since the writer keys End behavior on a per-Begin EScopeKind stack push, every End is paired with the kind its Begin pushed — a stray End cannot reach the emission path. The realistic source is now (a) a helper calling Dispatch::NetEndConnection() directly, bypassing FNetConnectionScope; or (b) a scope crossing a block-boundary buffer flush so its Begin and End land in different blocks (this is normal cross-block stitching, handled by the net_spans view — only flag it if the same block contains an unbalanced End).
Fix: only use MICROMEGAS_NET_CONNECTION_SCOPE (RAII) at function-scoped boundaries — there are no public Begin/End macros. If this surfaces, confirm no helper is calling Dispatch::NetEndConnection() directly (FNetConnectionScope is the only authorized caller).

RPC bits double-count into `NetConnectionEndEvent.bit_size`¶

Symptom: content bits appear inflated when RPCs are involved.
Cause: an RPC scope nested inside an object scope on the receive path (an RPC fires inside ReceivedBunch which is inside §3.7.3's ProcessBunch OBJECT_SCOPE). Both scopes would measure deltas of the same Reader.
Fix: already gated automatically by if (ObjectDepth == 0) in EndRPC — nested RPC bits are silently absorbed into the parent object's measurement, not double-counted. Don't try to "unfix" the gate.

Connection name shows as `NetConnection_0`, an opaque GUID, or an `addr:port` you can't recognize¶

Symptom: NetConnectionBeginEvent.connection_name is a generic FName (e.g. NetConnection_0), an internal player ID, or a raw addr:port instead of a player-facing handle.
Cause: MmDisplayName was never refreshed for that connection — the cached FName is still the early-fallback value from before player identity was available.
Fix: confirm the refresh function runs at every lifecycle hook listed in §3 (handshake complete, OnRep_PlayerState, OnRep_PlayerName, OnRep_UniqueId, custom "identity finalized" hooks). The NetConnection_0 form means the function never ran for that connection; the addr:port or internal-ID form means it ran but the higher-priority inputs were unavailable at the time.
Generalization: the resolution chain is PlayerName|PlayerId → PlayerId → addr:port → GetFName(). Anytime you see a low-priority value, the higher-priority sources weren't populated yet — refresh again at the lifecycle event that populates them.

Subobject hierarchy differs between classic and Iris¶

Symptom: queries that walk NetObject* nesting return different shapes from classic vs Iris processes.
Cause: classic emits subobjects as peers at depth 0 (because WriteSubObjectInBunch is called from DoSubObjectReplication after the root actor's scope closed). Iris emits them nested at depth 1+ (because WriteObjectAndSubObjects recurses inside the parent scope).
Fix: both shapes are correct — they reflect how each system actually replicates. Queries that traverse hierarchy must handle both, or filter on a per-process Iris/classic flag.

5. Verifying instrumentation¶

Scope

The net_spans JIT view — available via view_instance('net_spans', '<process_id>') — decodes each block into pre-paired Connection/Object/Property/RPC spans with cumulative begin_bits / end_bits offsets. See the net_spans schema for the full column list. For raw per-event inspection (debugging a suspect block, building custom aggregations), the generic parse_block(block_id) table function still works and returns (object_index, type_name, value as JSONB) rows where type_name is one of NetConnectionBeginEvent, NetConnectionEndEvent, NetObjectBeginEvent, NetObjectEndEvent, NetPropertyEvent, NetRPCBeginEvent, NetRPCEndEvent.

Top bandwidth spans via `net_spans`¶

For routine inspection prefer the net_spans view — it handles Begin/End pairing, cross-block stitching, and bit-offset math so queries stay one-liners.

SELECT kind, name, connection_name, is_outgoing, bit_size
FROM view_instance('net_spans', '<process_id>')
ORDER BY bit_size DESC
LIMIT 20;

Feed the same query to a Flame Graph cell (with span_id AS id, parent_span_id AS parent, begin_bits AS begin, end_bits AS end) to get a bandwidth-weighted flame chart of the process's replication traffic.

Stream registration¶

Confirms the net stream is created and tagged.

SELECT process_id, tags, properties
FROM streams
WHERE array_has(tags, 'net')
ORDER BY insert_time DESC
LIMIT 5;

If this returns zero rows, the writer isn't initializing or the sink isn't uploading — no point checking anything else.

Wire-bit metrics¶

Confirms §3.1 and §3.2 are firing. Independent of any net-events view — these go through the standard metric pipeline.

SELECT time, name, value, properties
FROM measures
WHERE name IN ('net.packet_sent_bits', 'net.packet_received_bits')
ORDER BY time DESC
LIMIT 20;

Find a net block to inspect¶

The blocks view joins stream tags under "streams.tags"; filter with array_has:

SELECT block_id, stream_id, "processes.exe", begin_time, end_time,
       nb_objects, payload_size, "streams.tags"
FROM blocks
WHERE array_has("streams.tags", 'net')
ORDER BY end_time DESC
LIMIT 5;

Decode events in a block¶

Pick a block_id from the query above and pass it to parse_block:

SELECT object_index, type_name, jsonb_format_json(value) AS event
FROM parse_block('<block_id>')
LIMIT 1000;

Or summarize what event types a block contains:

SELECT DISTINCT type_name FROM parse_block('<block_id>');

A NetConnectionBeginEvent without a matching EndEvent in the same block signals unbalanced instrumentation.

Notebook-friendly

In the Analytics Web App, the three queries above compose naturally into a notebook: a net-blocks table with a row selection → two parse_block cells bound to $blocks.selected.block_id. Ready-made inspection workflow.

Connection names and bit sizes¶

Extract specific fields with jsonb_get / jsonb_as_string:

SELECT object_index,
       type_name,
       jsonb_as_string(jsonb_get(value, 'connection_name')) AS connection_name,
       jsonb_as_string(jsonb_get(value, 'is_outgoing'))     AS is_outgoing,
       jsonb_as_string(jsonb_get(value, 'bit_size'))        AS bit_size
FROM parse_block('<block_id>')
WHERE type_name IN ('NetConnectionBeginEvent', 'NetConnectionEndEvent')
ORDER BY object_index;

connection_name should be a non-empty FName-backed string. bit_size on the End event should be a plausible content-bit total.

Content-vs-wire reconciliation¶

Pick one connection and one time window. Sum the bit_size of all root-level net_spans rows for that connection (content + named framing classes from §3.14), and sum net.packet_sent_bits from measures for the same connection over the same window. With §3.14 instrumentation in place, the ratio should land within a few percent of 1.0 over a steady window — a persistent shortfall points to a missing framing site, and an excess points to a missing SUSPEND_SCOPE somewhere.

Without §3.14, content alone typically lands at 0.5–0.7× wire — that gap is unattributed framing, not instrumentation error.

Framing class breakdown¶

After §3.14, slice net_spans by name to see which classes dominate:

SELECT name, sum(bit_size) AS bits
FROM view_instance('net_spans', '<process_id>')
WHERE kind = 'object' AND depth = 1
GROUP BY name
ORDER BY bits DESC
LIMIT 20;

depth = 1 filters to objects immediately under a Connection (root-level peers — actor scopes and §3.14 framing events), excluding nested Iris subobjects whose bits are already counted in their parent. Expect content names (actor classes) at the top followed by PacketHeader, BunchHeader, then the smaller framing classes. An unfamiliar dominant name often points to a missing SUSPEND_SCOPE (the demo connection most commonly).

No orphaned ends¶

Over a single block, every NetConnectionEndEvent should follow a preceding NetConnectionBeginEvent (same for NetObject* and NetRPC*). Walk the parse_block output and maintain a depth counter per event family — it must never go negative.

CVar runtime change¶

telemetry.net.verbosity 4
telemetry.net.verbosity 2

Confirm net-block insertion rate scales with verbosity by re-running the blocks filter query at each level. The change takes effect at the next outermost BeginConnection (verbosity is snapshotted per scope), not mid-scope.

6. Build configuration¶

MICROMEGAS_NET_TRACE_ENABLED defaults to !UE_BUILD_SHIPPING && !WITH_EDITOR:

Editor → off
Development / Test → on
Shipping → off

To enable in shipping for a focused investigation, add to your Target.cs:

GlobalDefinitions.Add("MICROMEGAS_NET_TRACE_ENABLED=1");

To force-off in non-shipping for a perf bake-off:

GlobalDefinitions.Add("MICROMEGAS_NET_TRACE_ENABLED=0");

When the macro is 0, every MICROMEGAS_NET_* call expands to nothing — zero overhead, zero divergence cost beyond the insertion lines themselves.

Network Tracing¶

1. Overview¶

2. Architecture¶

Data flow¶

Verbosity model¶

Content-attribution vs wire-bit semantics¶

Classic vs Iris hierarchy¶

Non-nesting invariant (Decision 6)¶

Suspend mechanism¶

RAII everywhere¶

3. Engine instrumentation recipe¶

3.1 Incoming packet scope¶

3.2 Outgoing packet metric¶

3.3 Classic server replication scope¶

3.4 RepGraph server replication scope¶

3.5 Iris outgoing scope¶

3.6 Queued-bunch deferred-flush scope¶

3.7 Classic object scopes¶

3.7.1 UActorChannel::ReplicateActor (root actor, send)¶

3.7.2 UActorChannel::WriteSubObjectInBunch (subobject, send)¶

3.7.3 UActorChannel::ProcessBunch content-block receive loop¶

3.7.4 FObjectReplicator::ReplicateCustomDeltaProperties (fast arrays / custom delta)¶

3.8 Classic property scopes¶

3.8.1 SendProperties_r, shared-property branch¶

3.8.2 SendProperties_r, regular-property branch¶

3.8.3 SendProperties_BackwardsCompatible_r¶

3.8.4 ReceiveProperties_r¶

3.8.5 ReceiveProperties_BackwardsCompatible_r¶

3.9 Classic RPCs¶

3.9.1 ProcessRemoteFunctionForChannelPrivate, send-side¶

3.9.2 ProcessRemoteFunctionForChannelPrivate, RPC-forced ReplicateActor block¶

3.9.3 FObjectReplicator::ReceivedRPC¶

3.10 Iris properties¶

3.11 Iris objects¶

3.11.1 FReplicationWriter::SerializeObjectStateDelta¶

3.11.2 FReplicationReader::DeserializeObjectStateDelta¶

3.12 Iris RPCs¶

3.12.1 FNetRPC::SerializeWithObject¶

3.12.2 FNetRPC::Serialize¶

3.12.3 FNetRPC::DeserializeWithObject¶

3.12.4 FNetRPC::Deserialize¶

3.13 Demo recording suspend¶

3.13.1 UDemoNetDriver::ProcessRemoteFunction¶

3.13.2 UDemoNetDriver::TickDemoRecordFrame¶

3.13.3 UDemoNetDriver::NotifyActorTornOff¶

3.13.4 Demo connection packet pipeline (TickFlush / TickDispatch)¶

3.14 Wire-framing instrumentation¶

3.14.1 PacketHandler chain bits (outgoing)¶

3.14.2 BitPadding (outgoing byte alignment)¶

3.14.3 PacketHeader / Ack¶

3.14.4 BunchHeader¶

3.14.5 Retransmits (NAK resend)¶

3.14.6 NetGUID exports¶

3.14.7 Picking sites in your fork¶

4. Pitfalls¶

Object events orphan outside any connection scope (server traces)¶

Multi-kilobit objects appear ~1 s after their packet (client traces, initial state)¶

LogMicromegasNet fires repeatedly with the same (inner, outer) pair¶

Children measure on a different bit stream than parent (reply RPC / NAK)¶

NetConnectionEndEvent.bit_size much smaller than sum(net.packet_*_bits)¶

Demo recording bits show up as live network bandwidth¶

EndConnection events with no preceding BeginConnection¶

RPC bits double-count into NetConnectionEndEvent.bit_size¶

Connection name shows as NetConnection_0, an opaque GUID, or an addr:port you can't recognize¶

Subobject hierarchy differs between classic and Iris¶

5. Verifying instrumentation¶

Top bandwidth spans via net_spans¶

Stream registration¶

Wire-bit metrics¶

Find a net block to inspect¶

Decode events in a block¶

Connection names and bit sizes¶

Content-vs-wire reconciliation¶

Framing class breakdown¶

No orphaned ends¶

CVar runtime change¶

6. Build configuration¶

3.7.1 `UActorChannel::ReplicateActor` (root actor, send)¶

3.7.2 `UActorChannel::WriteSubObjectInBunch` (subobject, send)¶

3.7.3 `UActorChannel::ProcessBunch` content-block receive loop¶

3.7.4 `FObjectReplicator::ReplicateCustomDeltaProperties` (fast arrays / custom delta)¶

3.8.1 `SendProperties_r`, shared-property branch¶

3.8.2 `SendProperties_r`, regular-property branch¶

3.8.3 `SendProperties_BackwardsCompatible_r`¶

3.8.4 `ReceiveProperties_r`¶

3.8.5 `ReceiveProperties_BackwardsCompatible_r`¶

3.9.1 `ProcessRemoteFunctionForChannelPrivate`, send-side¶

3.9.2 `ProcessRemoteFunctionForChannelPrivate`, RPC-forced ReplicateActor block¶

3.9.3 `FObjectReplicator::ReceivedRPC`¶

3.11.1 `FReplicationWriter::SerializeObjectStateDelta`¶

3.11.2 `FReplicationReader::DeserializeObjectStateDelta`¶

3.12.1 `FNetRPC::SerializeWithObject`¶

3.12.2 `FNetRPC::Serialize`¶

3.12.3 `FNetRPC::DeserializeWithObject`¶

3.12.4 `FNetRPC::Deserialize`¶

3.13.1 `UDemoNetDriver::ProcessRemoteFunction`¶

3.13.2 `UDemoNetDriver::TickDemoRecordFrame`¶

3.13.3 `UDemoNetDriver::NotifyActorTornOff`¶

`LogMicromegasNet` fires repeatedly with the same `(inner, outer)` pair¶

`NetConnectionEndEvent.bit_size` much smaller than `sum(net.packet_*_bits)`¶

`EndConnection` events with no preceding `BeginConnection`¶

RPC bits double-count into `NetConnectionEndEvent.bit_size`¶

Connection name shows as `NetConnection_0`, an opaque GUID, or an `addr:port` you can't recognize¶

Top bandwidth spans via `net_spans`¶