VReplication exports several metrics using the expvars interface. These are available at the /debug/vars endpoint of vttablet's http status pages. More details can be found here.

Target Tablet Metrics #

VReplicationCopyLoopCount, VReplicationCopyLoopCountTotal #

During the copy phase we run one loop of bulk copy for approximately an hour at a time (by default) before running catchup. VReplicationCopyLoopCount counts the number of times this loop has run for each stream and VReplicationCopyLoopCountTotal the total across all streams.

VReplicationCopyRowCount, VReplicationCopyRowCountTotal #

VReplicationCopyRowCount counts the number of rows copied during the copy phase per stream and VReplicationCopyRowCountTotal the total across all streams.

VReplicationDDLActions #

This metric counts the number of DDL actions — based on the value of the OnDDL strategy configured for the workflow — that have been taken in the current workflow streams.

VReplicationErrors #

VReplicationErrors counts the number of times errors occurred during vreplication. Errors are keyed by the type of error.

VReplicationHeartbeat #

VReplicationHeartbeat records, for each stream, the timestamp sent by the last heartbeat event for that stream.

VReplicationLagSeconds, VReplicationLagSecondsMax, VReplicationLagSecondsTotal #

These metrics show the replication lag of the target stream with respect to the source stream. VReplicationLagSeconds shows the current replication lag and VReplicationLagSecondsMax has the maximum lag in this stream. Note that these values are only valid during the replication phase of a workflow.

VReplicationMessages #

VReplicationMessages contains a stack of the last N (currently 3) messages of a vreplication stream.

VReplicationPhaseTimings, VReplicationPhaseTimingsCounts, VReplicationPhaseTimingsTotal #

This metric relates to the times each phase is run during the lifetime of a stream. VReplicationPhaseTimings counts the total time taken by the runs, VReplicationPhaseTimingsCounts the number of runs and VReplicationPhaseTimingsTotals the total runs across all streams.

VReplicationQPS #

VReplicationQPS is a list of QPS values for each loop of each phase of the workflow.

VReplicationQueryCount, VReplicationQueryCountTotal #

VReplicationQueryCount is the total number of queries in each phase of a workflow. VReplicationQueryCountTotal is the total queries across all phases and workflows.

VReplicationSource #

Shows the keyspace and shard of the source from which this target stream is replicating

VReplicationSourceTablet #

Shows the tablet from which this stream is currently replicating

VReplicationStreamCount #

The number of streams running on this target

VReplicationStreamState #

This shows the state of each stream.

VReplicationTableCopyRowCounts #

VReplicationTableCopyRowCounts counts the number of rows copied during the copy phase per table per stream.

VReplicationTableCopyTimings #

VReplicationTableCopyTimings counts the time taken per table per stream during the copy phase of the stream. Unlike VReplicationPhaseTimings, this metric updates continuously, rather than being set once at the end of the copy phase.

VReplicationThrottledCounts #

The number of times vreplication was throttled by workflow, id, throttler (trx or tablet), and the sub-component (vstreamer, vcopier, vplayer, etc) that was throttled (see the other tablet throttler metrics)

VReplicationThrottledCountTotal #

The total number of times that vreplication has been throttled (see the other tablet throttler metrics)

Source Tablet Metrics #

VStreamerCompressedTransactionsDecoded #

The total number of compressed transactions (MySQL's binlog_transaction_compression=ON) decoded by this vttablet across all workflows

VStreamerCount #

The current number of running vstreamers

VStreamersCreated #

The total number of vstreamers created during the lifetime of this tablet

VStreamersEndedWithErrors #

The total number of errors that caused a stream to stall

VStreamerErrors #

The number of errors per category across workflows

VStreamerEventsStreamed #

The total number of events streamed by this vttablet across all workflows

VStreamPacketSize #

The value of the vstream_packet_size flag specified for this tablet

VStreamerNumPackets #

The total number of packets sent by this vttablet across all workflows

Example #

A snippet from tablet 200 from the local example after running the MoveTables step

"VReplicationCopyLoopCount": {"commerce.0.commerce2customer.1": 2},
"VReplicationCopyLoopCountTotal": 2,
"VReplicationCopyRowCount": {"commerce.0.commerce2customer.1": 10},
"VReplicationCopyRowCountTotal": 10,
"VReplicationErrors": {},
"VReplicationHeartbeat": {"commerce.0.commerce2customer.1": 1618681048},
"VReplicationMessages": {"1": "2021-04-17T19:36:13.003858838+02:00:Picked source tablet: cell:\"zone1\" uid:100 "},
"VReplicationPhaseTimings": {"commerce.0.commerce2customer.1.catchup": 1000935083, "commerce.0.commerce2customer.1.fastforward": 15349583, "commerce.0.commerce2customer.1.copy": 63353125},
"VReplicationPhaseTimingsCounts": {"commerce.0.commerce2customer.1.copy": 2, "commerce.0.commerce2customer.1.All": 6, "commerce.0.commerce2customer.1.catchup": 2, "commerce.0.commerce2customer.1.fastforward": 2},
"VReplicationPhaseTimingsTotal": 1079637791,
"VReplicationQPS": {"All":[11.8,1,1.2,1.2,1,1.2,1,1.2,1,1.2,1,1.2,1.2,1,1.2,1,1.2],"Query":[11.2,1,1.2,1.2,1,1.2,1,1.2,1,1.2,1,1.2,1.2,1,1.2,1,1.2],"Transaction":[0.6,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]},
"VReplicationQueryCount": {"commerce.0.commerce2customer.1.copy": 2},
"VReplicationQueryCountTotal": 2,
"VReplicationLagSeconds": {"commerce.0.commerce2customer.1": 0},
"VReplicationLagSecondsMax": 0,
"VReplicationLagSecondsTotal": 0,
"VReplicationSource": {"1": "commerce/0"},
"VReplicationSourceTablet": {"1": "cell:\"zone1\" uid:100 "},
"VReplicationStreamCount": 1,
"VReplicationStreamState": {"commerce2customer.1": "Running"},
"VReplicationTableCopyRowCounts": {"commerce.0.commerce2customer.1.corder": 4, "commerce.0.commerce2customer.1.customer": 2},
"VReplicationTableCopyTimings": {"commerce.0.commerce2customer.1.customer": 6707583, "commerce.0.commerce2customer.1.corder": 13254250},
"VStreamPacketSize": 250000,
"VStreamerCount": 0,
"VStreamerErrors": {"Catchup": 0, "Copy": 0, "Send": 0, "TablePlan": 0},
"VStreamerEventsStreamed": 0,
"VStreamerNumPackets": 0,
"VStreamerPhaseTiming": {"TotalCount":0,"TotalTime":0,"Histograms":{}},
"VStreamersCreated": 0,
"VStreamersEndedWithErrors": 0,

VTGate Metrics #

VStreamsCreated #

The total number of vstreams created during the lifetime of this vtgate.

VStreamsLag #

The difference in seconds between the current time when the vstream event was sent and the time when the binlog event occurred.