Trace an experiment

The three handles

• Run address. Each run gets its own address, derived from the spec and a seed. Every transaction of the run pays to it or spends from it. Paste it into any explorer and you see the whole run, with nothing else to set up.
• Run id (16 hex characters). Written on chain in every transaction of the run, with that transaction's index: run:<run id>#<index>. The index matches tx_index in the export.
• Spec id (16 hex characters, the start of the spec's hash). The same for every run of the same spec, so it finds all runs of one experiment.

A run link, txflow.fun/run/<id>, opens the run in any browser: the experiment's spec, with that run on the graph.

The message on chain

Every transaction txflow builds carries a CIP-20 message (metadata label 674):

{ "674": { "msg": ["txflow/1 txflow.fun", "spec:<spec id>", "run:<run id>#<index>"] } }

The first entry names the tool, the version of this layout and where the transaction was built (txflow.fun, or cli for the command line). The sweeps that return a run's leftovers say ["txflow/1 txflow.fun", "sweep", "run:<run id>"]. Funding the lanes from outside txflow carries no message.

In db-sync

Every transaction of one run, with its block and time:

SELECT encode(tx.hash, 'hex') AS tx_id, block.block_no, block.time,
       m.json->'msg'->>2 AS which
FROM tx_metadata m
JOIN tx ON tx.id = m.tx_id
JOIN block ON block.id = tx.block_id
WHERE m.key = 674
  AND m.json->'msg'->>0 LIKE 'txflow/%'
  AND m.json->'msg'->>2 LIKE 'run:<run id>#%'
ORDER BY block.block_no;

For every run of an experiment, match m.json->'msg'->>1 = 'spec:<spec id>' instead. For everything txflow ever sent, keep only the first condition. A transaction that never landed (the losing side of a double spend) is not in db-sync; the export below still has it.

The export

Data on the run panel downloads the run, also at GET /api/runs/<id>/export. The format is txflow-run/1, in three parts:

• run: run_id, spec_id, seed, t0 (when its schedule started), the network and era, the watermark, the lane, the run address and the spec itself.
• txs, one per transaction: tx_index (the #index on chain), tx_name, kind, tx_id, the spec's labels, the conflict_sets it spends from and the txs it conflicts_with, its inputs and outputs (lovelace), where and when it was planned, and whether it was included and in which block (block_no when known, else block_min to block_max).
• events, one per thing that happened to a transaction at a relay, in time order: sent (with the relay's result, the reason when rejected, and reply_ms, how long the submit took), mempool_in, mempool_out (with the node's reason: AllInputsAreSpent when a block spent its inputs, normally because it landed), node_rejected, dropped (it left without landing, when the node did not say), in_eb (the first endorser block that referenced it, eb_hash) and in_ledger (with block_no and block_hash when that relay's block is exact).

Every event has t_ms, milliseconds after t0, and at, the same time in UTC to the microsecond, to line up with the relays' logs. source says whose clock it is: node is the relay node's own, from its log; poll is when txflow looked, which can be a few seconds late; txflow is when txflow sent it. Each relay's clock is its own, so differences of a few milliseconds between relays can be clock skew.

?format=json gives all three parts. ?format=csv&table=events (the default) or &table=txs gives one table, with run_id and spec_id on every row and lists joined by spaces, so the files of several runs can be stacked.

In the relays' logs

Each relay node logs every transaction it takes in or drops, with txIdFull: grep its trace (on Musashi, /var/log/musashi/traces/<host>_3010/, and the S3 archive of rotated files) for the run's tx_ids from the export. The namespaces are Mempool.AddedTx, Mempool.RemoveTxs (with the reason) and Mempool.RejectedTx; BlockFetch.Remote.Receive.Block lists the transactions of each block, and LeiosFetch.Remote.Receive.Block the endorser blocks' references.

What the times can and cannot say

• Node times come from each relay's own clock, so comparing them across relays is only as good as the relays' clock sync.
• The site reads them through a bridge on each relay. A bridge keeps a run's events for an hour after the run last asked; if a bridge restarts during a run, it only knows what is in the node's current log file (a minute or two under heavy load), and the times it missed are polled ones (≈).
• A transaction that lands through an endorser block is not listed in the ranking block's transactions, so its ledger_ms is the time the run saw it in the ledger.
• The run's own record (and so the export) is kept by the site; the chain and the relays' logs are the record that does not depend on it.