Meteorological Cooperative › Developers › Conformance test vectors

Conformance test vectors

A vector is a recorded or synthetic input together with the output an implementation should produce from it. The set covers detection, estimation, message encoding, decoding and device presentation. It is the same set the cooperative runs against its own code before any change reaches the warning path, and it is what a device is tested against for certification.

1,412 vectors
5 families
40 minutes for the whole set
4 events the cooperative got wrong

Repositorysrc.meteo.rof/rep-vectors, about 4 GB

Vectors1,412 across five families

Real eventsEvery located event above magnitude 3.0 since 1962 with station records

Run timeAbout 40 minutes on one ordinary machine for the whole set

ToleranceStated per vector; exact match required for encoding and decoding

CertificationA device passes the presentation family; a full implementation passes all five

The five families

Vector families
Family	Vectors	Input	Expected output	Tolerance
Detection	318	Raw three-component station records	Trigger time at each station	±0.1 s
Estimation	318	Trigger times and the first seconds of P record	Location, depth, magnitude	±5 km, ±3 km, ±0.2
Intensity	406	Event parameters and receiving location	Continuous intensity and whole step	±0.05 step
Encoding	204	Message contents as a structure	The exact bytes of the frame	Exact
Presentation	166	Frames on the wire, in order and out of order	What a device shows, sounds and withdraws	Exact, with timing limits

What the presentation family checks

This is the family a device must pass for certification, and it exercises every rule in the wire format. A test harness plays frames at a device and observes what the device does.

Selected presentation vectors
Vector	Input	Required result
`pres-001`	Segment 1, REP 4, onset 22 s	Tone, message and falling onset count within 400 ms
`pres-014`	Same frame delivered three times on three paths	One presentation; duplicates discarded silently
`pres-027`	Segment 3 arriving before segment 2	Segment 3 presented; segment 2 discarded on arrival
`pres-039`	Warning followed by a cancellation 12 s later	Presentation withdrawn within 400 ms; reason stated
`pres-051`	Frame with the test flag set	Presented as a test, distinguishable by sound and wording
`pres-068`	Frame with a bad signature	Nothing presented; failure recorded
`pres-072`	Version 4 frame, unknown to the decoder	Known fields presented; unknown fields ignored
`pres-089`	Frame arriving 40 s after its expiry	Age stated; message never presented as current
`pres-101`	Device muted, screen off, in low power	Wakes, sounds at the fixed level, presents within 400 ms
`pres-118`	Revision lowering REP 6 to REP 4	Values replaced; the revision stated in the presentation
`pres-133`	Event with Aporean areas flagged foreign	Domestic areas first, foreign areas after and marked as beyond the frontier
`pres-147`	Onset already negative on arrival	States that shaking has reached the location

The four events the cooperative got wrong

Four events since 2021 produced an intensity estimate two steps or more from the value observed. All four are in the vector set with the estimate the system made and the value that was afterwards measured. An implementation that reproduces the correct answer on easy events and the same error on these has learned nothing.

The misestimated events, as vectors
Vector	Event	Estimated	Observed	Cause
`err-2022-06`	June 2022, Belvacius margin	REP 6	REP 4	Depth estimated at 8 km against an actual 26 km
`err-2023-03`	March 2023, Ravelluri	REP 3	REP 5	Soft lagoon sediment amplified the motion beyond the model
`err-2024-11`	November 2024, Belvacius margin	REP 5	REP 3	Depth again, from a poor first station geometry
`err-2025-04`	April 2025, Ravelluri	REP 3	REP 5	Same amplification; the soil map for the sub-region was revised afterwards

What the four have in common

Two are depth errors and two are ground amplification errors, and both pairs remain open problems. Depth from the first three or four arrivals is unstable when the stations lie on one side of the epicentre. Amplification over soft sediment depends on a soil map whose resolution is coarser than the variation in the ground.

An implementation is expected to fail some of these vectors. The expected outputs record what was observed, and reproducing them requires information the system did not hold at the moment of issue. They are published so that anyone working on either problem has the cases to work against.

Synthetic vectors

Recorded events cover what has occurred in the Republic, which stops at magnitude 6.4 and at REP 8. Synthetic vectors cover the rest: ruptures to magnitude 7.5, events at REP 9 and REP 10, shallow events beneath a settlement, two events forty seconds apart, and an event on the Aporean side of the boundary.

The synthetic set also holds the failure cases. A station reporting nonsense, a station clock 400 ms out, three stations offline, the issuing node losing contact with half the network, and a frame injected with a valid structure and an invalid signature.

Running the set

git clone src.meteo.rof/rep-vectors
cd rep-vectors
./run --implementation /path/to/yours --family all

# a single family, with the report written out
./run --implementation /path/to/yours --family presentation --report report.txt

# the four misestimated events alone
./run --implementation /path/to/yours --select err-*

The runner prints one line per vector with the expected and actual values and the margin by which each passed or failed. Its exit status is the count of failures. Nothing in the harness communicates with the cooperative, and a device may be tested without notifying anybody.

Inside the cooperative the set gates the warning path. A change to any tree that touches detection, estimation, encoding or presentation runs the whole set before it may be merged, and a failure blocks the merge. Where a vector is expected to fail, the expectation is recorded in the tree with the reference of the defect it belongs to, and an unexplained failure blocks the merge regardless.

Certification testing

Certification consists of running the presentation family against the physical device, with the harness driving each of the five delivery paths in turn. The cooperative performs the test, and the results are published with the certificate, including the margin on the timing limits.

Any person may run the same set and reach their own conclusion. The harness, the vectors and the test procedure are public, and the cooperative holds no monopoly on the assessment.