Skip to content

test_vectors.md

Latest commit

 

History

History
65 lines (41 loc) · 3.05 KB

test_vectors.md

File metadata and controls

65 lines (41 loc) · 3.05 KB
Raw

TODO: Test Vectors

We will want to make our implementation-specific tests largely obsolete by a cross-implementation set of static test data, the single source of truth for the format. This will be a JSON file generated by the OCaml reference implementation and used by all test suites.

Initial strategy:

  • Write an OCaml program to generate an initial set of vectors
  • Make the OCaml test suite iterate through those vectors
  • Remove OCaml-specific tests which were made redundant by the vectors

Ongoing:

  • Testing a new VOF implementation would begin by testing all the vectors
  • Any coverage gaps found should raise the question: "could this be expressed as a new vector?"
    • Yes: add the vector in the OCaml vector generator program, remove local tests obsoleted by it
    • No: add tests in the current implementation

So first, we need to define a schema for this JSON file, probably create a brief Markdown file documenting it, and a strategy for populating it. We'll add Yojson as a dependency for that bit (it's a test-only dependency, so our production module remains lean).

We have the Base64 module as a dependency in our main project, so we could use that to represent binary data in the vectors as well, URL-safe. (i.e. Base64.(encode_string ~pad:false ~alphabet:uri_safe_alphabet))

CBOR Example, for ideas

The official "Appendix A" vectors of CBOR is a JSON file, a list of objects each with keys:

  • cbor: a base-64 encoded CBOR data item
  • hex: the same CBOR data item in hex encoding
  • roundtrip: a boolean that indicates whether a generic CBOR encoder would typically produce identical CBOR on re-encoding the decoded data item (your mileage may vary)
  • decoded: the decoded data item if it can be represented in JSON
  • diagnostic: the representation of the data item in CBOR diagnostic notation, otherwise

The whole file is just 82 objects. VOF is inherently much more sophisticated, though.

Layer 1: Binary primitives

Most similar to CBOR's test vectors.

  • Control byte decoding, integer forms, floats, short/long strings, lists, gaps, null, alt, tag.
  • Canonical encoding: very the smallest form is chosen
  • ZigZag signed integer codec
  • Invalid inputs that should cause rejection

This goes for Binary and CBOR wire formats, since we rolled our own CBOR codec (truly trivial for our subset of it).

Layer 2: Scalar types

Each vector would carry a type tag and expected representations in all 3 formats. Error cases too.

Layer 3: Typed Readers

Test minimum coercion/tolerance that every implementation should provide and what should be rejected.

Layer 4: Symbol Tables

Composites with schema context (records, variants, series) would rely on a second file (or embedded as an escaped JSON string), a symbol table, and would somehow need enough type information to allow encoding and decoding the same high-level value in all encodings.

PATCH application may be testable here. (i.e. object A + patch X = object B)

Not Covered

  • PATCH generation
  • select~ parsing an query filtering
  • $msg construction
  • Streaming behavior
  • Performance and limits