Skip to content

typedecide.data.augment

Option-order permutation for training and for auditing. Background: Position priors and debiasing.

typedecide.data.augment

Permuting the options, because position is the easiest signal in the dataset.

If every row lists the correct answer first, the cheapest hypothesis consistent with the training data is "answer A", and a model will find it long before it finds the task. Randomising the order at training time removes that shortcut; enumerating the orders at evaluation time measures whether any of it survived.

answer_id is carried through every permutation untouched -- the answer is an option id, not a position, which is the whole reason the schema works this way.

MAX_PERMUTATION_OPTIONS module-attribute 

MAX_PERMUTATION_OPTIONS = 8

randomise_option_order 

randomise_option_order(
    decisions: Sequence[Decision], *, seed: int
) -> list[Decision]

One new ordering per decision. Same length, same ids, same answers.

The per-row generator is seeded from (seed, decision.id) rather than from the row's position, so a row gets the same ordering whichever dataset it is part of and whatever order it was loaded in.

Source code in src/typedecide/data/augment.py 
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
def randomise_option_order(decisions: Sequence[Decision], *, seed: int) -> list[Decision]:
    """One new ordering per decision. Same length, same ids, same answers.

    The per-row generator is seeded from `(seed, decision.id)` rather than from the
    row's position, so a row gets the same ordering whichever dataset it is part of
    and whatever order it was loaded in.
    """
    out: list[Decision] = []
    for decision in decisions:
        count = len(decision.criterion.options)
        rng = random.Random(f"{seed}:{decision.id}")
        order = list(range(count))
        rng.shuffle(order)
        out.append(_reordered(decision, order, suffix=None))
    LOGGER.debug("randomised option order for %d decisions (seed=%d)", len(out), seed)
    return out

all_rotations 

all_rotations(decision: Decision) -> list[Decision]

The n cyclic shifts of one decision, identity first.

A cyclic set puts every option in every position exactly once, so a position prior contributes the same total to each option and cancels when the probabilities are averaged per option id. n forward passes instead of n!.

Source code in src/typedecide/data/augment.py 
49
50
51
52
53
54
55
56
57
58
59
60
def all_rotations(decision: Decision) -> list[Decision]:
    """The n cyclic shifts of one decision, identity first.

    A cyclic set puts every option in every position exactly once, so a position prior
    contributes the same total to each option and cancels when the probabilities are
    averaged per option id. n forward passes instead of n!.
    """
    count = len(decision.criterion.options)
    return [
        _reordered(decision, [(i + shift) % count for i in range(count)], suffix=f"r{shift}")
        for shift in range(count)
    ]

all_permutations 

all_permutations(decision: Decision) -> list[Decision]

Every ordering of one decision's options, identity first.

Exact rather than cyclic, and factorially expensive: use it to audit a small sample, and all_rotations for anything you run over a whole eval set.

Source code in src/typedecide/data/augment.py 
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
def all_permutations(decision: Decision) -> list[Decision]:
    """Every ordering of one decision's options, identity first.

    Exact rather than cyclic, and factorially expensive: use it to audit a small
    sample, and `all_rotations` for anything you run over a whole eval set.
    """
    count = len(decision.criterion.options)
    if count > MAX_PERMUTATION_OPTIONS:
        raise DataError(
            f"Decision {decision.id!r} has {count} options, so all_permutations would build "
            f"{count}! variants of a single row. Fix: use all_rotations(), which puts every "
            f"option in every position with {count} variants, or sample orderings instead."
        )
    return [
        _reordered(decision, list(order), suffix=f"p{index}")
        for index, order in enumerate(itertools.permutations(range(count)))
    ]