Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions emrg/server/competition_prompt.md
Original file line number Diff line number Diff line change
Expand Up @@ -160,6 +160,17 @@ Write the effective features/models/lessons into memory (so later competitions c

- **With prize money**: read the prize rules and payout conditions **in full** and record them in memory (amount, ranking thresholds, payout conditions, any offline award requirement). A prize competition whose payout requires an offline ceremony is still eligible — see the §3.2 exception.
- **Without prize money**: the goal is leaderboard standing / percentile. Record the metric and the current standing.
- **What decides the score is a reading, not background**: the platform's own terms say which
submissions count (best of / last one / a manual selection of N), whether it falls back to a
default when nothing is selected, the quota and when it resets, and when the standings freeze.
Each platform states this in its own words and none of them defers to another's, so an old
conclusion about the same platform is a hypothesis to re-check, not a fact to reuse.
- **Anything set by hand once goes stale**: a manual selection of submissions, a pending queue, a
freeze marker — none of them follows the results that arrive afterwards. A selection made in an
early round stays that selection while later rounds produce better work, so every round after
the first improvement is scored below what the round had already achieved. Re-read that state
each round, and treat a mismatch between what is set and this round's own best as this round's
work rather than a note for later.

---

Expand Down
48 changes: 48 additions & 0 deletions tests/test_competition_prompt.py
Original file line number Diff line number Diff line change
Expand Up @@ -295,6 +295,54 @@ def test_goal_line_covers_both_prize_and_standing():
assert "leaderboard standing / percentile" in text


def test_the_scoring_rule_is_read_each_round(rendered):
"""What decides the score is a per-round reading, and hand-set state goes stale.

Both levels are asked, and they are different questions. The file read answers "does the
source that ships carry this?" — the render read answers "is this what a round is sent?".
A rule wrapped in `{# … #}` keeps every word in the file and renders as nothing, so a
file-level assertion alone would report a rule no round ever reads; measured on this very
template in issue #1904 for §0.0. The render is therefore the reading that matters here,
and the file assertion is kept because a rule deleted from the source must also fail.

Whitespace is collapsed before the search, because these phrases are longer than the
template's wrap column: pinning a line break would make the guard fail on a reflow that
changed no word, and a guard that reports formatting as a missing rule teaches the next
reader to edit the assertion rather than the text.

What this does not measure: whether a round obeyed it. That is a reading of the round's
closing summary, not something a text assertion can answer.
"""
def flat(text: str) -> str:
return re.sub(r"\s+", " ", text)

source = flat(PROMPT.read_text(encoding="utf-8"))
prompt = flat(rendered)

for phrase, why in (
("What decides the score is a reading",
"the rule that the metric's own definition is re-read each round, rather than reused "
"as background or carried across platforms"),
("which submissions count",
"the first of the four things that decide a score — without it the rule states a "
"principle and leaves the round nothing to check"),
("when the standings freeze",
"the fourth check; a rule that lists only where submissions land has read a quarter "
"of what decides the score"),
("Anything set by hand once goes stale",
"the half that decays in silence: a selection or freeze marker made in an early round "
"does not follow the better work later rounds produce"),
):
assert phrase in prompt, (
f"the rendered competition prompt must carry {why}; {phrase!r} is not in what a "
f"round is sent"
)
assert phrase in source, (
f"{phrase!r} renders but is not in the source — these two readings are supposed to "
f"agree, so one of them is being produced by something other than the template"
)


def test_preparation_and_closing_summary_are_mandatory():
text = PROMPT.read_text(encoding="utf-8")
assert "0. Preparation (MUST run first every round)" in text
Expand Down
Loading