actions-queue-status: colour progress, totals and PMC grouping

[potiuk]

A full org sweep of actions-queue-status.py runs for minutes behind a handful of terse lines, which makes a slow or partial run hard to explain after the fact. This adds coloured progress and diagnostics.

What changed

• Reporter owns everything the script prints: the colour vocabulary (cyan phases, dim detail, green success, yellow warnings, red errors), the progress bars, and thread-safe run counters. Tracker is its per-phase handle - a live rich bar on a terminal, one line every N steps when piped, so logs stay greppable.
• Phase banners: Discovery / Status / REST re-count, numbered against the phases the run actually plans (Phase 2/3), each closing with elapsed time and GraphQL points spent. When the REST phase is skipped because the sample covered everything, it says so instead of silently vanishing.
• Bars show count, percent (speed for the indeterminate discovery pass), a live detail field, elapsed and ETA. The points balance is a traffic light - green above 2000, yellow down to the 200-point floor, red below - so an approaching budget stop is visible before it happens.
• Run diagnostics table at the end: queries, retries, give-ups, batches split, repos skipped, REST requests, repos re-counted, points spent, wall time. Failure counters stay dim at zero and colour when they are not.
• New flags: -v/--verbose (retries with backoff and error, batch splits, discovery cursors, each REST re-count next to what GraphQL sampled), -q/--quiet, --no-color (also honours NO_COLOR, for the bars too).
• Everything goes to stderr, so --json on stdout stays machine-readable.
• The open-PR distribution is now a table with the knee of the curve highlighted; result tables gained a showing top N of M caption and a coloured source column.

Drive-by fix

Writing the retry diagnostics surfaced a real defect: both retry loops slept their full backoff after the final attempt - up to 256s of dead wait before giving up. They now break out instead, which also makes the new "retrying in Ns" message truthful.

Verification

• An offline harness with canned GitHub payloads exercised discovery paging, a forced 502 -> retry -> batch split -> skipped repo, the REST re-count, and the piped / TTY / verbose / quiet / JSON / no-color paths.
• A live --repos-file run against apache/airflow (37 running jobs, re-counted over REST).
• ruff check clean.

README is updated in the same commit: the three new flags in the options table plus a "Progress and Diagnostics" section with sample output.

Example, mid-sweep:

> Phase 2/3  Status  62 queries of up to 20 repos
  Status ------------------  38/62  61% 4102 pts 0:01:12 eta 0:00:45

Follow-up: totals on the tables

The org-wide total is printed once, above both tables, which is where it cannot be
read: with --top 30 of sixty active repos it has scrolled off by the time the table
below it has been. Each table now closes with a TOTAL row - the same counts the
summary line and CSVs already carry, plus the active-repo count - and when --top
truncates, a dim shown (top N) row subtotals the visible rows.

Follow-up: the discovered repo list is stored in the repo

Discovery walks every repository in the org before a single job is counted - the
slowest and most rate-limit-hungry phase of a sweep - and what it finds changes slowly.
The current answer is committed as utils/apache-actions-repos.txt: 1258 apache
repositories that define workflow files, ready to hand back with --repos-file.

• --save-repos writes through write_repos_file(): ASF header, count, discovery date,
  names sorted rather than left in discovery's push order, which reshuffles every run.
  A refresh should diff as the repos that joined and left, nothing else.
• --repos-file skips # lines, so the header survives the round trip.
• Discovery now pages at 50 repositories instead of 100. The README tells people to
  refresh with --save-repos, and at 100 that command does not finish: two consecutive
  full runs died on HTTP 502, after 200 and 300 repositories, with every retry
  exhausted. The same paging at 50 walked all 3186 repositories without a single retry.
• README gains "The Stored Repository List": where it lives, how to pass it back, and why
  it must be refreshed - a stale list fails silently, reporting totals across the
  repositories it was given with no way to know which are missing.

The stored list: 1258 repositories across 225 PMC prefixes

casbin 161 | maven 86 | commons 47 | incubator 43 | cordova 38 | skywalking 35 | flink 28
dubbo 25 | pekko 22 | logging 19 | infrastructure 18 | camel 17 | openwhisk 17 | rocketmq 16
accumulo 15 | arrow 15 | datasketches 15 | pulsar 14 | cassandra 13 | doris 13 | datafusion 12
paimon 12 | shenyu 11 | airflow 10 | apisix 10 | couchdb 10 | openserverless 10 | cloudstack 9
fineract 8 | grails 8 | iotdb 8 | netbeans 8 | ozone 8 | qpid 8 | spark 8 | tooling 8 | beam 7
echarts 7 | mynewt 7 | solr 7 | aries 6 | daffodil 6 | eventmesh 6 | opendal 6
shardingsphere 6 | teaclave 6 | tomcat 6 | yunikorn 6 | cloudberry 5 | devlake 5 | directory 5
fluss 5 | geode 5 | hugegraph 5 | iceberg 5 | karaf 5 | nifi 5 | opennlp 5 | seatunnel 5
servicecomb 5 | airavata 4 | artemis 4 | axis 4 | cxf 4 | dolphinscheduler 4 | geronimo 4
guacamole 4 | hadoop 4 | ignite 4 | kyuubi 4 | nuttx 4 | phoenix 4 | ponymail 4 | sedona 4
tomee 4 | ws 4 | xmlgraphics 4 | activemq 3 | age 3 | answer 3 | calcite 3 | comdev 3
creadur 3 | db 3 | hertzbeat 3 | iggy 3 | jackrabbit 3 | kafka 3 | kvrocks 3 | myfaces 3
otava 3 | parquet 3 | plc4x 3 | ratis 3 | security 3 | struts 3 | trafficserver 3 | tvm 3
yetus 3 | ambari 2 | amoro 2 | apachecon 2 | apr 2 | asyncband 2 | auron 2 | avro 2 | bifromq 2
brpc 2 | buildstream 2 | bval 2 | celeborn 2 | curator 2 | drill 2 | druid 2 | felix 2 | fory 2
geaflow 2 | gravitino 2 | groovy 2 | hive 2 | httpcomponents 2 | httpd 2 | hudi 2 | inlong 2
libcloud 2 | linkis 2 | lucene 2 | lucenenet 2 | magpie 2 | mina 2 | nutch 2 | openoffice 2
openwebbeans 2 | orc 2 | ossie 2 | poi 2 | polaris 2 | ranger 2 | shiro 2 | sling 2
sourcelume 2 | storm 2 | stormcrawler 2 | streampark 2 | streampipes 2 | superset 2 | systemds 2
terraform 2 | thrift 2 | tika 2 | tsfile 2 | uima 2 | uniffle 2 | unomi 2 | wayang 2
zeppelin 2 | allura 1 | atlas 1 | attic 1 | bigtop 1 | bookkeeper 1 | burr 1 | caldera 1
carbondata 1 | causeway 1 | cayenne 1 | celix 1 | conferences 1 | ctakes 1 | datafu 1
deltaspike 1 | fesod 1 | flagon 1 | flex 1 | freemarker 1 | gluten 1 | gobblin 1 | gora 1
hamilton 1 | hbase 1 | helix 1 | hop 1 | jena 1 | jmeter 1 | johnzon 1 | jspwiki 1 | juneau 1
knox 1 | livy 1 | mahout 1 | maka 1 | manifoldcf 1 | ofbiz 1 | openjpa 1 | openmeetings 1
ozhera 1 | pdfbox 1 | petri 1 | pinot 1 | pouchdb 1 | rai 1 | roller 1 | royale 1 | samza 1
santuario 1 | sdap 1 | serf 1 | singa 1 | spamassassin 1 | steve 1 | subversion 1 | synapse 1
syncope 1 | tapestry 1 | template 1 | texera 1 | tez 1 | tinkerpop 1 | voxpupuli 1 | whimsy 1
wicket 1 | www 1 | xalan 1 | xerces 1 | zookeeper 1

Repositories per PMC prefix, largest first. 69 prefixes hold a single
repository. The names themselves are in utils/apache-actions-repos.txt.

## Follow-up: `--by-pmc`

"Which project is eating the runners" is a different question from "which repository is",
and several PMCs spread their CI over a dozen repos. --by-pmc changes the unit of the
report; both orderings, the TOTAL footer, --top, --csv and --json are unchanged.

| PMC     | Queued | Running | Repos | Repositories                   |
| airflow |     13 |     133 |     2 | airflow, airflow-client-python |
| spark   |      1 |      83 |     1 | spark                          |
| TOTAL   |     14 |     216 |     3 | 2 PMCs                         |

The Repos column reads active / total - how many of a PMC's repositories have jobs now,
out of every repository of theirs the sweep covered (60 / 1258 for a full org sweep).
Four busy repos read differently for a PMC of four than for one of forty-seven.

A repo's PMC is the text before the first hyphen, the whole name when there is none -
the rule actions-audit.py's --pmc already uses and the README documents, so the two
scripts agree on what spark covers, with no committee list and no network call. It is
a naming convention, not authoritative ownership: an incubator- repo groups under
incubator, which the README states. The Source column gives way to a repo count
(which API counted a row is a per-repo fact a PMC can only blur); the CSV keeps it and
reads mixed where a PMC's repos were counted different ways.

Verification of the follow-ups

• Totals and grouped tables rendered in truncated and untruncated form; subtotals plus
  hidden rows tie out to TOTAL; grouped sums equal the per-repo sums they came from.
• pmc_of checked on no-hyphen, multi-hyphen, incubator-* and owner-prefixed names.
• Round-trip: the committed list loads back as 1258 names, sorted, no comments leaking.
• Live runs: a 3-repo list with comment and blank lines; a mixed-PMC list grouped and
  ungrouped; --json and --csv shapes checked in both modes.
• A full org sweep produced the stored list and the snapshot behind it: 3186 repos
  scanned, 1258 with Actions, 881 jobs running and 330 queued across 60 repos.
• pre-commit clean.

Not verified: RAT could not be run locally (archive download blocked, and
utils/rat.sh fails on macOS - its Java-version parse uses \?, a GNU-sed extension
BSD sed does not support). The header in the new .txt is byte-identical to
approved_patterns.yml's and no insert-license hook covers .txt, so nothing strips
it - but CI's RAT job is the real check.

🤖 Generated with Claude Code

https://claude.ai/code/session_015veDEPK6ZLK9ot4NBQaC13
https://claude.ai/code/session_01LefMKj1fh7D2RxwNC6A4L5

[potiuk]

cc: @ppkarwasz

[potiuk]

Gentle nudge on this one, @dfoulks1 @ppkarwasz - the ASF Actions queue is looking very long again, and these are exactly the small fixes that make the tool we use to look at it more robust: stored repo list so a sweep does not re-discover 1200+ repos every time, progress output so a long sweep is not a black box, PMC grouping and totals so the snapshot is readable.

Low risk - utils/ tooling only, no workflow or action surface. CI is green.

Note these two overlap (both add utils/apache-actions-repos.txt and touch actions-queue-status.py), so whichever lands first will need the other rebased on top - happy to do that as soon as one goes in.