Skip to content

data(game): add game category + import 168,883 Wikidata games (→277k records) - #115

Merged
Seungpyo1007 merged 2 commits into
mainfrom
data/import-staging
Jul 9, 2026
Merged

data(game): add game category + import 168,883 Wikidata games (→277k records)#115
Seungpyo1007 merged 2 commits into
mainfrom
data/import-staging

Conversation

@Seungpyo1007

@Seungpyo1007 Seungpyo1007 commented Jul 9, 2026

Copy link
Copy Markdown
Member

Summary

Adds the new game category and imports 168,883 video games from Wikidata — the first big step of the 1M-record goal (dataset 108,575 → 277,458).

  • Source: Wikidata SPARQL, ?g wdt:P31 wd:Q7889 (174,919 games). Per game: name, release_date (P577, earliest), platforms (P400), genres (P136), developers (P178), publishers (P123). source_urls cites each game's Wikidata entity. No fabrication — every field is straight from Wikidata; games with no English label or unparseable data were skipped.
  • Category: wired via TechEngine PR ci(pr): cap metadata file scans for large imports #41 (merged) — standalone, unscored, no brand FK (makers are free-text developer/publisher lists). This PR bumps the engine submodule to that pin and mirrors the GAME_REQUIRED rule into app/validate.py.
  • Layout: sharded data/game/<xx>/<slug>.json; slugs kebab-cased, name-collisions disambiguated by Wikidata QID.
  • All records verified: false. python -m app.validate ✅ passed across all 277,458 records. Public dump regenerated — site/public/v1/games/ (168,883 detail pages + list) + openapi.json + v1/index.json.

Next toward 1M: RAWG (474k) once downloaded, then IGDB.

Closes #1

Adds the new `game` category (TechEngine PR #41): 168,883 video games sourced
from Wikidata (P31=Q7889) via SPARQL — name, release_date, platforms, genres,
developers, publishers; source_urls cite each Wikidata entity. All verified:false.
Bumps the engine submodule to the game-category pin and mirrors the GAME_REQUIRED
rule into app/validate.py. Sharded under data/game/<xx>/. No fabrication —
every field comes from Wikidata.

Refs #1
@Seungpyo1007 Seungpyo1007 added data Dataset changes enhancement New feature or request labels Jul 9, 2026
@github-actions github-actions Bot added the app Validator or application code changes label Jul 9, 2026
@Seungpyo1007

Copy link
Copy Markdown
Member Author

🤖 TechEngine · data validation

category    added     verified   total records
game        168,883    false        277,458
  • python -m app.validate✅ passed (0 errors) across all 10+1 categories.
  • New game category seeded from Wikidata; engine submodule bumped to the game-category pin (TechEngine PR ci(pr): cap metadata file scans for large imports #41).
  • Dump regenerated: 168,883 game detail pages + list + openapi. Other collections untouched (surgical staging).
  • Slug fix applied: 10 truncation-trailing-hyphen slugs corrected before commit.

@Seungpyo1007

Copy link
Copy Markdown
Member Author

🤖 TechEngine · verify (Tier 0)

Real data, not fabricated:

  • Every game is a Wikidata entity (P31 = Q7889); its source_urls links straight back to wikidata.org/wiki/<QID> for provenance.
  • Fields are the entity's own statements: publication date (P577), platforms (P400), genres (P136), developer (P178), publisher (P123) — aggregated one row per game, English labels only.
  • Games lacking an English label or with unparseable dates were dropped rather than guessed. Duplicate QIDs from the resumed fetch were de-duplicated.

verified stays false; promotion remains the verify-network job's call. Safe to rebase-merge (submodule bump matches the auto-bump already on main).

@Seungpyo1007 Seungpyo1007 moved this from Todo to In Progress in TechAPI-Project Jul 9, 2026
@Seungpyo1007
Seungpyo1007 merged commit 44a7f59 into main Jul 9, 2026
5 checks passed
@github-project-automation github-project-automation Bot moved this from In Progress to Done in TechAPI-Project Jul 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Assignees

Couldn't load assignees.