fix: Update choreographer and rely on its message chunking

[camdecoster]

Description

Update choreographer to v1.4.0 and rely on its message chunking. Remove the chunking code added in #442 (and the problematic serialization).

Closes #343.
Closes #423.
Closes #465.
Closes #467.

Changes

• Update choregrapher
  • Drop support for Python 3.8 per above
• Remove Kaleido chunking
• Add encoding test
• Drop orjson as dependency

Testing

• Be on main
• Run the following script:

  import json
  import time
  from decimal import Decimal
  from importlib.metadata import version
  
  import numpy as np
  import pandas as pd
  import plotly.graph_objects as go
  
  import kaleido
  
  print(f"kaleido {version('kaleido')}, choreographer {version('choreographer')}")
  
  # Each case: trace values, the key to read back, and what plotly.py writes
  cases = {
      "Decimal": ({"y": [Decimal("10.5"), Decimal(20)]}, "y", [10.5, 20]),
      "Timestamp": ({"x": [pd.Timestamp("2024-01-01")]}, "x", ["2024-01-01T00:00:00"]),
      "NaT": ({"x": [pd.Timestamp("2024-01-01"), pd.NaT]}, "x", ["2024-01-01T00:00:00", None]),
      "NA": ({"customdata": [1, pd.NA]}, "customdata", [1, None]),
  }
  for name, (trace, key, expected) in cases.items():
      fig = go.Figure(go.Scatter(**{"x": [1, 2], "y": [1, 2], **trace}))
      try:
          out = json.loads(kaleido.calc_fig_sync(fig, opts={"format": "json"}))
          got = out["data"][0][key]
          status = "OK  " if got == expected else "FAIL"
          print(f"{status} {name:9} got {got!r}")
      except Exception as e:  # noqa: BLE001
          print(f"FAIL {name:9} {type(e).__name__}: {e}")
  
  # About 140 MiB of JSON, over Chrome's 100 MiB devtools buffer
  n = 6_000_000
  x = np.arange(n, dtype=float)
  big = go.Figure(go.Scatter(x=x, y=np.sin(x / 100_000), mode="lines"))
  start = time.perf_counter()
  png = kaleido.calc_fig_sync(big, opts={"format": "png"})
  print(f"OK   large figure: {len(png):,} byte PNG in {time.perf_counter() - start:.1f}s")

• Note that the script fails with the pd.NaT and pd.NA values
• Switch to this branch
• Run the script again
• Note that the Pandas values are serialized properly
• The large message chunking should work on both branches

Notes

• Python 3.8 support was dropped in choreographer 1.4.0 and I brought that change to Kaleido in this PR

[KoolADE85]

I tested a number of python versions on my end, and this is working well.
Left a couple of small comments for you, but I'm basically good with this.

As for dropping 3.8, should this be a major version bump?

[KoolADE85]

It looks like we're missing a test here for #467 (integers > 2**64). I confirmed manually that it works, but would be good to have a test for it.

[camdecoster]

Good point. I'll add a test.

[KoolADE85]

nit: 3.8 is now dropped.

[camdecoster]

I'll update the comment to make it clearer. The Listish is still needed.

[camdecoster]

I'll make the next release a patch or minor bump since 3.8 is so old that no one should still be using it.

[camdecoster]

I'll update the comment to make it clearer. The Listish is still needed.

[camdecoster]

Good point. I'll add a test.