Skip to content

Unable to update Unicode version to 16.0.0 #138

Closed
#139
Closed
Unable to update Unicode version to 16.0.0#138
#139

Description

@adithyaov
adithyaov
opened on Jul 15, 2025

I'm trying to upgrade Unicode to 16.0.0
On running ./ucd generate (after ./ucd/download) I'm greeted with this:

Too many script extensions: 271

Interesting code blocks to look at:

UCD2Haskell/Modules/ScriptExtensions.hs:150

        encodedExtensions :: Map.Map (NE.NonEmpty BS.ShortByteString) Word8
        encodedExtensions = let len = length extensionsList in if len > 0xff
            then error ("Too many script extensions: " <> show len)
            else Map.fromList (zip extensionsList [0..])
        -- Encode single script as their script value
        extensionsList = singleScriptExtensions
                      <> Set.toList multiScriptExtensions

It looks like
length (singleScriptExtensions <> Set.toList multiScriptExtensions)
is > 271.

UCD2Haskell/Modules/ScriptExtensions.hs:141

        singleScriptExtensions = pure . getScriptAbbr <$> scripts
        singleScriptExtensionsSet = Set.fromList singleScriptExtensions
        multiScriptExtensions :: Set.Set (NE.NonEmpty BS.ShortByteString)
        multiScriptExtensions = Set.fromList (Map.elems extensions)
                                Set.\\ singleScriptExtensionsSet

singleScriptExtensionsSet cannot be more than 204 as:

$ cat data/16.0.0/ucd/ScriptExtensions.txt | grep -v '^\s*#' | grep -v '^\s*$' | wc -l
204

So multiScriptExtensions bumps the value up?

I've not investigated this further. We use genEnumBitmapShamochu to generate
the bitmap, and Shamochu restricts the number of elements to be < 0xff. So by
design, we don't allow extensionsList to have more than 0xff elements.

This is a bug if there SHOULD NOT be more than 0xff extensions, or this is a
LIMITATION of the generator if we don't support more than 0xff extensions.

I'm not sure how to fix this as I don't fully understand the problem.

@wismill could you please suggest how I should proceed?

Activity

  1. wismill commented on Jul 16, 2025

    @wismill
    wismill
    on Jul 16, 2025
    Collaborator

    I have not touched this for a long time 😅.

    This is a bug if there SHOULD NOT be more than 0xff extensions, or this is a
    LIMITATION of the generator if we don't support more than 0xff extensions.

    This is the current limitation: I tried to compress the data as much as possible, and it was fine using 8 bits with the previous releases of Unicode. However, the new scripts in 16.0 are incompatible with the previous optimization. So the obvious fix is to encode the script extensions on 16 bits rather than 8. This does not seems completely trivial though, because IIRC, most of the code relies on enums being encodable on 8 bits. We would need to write alternative versions of the relevant functions to make it work on 16 bits.

  2. adithyaov commented on Jul 16, 2025

    @adithyaov
    adithyaov
    on Jul 16, 2025
    MemberAuthor

    Hi @wismill

    Could you please let me know if I'm going about this in the right direction:

    The shamochu library provides lossless compression.

    Looking at:

    compress
    :: forall e. (FiniteBits e, Unbox e, Ord e)
    => NonEmpty Word
    -> [Word]
    -> Vector e
    -> CompressedBlob e

    It does not look like this has a size-based restriction on what it can
    compress. In our case, we pass in a Vector Word8, but we can just as well pass a
    Vector Word16.

    That said, there is already a more flexible bitmap generator:

    genBitmapShamochu ::
     HasCallStack
  => String
  -> NE.NonEmpty Word
  -- ^ Chunk size stage 1
  -> [Word]
  -- ^ Chunk size stage 2
  -> [Int]
  -> ShamochuCode

    as opposed to,

    genEnumBitmapShamochu
  :: forall a. (HasCallStack, Bounded a, Enum a, Eq a, Show a)
  => String
  -- ^ Function name
  -> Bool
  -- ^ If true, use raw Int
  -> NE.NonEmpty Word
  -- ^ Chunk size stage 1
  -> [Word]
  -- ^ Chunk size stage 2
  -> (a -> Word8)
  -- ^ Conversion
  -> (a, BB.Builder)
  -- ^ Value for planes 15-16
  -> (a, BB.Builder)
  -- ^ Default value
  -> [a]
  -- ^ List of values to encode for planes 0 to 3
  -> [a]
  -- ^ List of values to encode for plane 14
  -> ShamochuCode

    Would the implementation still be correct if I use genBitmapShamochu over
    genEnumBitmapShamochu?

    In this case, the Enums would be encoded in 64 bits. There probably will be some
    cascading changes, which I'll also have to fix.

  3. wismill commented on Jul 16, 2025

    @wismill
    wismill
    on Jul 16, 2025
    Collaborator

    The shamochu library provides lossless compression.

    Looking at:

    compress
    :: forall e. (FiniteBits e, Unbox e, Ord e)
    => NonEmpty Word
    -> [Word]
    -> Vector e
    -> CompressedBlob e

    It does not look like this has a size-based restriction on what it can compress. In our case, we pass in a Vector Word8, but we can just as well pass a Vector Word16.

    That’s correct.

    That said, there is already a more flexible bitmap generator:

    genBitmapShamochu ::
     HasCallStack
  => String
  -> NE.NonEmpty Word
  -- ^ Chunk size stage 1
  -> [Word]
  -- ^ Chunk size stage 2
  -> [Int]
  -> ShamochuCode

    […]

    Would the implementation still be correct if I use genBitmapShamochu over genEnumBitmapShamochu?

    The issue is that genBitmapShamochu actually encodes Bool, not other Ints. The last parameter is the list of indexes of the True values.

    In this case, the Enums would be encoded in 64 bits.

    While using more space seems inevitable and should not block updating to Unicode 16.0, a 64-bit (×8) encoding seems excessive here, so we should limit to 16 bits (×2)

    Do you have an agenda for the update? I would like to take a look at it, but I confess that my Haskell got rusty.

    It seems that the first step would be to better document all those functions 😅.

  4. adithyaov commented on Jul 16, 2025

    @adithyaov
    adithyaov
    on Jul 16, 2025
    MemberAuthor

    While using more space seems inevitable and should not block updating to Unicode 16.0, a 64-bit (×8) encoding seems excessive here, so we should limit to 16 bits (×2)

    I agree. I was trying to confirm the usage of genBitmapShamochu.

    Do you have an agenda for the update?

    Please see: #137

    I would like to take a look at it, but I confess that my Haskell got rusty.
    It seems that the first step would be to better document all those functions 😅.

    Please let me know if you need any help. Updating things as pseudo-code will
    also help a lot. I can take care of getting it to compile.

  5. wismill commented on Jul 16, 2025

    @wismill
    wismill
    on Jul 16, 2025
    Collaborator

    Do you have an agenda for the update?

    Please see: #137

    I meant do you have a dead line?

  6. adithyaov commented on Jul 16, 2025

    @adithyaov
    adithyaov
    on Jul 16, 2025
    MemberAuthor

    I meant do you have a dead line?

    Not really. There's no urgency per se.
    But the sooner, the better, I suppose. :-)

  7. wismill commented on Aug 26, 2025

    @wismill
    wismill
    on Aug 26, 2025
    Collaborator

    I finally found time to work on this, see #139. While I am confident that the shamochu package works correctly, I am still a bit uneasy there could be still some hard-coded bits in ucd2haskell that assume all values can be encoded as Word8. Tests are exhaustive and pass against ICU though, so I am maybe a bit paranoid 😅

  8. wismill
    closed this as completedin #139on Sep 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions