Skip to content

LT-21834: Update libpalaso to the build with the abandoned mutex fix - #1164

Merged
thejambi merged 1 commit into
mainfrom
LT-21834-update-libpalaso
Sep 29, 2026
Merged

thejambi merged 1 commit into
mainfrom
LT-21834-update-libpalaso

Conversation

@thejambi

@thejambi thejambi commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Start here: Build/SilVersions.props. The whole diff is one line; review what that line pulls in, not the line.

Moves SilLibPalasoVersion from 18.0.0-beta0030 to 18.0.0-beta0042, the first libpalaso build carrying the LT-21834 fix (libpalaso#1544). A mutex abandoned by a crashed process made FLEx crash on every later project open; GlobalMutex now recovers from the abandonment instead. It also fixes LT-21467, open since 2023: re-entering Crop in Picture Properties failed with a GDI+ error. No FieldWorks code changes.

What else comes with it? The pin moves all fourteen libpalaso packages, so twelve upstream commits land, not one. Eight are test repairs, CI plumbing, or fixes in code FieldWorks does not call. Two are fixes in code FLEx reaches: #1535 (writing system Replace after an interrupted update) and #1536 (ImageCropper disposal). One is a declared breaking change that FLEx also reaches.

Where to look

  • libpalaso#1530 makes a cropped image's RawFormat MemoryBmp rather than Jpeg. PicturePropertiesDialog saves through PalasoImage.Save, which picks the encoder from the file extension. Verified by hand: a crop saved as .jpg is a genuine, complete JPEG.
  • FwNewLangProjectModel_CanFinish_FalseIfNoneComplete is flaky, in the writing-system code this fix touches. A control run shows it is already flaky on main: it failed 2 of 3 full runs without the bump and 1 of 3 with it.

Not here

  • 18.0.0-beta0043, a CHANGELOG-only rebuild. beta0042 is pinned because its package provenance is the fix's merge commit.
  • A ticket for the flaky test.

Verification

  • build.ps1 -CommentHygiene -TokenHygiene clean. test.ps1: 6317 tests, 6255 passed, 62 skipped, 0 failed.
  • LT-21834 and LT-21467 each reproduced on installed FLEx 9.3.9 and fixed on this branch; steps and stacks below.
  • Testers: the first launch after upgrading out of the crash loop shows "The last time FLEx started, it stopped responding…" once. That is FLEx reporting the previous crash, not a new one.

Next: approve, or tell me to take beta0043 instead. On merge, resolve LT-21467 with LT-21834.


Reading this a year from now -- start here

This is a version-pin PR. The fix itself is in libpalaso, and the reasoning behind it lives on libpalaso#1544. What this record keeps is the FieldWorks side: which build to pin, why no FieldWorks code changed, how both bugs were reproduced, and the evidence behind the risk claims above. No working documents were written on this branch, so nothing was deleted from the tree.

Decisions, and why

No FieldWorks code change. The recovery has to happen where the mutex is acquired. The GlobalMutex that throws is private to GlobalWritingSystemRepository, so FieldWorks can only catch the exception after the fact, by which point the catching thread already owns the mutex and nothing it can reach will release it. See Paths not taken.

beta0042, not beta0043. beta0043 adds a CHANGELOG section and nothing else (libpalaso#1548), so the two are functionally identical. beta0042's nuspec records repository commit="8948e6c5…", the LT-21834 merge itself, which makes the pin self-describing.

One bump, not a catch-up followed by the fix. beta0041 is built from 5ecf001f, the commit immediately before the fix, so the other eleven commits could have landed separately. They were taken together because the full suite and both manual checks ran against beta0042, and a separate catch-up would repeat that validation without changing what ships.

Paths not taken

Catching the exception in FieldWorks (#1141, closed). A catch in FieldWorks.cs turned the crash into an "Unable to Open Project" dialog, and retrying from it opened the project. Measured, it was worse than the crash. The thread that receives AbandonedMutexException owns the mutex, so the retry was a recursive acquisition that returned immediately. The mutex was then held for the whole session, blocking other SIL programs that share the writing system store, and abandoned again on exit, so the dialog returned on every launch.

Reproducing LT-21467 -- for testers
  1. On a sense, choose Insert → Picture.
  2. Click Get Image and choose a JPEG. Only JPEGs take the failing path.
  3. Click Crop and drag one edge. An untouched crop is now skipped, so without the drag the fix is never exercised.
  4. Click Get Image, then Crop again.

Before the fix: "Sorry, something went wrong with the ImageToolbox", whose details show A generic error occurred in GDI+ thrown from ImageCropper.set_Image. Re-entering Crop hands the cropper an image backed by a stream disposed when it was made. With the fix, the cropper reopens with the image intact.

On a developer machine, an installed FLEx reads the dev tree's configuration: every build writes HKCU\SOFTWARE\SIL\FieldWorks\9\RootCodeDir and RootDataDir (setKeysInHKCU, Build/mkall.targets:279), and FwDirectoryFinder.GetDirectory prefers them over the install path. The symptom is an XCore error naming AiAnalysisExportListener. For a clean before, back up that key and remove the two values; restore them, or rebuild, before running the branch build.

Reproducing LT-21834 -- for testers

A named mutex exists only while some process holds a handle to it, so abandoning it takes two processes: one that keeps a handle open, standing in for a second SIL program, and one that acquires it and exits without releasing it.

  1. Close every copy of FLEx.
  2. In PowerShell, keep a handle open for ten minutes:
    $name = 'C:_ProgramData_SIL_WritingSystemRepository_3'
    Start-Process powershell -WindowStyle Hidden -ArgumentList '-NoProfile','-Command',"`$m = New-Object System.Threading.Mutex(`$false, '$name'); Start-Sleep -Seconds 600"
  3. Acquire the mutex and exit without releasing it:
    Start-Process powershell -WindowStyle Hidden -Wait -ArgumentList '-NoProfile','-Command',"`$m = New-Object System.Threading.Mutex(`$false, '$name'); [void]`$m.WaitOne(); [Environment]::Exit(0)"
  4. Start FLEx and open any project.

Before the fix, FLEx dies before its main window appears, with the crash reporter showing The wait completed due to an abandoned mutex. (AbandonedMutexException, raised from SIL.Threading.GlobalMutex.Lock() under FieldWorks.EnsureDefaultCollationsPresent). The crash abandons the mutex again, so it repeats on every launch. With the fix, the project opens with no error.

Run only one FLEx at a time. A second instance opening the same project meets FieldWorks' own project-in-use handling, which looks like a failure but is unrelated.

Surprising findings
  • The first launch after upgrading shows a notice. kstidUnableToOpenLastProject ("The last time FLEx started, it stopped responding when attempting to open the FieldWorks project: …") is raised at FieldWorks.cs:1434 when the previous launch left LoadingProcessId set (:3081) and its process is gone. A successful load clears it (:3101), so a user leaving the crash loop sees the notice once, naming the project that last failed.
  • A broken crop would have looked like a locked file. PicturePropertiesDialog.ApplySaveFile catches every exception from PalasoImage.Save and shows ksErrorFileInUse ("This file is probably open somewhere"), writing the real exception only to the log. Had #1530 broken the save path, a tester would most likely have dismissed it as a file-locking problem.
Preflight review details

Code Review Summary

Branch: LT-21834-update-libpalaso
Base: origin/main (30564cc)
Date: 2026-09-28
Review model: Claude Opus 5 and Claude Opus 5.5 (Claude Code)
Files changed: 1

Overview

Raises SilLibPalasoVersion from 18.0.0-beta0030 to 18.0.0-beta0042 so that
FieldWorks picks up the fix for LT-21834, where an abandoned GlobalMutex left
FLEx unable to open a project again until the machine was rebooted. The fix
itself is in libpalaso (#1544,
merged as 8948e6c5); no FieldWorks code change is needed, so the version pin is
the whole of the FieldWorks-side fix.

The pin is a single property consumed by Directory.Packages.props and the
restore targets, so one line moves all fourteen libpalaso packages. That makes
the diff trivial and the risk surface large: twelve libpalaso commits come with
it, not one. The analysis below is therefore about the delta, not the diff.

The bump also fixes LT-21467, open since
2023: cropping in the Picture Properties dialog, switching to Get Image and
returning to Crop failed with a GDI+ error. The fix is libpalaso
#1530, which closed
libpalaso #1275, the issue filed against it; nothing had carried it into
FieldWorks until now.

One commit: 830daeedf. Branch is current with main as of 2026-09-28.

Contract/API Changes

None in FieldWorks. One in the dependency:

  • libpalaso #1530 is a
    declared BREAKING CHANGE. ImageCropper.GetCroppedImage now returns the
    cropped bitmap directly, so its RawFormat is MemoryBmp rather than Jpeg.
    Callers that read RawFormat, or that call Image.Save(path) and rely on the
    JPEG encoder being chosen implicitly, must now pass an explicit ImageFormat.
  • The remaining eleven commits are bug fixes, resource-leak fixes, test-flakiness
    fixes and CI plumbing. Notably
    #1535 fixes Replace
    failing forever after an interrupted writing system update, which is
    complementary to the LT-21834 fix rather than incidental.

Findings

Critical - Must address before merge

None.

Important - Should address before merge

  • The #1530 breaking change is reachable from FLEx. (validated
    during review: author exercised Insert Picture → crop → Save As on the branch
    build and reports it working; the saved file was then checked directly and is
    a genuine, complete JPEG — see Required Validation)
    PicturePropertiesDialog hosts ImageToolboxControl
    (Src/FwCoreDlgs/PicturePropertiesDialog.cs:21, .Designer.cs:45), which is
    the Insert Picture flow. Reading the code, FieldWorks is on the safe side of
    the change: it never reads RawFormat; it saves through
    imageToolbox.ImageInfo.Save(savePath) at PicturePropertiesDialog.cs:414,
    which is PalasoImage.Save and picks the encoder from the file extension, the
    path the libpalaso changelog names as safe; IsCropped (:68) compares image
    sizes, not formats; and FileFormatSupportsMetadata (:84) resolves through
    Metadata.FileFormatSupportsMetadata(path), a file path rather than
    RawFormat. No automated test covers the crop-and-save flow; the manual
    check below is what confirms the reading.

Minor - Consider

  • 18.0.0-beta0043 exists and was not taken. It is a CHANGELOG-only
    rebuild (Add missing 17.0.0 section to CHANGELOG,
    #1548), so it is
    functionally identical to beta0042. beta0042 was chosen deliberately
    because its nuspec repository commit is the LT-21834 merge itself, which
    makes the pin self-documenting. Worth a sentence in the PR so a reviewer does
    not read it as an oversight.
  • Untracked test output in the working tree (fixed during review:
    deleted AlloGenServiceTests/TestData/SharedSettings/ and
    TestData/WritingSystemStore/, left over from an interrupted test run; one
    file carried the developer's account name)
    .

Required Validation / Evidence

Run on the current head, after merging main:

  • .\build.ps1 -CommentHygiene -TokenHygiene — exit 0, 0 warnings, 0 errors.
    comment-hygiene clean, token-hygiene clean (176 files scanned),
    powershell-compat clean (5.1 and 7.0).
  • .\test.ps1 -CommentHygiene -TokenHygiene — 6317 tests, 6255 passed, 62
    skipped, 0 failed.
  • gitlint --ignore body-is-missing --commits origin/main..HEAD — exit 0.
  • Restored assemblies confirmed as the intended build, not a stale cache:
    Output\Debug\SIL.WritingSystems.dll and SIL.Core.dll report
    18.0.0-beta.42+Branch.master.Sha.8948e6c5e5a97a1c82b802ec9e87590078fb5a6d —
    the LT-21834 merge commit.

One pre-existing flaky test, investigated rather than waved through:

FwNewLangProjectModel_CanFinish_FalseIfNoneComplete failed once with
KeyNotFoundException : The writing system en was not found in this manager.
Because materialising en runs through the SLDR and global writing system store
paths that libpalaso #1544 rewrote, this could not be assumed unrelated. Control
experiment, reverting only the version pin and rebuilding:

Runs beta0030 (base) beta0042 (this branch)
Full suite failed 2 of 3 failed 1 of 3

The same test, the same exception, failing more often without the bump than with
it. It also passes in isolation (-TestProject FwCoreDlgsTests -TestFilter FwNewLangProjectModel_CanFinish_FalseIfNoneComplete, 1/1). Pre-existing on
main, order- or parallelism-sensitive, not caused by this change. It is not
documented as flaky anywhere in the repo and has no ticket.

Manual validation, on the branch build (Output\Debug\FieldWorks.exe), 2026-09-28:

  • LT-21834, reproduced and then fixed. The writing system store's mutex
    (C:_ProgramData_SIL_WritingSystemRepository_3) was left abandoned with a
    second handle held open, using Abandon-FwMutex.ps1. The abandoned state was
    confirmed from a separate process, whose WaitOne threw
    AbandonedMutexException, before either FLEx was started. Project:
    Lex Training Sample Project 1, one FLEx running at a time.

    Before — installed FLEx 9.3.9.1439 (SIL.WritingSystems 18.0.0-beta.13).
    Crashed during startup, before any main window, with the crash reporter:

    Msg: The wait completed due to an abandoned mutex.
    Class: System.Threading.AbandonedMutexException
       at SIL.Threading.GlobalMutex.Lock()
       at SIL.WritingSystems.GlobalWritingSystemRepository`1.TryGet(String id, T& ws)
       at SIL.WritingSystems.LocalWritingSystemRepositoryBase`1.OnChangeNotifySharedStore(T ws)
       at SIL.WritingSystems.LdmlInFolderWritingSystemRepository`1.Save()
       at SIL.LCModel.Core.WritingSystems.WritingSystemManager.Save()
       at SIL.FieldWorks.FieldWorks.EnsureDefaultCollationsPresent(LcmCache cache) FieldWorks.cs:909
       at SIL.FieldWorks.FieldWorks.CreateCache(ProjectId projectId) FieldWorks.cs:864
    

    The same path as the report on the ticket. The thread that received the
    exception owned the mutex and died with it, so the crash left it abandoned
    again — the repeating half of the bug, and the state the next step starts
    from.

    After — branch build. The Open Project dialog first showed "The last time
    FLEx started, it stopped responding when attempting to open the FieldWorks
    project: Lex Training Sample Project 1". That is
    kstidUnableToOpenLastProject, raised at FieldWorks.cs:1434 when the
    previous launch recorded StartupStatus.Failed in the registry settings both
    builds share: it reports step one's crash, not anything in this change. The
    project then opened normally, with no error.

    A user upgrading out of the crash loop should expect that notice once, on the
    first launch after the upgrade, naming the project that last failed.

  • #1530 crop path. Author inserted a JPEG through Insert Picture, cropped
    it and saved it, and reports it working. The saved file,
    LinkedFiles\Pictures\LT-21834-crop-test_croppped.jpg, was then inspected
    directly:

    Check Result
    Header FF D8 FF E0 — JPEG (JFIF)
    Trailer FF D9 — end-of-image marker present, file complete
    Decoded size 717 × 539, cropped from 1920 × 1200
    RawFormat read back from disk Jpeg

    The risk was a bitmap written under a .jpg name, since the in-memory crop is
    now MemoryBmp. PalasoImage.Save chose the encoder from the extension, as
    the code reading predicted.

The mutex holder was released with Abandon-FwMutex.ps1 -Stop afterwards.

  • LT-21467, reproduced and then fixed. Steps: Insert Picture on a sense,
    Get Image with a JPEG, Crop with one edge dragged, Get Image, Crop again.

    Before — installed FLEx 9.3.9. "Sorry, something went wrong with the
    ImageToolbox", with the details:

    Msg: A generic error occurred in GDI+.
    Class: System.Runtime.InteropServices.ExternalException
       at System.Drawing.Image.Save(String filename, ImageCodecInfo encoder, EncoderParameters encoderParams)
       at SIL.Windows.Forms.ImageToolbox.Cropping.ImageCropper.set_Image(PalasoImage value)
       at SIL.Windows.Forms.ImageToolbox.ImageToolboxControl.listView1_SelectedIndexChanged(Object sender, EventArgs e)
    

    Re-entering Crop hands the cropper an image backed by a stream that was
    disposed when it was created, and saving it throws — the mechanism #1530
    describes.

    After — branch build. The same steps return to the cropper with the image
    intact and no error. Author reports it working.

    For a clean before, the installed FLEx was run with the two
    HKCU\SOFTWARE\SIL\FieldWorks\9 values RootCodeDir and RootDataDir
    removed. Every build writes them (setKeysInHKCU, Build/mkall.targets:279),
    pointing at DistFiles, and FwDirectoryFinder.GetDirectory prefers them
    over the machine-wide install path, so on a developer machine the installed
    FLEx otherwise reads main's configuration over 9.3.9 binaries. With them
    present it reported AiAnalysisExportListener, a class main's Main.xml
    declares and 9.3.9 lacks; with them removed it opened normally. The key was
    backed up first and restored afterwards, matching the backup.

Positive Observations

  • The change is the minimum that carries the fix: one property, no code, no
    conditional compilation, no shims.
  • Manage-LocalLibraries.ps1 was used rather than hand-editing the props file,
    so the 15 stale package folders were cleared as part of the change. That
    avoids the LT-22728 trap where NuGet keeps serving an already-extracted
    package after its .nupkg is gone.
  • The delta was read commit by commit against the libpalaso clone rather than
    trusted as "twelve beta bumps", which is what surfaced the #1530 breaking
    change.

Interview Notes

  • Ticket: LT-21834 exists and this branch carries the key. The change is
    user-visible (FLEx failing to reopen a project), so it belongs in a tester's
    queue; no new ticket needed.
  • #1530 validation: author chose to verify the picture crop flow manually
    rather than ship on code reading alone or have the agent drive the UI.
    Performed on 2026-09-28 and reported good; the output file was independently
    verified.
  • LT-21834 validation: a first attempt was not counted. The installed FLEx
    was started while the branch build still had the project open, so it met
    FieldWorks' own project-in-use handling rather than the mutex. Rerun with one
    FLEx at a time: reproduced on 9.3.9, fixed on the branch. Author supplied the
    crash report text and the dialog text verbatim.
  • LT-21467: found while reviewing #1530's history. The issue it closed was
    filed with a link to this FLEx ticket, and the ticket was still open. Author
    ran the ticket's steps on both builds and reports reproduction on 9.3.9 and a
    clean pass on the branch.
  • Untracked test output: author chose deletion over leaving it untracked.
  • Unresolved: none.

In-Review Quality Check

Two directories of leftover test output were deleted. No source was modified
during review; the branch remains a one-line version pin.

Suggested Review Focus

  • Does taking twelve commits at once belong in an LT-21834 PR, or should the
    catch-up be separated? 18.0.0-beta0041 is built from 5ecf001f, the
    commit immediately before the fix, so it carries the other eleven
    without it. A separate catch-up to beta0041 is possible, at the cost of
    repeating this validation against it.
  • FwNewLangProjectModel_CanFinish_FalseIfNoneComplete is flaky on main
    today. Worth a ticket of its own?
  • LT-21467 should be resolved alongside LT-21834 when this merges, so it
    reaches testers.

🤖 Generated with Claude Code


This change is Reviewable

A GlobalMutex whose owner died without releasing it was left abandoned,
and every later attempt to take it threw. FLEx hit that when opening a
project, and because the failing caller abandoned the mutex again, the
crash repeated on every launch until the machine was rebooted. The fix
is in libpalaso, so nothing in FieldWorks changes but the version pin.

The pin moves all fourteen libpalaso packages at once, so twelve
upstream commits come with it rather than one. Eleven are bug fixes,
resource-leak fixes and test repairs. The twelfth is a declared
breaking change: ImageCropper.GetCroppedImage now returns a bitmap
whose RawFormat is MemoryBmp rather than Jpeg. PicturePropertiesDialog
is the only place FieldWorks reaches that code, and it saves through
PalasoImage.Save, which chooses the encoder from the file extension,
so it is on the safe side of the change.

beta0043 exists but only adds a CHANGELOG section. beta0042 is pinned
because its package provenance points at the fix's merge commit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

NUnit Tests

    1 files  ±0      1 suites  ±0   13m 38s ⏱️ +38s
6 301 tests ±0  6 216 ✅ ±0  85 💤 ±0  0 ❌ ±0 
6 310 runs  ±0  6 225 ✅ ±0  85 💤 ±0  0 ❌ ±0 

Results for commit 830daee. ± Comparison against base commit 30564cc.

@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 39.02%. Comparing base (30564cc) to head (830daee).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1164      +/-   ##
==========================================
- Coverage   39.02%   39.02%   -0.01%     
==========================================
  Files        1522     1522              
  Lines      352937   352937              
  Branches    40726    40726              
==========================================
- Hits       137741   137733       -8     
- Misses     185893   185900       +7     
- Partials    29303    29304       +1     

see 3 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@thejambi
thejambi marked this pull request as ready for review September 28, 2026 21:07

@mark-sil mark-sil left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mark-sil reviewed 1 file and all commit messages.
Reviewable status: :shipit: complete! all files reviewed, all discussions resolved (waiting on thejambi).

@thejambi
thejambi merged commit 506c2a6 into main Sep 29, 2026
9 checks passed
@thejambi
thejambi deleted the LT-21834-update-libpalaso branch September 29, 2026 20:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants