Skip to content

More locking around flash operations - #3503

Open
cammeresi wants to merge 3 commits into
meshcore-dev:mainfrom
cammeresi:extrafs-flash-lock
Open

cammeresi wants to merge 3 commits into
meshcore-dev:mainfrom
cammeresi:extrafs-flash-lock

Conversation

@cammeresi

Copy link
Copy Markdown

While hacking on a custom client talking to a RAK WisMesh 1W Booster, I observed that after several connections, I would get permanent BLE failure until I reflashed the device. The BLE connection would initially start but fail during negotiation, and trial and error revealed that removing a request of the battery status (which also includes filesystem state) caused the problem to not recur. Thus the failure was guessed to be related to some kind of corruption of BLE state stored in flash caused by concurrent flash access, so at length the firmware was consulted.

A couple of locking irregularities were identified, the most serious of which was the lack of synchronization between InternalFS and ExtraFS on the flash cache. The lfs_traverse function was also insufficiently synchronized, and one other minor off-by-one error in the flash was found.

Before these patches, I could trigger permanent BLE failure within 5 to 10 connections. With these patches, I was able to connect 100 times without a failure.

On nRF52 boards with EXTRAFS, InternalFS and ExtraFS both share the same
flash cache, which has no locking, and the two filesystems each had
their own mutex.  Thus Bluefruit could write bonds to InternalFS from
its task while the main loop read and wrote ExtraFS.

During testing of a custom client, permanent BLE connection failure was
observed after around ten connections.  The problem was traced to a
request for battery level (which includes filesystem state for free)
that appeared to occasionally corrupt the flash.

Add a LockedLFS wrapper around CustomLFS so ExtraFS operations go
through InternalFS's lock.  The lock order is ExtraFS then InternalFS.

After this fix, a run of 100 connections was made with no BLE failure.
CMD_GET_BATT_AND_STORAGE computes used storage with lfs_traverse on the
raw lfs_t from _getFS().  Adafruit_LittleFS takes its mutex around all
of its own LittleFS calls, but this call skips it.

LittleFS is not thread safe, so this call should be synchronized with
the rest of them.
_countLfsBlock rejects blocks greater than the filesystem's block
count, but valid blocks run from 0 to count - 1, so a block equal to
the count was let through.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant