Skip to content

De-lock mmap readers. #866

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
evoskuil merged 6 commits into from
Jul 21, 2026
Merged

De-lock mmap readers. #866

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
9f27861
Remove dead code.
evoskuil Jul 21, 2026
b9ba0cd
Bump ultrafast NuGet to 4.5.0.0
evoskuil Jul 21, 2026
657e600
mmap mechanical atomics conversion.
evoskuil Jul 21, 2026
0dc4cf5
De-lock mmap readers.
evoskuil Jul 21, 2026
e857229
Patch macOS fallocate shim.
evoskuil Jul 21, 2026
015a72e
Regenerate test runner.
evoskuil Jul 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 1 addition & 7 deletions builds/gnu/test_runner.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,7 @@

# Define tests and options.
#==============================================================================
BOOST_UNIT_TEST_OPTIONS=\
"--log_level=warning "\
"--show_progress=no "\
"--detect_memory_leak=0 "\
"--report_level=no "\
"--build-info=yes "\
"--run_tests=*"
BOOST_UNIT_TEST_OPTIONS="--log_level=warning --show_progress=no --detect_memory_leak=0 --report_level=no --build_info=yes --run_test=*"

# Run tests.
#==============================================================================
Expand Down
3 changes: 0 additions & 3 deletions builds/msvc/vs2022/libbitcoin-database-test/libbitcoin-database-test.props
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,9 +48,6 @@
<Linkage-libbitcoin-system>static</Linkage-libbitcoin-system>
<Linkage-libbitcoin-database>static</Linkage-libbitcoin-database>
</PropertyGroup>
<PropertyGroup Condition="'$(Option-ultrafast)' == 'true' And '$(Option-cuda)' == 'true'">
<Linkage-ultrafast>cuda</Linkage-ultrafast>
</PropertyGroup>

<!-- Messages -->

Expand Down
3 changes: 0 additions & 3 deletions builds/msvc/vs2022/libbitcoin-database-tools/libbitcoin-database-tools.props
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,9 +45,6 @@
<Linkage-libbitcoin-system>static</Linkage-libbitcoin-system>
<Linkage-libbitcoin-database>static</Linkage-libbitcoin-database>
</PropertyGroup>
<PropertyGroup Condition="'$(Option-ultrafast)' == 'true' And '$(Option-cuda)' == 'true'">
<Linkage-ultrafast>cuda</Linkage-ultrafast>
</PropertyGroup>

<!-- Messages -->

Expand Down
3 changes: 0 additions & 3 deletions builds/msvc/vs2022/libbitcoin-database/libbitcoin-database.props
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,9 +44,6 @@
<Linkage-ultrafast Condition="'$(Option-ultrafast)' == 'true'">static</Linkage-ultrafast>
<Linkage-libbitcoin-system>static</Linkage-libbitcoin-system>
</PropertyGroup>
<PropertyGroup Condition="'$(Option-ultrafast)' == 'true' And '$(Option-cuda)' == 'true'">
<Linkage-ultrafast>cuda</Linkage-ultrafast>
</PropertyGroup>

<!-- Messages -->

Expand Down
3 changes: 0 additions & 3 deletions builds/msvc/vs2026/libbitcoin-database-test/libbitcoin-database-test.props
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,9 +48,6 @@
<Linkage-libbitcoin-system>static</Linkage-libbitcoin-system>
<Linkage-libbitcoin-database>static</Linkage-libbitcoin-database>
</PropertyGroup>
<PropertyGroup Condition="'$(Option-ultrafast)' == 'true' And '$(Option-cuda)' == 'true'">
<Linkage-ultrafast>cuda</Linkage-ultrafast>
</PropertyGroup>

<!-- Messages -->

Expand Down
4 changes: 2 additions & 2 deletions builds/msvc/vs2026/libbitcoin-database-test/libbitcoin-database-test.vcxproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -270,7 +270,7 @@
<Import Project="$(NuGetPackageRoot)libbitcoin-boost_unit_test_framework-vc145.1.91.0\build\libbitcoin-boost_unit_test_framework-vc145.targets" Condition="Exists('$(NuGetPackageRoot)libbitcoin-boost_unit_test_framework-vc145.1.91.0\build\libbitcoin-boost_unit_test_framework-vc145.targets')" />
<Import Project="$(NuGetPackageRoot)libbitcoin-boost_url-vc145.1.91.0\build\libbitcoin-boost_url-vc145.targets" Condition="Exists('$(NuGetPackageRoot)libbitcoin-boost_url-vc145.1.91.0\build\libbitcoin-boost_url-vc145.targets')" />
<Import Project="$(NuGetPackageRoot)secp256k1_vc145.0.6.1.2\build\native\secp256k1_vc145.targets" Condition="Exists('$(NuGetPackageRoot)secp256k1_vc145.0.6.1.2\build\native\secp256k1_vc145.targets')" />
<Import Project="$(NuGetPackageRoot)UltrafastSecp256k1-vc145.4.4.0.4\build\native\UltrafastSecp256k1-vc145.targets" Condition="Exists('$(NuGetPackageRoot)UltrafastSecp256k1-vc145.4.4.0.4\build\native\UltrafastSecp256k1-vc145.targets')" />
<Import Project="$(NuGetPackageRoot)UltrafastSecp256k1-vc145.4.5.0.0\build\native\UltrafastSecp256k1-vc145.targets" Condition="Exists('$(NuGetPackageRoot)UltrafastSecp256k1-vc145.4.5.0.0\build\native\UltrafastSecp256k1-vc145.targets')" />
</ImportGroup>
<Target Name="EnsureNuGetPackageBuildImports" BeforeTargets="PrepareForBuild">
<PropertyGroup>
Expand All @@ -286,7 +286,7 @@
<Error Condition="!Exists('$(NuGetPackageRoot)libbitcoin-boost_unit_test_framework-vc145.1.91.0\build\libbitcoin-boost_unit_test_framework-vc145.targets')" Text="$([System.String]::Format('$(ErrorText)', '$(NuGetPackageRoot)libbitcoin-boost_unit_test_framework-vc145.1.91.0\build\libbitcoin-boost_unit_test_framework-vc145.targets'))" />
<Error Condition="!Exists('$(NuGetPackageRoot)libbitcoin-boost_url-vc145.1.91.0\build\libbitcoin-boost_url-vc145.targets')" Text="$([System.String]::Format('$(ErrorText)', '$(NuGetPackageRoot)libbitcoin-boost_url-vc145.1.91.0\build\libbitcoin-boost_url-vc145.targets'))" />
<Error Condition="!Exists('$(NuGetPackageRoot)secp256k1_vc145.0.6.1.2\build\native\secp256k1_vc145.targets')" Text="$([System.String]::Format('$(ErrorText)', '$(NuGetPackageRoot)secp256k1_vc145.0.6.1.2\build\native\secp256k1_vc145.targets'))" />
<Error Condition="!Exists('$(NuGetPackageRoot)UltrafastSecp256k1-vc145.4.4.0.4\build\native\UltrafastSecp256k1-vc145.targets')" Text="$([System.String]::Format('$(ErrorText)', '$(NuGetPackageRoot)UltrafastSecp256k1-vc145.4.4.0.4\build\native\UltrafastSecp256k1-vc145.targets'))" />
<Error Condition="!Exists('$(NuGetPackageRoot)UltrafastSecp256k1-vc145.4.5.0.0\build\native\UltrafastSecp256k1-vc145.targets')" Text="$([System.String]::Format('$(ErrorText)', '$(NuGetPackageRoot)UltrafastSecp256k1-vc145.4.5.0.0\build\native\UltrafastSecp256k1-vc145.targets'))" />
</Target>
<ItemGroup>
<ProjectReference Include="..\libbitcoin-database\libbitcoin-database.vcxproj">
Expand Down
2 changes: 1 addition & 1 deletion builds/msvc/vs2026/libbitcoin-database-test/packages.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
|
-->
<packages>
<package id="UltrafastSecp256k1-vc145" version="4.4.0.4" targetFramework="Native" />
<package id="UltrafastSecp256k1-vc145" version="4.5.0.0" targetFramework="Native" />
<package id="libbitcoin-boost" version="1.91.0" targetFramework="Native" />
<package id="libbitcoin-boost_container-vc145" version="1.91.0" targetFramework="Native" />
<package id="libbitcoin-boost_iostreams-vc145" version="1.91.0" targetFramework="Native" />
Expand Down
3 changes: 0 additions & 3 deletions builds/msvc/vs2026/libbitcoin-database-tools/libbitcoin-database-tools.props
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,9 +45,6 @@
<Linkage-libbitcoin-system>static</Linkage-libbitcoin-system>
<Linkage-libbitcoin-database>static</Linkage-libbitcoin-database>
</PropertyGroup>
<PropertyGroup Condition="'$(Option-ultrafast)' == 'true' And '$(Option-cuda)' == 'true'">
<Linkage-ultrafast>cuda</Linkage-ultrafast>
</PropertyGroup>

<!-- Messages -->

Expand Down
4 changes: 2 additions & 2 deletions builds/msvc/vs2026/libbitcoin-database-tools/libbitcoin-database-tools.vcxproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -139,7 +139,7 @@
<Import Project="$(NuGetPackageRoot)libbitcoin-boost_thread-vc145.1.91.0\build\libbitcoin-boost_thread-vc145.targets" Condition="Exists('$(NuGetPackageRoot)libbitcoin-boost_thread-vc145.1.91.0\build\libbitcoin-boost_thread-vc145.targets')" />
<Import Project="$(NuGetPackageRoot)libbitcoin-boost_url-vc145.1.91.0\build\libbitcoin-boost_url-vc145.targets" Condition="Exists('$(NuGetPackageRoot)libbitcoin-boost_url-vc145.1.91.0\build\libbitcoin-boost_url-vc145.targets')" />
<Import Project="$(NuGetPackageRoot)secp256k1_vc145.0.6.1.2\build\native\secp256k1_vc145.targets" Condition="Exists('$(NuGetPackageRoot)secp256k1_vc145.0.6.1.2\build\native\secp256k1_vc145.targets')" />
<Import Project="$(NuGetPackageRoot)UltrafastSecp256k1-vc145.4.4.0.4\build\native\UltrafastSecp256k1-vc145.targets" Condition="Exists('$(NuGetPackageRoot)UltrafastSecp256k1-vc145.4.4.0.4\build\native\UltrafastSecp256k1-vc145.targets')" />
<Import Project="$(NuGetPackageRoot)UltrafastSecp256k1-vc145.4.5.0.0\build\native\UltrafastSecp256k1-vc145.targets" Condition="Exists('$(NuGetPackageRoot)UltrafastSecp256k1-vc145.4.5.0.0\build\native\UltrafastSecp256k1-vc145.targets')" />
</ImportGroup>
<Target Name="EnsureNuGetPackageBuildImports" BeforeTargets="PrepareForBuild">
<PropertyGroup>
Expand All @@ -154,7 +154,7 @@
<Error Condition="!Exists('$(NuGetPackageRoot)libbitcoin-boost_thread-vc145.1.91.0\build\libbitcoin-boost_thread-vc145.targets')" Text="$([System.String]::Format('$(ErrorText)', '$(NuGetPackageRoot)libbitcoin-boost_thread-vc145.1.91.0\build\libbitcoin-boost_thread-vc145.targets'))" />
<Error Condition="!Exists('$(NuGetPackageRoot)libbitcoin-boost_url-vc145.1.91.0\build\libbitcoin-boost_url-vc145.targets')" Text="$([System.String]::Format('$(ErrorText)', '$(NuGetPackageRoot)libbitcoin-boost_url-vc145.1.91.0\build\libbitcoin-boost_url-vc145.targets'))" />
<Error Condition="!Exists('$(NuGetPackageRoot)secp256k1_vc145.0.6.1.2\build\native\secp256k1_vc145.targets')" Text="$([System.String]::Format('$(ErrorText)', '$(NuGetPackageRoot)secp256k1_vc145.0.6.1.2\build\native\secp256k1_vc145.targets'))" />
<Error Condition="!Exists('$(NuGetPackageRoot)UltrafastSecp256k1-vc145.4.4.0.4\build\native\UltrafastSecp256k1-vc145.targets')" Text="$([System.String]::Format('$(ErrorText)', '$(NuGetPackageRoot)UltrafastSecp256k1-vc145.4.4.0.4\build\native\UltrafastSecp256k1-vc145.targets'))" />
<Error Condition="!Exists('$(NuGetPackageRoot)UltrafastSecp256k1-vc145.4.5.0.0\build\native\UltrafastSecp256k1-vc145.targets')" Text="$([System.String]::Format('$(ErrorText)', '$(NuGetPackageRoot)UltrafastSecp256k1-vc145.4.5.0.0\build\native\UltrafastSecp256k1-vc145.targets'))" />
</Target>
<ItemGroup>
<ProjectReference Include="..\libbitcoin-database\libbitcoin-database.vcxproj">
Expand Down
2 changes: 1 addition & 1 deletion builds/msvc/vs2026/libbitcoin-database-tools/packages.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
|
-->
<packages>
<package id="UltrafastSecp256k1-vc145" version="4.4.0.4" targetFramework="Native" />
<package id="UltrafastSecp256k1-vc145" version="4.5.0.0" targetFramework="Native" />
<package id="libbitcoin-boost" version="1.91.0" targetFramework="Native" />
<package id="libbitcoin-boost_container-vc145" version="1.91.0" targetFramework="Native" />
<package id="libbitcoin-boost_iostreams-vc145" version="1.91.0" targetFramework="Native" />
Expand Down
3 changes: 0 additions & 3 deletions builds/msvc/vs2026/libbitcoin-database/libbitcoin-database.props
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,9 +44,6 @@
<Linkage-ultrafast Condition="'$(Option-ultrafast)' == 'true'">static</Linkage-ultrafast>
<Linkage-libbitcoin-system>static</Linkage-libbitcoin-system>
</PropertyGroup>
<PropertyGroup Condition="'$(Option-ultrafast)' == 'true' And '$(Option-cuda)' == 'true'">
<Linkage-ultrafast>cuda</Linkage-ultrafast>
</PropertyGroup>

<!-- Messages -->

Expand Down
4 changes: 2 additions & 2 deletions builds/msvc/vs2026/libbitcoin-database/libbitcoin-database.vcxproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -309,7 +309,7 @@
<Import Project="$(NuGetPackageRoot)libbitcoin-boost_thread-vc145.1.91.0\build\libbitcoin-boost_thread-vc145.targets" Condition="Exists('$(NuGetPackageRoot)libbitcoin-boost_thread-vc145.1.91.0\build\libbitcoin-boost_thread-vc145.targets')" />
<Import Project="$(NuGetPackageRoot)libbitcoin-boost_url-vc145.1.91.0\build\libbitcoin-boost_url-vc145.targets" Condition="Exists('$(NuGetPackageRoot)libbitcoin-boost_url-vc145.1.91.0\build\libbitcoin-boost_url-vc145.targets')" />
<Import Project="$(NuGetPackageRoot)secp256k1_vc145.0.6.1.2\build\native\secp256k1_vc145.targets" Condition="Exists('$(NuGetPackageRoot)secp256k1_vc145.0.6.1.2\build\native\secp256k1_vc145.targets')" />
<Import Project="$(NuGetPackageRoot)UltrafastSecp256k1-vc145.4.4.0.4\build\native\UltrafastSecp256k1-vc145.targets" Condition="Exists('$(NuGetPackageRoot)UltrafastSecp256k1-vc145.4.4.0.4\build\native\UltrafastSecp256k1-vc145.targets')" />
<Import Project="$(NuGetPackageRoot)UltrafastSecp256k1-vc145.4.5.0.0\build\native\UltrafastSecp256k1-vc145.targets" Condition="Exists('$(NuGetPackageRoot)UltrafastSecp256k1-vc145.4.5.0.0\build\native\UltrafastSecp256k1-vc145.targets')" />
</ImportGroup>
<Target Name="EnsureNuGetPackageBuildImports" BeforeTargets="PrepareForBuild">
<PropertyGroup>
Expand All @@ -324,7 +324,7 @@
<Error Condition="!Exists('$(NuGetPackageRoot)libbitcoin-boost_thread-vc145.1.91.0\build\libbitcoin-boost_thread-vc145.targets')" Text="$([System.String]::Format('$(ErrorText)', '$(NuGetPackageRoot)libbitcoin-boost_thread-vc145.1.91.0\build\libbitcoin-boost_thread-vc145.targets'))" />
<Error Condition="!Exists('$(NuGetPackageRoot)libbitcoin-boost_url-vc145.1.91.0\build\libbitcoin-boost_url-vc145.targets')" Text="$([System.String]::Format('$(ErrorText)', '$(NuGetPackageRoot)libbitcoin-boost_url-vc145.1.91.0\build\libbitcoin-boost_url-vc145.targets'))" />
<Error Condition="!Exists('$(NuGetPackageRoot)secp256k1_vc145.0.6.1.2\build\native\secp256k1_vc145.targets')" Text="$([System.String]::Format('$(ErrorText)', '$(NuGetPackageRoot)secp256k1_vc145.0.6.1.2\build\native\secp256k1_vc145.targets'))" />
<Error Condition="!Exists('$(NuGetPackageRoot)UltrafastSecp256k1-vc145.4.4.0.4\build\native\UltrafastSecp256k1-vc145.targets')" Text="$([System.String]::Format('$(ErrorText)', '$(NuGetPackageRoot)UltrafastSecp256k1-vc145.4.4.0.4\build\native\UltrafastSecp256k1-vc145.targets'))" />
<Error Condition="!Exists('$(NuGetPackageRoot)UltrafastSecp256k1-vc145.4.5.0.0\build\native\UltrafastSecp256k1-vc145.targets')" Text="$([System.String]::Format('$(ErrorText)', '$(NuGetPackageRoot)UltrafastSecp256k1-vc145.4.5.0.0\build\native\UltrafastSecp256k1-vc145.targets'))" />
</Target>
<ItemGroup>
<Natvis Include="..\..\debug.natvis" />
Expand Down
2 changes: 1 addition & 1 deletion builds/msvc/vs2026/libbitcoin-database/packages.config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
|
-->
<packages>
<package id="UltrafastSecp256k1-vc145" version="4.4.0.4" targetFramework="Native" />
<package id="UltrafastSecp256k1-vc145" version="4.5.0.0" targetFramework="Native" />
<package id="libbitcoin-boost" version="1.91.0" targetFramework="Native" />
<package id="libbitcoin-boost_container-vc145" version="1.91.0" targetFramework="Native" />
<package id="libbitcoin-boost_iostreams-vc145" version="1.91.0" targetFramework="Native" />
Expand Down
16 changes: 7 additions & 9 deletions include/bitcoin/database/impl/memory/mmap.ipp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,9 +59,9 @@ CLASS::mmap(const paths& filenames, size_t minimum, size_t expansion,
TEMPLATE
CLASS::~mmap() NOEXCEPT
{
BC_ASSERT(!loaded_);
BC_ASSERT(is_zero(logical_));
BC_ASSERT(is_zero(capacity_));
BC_ASSERT(!loaded_.load());
BC_ASSERT(is_zero(logical_.load()));
BC_ASSERT(is_zero(capacity_.load()));
BC_ASSERT(std::ranges::all_of(memory_map_,
[](auto map) NOEXCEPT { return is_null(map); }));
BC_ASSERT(std::ranges::all_of(opened_,
Expand All @@ -78,8 +78,7 @@ bool CLASS::is_open() const NOEXCEPT
TEMPLATE
bool CLASS::is_loaded() const NOEXCEPT
{
std::shared_lock field_lock(field_mutex_);
return loaded_;
return loaded_.load();
}

// protected
Expand All @@ -94,14 +93,13 @@ size_t CLASS::to_capacity(size_t required) const NOEXCEPT
return std::max(minimum_, ceilinged_add(required, growth));
}

// Read-write protected by atomic, write-write protected by remap_mutex.
// Write-write protected by remap_mutex.
TEMPLATE
void CLASS::set_first_code(const error::error_t& ec) NOEXCEPT
{
if (!fault_)
if (!fault_.load())
{
// fault is not exposed so requires no atomic (fast read).
fault_ = true;
fault_.store(true);

// error is atomic for public read exposure.
error_.store(ec);
Expand Down
24 changes: 14 additions & 10 deletions include/bitcoin/database/impl/memory/mmap_dispatch.ipp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,11 +41,12 @@ memory CLASS::get_filled(size_t offset, size_t size,
{
std::unique_lock field_lock(field_mutex_);

if (fault_ || !loaded_ || system::is_add_overflow(offset, size))
if (fault_.load() || !loaded_.load() ||
system::is_add_overflow(offset, size))
return {};

const auto end = std::max(logical_, offset + size);
if (end > capacity_)
const auto end = std::max(logical_.load(), offset + size);
if (end > capacity_.load())
{
const auto extended = to_capacity(end);

Expand All @@ -58,13 +59,16 @@ memory CLASS::get_filled(size_t offset, size_t size,

// Fill new capacity as offset may not be at end due to expansion.
auto data = memory_map_.front();
const auto logical = to_width<zero>(logical_);
const auto capacity = to_width<zero>(capacity_);
const auto logical = to_width<zero>(logical_.load());
const auto capacity = to_width<zero>(capacity_.load());
const auto start = std::next(data, logical);
std::fill_n(start, capacity - logical, backfill);
}

logical_ = end;
// Raise to at least end (concurrent claims may already exceed it).
for (auto current = logical_.load(); current < end;)
if (logical_.compare_exchange_weak(current, end))
break;
}

return get(offset);
Expand All @@ -77,7 +81,7 @@ memory CLASS::get_capacity(size_t offset) const NOEXCEPT

memory out{ remap_mutex_ };

if (!loaded_)
if (!loaded_.load())
return {};

auto data = memory_map_.front();
Expand Down Expand Up @@ -114,14 +118,14 @@ memory CLASS::get_at(size_t column, size_t offset) const NOEXCEPT
if (column >= columns)
return {};

// Obtaining size before access prevents mutual mutex wait (deadlock).
// size() is a lock-free atomic read.
const auto allocated = size() * widths.at(column);

// Takes a shared lock on remap_mutex_ until destruct, blocking remap.
memory out{ remap_mutex_ };

// loaded_ update is precluded by above lock, making this read atomic.
if (!loaded_)
// loaded_ update is precluded by above lock.
if (!loaded_.load())
return {};

// With offset > size the assignment is negative (stream is exhausted).
Expand Down
Loading
Loading