Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 18 additions & 6 deletions docs/website-src/docs/buffering.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ This page explains how to create arrays from existing buffers without copying, h

**Predictable Layout.** Managed arrays can be moved by the garbage collector at any time. Unmanaged memory stays put, which is essential when passing pointers to native libraries or GPU drivers.

**No GC Pauses.** Large managed arrays cause GC pressure. A 1GB NDArray in unmanaged memory doesn't affect GC at all.
**Reduced GC Overhead.** Large managed arrays cause GC pressure and can trigger expensive collections. Unmanaged memory avoids this—though NumSharp still informs the GC about allocation sizes so it can schedule collections appropriately.

**Interop Efficiency.** When calling into native code (BLAS, CUDA, image processing libraries), unmanaged memory can be passed directly without marshaling.

Expand Down Expand Up @@ -50,6 +50,14 @@ User Code

**Internal Infrastructure** handles the low-level details: pinning managed arrays so the GC won't move them, tracking ownership so memory gets freed at the right time, and managing the raw pointers. You don't need to interact with these directly—the external APIs handle it for you.

### GC Pressure Tracking

Although NumSharp uses unmanaged memory, the .NET garbage collector still needs to know about it. Otherwise, the GC sees only the small managed wrappers (~100 bytes each) and doesn't realize there's megabytes of unmanaged data attached. This can cause memory to grow unbounded before the GC kicks in.

NumSharp solves this by calling `GC.AddMemoryPressure()` when allocating native memory and `GC.RemoveMemoryPressure()` when freeing it. This applies to arrays created with `np.array()`, `np.zeros()`, `np.empty()`, and similar functions.

For external memory (via `np.frombuffer()` with a dispose callback), the caller is responsible for pressure tracking since NumSharp doesn't know how the memory was allocated.

---

## Creating Arrays from Buffers
Expand Down Expand Up @@ -163,17 +171,21 @@ This is appropriate when you're borrowing memory temporarily. You must ensure th

```csharp
// We allocate native memory
IntPtr ptr = Marshal.AllocHGlobal(1024 * sizeof(float));
int bytes = 1024 * sizeof(float);
IntPtr ptr = Marshal.AllocHGlobal(bytes);
GC.AddMemoryPressure(bytes); // Tell GC about this allocation

// Transfer ownership to NumSharp
var arr = np.frombuffer(ptr, 1024 * sizeof(float), typeof(float),
dispose: () => Marshal.FreeHGlobal(ptr));
var arr = np.frombuffer(ptr, bytes, typeof(float),
dispose: () => {
Marshal.FreeHGlobal(ptr);
GC.RemoveMemoryPressure(bytes);
});

// When arr is garbage collected, the dispose action runs
// No manual free needed
```

The `dispose` parameter takes an action that NumSharp calls when the array is no longer needed. This is cleaner for memory you've allocated, but be careful: if you free the memory yourself AND provide a dispose action, you'll double-free.
The `dispose` parameter takes an action that NumSharp calls when the array is no longer needed. For large allocations, pair `GC.AddMemoryPressure()` with `GC.RemoveMemoryPressure()` so the GC knows about your memory. Be careful: if you free the memory yourself AND provide a dispose action, you'll double-free.

### From .NET Buffer Types

Expand Down
45 changes: 36 additions & 9 deletions src/NumSharp.Core/Backends/Unmanaged/UnmanagedMemoryBlock`1.cs
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ public UnmanagedMemoryBlock(long count)
{
var bytes = BytesCount = count * InfoOf<T>.Size;
var ptr = (IntPtr)NativeMemory.Alloc((nuint)bytes);
_disposer = new Disposer(ptr);
_disposer = new Disposer(ptr, bytes);
Address = (T*)ptr;
Count = count;
}
Expand All @@ -52,16 +52,19 @@ public UnmanagedMemoryBlock(T* ptr, long count)
/// <summary>
/// Construct with externally allocated memory and a custom <paramref name="dispose"/> function.
/// </summary>
/// <param name="start"></param>
/// <param name="start">Pointer to externally allocated unmanaged memory.</param>
/// <param name="count">The length in objects of <typeparamref name="T"/> and not in bytes.</param>
/// <param name="dispose"></param>
/// <remarks>Does claim ownership.</remarks>
/// <param name="dispose">Cleanup action called when memory is released.</param>
/// <remarks>
/// Claims ownership of the memory. Caller is responsible for GC.AddMemoryPressure
/// if the memory is unmanaged and large enough to warrant it.
/// </remarks>
[MethodImpl(OptimizeAndInline)]
public UnmanagedMemoryBlock(T* start, long count, Action dispose)
{
Count = count;
BytesCount = InfoOf<T>.Size * count;
_disposer = new Disposer(dispose);
_disposer = new Disposer(dispose); // Caller tracks pressure for their allocation
Address = start;
}

Expand Down Expand Up @@ -984,16 +987,24 @@ private enum AllocationType
private readonly IntPtr Address;
private readonly GCHandle _gcHandle;
private readonly Action _dispose;
private readonly long _bytesCount;


/// <summary>
/// Construct a AllocationType.Native (NativeMemory.Alloc)
/// </summary>
/// <param name="address"></param>
public Disposer(IntPtr address)
/// <param name="address">The address of the allocated memory.</param>
/// <param name="bytesCount">The size in bytes of the allocation (for GC memory pressure tracking).</param>
public Disposer(IntPtr address, long bytesCount)
{
Address = address;
_bytesCount = bytesCount;
_type = AllocationType.Native;

// Inform the GC about unmanaged memory allocation so it can
// schedule collections appropriately (fixes GitHub issue #501)
if (bytesCount > 0)
GC.AddMemoryPressure(bytesCount);
}

/// <summary>
Expand All @@ -1009,11 +1020,21 @@ public Disposer(GCHandle gcHandle)
/// <summary>
/// Construct a AllocationType.External
/// </summary>
/// <param name="dispose"></param>
public Disposer(Action dispose)
/// <param name="dispose">The cleanup action to invoke on disposal.</param>
/// <param name="bytesCount">
/// Optional: Size in bytes for GC memory pressure tracking.
/// Pass 0 for managed memory (GCHandle) or when caller manages pressure.
/// Pass actual bytes for unmanaged memory to inform GC.
/// </param>
public Disposer(Action dispose, long bytesCount = 0)
{
_dispose = dispose;
_bytesCount = bytesCount;
_type = AllocationType.External;

// Track memory pressure for external unmanaged memory
if (bytesCount > 0)
GC.AddMemoryPressure(bytesCount);
}

/// <summary>
Expand All @@ -1036,11 +1057,17 @@ private void ReleaseUnmanagedResources()
{
case AllocationType.Native:
NativeMemory.Free((void*)Address);
// Remove GC memory pressure that was added during allocation
if (_bytesCount > 0)
GC.RemoveMemoryPressure(_bytesCount);
return;
case AllocationType.Wrap:
return;
case AllocationType.External:
_dispose();
// Remove GC memory pressure if it was added for external unmanaged memory
if (_bytesCount > 0)
GC.RemoveMemoryPressure(_bytesCount);
return;
case AllocationType.GCHandle:
_gcHandle.Free();
Expand Down
Loading