Optimize local placement

CHANGELOG.md

@@ -1,5 +1,7 @@
 ## Unreleased
 
+- Balance new child starts using the existing eligible-node ranking, considering live local capacity alongside remote heartbeat capacity instead of filling local first. Eligible local remains a candidate even when it ranks beyond the remote candidate-pool limit. Explicit local starts, restart/rehome, and sticky-placement behavior are unchanged.
+- Skip dumping, serialization, and hashing during periodic and automatic sync when callback state is exactly unchanged and no earlier change is pending. Runtime state changes still use `dump_state/1` to determine whether a storage write is needed; explicit sync and lifecycle writes continue to evaluate the dump.
 - Resolve duplicate DurableServer ownership by a persisted, monotonically increasing lock epoch. Group now keeps the latest successful object-store claimant and retires only stale owners instead of killing every claimant and causing repeated anti-entropy restart loops.
 - Enforce cumulative sticky-placement gates during request-driven remote placement so an existing server cannot move to a fallback node before that level unlocks.
 - Allow an expired restart attempt to be reclaimed immediately by another node at an allowed sticky-placement level, while preserving strict placement when no level matches.

README.md

@@ -8,7 +8,7 @@ It implements fault-tolerant, stateful processes that can survive node failures,
 
 - **Durable state**: Automatically persists state to storage with configurable sync intervals
 - **Cluster coordination**: Uses distributed registry for process discovery and health monitoring
-- **Capacity-aware placement**: Monitors CPU, memory, and disk usage to route new processes to nodes with available capacity
+- **Capacity-aware placement**: Ranks local and remote nodes together by utilization, spreading new processes before local fills while respecting child-count and resource limits
 - **Sticky placement**: Environment variable-based placement preferences (e.g., same machine, same region, etc.) with time-gated fallback to ensure servers restart on preferred nodes when possible
 - **Automatic recovery**: Failed processes are detected and restarted across the cluster
 - **Graceful shutdown**: Ensures state is synchronized before termination
@@ -186,7 +186,7 @@ See `DurableServer.Backends.MirrorStore` for usage and an example rollout.
 
 DurableServer supports these options in the `init/1` return tuple:
 
-- `:auto_sync` - Enable automatic periodic syncing (default: false)
+- `:auto_sync` - Enable automatic syncing on callback return (default: false)
 - `:sync_every_ms` - Sync interval in milliseconds (default: 30_000)
 - `:meta` - Optional metadata included in the global registry
 
@@ -205,6 +205,16 @@ State is synchronized to storage in these scenarios:
    eligible for a later sync. Storage conflicts remain fatal.
 3. **Graceful shutdown**: State is always synced before termination
 
+Periodic and automatic sync skip dumping, serialization, and hashing when callback state is
+exactly unchanged and no earlier change is pending. Any runtime state change schedules a
+check of `dump_state/1`; if its output is unchanged, the check skips the storage write and
+clears the pending flag. Failed writes remain pending for a later sync.
+
+`dump_state/1` should deterministically derive its output from the supplied state. Changes
+to external data, such as ETS or the process dictionary, cannot be detected by state tracking;
+use an explicit sync to re-evaluate the dump when needed. Explicit sync and lifecycle writes
+always evaluate `dump_state/1`.
+
 Manual synchronization performs storage work inline. A `GenServer.call/3` timeout should be
 long enough to cover the configured backend's retry window. A caller timeout does not cancel
 storage work already running in the DurableServer process.

lib/durable_server.ex

@@ -139,7 +139,7 @@ defmodule DurableServer do
 
   DurableServer supports these options in the `init/1` or `init/2` return tuple:
 
-  - `:auto_sync` - Enable automatic periodic syncing (default: false)
+  - `:auto_sync` - Enable automatic syncing on callback return (default: false)
   - `:sync_every_ms` - Sync interval in milliseconds (default: 30_000)
   - `:meta` - Optional metadata to include for the globally registered server which is
     returned alongside the pid with `DurableServer.Supervisor.lookup/2`.
@@ -203,6 +203,11 @@ defmodule DurableServer do
   3. **Graceful shutdown**: Automatically synced during normal termination, ie: cold deploys
   4. **Before stopping**: When returning `{:stop, reason, state}` from callbacks
 
+  Periodic and automatic sync skip dumping and hashing while callback state is exactly
+  unchanged and no earlier change is pending. Runtime changes are checked with `dump_state/1`
+  before deciding whether a storage write is needed. Explicit sync and lifecycle writes
+  always evaluate the dump.
+
   ## Stopping Behavior
 
   DurableServer supports different stop reasons with specific behaviors regarding exit signal propagation:
@@ -269,20 +274,15 @@ defmodule DurableServer do
 
   ## Capacity-Aware Placement
 
-  DurableServers support automatic capacity-aware placement with remote fallback.
-
-  ### Local Placement (Default)
-
-  When starting a child, the local node is tried first. If capacity limits are exceeded,
-  remote placement is attempted automatically.
-
-  ### Remote Placement
-
-  If local capacity is exhausted, DurableServer automatically tries remote nodes:
+  New children use the existing least-busy eligible-node ranking, with local considered
+  alongside remote nodes instead of filling local capacity first. Ranking uses the
+  highest utilization across configured child-count, CPU, and memory limits. Full,
+  draining, or resource-limited nodes are excluded; connected nodes are preferred.
 
-  1. **Same-region nodes first** - Prioritizes nodes in the same region for lower latency
-  2. **Least busy nodes** - Selects nodes with the lowest utilization across all limits
-  3. **Configurable retries** - Default 3 remote nodes tried, configurable via `max_placement_retries`
+  Local counts include in-flight start reservations. Remote load remains heartbeat-based,
+  so placement is approximate rather than strictly even during bursts. Admission checks
+  on the selected node still enforce its limits. Restart, rehome, and sticky-placement
+  behavior is unchanged.
 
   ### Capacity Limits
 
@@ -306,14 +306,14 @@ defmodule DurableServer do
 
   Control remote placement behavior per start_child call:
 
-      # Default: Try local, then up to 3 remote nodes
+      # Default: Rank local alongside up to 3 remote nodes
       DurableServer.Supervisor.start_child(sup, {MyServer, key: "user_1", initial_state: %{}})
 
       # Local only, no remote fallback
       DurableServer.Supervisor.start_child(sup, {MyServer, key: "user_1", initial_state: %{}},
         max_placement_retries: 0)
 
-      # Try local, then up to 5 remote nodes
+      # Rank local alongside up to 5 remote nodes
       DurableServer.Supervisor.start_child(sup, {MyServer, key: "user_1", initial_state: %{}},
         max_placement_retries: 5)
 
@@ -757,6 +757,10 @@ defmodule DurableServer do
   - Transform the state shape for storage
   - Remove ephemeral data
 
+  Derive the output deterministically from the supplied state. Periodic and automatic
+  sync skip this callback when runtime state has not changed since the last successful
+  check. If the dump depends on external data, use explicit sync to re-evaluate it.
+
   The returned value must be a plain map at the top level. Nested values are passed
   through to the configured backend as-is, so they only need to be encodable by the
   backend you are using.
@@ -864,6 +868,7 @@ defmodule DurableServer do
             crash_history: [],
             module: nil,
             last_synced_user_state_hash: nil,
+            needs_sync_check: true,
             final_status_set: nil,
             terminator_handled: false,
             delete_requester: nil,
@@ -2111,7 +2116,12 @@ defmodule DurableServer do
   end
 
   def handle_info({@durable, :sync}, %__MODULE__{} = state) do
-    case sync_to_storage(state) do
+    state = maybe_migrate_on_callback(state)
+
+    sync_result =
+      if state.needs_sync_check, do: sync_to_storage(state), else: {:ok, state}
+
+    case sync_result do
       {:ok, %DurableServer{} = new_state} ->
         {:noreply, schedule_sync(new_state)}
 
@@ -2814,7 +2824,8 @@ defmodule DurableServer do
   def code_change(old_vsn, %__MODULE__{} = state, extra) do
     case state.module.code_change(old_vsn, state.user_state, extra) do
       {:ok, new_user_state} ->
-        {:ok, %{state | user_state: new_user_state}}
+        # The dumped shape can change across upgrades even if user state is unchanged.
+        {:ok, %{state | user_state: new_user_state, needs_sync_check: true}}
 
       {:error, reason} ->
         {:error, reason}
@@ -2966,7 +2977,13 @@ defmodule DurableServer do
 
       {:ok, new_user_state} = state.module.code_change(state.vsn, state.user_state, current_vsn)
 
-      %{state | vsn: current_vsn, old_vsn: state.vsn, user_state: new_user_state}
+      %{
+        state
+        | vsn: current_vsn,
+          old_vsn: state.vsn,
+          user_state: new_user_state,
+          needs_sync_check: true
+      }
     end
   end
 
@@ -3340,11 +3357,16 @@ defmodule DurableServer do
 
   defp update_state(%__MODULE__{} = state, new_user_state) do
     new_user_state = validate_user_state!(new_user_state)
-    %{state | user_state: new_user_state}
+
+    %{
+      state
+      | user_state: new_user_state,
+        needs_sync_check: state.needs_sync_check or new_user_state !== state.user_state
+    }
   end
 
   defp auto_sync_to_storage(%DurableServer{module: module, key: key} = state, old_user_state) do
-    if state.auto_sync do
+    if state.auto_sync && state.needs_sync_check do
       old_user_state = old_user_state || state.user_state
 
       # if auto sync fails we continue, but log
@@ -3415,14 +3437,14 @@ defmodule DurableServer do
 
       case put_object(new_state, storage_key(new_state), data) do
         {:ok, %DurableServer{} = new_state} ->
-          synced_state = %{new_state | status: new_meta.status}
+          synced_state = %{new_state | status: new_meta.status, needs_sync_check: false}
           maybe_apply_handle_sync(synced_state, old_user_state, user_state_changed?)
 
         {:error, reason} ->
           {:error, reason}
       end
     else
-      {:ok, new_state}
+      {:ok, %{new_state | needs_sync_check: false}}
     end
   end
 

lib/durable_server/lifecycle_manager.ex

@@ -3329,12 +3329,14 @@ defmodule DurableServer.LifecycleManager do
   (global capacity, module capacity, CPU, memory). Nodes with lower utilization
   are prioritized for better load distribution.
 
-  Always excludes the local node since we only lookup eligible remote nodes
-  after local placement fails.
+  Excludes the local node unless `:include_local` is set. When included, local
+  capacity and resources are read directly instead of using its last heartbeat.
 
   ## Options
 
-  - `:limit` - Maximum number of nodes to return (default: 3)
+  - `:limit` - Maximum number of ranked candidates to return (default: 3).
+    With `:include_local`, an eligible local node is also retained if it falls beyond this limit.
+  - `:include_local` - Include the local node using live capacity (default: false)
   - `:key` - The server key, used to load augmented sticky placement preferences and timing metadata
 
   ## Examples
@@ -3357,6 +3359,7 @@ defmodule DurableServer.LifecycleManager do
         limit = Keyword.get(opts, :limit, 3)
         key = Keyword.get(opts, :key)
         my_node = Node.self()
+        include_local = Keyword.get(opts, :include_local, false)
 
         # Get both the effective placement and its persisted metadata. Candidate
         # selection needs the metadata heartbeat to enforce cumulative time gates.
@@ -3374,6 +3377,19 @@ defmodule DurableServer.LifecycleManager do
         # Liveness is ALWAYS computed from timestamp, never from mere Group presence.
         merged_nodes = merge_heartbeat_sources(supervisor_name, heartbeat_table, now)
 
+        merged_nodes =
+          if include_local do
+            local_entry =
+              {to_string(my_node), DurableServer.Supervisor.node_ref(supervisor_name), now,
+               DurableServer.Supervisor.current_capacity(supervisor_name),
+               calculate_resource_map(supervisor_name), get_my_env_vars(supervisor_name),
+               %{"draining" => check_shutting_down(supervisor_name) != :ok}}
+
+            [local_entry | Enum.reject(merged_nodes, &(elem(&1, 0) == to_string(my_node)))]
+          else
+            merged_nodes
+          end
+
         merged_nodes
         |> Enum.map(fn {node_str, node_ref, timestamp, capacity, resources, env_vars,
                         heartbeat_meta} ->
@@ -3409,7 +3425,7 @@ defmodule DurableServer.LifecycleManager do
         end)
         |> Enum.reject(&is_nil/1)
         |> Enum.filter(fn {node, health, matching_level, _timestamp} ->
-          node != my_node and
+          (include_local or node != my_node) and
             can_node_accept_module?(health, module, matching_level: matching_level) and
             sticky_candidate_gate_open?(
               sticky_placement,
@@ -3424,8 +3440,15 @@ defmodule DurableServer.LifecycleManager do
           staleness = -timestamp
           {level_priority, busyness, staleness}
         end)
-        |> Enum.take(limit)
         |> Enum.map(fn {node, _health, _matching_level, _timestamp} -> node end)
+        |> Enum.split(limit)
+        |> then(fn {candidates, rest} ->
+          # Keep local available as a fallback even when busier than the whole
+          # remote pool. It has already passed the same eligibility checks.
+          if include_local,
+            do: candidates ++ Enum.filter(rest, &(&1 == my_node)),
+            else: candidates
+        end)
     end
   end