Skip to content

perf: filter source networks using overrides in network mapping#40

Open
albertoflorez wants to merge 2 commits into
redhat-cop:mainfrom
albertoflorez:perf/optimize-network-map-processing
Open

perf: filter source networks using overrides in network mapping#40
albertoflorez wants to merge 2 commits into
redhat-cop:mainfrom
albertoflorez:perf/optimize-network-map-processing

Conversation

@albertoflorez
Copy link
Copy Markdown

Description

This PR optimizes the network map creation process by filtering the source networks (vSphere/Ovirt) based on the provided overrides before entering the task loop.

Motivation

Beyond performance, this change addresses a critical scalability issue. In large-scale vSphere environments (e.g., thousands of PortGroups), attempting to map every available network results in an oversized NetworkMap Custom Resource.

This often leads to:

  1. API Server Rejection: The generated resource exceeds the standard 1MB limit for objects in etcd, causing the k8s module to fail with a "Request entity too large" error.

  2. Controller Timeout: The MTV controller struggles to reconcile objects with massive spec.map arrays, leading to timeouts and instability.

By filtering the networks at the source, we ensure that the resulting NetworkMap object only contains relevant entries, keeping the resource footprint small and within Kubernetes API limits.

Changes

  • Added a mtv_allowed_ids fact to capture required network IDs from overrides.
  • Updated the loop logic in Process VMware Networks and Process Ovirt Networks to only iterate over the filtered list when overrides are defined.
  • Maintained backward compatibility: if no overrides are provided, it still processes all networks.

Impact

Significant reduction in playbook execution time for large-scale environments.

@albertoflorez albertoflorez requested a review from sabre1041 as a code owner May 26, 2026 10:09
@albertoflorez albertoflorez changed the title perf: filter source networks using overrides before loop in network map creation perf: filter source networks using overrides in network mapping May 26, 2026
@sabre1041 sabre1041 changed the title perf: filter source networks using overrides in network mapping perf: filter source networks using overrides in network mapping [skip ci] May 27, 2026
@burigolucas burigolucas changed the title perf: filter source networks using overrides in network mapping [skip ci] perf: filter source networks using overrides in network mapping May 27, 2026
@stevefulme1
Copy link
Copy Markdown
Contributor

Great optimization — the etcd 1MB limit issue in large vSphere environments is a real problem and this is the right approach to fix it.

One issue I spotted: exclude-only overrides will break with this change.

If a user provides overrides with only exclude: true entries (no include keys), the "Extract allowed network IDs" task runs but produces an empty list (no include attributes to select). The loop condition then sees mtv_management_network_map_overrides | length > 0 is true, so it filters mtv_networks against the empty mtv_allowed_ids — resulting in zero networks processed. The downstream exclude logic in _mtv_network_map_process_network.yml never gets a chance to run.

The fix is to key the loop condition on whether include IDs were actually extracted, not on whether overrides exist:

loop: >-
  {{
    (mtv_networks | selectattr('id', 'in', mtv_allowed_ids) | list)
    if (mtv_allowed_ids | default([]) | length > 0)
    else mtv_networks
  }}

This way the filter only kicks in when there are actual include IDs, and exclude-only overrides still iterate the full list (letting the existing downstream task handle the exclusion as it does today).

Minor nit: might also be worth initializing mtv_allowed_ids: [] in the "Initialize data structures" task at the top to prevent fact leakage if the role is called twice in one play (once with overrides, once without).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants