fix: rotate the LinkContentFetcher user agent per fetch, not per component

[pcbeingused333]

Related Issues

• fixes LinkContentFetcher's User-Agent rotation on retry is scrambled across concurrent multi-URL fetches #12287

Proposed Changes:

The rotation cursor lived on the component (self.current_user_agent_idx), but run() fetches the URLs through a ThreadPoolExecutor and run_async() gathers them. Every in-flight request read and wrote that one counter, so with several URLs in the same call:

• a retry triggered by one URL rotated the user agent for all the others, and
• each fetch that completed reset the counter to 0 (in the finally of _fetch/_fetch_async) underneath the requests still running.

The retries then mostly went out with the un-rotated user agent, which is the behaviour reported in #12287. A single-URL call rotates correctly, which is why this went unnoticed.

Each fetch now keeps its own cursor: a local in _get_response that the tenacity after callback advances, and a local in _get_response_async. _get_headers takes the user agent for the attempt instead of reading component state.

current_user_agent_idx and _switch_user_agent were that shared state and are removed. Neither is part of the documented API and neither is serialized by to_dict, so nothing that survives a pipeline round-trip changes.

How did you test it?

• New unit test test_user_agent_rotation_is_independent_per_url: fetches 8 URLs in one run() call with 4 user agents, each URL failing once. Every URL must send user_agents[1] on its successful retry. On main they mostly send user_agents[0]; with this change every URL sends user_agents[1].
• hatch run test:unit — 6116 passed, 10 skipped.
• hatch run test:types and hatch run fmt clean.

Notes for the reviewer

The behaviour pinned by the existing tests is unchanged: the first attempt still uses user_agents[0] and the first retry still uses user_agents[1].

The one judgement call is deleting current_user_agent_idx rather than leaving it as a vestigial attribute. It was only ever the shared cursor, and keeping it would leave an attribute that no longer describes anything. Happy to keep it (unused) if you would rather not touch the surface at all.

Checklist

• I have read the contributors guidelines and the code of conduct.
• I have added unit tests and updated the docstrings.
• I've used one of the conventional commit types for my PR title.
• I have documented my code.
• I have added a release note file.
• I have run pre-commit hooks and fixed any issue.

I used an AI assistant while writing this change. I have reviewed it, reproduced the bug, and run the tests.

[vercel]

Someone is attempting to deploy a commit to the deepset Team on Vercel.

A member of the Team first needs to authorize it.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

---

Alex Castillo seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[github-actions]

Hi @pcbeingused333, thanks for your interest in contributing to Haystack! 🙏

⚠️ Issue #12287 is already being addressed by open pull request(s) #12289. Before opening a PR for an issue, please check whether a PR is already linked to it, and consider contributing to the existing PR instead. We may close duplicate PRs to keep the review queue manageable.

This is an automated message to help us keep the review queue healthy.

[pcbeingused333]

Thanks for the flag — the guard is right and I missed it. I checked #12289 by its title, which describes adding a component, and did not read the body, where it is the same fix for the same race in the same file. Closing this as the duplicate; #12289 got there first.

Leaving one note in case it is useful to whoever reviews #12289: the async path has the same shared-cursor problem (_get_response_async mutates current_user_agent_idx between attempts), and _get_headers reads that state, so a fix that only covers the sync path leaves run_async scrambled. There is also a regression test in this branch that fetches 8 URLs in one call and asserts each one sends its own rotated agent, if it is worth pulling across.