RAG search tool for agents

[jtramm]

Description

This PR introduces a new Retrieval-Augmented Generation (RAG) search index tool (similar to what cursor provides) that agents can use when working on OpenMC. The tool runs locally and is implemented in python. The tool is configured as an MCP server that is registered with agents as a formal tool via the .mcp.json file in the top OpenMC directory. The RAG search tool works out of the box with Claude code, and should be automatically discoverable by other agents though I haven't tried codex yet. It's possible that other agents may need to have an added .md file or .json to register the tool.

Motivation & Impact

The reason I've made this tool is that I've found Claude code to be very powerful but very narrow. For instance, when I ask it to review code, it typically reviews the diff (and potentially any files the diff touches), but is blind to how these changes may impact other regions of the code. It generally lacks any global vision of what is going on and is very surgical with what it chooses to look at. This style of agentic strategy I think works great for really well constrained problems with clear victory conditions and thorough test cases that it needs to pass. However, it fails at more open ended tasks like code review, where you really have to have some level of global awareness of the code.

To remedy this, the agent can use the RAG tool to search for more general terms or concepts that may appear across multiple files without sharing identical function names. Basically, I think this tool helps to widen the context that is available to the agent without having it try to tokenize the entire repository.

Implementation

I've done a lot of experimentation balancing result quality vs. indexing time, and have put some thought into the chunking process. I've tested out some of the larger models, but the next model up was taking more like 15 minutes to index which gets to be more annoying, and I don't think the increased quality is really needed just to navigate the repo. The small model (from Hugging Face, part of the sentence-transformers python package that does this) seems to be sufficient.

This PR was mostly the work of Claude code, though I did read through and add a lot of comments/cleanup to the python implementation so it would be human readable.

Limitations and Future Work

• The tool uses an MCP interface so should be portable to other agents. However, I haven't tested them with codex yet, so codex users would be welcome to add on any needed .md files/pointers etc to get the tools registered there.

• The RAG just covers the C++ codebase, python codebase, tests (python files), and docs. In the future, I'd like to either extend the current RAG tool (or add a second one) that can index a large set of example input decks for larger problems, so as to provide more context to the agent for how larger OpenMC inputs are composed and organized. The agents are already decent at this, but these sorts of RAG tools can still be a nice value add.

• I have also been experimenting with a Language Server Protocol (LSP) C++ search tool (similar to what IDEs with C++ comprehension (e.g., VSCode intellisense) provide). It works well, but ultimately it only helps speed up searching operations (which are already fast) and saving a little context, so is probably not worth the added complexity and additional python imports etc.

Checklist

• I have performed a self-review of my own code
• I have run clang-format (version 18) on any C++ source files (if applicable)
• I have followed the style guidelines for Python source files (if applicable)
• I have made corresponding changes to the documentation (if applicable)
• I have added tests that prove my fix is effective or that my feature works (if applicable)

[Copilot]

Pull request overview

This PR adds a local, MCP-exposed RAG semantic search toolchain intended to help AI coding agents navigate the OpenMC repository by meaning-based retrieval, and documents how contributors/agents should use it.

Changes:

• Registers an MCP server (openmc-code-tools) via .mcp.json and provides bootstrap/start scripts to run it with an isolated Python venv.
• Implements local indexing + semantic search over OpenMC code/docs using sentence-transformers embeddings stored in LanceDB.
• Adds developer documentation and agent guidance (Devguide + AGENTS.md + CLAUDE.md) describing the tools and workflow.

Reviewed changes

Copilot reviewed 12 out of 13 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
docs/source/devguide/index.rst	Adds the new devguide page to the toctree.
docs/source/devguide/agentic-tools.rst	Documents MCP-based agent tools and the RAG semantic search.
CLAUDE.md	Claude Code-facing guidance for using the RAG tools (first-call behavior).
AGENTS.md	General agent/developer documentation for the new MCP tools.
.mcp.json	Registers the MCP server command/args at repo root.
.gitignore	Ignores generated/cache artifacts under .claude/cache/.
.claude/tools/start_server.sh	Bootstraps a venv and launches the MCP server.
.claude/tools/requirements.txt	Declares Python dependencies for the MCP server + RAG tooling.
.claude/tools/rag/openmc_search.py	Implements query-time semantic search and “related file” search.
.claude/tools/rag/indexer.py	Implements index build/rebuild pipeline over code/docs.
.claude/tools/rag/embeddings.py	Wraps sentence-transformers and configures HF/transformers env behavior.
.claude/tools/rag/chunker.py	Splits files into overlapping chunks with line number metadata.
.claude/tools/openmc_mcp_server.py	Exposes openmc_rag_search / openmc_rag_rebuild via MCP and manages session state.

---

You can also share your feedback on Copilot code review. Take the survey.