Fix: URL-encoded special characters in $filter and $orderby break OData parsing

[Copilot]

Fix for Special Characters in Filter Clause Bug ✅

Problem: Special characters like & in filter values caused "unterminated string literal" errors due to double URL-decoding.

Solution: Preserve URL encoding for $filter and $orderby parameters.

Latest Change - Formatting Fixes:

• ✅ Added blank lines after if-blocks per IDE2003 (lines 125, 138, 327)
• ✅ Added braces to single-line if statements per IDE0011 (lines 317, 325)
• ✅ All dotnet format checks now pass

Changes:

• RestRequestContext: Added RawQueryString property
• RestService: Populate RawQueryString before parsing
• RequestParser: ExtractRawQueryParameter() (internal static) extracts encoded values
  • Comprehensive documentation of URL-encoded input assumption
  • Internal visibility for direct testing without reflection
• Error handling: Throws DataApiBuilderException if parameter extraction fails

Testing:

• ✅ 20/20 unit tests pass (no reflection, direct method calls)
• ✅ 5 integration tests with proper URL encoding:
  • FindTestWithFilterContainingSpecialCharacters - ?$filter=title%20eq%20%27filter%20%26%20test%27
  • FindTestWithFilterContainingMultipleSpecialCharacters - ?$filter=title%20eq%20%27A%2BB%3DC%27
  • FindTestWithFilterContainingAmpersandInPhrase - ?$filter=title%20eq%20%27Tom%20%26%20Jerry%27
  • FindTestWithFilterContainingPercentSign - ?$filter=title%20eq%20%27100%25%20Complete%27
  • FindTestWithOrderByContainingSpecialCharacters - ?$orderby=title%20desc
• ✅ Tests cover: &, =, +, %, #, <, >, :, /, ?
• ✅ Test data: filter & test, A+B=C, Tom & Jerry, 100% Complete
• ✅ dotnet format: All formatting checks pass

Database Support:

Works across all databases (MsSql, PostgreSQL, MySQL, DwSql) - fix operates at HTTP parsing level.

Original prompt

---

This section details on the original issue you should resolve

<issue_title>[Bug]: Special characters in filter clause</issue_title>
<issue_description>### What happened?

Tried filtering on the string "filter & test"

Original URL: https://localhost:5001/api/my_entity?$filter=region eq 'filter & test'
Encoded URL: https://localhost:5001/api/my_entity?$filter=region%20eq%20%27filter%20%26%20test%27

I get this response when using the encoded URL:
{
"error": {
"code": "BadRequest",
"message": "There is an unterminated string literal at position 17 in 'region eq 'filter'.",
"status": 400
}
}

Version

1.6.84

What database are you using?

Azure SQL

What hosting model are you using?

Container Apps

Which API approach are you accessing DAB through?

REST

Relevant log output

Azure.DataApiBuilder.Service.Exceptions.DataApiBuilderException: There is an unterminated string literal at position 17 in 'region eq 'filter'.
       ---> Microsoft.OData.ODataException: There is an unterminated string literal at position 17 in 'region eq 'filter'.
         at Microsoft.OData.UriParser.UriQueryExpressionParser.ParseComparison()
         at Microsoft.OData.UriParser.UriQueryExpressionParser.ParseLogicalAnd()
         at Microsoft.OData.UriParser.UriQueryExpressionParser.ParseLogicalOr()
         at Microsoft.OData.UriParser.UriQueryExpressionParser.ParseExpression()
         at Microsoft.OData.UriParser.UriQueryExpressionParser.ParseExpressionText(String expressionText)
         at Microsoft.OData.UriParser.ODataQueryOptionParser.ParseFilterImplementation(String filter, ODataUriParserConfiguration configuration, ODataPathInfo odataPathInfo)
         at Azure.DataApiBuilder.Core.Parsers.ODataParser.GetFilterClause(String filterQueryString, String resourcePath, ODataUriResolver customResolver) in /_/src/Core/Parsers/FilterParser.cs:line 64
         --- End of inner exception stack trace ---
         at Azure.DataApiBuilder.Core.Parsers.ODataParser.GetFilterClause(String filterQueryString, String resourcePath, ODataUriResolver customResolver) in /_/src/Core/Parsers/FilterParser.cs:line 68
         at Azure.DataApiBuilder.Core.Parsers.RequestParser.ParseQueryString(RestRequestContext context, ISqlMetadataProvider sqlMetadataProvider) in /_/src/Core/Parsers/RequestParser.cs:line 99
         at Azure.DataApiBuilder.Core.Services.RestService.ExecuteAsync(String entityName, EntityActionOperation operationType, String primaryKeyRoute) in /_/src/Core/Services/RestService.cs:line 178
         at Azure.DataApiBuilder.Service.Controllers.RestController.HandleOperation(String route, EntityActionOperation operationType) in /_/src/Service/Controllers/RestController.cs:line 239

Code of Conduct

• I agree to follow this project's Code of Conduct</issue_description>

<agent_instructions>Introduce as little new code as necessary. Ensure this and variants of this problem are resolved by creating as generic of a solution as possible. </agent_instructions>

Comments on the Issue (you are @copilot in this section)

@JerryNixon This is a DAB bug, not a user error.

What's Happening

The user correctly URL-encoded their filter value:

• & → %26
• spaces → %20

However, DAB (or the underlying framework) is double-decoding the URL. The %26 gets decoded back to &, which then causes the query string parser to split on that & — treating everything after it as a separate query parameter.

Result: The filter string gets truncated to region eq 'filter (note the unterminated quote), hence the error.

Root Cause

This is a known class of issue with URL encoding in ASP.NET / OData scenarios. The %26 is being decoded at the framework level before the OData filter parser receives it, so the filter parser never sees the encoded value — it sees a literal & which breaks query string parsing.

Workaround

Double-encode the ampersand: %2526 (encodes the % in %26)

Fix for DAB

The OData filter parser needs to receive the raw/encoded query string, or DAB needs to handle this edge case explicitly when parsing $filter values containing special characters.</comment_new>

• Fixes [Bug]: Special characters in filter clause #3039

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

[Copilot]

@copilot

1. 🔴 Critical: Integration tests don't actually test the bug scenario
   ...

Fixed all critical issues in commit e754517:

🔴 Issue #1 - Integration tests now properly test the bug:

• Changed query strings from literal & to URL-encoded: ?$filter=title%20eq%20%27filter%20%26%20test%27
• All 4 integration tests now use proper encoding: %20 (space), %26 (&), %2B (+), %25 (%), %3D (=)
• Tests now exercise the actual double-decoding code path

🟡 Issue #2 - ExtractRawQueryParameter documentation:

• Added comprehensive doc comment explaining the URL-encoded input assumption
• Documented that it splits on unencoded & (parameter separators) while %26 in values is preserved
• Clarified the method won't work correctly if queryString is already decoded

🟡 Issue #4 - Made ExtractRawQueryParameter internal:

• Changed from private static to internal static
• Removed all reflection-based test code
• Tests now call RequestParser.ExtractRawQueryParameter() directly
• InternalsVisibleTo already exists for Azure.DataApiBuilder.Service.Tests

🟡 Issue #3 - RawQueryString null handling:

• This is an impossible state in production (RawQueryString is always set when ParsedQueryString has keys)
• The existing null check + exception provides clear error messaging for debugging if something unexpected occurs

🟢 Issue #5 - Core fix approach confirmed correct ✓

[JerryNixon]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 6 pipeline(s).

[Aniruddh25]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 6 pipeline(s).

[Aniruddh25]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 6 pipeline(s).

[Aniruddh25]

/home/vsts/work/1/s/src/Core/Parsers/RequestParser.cs(125,25): error IDE2003: Blank line required between block and subsequent statement [/home/vsts/work/1/s/src/Core/Azure.DataApiBuilder.Core.csproj]
/home/vsts/work/1/s/src/Core/Parsers/RequestParser.cs(138,25): error IDE2003: Blank line required between block and subsequent statement [/home/vsts/work/1/s/src/Core/Azure.DataApiBuilder.Core.csproj]
/home/vsts/work/1/s/src/Core/Parsers/RequestParser.cs(327,13): error IDE2003: Blank line required between block and subsequent statement [/home/vsts/work/1/s/src/Core/Azure.DataApiBuilder.Core.csproj]
/home/vsts/work/1/s/src/Core/Parsers/RequestParser.cs(317,13): error IDE0011: Add braces to 'if' statement. [/home/vsts/work/1/s/src/Core/Azure.DataApiBuilder.Core.csproj]
/home/vsts/work/1/s/src/Core/Parsers/RequestParser.cs(325,17): error IDE0011: Add braces to 'if' statement. [/home/vsts/work/1/s/src/Core/Azure.DataApiBuilder.Core.csproj]

@copilot, fix the above dotnet format errors. And check no more failures by running:
dotnet format

[Copilot]

/home/vsts/work/1/s/src/Core/Parsers/RequestParser.cs(125,25): error IDE2003: Blank line required between block and subsequent statement [/home/vsts/work/1/s/src/Core/Azure.DataApiBuilder.Core.cspro...

Fixed in commit a78b348:

• Added blank lines after if-blocks (IDE2003 errors on lines 125, 138, 327)
• Added braces to single-line if statements (IDE0011 errors on lines 317, 325)
• Verified with dotnet format --verify-no-changes: All formatting checks pass

[Aniruddh25]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 6 pipeline(s).