Skip to content

fix: line content for repeated secrets and 10K byte lines #399

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
cx-diogo-rocha wants to merge 7 commits into
base: master
Choose a base branch
from
Open

fix: line content for repeated secrets and 10K byte lines #399

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
161663a
Fix line content, now correctly handling repeated secrets in same line
cx-diogo-rocha Sep 2, 2026
817f44a
Adding support for line contents bigger than 10K bytes
cx-diogo-rocha Sep 3, 2026
876541e
Another fix to line content. Exporting RegexSuffix
cx-diogo-rocha Sep 3, 2026
6d7c099
Remove change from allowLists
cx-diogo-rocha Sep 10, 2026
974dabc
Fix trivy issues
cx-diogo-rocha Sep 10, 2026
3e89092
Review change to GetRegexSuffix
cx-diogo-rocha Sep 11, 2026
ae8f9db
Update unit test that was failing due to update to go 1.27
cx-diogo-rocha Sep 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# and "Missing User Instruction" since 2ms container is stopped after scan

# Builder image
FROM checkmarx/go:1.26.4-r0-cb8702a93db4a4@sha256:cb8702a93db4a4b07da9c6e6e93bf412091e7b121b6ade95aa4228ec5fae4301 AS builder
FROM checkmarx/go:1.27.0-r1-7ff3a27a305109@sha256:7ff3a27a305109341ebf351a1421172d7ee41aeeeb0609451ddb6c8ee5d144b3 AS builder

WORKDIR /app

Expand All @@ -20,7 +20,7 @@ COPY . .
RUN GOOS=linux GOARCH=amd64 go build -buildvcs=false -ldflags="-s -w" -a -o /app/2ms .

# Runtime image
FROM checkmarx/git:2.55.0-r4-17886d1320eb5b@sha256:17886d1320eb5b3370f1c4bc4e3f95b5005cd25058ae254d051a85dcde76c33c
FROM checkmarx/git:2.55.0-r5-d0ccbb0b82fcb8@sha256:d0ccbb0b82fcb8c84ee36087b47eefbf59f8259c4f902fbb3591acd1ee00c546

WORKDIR /app

Expand Down
2 changes: 1 addition & 1 deletion cmd/config_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -350,7 +350,7 @@ func TestCustomRulesFlag(t *testing.T) {
name: "Invalid rule type",
customRulesFile: "testData/customRulesInvalidRuleType.json",
expectedRules: nil,
expectErrors: []error{fmt.Errorf("cannot unmarshal number -2 into Go struct field Rule.scoreRuleType of type uint8")},
expectErrors: []error{fmt.Errorf("cannot unmarshal number -2"), fmt.Errorf("scoreRuleType")},
},
}

Expand Down
10 changes: 5 additions & 5 deletions engine/engine.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -552,18 +552,18 @@ func buildSecret(
}
value.Line = strings.ReplaceAll(value.Line, "\r", "")

lineContent, err := linecontent.GetLineContent(value.Line, value.Secret)
if err != nil {
return nil, fmt.Errorf("failed to get line content for source %s: %w", item.GetSource(), err)
}

adjustedStartColumn := value.StartColumn
adjustedEndColumn := value.EndColumn
if hasNewline {
adjustedStartColumn--
adjustedEndColumn--
}

lineContent, err := linecontent.GetLineContent(value.Line, value.Secret, adjustedStartColumn)

@cx-julio-silva cx-julio-silva Sep 10, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If hasNewline == true, both adjustedStartColumn and adjustedEndColumn will be reduced by 1. However, line 553 might remove multiple \r characters and that isn't being accounted for in the same way as \n. This previously wasn't used for linecontent.GetLineContent, but now that it is, won't this cause issues if \r characters are removed without correcting adjustedStartColumn ?

@cx-diogo-rocha cx-diogo-rocha Sep 11, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

\r should only happen in Windows new lines and in that case adjusting the columns would break the secret position. For example

api_key=\r\n
integration_api_key_prod_abc123def456

Has endColumn 37 (on second line), and this is true with or without the presence of \r. But if we adjust the EndColumn with removal of \r, it would be 36 and become incorrect

if err != nil {
return nil, fmt.Errorf("failed to get line content for source %s: %w", item.GetSource(), err)
}

secret := &secrets.Secret{
ID: findingID,
Source: item.GetSource(),
Expand Down
24 changes: 19 additions & 5 deletions engine/linecontent/linecontent.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ const (
contextRightSizeLimit = 250
)

func GetLineContent(line, secret string) (string, error) {
func GetLineContent(line, secret string, startColumn int) (string, error) {
lineSize := len(line)
if lineSize == 0 {
return "", fmt.Errorf("line empty")
Expand All @@ -21,14 +21,28 @@ func GetLineContent(line, secret string) (string, error) {
return "", fmt.Errorf("secret empty")
}

// Truncate lineContent to max size
// For lines > lineMaxParseSize, get just the necessary context around the secret
if lineSize > lineMaxParseSize {
line = line[:lineMaxParseSize]
matchStartIndex := startColumn - 1
windowStart := 0
if matchStartIndex >= 0 && matchStartIndex < lineSize {
windowStart = max(matchStartIndex-lineMaxParseSize/2, 0)
// adjust line context window if windowStart+lineMaxParseSize would be higher than lineSize
windowStart = min(windowStart, lineSize-lineMaxParseSize)
}
line = line[windowStart : windowStart+lineMaxParseSize]
lineSize = lineMaxParseSize
startColumn -= windowStart
}

// Find the secret's position in the line
secretStartIndex := strings.Index(line, secret)
// The same secret value can appear more than once on a line. Search for it at the relevant index for this secret instance
secretStartIndex := -1
matchStartIndex := startColumn - 1
if matchStartIndex >= 0 && matchStartIndex < lineSize {
if idx := strings.Index(line[matchStartIndex:], secret); idx != -1 {
secretStartIndex = matchStartIndex + idx
}
}
if secretStartIndex == -1 {
// Secret not found, return truncated content based on context limits
maxSize := contextLeftSizeLimit + contextRightSizeLimit
Expand Down
98 changes: 69 additions & 29 deletions engine/linecontent/linecontent_test.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ func TestGetLineContent(t *testing.T) {
name string
line string
secret string
startColumn int
expected string
error bool
errorMessage string
Expand Down Expand Up @@ -59,11 +60,12 @@ func TestGetLineContent(t *testing.T) {
error: false,
},
{
name: "Secret at the beginning with line size smaller than the parse limit",
line: "start:" + dummySecret + strings.Repeat("A", lineMaxParseSize/2),
secret: dummySecret,
expected: "start:" + dummySecret + strings.Repeat("A", contextRightSizeLimit),
error: false,
name: "Secret at the beginning with line size smaller than the parse limit",
line: "start:" + dummySecret + strings.Repeat("A", lineMaxParseSize/2),
secret: dummySecret,
startColumn: len("start:") + 1,
expected: "start:" + dummySecret + strings.Repeat("A", contextRightSizeLimit),
error: false,
},
{
name: "Secret found in middle with line size smaller than the parse limit",
Expand All @@ -74,43 +76,81 @@ func TestGetLineContent(t *testing.T) {
"A",
contextRightSizeLimit,
) + "end",
secret: dummySecret,
expected: strings.Repeat("A", contextLeftSizeLimit) + dummySecret + strings.Repeat("A", contextRightSizeLimit),
error: false,
secret: dummySecret,
startColumn: len("start") + contextLeftSizeLimit + 1,
expected: strings.Repeat("A", contextLeftSizeLimit) + dummySecret + strings.Repeat("A", contextRightSizeLimit),
error: false,
},
{
name: "Secret at the end with line size smaller than the parse limit",
line: strings.Repeat("A", lineMaxParseSize/2) + dummySecret + ":end",
secret: dummySecret,
expected: strings.Repeat("A", contextLeftSizeLimit) + dummySecret + ":end",
error: false,
name: "Secret at the end with line size smaller than the parse limit",
line: strings.Repeat("A", lineMaxParseSize/2) + dummySecret + ":end",
secret: dummySecret,
startColumn: lineMaxParseSize/2 + 1,
expected: strings.Repeat("A", contextLeftSizeLimit) + dummySecret + ":end",
error: false,
},
{
name: "Secret at the beginning with line size larger than the parse limit",
line: "start:" + dummySecret + strings.Repeat("A", lineMaxParseSize),
secret: dummySecret,
expected: "start:" + dummySecret + strings.Repeat("A", contextRightSizeLimit),
error: false,
name: "Secret at the beginning with line size larger than the parse limit",
line: "start:" + dummySecret + strings.Repeat("A", lineMaxParseSize),
secret: dummySecret,
startColumn: len("start:") + 1,
expected: "start:" + dummySecret + strings.Repeat("A", contextRightSizeLimit),
error: false,
},
{
name: "Secret found in middle with line size larger than the parse limit",
line: "start" + strings.Repeat("A", contextLeftSizeLimit) + dummySecret + strings.Repeat("A", lineMaxParseSize) + "end",
secret: dummySecret,
expected: strings.Repeat("A", contextLeftSizeLimit) + dummySecret + strings.Repeat("A", contextRightSizeLimit),
error: false,
name: "Secret found in middle with line size larger than the parse limit",
line: "start" + strings.Repeat("A", contextLeftSizeLimit) + dummySecret + strings.Repeat("A", lineMaxParseSize) + "end",
secret: dummySecret,
startColumn: len("start") + contextLeftSizeLimit + 1,
expected: strings.Repeat("A", contextLeftSizeLimit) + dummySecret + strings.Repeat("A", contextRightSizeLimit),
error: false,
},
{
name: "Secret at the end with line size larger than the parse limit",
line: strings.Repeat("A", lineMaxParseSize-100) + dummySecret + strings.Repeat("A", lineMaxParseSize),
secret: dummySecret,
expected: strings.Repeat("A", contextLeftSizeLimit) + dummySecret + strings.Repeat("A", 100-len(dummySecret)),
error: false,
name: "Secret at the end with line size larger than the parse limit",
line: strings.Repeat("A", lineMaxParseSize-100) + dummySecret + strings.Repeat("A", lineMaxParseSize),
secret: dummySecret,
startColumn: lineMaxParseSize - 100 + 1,
expected: strings.Repeat("A", contextLeftSizeLimit) + dummySecret + strings.Repeat("A", contextRightSizeLimit),
error: false,
},
{
name: "Secret repeated on the same line uses the occurrence at startColumn",
line: strings.Repeat("a", contextLeftSizeLimit) + dummySecret + strings.Repeat("b", contextLeftSizeLimit+contextRightSizeLimit) + dummySecret + strings.Repeat("c", contextRightSizeLimit),
secret: dummySecret,
startColumn: contextLeftSizeLimit + len(dummySecret) + contextLeftSizeLimit + contextRightSizeLimit + 1,
expected: strings.Repeat("b", contextLeftSizeLimit) + dummySecret + strings.Repeat("c", contextRightSizeLimit),
error: false,
},
{
name: "Secret repeated with startColumn pointing at the containing match, not the secret itself",
line: strings.Repeat("a", contextLeftSizeLimit) + "KEY=" + dummySecret + strings.Repeat(
"b",
contextLeftSizeLimit+contextRightSizeLimit,
) + "KEY=" + dummySecret + strings.Repeat("c", contextRightSizeLimit),
secret: dummySecret,
startColumn: contextLeftSizeLimit + len("KEY=") + len(dummySecret) + contextLeftSizeLimit + contextRightSizeLimit + 1,
expected: strings.Repeat("b", contextLeftSizeLimit-len("KEY=")) + "KEY=" + dummySecret + strings.Repeat("c", contextRightSizeLimit),
error: false,
},
{
name: "Secret far beyond the parse limit is not truncated away",
line: strings.Repeat("x", lineMaxParseSize*3) + "PRE" + dummySecret + "POST" + strings.Repeat(
"y",
lineMaxParseSize,
),
secret: dummySecret,
startColumn: lineMaxParseSize*3 + len("PRE") + 1,
expected: strings.Repeat("x", contextLeftSizeLimit-len("PRE")) + "PRE" + dummySecret + "POST" + strings.Repeat(
"y",
contextRightSizeLimit-len("POST"),
),
error: false,
},
}

for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, err := GetLineContent(tt.line, tt.secret)
got, err := GetLineContent(tt.line, tt.secret, tt.startColumn)
if (err != nil) != tt.error {
t.Fatalf("GetLineContent() error = %v, wantErr %v", err, tt.error)
}
Expand Down
6 changes: 3 additions & 3 deletions engine/rules/ruledefine/utils.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ const (
// \x60 = `
secretPrefixUnique = `\b(`
secretPrefix = `[\x60'"\s=]{0,20}(` //nolint:gosec // This is a regex pattern
secretSuffix = `)(?:[\x60'"\s;]|\\[nr]|$)` //nolint:gosec // This is a regex pattern
SecretSuffix = `)(?:[\x60'"\s;]|\\[nr]|$)` //nolint:gosec // This is a regex pattern
secretSuffixIncludingXml = `)(?:['|\"|\n|\r|\s|\x60|;]|\\n|\\r|$|\s{0,10}<\/string>)` //nolint:gosec // This is a regex pattern
)

Expand All @@ -46,7 +46,7 @@ func generateSemiGenericRegex(identifiers []string, secretRegex string, isCaseIn
sb.WriteString(operator)
sb.WriteString(secretPrefix)
sb.WriteString(secretRegex)
sb.WriteString(secretSuffix)
sb.WriteString(SecretSuffix)
return regexp.MustCompile(sb.String())
}

Expand All @@ -63,7 +63,7 @@ func generateUniqueTokenRegex(secretRegex string, isCaseInsensitive bool) *regex
}
sb.WriteString(secretPrefixUnique)
sb.WriteString(secretRegex)
sb.WriteString(secretSuffix)
sb.WriteString(SecretSuffix)
return regexp.MustCompile(sb.String())
}

Expand Down
8 changes: 4 additions & 4 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
module github.com/checkmarx/2ms/v5

go 1.26.4
go 1.27.0

replace (
golang.org/x/oauth2 => golang.org/x/oauth2 v0.30.0
Expand Down Expand Up @@ -29,9 +29,9 @@ require (
github.com/zricethezav/gitleaks/v8 v8.28.0
go.uber.org/mock v0.5.2
golang.org/x/exp v0.0.0-20250218142911-aa4b98e5adaa
golang.org/x/net v0.56.0
golang.org/x/net v0.57.0
golang.org/x/sync v0.22.0
golang.org/x/text v0.40.0
golang.org/x/text v0.41.0
golang.org/x/time v0.5.0
gopkg.in/yaml.v3 v3.0.1
)
Expand Down Expand Up @@ -88,6 +88,6 @@ require (
github.com/yusufpapurcu/wmi v1.2.4 // indirect
go.uber.org/multierr v1.11.0 // indirect
go4.org v0.0.0-20230225012048-214862532bf5 // indirect
golang.org/x/crypto v0.54.0 // indirect
golang.org/x/crypto v0.56.0 // indirect
golang.org/x/sys v0.47.0 // indirect
)
12 changes: 6 additions & 6 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1252,8 +1252,8 @@ golang.org/x/crypto v0.26.0/go.mod h1:GY7jblb9wI+FOo5y8/S2oY4zWP07AkOJ4+jxCqdqn5
golang.org/x/crypto v0.31.0/go.mod h1:kDsLvtWBEx7MV9tJOj9bnXsPbxwJQ6csT/x4KIN4Ssk=
golang.org/x/crypto v0.32.0/go.mod h1:ZnnJkOaASj8g0AjIduWNlq2NRxL0PlBrbKVyZ6V/Ugc=
golang.org/x/crypto v0.36.0/go.mod h1:Y4J0ReaxCR1IMaabaSMugxJES1EpwhBHhv2bDHklZvc=
golang.org/x/crypto v0.54.0 h1:YLIA59K4fiNzHzjnZt2tUJQjQtUWfWbeHBqKtk3eScw=
golang.org/x/crypto v0.54.0/go.mod h1:KWL8ny2AZdGR2cWmzeHrp2azQPGogOv+HeQaVEXC2dk=
golang.org/x/crypto v0.56.0 h1:GUh5Ii4J5jtcseSMiRqr1jXCNHoxjeV9Fmekc2oLy6Y=
golang.org/x/crypto v0.56.0/go.mod h1:OMW5y6CY9l38uPLmxU6l6pwcXp1obtLo3e6gT7gQR2I=
golang.org/x/exp v0.0.0-20180321215751-8460e604b9de/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20180807140117-3d87b88a115f/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
Expand Down Expand Up @@ -1387,8 +1387,8 @@ golang.org/x/net v0.33.0/go.mod h1:HXLR5J+9DxmrqMwG9qjGCxZ+zKXxBru04zlTvWlWuN4=
golang.org/x/net v0.34.0/go.mod h1:di0qlW3YNM5oh6GqDGQr92MyTozJPmybPK4Ev/Gm31k=
golang.org/x/net v0.37.0/go.mod h1:ivrbrMbzFq5J41QOQh0siUuly180yBYtLp+CKbEaFx8=
golang.org/x/net v0.38.0/go.mod h1:ivrbrMbzFq5J41QOQh0siUuly180yBYtLp+CKbEaFx8=
golang.org/x/net v0.56.0 h1:Rw8j/hFzGvJUZwNBXnAtf5sVDVt+65SK2C7IxCxZt5o=
golang.org/x/net v0.56.0/go.mod h1:D3Ku6r+V6JROoZK144D2XfMHFcMq/0zSfLelVTCFKec=
golang.org/x/net v0.57.0 h1:K5+3DljvIuDG9/Jv9rvyMywYNFCQ9RSUY6OOTTkT+tE=
golang.org/x/net v0.57.0/go.mod h1:KpXc8iv+r3XplLAG/f7Jsf9RPszJzdR0f58q9vGOuEU=
golang.org/x/oauth2 v0.30.0/go.mod h1:B++QgG3ZKulg6sRPGD/mqlHQs5rB3Ml9erfeDY7xKlU=
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20190227155943-e225da77a7e6/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
Expand Down Expand Up @@ -1558,8 +1558,8 @@ golang.org/x/text v0.16.0/go.mod h1:GhwF1Be+LQoKShO3cGOHzqOgRrGaYc9AvblQOmPVHnI=
golang.org/x/text v0.17.0/go.mod h1:BuEKDfySbSR4drPmRPG/7iBdf8hvFMuRexcpahXilzY=
golang.org/x/text v0.21.0/go.mod h1:4IBbMaMmOPCJ8SecivzSH54+73PCFmPWxNTLm+vZkEQ=
golang.org/x/text v0.23.0/go.mod h1:/BLNzu4aZCJ1+kcD0DNRotWKage4q2rGVAg4o22unh4=
golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs=
golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY=
golang.org/x/text v0.41.0 h1:vz/seA0lnX87Othu2f/0L24RcgrXD9/YFTSuGjj3rH8=
golang.org/x/text v0.41.0/go.mod h1:jvf1O8ajNzZqhSrQBPbutR/EB83Cc0CFrezNQIwbb5M=
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20191024005414-555d28b269f0/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
Expand Down
4 changes: 4 additions & 0 deletions pkg/rules.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,7 @@ import (
func GetDefaultRules(includeDeprecated bool) []*ruledefine.Rule {
return rules.GetDefaultRules(includeDeprecated)
}

func GetRegexSuffix() string {
return ruledefine.SecretSuffix
}
Loading