ci: adjust CPU and memory settings

[morrisonlevi]

Description

K8s is scheduling based on REQUEST, but allowed to run up to LIMIT. So if a job requests 3 GiB of memory, it's allowed to go up to 4 GiB. On a whole-node basis, if enough jobs do this then node can run out of memory and start using OOM-kill. Note that for CPU, the limit is not enforced unless the request is equal to the limit (but I'm not sure that's actually happening either, but it's what it's supposed to be doing).

These specific job families failed for me in PRs, which is why I've adjusted these specific jobs. Note that in the case of appsec, it did not reach its own limit, but rather the node it was on became full.

I am increasing the limits by feedback from a coworker who supports our CI infrastructure.

Reviewer checklist

• Test coverage seems ok.
• Appropriate labels assigned.

[datadog-datadog-prod-us1]

⚠️ Tests

✨ Fix all issues with BitsAI or with Cursor

⚠️ Warnings

🧪 25 Tests failed

testSearchPhpBinaries from integration.DDTrace\Tests\Integration\PHPInstallerTest (Datadog) (Fix with Cursor)

Risky Test
phpvfscomposer://tests/vendor/phpunit/phpunit/phpunit:52

testSimplePushAndProcess from laravel-58-test.DDTrace\Tests\Integrations\Laravel\V5_8\QueueTest (Datadog) (Fix with Cursor)

DDTrace\Tests\Integrations\Laravel\V5_8\QueueTest::testSimplePushAndProcess
Test code or tested code printed unexpected output: spanLinksTraceId: 699dda4f00000000ef6251faafd4ca84
tid: 699dda4f00000000
hexProcessTraceId: ef6251faafd4ca84
hexProcessSpanId: 22ce8e2d0b008ae0
processTraceId: 17249439659916053124
processSpanId: 2508098366601202400

phpvfscomposer://tests/vendor/phpunit/phpunit/phpunit:106

testSimplePushAndProcess from laravel-8x-test.DDTrace\Tests\Integrations\Laravel\V8_x\QueueTest (Datadog) (Fix with Cursor)

DDTrace\Tests\Integrations\Laravel\V8_x\QueueTest::testSimplePushAndProcess
Test code or tested code printed unexpected output: spanLinksTraceId: 699ddacc00000000706fc6488268aaf5
tid: 699ddacc00000000
hexProcessTraceId: 706fc6488268aaf5
hexProcessSpanId: c7d8fa6f3652c525
processTraceId: 8101912269390654197
processSpanId: 14400535164076868901

View all

ℹ️ Info

❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 8e059d1 | Docs | Datadog PR Page | Was this helpful? Give us feedback!}

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 62.17%. Comparing base (a3409cd) to head (8e059d1).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3671      +/-   ##
==========================================
- Coverage   62.22%   62.17%   -0.06%     
==========================================
  Files         141      141              
  Lines       13352    13352              
  Branches     1746     1746              
==========================================
- Hits         8308     8301       -7     
- Misses       4253     4260       +7     
  Partials      791      791              

see 2 files with indirect coverage changes

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a3409cd...8e059d1. Read the comment docs.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[morrisonlevi]

Note that this reports a quite large number, such as 66. This means we should be setting the number of works for parallel systems and not let them discover it!

[morrisonlevi]

These numbers look good in one of dashboards. Note that we are really memory bound and for parts of it we are only engaging a single core so there's not a lot of value in increasing CPUs at the moment.

5 GiB is too small, I've observed 4.9 GiB in practice.