feat: slack reporting

[redpanda-f]

It bases itself on #66, merge that before so diff is smaller.

See issue #8 for context.

Motivation: Introduce reporting of select workflows as needed

[Copilot]

Pull request overview

Adds Slack notifications for CI runs, including deep links to uploaded run artifacts, to improve visibility into nightly/CI health and debugging.

Changes:

• Introduces a new Notify Slack workflow that runs on workflow_run completion of the CI workflow and posts a structured Slack message.
• Updates the CI workflow to run nightly on a cron schedule.
• Adds AWS/S3 setup and a post-run upload step to sync ~/.foc-devnet/state/latest to S3 for later inspection.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
.github/workflows/notify-slack.yml	New workflow that queries CI job results and posts a Slack message with job status + S3 artifact links.
.github/workflows/ci.yml	Adds nightly schedule, AWS/S3 bootstrap, S3 upload of run state, and adjusts start invocation.

[Copilot]

The comments above this step say CI_LOGS_BUCKET is a "repository variable" and mention "AWS_REGION" secrets, but the workflow actually sources CI_LOGS_BUCKET and region from secrets.* (and uses AWS_DEFAULT_REGION). Please align the comments with the actual configuration to avoid confusion during setup/rotation.

[BigLep]

A few things:

1. Where did the slack reporting requirement come from? In #8 we discussed creating a github issue if the nightly job fails. I agree slack is nice for more "in your face" visibility, but the problem I see with slack threads are:
   • Easy to get lost in the stream of other threads.
   • Doesn't provide a durable place to have followup task tracking, investigation, etc.
   • They don't show up on our standup board. If nightly jobs are failing, I would want this talked about during standup and I'd like to assign someone to it.
     I think all of the above is facilitated better with an issue, but I am open to change my mind if the team feels differently.
2. (probably a misunderstanding by me) If "it bases itself on #66", then why isn't this PR targeting https://github.com/FilOzone/foc-devnet/tree/feat/redpanda/s3-uploads ?
3. (nit) I think it's ideal if PRs give more context for the motivation of the change. That can be as simple as linking to the issue that motivated the work (which I assume is #8 in this case).

[BigLep]

Why package this as a an action vs. just a secript that the ci.yml workflow can use?

[redpanda-f]

A few things:

1. Where did the slack reporting requirement come from? In CI/Nightly End-to-End Validation of FOC as a whole #8 we discussed creating a github issue if the nightly job fails. I agree slack is nice for more "in your face" visibility, but the problem I see with slack threads are:

   • Easy to get lost in the stream of other threads.
   • Doesn't provide a durable place to have followup task tracking, investigation, etc.
   • They don't show up on our standup board. If nightly jobs are failing, I would want this talked about during standup and I'd like to assign someone to it.
     I think all of the above is facilitated better with an issue, but I am open to change my mind if the team feels differently.

2. (probably a misunderstanding by me) If "it bases itself on feat: upload CI logs to S3, CI gets called every midnight #66", then why isn't this PR targeting https://github.com/FilOzone/foc-devnet/tree/feat/redpanda/s3-uploads ?

3. (nit) I think it's ideal if PRs give more context for the motivation of the change. That can be as simple as linking to the issue that motivated the work (which I assume is CI/Nightly End-to-End Validation of FOC as a whole #8 in this case).

1. I can extend this PR itself for including GH issue posting. In fact, slack reporting can be removed entirely if bots are not useful (I think they are a nice to have).
2. I try keeping PRs smaller so single functionalities are introduced in one PR at a time. Sometimes, that requires PRs to use other running PRs codebase. There are some ways of tackling it:
   2.1. Make the dependent PR (this) target the base (in this case feat/redpanda/s3-uploads). This holds back the base PR merge till all dependents are ready.
   2.2. Make the dependent PR (this) target the main, but call it out. Base PR merge is not held back, and dependent PR is blocked till base goes in mainline. The pro is the dependency is clear, the con is that PR diff looks larger than it really is.
   I generally go with 2.2 as the is clarity of dependency, and PRs are single utility. However, no strong opinions. Composite large PRs are also okay, just clunky imo.
3. Will do.

Why package this as a an action vs. just a secript that the ci.yml workflow can use?

I am assuming there will be workflows, one per composite e2e test. Each of them can decide whether they want to report it or not. For example, ci run where --notest is run may not need an issue created.

[redpanda-f]

Slack reporting is deprioritized / removed. #68 will be closed. Only GH issues are focus for now, since that's where a lot of our tracking happens anyways. There is @rvagg and @BigLep buy-in here.

will raise a separate composite PR for this.