Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,7 @@ jobs:
- uses: ./.github/actions/setup
- uses: ./.github/actions/run-bazel-test
with:
target: //test/integration/submitqueue/extension/counter/...
target: //test/integration/extension/counter/...

queue-integration-test:
name: Queue Extension Test
Expand All @@ -128,7 +128,7 @@ jobs:
- uses: ./.github/actions/setup
- uses: ./.github/actions/run-bazel-test
with:
target: //test/integration/extension/queue/...
target: //test/integration/extension/messagequeue/...

storage-integration-test:
name: Storage Extension Test
Expand Down
8 changes: 4 additions & 4 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -140,8 +140,8 @@ Paths follow the directory layout: shared code is top-level, domain code nests u
- Domain entities: `github.com/uber/submitqueue/{domain}/entity` (e.g. `.../submitqueue/entity`)
- Domain extensions: `github.com/uber/submitqueue/{domain}/extension/{ext}[/{impl}]` (e.g. `.../submitqueue/extension/storage/mysql`)
- Domain-internal infra: `github.com/uber/submitqueue/{domain}/core/{pkg}` (e.g. `.../submitqueue/core/consumer`, `.../submitqueue/core/request`)
- Shared entities: `github.com/uber/submitqueue/entity/{name}` (e.g. `.../entity/queue`)
- Shared extensions: `github.com/uber/submitqueue/extension/{name}` (e.g. `.../extension/queue`)
- Shared entities: `github.com/uber/submitqueue/entity/{name}` (e.g. `.../entity/messagequeue`)
- Shared extensions: `github.com/uber/submitqueue/extension/{name}` (e.g. `.../extension/messagequeue`)
- Cross-domain infra: `github.com/uber/submitqueue/core/{pkg}` (e.g. `.../core/errs`, `.../core/metrics`)

## Development
Expand All @@ -167,7 +167,7 @@ Generated proto files are committed. When modifying `.proto` files:
- **Directories**: singular (`mock/`, `entity/`, not `mocks/`, `entities/`)
- **Files**: `{method}.go`, `{entity}.go`, `{file}_test.go`, `BUILD.bazel`
- **Proto files**: `{service}.proto`
- **Test compose contexts**: the `testContext` passed to `NewComposeStack` (and thus the `sq-test-{context}-…` Docker project/container names) must be **domain-qualified** — `{category}-{domain}-{name}` where `{category}` is `svc`/`ext`/`core`/`e2e` and `{domain}` is `submitqueue`/`stovepipe`/… (omit the domain only for shared/cross-domain suites, e.g. `ext-queue-sql`). This keeps containers unambiguous and lets suites run in parallel. See [doc/howto/TESTING.md](doc/howto/TESTING.md#container-naming).
- **Test compose contexts**: the `testContext` passed to `NewComposeStack` (and thus the `sq-test-{context}-…` Docker project/container names) must be **domain-qualified** — `{category}-{domain}-{name}` where `{category}` is `svc`/`ext`/`core`/`e2e` and `{domain}` is `submitqueue`/`stovepipe`/… (omit the domain only for shared/cross-domain suites, e.g. `ext-messagequeue-sql`). This keeps containers unambiguous and lets suites run in parallel. See [doc/howto/TESTING.md](doc/howto/TESTING.md#container-naming).
- **README files**: Do not duplicate interface or type definitions as code blocks in READMEs. Describe behavior in prose and let readers navigate to the source. Only include code samples when explicitly instructed.
- **Markdown prose width**: Do not hard-wrap prose in Markdown docs (RFCs under `doc/`, READMEs). Write one line per paragraph and one line per list item, and let the editor soft-wrap — hard wrapping at a fixed column renders as a narrow fixed-width column regardless of window size. Code blocks, tables, and ASCII diagrams keep their own line breaks.

Expand Down Expand Up @@ -241,7 +241,7 @@ To create a mock package for a new extension (e.g., `submitqueue/extension/newex
3. Run `make mocks` to generate mock files into the new directory.
4. Run `make gazelle` to create the `BUILD.bazel` file automatically.

For inline mocks (mock in the same package, e.g., `extension/queue/mysql/mock_stores.go`):
For inline mocks (mock in the same package, e.g., `extension/messagequeue/mysql/mock_stores.go`):

1. Add a `//go:generate` directive with `-package=mypkg` and `-destination=mock_file.go`.
2. Run `make mocks` and `make gazelle`.
Expand Down
10 changes: 5 additions & 5 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -194,20 +194,20 @@ local-init-submitqueue-schemas: ## Manually apply all database schemas
docker exec -i $(SUBMITQUEUE_LOCAL_PROJECT)-mysql-app-1 mysql -uroot -proot submitqueue < $$file 2>&1 | grep -v "Using a password" || true; \
done
@echo "Applying counter schema to mysql-app..."
@for file in submitqueue/extension/counter/mysql/schema/*.sql; do \
@for file in extension/counter/mysql/schema/*.sql; do \
echo " - Applying $$(basename $$file)..."; \
docker exec -i $(SUBMITQUEUE_LOCAL_PROJECT)-mysql-app-1 mysql -uroot -proot submitqueue < $$file 2>&1 | grep -v "Using a password" || true; \
done
@echo "Applying queue schema to mysql-queue..."
@for file in extension/queue/mysql/schema/*.sql; do \
@for file in extension/messagequeue/mysql/schema/*.sql; do \
echo " - Applying $$(basename $$file)..."; \
docker exec -i $(SUBMITQUEUE_LOCAL_PROJECT)-mysql-queue-1 mysql -uroot -proot submitqueue < $$file 2>&1 | grep -v "Using a password" || true; \
done
@echo "✅ All schemas applied successfully"

local-init-stovepipe-queue-schema: ## Apply queue schema only (mysql-queue) for Stovepipe compose stacks
@echo "Applying queue schema to mysql-queue (Stovepipe; no app storage/counter schema yet)..."
@for file in extension/queue/mysql/schema/*.sql; do \
@for file in extension/messagequeue/mysql/schema/*.sql; do \
echo " - Applying $$(basename $$file)..."; \
docker exec -i $(STOVEPIPE_LOCAL_PROJECT)-mysql-queue-1 mysql -uroot -proot submitqueue < $$file 2>&1 | grep -v "Using a password" || true; \
done
Expand Down Expand Up @@ -334,7 +334,7 @@ local-stovepipe-gateway-start: build-stovepipe-gateway-linux ## Start Stovepipe

mocks: ## Generate mock files using mockgen
@echo "Generating mocks..."
@$(BAZEL) run @rules_go//go -- generate ./submitqueue/extension/storage/... ./submitqueue/extension/buildrunner/... ./submitqueue/extension/changestore/... ./submitqueue/extension/counter/... ./extension/queue/... ./submitqueue/extension/queueconfig/... ./submitqueue/extension/mergechecker/... ./submitqueue/extension/pusher/... ./submitqueue/extension/scorer/... ./submitqueue/extension/conflict/... ./submitqueue/core/consumer/...
@$(BAZEL) run @rules_go//go -- generate ./submitqueue/extension/storage/... ./submitqueue/extension/buildrunner/... ./submitqueue/extension/changestore/... ./extension/counter/... ./extension/messagequeue/... ./submitqueue/extension/queueconfig/... ./submitqueue/extension/mergechecker/... ./submitqueue/extension/pusher/... ./submitqueue/extension/scorer/... ./submitqueue/extension/conflict/... ./submitqueue/core/consumer/...
@echo "Mocks generated successfully!"

proto: ## Generate protobuf files from .proto definitions
Expand Down Expand Up @@ -381,7 +381,7 @@ run-client-stovepipe-orchestrator:
@$(BAZEL) run //example/stovepipe/orchestrator/client:orchestrator -- -addr $(or $(SERVER_ADDR),localhost:8084) -message "$(or $(MESSAGE),ping)"

run-queue-admin: ## Run queue-admin CLI (use ARGS to pass arguments, e.g. make run-queue-admin ARGS="list-topics")
@$(BAZEL) run //extension/queue/mysql/ctl -- $(ARGS)
@$(BAZEL) run //extension/messagequeue/mysql/ctl -- $(ARGS)

test: ## Run unit tests
@echo "Running unit tests..."
Expand Down
10 changes: 5 additions & 5 deletions doc/howto/TESTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,13 +23,13 @@ SubmitQueue uses **two separate databases** to demonstrate proper architectural

### 1. Application Database
- **Purpose**: Business data (requests, counters, batches)
- **Schema**: `submitqueue/extension/storage/mysql/schema`, `submitqueue/extension/counter/mysql/schema`
- **Schema**: `submitqueue/extension/storage/mysql/schema`, `extension/counter/mysql/schema`
- **Used by**: Gateway (stores requests), Orchestrator (reads/updates request state)
- **Connection**: `MYSQL_DSN`

### 2. Queue Database
- **Purpose**: Messaging infrastructure (queue messages, offsets, partition leases)
- **Schema**: `extension/queue/mysql/schema`
- **Schema**: `extension/messagequeue/mysql/schema`
- **Used by**: Gateway (publishes), Orchestrator (consumes)
- **Connection**: `QUEUE_MYSQL_DSN`

Expand Down Expand Up @@ -128,7 +128,7 @@ The `{context}` passed to `NewComposeStack` is **domain-qualified** so that the
- `{domain}` — `submitqueue`, `stovepipe`, … — **omit for shared/cross-domain code**
- `{name}` — the specific service/extension (e.g. `gateway`, `storage-mysql`)

Shared (cross-domain) suites carry no domain segment — e.g. the shared queue extension uses `ext-queue-sql`.
Shared (cross-domain) suites carry no domain segment — e.g. the shared queue extension uses `ext-messagequeue-sql`.

### Context reference

Expand All @@ -138,9 +138,9 @@ Shared (cross-domain) suites carry no domain segment — e.g. the shared queue e
| SubmitQueue orchestrator | `svc-submitqueue-orchestrator` | `sq-test-svc-submitqueue-orchestrator-xyz789-orchestrator-service-1` |
| Stovepipe gateway | `svc-stovepipe-gateway` | `sq-test-svc-stovepipe-gateway-abc123-gateway-service-1` |
| SubmitQueue storage extension | `ext-submitqueue-storage-mysql` | `sq-test-ext-submitqueue-storage-mysql-2ce1d0-mysql-1` |
| SubmitQueue counter extension | `ext-submitqueue-counter-mysql` | `sq-test-ext-submitqueue-counter-mysql-…-mysql-1` |
| Counter extension (shared) | `ext-counter-mysql` | `sq-test-ext-counter-mysql-…-mysql-1` |
| SubmitQueue changestore extension | `ext-submitqueue-changestore-mysql` | `sq-test-ext-submitqueue-changestore-mysql-…-mysql-1` |
| Shared queue extension | `ext-queue-sql` | `sq-test-ext-queue-sql-a1b2c3-mysql-1` |
| Shared queue extension | `ext-messagequeue-sql` | `sq-test-ext-messagequeue-sql-a1b2c3-mysql-1` |
| SubmitQueue consumer (core) | `core-submitqueue-consumer` | `sq-test-core-submitqueue-consumer-…-mysql-1` |
| SubmitQueue e2e (full stack) | `e2e-submitqueue` | `sq-test-e2e-submitqueue-def456-gateway-service-1` |

Expand Down
10 changes: 5 additions & 5 deletions doc/rfc/sql-queue-rfc.md
Original file line number Diff line number Diff line change
Expand Up @@ -172,7 +172,7 @@ Messages are append-only. No per-message mutation during delivery.
- `(topic, partition_key, offset)`: Core fetch query — poll messages in partition ordered by offset
- `(topic, partition_key, id)`: Unique constraint and idempotent publish

See `extension/queue/mysql/schema/queue_messages.sql` for full schema.
See `extension/messagequeue/mysql/schema/queue_messages.sql` for full schema.

### Delivery State Table

Expand All @@ -184,7 +184,7 @@ Per-consumer-group delivery tracking with explicit ack state.
- `invisible_until`: Visibility timeout in epoch milliseconds (only meaningful when `acked = FALSE`)
- `retry_count`: Number of times message has been redelivered to this consumer group

See `extension/queue/mysql/schema/queue_delivery_state.sql` for full schema.
See `extension/messagequeue/mysql/schema/queue_delivery_state.sql` for full schema.

### Partition Leases Table

Expand All @@ -197,7 +197,7 @@ See `extension/queue/mysql/schema/queue_delivery_state.sql` for full schema.
- `(leased_by)`: Find all partitions owned by a worker
- `(lease_renewed_at)`: Detect stale leases across workers

See `extension/queue/mysql/schema/queue_partition_leases.sql` for full schema.
See `extension/messagequeue/mysql/schema/queue_partition_leases.sql` for full schema.

### Consumer Offsets Table

Expand All @@ -210,7 +210,7 @@ See `extension/queue/mysql/schema/queue_partition_leases.sql` for full schema.
- `(consumer_group)`: Monitor all offsets for a consumer group
- `(topic)`: Find all consumers for a topic

See `extension/queue/mysql/schema/queue_offsets.sql` for full schema.
See `extension/messagequeue/mysql/schema/queue_offsets.sql` for full schema.

### Subscriber Heartbeats Table

Expand All @@ -219,7 +219,7 @@ See `extension/queue/mysql/schema/queue_offsets.sql` for full schema.
- `heartbeat_at`: Unix timestamp in milliseconds of last heartbeat
- `deregistered_at`: Soft-delete timestamp (0 = active, >0 = deregistered during graceful shutdown)

See `extension/queue/mysql/schema/queue_subscriber_heartbeats.sql` for full schema.
See `extension/messagequeue/mysql/schema/queue_subscriber_heartbeats.sql` for full schema.

### Dead Letter Queue

Expand Down
Original file line number Diff line number Diff line change
@@ -1,15 +1,15 @@
load("@rules_go//go:def.bzl", "go_library", "go_test")

go_library(
name = "queue",
name = "messagequeue",
srcs = ["message.go"],
importpath = "github.com/uber/submitqueue/entity/queue",
importpath = "github.com/uber/submitqueue/entity/messagequeue",
visibility = ["//visibility:public"],
)

go_test(
name = "queue_test",
name = "messagequeue_test",
srcs = ["message_test.go"],
embed = [":queue"],
embed = [":messagequeue"],
deps = ["@com_github_stretchr_testify//assert"],
)
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.

package queue
package messagequeue

import (
"maps"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.

package queue
package messagequeue

import (
"testing"
Expand Down
4 changes: 2 additions & 2 deletions example/submitqueue/gateway/server/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,9 @@ go_library(
importpath = "github.com/uber/submitqueue/example/submitqueue/gateway/server",
visibility = ["//visibility:private"],
deps = [
"//extension/queue/mysql",
"//extension/counter/mysql",
"//extension/messagequeue/mysql",
"//submitqueue/core/consumer",
"//submitqueue/extension/counter/mysql",
"//submitqueue/extension/queueconfig/yaml",
"//submitqueue/extension/storage/mysql",
"//submitqueue/gateway/controller",
Expand Down
4 changes: 2 additions & 2 deletions example/submitqueue/gateway/server/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -28,9 +28,9 @@ import (

_ "github.com/go-sql-driver/mysql"
"github.com/uber-go/tally/v4"
queueMySQL "github.com/uber/submitqueue/extension/queue/mysql"
mysqlcounter "github.com/uber/submitqueue/extension/counter/mysql"
queueMySQL "github.com/uber/submitqueue/extension/messagequeue/mysql"
"github.com/uber/submitqueue/submitqueue/core/consumer"
mysqlcounter "github.com/uber/submitqueue/submitqueue/extension/counter/mysql"
yamlqueueconfig "github.com/uber/submitqueue/submitqueue/extension/queueconfig/yaml"
mysqlstorage "github.com/uber/submitqueue/submitqueue/extension/storage/mysql"
"github.com/uber/submitqueue/submitqueue/gateway/controller"
Expand Down
8 changes: 4 additions & 4 deletions example/submitqueue/orchestrator/server/BUILD.bazel
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,10 @@ go_library(
"//core/errs/generic",
"//core/errs/mysql",
"//core/httpclient",
"//extension/queue",
"//extension/queue/mysql",
"//extension/counter",
"//extension/counter/mysql",
"//extension/messagequeue",
"//extension/messagequeue/mysql",
"//submitqueue/core/consumer",
"//submitqueue/entity",
"//submitqueue/extension/buildrunner",
Expand All @@ -25,8 +27,6 @@ go_library(
"//submitqueue/extension/changestore",
"//submitqueue/extension/changestore/mysql",
"//submitqueue/extension/conflict/all",
"//submitqueue/extension/counter",
"//submitqueue/extension/counter/mysql",
"//submitqueue/extension/mergechecker",
"//submitqueue/extension/mergechecker/github",
"//submitqueue/extension/pusher",
Expand Down
10 changes: 5 additions & 5 deletions example/submitqueue/orchestrator/server/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -33,8 +33,10 @@ import (
genericerrs "github.com/uber/submitqueue/core/errs/generic"
mysqlerrs "github.com/uber/submitqueue/core/errs/mysql"
"github.com/uber/submitqueue/core/httpclient"
extqueue "github.com/uber/submitqueue/extension/queue"
queueMySQL "github.com/uber/submitqueue/extension/queue/mysql"
"github.com/uber/submitqueue/extension/counter"
mysqlcounter "github.com/uber/submitqueue/extension/counter/mysql"
extqueue "github.com/uber/submitqueue/extension/messagequeue"
queueMySQL "github.com/uber/submitqueue/extension/messagequeue/mysql"
"github.com/uber/submitqueue/submitqueue/core/consumer"
"github.com/uber/submitqueue/submitqueue/entity"
"github.com/uber/submitqueue/submitqueue/extension/buildrunner"
Expand All @@ -44,8 +46,6 @@ import (
"github.com/uber/submitqueue/submitqueue/extension/changestore"
mysqlchangestore "github.com/uber/submitqueue/submitqueue/extension/changestore/mysql"
"github.com/uber/submitqueue/submitqueue/extension/conflict/all"
"github.com/uber/submitqueue/submitqueue/extension/counter"
mysqlcounter "github.com/uber/submitqueue/submitqueue/extension/counter/mysql"
"github.com/uber/submitqueue/submitqueue/extension/mergechecker"
githubchecker "github.com/uber/submitqueue/submitqueue/extension/mergechecker/github"
"github.com/uber/submitqueue/submitqueue/extension/pusher"
Expand Down Expand Up @@ -203,7 +203,7 @@ func run() error {
// Create consumer.
c := consumer.New(logger.Sugar(), scope.SubScope("consumer"), registry,
genericerrs.Classifier,
// Storage (extension/storage/mysql) and queue (extension/queue/mysql)
// Storage (extension/storage/mysql) and queue (extension/messagequeue/mysql)
// both run on the same MySQL driver, so a single classifier covers
// errors surfaced from either backend.
mysqlerrs.Classifier,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,6 @@ load("@rules_go//go:def.bzl", "go_library")
go_library(
name = "counter",
srcs = ["counter.go"],
importpath = "github.com/uber/submitqueue/submitqueue/extension/counter",
importpath = "github.com/uber/submitqueue/extension/counter",
visibility = ["//visibility:public"],
)
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ load("@rules_go//go:def.bzl", "go_library")
go_library(
name = "mock",
srcs = ["counter_mock.go"],
importpath = "github.com/uber/submitqueue/submitqueue/extension/counter/mock",
importpath = "github.com/uber/submitqueue/extension/counter/mock",
visibility = ["//visibility:public"],
deps = ["@org_uber_go_mock//gomock"],
)
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,11 @@ load("@rules_go//go:def.bzl", "go_library")
go_library(
name = "mysql",
srcs = ["counter.go"],
importpath = "github.com/uber/submitqueue/submitqueue/extension/counter/mysql",
importpath = "github.com/uber/submitqueue/extension/counter/mysql",
visibility = ["//visibility:public"],
deps = [
"//core/metrics",
"//submitqueue/extension/counter",
"//extension/counter",
"@com_github_uber_go_tally_v4//:tally",
],
)
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ import (

"github.com/uber-go/tally/v4"
"github.com/uber/submitqueue/core/metrics"
"github.com/uber/submitqueue/submitqueue/extension/counter"
"github.com/uber/submitqueue/extension/counter"
)

type mysqlCounter struct {
Expand Down
Original file line number Diff line number Diff line change
@@ -1,23 +1,23 @@
load("@rules_go//go:def.bzl", "go_library", "go_test")

go_library(
name = "queue",
name = "messagequeue",
srcs = [
"delivery.go",
"publisher.go",
"queue.go",
"subscriber.go",
"subscription_config.go",
],
importpath = "github.com/uber/submitqueue/extension/queue",
importpath = "github.com/uber/submitqueue/extension/messagequeue",
visibility = ["//visibility:public"],
deps = ["//entity/queue"],
deps = ["//entity/messagequeue"],
)

go_test(
name = "queue_test",
name = "messagequeue_test",
srcs = ["subscription_config_test.go"],
embed = [":queue"],
embed = [":messagequeue"],
deps = [
"@com_github_stretchr_testify//assert",
"@com_github_stretchr_testify//require",
Expand Down
File renamed without changes.
Loading
Loading