Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions docs/components/security.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ Security components wrap popular open-source tools for subdomain discovery, DNS
### Subfinder

<Info>
[GitHub](https://github.com/projectdiscovery/subfinder) · Docker: `projectdiscovery/subfinder`
[GitHub](https://github.com/projectdiscovery/subfinder) · Docker: `ghcr.io/shipsecai/subfinder`
</Info>

Discovers subdomains using passive sources.
Expand All @@ -35,7 +35,7 @@ Discovers subdomains using passive sources.
### Amass

<Info>
[GitHub](https://github.com/owasp-amass/amass) · Docker: `owaspamass/amass`
[GitHub](https://github.com/owasp-amass/amass) · Docker: `ghcr.io/shipsecai/amass`
</Info>

Active and passive subdomain enumeration.
Expand Down Expand Up @@ -73,7 +73,7 @@ High-performance DNS bruteforcing and resolution. This is a combined image that
### DNSX

<Info>
[GitHub](https://github.com/projectdiscovery/dnsx) · Docker: `projectdiscovery/dnsx`
[GitHub](https://github.com/projectdiscovery/dnsx) · Docker: `ghcr.io/shipsecai/dnsx`
</Info>

Resolves DNS records with support for multiple record types and custom resolvers.
Expand Down Expand Up @@ -105,7 +105,7 @@ Resolves DNS records with support for multiple record types and custom resolvers
### httpx

<Info>
[GitHub](https://github.com/projectdiscovery/httpx) · Docker: `projectdiscovery/httpx`
[GitHub](https://github.com/projectdiscovery/httpx) · Docker: `ghcr.io/shipsecai/httpx`
</Info>

Probes hosts for live HTTP services and captures response metadata.
Expand Down Expand Up @@ -137,7 +137,7 @@ Probes hosts for live HTTP services and captures response metadata.
### Naabu

<Info>
[GitHub](https://github.com/projectdiscovery/naabu) · Docker: `projectdiscovery/naabu`
[GitHub](https://github.com/projectdiscovery/naabu) · Docker: `ghcr.io/shipsecai/naabu`
</Info>

Fast active port scanning using SYN/CONNECT probes.
Expand Down Expand Up @@ -196,7 +196,7 @@ Template-based vulnerability scanning. This is nuclei custom image with nuclei-t
### TruffleHog

<Info>
[GitHub](https://github.com/trufflesecurity/trufflehog) · Docker: `trufflesecurity/trufflehog`
[GitHub](https://github.com/trufflesecurity/trufflehog) · Docker: `ghcr.io/shipsecai/trufflehog`
</Info>

Scans for leaked credentials across repositories, filesystems, and cloud storage.
Expand Down Expand Up @@ -227,7 +227,7 @@ Scans for leaked credentials across repositories, filesystems, and cloud storage
### Prowler Scan

<Info>
[GitHub](https://github.com/prowler-cloud/prowler) · Docker: `prowlercloud/prowler`
[GitHub](https://github.com/prowler-cloud/prowler) · Docker: `ghcr.io/shipsecai/prowler`
</Info>

Cloud (AWS, Azure, GCP) security posture management. Best practices auditing.
Expand Down Expand Up @@ -262,7 +262,7 @@ Scans Supabase instances for misconfigurations.
### Notify

<Info>
[GitHub](https://github.com/projectdiscovery/notify) · Docker: `projectdiscovery/notify`
[GitHub](https://github.com/projectdiscovery/notify) · Docker: `ghcr.io/shipsecai/notify`
</Info>

Sends alerts to Slack, Discord, Telegram, or email.
Expand Down
2 changes: 1 addition & 1 deletion docs/development/component-development.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -733,7 +733,7 @@ export default defineComponent({
category: 'security',
runner: {
kind: 'docker',
image: 'projectdiscovery/dnsx:latest',
image: 'ghcr.io/shipsecai/dnsx:latest',
entrypoint: 'sh',
command: ['-c', 'dnsx "$@"', '--'],
network: 'bridge',
Expand Down
2 changes: 1 addition & 1 deletion scratch/opencode-mcp-test/run_test.sh
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ docker run --rm \
--network host \
-v "$DIR:/workspace" \
-e OPENROUTER_API_KEY="$OPENROUTER_API_KEY" \
ghcr.io/anomalyco/opencode \
ghcr.io/shipsecai/opencode:1.1.53 \
run --log-level INFO "$(cat prompt.txt)"

# Kill MCP server
Expand Down
21 changes: 19 additions & 2 deletions worker/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,11 +3,13 @@
Node.js component execution engine with Temporal.io integration for running security workflows in isolated environments.

## Prerequisites

- Bun latest (see root `README.md` for install instructions)
- Infrastructure services running (`just dev` from repo root)
- Docker for containerized component execution

## Development Commands

```bash
# Install workspace dependencies (run once from repo root)
bun install
Expand All @@ -26,6 +28,7 @@ bun run test
## Architecture Overview

### Core Technologies

- **Node.js** with TypeScript for component execution
- **Temporal.io** for workflow orchestration and activities
- **Docker** for isolated component execution
Expand Down Expand Up @@ -56,26 +59,31 @@ const runComponentActivity = async (componentId, input, context) => {
The worker provides concrete implementations of SDK interfaces:

#### File Storage Adapter

- **MinIO Integration**: S3-compatible object storage
- **PostgreSQL Metadata**: File metadata and organization
- **Artifact Management**: Component outputs and execution results

#### Secrets Adapter

- **HashiCorp Vault**: Enterprise-grade secret management
- **AES-256 Encryption**: Secure secret storage
- **Version Control**: Multiple secret versions with rollback

#### Trace Adapter

- **Event Streaming**: Kafka-based event publishing
- **Redis Transport**: Real-time event delivery
- **Timeline Generation**: Sequential event numbering

#### Logging Adapters

- **Kafka Log Transport**: Structured log streaming
- **Loki Integration**: Log aggregation and querying
- **PostgreSQL Persistence**: Log metadata and indexing

#### Terminal Adapter

- **Redis Streams**: Real-time terminal output streaming
- **Base64 Encoding**: Efficient binary data transport
- **Monotonic Timestamps**: Precise chronological ordering
Expand All @@ -85,12 +93,14 @@ The worker provides concrete implementations of SDK interfaces:
### Component Categories

#### Core Components

- **file-loader**: File upload and content extraction
- **trigger-manual**: Manual workflow execution trigger
- **text-block**: Markdown documentation and notes
- **text-joiner**: Text concatenation and formatting

#### Security Components

- **subfinder**: Subdomain discovery
- **dnsx**: DNS resolution and enumeration
- **nmap**: Network scanning and discovery
Expand All @@ -106,7 +116,7 @@ const definition: ComponentDefinition = {
id: 'security.subfinder',
label: 'Subfinder',
category: 'discovery',
runner: { kind: 'docker', image: 'projectdiscovery/subfinder' },
runner: { kind: 'docker', image: 'ghcr.io/shipsecai/subfinder' },
inputSchema: z.object({
domain: z.string(),
timeout: z.number().default(30),
Expand All @@ -122,17 +132,20 @@ componentRegistry.register(definition);
## Temporal Integration

### Activities

- **runComponentActivity**: Execute individual components
- **setRunMetadataActivity**: Store workflow execution metadata
- **finalizeRunActivity**: Complete workflow and cleanup resources

### Workflows

- **Workflow Orchestration**: Topological sorting and dependency resolution
- **Join Strategies**: Handle multiple parent dependencies (all, any, first)
- **Error Handling**: Retry policies and graceful degradation
- **Heartbeating**: Long-running activity support

### Worker Configuration

```typescript
const worker = await Worker.create({
connection,
Expand All @@ -148,6 +161,7 @@ const worker = await Worker.create({
```

## Project Structure

```
src/
├── components/ # Component implementations
Expand All @@ -169,12 +183,14 @@ src/
## Container Execution

### Docker Integration

- **Isolated Execution**: Components run in isolated Docker containers
- **Resource Limits**: CPU and memory constraints enforced
- **Network Isolation**: Bridge network configuration
- **Volume Management**: Isolated storage volumes for file operations

### Terminal Capture

- **PTY Allocation**: Pseudo-terminal for interactive tools
- **Stream Multiplexing**: stdout, stderr, and PTY stream capture
- **Chunk Encoding**: Base64 encoding for efficient transport
Expand All @@ -189,6 +205,7 @@ src/
5. **Validation**: Check types with `bun run typecheck`

## Where To Read More

- **[Architecture Overview](../docs/architecture.md)** - Complete system design and component execution
- **[Component Development](../docs/component-development.md)** - Building security components
- **[Component SDK](../packages/component-sdk)** - Framework-agnostic component interfaces
- **[Component SDK](../packages/component-sdk)** - Framework-agnostic component interfaces
2 changes: 1 addition & 1 deletion worker/src/components/ai/__tests__/opencode.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,7 @@ describe('shipsec.opencode.agent', () => {

expect(runSpy).toHaveBeenCalled();
const runnerCall = runSpy.mock.calls[0][0];
expect(runnerCall.image).toBe('ghcr.io/anomalyco/opencode');
expect(runnerCall.image).toBe('ghcr.io/shipsecai/opencode:1.1.53');
expect(runnerCall.network).toBe('host');
expect(runnerCall.env.OPENAI_API_KEY).toBe('sk-test');
});
Expand Down
2 changes: 1 addition & 1 deletion worker/src/components/ai/opencode.ts
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,7 @@ const definition = defineComponent({
category: 'ai',
runner: {
kind: 'docker',
image: 'ghcr.io/anomalyco/opencode',
image: 'ghcr.io/shipsecai/opencode:1.1.53',
entrypoint: 'opencode', // We will override this in execution
network: 'host' as const, // Required to access localhost gateway
command: ['help'],
Expand Down
4 changes: 2 additions & 2 deletions worker/src/components/security/__tests__/amass.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -107,13 +107,13 @@ describe.skip('amass component', () => {
expect(result.rawOutput).toBe(rawOutput);
});

it('should configure docker runner for owaspamass/amass image', () => {
it('should configure docker runner for ghcr.io/shipsecai/amass image', () => {
const component = componentRegistry.get<AmassInput, AmassOutput>('shipsec.amass.enum');
if (!component) throw new Error('Component not registered');

expect(component.runner.kind).toBe('docker');
if (component.runner.kind === 'docker') {
expect(component.runner.image).toBe('owaspamass/amass:v5.0.1');
expect(component.runner.image).toBe('ghcr.io/shipsecai/amass:v5.0.1');
expect(component.runner.entrypoint).toBe('sh');
expect(component.runner.command).toBeInstanceOf(Array);
}
Expand Down
2 changes: 1 addition & 1 deletion worker/src/components/security/__tests__/dnsx.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -195,7 +195,7 @@ describe.skip('dnsx component', () => {

expect(component.runner.kind).toBe('docker');
if (component.runner.kind === 'docker') {
expect(component.runner.image).toBe('projectdiscovery/dnsx:v1.2.2');
expect(component.runner.image).toBe('ghcr.io/shipsecai/dnsx:v1.2.2');
expect(component.runner.entrypoint).toBe('sh');
}
});
Expand Down
2 changes: 1 addition & 1 deletion worker/src/components/security/__tests__/naabu.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -105,7 +105,7 @@ describe('naabu component', () => {

expect(component.runner.kind).toBe('docker');
if (component.runner.kind === 'docker') {
expect(component.runner.image).toBe('projectdiscovery/naabu:v2.3.7');
expect(component.runner.image).toBe('ghcr.io/shipsecai/naabu:v2.3.7');
expect(component.runner.entrypoint).toBe('sh');
expect(component.runner.command).toBeInstanceOf(Array);
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ describeNotify('Notify component registration', () => {
expect(component!.ui?.slug).toBe('notify');

if (component!.runner.kind === 'docker') {
expect(component!.runner.image).toContain('projectdiscovery/notify');
expect(component!.runner.image).toContain('ghcr.io/shipsecai/notify');
expect(component!.runner.entrypoint).toBe('sh');
} else {
throw new Error('Expected docker runner for notify component');
Expand Down
2 changes: 1 addition & 1 deletion worker/src/components/security/__tests__/subfinder.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,7 @@ describe.skip('subfinder component', () => {

expect(component.runner.kind).toBe('docker');
if (component.runner.kind === 'docker') {
expect(component.runner.image).toBe('projectdiscovery/subfinder:v2.12.0');
expect(component.runner.image).toBe('ghcr.io/shipsecai/subfinder:v2.12.0');
}
});
});
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ describe('trufflehog component', () => {

expect(component.runner.kind).toBe('docker');
if (component.runner.kind === 'docker') {
expect(component.runner.image).toBe('trufflesecurity/trufflehog:v3.92.1');
expect(component.runner.image).toBe('ghcr.io/shipsecai/trufflehog:v3.93.1');
}
});

Expand Down
2 changes: 1 addition & 1 deletion worker/src/components/security/amass.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ import {
} from '@shipsec/component-sdk';
import { IsolatedContainerVolume } from '../../utils/isolated-volume';

const AMASS_IMAGE = 'owaspamass/amass:v5.0.1';
const AMASS_IMAGE = 'ghcr.io/shipsecai/amass:v5.0.1';
const AMASS_TIMEOUT_SECONDS = (() => {
const raw = process.env.AMASS_TIMEOUT_SECONDS;
const parsed = raw ? Number.parseInt(raw, 10) : NaN;
Expand Down
2 changes: 1 addition & 1 deletion worker/src/components/security/dnsx.ts
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ const recordTypeEnum = z.enum([

const outputModeEnum = z.enum(['silent', 'json']);

const DNSX_IMAGE = 'projectdiscovery/dnsx:v1.2.2';
const DNSX_IMAGE = 'ghcr.io/shipsecai/dnsx:v1.2.2';
const DNSX_TIMEOUT_SECONDS = 180;
const INPUT_MOUNT_NAME = 'inputs';
const CONTAINER_INPUT_DIR = `/${INPUT_MOUNT_NAME}`;
Expand Down
2 changes: 1 addition & 1 deletion worker/src/components/security/httpx.ts
Original file line number Diff line number Diff line change
Expand Up @@ -202,7 +202,7 @@ const definition = defineComponent({
category: 'security',
runner: {
kind: 'docker',
image: 'projectdiscovery/httpx:v1.7.4',
image: 'ghcr.io/shipsecai/httpx:v1.7.4',
entrypoint: 'httpx',
network: 'bridge',
timeoutSeconds: dockerTimeoutSeconds,
Expand Down
2 changes: 1 addition & 1 deletion worker/src/components/security/naabu.ts
Original file line number Diff line number Diff line change
Expand Up @@ -179,7 +179,7 @@ const definition = defineComponent({
category: 'security',
runner: {
kind: 'docker',
image: 'projectdiscovery/naabu:v2.3.7',
image: 'ghcr.io/shipsecai/naabu:v2.3.7',
entrypoint: 'sh',
network: 'bridge',
timeoutSeconds: dockerTimeoutSeconds,
Expand Down
2 changes: 1 addition & 1 deletion worker/src/components/security/notify.ts
Original file line number Diff line number Diff line change
Expand Up @@ -213,7 +213,7 @@ const definition = defineComponent({
category: 'security',
runner: {
kind: 'docker',
image: 'projectdiscovery/notify:v1.0.7',
image: 'ghcr.io/shipsecai/notify:v1.0.7',
entrypoint: 'sh',
network: 'bridge',
timeoutSeconds: dockerTimeoutSeconds,
Expand Down
6 changes: 3 additions & 3 deletions worker/src/components/security/prowler-scan.ts
Original file line number Diff line number Diff line change
Expand Up @@ -399,14 +399,14 @@ const definition = defineComponent({
retryPolicy: prowlerRetryPolicy,
runner: {
kind: 'docker',
image: 'prowlercloud/prowler:5.14.2',
image: 'ghcr.io/shipsecai/prowler:5.14.2',
platform: 'linux/amd64',
command: [], // Placeholder - actual command built dynamically in execute()
},
inputs: inputSchema,
outputs: outputSchema,
parameters: parameterSchema,
docs: 'Execute Prowler inside Docker using `prowlercloud/prowler` (amd64 enforced on ARM hosts). Supports AWS account scans and the multi-cloud `prowler cloud` overview, with optional CLI flag customisation.',
docs: 'Execute Prowler inside Docker using `ghcr.io/shipsecai/prowler` (amd64 enforced on ARM hosts). Supports AWS account scans and the multi-cloud `prowler cloud` overview, with optional CLI flag customisation.',
ui: {
slug: 'prowler-scan',
version: '2.0.0',
Expand Down Expand Up @@ -566,7 +566,7 @@ const definition = defineComponent({
// Prepare a one-off runner with dynamic command and volume
const dockerRunner: DockerRunnerConfig = {
kind: 'docker',
image: 'prowlercloud/prowler:5.14.2',
image: 'ghcr.io/shipsecai/prowler:5.14.2',
platform: 'linux/amd64',
network: 'bridge',
timeoutSeconds: 900,
Expand Down
4 changes: 2 additions & 2 deletions worker/src/components/security/subfinder.ts
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ import {
} from '@shipsec/component-sdk';
import { IsolatedContainerVolume } from '../../utils/isolated-volume';

const SUBFINDER_IMAGE = 'projectdiscovery/subfinder:v2.12.0';
const SUBFINDER_IMAGE = 'ghcr.io/shipsecai/subfinder:v2.12.0';
const SUBFINDER_TIMEOUT_SECONDS = 1800; // 30 minutes
const INPUT_MOUNT_NAME = 'inputs';
const CONTAINER_INPUT_DIR = `/${INPUT_MOUNT_NAME}`;
Expand Down Expand Up @@ -280,7 +280,7 @@ const definition = defineComponent({
inputs: inputSchema,
outputs: outputSchema,
parameters: parameterSchema,
docs: 'Runs projectdiscovery/subfinder to discover subdomains for a given domain. Optionally accepts a provider config secret to enable authenticated sources.',
docs: 'Runs subfinder to discover subdomains for a given domain. Optionally accepts a provider config secret to enable authenticated sources.',
ui: {
slug: 'subfinder',
version: '1.0.0',
Expand Down
2 changes: 1 addition & 1 deletion worker/src/components/security/trufflehog.ts
Original file line number Diff line number Diff line change
Expand Up @@ -303,7 +303,7 @@ const definition = defineComponent({
category: 'security',
runner: {
kind: 'docker',
image: 'trufflesecurity/trufflehog:v3.92.1',
image: 'ghcr.io/shipsecai/trufflehog:v3.93.1',
entrypoint: 'trufflehog',
network: 'bridge',
command: [], // Will be built dynamically in execute
Expand Down