-
Notifications
You must be signed in to change notification settings - Fork 627
[MISC] Decommission prompt-service, old tools, SDK1 prompt module #1978
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
Changes from all commits
180297b
2e1bc54
7bdff5a
4774926
fd6e3e7
c3c729d
de78b7a
c1a42aa
a7a035a
0619756
093a6b4
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
@@ -10,13 +10,11 @@ on: | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| service_name: | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| description: "Tool to build" | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| required: true | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| default: "tool-structure" # Provide a default value | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| default: "tool-sidecar" # Provide a default value | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| type: choice | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| options: # Define available options | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| - tool-classifier | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| - tool-structure | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| - tool-text-extractor | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| - tool-sidecar | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| - tool-text-extractor | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| add_latest_tag: | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| description: "Also tag as 'latest'" | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| required: false | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
@@ -58,18 +56,12 @@ jobs: | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| - name: Set build configuration | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| id: build-config | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| run: | | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| if [ "${{ github.event.inputs.service_name }}" == "tool-classifier" ]; then | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| echo "context=." >> $GITHUB_OUTPUT | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| echo "dockerfile=./tools/classifier/Dockerfile" >> $GITHUB_OUTPUT | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| elif [ "${{ github.event.inputs.service_name }}" == "tool-structure" ]; then | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| if [ "${{ github.event.inputs.service_name }}" == "tool-sidecar" ]; then | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| echo "context=." >> $GITHUB_OUTPUT | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| echo "dockerfile=./tools/structure/Dockerfile" >> $GITHUB_OUTPUT | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| echo "dockerfile=docker/dockerfiles/tool-sidecar.Dockerfile" >> $GITHUB_OUTPUT | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| elif [ "${{ github.event.inputs.service_name }}" == "tool-text-extractor" ]; then | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| echo "context=." >> $GITHUB_OUTPUT | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| echo "dockerfile=./tools/text_extractor/Dockerfile" >> $GITHUB_OUTPUT | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| elif [ "${{ github.event.inputs.service_name }}" == "tool-sidecar" ]; then | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| echo "context=." >> $GITHUB_OUTPUT | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| echo "dockerfile=docker/dockerfiles/tool-sidecar.Dockerfile" >> $GITHUB_OUTPUT | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| fi | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
Comment on lines
+59
to
65
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
# Verify this workflow no longer embeds GitHub expression templates directly in shell conditionals.
rg -n -C2 '\$\{\{\s*github\.event\.inputs\.service_name\s*\}\}' .github/workflows/docker-tools-build-push.yaml
# Expected after fix: only appears in YAML fields (e.g., run-name/tags), not inside shell branching logic.Repository: Zipstack/unstract Length of output: 1382 🏁 Script executed: rg -n -C2 '\$\{\{\s*github\.event\.inputs\.service_name\s*\}\}' .github/workflows/docker-tools-build-push.yaml
# Expected: no occurrences inside the bash branching logic; `${{ github.event.inputs.service_name }}` should be assigned to an env var or used in non-branching YAML fields.Repository: Zipstack/unstract Length of output: 1382 Harden In Suggested patch - name: Set build configuration
id: build-config
+ env:
+ SERVICE_NAME: ${{ github.event.inputs.service_name }}
run: |
- if [ "${{ github.event.inputs.service_name }}" == "tool-sidecar" ]; then
- echo "context=." >> $GITHUB_OUTPUT
- echo "dockerfile=docker/dockerfiles/tool-sidecar.Dockerfile" >> $GITHUB_OUTPUT
- elif [ "${{ github.event.inputs.service_name }}" == "tool-text-extractor" ]; then
- echo "context=." >> $GITHUB_OUTPUT
- echo "dockerfile=./tools/text_extractor/Dockerfile" >> $GITHUB_OUTPUT
- fi
+ case "$SERVICE_NAME" in
+ tool-sidecar)
+ echo "context=." >> "$GITHUB_OUTPUT"
+ echo "dockerfile=docker/dockerfiles/tool-sidecar.Dockerfile" >> "$GITHUB_OUTPUT"
+ ;;
+ tool-text-extractor)
+ echo "context=." >> "$GITHUB_OUTPUT"
+ echo "dockerfile=./tools/text_extractor/Dockerfile" >> "$GITHUB_OUTPUT"
+ ;;
+ *)
+ echo "Unsupported service_name: $SERVICE_NAME" >&2
+ exit 1
+ ;;
+ esac📝 Committable suggestion
Suggested change
🧰 Tools🪛 zizmor (1.25.2)[error] 59-59: code injection via template expansion (template-injection): may expand into attacker-controllable code (template-injection) [error] 62-62: code injection via template expansion (template-injection): may expand into attacker-controllable code (template-injection) 🤖 Prompt for AI Agents |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| # Determine tags based on inputs | ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -101,7 +101,6 @@ jobs: | |
| backend, | ||
| frontend, | ||
| platform-service, | ||
| prompt-service, | ||
| runner, | ||
| worker-unified, | ||
| x2text-service, | ||
|
|
@@ -225,7 +224,7 @@ jobs: | |
| id: summary | ||
| run: | | ||
| # Initialize variables | ||
| TOTAL_SERVICES=7 | ||
| TOTAL_SERVICES=6 | ||
| OVERALL_RESULT='${{ needs.build-and-push.result }}' | ||
| SUCCESS_COUNT=0 | ||
| FAILED_COUNT=0 | ||
|
|
@@ -316,7 +315,7 @@ jobs: | |
| echo "|---------|--------|" >> $GITHUB_STEP_SUMMARY | ||
|
|
||
| # Define services in order | ||
| for service in backend frontend platform-service prompt-service runner worker-unified x2text-service; do | ||
| for service in backend frontend platform-service runner worker-unified x2text-service; do | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If we are removing the tools, we should remove the runner as well |
||
| if [ -f "build-status/${service}.json" ]; then | ||
| STATUS=$(jq -r '.status' "build-status/${service}.json") | ||
| if [ "$STATUS" = "success" ]; then | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -20,36 +20,21 @@ services: | |
| build: | ||
| dockerfile: docker/dockerfiles/tool-sidecar.Dockerfile | ||
| context: .. | ||
| tool-structure: | ||
| image: unstract/tool-structure:${VERSION} | ||
| build: | ||
| dockerfile: tools/structure/Dockerfile | ||
| context: .. | ||
| tool-text_extractor: | ||
| image: unstract/tool-text_extractor:${VERSION} | ||
| build: | ||
| dockerfile: tools/text_extractor/Dockerfile | ||
| context: .. | ||
| tool-classifier: | ||
| image: unstract/tool-classifier:${VERSION} | ||
| build: | ||
| dockerfile: tools/classifier/Dockerfile | ||
| context: .. | ||
| platform-service: | ||
| image: unstract/platform-service:${VERSION} | ||
| build: | ||
| dockerfile: docker/dockerfiles/platform.Dockerfile | ||
| context: .. | ||
| prompt-service: | ||
| image: unstract/prompt-service:${VERSION} | ||
| build: | ||
| dockerfile: docker/dockerfiles/prompt.Dockerfile | ||
| context: .. | ||
| x2text-service: | ||
| image: unstract/x2text-service:${VERSION} | ||
| build: | ||
| dockerfile: docker/dockerfiles/x2text.Dockerfile | ||
| context: .. | ||
| tool-text_extractor: | ||
| image: unstract/tool-text_extractor:${VERSION} | ||
| build: | ||
| dockerfile: tools/text_extractor/Dockerfile | ||
| context: .. | ||
|
Comment on lines
+33
to
+37
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
# Inventory both naming variants to quantify blast radius before renaming.
rg -n -C2 'tool-text[-_]extractor|tool-text[_-]extractor|tool-text_extractor|tool-text-extractor'
# Expected: pick one canonical form and eliminate the other across workflows/compose/scripts/docs.Repository: Zipstack/unstract Length of output: 3034 🏁 Script executed: #!/bin/bash
set -euo pipefail
# Find all occurrences and show them grouped by file
rg -n --hidden --no-ignore-vcs 'tool-text[-_]extractor|tool-text_extractor' . \
| sed -E 's/^(.*):([0-9]+):(.*)$/\1:\2:\3/' \
| awk -F: '{file=$1":"$2; print}' >/dev/null
# Full match listing (limited context per match)
rg -n -C1 --hidden --no-ignore-vcs 'tool-text_extractor|tool-text-extractor' .
echo "---- public_tools.json image_name/image_url entries ----"
rg -n -C2 'tool-text[_-]extractor' unstract/tool-registry/tool_registry_config/public_tools.json
echo "---- docker-tools-build-push workflow ----"
rg -n -C2 'tool-text[_-]extractor' .github/workflows/docker-tools-build-push.yaml
echo "---- docker compose build yaml ----"
rg -n -C2 'tool-text[_-]extractor' docker/docker-compose.build.yaml
echo "---- run-platform.sh ignore_services / service wiring ----"
rg -n -C2 'tool-text[_-]extractor' run-platform.shRepository: Zipstack/unstract Length of output: 6334 🏁 Script executed: #!/bin/bash
set -euo pipefail
# Look for image/tag construction in the workflow
rg -n --hidden --no-ignore-vcs 'unstract/tool-[^"\s]+' .github/workflows/docker-tools-build-push.yaml
# Look for any usage of underscore variant outside compose
rg -n --hidden --no-ignore-vcs 'tool-text_extractor' . \
| head -n 50
# Look for any usage of hyphen variant outside the workflow list
rg -n --hidden --no-ignore-vcs 'tool-text-extractor' . \
| head -n 50
# Show the relevant portion of the workflow where image name is set (keep output small)
python3 - <<'PY'
import itertools, pathlib
p = pathlib.Path(".github/workflows/docker-tools-build-push.yaml")
lines = p.read_text().splitlines()
for i,l in enumerate(lines,1):
if "service_name" in l or "image" in l or "docker build" in l or "DOCKER" in l:
pass
# Print a window around likely blocks
keys = ["service_name", "image", "dockerfile=", "build-push", "docker/build-push-action", "repository", "tags"]
idxs=[]
for i,l in enumerate(lines,1):
if any(k in l for k in keys):
idxs.append(i)
if not idxs:
idxs=[1]
start=max(1,min(idxs)-5)
end=min(len(lines),max(idxs)+30)
for i in range(start,end+1):
print(f"{i:4d}:{lines[i-1]}")
PYRepository: Zipstack/unstract Length of output: 43 🏁 Script executed: #!/bin/bash
set -euo pipefail
# 1) Find where workflow sets service_name -> dockerfile/context (avoid exit 1 on no matches)
rg -n --hidden --no-ignore-vcs 'tool-text_extractor|tool-text-extractor|service_name|dockerfile|image|DOCKER' .github/workflows/docker-tools-build-push.yaml || true
echo "---- workflow full (first 120 lines) ----"
sed -n '1,120p' .github/workflows/docker-tools-build-push.yaml
echo "---- workflow remainder (next 120 lines, if any) ----"
sed -n '121,260p' .github/workflows/docker-tools-build-push.yaml || true
# 2) Check for underscore variant service names in other CI/scripts
echo "---- repo occurrences of tool-text_extractor (service naming) ----"
rg -n --hidden --no-ignore-vcs 'tool-text_extractor' . || true
echo "---- repo occurrences of tool-text-extractor (service naming) ----"
rg -n --hidden --no-ignore-vcs 'tool-text-extractor' . || trueRepository: Zipstack/unstract Length of output: 5478 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "---- docker/docker-compose.build.yaml (lines 1-80) ----"
sed -n '1,80p' docker/docker-compose.build.yaml
echo "---- run-platform.sh (lines 300-380) ----"
sed -n '300,380p' run-platform.sh
echo "---- run-platform.sh: find where ignore_services are applied (small neighborhood) ----"
rg -n --no-ignore-vcs 'ignore_services=' run-platform.sh
rg -n --no-ignore-vcs 'ignore_services' run-platform.sh | head -n 20Repository: Zipstack/unstract Length of output: 3582 🏁 Script executed: #!/bin/bash
set -euo pipefail
sed -n '160,230p' run-platform.shRepository: Zipstack/unstract Length of output: 3384 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "---- Search compose.yaml for tool-text-extractor / tool-text_extractor ----"
rg -n --hidden --no-ignore-vcs 'tool-text-extractor|tool-text_extractor' docker/docker-compose*.yaml
echo "---- Search tool registry config (public_tools.json already saw hyphen) for underscore variant ----"
rg -n --hidden --no-ignore-vcs 'tool-text_extractor' unstract/tool-registry || true
echo "---- Search for underscore/hyphen image repo usage in general (limit output) ----"
rg -n --hidden --no-ignore-vcs 'unstract/tool-text_extractor|unstract/tool-text-extractor' . | head -n 50Repository: Zipstack/unstract Length of output: 811 🏁 Script executed: #!/bin/bash
set -euo pipefail
echo "---- docker/docker-compose.yaml tool image references ----"
rg -n --hidden --no-ignore-vcs 'tool-text-extractor|tool-text_extractor' docker/docker-compose.yaml docker/docker-compose*.yaml
echo "---- any other compose files beyond docker/docker-compose*.yaml ----"
fd -e yaml .github docker | rg -n --hidden --no-ignore-vcs 'tool-text-extractor|tool-text_extractor' || trueRepository: Zipstack/unstract Length of output: 335 🏁 Script executed: #!/bin/bash
set -euo pipefail
# Check whether runtime references the public tool registry (likely tool image names)
rg -n --hidden --no-ignore-vcs 'public_tools\.json|tool_registry_config|tool registry|tool-registry|public_tools' . \
| head -n 80
# Check for any other references to the image repo name patterns outside the compose build file
rg -n --hidden --no-ignore-vcs 'unstract/tool-(text[-_]extractor)' . \
| head -n 80
# Check docker-compose.yaml for any service wiring that might reference tool image names indirectly
rg -n --hidden --no-ignore-vcs 'tool-(text[-_]extractor)|text_extractor|text-extractor|adapter|tool.*image|image_name|image_url' docker/docker-compose.yaml \
|| trueRepository: Zipstack/unstract Length of output: 8928 🏁 Script executed: rg -n --hidden --no-ignore-vcs 'tool-text[-_]extractor|unstract/tool-text[-_]extractor' docker docker/scripts .github/workflows unstract/tool-registryRepository: Zipstack/unstract Length of output: 684 🏁 Script executed: #!/bin/bash
set -euo pipefail
rg -n --hidden --no-ignore-vcs 'tool-text_extractor' . && trueRepository: Zipstack/unstract Length of output: 340 🏁 Script executed: #!/bin/bash
set -euo pipefail
# Where tool registry fields image_name/image_url are used to run/pull containers
rg -n --hidden --no-ignore-vcs 'image_name|image_url' unstract backend workers docker | head -n 120Repository: Zipstack/unstract Length of output: 13915 🏁 Script executed: #!/bin/bash
set -euo pipefail
# Check whether env directories exist for both delimiters (what run-platform.sh expects)
ls -ld tool-text_extractor tool-text-extractor 2>/dev/null || true
find . -maxdepth 1 -type d -name 'tool-text*' -print
# Also verify whether compose service keys map to actual env sample.env paths
for d in tool-text_extractor tool-text-extractor; do
if [ -d "$d" ]; then
echo "---- $d contents ----"
ls -la "$d" | head -n 50
[ -f "$d/sample.env" ] && echo "has sample.env" || echo "missing sample.env"
fi
doneRepository: Zipstack/unstract Length of output: 43 🏁 Script executed: rg -n --hidden --no-ignore-vcs 'tool-text[-_]extractor|unstract/tool-text[-_]extractor' docker .github/workflows unstract/tool-registryRepository: Zipstack/unstract Length of output: 684 🏁 Script executed: #!/bin/bash
set -euo pipefail
rg -n --no-ignore-vcs 'build_services\(\)|build_services\s*\{' run-platform.sh
sed -n '230,320p' run-platform.shRepository: Zipstack/unstract Length of output: 4093 Fix tool-text-extractor image repository name mismatch (
🤖 Prompt for AI Agents |
||
| # Unified worker image (replaces all individual worker images) | ||
| worker-unified: | ||
| image: unstract/worker-unified:${VERSION} | ||
|
|
||
This file was deleted.
This file was deleted.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@harini-venkataraman tool sidecar can also be removed right? Essentially this workflow file itself