[test_operator] Add stackviz integration for Tempest test visualization#3642
[test_operator] Add stackviz integration for Tempest test visualization#3642sauragar wants to merge 16 commits intoopenstack-k8s-operators:mainfrom
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Merge Failed. This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset. |
|
Merge Failed. This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset. |
|
Merge Failed. This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset. |
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/9eebbe61675c4dcc9d088d2c2d5ea48d ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 37m 18s |
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/8f1eb8e960fb4d9e8ca4f4a98fc80cda ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 19m 50s |
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/11f7659c426842b28edff49ed1d8f0cf ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 11m 41s |
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/75bb82277de440b3ac6855780f9c6fa9 ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 09m 12s |
sauragar
force-pushed
the
OSPRH-12077
branch
3 times, most recently
from
4000706 to
a8b3e36
Compare
|
PR Summary: Fix Stackviz Report Generation for Tempest Tests Problem Statement The stackviz integration for visualizing Tempest test results was not generating reports in Zuul CI runs. Investigation
Solution Overview This PR fixes the stackviz integration through 13 commits that address all identified issues and improve documentation: Core Fixes (5 commits)
Cleanup & Documentation (8 commits)
Foundation (3 commits)
Configuration Variables Four new variables control stackviz behavior (all documented in roles/test_operator/README.md):
Files Modified Core functionality:
Documentation:
Files Deleted
Testing & Verification ✅ Verified in Zuul CI:
Expected Output After Tempest tests complete, users will find: Single test stage: Multiple test stages (workflows): Report Features The generated stackviz HTML reports include: 📊 Core Visualization:
🎨 Theme Support:
🔍 Interaction:
Benefits
Technical Highlights Python Script Improvements:
Ansible Role Improvements:
Related Issues
|
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/1ad35fe4e1874a5d995b1b081965dc9a ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 18m 11s |
Add automatic stackviz report generation for Tempest tests with comprehensive dependency checking and optional auto-installation. New files: - scripts/generate-stackviz-report.py: Python script to generate interactive HTML reports from subunit test results - roles/test_operator/tasks/generate-stackviz.yml: Ansible tasks for automatic stackviz generation with dependency validation - scripts/README-stackviz.md: Complete documentation for stackviz integration - test-stackviz-integration.yml: Example test playbook - STACKVIZ-INTEGRATION-SUMMARY.md: Integration overview Modified files: - roles/test_operator/tasks/run-test-operator-job.yml: Added stackviz generation after log collection for tempest tests - roles/test_operator/defaults/main.yml: Added configuration variables for stackviz (generation, debug, auto-install dependencies) Features: - Automatic decompression of tempest_results.subunit.gz files - Individual validation of python-subunit and testtools packages - Detailed error messages with platform-specific installation commands - Optional automatic dependency installation via pip3 - Interactive HTML visualization with timeline, search, and filtering - Single self-contained HTML file output at stackviz/tempest-viz.html Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Enhanced stackviz integration to handle multiple test stages/workflow steps with individual reports and summary index. Changes: - Split processing into main orchestrator and per-file processor - generate-stackviz.yml: Main orchestrator, dependency checking, index - generate-stackviz-single.yml: Process individual subunit files - Each stage gets report in its own directory (next to subunit file) - Summary index created at stackviz/index.html linking all reports - Dependency checking done once (not per file) for efficiency New features: - Automatic detection of multiple tempest_results.subunit.gz files - Individual HTML reports generated in each test stage directory - Summary index page with links to all reports - Progress tracking during multi-file processing - Configurable index creation via cifmw_test_operator_stackviz_create_index Example directory structure with workflows: tempest-tests-tempest-s00-ironic-scenario-testing/tempest-viz.html tempest-tests-tempest-s01-multi-thread-testing/tempest-viz.html stackviz/index.html (summary) Documentation: - Updated README-stackviz.md with multi-stage examples - Added MULTI-STAGE-STACKVIZ-EXAMPLE.md with detailed walkthrough - Updated STACKVIZ-INTEGRATION-SUMMARY.md with workflow diagrams Configuration: - Added cifmw_test_operator_stackviz_create_index (default: true) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Renamed stackviz task files to clarify the main/worker relationship: - generate-stackviz.yml → generate-stackviz-main.yml (orchestrator) - generate-stackviz-single.yml → generate-stackviz-worker.yml (per-file processor) This makes the architecture more obvious: - Main: Finds files, checks dependencies, loops, creates index - Worker: Processes one file at a time (called by main in loop) Updated all references: - generate-stackviz-main.yml: Updated include_tasks to worker file - run-test-operator-job.yml: Updated to call main file - Header comments: Clarified main vs worker roles - Documentation: Updated file references in README and summary No functional changes - purely organizational for better clarity. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> Signed-off-by: sauragar <sauragar@redhat.com>
Add missing leading slash to pod_path in log collection task. The volumes are mounted at absolute paths (/mnt/logs-...), but oc cp was using a relative path (mnt/logs-...), causing log files including tempest_results.subunit.gz to not be copied. This prevented stackviz report generation from finding the required subunit files. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> Signed-off-by: sauragar <sauragar@redhat.com>
- Add debug task to list PVC contents before copying to diagnose missing files - Fix oc cp to copy directory contents (/. suffix) to prevent issues - Create separate step-X subdirectories for each workflow step to prevent files from overwriting each other and maintain proper directory structure for stackviz generation Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> Signed-off-by: sauragar <sauragar@redhat.com>
The test-operator creates tempest_results.subunit files (uncompressed), not tempest_results.subunit.gz. Update stackviz generation to: - Search for tempest_results.subunit* (both .subunit and .subunit.gz) - Detect if file is compressed by checking .gz extension - Only decompress if file is compressed - Use file directly if already uncompressed This fixes the issue where stackviz generation was skipped because it couldn't find .gz files when only uncompressed files existed. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> Signed-off-by: sauragar <sauragar@redhat.com>
Change cifmw_test_operator_stackviz_auto_install_deps from false to true to automatically install python-subunit and testtools packages when missing. This ensures stackviz report generation works out of the box without requiring manual dependency installation. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> Signed-off-by: sauragar <sauragar@redhat.com>
…llection
The PVC already contains the proper directory structure (tempest-tests-tempest/),
so copying to step-X creates an unnecessary extra directory level.
Changed from: {{ cifmw_test_operator_artifacts_basedir }}/step-{{ index }}
Changed to: {{ cifmw_test_operator_artifacts_basedir }}
The /. suffix ensures we copy directory contents directly without the wrapper.
Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Signed-off-by: sauragar <sauragar@redhat.com>
Remove the temporary debug tasks that were added to troubleshoot missing subunit files. The issue has been resolved, so these debug tasks are no longer needed. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> Signed-off-by: sauragar <sauragar@redhat.com>
Add documentation for the four stackviz-related configuration variables: - cifmw_test_operator_generate_stackviz: Enable/disable stackviz generation - cifmw_test_operator_stackviz_debug: Debug mode for stackviz - cifmw_test_operator_stackviz_auto_install_deps: Auto-install dependencies - cifmw_test_operator_stackviz_create_index: Create summary index for workflows This helps users understand the available stackviz options and their default values. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> Signed-off-by: sauragar <sauragar@redhat.com>
Remove three separate documentation files and consolidate them into a single comprehensive documentation: - scripts/README-stackviz.md - STACKVIZ-INTEGRATION-SUMMARY.md - MULTI-STAGE-STACKVIZ-EXAMPLE.md These files contained overlapping information. A consolidated version has been created for internal development reference (not committed). The main user-facing documentation remains in roles/test_operator/README.md. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> Signed-off-by: sauragar <sauragar@redhat.com>
Remove test-stackviz-integration.yml as it's no longer needed: - Stackviz functionality is production-ready and automatic - Complete documentation exists in roles/test_operator/README.md - Test file had outdated path references after recent fixes - Manual testing playbook is not required for end users Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> Signed-off-by: sauragar <sauragar@redhat.com>
Add technical terms used in stackviz documentation to the custom dictionary to fix spell-check errors: - Stackviz: The visualization tool name - subunit: Python package for test result streams - testtools: Python package for test result processing These are legitimate technical terms that should not be flagged as misspelled in the documentation. Signed-off-by: sauragar <sauragar@redhat.com>
Add theme switching functionality to stackviz HTML reports: - CSS variables for complete dark/light theme support - Toggle button in header to switch themes - Theme preference saved to localStorage - Smooth transitions between themes - Canvas elements (timeline/minimap) update with theme - All UI components (modals, tooltips, search) are theme-aware - Defaults to dark mode Users can now switch between dark and light modes based on their preference, and the choice persists across page reloads. Signed-off-by: sauragar <sauragar@redhat.com>
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/7f226ed574e54971acc665beca6f4750 ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 05m 33s |
…results Updates stackviz generation to create distinct HTML visualizations for tempest retry test results alongside original test run reports. When retries are enabled, generates tempest_retry_viz.html in addition to tempest-viz.html for easier comparison of test runs. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/3f99653ac2a64483bf6d1550aeaf7f6a ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 08m 27s |
Use cifmw_repo variable instead of playbook_dir to reference the stackviz generation script. This fixes failures in adoption jobs where playbooks are executed from rdo-jobs repository, causing playbook_dir to point to the wrong location. The cifmw_repo variable always points to the ci-framework checkout directory, ensuring the script is found regardless of where the playbook is executed from. Fixes stackviz generation in periodic-internal-adoption-* jobs. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
|
Build failed (check pipeline). Post https://softwarefactory-project.io/zuul/t/rdoproject.org/buildset/d8974b19497244cc8715428016054050 ✔️ openstack-k8s-operators-content-provider SUCCESS in 2h 07m 15s |
2 participants
Add automatic stackviz report generation for Tempest tests with comprehensive dependency checking and optional auto-installation.
New files:
Modified files:
Features:
Signed-off-by: Saurabh Agarwal sauragar@redhat.com