From 29a1c16bb3b5f5a0ff7f89ce6ecb0b9aa590d7bd Mon Sep 17 00:00:00 2001 From: Wayn_Liu Date: Fri, 4 Sep 2026 22:55:24 +0800 Subject: [PATCH] docs: add docs/ARCHITECTURE.md and explain the scar tissue in client.ts Closes #49. Split by distance, as agreed: the document carries what someone needs before opening a file, the comments carry what is only understandable at the line. docs/ARCHITECTURE.md states the premise everything follows from - there is no public Overleaf API, so the client authenticates as a logged-in browser session and calls the endpoints the web editor calls - then works through authentication, the HTML scraping and the Socket.IO fallback for the file tree, the write endpoints, the transport, and which modules can be tested without an account. In client.ts: - The file header claimed to provide "programmatic access to Overleaf's REST APIs", which is the exact misconception the document exists to correct. - extractCsrfToken and listProjects each carried numbered fallbacks with no hint that they are successive Overleaf redesigns rather than defensive habit, or that dropping the older ones breaks self-hosted instances without breaking overleaf.com - so the failure would not show up for whoever removed them. - getProjectFromSocket now says it is a hand-written Socket.IO 0.9 client, that it exists because the file tree left the meta tags with no HTTP route returning it, and that it is the first thing to suspect when the tree breaks. - uploadFile now says it overwrites rather than edits, and connects that to why push has no merge semantics and why diff exists. --- README.md | 3 + docs/ARCHITECTURE.md | 164 +++++++++++++++++++++++++++++++++++++++++++ src/client.ts | 65 +++++++++++++++-- 3 files changed, 228 insertions(+), 4 deletions(-) create mode 100644 docs/ARCHITECTURE.md diff --git a/README.md b/README.md index ee1d180..211519a 100644 --- a/README.md +++ b/README.md @@ -399,6 +399,9 @@ import { Contributions are welcome! Please open an issue or submit a pull request. +[docs/ARCHITECTURE.md](docs/ARCHITECTURE.md) explains how the client works — +there is no public Overleaf API, so it authenticates as a browser session. + ## License MIT © [Alexander Loth](https://alexloth.com) diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md new file mode 100644 index 0000000..2e604e2 --- /dev/null +++ b/docs/ARCHITECTURE.md @@ -0,0 +1,164 @@ +# Architecture + +Orientation for anyone about to open `src/client.ts` and wonder why it looks +like that. For setup and pull request mechanics, see +[CONTRIBUTING.md](../CONTRIBUTING.md). + +## The premise: there is no Overleaf API + +Overleaf publishes no REST API for the free tier. olcli is not an API client — +it **authenticates as a logged-in browser session and calls the same endpoints +the web editor's own JavaScript calls.** + +Almost every design decision follows from that one fact: + +- Authentication is a session cookie, because that is what a browser holds. +- Reading project data means parsing Overleaf's HTML, because that is where a + server-rendered page puts it. +- Write endpoints are whatever the web editor posts to. +- Nothing is versioned or documented, so **anything here can break when + Overleaf ships a redesign.** The layered fallbacks scattered through + `client.ts` are not defensive habit; each one is a redesign that already + happened. + +Requests carry a `User-Agent` of `olcli/`. olcli does not pretend to +be Chrome — the session is a real one belonging to the user running it. + +## Authentication + +Two ways in, both ending at the same place: + +| Entry point | What it does | +|---|---| +| `OverleafClient.fromSessionCookie()` | Takes a cookie the user copied from their browser | +| `OverleafClient.fromPasswordLogin()` | Submits the login form (self-hosted instances without reCAPTCHA) | + +Both then need a second credential. Overleaf requires a **CSRF token** on every +state-changing request, which is what stops another site using your cookie +against it. The token is not secret — the web editor needs it in the page to +make its own requests — so olcli fetches a page with the cookie and reads the +token out of the HTML (`extractCsrfToken`). From then on every request carries +both: + +``` +Cookie: overleaf_session2=... +X-Csrf-Token: ... +``` + +`applySetCookieHeaders()` folds any `Set-Cookie` from each response back into +the in-memory jar, so a session that rotates mid-run keeps working — the same +bookkeeping a browser does. + +Credential *storage* lives in `src/config.ts`, deliberately apart from the +client: env var, then `.olauth` in the current directory, then the global +config file. The client itself never reads any of them. + +## Reading: HTML scraping, then Socket.IO + +Project data is server-rendered into `` tags, so +`listProjects()` and `getProjectInfo()` parse the page with `cheerio`. Each has +several fallbacks tried in order, because the tag names and shapes have changed +more than once. + +The file tree is the awkward one. It used to live in `ol-project`; it no longer +does. `getProjectFromSocket()` recovers it by **speaking Socket.IO 0.9 by +hand** — handshake for a session id, `xhr-polling` for packets, decode the +frames, answer the `2::` heartbeats, and pull the tree out of the +`joinProjectResponse` event. + +This is the most fragile surface in the repository, and the least +self-evident. It is also unavoidable: that payload is where the tree is now. +Results are cached per project in `folderTreeCache` so a multi-file upload +does not repeat the whole dance for every file. + +## Writing: upload replaces, it does not edit + +| Operation | Request | +|---|---| +| Read all files | `GET /project//download/zip` | +| Write a file | `POST /project//upload?folder_id=` (multipart, field `qqfile`) | +| Delete | `DELETE /project//{doc,file,folder}/` | +| Rename an entity | `POST /project////rename` | +| Rename the project | `POST /project//rename` | +| Compile | `POST /project//compile` | + +**The most important thing to understand about writes:** typing in the Overleaf +editor sends character-level operations over the collaboration socket — an +operational transform stream that merges concurrent edits. `uploadFile()` does +not do that. It posts a whole file to the upload endpoint, exactly as if you +had dragged a same-named file into the web UI. + +So a `push` **overwrites**. It does not merge, and it cannot: there is no +three-way merge to perform, only a file replacing a file. That is why +`olcli diff` exists — previewing what a push will overwrite is the only +protection against a collaborator's edit being replaced — and why `diff` +fetches the remote fresh rather than comparing against the last pull. + +Reading the whole project is one request, not one per file: `downloadProject()` +returns the entire project as a zip. `pull`, `sync` and `diff` all use it. + +## The transport + +Everything goes through one private method, `httpRequest()`, built on +`node:http`/`node:https` rather than `fetch`. That is not preference: `fetch` +validates response headers as Latin-1 and throws on a `Content-Disposition` +carrying a non-ASCII project name, which made downloads fail for anyone with an +accented title ([#2](https://github.com/aloth/olcli/issues/2)). It also handles +redirects, timeouts, and serialising `FormData` into a multipart body. + +`--verbose` makes it log every request and response to stderr, which is the +first thing to reach for when Overleaf changes something. + +## Module map + +Which files need an Overleaf account to exercise, and which do not. This is the +main thing to know before adding a feature, because it decides where the logic +should go. + +**Pure — data in, data out. No network, no filesystem, unit-tested:** + +| Module | Responsibility | +|---|---| +| `diff.ts` | Compare two file trees; render unified diffs | +| `ignore.ts` | The three ignore layers and the `.pdf`-next-to-`.tex` rule | +| `paths.ts` | Remote path normalisation; zip-slip containment | +| `rename-plan.ts` | Plan bulk project renames before applying any | +| `prompt.ts` | Keystroke handling for the password prompt | +| `scan.ts` | Walk a local directory, applying ignore rules | + +**Talks to Overleaf:** + +| Module | Responsibility | +|---|---| +| `client.ts` | Every request. The browser-session model lives here | +| `config.ts` | Credential resolution and storage | + +**Entry points, all thin over the two above:** + +| Module | Binary | +|---|---| +| `cli.ts` | `olcli` — argument parsing and terminal output | +| `mcp.ts` | `olcli-mcp` — the same operations as MCP tools | +| `remote-helper.ts` | `git-remote-overleaf` — `gitremote-helpers(7)` protocol | +| `index.ts` | The programmatic API re-exported from the package root | + +New logic belongs in the pure column wherever it can go. That is why `scan.ts` +exists at all: `push` and `sync` each carried their own copy of the same walk +loop and had already drifted apart, and `diff` would have made a third. The +same reasoning produced `rename-plan.ts` and `diff.ts`. + +`client.ts` request *construction* can also be tested without an account, by +pointing the client at a local HTTP server that captures the outgoing request — +see `test/client.test.ts`. + +## When Overleaf breaks it + +The usual failure is a redesign moving data somewhere else. Reliable order: + +1. `olcli --verbose ` — see the actual request and response. +2. If a page parse returns nothing, fetch the page in a browser with devtools + and look for the `ol-*` meta tag. Add a fallback; keep the existing ones, + since self-hosted instances run older versions. +3. If the file tree is what broke, suspect `getProjectFromSocket()` first. +4. `olcli check` reports which credential source is in play, without printing + any secret. diff --git a/src/client.ts b/src/client.ts index 1d2c9f5..4dbd3fe 100644 --- a/src/client.ts +++ b/src/client.ts @@ -1,8 +1,18 @@ /** - * Overleaf API Client + * Overleaf client * - * Provides programmatic access to Overleaf's REST APIs for project - * management, file operations, and LaTeX compilation. + * Project management, file operations and LaTeX compilation against an + * Overleaf instance. + * + * These are not Overleaf's public APIs - there are none for the free tier. + * This client authenticates as a logged-in browser session and calls the same + * endpoints the web editor's own JavaScript calls: a session cookie plus a + * CSRF token scraped from the page, project data parsed out of `ol-*` meta + * tags, and the file tree recovered over the collaboration socket. Nothing + * here is versioned or documented by Overleaf, so the layered fallbacks below + * are not defensive habit - each one is a redesign that already happened. + * + * Read docs/ARCHITECTURE.md before changing anything in this file. */ import * as cheerio from 'cheerio'; @@ -391,6 +401,22 @@ export class OverleafClient { return new OverleafClient({ cookies: bootstrapClient.cookies, csrf: projectCsrf, baseUrl }); } + /** + * Pull the CSRF token out of a rendered Overleaf page. + * + * Overleaf requires this on every state-changing request, which is what + * stops another site from using your session cookie against it. It is not a + * secret - the web editor needs it in the page to make its own requests - so + * reading it back out of the HTML is the intended way for a session to + * obtain one. See docs/ARCHITECTURE.md. + * + * The three lookups are not belt-and-braces. Each is where the token lived + * at some point: the `ol-csrfToken` meta tag is current, the hidden form + * input is what older releases shipped, and the inline-script scrape catches + * self-hosted instances older still. Removing the later ones breaks + * self-hosted users without breaking anything on overleaf.com, so the + * failure would not show up here. + */ private static extractCsrfToken($: cheerio.CheerioAPI): string | undefined { let csrf = $('meta[name="ol-csrfToken"]').attr('content'); if (!csrf) { @@ -571,7 +597,11 @@ export class OverleafClient { const html = response.body as string; const $ = cheerio.load(html); - // Try new Overleaf structure first (PR #82) + // There is no projects API; the list is server-rendered into a meta tag, + // so this parses Overleaf's own HTML. The three methods below are three + // successive shapes that tag has had - newest first, oldest last. A + // self-hosted instance can be running any of them, which is why the older + // ones stay. See docs/ARCHITECTURE.md. let projectsData: any[] = []; // Method 1: ol-prefetchedProjectsBlob (newest Overleaf) @@ -757,6 +787,17 @@ export class OverleafClient { * Fetch the full project object via the collaboration socket. * Returns the `project` field of the joinProjectResponse, which contains * the rootFolder tree and other metadata that used to live in ol-project. + * + * This is a hand-written Socket.IO 0.9 client: handshake for a session id, + * `xhr-polling` for packets, decode the frames, answer the `2::` heartbeats, + * disconnect with `0::`. No library - the protocol is old enough that + * depending on one to speak it would cost more than the forty lines below. + * + * It is the most fragile surface in the repository and the least obvious, + * because it reimplements an undocumented internal protocol rather than + * calling an endpoint. It exists because the file tree left the meta tags + * and this payload is where it went; there is no HTTP route that returns it. + * When the tree is what broke, suspect this method first. */ private async getProjectFromSocket(projectId: string): Promise { let sid: string | null = null; @@ -1631,6 +1672,22 @@ export class OverleafClient { * If folderTree is provided and fileName contains a path (e.g. 'figures/img.png'), * the file will be uploaded into the correct subfolder, creating it if needed. */ + /** + * Upload a file, replacing any file of the same name. + * + * This **overwrites**; it does not edit. Typing in the Overleaf editor sends + * character-level operations over the collaboration socket, and those merge + * with concurrent edits. This posts a whole file to the upload endpoint - + * the same thing as dragging a same-named file into the web UI - so whatever + * was there is gone. + * + * That is why `push` has no merge semantics and cannot grow any: there is no + * three-way merge available, only a file replacing a file. It is also why + * `olcli diff` exists, and why it fetches the remote fresh rather than + * comparing against the last pull - previewing what a push will overwrite is + * the only thing standing between a collaborator's edit and its replacement. + * See docs/ARCHITECTURE.md. + */ async uploadFile( projectId: string, folderId: string | null,