diff --git a/sources/academy/platform/scraping_with_apify_and_ai/03_docs_driven_prompting.md b/sources/academy/platform/scraping_with_apify_and_ai/03_docs_driven_prompting.md index 309b48bf03..05d58ba1e1 100644 --- a/sources/academy/platform/scraping_with_apify_and_ai/03_docs_driven_prompting.md +++ b/sources/academy/platform/scraping_with_apify_and_ai/03_docs_driven_prompting.md @@ -1,5 +1,5 @@ --- -title: Developing a scraper with docs-driven prompting +title: Using docs as a spec for AI description: Improve your Apify scraper by documenting its behavior first and letting an AI agent follow the documentation as a practical spec. slug: /scraping-with-apify-and-ai/docs-driven-prompting unlisted: true @@ -13,9 +13,9 @@ We made our lives easier with an AI agent. Improving our scraper now takes way l Prompting a chat or agent is quick and straightforward, but it doesn't leave much trace of our intentions: -- If we want someone else to take over later, it'll be hard for them to figure out why we made some decisions and whether behavior is intentional or accidental. -- If we get busy with other things and return after a few months, we'll basically become that “someone else” who needs to figure out the project. After a week, we might still remember why we process prices a certain way. After a year, it's mostly fuzzy memories. -- If we want other people to use our scraper, they need simple instructions on how to run it and what to expect. +- _Hard to hand over:_ If we want someone else to take over later, it'll be hard for them to figure out why we made some decisions and whether behavior is intentional or accidental. +- _Easy to forget:_ If we get busy with other things and return after a few months, we'll basically become that “someone else” who needs to figure out the project. After a week, we might still remember why we process prices a certain way. After a year, it's mostly fuzzy memories. +- _No instructions:_ If we want other people to run or even develop our scraper, they need simple instructions on how to do it and what to expect. Traditionally, we write the documentation after finishing the software. With AI, we can describe how the program should work before it's done, point the agent to that spec, and ask it to make it real. diff --git a/sources/academy/platform/scraping_with_apify_and_ai/04_tests_driven_prompting.md b/sources/academy/platform/scraping_with_apify_and_ai/04_tests_driven_prompting.md index c21390efd0..f1a7d472cc 100644 --- a/sources/academy/platform/scraping_with_apify_and_ai/04_tests_driven_prompting.md +++ b/sources/academy/platform/scraping_with_apify_and_ai/04_tests_driven_prompting.md @@ -1,33 +1,159 @@ --- -title: Tests driven prompting -description: TBD +title: Using examples as a spec for AI +description: Improve your Apify scraper by adding automated tests with real-world examples that an AI agent can use as a spec. slug: /scraping-with-apify-and-ai/tests-driven-prompting unlisted: true --- - +--- + +The README as a source of truth for the AI agent gets us far, but it has limits: + +- _Messy edge cases:_ Describing a large set of edge cases is tedious. "If this tiny detail is a certain way, process it as X, otherwise Y" for each situation is possible, but messy. +- _Bulky examples:_ Sometimes the edge case lies in the page's HTML, the text format that describes its content and structure. We'd have to say, "if you encounter exactly this HTML code, process it like this." Pasting long snippets of HTML into a README isn't great. +- _No safety net:_ After each change, we have to trust that the agent didn't break what already worked. We can prompt it to "go through the whole README and verify all the behavior," but that's slow and unreliable. +- _No reference point:_ Our scraper assumes a certain page structure, but that structure can change over time. The README says what data we want, not what the page looked like when everything still worked. + +There's a better way. We can save real-world examples of the pages we scrape, along with the data we expect to get from them. A program can then load each example, process it as if it were scraping the live page, and compare the result with our expectations. + +Software developers do this all the time, so each piece has a name. The saved examples are _fixtures_ or _snapshots_. The expected results are _expectations_. The setup that runs all the checks is a _test suite_. + +Running these _tests_ is much faster than checking everything by hand. Both we and the AI agent can run them anytime. When the website changes, we can replace the saved example with a fresh one and let the AI agent fix the code. + +## Setting up a test suite + +Let's start by adding a new section to the README: + +```md +## Testing + +- The `tests` directory contains real-world HTML snapshots of each page type we scrape. +- Each snapshot also has a JSON file of the same name with the data we expect to get out of it. +- Run `npm test` to run all automated tests. +- Use red-green test-driven development. +``` + +_JSON_ is a text format for storing structured data. A JSON file can represent the same rows and fields you see in the Apify output table, but in a form that programs can read and compare. You won't need to write these files yourself. The AI agent will create them, and you'll only check whether their contents look right. + +_Red-green test-driven development_, often shortened to TDD, means that whenever the AI agent modifies our project, it starts with the expectations. It runs the tests and watches them fail, which proves that they actually check something. Only then does it change the code to make them pass. This technique makes development more reliable. + +Now let's send this prompt to the AI agent: + +```text +Read the Testing section in README and set up a test suite +covering the behavior we already have. +``` + +When the AI agent gets to work, you should see it create a new directory called `tests`. It'll probably use `curl`, a command-line program for downloading files, to create an HTML snapshot of the Sales listing. + +When it's done, you should see several new files inside `tests`. Most likely, `sales.html` will contain the downloaded page, `sales.json` will contain the expected data, and a test file will run the checks. + +When we run `npm test`, all the tests should pass. This is an example output of the command: + +```text +npm notice run node --test tests/*.test.js +✔ parseMinPrice handles sale price text (0.532625ms) +✔ parseMinPrice returns null when no price is found (0.06775ms) +✔ parseSku handles inventory text (0.129875ms) +✔ parseSku returns null for empty inventory text (0.058833ms) +✔ toAbsoluteUrl resolves protocol-relative and relative URLs (0.325541ms) +✔ extractProducts matches expected output for sales (58.530583ms) +ℹ tests 6 +ℹ suites 0 +ℹ pass 6 +ℹ fail 0 +ℹ cancelled 0 +ℹ skipped 0 +ℹ todo 0 +ℹ duration_ms 165.677875 +``` + +Depending on what the agent created, your output may look different, but it should follow a similar pattern. You don't need to understand every line. The important parts are `pass 6` and `fail 0`, which confirm that all tests passed. + +## Handling product variants + +Some prices in our data are "from" values because many items in the listing represent several product variants. Let's scrape each variant as a separate product with its actual price. + +Each product in the listing links to a _product detail page_, or PDP. If we open a product URL in the browser, such as the page for the [Sony XBR-950G BRAVIA](https://warehouse-theme-metal.myshopify.com/products/sony-xbr-65x950g-65-class-64-5-diag-bravia-4k-hdr-ultra-hd-tv), we can see its vendor name, [SKU](https://en.wikipedia.org/wiki/Stock_keeping_unit), reviews, images, variants, stock availability, description, and more. + +![Product detail page](images/pdp.webp) + +Before we tell the AI agent to handle variants, let's check what we're dealing with. This will help us choose the right design and cover all possible situations. + +## Identifying and handling edge cases + +When we [browse the Sales page](https://warehouse-theme-metal.myshopify.com/collections/sales), we can see several situations to cover: + +- A product with one price and no variants: [Sony SACS9 10" Active Subwoofer](https://warehouse-theme-metal.myshopify.com/products/sony-sacs9-10-inch-active-subwoofer) +- A product with several variants, each with a different price: [Sony XBR-950G BRAVIA 4K HDR Ultra HD TV](https://warehouse-theme-metal.myshopify.com/products/sony-xbr-65x950g-65-class-64-5-diag-bravia-4k-hdr-ultra-hd-tv) +- A product with several variants, all with the same price: [JBL Flip 4 Waterproof Portable Bluetooth Speaker](https://warehouse-theme-metal.myshopify.com/products/jbl-flip-4-waterproof-portable-bluetooth-speaker?variant=17549970440243) +- Variants that represent colors, like the JBL speaker, or sizes, like the Sony TV. + +Let's save each variant as a separate product with its own price and one extra field for the variant name, such as `Green` or `55"`. We'll add a new section to the README right after _Prices handling_: + +```md +### Variants handling + +Downloads product detail pages. Instead of saving each listing item as a single product, it saves each variant as an individual product. Variants have these extra properties: + +- Variant name +- Exact variant price as `price` + +Minimum price from the listing stays as `minPrice`. Products without variants have empty variant name and `price` equal to `minPrice`. +``` + +Let's save the README and prepare a prompt for the AI agent. We'll tell it which pages to use as fixtures for each edge case. + +It's best to focus each fixture on a single situation. Our findings give us five snapshots, although some will contain the same HTML: + +```text +Read the new Variants handling section and change +the project accordingly. Use red-green TDD. Fixtures: + +one-price-no-variants.html +https://warehouse-theme-metal.myshopify.com/products/sony-sacs9-10-inch-active-subwoofer + +variants-different-prices.html +https://warehouse-theme-metal.myshopify.com/products/sony-xbr-65x950g-65-class-64-5-diag-bravia-4k-hdr-ultra-hd-tv + +variants-same-price.html +https://warehouse-theme-metal.myshopify.com/products/jbl-flip-4-waterproof-portable-bluetooth-speaker?variant=17549970440243 + +variants-colors.html +https://warehouse-theme-metal.myshopify.com/products/jbl-flip-4-waterproof-portable-bluetooth-speaker?variant=17549970440243 + +variants-sizes.html +https://warehouse-theme-metal.myshopify.com/products/sony-xbr-65x950g-65-class-64-5-diag-bravia-4k-hdr-ultra-hd-tv +``` + +After several `curl` calls and much crunching, we should see the new files in the `tests` directory, together with their JSON expectations. We can open them and eyeball whether they look correct. + +For example, `variants-colors.json` should contain seven products. Their colors should match those on the JBL speaker page. Similarly, `variants-different-prices.json` should contain the correct price for each variant, while `one-price-no-variants.json` should contain just one product. + +If everything looks right and `npm test` passes, we can be pretty sure that our scraper handles variants according to our spec. And we haven't even run it against the live website yet. Let's do that now for one final check: + +```text +apify run +``` + +In the output, we should see products with variant names and exact prices, like this: + +```text +INFO Saving product {"productName":"Sony XB-950B1 Extra Bass Wireless Headphones with App Control","productUrl":"https://warehouse-theme-metal.myshopify.com/products/sony-xb950-extra-bass-wireless-headphones-with-app-control","vendorName":"Sony","imageUrl":"https://warehouse-theme-metal.myshopify.com/cdn/shop/products/13261_147__1_2e3211f9-de49-4919-9e67-006800a5c5a0.jpg?v=1559727794","minPrice":128,"variantName":"Red","price":178,"sku":14} +``` -:::note Course under construction -This page hasn't been written yet. Come later, please! -::: +With a bit of effort, we can spot that `minPrice` is `128`, `price` is `178`, and `variantName` is `Red`. - +If the target website introduces new edge cases, all we have to do is identify them and ask the AI agent to add snapshots and expectations for them. - +If you build a stable scraper like this, it would be a shame to keep it to yourself. In the next lesson, we'll publish our scraper to Apify Store so that other people can use it and pay us for developing and maintaining it. diff --git a/sources/academy/platform/scraping_with_apify_and_ai/05_before_publishing.md b/sources/academy/platform/scraping_with_apify_and_ai/05_before_publishing.md new file mode 100644 index 0000000000..ab0828e209 --- /dev/null +++ b/sources/academy/platform/scraping_with_apify_and_ai/05_before_publishing.md @@ -0,0 +1,137 @@ +--- +title: Before publishing to Apify Store +description: Use AI to prepare your scraper for other users by improving its first-run experience, Apify Store listing, documentation, and maintenance plan. +slug: /scraping-with-apify-and-ai/before-publishing-to-apify-store +unlisted: true +--- + +**In this lesson, we'll prepare our app for tracking prices on an e-commerce website for other people to use. We'll use Cursor to inspect and polish its first-run experience and documentation, prepare its Apify Store listing, and make a plan for keeping it reliable.** + +--- + +Our scraper works, and its behavior is backed by documentation and tests. However, we've built it only for ourselves. If we wanted other people to use it, they'd run into several problems: + +- _Wrong kind of README:_ It tells its developers how the code should behave, not users how to get useful data. +- _Rough first run:_ It can be the case that the scraper inputs are not designed, documented, or properly tested with a first-time user in mind. +- _Empty storefront:_ The Actor has no convincing name, description, presentation, or clear pricing. +- _No maintenance strategy:_ There will be failed runs, user questions, or changes to the target website. We need to be prepared. + +Before publishing our Actor to the Apify Store we'll rework the README, make sure first-time users know what to do, prepare the Store listing, and decide how to keep the scraper working after launch. + +## Turning the README into a landing page + +Right now, the README explains how to develop the project, how it works, and why we made certain design decisions. That's useful information, but not for most users of Actors. + +They want to know what data the Actor provides, what inputs it takes, and what its output looks like. When they need to understand the scraper's limitations, they might care about some technical details, but as long as the scraper delivers data they need, they'll be perfectly happy without them. + +Let's move the current README to a different file, such as `CONTRIBUTING.md`, and create a new `README.md` that serves as the Actor's landing page. Ask the AI agent to draft it: + +```text +Move the current README content to CONTRIBUTING.md. +Then read https://docs.apify.com/actors/publishing/actor-readme +and draft a new README focused on users. +``` + +After a short wait, we'll have a new README ready. Cursor has a built-in Markdown preview, so let's make it easier to read. Open the [command palette](https://docs.cursor.com/advanced/keyboard-shortcuts) with ⌘+⇧+P on macOS or Ctrl+Shift+P on Windows and Linux. Type "mark pre", select **Markdown: Open Preview**, and press ↵. You should see a preview of how the document would look on Apify Store, GitHub, or another service. + +![Preview of the new README in Cursor](images/cursor-user-readme-top.webp) + +Each AI agent run is different, but the result will probably include sections similar to these: + +- What does this Actor do? +- How to scrape Shopify product prices +- How much does it cost? +- Input and output +- FAQ + +Cursor can read the contributing docs, inspect the code, and follow the [guide to writing a good Actor README](/actors/publishing/actor-readme) we gave it. That gives it enough context to draft a useful document. It can also anticipate questions and answers like the following: + +- What websites does this Actor support? +- How are prices parsed? +- How is stock availability handled? +- Can I schedule regular price checks? +- Something went wrong - what should I check? + +![Output fields and FAQ in the README preview](images/cursor-user-readme-output-faq.webp) + +Read the whole README and make sure everything is accurate and sounds like you. It will set users' expectations, and it's you who is responsible for every promise it makes, not the AI agent. + +This new README will eventually become the page that sells your Actor, so keep prompting the AI agent to improve it. And most importantly, ask it to rename the Actor to something catchier than "My Actor"! + +## Making the first run easy + +Now let's make sure people can understand the Actor and get through their first run without getting stuck. We'll run a small experiment. + +If you have a friend who's at least a tiny little bit tech-savvy, ask for 30 minutes of their time and let them try your Actor. Ideally, choose someone who doesn't know what you've been working on. + +It might sound a bit silly, but it really isn't! This is called _user testing_. + +Run `apify push`, give your friend the README, and open the Actor in Apify Console. Then let them take control of the computer with a single goal: run the Actor for the first time. Watch over their shoulder and take notes, but don't help. Within 30 minutes, you'll almost certainly uncover a few loose ends: + +- Does the README explain the quickest way to get useful results? +- Are the input field names clear, with helpful tooltips where needed? +- Are the default and prefilled values safe, inexpensive, and quick to run while still showing the Actor's value? +- Does the sample output make it obvious what useful data the Actor provides? +- Is the output consistent, with predictable fields and formats? +- When the Actor fails, does it provide a useful, actionable error message? + +If you can't find such a friend, you can try the experiment yourself and pretend you're seeing the Actor for the first time, but it won't match a genuine second pair of eyes. + +A better alternative is to ask an AI chat or agent other than the one that wrote the README. Use this prompt as a starting point: + +```text +Imitate a user testing session. You are an Apify Store user +who has just found this Actor and its README. Work out what +the Actor does, what it's useful for, and how to run it for +the first time. Then suggest improvements that would make +the first-run experience clearer and smoother. +``` + +For example, here's what a response from Claude, Anthropic's AI chat, might look like: + +![Claude reviewing an Actor README as a first-time user](images/claude-user-testing.webp) + +:::info Apify Store test + +Once you publish your Actor, Apify Store itself will join the feedback party. Apify [automatically tests public Actors](/actors/publishing/test) every day using each Actor's prefilled input. The run must succeed and produce a non-empty output within 5 minutes. If it fails, the Actor gets flagged. + +::: + +## Preparing the Store listing + +:::note Course under construction +This page hasn't been written yet. Please come back later! +::: + + + +## Planning for maintenance + +:::note Course under construction +This page hasn't been written yet. Please come back later! +::: + + + +## Helping the right users find it + +:::note Course under construction +This page hasn't been written yet. Please come back later! +::: + + diff --git a/sources/academy/platform/scraping_with_apify_and_ai/05_publishing.md b/sources/academy/platform/scraping_with_apify_and_ai/05_publishing.md deleted file mode 100644 index 1eacdd0275..0000000000 --- a/sources/academy/platform/scraping_with_apify_and_ai/05_publishing.md +++ /dev/null @@ -1,20 +0,0 @@ ---- -title: Publishing to Apify Store -description: TBD -slug: /scraping-with-apify-and-ai/publishing-to-apify-store -unlisted: true ---- - - - -:::note Course under construction -This page hasn't been written yet. Come later, please! -::: - - diff --git a/sources/academy/platform/scraping_with_apify_and_ai/images/claude-user-testing.webp b/sources/academy/platform/scraping_with_apify_and_ai/images/claude-user-testing.webp new file mode 100644 index 0000000000..547dc927bf Binary files /dev/null and b/sources/academy/platform/scraping_with_apify_and_ai/images/claude-user-testing.webp differ diff --git a/sources/academy/platform/scraping_with_apify_and_ai/images/cursor-user-readme-output-faq.webp b/sources/academy/platform/scraping_with_apify_and_ai/images/cursor-user-readme-output-faq.webp new file mode 100644 index 0000000000..f82cfef7d0 Binary files /dev/null and b/sources/academy/platform/scraping_with_apify_and_ai/images/cursor-user-readme-output-faq.webp differ diff --git a/sources/academy/platform/scraping_with_apify_and_ai/images/cursor-user-readme-top.webp b/sources/academy/platform/scraping_with_apify_and_ai/images/cursor-user-readme-top.webp new file mode 100644 index 0000000000..b05142de86 Binary files /dev/null and b/sources/academy/platform/scraping_with_apify_and_ai/images/cursor-user-readme-top.webp differ diff --git a/sources/academy/platform/scraping_with_apify_and_ai/images/pdp.webp b/sources/academy/platform/scraping_with_apify_and_ai/images/pdp.webp new file mode 100644 index 0000000000..6ee465c652 Binary files /dev/null and b/sources/academy/platform/scraping_with_apify_and_ai/images/pdp.webp differ