Starter project for the Expert scraping with Apify course. Each lesson of the course builds on top of this scraper, so you don't need to write one from scratch before you start.
The scraper takes a keyword, searches Amazon for it, and saves every offer for each product it finds. One product can have many offers, because there are many sellers on Amazon selling the same thing for different prices.
The scraper is built with Crawlee and the Apify SDK, on top of Apify's Crawlee + Cheerio template. The project setup, such as the Dockerfile, the schemas in .actor/, and the linter configuration, follows the template. It uses CheerioCrawler, which downloads pages over plain HTTP and parses their HTML. Each request has a label, and the router in src/routes.js handles each label differently:
START- The search results page. The scraper finds all products on it, skipping sponsored results, and enqueues a request to each product's page.PRODUCT- The product page. The scraper extracts the product's description and enqueues a request to the product's offers.OFFERS- The list of offers, which Amazon loads from a separate URL when you click Other sellers on Amazon on the product page. The scraper saves each offer as one item in the default dataset.
Data scraped in earlier steps travels to the next ones in the request's userData.
The project files:
src/main.js- Reads the input, sets up the proxies and the crawler, and starts the crawl.src/routes.js- The handlers for each label, which do the actual scraping.src/constants.js- The labels and URLs shared by the other files..actor/- Configuration of the Actor for the Apify platform: name, and the schemas of the input and of the output.Dockerfile- How the Apify platform builds the Actor.
The only input is keyword, the text to search Amazon for. It defaults to headphones.
Each dataset item represents a single offer:
{
"title": "Bose QuietComfort Bluetooth Headphones, Wireless Headphones with Active Over Ear Noise Cancelling and Mic, Deep Bass, Up to 24 Hours of Playtime, Cypress Green - Limited Edition",
"asin": "B0CCZ1SQ8G",
"itemUrl": "https://www.amazon.com/dp/B0CCZ1SQ8G",
"keyword": "headphones",
"description": "Go beyond the beat and take control of the sound that puts you on top of the world with Bose QuietComfort Wireless Noise Cancelling Headphones...",
"sellerName": "Amazon.com",
"offer": "$169.00"
}The asin is Amazon's ID of the product. Items for the same product share the asin and differ in sellerName and offer. The scraper only saves the first page of offers for each product, which is up to 11 offers.
Amazon blocks scrapers quickly, so the scraper uses Apify Proxy with residential IP addresses from the US. US addresses also make Amazon show prices in US dollars.
Even with residential proxies, Amazon sometimes responds with errors or an empty page. The crawler retries such requests with a different IP address, so warnings about retries in the log are expected. Residential proxies are paid by traffic. A run with the default input costs a few cents.
You need Node.js 24 or newer, the Apify CLI, and an Apify account.
npm install
apify login
apify runConnecting to Apify Proxy from your own computer requires a paid Apify plan. On the free plan, the local run fails on proxy errors, so deploy the scraper to the Apify platform and run it there:
apify pushThen open the Actor in Apify Console and start it, or run apify call.
The code is checked by ESLint and formatted by Prettier. Before you commit, run:
npm run lint
npm run format
apify validate-schemaGitHub Actions run the same checks on every pull request.
Amazon changes its website from time to time, and the scraper can stop finding data. If you get empty items or no items at all, open an issue.