Build a Web Scraper with Node.js, TypeScript, and Cheerio

What you'll build

By the end of this tutorial you'll have a working web scraper written in TypeScript. It fetches a web page, parses the HTML with Cheerio (jQuery-style selectors on the server), pulls a list of items out of the page (book titles, prices, and links), and writes them to a clean JSON file. Then you'll turn it into a small command-line tool that takes any URL as an argument.

Every file you need is in this page: a package.json, a tsconfig.json, and a single src/scrape.ts. Paste each one into a fresh project, follow the setup steps, and it runs. No repo to clone, no hidden files.

We scrape books.toscrape.com, a site the Scrapy team built specifically for practicing scraping. That matters: it is a static HTML page that is meant to be scraped, so you can follow along without worrying about hammering someone's real server or breaking a rule. The last section covers how to scrape responsibly once you point this at a live site.

fetch or Axios? The honest verdict

A scraper does two jobs: get the HTML, then read it. Those are two different tools, and people often confuse which one does what.

Getting the HTML is an HTTP request. You have three common choices. Node's built-in fetch (stable since Node 18) needs zero extra packages and is what this tutorial uses. Axios is the most popular third-party HTTP client and is worth reaching for once you need request retries, interceptors, or automatic JSON handling across a bigger project. node-fetch was the old way to get fetch before Node shipped its own, and you no longer need it on a modern Node version.

Reading the HTML is where Cheerio comes in, and there is no real debate here: Cheerio is the standard library for parsing static HTML in Node. It loads the HTML string into a document you query with the same selectors you'd use in the browser ($('h3 a')), which is why anyone who has touched jQuery feels at home in minutes.

We use built-in fetch to keep the runtime dependency count at exactly one (Cheerio). If you already know Axios or your project uses it, swapping it in is a two-line change, and the FAQ at the bottom shows how.

What you need

• Node.js 18.17 or newer. The scraper uses the built-in fetch, stable since Node 18, and Cheerio v1 needs Node 18.17+. Check with node --version; if you are older, update.
• A code editor and a terminal. That is the whole toolchain. No API keys, no accounts, no paid services.
• Basic TypeScript and CSS-selector familiarity. If you can read document.querySelector('.price'), you already know enough to follow along. New to selectors? Our CSS courses cover them.

Set up the project

Create a fresh project and install the one runtime dependency plus the TypeScript tooling:

mkdir web-scraper-ts
cd web-scraper-ts
npm init -y
npm install cheerio
npm install -D typescript tsx @types/node

Cheerio is the only thing that ships to production. tsx runs TypeScript directly so there is no build step while you follow along, and @types/node gives you typed process, fetch, and the fs module. Open package.json, add "type": "module", and add these scripts:

{
  "type": "module",
  "scripts": {
    "start": "tsx src/scrape.ts",
    "build": "tsc"
  }
}

Last, add a tsconfig.json at the project root. This runs cleanly with tsx in development and compiles with tsc for a production build:

{
  "compilerOptions": {
    "target": "ES2022",
    "module": "ESNext",
    "moduleResolution": "bundler",
    "strict": true,
    "esModuleInterop": true,
    "skipLibCheck": true,
    "outDir": "dist",
    "types": ["node"]
  },
  "include": ["src"]
}

Step 1: fetch the HTML

Create src/scrape.ts. We'll build it up one function at a time, then run the finished file at the end. First, a typed function that fetches a page and returns its HTML as a string, or throws a clear error if the request fails:

async function fetchHtml(url: string): Promise<string> {
  const response = await fetch(url, {
    headers: {
      // Identify the scraper honestly. Some servers reject requests with no
      // User-Agent, and a real one is polite: it tells the site who is calling.
      "User-Agent": "FreeCodingCourses-tutorial-scraper/1.0",
    },
  });

  if (!response.ok) {
    throw new Error(`Request failed: ${response.status} ${response.statusText}`);
  }

  return response.text();
}

Two decisions worth calling out. The response.ok check is not optional: fetch only rejects on a network error, so a 404 or 500 still resolves. Without this guard you would happily try to parse an error page and get confusing empty results. And the User-Agent header is the polite default. A blank User-Agent is a common reason a scraper gets a 403 back, and setting an honest one tells the site who is calling if they look at their logs.

Step 2: parse with Cheerio

Now the part that makes this a scraper. Add an interface for one record, then a function that loads the HTML into Cheerio and pulls out every book on the page. On books.toscrape.com, each book sits in an <article class="product_pod">, the title is in the title attribute of an <h3> link, and the price is in a .price_color element. Open the page and "View Source" to confirm; reading the real markup is the first step of every scrape.

import * as cheerio from "cheerio";

// The shape of one scraped record. Everything downstream is typed off this.
interface ScrapedBook {
  title: string;
  price: string;
  url: string;
}

function parseBooks(html: string, baseUrl: string): ScrapedBook[] {
  const $ = cheerio.load(html);
  const books: ScrapedBook[] = [];

  $("article.product_pod").each((_index, element) => {
    const link = $(element).find("h3 a");
    const title = link.attr("title")?.trim() ?? "";
    const price = $(element).find(".price_color").text().trim();
    const href = link.attr("href") ?? "";
    // Listing links are relative ("catalogue/..."), so resolve them to a full
    // URL against the page we fetched. new URL does this for you.
    const url = href ? new URL(href, baseUrl).href : "";

    if (title) {
      books.push({ title, price, url });
    }
  });

  return books;
}

Read the flow top to bottom. cheerio.load(html) parses the string into a document and hands you $, the same name jQuery uses. $("article.product_pod") selects every book card. .each walks them, and inside the callback $(element) wraps the current card so you can search within it with .find.

The small choices are what make this reliable. Pulling the title from link.attr("title") rather than the link's visible text avoids the truncated "..." that the site shows in the listing. The ?? "" and the if (title) guard mean a malformed card gets skipped instead of pushing a half-empty record into your results. And new URL(href, baseUrl) turns a relative link into a full, clickable one, which is the difference between data you can use and data you have to clean later.

Step 3: write the results to disk

Scraped data is only useful if you keep it. Add a function that writes the list to a pretty-printed JSON file using fs/promises, so the output is easy to read and easy to feed into whatever comes next:

import { writeFile } from "node:fs/promises";

async function saveJson(books: ScrapedBook[], file: string): Promise<void> {
  await writeFile(file, JSON.stringify(books, null, 2), "utf8");
}

JSON.stringify(books, null, 2) is the whole trick: the 2 is the indent, which turns a single dense line into readable, diff-friendly JSON. Swap the .json file for a .csv builder later if a spreadsheet is where the data needs to land, but JSON is the right default while you are still shaping the scrape.

Step 4: make it a CLI

Tie the three functions together with a main that reads the target URL from the command line, so you can point the scraper at any page without editing the code. If no URL is passed, it falls back to the practice site. Here is the complete src/scrape.ts, all four pieces in order, ready to paste as one file:

import { writeFile } from "node:fs/promises";
import * as cheerio from "cheerio";

// The shape of one scraped record. Everything downstream is typed off this.
interface ScrapedBook {
  title: string;
  price: string;
  url: string;
}

// Step 1: fetch the page HTML, or throw a clear error.
async function fetchHtml(url: string): Promise<string> {
  const response = await fetch(url, {
    headers: {
      "User-Agent": "FreeCodingCourses-tutorial-scraper/1.0",
    },
  });

  if (!response.ok) {
    throw new Error(`Request failed: ${response.status} ${response.statusText}`);
  }

  return response.text();
}

// Step 2: turn the HTML into a typed list of books.
function parseBooks(html: string, baseUrl: string): ScrapedBook[] {
  const $ = cheerio.load(html);
  const books: ScrapedBook[] = [];

  $("article.product_pod").each((_index, element) => {
    const link = $(element).find("h3 a");
    const title = link.attr("title")?.trim() ?? "";
    const price = $(element).find(".price_color").text().trim();
    const href = link.attr("href") ?? "";
    const url = href ? new URL(href, baseUrl).href : "";

    if (title) {
      books.push({ title, price, url });
    }
  });

  return books;
}

// Step 3: write the results to a pretty-printed JSON file.
async function saveJson(books: ScrapedBook[], file: string): Promise<void> {
  await writeFile(file, JSON.stringify(books, null, 2), "utf8");
}

// Step 4: wire it together. Read the target URL from argv, fall back to the
// practice site if none is given.
async function main(): Promise<void> {
  const targetUrl = process.argv[2] ?? "https://books.toscrape.com/";
  console.log(`Fetching ${targetUrl} ...`);

  const html = await fetchHtml(targetUrl);
  const books = parseBooks(html, targetUrl);
  await saveJson(books, "books.json");

  console.log(`Scraped ${books.length} items -> books.json`);
}

main().catch((err) => {
  console.error(err);
  process.exit(1);
});

process.argv[2] is the first argument after the script name, so npm start -- https://example.com passes a URL straight through. The main().catch(...) at the bottom is the top-level error handler: any failure (a bad URL, a non-200 response, a write error) prints a readable message and exits with a non-zero code, which is exactly what you want if you ever run this from a cron job or a CI step.

Run it and read the output

Run the scraper against the default practice site:

npm start
# -> Fetching https://books.toscrape.com/ ...
# -> Scraped 20 items -> books.json

Open books.json and you'll see 20 records, one per book on the first page, each with a full title, a price, and an absolute URL. To scrape a different page, pass it as an argument (the -- tells npm the rest is for your script, not for npm):

npm start -- https://books.toscrape.com/catalogue/page-2.html

That second page has the next 20 books. Change the two selectors in parseBooks (article.product_pod and .price_color) to match a different site's markup and you have scraped something new. The whole pattern is: find the repeating element, find the fields inside it, map each one to a typed object.

What Cheerio can't do, and what to reach for

This is the honest limitation, and it is the single most useful thing to understand about scraping. Cheerio parses the HTML the server sends. It does not run JavaScript. So if you fetch a page and the data you want is missing from the result, the page is almost certainly building that content in the browser after load: an infinite-scroll feed, a single-page React or Vue app that ships an empty shell, a "load more" button that fires an API call. Cheerio will only ever see the empty shell.

You have two good options when that happens. First, open your browser's Network tab and look for the API call the page makes; often the data comes from a clean JSON endpoint you can request directly, which is faster and kinder than rendering the whole page. Second, when there is no such endpoint, reach for a headless browser like Playwright or Puppeteer. Those drive a real Chromium instance, run the page's JavaScript, and then hand you the finished HTML, which you can still parse with Cheerio if you like. The tradeoff is weight: a headless browser is much slower and heavier than a plain fetch, so use Cheerio alone whenever the data is already in the HTML, and only add a browser when it truly is not.

The other limit is not technical, it is about permission. Scraping a site sends real traffic to someone else's server, and not every site wants to be scraped. Before you point this at a live site, check its robots.txt (at example.com/robots.txt) and its terms of service, scrape at a slow rate rather than hammering the server, and never collect personal data you do not have a right to. The practice site we used here exists precisely so you can learn without any of that being a concern. Treat it as the norm, not the exception: be a polite guest on other people's servers.

Keep a reference handy

You now have a scraper in one file: fetch, parse, save, and a CLI wrapper. Every change from here is a variation on those four steps. Want a quick-reference card for the TypeScript syntax you'll lean on (types, async/await, modules)? Our Node.js cheat sheet and TypeScript cheat sheet are worth keeping open while you build.

Where to go next

You have the core loop. Three moves take it further.

Scrape every page, not just one. The site paginates at /catalogue/page-N.html. Wrap the fetch-and-parse in a loop that increments N and stops when a page returns no books. Add a short delay between requests so you are not hammering the server.

Scrape detail pages. For each book URL you collected, fetch that page and pull the fields the listing does not show (description, stock count, rating). This is the same fetchHtml plus parseBooks pattern, one level deeper.

Store it somewhere real. A JSON file is fine to start, but once you are collecting a lot, a database is better. Our tutorial on building a REST API with Express and TypeScript and the URL shortener with SQLite both cover the TypeScript data patterns you would use to save scraped records into SQLite or Postgres.