I have just implemented a simple cookie consent banner on my website, and I want to share with you in this article how I've done it with a step-by-step guide.

The cookie consent banner is a way to inform your users how your website and the services it uses their browser cookies.

It does contain a link to your website privacy policy where they can learn in more detail how their personal data is stored and used.

Each website's policy is different, that's why you may need to change the content of the consent banner, as well as the type of action buttons your website needs.

In this article, we will use a fresh Next.js installation with TypeScript and Tailwind CSS, so it's easy for you to follow the same steps without getting lost in the project complexity.

You don't have to create a new Next.js project as long as you already have one, Luckily I already have an article that explains all about setting up a new project.

If you like to follow along, please go ahead and set up your project using the <a href="https://codersteps.com/articles/how-to-set-up-and-structure-your-full-stack-next-js-application">How To Set Up And Structure Your Full-Stack Next.Js Application</a> article.

Please skip the unnecessary sections for the purpose of this article, you can read up to the Tailwind CSS section by skipping the Google Fonts section.

I went ahead and did that myself, so now I have a fresh Next.js project with TypeScript and Tailwind CSS set up and ready to go.

In order to include the consent banner on all our public pages, we will need to create a shared layout that wraps all the pages we want the banner to show on.

We will create a simple layout that we will apply on the homepage and our privacy policy page, but for you, you may want to use a different approach that sweets your needs better.

You can learn more about how you can do that on the <a href="https://nextjs.org/docs/basic-features/layouts" rel="noopener noreferrer" target="_blank">Next.js Layouts</a> docs page.

Let's first, create a new file at `components/layouts/PublicLayout.tsx` and have it just add a simple header at the top and the footer at the bottom.

import Link from 'next/link'
import { ReactNode } from 'react'

export const Container = ({ children }: { children: ReactNode }) => (
  <div className="w-full max-w-5xl px-2 mx-auto md:px-4">{children}</div>
)

const PublicLayout = ({ children }: { children: ReactNode }) => {
  return (
    <div className="flex flex-col min-h-screen">
      <header className="flex items-center border-b border-gray-200 h-14">
        <Container>
          <nav>
            <ul>
              <li>
                <Link href="/">
                  <a className="text-sm font-medium text-center capitalize">
                    Home
                  </a>
                </Link>
              </li>
            </ul>
          </nav>
        </Container>
      </header>
      <main className="flex-grow">
        <Container>{children}</Container>
      </main>
      <footer className="py-5 border-t border-gray-200">
        <Container>
          <nav className="my-5">
            <ul>
              <li>
                <Link href="/privacy-policy">
                  <a className="text-sm font-medium text-center capitalize">
                    Privacy Policy
                  </a>
                </Link>
              </li>
            </ul>
          </nav>
        </Container>
        <Container>
          <p className="text-sm font-medium text-center capitalize">
            copyright © {new Date().getFullYear()} all rights reserved
          </p>
        </Container>
      </footer>
    </div>
  )
}

export default PublicLayout

As you can see this is a simplified not responsive `PublicLayout` example component, but it will serve us very well in this article.

At the top we declared a simple React component `Container` which we reuse it inside our layout component, it's just a simple div.

Then we added a simple header, main, and footer sections, we used `flex-col` and `flex-grow` to have the main section take the available height, in order to push the footer to the bottom.

Applying the public layout component to our public pages

Let's go ahead and have our homepage use the `PublicLayout` component

import Head from 'next/head'
import type { NextPage } from 'next'
import PublicLayout from '../components/layouts/PublicLayout'

const Home: NextPage = () => {
  return (
    <>
      <Head>
        <title>Next.js cookie consent banner</title>
        <meta
          name="description"
          content="A Next.js cookie consent banner with TypeScript and Tailwind CSS."
        />
      </Head>

      <PublicLayout>
        <h1 className="text-3xl font-bold font-open">
          Next.js cookie consent banner
        </h1>
      </PublicLayout>
    </>
  )
}

export default Home

We simply imported it and wrapped our content using it, now if we opened our homepage, we should see a header and a footer.

As we mentioned above we will link to the privacy policy page from the banner content. I know if you already own a website there's a high chance you already have it.

But I wanted to add it, so I can have a fully working example at the end of this article for you as a reference, as well as having a page link I can refer to.

I will go ahead and create a new file at `pages/privacy-policy.tsx` and a simple empty page with a title.

import Head from 'next/head'
import type { NextPage } from 'next'
import PublicLayout from '../components/layouts/PublicLayout'

const PrivacyPolicy: NextPage = () => {
  return (
    <>
      <Head>
        <title>Our Privacy Policy</title>
        <meta name="description" content="Website privacy policy page" />
      </Head>

      <PublicLayout>
        <h1 className="text-3xl font-bold font-open">
          Website privacy policy page content
        </h1>
      </PublicLayout>
    </>
  )
}

export default PrivacyPolicy

You may notice we have also wrapped its content with the `PublicLayout` component just like we did with the homepage.

It's time to create the main component that will display the cookie consent banner to the user. but first, let's install a single dependency that we will need to manage browser cookies.

The `js-cookie` npm package is a handy tool to set and get our browser cookies, and we need that to remember that a user has already accepted our consent.

To install it we will use the commands `npm i js-cookie` and `npm i --save-dev @types/js-cookie` for the package types.

CookieConsent component structuring and styling

Now we are ready to start with our component structure, content, and styling, we will have a fixed banned with standard content and a single button that says `"Got it"`.

Go ahead and create a file for our component at `components/banners/CookieConsent.tsx` and put the following.

import Link from 'next/link'
import { MouseEvent } from 'react'
import { Container } from '../layouts/PublicLayout'

const CookieConsent = () => {
  const onClick = (e: MouseEvent<HTMLButtonElement>) => {
    e.preventDefault()
    console.log('remember that I accept')
  }

  return (
    <section className="fixed bottom-0 left-0 w-full py-2 md:py-4">
      <Container>
        <div className="flex flex-col items-start px-5 py-3 space-y-2 bg-gray-200 md:flex-row md:space-y-0 md:items-stretch md:space-x-2">
          <div className="flex items-center flex-grow text-gray-900">
            <p className="text-sm font-medium">
              This site uses services that use cookies to deliver better
              experience and analyze traffic. You can learn more about the
              services we use at our{' '}
              <Link href="/privacy-policy">
                <a className="text-sm underline hover:text-lightAccent">
                  privacy policy
                </a>
              </Link>
              .
            </p>
          </div>
          <div className="flex items-center">
            <button
              className="p-3 text-sm font-bold text-white uppercase bg-gray-700 whitespace-nowrap"
              onClick={onClick}
            >
              Got it
            </button>
          </div>
        </div>
      </Container>
    </section>
  )
}

export default CookieConsent

I will assume that you know your way around CSS and Tailwind so I will focus on explaining the idea behind the styles instead of the actually applied styles.

We have added a simple fixed wrapper to the bottom, with a vertical padding, inside of it we have added a container which by default it has horizontal padding.

Inside the container, we added our banner saying that we use services that use the user's cookie and we give them a button to consent to it.

CookieConsent button action implementation

When the button is clicked, currently we just log to the console, but in this subsection, we will have it set a cookie with the key `cookie_consent_is_true` to `"true"` to mark a user as consented.

We will use the `js-cookie` package to easily do that, this package gives us two methods `set` and `get` which enable us to set a new cookie with `key=value` pairs.

Let's write the code first then explain it, go ahead and update the `CookieConsent` component to match the following changes.

import Link from 'next/link'
import Cookies from 'js-cookie'
import { MouseEvent, useCallback, useEffect, useState } from 'react'
import { Container } from '../layouts/PublicLayout'

const USER_CONSENT_COOKIE_KEY = 'cookie_consent_is_true'
const USER_CONSENT_COOKIE_EXPIRE_DATE = 365

const CookieConsent = () => {
  const [cookieConsentIsTrue, setCookieConsentIsTrue] = useState(true)

  useEffect(() => {
    const consentIsTrue = Cookies.get(USER_CONSENT_COOKIE_KEY) === 'true'
    setCookieConsentIsTrue(consentIsTrue)
  }, [])

  const onClick = (e: MouseEvent<HTMLButtonElement>) => {
    e.preventDefault()

    if (!cookieConsentIsTrue) {
      Cookies.set(USER_CONSENT_COOKIE_KEY, 'true', {
        expires: USER_CONSENT_COOKIE_EXPIRE_DATE,
      })
      setCookieConsentIsTrue(true)
    }
  }

  if (cookieConsentIsTrue) {
    return null
  }

  return (
    <section className="fixed bottom-0 left-0 w-full py-2 md:py-4">
      ...
    </section>
  )
}

export default CookieConsent

For a little best practice 😁 we declared two constants `USER_CONSENT_COOKIE_KEY` for the cookie key and `USER_CONSENT_COOKIE_EXPIRE_DATE` for the cookie expires property.

Then we created a new state for our user cookie consent value `cookieConsentIsTrue` we set it to be `true` by default.

Using the `useEffect` hook which runs once after the component has been mounted, we check if there's a cookie with the key `"cookie_consent_is_true"` and value `"true"`.

When the user clicks on the "Got it" button, we simply check if the `cookieConsentIsTrue` is not true, which means the cookie wasn't set before, if not then we set it with the right value and an expiration date of a single year.

Finally, before rendering our component we check if the user has already accepted, if yes we just return `null` which means the component won't be rendering anything.

That was it for our component but sure you can extend it to whatever extent your website needs, Now it's time to add it to our public layout.

This is simply importing the component and inserting it after our footer, where you put it doesn’t matter because we give it a fixed position.

// imports
import CookieConsent from '../banners/CookieConsent'

...

const PublicLayout = ({ children }: { children: ReactNode }) => {
  return (
    <div className="flex flex-col min-h-screen">
      <header className="flex items-center border-b border-gray-200 h-14">
        ...
      </header>
      <main className="flex-grow">
        <Container>{children}</Container>
      </main>
      <footer className="py-5 border-t border-gray-200">
        ...
      </footer>
      <CookieConsent />
    </div>
  )
}

export default PublicLayout

Now if we checked out our home and privacy policy pages, we should see a cookie consent banner appears.

And if we click on the "Got it" button it should disappear, even if we reload it won't show, because we set the cookie to mark the user as consented.

In this article, you have learned about how to add a cookie consent banner to your Next.js website with Tailwind CSS as our styling framework of choice.

Source code is available at: https://github.com/codersteps/nextjs_cookie_consent_banner.

That's it for this one, I hope it was helpful, see you in the next one 😉.

Learn how to add a cookie consent banner with Next.js and Tailwind CSS, and how to remember when a user has already accepted the cookie consent banner.

I have just implemented a simple cookie consent banner on my website, and I want to share with you in this article how I've done it with a step-by-step guide.


The cookie consent banner is a way to inform your users how your website and the services it uses their browser cookies.


It does contain a link to your website privacy policy where they can learn in more detail how their personal data is stored and used.


Each website's policy is different, that's why you may need to change the content of the consent banner, as well as the type of action buttons your website needs.


In this article, we will use a fresh Next.js installation with TypeScript and Tailwind CSS, so it's easy for you to follow the same steps without getting lost in the project complexity.


You don't have to create a new Next.js project as long as you already have one, Luckily I already have an article that explains all about setting up a new project.


If you like to follow along, please go ahead and set up your project using the <a href="https://codersteps.com/articles/how-to-set-up-and-structure-your-full-stack-next-js-application">How To Set Up And Structure Your Full-Stack Next.Js Application</a> article.


Please skip the unnecessary sections for the purpose of this article, you can read up to the Tailwind CSS section by skipping the Google Fonts section.


I went ahead and did that myself, so now I have a fresh Next.js project with TypeScript and Tailwind CSS set up and ready to go.


In order to include the consent banner on all our public pages, we will need to create a shared layout that wraps all the pages we want the banner to show on.


We will create a simple layout that we will apply on the homepage and our privacy policy page, but for you, you may want to use a different approach that sweets your needs better.


You can learn more about how you can do that on the <a href="https://nextjs.org/docs/basic-features/layouts" rel="noopener noreferrer" target="_blank">Next.js Layouts</a> docs page.


Let's first, create a new file at <code>components/layouts/PublicLayout.tsx</code> and have it just add a simple header at the top and the footer at the bottom.


As you can see this is a simplified not responsive <code>PublicLayout</code> example component, but it will serve us very well in this article.


At the top we declared a simple React component <code>Container</code> which we reuse it inside our layout component, it's just a simple div.


Then we added a simple header, main, and footer sections, we used <code>flex-col</code> and <code>flex-grow</code> to have the main section take the available height, in order to push the footer to the bottom.


Let's go ahead and have our homepage use the <code>PublicLayout</code> component


We simply imported it and wrapped our content using it, now if we opened our homepage, we should see a header and a footer.


As we mentioned above we will link to the privacy policy page from the banner content. I know if you already own a website there's a high chance you already have it.


But I wanted to add it, so I can have a fully working example at the end of this article for you as a reference, as well as having a page link I can refer to.


I will go ahead and create a new file at <code>pages/privacy-policy.tsx</code> and a simple empty page with a title.


You may notice we have also wrapped its content with the <code>PublicLayout</code> component just like we did with the homepage.


It's time to create the main component that will display the cookie consent banner to the user. but first, let's install a single dependency that we will need to manage browser cookies.


The <code>js-cookie</code> npm package is a handy tool to set and get our browser cookies, and we need that to remember that a user has already accepted our consent.


To install it we will use the commands <code>npm i js-cookie</code> and <code>npm i --save-dev @types/js-cookie</code> for the package types.


Now we are ready to start with our component structure, content, and styling, we will have a fixed banned with standard content and a single button that says <code>"Got it"</code>.


Go ahead and create a file for our component at <code>components/banners/CookieConsent.tsx</code> and put the following.


I will assume that you know your way around CSS and Tailwind so I will focus on explaining the idea behind the styles instead of the actually applied styles.


We have added a simple fixed wrapper to the bottom, with a vertical padding, inside of it we have added a container which by default it has horizontal padding.


Inside the container, we added our banner saying that we use services that use the user's cookie and we give them a button to consent to it.


When the button is clicked, currently we just log to the console, but in this subsection, we will have it set a cookie with the key <code>cookie_consent_is_true</code> to <code>"true"</code> to mark a user as consented.


We will use the <code>js-cookie</code> package to easily do that, this package gives us two methods <code>set</code> and <code>get</code> which enable us to set a new cookie with <code>key=value</code> pairs.


Let's write the code first then explain it, go ahead and update the <code>CookieConsent</code> component to match the following changes.


For a little best practice 😁 we declared two constants <code>USER_CONSENT_COOKIE_KEY</code> for the cookie key and <code>USER_CONSENT_COOKIE_EXPIRE_DATE</code> for the cookie expires property.


Then we created a new state for our user cookie consent value <code>cookieConsentIsTrue</code> we set it to be <code>true</code> by default.


Using the <code>useEffect</code> hook which runs once after the component has been mounted, we check if there's a cookie with the key <code>"cookie_consent_is_true"</code> and value <code>"true"</code>.


When the user clicks on the "Got it" button, we simply check if the <code>cookieConsentIsTrue</code> is not true, which means the cookie wasn't set before, if not then we set it with the right value and an expiration date of a single year.


Finally, before rendering our component we check if the user has already accepted, if yes we just return <code>null</code> which means the component won't be rendering anything.


That was it for our component but sure you can extend it to whatever extent your website needs, Now it's time to add it to our public layout.


This is simply importing the component and inserting it after our footer, where you put it doesn’t matter because we give it a fixed position.


Now if we checked out our home and privacy policy pages, we should see a cookie consent banner appears.


And if we click on the "Got it" button it should disappear, even if we reload it won't show, because we set the cookie to mark the user as consented.


In this article, you have learned about how to add a cookie consent banner to your Next.js website with Tailwind CSS as our styling framework of choice.


Source code is available at: <a href="https://github.com/codersteps/nextjs_cookie_consent_banner">https://github.com/codersteps/nextjs_cookie_consent_banner</a>.


That's it for this one, I hope it was helpful, see you in the next one 😉.


How to add cookie consent banner with Next.js and Tailwind CSS

**Notice**: Before we start I just want to say that this article is purely for educational purposes, please use it responsibly at your own risk.

`1337x.to` is a popular torrents website, with multiple categories: movies, music, and so on.

The goal of this article is to automate downloading torrents on a page like trending music, top-100 movies, and so on.

To download a torrent from `1337x.to` we need to go to the torrent details page, then click the magnet link, which will then be opened with your default torrent downloader.

Imagine you want to download 50 torrents on a page, you will have to visit each page separately, then download the files by clicking on the magnet link on that page.

I will focus on explaining the principles used to scrape the links from the `1337x.to`, so you will be able to reuse the same technologies and adapt them to your use case.

In this article, we will use Node.js with TypeScript support, even though it’s a small project I like to use TypeScript just for fun.

Get your terminal ready, and change your current directory to the location you want to store your projects.

We will create a new directory with `mkdir javascript_extract_torrents_magnet_links`, then change to it with `cd javascript_extract_torrents_magnet_links`.

Let's initialize our git repository `git init`, our npm repository `npm init -y`, and our tsconfig.json by installing TypeScript `npm install typescript --save-dev` then `npx tsc --init`. 

This is the `tsconfig.json` file after removing comments, and adding the `"outDir": ".out"` property.

When using Node.js with TypeSctipt it's helpful to install `@types/node` with `npm install @types/node@14 --save-dev` please specify your own version instead of 14.

Let's start by creating a new file at `src/app.ts` with a simple async function that logs hello world to the console.

The app function will be the main starting point of our application.

async function app() {
  console.log("hello world");
}

app();

To execute it, we will add an npm script called `fetch` to compile and execute our app.ts file.

{
    ...
    "scripts": {
      "fetch": "tsc && node .out/app.js"
    },
    ...
}

Because we use git, we need to add a `.gitignore`

To test that everything is working as expected we will run `npm run fetch` we should see a *hello world* message.

It's time to save our change to our git repository with `git add .` then `git commit -m "initial commit"`.

Web scraping links for torrent detail pages

As we said before, there are two types of pages we will scrape, a listing page that contains multiple links to detail pages, and a single detail page.

Let's first install both dependencies with `npm install cheerio node-libcurl` and `npm install @types/cheerio --save-dev`.

The main idea behind web scraping is fetching the HTML content for a page, then extracting the required information in an automatic way.

The same principle applies here, this is the structure of links on the "/popular-music" page.

<table class="...">
  <thead>
    ...
  </thead>
  <tbody>
    <tr>
      <td class="...">
        <a href="..." class="icon"><i class="flaticon-hd"></i></a>
        <a href="the-link-we-are-looking-for">Torrent title</a>
        <span class="..."><i class="..."></i>1</span>
      </td>
      ...
    </tr>
  </tbody>
</table>

To extract the links we're looking for, we will need a css selector to use with `cheerio` to query the exact anchor elements of our choice.

To make it readable and clean we will create a new object `selectors` which will store all of our css selectors, this becomes handy in larger web scraping projects.

It's always easier to explain after the code is already written, let's do just that and then explain what's going on.

import * as cheerio from "cheerio";
import { curly } from "node-libcurl";

const config = {
  baseUrl: "https://1337x.to",
  selectors: {
    anchorLink:
      'table > tbody> tr > td:first-child > a:last-child[href^="/torrent/"]',
    magnetAnchorLink: 'ul>li>a[onclick="javascript: count(this);"]',
  },
};

async function get(path: string): Promise<string> {
  const result = await curly.get(`${config.baseUrl}${path}`);
  if (result.statusCode !== 200) {
    throw `error with status code: ${result.statusCode}`;
  }
  return result.data;
}

....

Because this project is not that big we will keep all code in a single file, but please feel free to move any helper function to a `helpers.ts` file, and wrappers for packages like our `get` function to `lib.ts`.

The important thing is to be comfortable with your file structure, so it's easy for you to modify or add new features.

As you can see we have declared a new object `config` which stores our app configurations, we have a `baseUrl` which is the homepage of the website we are scraping.

And the `selectors` config which includes the css selectors we need to extract data from the scrapped HTML content.

Then we created a wrapper function around our `curly` function from the `node-libcurl` npm package.

This way it's easy to use and without unnecessary duplication, it will also give us the possibility to easily replace the current package with another one without having to do changes all over the place.

The `get` function will simply try to fetch the HTML content from the page path, by concatenating it with the `baseUrl`, it will return the HTML in case of success otherwise it will just throw an error.

...

async function app(path: string) {
  const html = await get(path);
  const $ = cheerio.load(html);

  const anchors = Array.from($(config.selectors.anchorLink));
  const hrefs = anchors.map((a) => $(a).attr("href")) as string[];

  console.log({ hrefs });
}

app("/popular-music");

Inside our main `app` function, we accept the `path` of the page we want to scrape, we use the `get` function to fetch the page HTML content.

Then we will create a dom instance using `cheerio.load` passing to it the HTML content we got back from `curl`. `cheerio.load` returns something similar to `jQuery` but on the server.

Now it's time to extract some data, using our `anchorLink` selector we will select all the anchor elements related to the torrent detail pages.

This will return something called `cheerio.Cheerio` which contains the elements that match our selector, used with `Array.from` we get an array of `cheerio.Element[]`.

You can imagine the `cheerio.Element` as an `HTMLAnchorElement` and object with the properties we need like the `href` property in our case.

Using the `Array.prototype.map() ` we map the `cheerio.Element[]` to elements `href` attribute `string[]`.

At this point, if you run the `fetch` script with `npm run fetch`, you should see an array of the href attribute values logged to your console.

Web scraping magnet link from each torrent detail page

Now that we have the list of relative links to all torrent detail pages, we can scrape each page and extract the torrent magnet link needed to download the torrent files.

Just after where we map elements to links, we will add the following.

  ...
  const hrefs = anchors.map((a) => $(a).attr("href"));

  let fetched = 1;
  const magnets: string[] = [];
  for (const href of hrefs) {
    const html = await get(href);
    const $ = cheerio.load(html);

    const magnet = $(config.selectors.magnetAnchorLink).attr("href") as string;
    magnets.push(magnet);

    console.log("magnet fetched: ", fetched);
    fetched++;
  }

  console.log({ magnets });
  ...

We loop through the `hrefs` array from the previous step, and for each `href` we do a similar thing as we did before.

We fetch the page HTML content using the `get` function, then we create a cheerio instance with the `cheerio.load` function.

Here because we only have a single link per page, we don't need to map anything, we just access the `href` attribute and push it to the array `magnets`.

We also declare the `fetched` variable which starts at `1` and increments by `1` each time a page was scrapped. We then `console.log` it to have some feedback.

At this point, if you run `npm run fetch` again, you should see an array of the magnet links we extracted from each torrent detail page.

Create an HTML page with our magnet links

The purpose of doing all of this was to be able to automate downloading torrents through extracting magnet links and using them in an automatic fashion.

In this section, we will have our app generate a simple HTML page with an anchor link to each magnet link.

The idea is to create a `public/index.html` file, populate it with the links we extracted, then server using the `live-server` package.

When this page loads we will have it click on the links every 3 seconds, at first you will need to allow the page to open links (you will get a notice that links were blocked, so you need to allow all).

By default when you click a magnet link on your browser you get a popup asking you if you want to open it with your default torrent downloader or cancel.

What you want to do is check the *Always allow site_domain to open links ...* and click *Open torrent_app_name* this will allow our script to work as we intended to use it.

Enough theory let's write some code, we will use `fs/promises` to create and put content to our file by creating a new `writePublicIndex` function that takes the scrapped magnet links.

Just after the `get` function add the following.

// imports
import path from 'path'
import fs from 'fs/promises'

...

async function writePublicIndex(magnets: string[]) {
  const indexPath = path.resolve("./public/index.html");

  const html = ['<div>'];
  for (const magnet of magnets) {
    html.push(`
  <a href="${magnet}" target="_blank">Link</a>`);
  }
  html.push(`
  <script>
    window.onload = async (event) => {
      let executed = 1
      for (let anchor of Array.from(document.getElementsByTagName('a'))) {
        await new Promise((res, rej) => {
          setTimeout(() => {
            anchor.click()
            console.log('magnet executed: ', executed)
            res()
          }, 3000)
        })
        executed++
      }
    }
  </script>
</div>
  `);

  try {
    await fs.writeFile(indexPath, html.join(""), {
      encoding: "utf-8",
    });
  } catch (e: any) {
    console.log(`Error while writing the ${indexPath}: ${e.message}`);
  }
}

...

For the `writePublicIndex` function to work properly we need to manually create the `public/index.html` file and leave it empty.

At first, we declare and set the `indexPath` constant with the help of the `path.resolve` function, which enables us to get an absolute path from a relative one.

Then we declare an `html` array, which we use to construct our HTML page, first we push a `<div>` to it, then we use the array `map` prototype to map our magnet links to HTML anchor tags.

After that, we add a script that will run after the page has loaded, the script's purpose is to click on each link on the page with a delay of 3 seconds.

After that, we just close our `</div>` tag, and we try to write the content to the `public/index.html` file, if an error occurs like the *file does not exist*, we just log the error message to our terminal.

After creating the `writePublicIndex` function, it's time to make use of in order to populate our `public/index.html` with the magnet links we scrapped in the previous section.

Locate where we logged the magnets array to the console like so `console.log({ magnets });` change it with `await writePublicIndex(magnets);`.

...

async function writePublicIndex(magnets: string[]) {
  ...
}

async function app(path: string) {
  ...

  for (const href of hrefs) {
    ...
  }

  await writePublicIndex(magnets);
}

app("/cat/Games/1/");

Now, if we run our `fetch` script again it should populate our `public/index.html` file with links to all the magnet links we scrapped.

This automation wouldn’t be complete without making the page path dynamic, in our example we use a static path like `"/cat/Games/1/"` or `"/popular-music"`.

But what about passing it as an argument from the terminal, and maybe having a default page path in case no page was provided.

It's pretty straightforward, we will use the `process.argv` object to extract the first argument we provide to our app.

At the end of the `src/app.ts` file, let's change this `app("/cat/Games/1/");` with the following.

...

const pagePath = process.argv.length >= 3 ? process.argv[2] : "/popular-music";
app(pagePath);

The first argument we pass, the third in the `process.argv` as the two args are reserved for the node binary path, and the script file path.

So we check if we have more than 3 args, if so we take the third argument otherwise we use the `"/popular-music"` as the default page path.

Let's first install it as a dev dependency using `npm install live-server --save-dev` this package will help us quickly serve our public directory which will serve by default the `index.html` file.

Cause we're using typescript, we also need to install its types by `npm i --save-dev @types/live-server`.

So we will have a single npm script that does everything for us, from fetching and scraping the magnet links to serving them and opening them with our default torrent downloader.

After the installation is complete, we will create a new function `servePublicDir` which will do what we explained above. 

Just before our `app` function, we will define our function as shown below.

// imports
import liveServer from "live-server";

...

async function servePublicDir() {
  const params = {
    root: "public",
    open: false,
  };
  liveServer.start(params);
}

Nothing much to explain here, its self-explanatory we use the `start` method on the `liveServer` object we imported.

Now we just need to use it, after the `writePublicIndex` completed writing to our `public/index.html` file.

I also added some exception handling for our application, so we won't serve the page if the scraping wasn’t successful.

This is the `app` function content, after all the modifications we did.

...

async function app(path: string) {
  try {
    const html = await get(path);
    const $ = cheerio.load(html);

    const anchors = Array.from($(config.selectors.anchorLink));
    const hrefs = anchors.map((a) => $(a).attr("href")) as string[];

    let fetched = 1;
    const magnets: string[] = [];
    for (const href of hrefs) {
      const html = await get(href);
      const $ = cheerio.load(html);

      const magnet = $(config.selectors.magnetAnchorLink).attr(
        "href"
      ) as string;
      magnets.push(magnet);

      console.log("magnet fetched: ", fetched);
      fetched++;
    }

    await writePublicIndex(magnets);
    servePublicDir();
  } catch (e: any) {
    console.error(e);
  }
}

const pagePath = process.argv.length >= 3 ? process.argv[2] : "/popular-music";
app(pagePath);

Now you can use it with something like `npm run fetch /cat/Music/1/` and it should scrape all the torrent magnet links for the  **/cat/Music/1/** page.

A practical guide on how you can use the cheerio and curl npm packages to web scrape magnet links from a torrent website with Node.js and TypeScript.

<strong>Notice</strong>: Before we start I just want to say that this article is purely for educational purposes, please use it responsibly at your own risk.


<code>1337x.to</code> is a popular torrents website, with multiple categories: movies, music, and so on.


The goal of this article is to automate downloading torrents on a page like trending music, top-100 movies, and so on.


To download a torrent from <code>1337x.to</code> we need to go to the torrent details page, then click the magnet link, which will then be opened with your default torrent downloader.


Imagine you want to download 50 torrents on a page, you will have to visit each page separately, then download the files by clicking on the magnet link on that page.


I will focus on explaining the principles used to scrape the links from the <code>1337x.to</code>, so you will be able to reuse the same technologies and adapt them to your use case.


In this article, we will use Node.js with TypeScript support, even though it’s a small project I like to use TypeScript just for fun.


Codersteps
Is For Ambitious Learners

Learn it by doing it for Free with Practical step by step Series and Articles.

Featured Articles

How to add cookie consent banner with Next.js and Tailwind CSS

Codersteps

Next.js

Web Scraping magnet links from a website with cheerio and curl

Codersteps

JavaScript

How to set up and structure your full-stack Next.js application

Codersteps

Next.js

Looking for something in particular?

Recent Articles

How to add OpenGraph to your Next.js website

Next.js

How to add a Sitemap to your Next.js website

Next.js

How to add Google Analytics to your Next.js website

Next.js

Next.js file upload progress bar using Axios

Next.js

Let's build a file uploader with Next.js and formidable

Next.js

How I Integrated a Simple CI/CD for Next.js website on a VPS

Next.js