Automation
Jun 3, 2024

Web Scraper and Airtable Integration with Next.js

Web Scraper and Airtable Integration with Next.js

Low-code tools are going mainstream

Purus suspendisse a ornare non erat pellentesque arcu mi arcu eget tortor eu praesent curabitur porttitor ultrices sit sit amet purus urna enim eget. Habitant massa lectus tristique dictum lacus in bibendum. Velit ut viverra feugiat dui eu nisl sit massa viverra sed vitae nec sed. Nunc ornare consequat massa sagittis pellentesque tincidunt vel lacus integer risu.

  1. Vitae et erat tincidunt sed orci eget egestas facilisis amet ornare
  2. Sollicitudin integer  velit aliquet viverra urna orci semper velit dolor sit amet
  3. Vitae quis ut  luctus lobortis urna adipiscing bibendum
  4. Vitae quis ut  luctus lobortis urna adipiscing bibendum

Multilingual NLP will grow

Mauris posuere arcu lectus congue. Sed eget semper mollis felis ante. Congue risus vulputate nunc porttitor dignissim cursus viverra quis. Condimentum nisl ut sed diam lacus sed. Cursus hac massa amet cursus diam. Consequat sodales non nulla ac id bibendum eu justo condimentum. Arcu elementum non suscipit amet vitae. Consectetur penatibus diam enim eget arcu et ut a congue arcu.

Vitae quis ut  luctus lobortis urna adipiscing bibendum

Combining supervised and unsupervised machine learning methods

Vitae vitae sollicitudin diam sed. Aliquam tellus libero a velit quam ut suscipit. Vitae adipiscing amet faucibus nec in ut. Tortor nulla aliquam commodo sit ultricies a nunc ultrices consectetur. Nibh magna arcu blandit quisque. In lorem sit turpis interdum facilisi.

  • Dolor duis lorem enim eu turpis potenti nulla  laoreet volutpat semper sed.
  • Lorem a eget blandit ac neque amet amet non dapibus pulvinar.
  • Pellentesque non integer ac id imperdiet blandit sit bibendum.
  • Sit leo lorem elementum vitae faucibus quam feugiat hendrerit lectus.
Automating customer service: Tagging tickets and new era of chatbots

Vitae vitae sollicitudin diam sed. Aliquam tellus libero a velit quam ut suscipit. Vitae adipiscing amet faucibus nec in ut. Tortor nulla aliquam commodo sit ultricies a nunc ultrices consectetur. Nibh magna arcu blandit quisque. In lorem sit turpis interdum facilisi.

“Nisi consectetur velit bibendum a convallis arcu morbi lectus aecenas ultrices massa vel ut ultricies lectus elit arcu non id mattis libero amet mattis congue ipsum nibh odio in lacinia non”
Detecting fake news and cyber-bullying

Nunc ut facilisi volutpat neque est diam id sem erat aliquam elementum dolor tortor commodo et massa dictumst egestas tempor duis eget odio eu egestas nec amet suscipit posuere fames ded tortor ac ut fermentum odio ut amet urna posuere ligula volutpat cursus enim libero libero pretium faucibus nunc arcu mauris sed scelerisque cursus felis arcu sed aenean pharetra vitae suspendisse ac.

The Problem

Imagine you want to keep track of the templates available on the Webflow website. You’re interested in monitoring when new templates are added and when existing templates are removed from the homepage. Manually checking the website and updating a spreadsheet would be a tedious and time-consuming task. That’s where our Next.js web scraper and Airtable integration come into play.

The Solution

We’ll build a Next.js application that scrapes the Webflow templates page, extracts relevant information, and stores it in an Airtable base. The application will consist of two main components:

  1. Webflow Scraper API Route:
  • The webflow-scraper.js file defines an API route that fetches the HTML content of the Webflow templates page using the axios library.
  • It then uses the cheerio library to parse the HTML and extract the desired data, such as the template name, URL, and publish date.
  • The scraped data is returned as a JSON response.

2. Compare and Update API Route:

  • The compare-and-update.js file defines another API route that compares the scraped data with the existing records in the Airtable base.
  • It fetches the scraped data from the Webflow Scraper API route and retrieves all records from the specified Airtable table.
  • It compares the scraped data with the Airtable records to identify new templates and templates that have been removed from the homepage.
  • For new templates, it creates new records in the Airtable table.
  • For templates that exist in Airtable but are no longer on the homepage, it updates the “Last Time on Home Page” field with the current date.

Airtable Integration

To integrate with Airtable, we use the airtable package and configure it with our Airtable API key and base ID. We define the necessary field IDs and table name in the lib/airtable.js file, making it easy to reference them throughout the application.

Scheduled Execution

To automate the scraping and comparison process, we utilize the Vercel Cron Jobs feature. By adding a vercel.json file with the desired cron schedule, we can configure the compare-and-update API route to run at specific intervals (e.g., every 12 hours). This ensures that our Airtable base stays up to date with the latest changes on the Webflow templates page.

Favicon - AI X+ Webflow Template

At Create Something, our mission is to automate every integration through innovative AI and API solutions. Explore our popular papers to gain insights into the latest advancements and practical applications in automation and integration.

Subscribe to our weekly newsletter

Thanks for joining our newsletter.
Oops! Something went wrong while submitting the form.