Jun 21 2018 • 6 min read
In this tutorial you will learn how to build a Silverbird Accra API on Glitch with Node.JS and Express.js.
Glitch (opens new window) is basically SoundCloud for simple* apps or bots. You can roll out a real web application easily and for free. No complex setup process, just create a new project or remix an existing one, write your code and you’re good to go.
I use Glitch a lot. It’s my go-to for experimenting on the numerous fun ideas I think up. I’ve made a twitter bot (opens new window) and a back-end for a polling app (opens new window) & shark (opens new window) all with Glitch. I honestly cannot recommend Glitch enough.
Table of contents
At this point, your glitch app should be running. To view your app click on the “Show” button.
NB: If there are no errors in your project, your “Show” button would have a green “Live” badge next to it. Otherwise it’d be a grey “Off” badge.
On the left side of the Glitch project webpage you should see a few folders and files Glitch created for you. Some of the files are only necessary based on the kind of application you’re building.
Since we’re only building an API we can get rid of the public and views folder. In server.js, we can also get rid of the code that sets up Express static files. At the end of clean up, your project should look something like this ( 👇👇).
For our API there are a few dependencies we’d need to add to our project.
After adding all the packages, your package.json should look like this 👇
We only need two (2) routes. GET /movies & GET /movies/:id.
I don’t like to litter my server.js file so for anything that isn’t related to setting up the server, middle-ware, and routes I like to create a separate file for it.
Create a scraper.js file using the “New file” button. It’s where we’re going to put the code that scrapes the Silverbird website.
Our most important function is scrapeMovies(). It downloads the HTML from "https://silverbirdcinemas.com/cinema/accra/" (opens new window), parses it with Cheerio and creates a list of movies with the relevant bits of the parsed HTML.
Downloading, parsing, and generating a list of movies on every request isn’t a trivial task. Since Silverbird Cinemas rarely change the listings on their website more than once a day, we can cache the scraping in other to speed up the response time of requests. The getMovies() function does exactly that.
When a user requests /movies/:id we need to provide more details about a movie. Details like a header image, director, cast, and synopsis. These details aren’t available on the Silverbird Accra landing page. Instead, they’re available on a separate detail page which we store in each movie’s url property during scrapeMovies(). The getMovieDetails() function will download the detail page, parse and generate a details object which we’ll then store in each movie.
Scraping the detail page for each movie for each request is time consuming so if the details for a movie does not exist, we’ll scrape and return the details. Otherwise, we return the already scraped details. The getMovie() function does that for us.
Now that we’re done with scraper.js we need to export our functions so we can use it in server.js. Exporting is as simple as 👇
module.exports = {
getMovies,
getMovie
};
We now need to connect the scraper to our routes so that when someone calls /movies or /movies/:id in server.js, we’ll return the results of getMovies or getMovie in scraper.js respectively.
To use the functions we exported in scraper.js we need to import them. Importing is as simple as 👇
const scraper = require('./scraper');
To test if our API works, open a new tab in your browser of choice — mine is Firefox Quantum. Navigate to <your_project_name>.glitch.me/movies
So that’s it for this tutorial. We learnt how to create a Node.js based website scraper from scratch. In the next tutorial we will be working on the accompanying front-end application with Vue.js.
Thank you for reading and don’t forget to check the next part soon!
You can browse the source code of this application on Github (opens new window).