In this article, we learn about the installation of node unblocker. You will need a web proxy to route your requests through if you want to surf the web without being seen, scrape the web without getting blocked, or get to the information that is only available in certain places. You can buy a ready-made web proxy, which, depending on your needs, may be expensive and overkill. You can also make your own with a web proxy library like node-unblocker, which lets you get around internet filters and scrape sites.
What is the Node Unblocker?
Node Unblocker is a general-purpose library for proxying and rewriting external websites, which allows you to do the following:
- When viewing a website, conceal your true identity.
- Scrape information from a website without being banned.
- Use a website to access geo-restricted material.
Node Unblocker is a Node.js library built on top of the well-known Express framework. This makes it incredibly familiar and simple to use for any Node.js developer who has previously used Express.
Scrapers and browsers communicate with Node Unblocker using the REST API endpoints it provides.
Simply attach the URL of the page you wish to obtain to the /proxy/endpoint, and the Node Unblocker server will retrieve the content using the machine’s IP address.
Node-Unblocker is a fast and effective proxy alternative because all data is processed and sent to the client only when it is needed.
How to Install and Run a Node-Unblocker Server
Setting up a Node-Unblocker server is simple, especially if you’ve used Express before.
Step 1: Install Node-Unblocker and Express
- The first step is to put our dependencies in place.
- Express is a popular, simple web framework for building servers using Node.js.
- The package name for node unblocker is unblocker.
Step 2: Build Our Node-Unblocker Server
- Create an app.js file in our folder with the following code:
- When started, this simple server will run on localhost:8080 or the port you set in the PORT environment variable.
- In addition, we added the event handler.on(‘update’, unblocker.onUpgrade) to our app.
- listen() This tells our proxy what to do if, for example, the connection protocol changes from HTTP to WebSocket.
Step 3: Start and Run Our Server
- To start our server, simply execute:
- And we can put our Node-Unblocker server to the test by sending requests to it using the URL format:
- You may try it on your terminal by typing curl:
- Alternatively, in your browser, put the following URL into the URL bar:
- When you send a request through your Node Unblocker server, the request will now include the IP address of the computer where the server is running.
- Since you are running the node-unblocker server on your own computer, the IP address will not change.
- To change your IP address, you must deploy your unblocker server to a remote server, as we will see later.
Install the Node-unblocker Proxy Server
We can now use our simple node-unblocker server as a proxy to pass our web scraping and browser traffic through.
- To do this, we simply install our server on a distant server and transmit queries to it.
- You may install your proxy server anywhere. However, Heroku is one of the easiest solutions because you can utilize their free plan.
Setup Our Server for Deployment
Before we can put our proxy server on Heroku (or any other hosting service) and use the IP address of the Heroku machine, we need to update our package.
- json file with server configuration information.
- Here are several examples:
- So Heroku knows which Node.js version to use.
- start script—which instructs Heroku on how to launch our proxy server.
- We should also build an a.gitignore file to avoid publishing extraneous files and dependencies to our Github repositories and Heroku server.
Heroku publishing
Create a free Heroku account, install the Heroku CLI Tool, and create a new app on the Heroku dashboard if you haven’t already.
- Then, using the command line, log into your Heroku account and follow the steps to generate a new SSH public key.
- Next, open Git in the directory where you made the server and set up the Heroku remote to point to the app you just set up in the Heroku dashboard.
- In this example, we referred to the app as
- Finally, we must use Git to push our code to Heroku.
- Now, with node unblocker installed, we can navigate to https://node-unblocker-demo.herokuapp.com/proxy/.
- https://www.amazon.com We will route our request through our Heroku server, and when Amazon’s server receives the request, it will see a Heroku IP address.
- The graphic above shows that the location Amazon believes we are in has changed from Germany to the United States since the request now utilizes the Heroku server’s US IP address rather than my machine’s IP address.
Construct a Node Unblocker Proxy Network
We can scale this up and build our own proxy network by putting node-unblocker servers on several computers.
Install the server code on many Heroku servers or hire cheap remote servers like Vultr (servers start at $2.50 per month), and then set your web scrapers or bots to use a random server from this proxy network with each request:
For example, using Python Request Scraper, you might execute the following:
Personalizing Our Node-Unblocker Server
Okay, you now understand how to set up and install a basic proxy server.
However, by utilizing Node-Unblocker middleware, we can have access to a plethora of additional functionality that may be used to make our proxy servers more powerful and useful when web scraping or employing bots. https://www.amazon.com
Middleware for Unblocking Nodes
“Middlewares” are simple functions that allow us to change requests and answers as they are received by our proxy server.
The majority of Unblockers’ functionality is provided via middleware. However, we can simply set our own custom middleware to replace or supplement the default middleware.
By adding them to the request middleware or response middleware arrays, we may create our own custom request or response middleware:
To alter Unblocker’s default middleware, set standard middleware: as false and then only activate the middleware you want.
The Benefits of Node Unblocker Middlewares
Using Requests & Responses Middlewares with Unblocker is helpful because you can easily abstract a lot of the details of your proxy setup to your proxy server and then let your scrapers and browsers send requests there.
Instead of keeping and managing the proxy header and user-agent settings in each scraper, this logic would be stored in your proxy server and applied to every scraper request.
you are making your proxy architecture much easier to maintain.
Request Middleware Illustration
- The Request Middlewares feature is often used to add custom headers and user agents to requests.
- Every request might have a user-agent like this:
- You can even tell it to send certain headers when you go to certain URLs if the domain needs custom headers.
Response Middleware as an Example
- Response middleware is used to do things like changing the response you get from the target website.
- In this example, we will look for and replace the following content in the HTML response: