In this blog series I want to create a a web scraper that gathers job listing informations from real sites around my location, Eastern Europe.
Targets of the scraper
My primary goal is to target HelloWorld.rs, Serbia's biggest job listing page, since I haven't found a scraper that already mines data from it. Without abusing their terms of use, I will run through what offers they have.
Later I plan to expand my scrapers scope by forking an already existing Joberty scraper and if I have the time, I'll add LinkedIn to the targets as well.
Purpose of extracted data
- Faster and more precise job results without distraction
- Filter out fake and weak listings
- Gather information about what local companies looking for
Obstacles
- Need to bypass blockers such as reCaptcha
- Target website layout can change drastically (e.g. design update)
- Network outages, unexpected events (e.g. all my IP addresses get blocked)
Technical part
I'm going to use Playwright for scraping inside NestJS as the backbone of the whole project. I will use SQLite to store the scraped data, since it is simple to setup and later one can easily switch to another RDBMS. Futhermore, I will use proxy servers as well to evade complete blockage. Later on, I might integrate AI to make job description filtering easier.
The project will be available on Github for those who want to check it out.
Next up: Defining the data model