Java Pills

🕐 Reading time: 8 minutes

This new article comes out of a small side project I built in Python, driven by a very concrete curiosity – how much do developer job ads in Italy really pay? The offers that land in your LinkedIn chats are one thing; the public ads, with the salary written in black and white, are another. I wanted a sample to extract a few little numbers from.

So I wrote a semi-automatic scraper for LinkedIn, targeted at software development roles (backend, full stack, architects, tech leads, DevOps…). It's a first draft – a proof of concept – and it collected 319 positions before LinkedIn started slowing me down. To be quick, I had the list of roles to search for generated directly by the AI: I briefly described my profile to it and got back a good thirty-odd related job titles, to feed as input to the search. The tool is therefore parameterisable: just change that list to point it at any other stack or professional figure.

📌 First I download, then I process

The guiding principle of the whole architecture is: first I download, then I process. The scraping saves only the raw HTML of the ads to disk. All the data extraction – title, company, technologies, salary – happens at a later stage, without ever going back to LinkedIn.

The pipeline is a sequence of decoupled scripts, each with a single task:

roles.py             → list of roles to search for
scrape_html.py       → PHASE 1:   downloads ONLY the HTML
clean_resources.py   → PHASE 1.5: discards ads from other sectors
parse_jobs.py        → PHASE 2:   HTML → structured data (JSON + CSV)
apply_corrections.py → PHASE 2.5: AI cleanup of the salary (optional)
to_excel.py          → PHASE 3:   JSON/CSV → formatted Excel

Why so much care in separating them? Because the network is the scarce resource and downloading is risky, whereas parsing isn't. Once I have the HTML on disk, I can re-process it as many times as I want: add a field, improve a regex, fix an extraction bug – all without a single new call to LinkedIn and without new ban risks. If I had mixed downloading and parsing in the same step, every little change to the extraction would have forced me to re-scrape from scratch. The logic is to keep the raw material untouched and immutable, and to rebuild on top of it each time.

📌 The scraping, and the anti-bot cat and mouse

Phase 1 is the most delicate: automation that's too aggressive can trigger a shadow ban, if not an outright ban of the account. It's best not to overdo it and – since it's a proof of concept – to use a secondary account: the measures I describe here counter the anti-bot but don't cancel it out. For each role the script builds LinkedIn's search URL, scrolls through the list of results (scrolling for lazy loading), gathers the links to the ads and downloads the full HTML of each one, updating a manifest that maps every file to its role and original URL.

A couple of sound implementation choices saved me several problems. The first is manual login: the user logs in in the visible browser, then the script carries on from there. No credentials to handle, and the session is indistinguishable from normal browsing. The second concerns robustness against layout changes: LinkedIn obfuscates and frequently changes its CSS classes, so I don't rely on the card "wrappers" – too fragile. Instead I gather all the /jobs/view/<id> links present on the page and extract the job_id directly from the URL, a pattern that never changes. Title and company I read from the <title> tag (Title | Company | LinkedIn). When you scrape, the principle is to anchor yourself to what's stable, not to what's convenient.

Then there's the throttling. "Aggressive" scraping gets detected and blocked, and the most typical signal of a bot is regularity: a constant, millimetric rhythm that no human would have. So the whole job is in breaking that regularity:

1. randomised delays between actions – not fixed pauses but random values (jitter) between a minimum and a maximum threshold

2. "human" pauses between one role and the next (≈ 15–30 s), also random, and micro scroll-pauses to simulate reading

3. progressive backoff: if several consecutive searches come back empty – a sign of a soft block – a long pause kicks in (≈ 90–150 s) and, if the situation persists, the script stops cleanly, inviting you to try again later, without hammering on an already-limited session

import random, time

def human_pause(min_s: float, max_s: float) -> None:
    """Randomized wait: adds jitter, breaks the constant rhythm."""
    time.sleep(random.uniform(min_s, max_s))

On top of this there's the supporting cast: a non-headless browser (with a visible GUI, not driven in the background in a hidden window – much more like a real user), a realistic user-agent, and the removal of the navigator.webdriver flag (a property the browser exposes when it's driven by automation, and which would otherwise scream "I'm a bot"). The scraping is also resumable: at startup it reloads the ads already downloaded and skips them, picking back up only from the missing roles. Outcome of the first campaign: 356 ads downloaded, then LinkedIn started returning empty pages (session throttling, not an account ban) and the collection stopped on its own, in a controlled way. Data intact.

📌 Cleaning the noise: the building architect under «API Architect»

Searching by keyword inevitably catches ads from other sectors. Searching for API Architect you get Architetto edile (building architect), Geometra (surveyor), BIM Coordinator; under Team Leader I even got a Powder Coat Team Lead from a manufacturing company. Phase 1.5 (clean_resources.py) classifies each ad by its title into keep / remove / review, with rules based on positive IT signals (developer, software, engineer, backend…), signals from other sectors (building, construction site, construction, surveyor…), and special handling of the ambiguous term architect/architetto – considered IT only in a software/cloud/integration context. Result? From 356 to 319 cleaned ads, 37 removed, almost all of them building "architects" that ended up in the API Architect search.

📌 Extracting meaning: the case of the salary

Phase 2 turns each HTML file into data. I extract the visible text, then isolate the ad's description between two stable markers ("Informazioni sull'offerta di lavoro" → "Offerte di lavoro simili") to throw away header, sidebar and footer and reduce the noise. Regexes and heuristics run on the clean text.

The most delicate field is the salary, and here the point isn't reading a number: it's understanding what that number means. It's an entirely semantic distinction, and I tackle it with context-driven regexes and heuristics:

• explicit range (tra 30.000 e 45.000, even when dirtied by currency and decimals)
• fino a / massima 35.000 (up to / maximum) → only ral_max – it's a ceiling, not a minimum!
• a partire da / da 33.000 (starting from / from) → only ral_min – it's a base
• flat amount (RAL €35.000) → ral_min = ral_max

Treating an "up to 35,000" as if it were the minimum offered would completely skew the statistics. And then you need the common-sense filters: discarding non-salary numbers (e.g. "21,000 employees") and amounts in foreign currency (£/CHF/USD…), not comparable with the Italian market. In practice I look for all the plausible amounts (≈ 15k–300k), keep only those with a nearby salary anchor (€, "RAL", "lordo", "retribuzione"…) and decide the direction from the context. Another interesting part is teaching the regexes the real formats of the ads: not just 30.000, but also 30-40k, 28K-35K and the English-style thousands €45,000 – very common notations that a naive match skips. By handling them, salary coverage went from 24% to 36% of the ads, without a single line of AI.

There remains, though, a grey area where pattern matching isn't enough: the "multiple band" ads. A "Graduate software engineer" that publishes both the entry band (≈€66k "with little experience") and the senior one (€107k–€188k): the regex, seeing two increasing, nearby numbers, grabs the wrong pair. Here I did a semantic cleanup pass with the AI – I gave Claude the doubtful descriptions, asking it to choose the right band for the role. The balance is the answer to the question "do you need AI?": out of 116 ads with a salary the regex was correct in 108, the AI fixed the 8 with multiple bands – the graduate above and a few consulting firms that put two grades (junior and senior) in the same ad. For the bulk of the work a well-made regex is enough; the AI is needed in that handful of cases that also require understanding the context and not just recognising a pattern. This way the base pipeline stays AI-free and without pay-per-use APIs – usable by anyone – with the AI as an optional refinement where the heuristics give way. The output is two files: jobs.json (the rich source – technologies as an array, salary as numbers) and jobs.csv ready for Excel. Phase 3, finally, generates a formatted jobs.xlsx with automatic filters, sortable columns and clickable links to the original ads.

📌 The numbers (finally)

Out of 319 positions, the field coverage already tells us something: title, company, location, work mode and publication date are, obviously, at 100%; the required years of experience at ~54%; the number of applicants at ~40%. And the salary? Present in just over a third of the ads (~36%, 116 out of 319).

Let's get to the meat: where the salary was there, here's the picture (€, gross annual salary):

And the extreme values of the sample, to give an idea of the spread:

To calculate them I added formulas directly to the output Excel sheet, and next to the means I put the medians: with a small sample and a few off-scale ads (a scale-up that pays at international levels) the mean gets pulled around, the median doesn't. An interesting detail: almost all the ads that state a minimum also state a maximum, and vice versa. As a rule there's a range, not a flat value.

📌 Pay transparency

Two observations, beyond the code. The first is about pay transparency: despite the regulation that came into force on 7 June, the salary appears in just over one ad in three. It has to be said that publishing it in the ad isn't mandatory, but the figure remains interesting. The second is an observation about the sample: the salaries I measured are on average lower than the offers I get in chat, where the offers are more targeted to the profile (senior, in my case). The scraper, instead, caught a mix that also includes juniors and middles, bands where the values are naturally lower. It's an unbalanced sample: the averages should be read as indicative of the sample, not of the market.

I know this prototype's limits well: the throttling that breaks the collection, the partial salary coverage (~36%, but that's a limit of the source data, not of the parser), the heuristic parsing that can still get synonyms and creative phrasing wrong. The next step is to re-run it with the anti-bot improvements I've added and widen it by an order of magnitude: aiming for ~3,000 ads to have, at the same coverage, over a thousand salary ranges instead of today's ~116. Only then will the averages say something about the market, and not just about this sample. The natural evolution, then, is to extend the use of AI – today limited to refining the salary in the tricky cases – to extracting all the fields, and then to keep historical data over time and normalise the technologies. A local model, or pay-per-use APIs (weighing up the costs and cleaning the HTML first, often heavy, to reduce the tokens). But even as it is – a Python draft of a few hundred lines – it scratched my curiosity with real numbers in hand, instead of going by gut feeling. And I'd say a proof of concept doesn't have to do more than that.

The code is public on GitLab, link to the repo: linkedin-scraper. Inside the first_extraction folder you'll also find a sample extraction, with the links to the over 300 positions downloaded, if you fancy taking a look at the raw data.

If you enjoyed the article leave a like, and see you at the next article! ☕