Enrichment

Turning an address into a device, a hostname into a company, an answer into a city.

internalsHow it is built. Never needed to run it.page 15 of 24

Turning 192.168.4.159 into “Kitchen iPad” and sb.scorecardresearch.com into “Comscore”. All of it happens locally, against files on disk.

In plain terms

A raw log line says an address asked for a hostname. Neither half is something you recognise, and a dashboard full of numbers and tracker domains is a dashboard nobody reads.

Four reference files sit on disk — who owns which domain, which country an address is in, which network it belongs to, and which manufacturer a device’s hardware address belongs to. Every row is matched against them as it is written. Nothing is looked up online, so your household’s browsing is never sent anywhere to be identified.

The four datasets#

Dataset	Size	Gives	Licence
AdGuard companiesdb	~1.5MB	Domain → tracker → company → category	CC BY-SA 4.0
sapics/ip-location-db	~40MB	IP → country, IP → ASN	Public domain / CC0
MaxMind GeoLite2 City	~45MB	IP → city and coordinates	GeoLite2 EULA, operator-supplied
IEEE OUI registry	~4MB	MAC prefix → hardware vendor	Public

When enrichment happens#

01
Once per newly-seen domain, not once per query
Attribution walks the hostname’s suffixes against the tracker table until one matches. The result is stored on the domains row and never recomputed.
02
Once per answer, at write time
The first resolved address is geolocated and denormalised onto the query row. Doing it at read time would mean opening a 45MB MMDB file for every row in a live stream.Skipped entirely. Pi-hole’s log carries no answer addresses, so there is nothing to geolocate — which is the whole reason the destinations map is absent rather than empty on this resolver.
03
On device sync, from the resolver
Names, MACs and DHCP lease hostnames come from AdGuard Home’s client and DHCP APIs. The MAC’s OUI prefix gives the hardware vendor.Skipped on Pi-hole, which reports the same facts through different endpoints that are not read yet. Less is lost than that suggests — see below.

stats.g.doubleclick.netno match
g.doubleclick.netno match
doubleclick.netno match

Each suffix is looked up in turn and the walk stops at the first hit. The result is written onto the domain row and never recomputed, which is why a domain first seen before a dataset update keeps its old attribution until it is seen again.

Where a name comes from#

Four sources, in priority order: the name you set in this dashboard, AdGuard HomePi-hole’s own client name, the DHCP lease hostname, then the raw address.

On AdGuard Home, naming a device here writes back, so the two stay in step rather than disagreeing.

Pi-hole gets there differently and loses less than the skipped sync suggests: FTL puts the client’s reverse-resolved hostname on every query, so identity arrives with the log itself rather than needing a second call. What it costs is the MAC, and therefore the hardware vendor — a Pi-hole install names devices but does not label them “Apple” or “Sonos”.

The registrable-domain approximation#

Grouping needs eTLD+1, and the correct way to compute it is the Public Suffix List: 9,000 entries that need periodic refreshing. The product approximates instead, and says so.

stats.g.doubleclick.net  →  doubleclick.net     ✓
bbc.co.uk                →  bbc.co.uk           ✓ (special-cased)
foo.s3.amazonaws.com     →  amazonaws.com       ~ (PSL would say s3.amazonaws.com)

It is used for grouping in the interface and never for a security decision, which is the trade that makes the approximation acceptable.

Cost#

The IP tables are SQLite with the range start as the primary key, so a lookup is WHERE start_ip <= ? ORDER BY start_ip DESC LIMIT 1: a single b-tree descent and a bounds check. City data is read straight from the memory-mapped MMDB rather than loaded into SQLite, because 3.5 million city ranges would take the database from 90MB to something no longer sensible on a Pi.

Answer addresses repeat heavily, since a handful of CDN endpoints back most of a network’s traffic, so a small in-memory cache removes almost all of the index scans during backfill.