find-commons

Resolves wikimedia_commons tags for OSM objects via Wikidata and Wikipedia.

Usage

Pipe OSM object references into stdin, one per line:

echo "node/12969207609" | ./find-commons

Accepted input formats: full URLs (http://osm.org/node/123), short form (node/123, way/456, relation/789), or bare numbers (treated as node).

Both streams can also be named explicitly, so a run needs no shell redirection:

./find-commons -i testurls.txt -o results.html

- (the default) keeps stdin and stdout. The output file is created and truncated only on the first byte written, so a run without a single result leaves an existing file untouched. A missing input file exits with code 1 before the output file is touched at all, a failing write exits with code 1 as well.

The default output is an HTML table with one row per object, and every row is built for checking it against the live pages:

• the object cell links to openstreetmap.org/<type>/<id>
• the source cell links to the wiki page the value came from (the Wikidata item for P373/P18, the article for wikipedia:de)
• the value cell links every component of the value to its own Commons page; a verified tag (ok) shows the value the object already carries, comma separated when the tag holds several
• each row whose value still has to be written carries an L0 link that opens the object in the Level0 editor with a prefilled changeset comment (add <value> as wikimedia_commons) and copies the full wikimedia_commons=<value> line to the clipboard. Rows that only report a status (ok) or a reason stay link free. Level0 only honors the url and comment parameters with an active Level0 login session.

Objects without a value get their own row as well: the source cell carries the concrete reason, linked to the page that was searched, for example nothing found in wikidata Q123 (P373, P18).

The footer sums the run up: the total number of objects, how many of them carry a Commons value and how many do not, next to the build revision, the generation time and the duration of the run.

With -tsv the output is tab-separated instead:

node/12969207609	P373	wikimedia_commons=Category:Stolpersteine in München

Lookup order

1. P373 on Wikidata (Commons category), and the item's commonswiki sitelink, which some items keep instead of the claim
2. P910 — the "main category" chain on Wikidata, category or sitelink at every hop
3. Wikipedia article — the commonscat template in the article source, falling back to the article title when the template names no category, else the first commons.wikimedia.org/wiki/Category: link
4. P18 on Wikidata (image file)

Every category a step finds is checked against Commons first: a category page that does not exist counts as no hit and the chain moves on.

No match means no output line in -tsv mode; the reason goes to stderr there, behind -verbose. The HTML table instead shows the reason as a row (see above), so one document accounts for every input URL.

When the input contains no OSM reference at all, neither output mode writes anything to stdout. The default HTML mode additionally prints no OSM objects found in input on stderr, which -quiet drops.

Every run opens with a line that repeats where it reads from, where it writes to and every option that was passed on the command line, so a job over a large list shows its settings before the first request goes out. Options at their default stay off the line:

2026/10/25 23:37:03 main.go:241: config: in=testurls.txt out=results.html format=html -refresh -verbose

Every stderr message carries date, time and the calling line from the standard logger. One line per object without a value would flood the console on a large list, so those reasons need -verbose and link to their object in the short form:

2026/10/25 10:37:28 main.go:185: http://osm.org/way/30915563 nothing found in wikidata Q98741107 (P373, P18)

Without the flag a run reports run level messages only (see the progress lines below). Logging writes to stderr only, stdout stays reserved for the document.

Runs with more than 100 objects additionally report their progress: a start line with the object count, one line every 10 percent of the run and one line every five minutes, so even a list of hundreds of thousands speaks up right away. The status line carries the processed objects against the total and how many of them carry a Commons value versus how many do not:

2026/10/25 11:30:00 main.go:221: progress: 40% 400/1000 objects, 123 with commons value, 277 without

-quiet keeps stderr down to the messages that end the run with exit code 1: per object reasons, progress lines, the startup line and the empty input note disappear, while open input, read error and write error stay, and it wins over -verbose. The document on stdout is never affected.

Response cache

Responses from all four APIs are cached on disk for cacheTTL (7 days, defined once in cache.go). The cache lives in ./cache/ relative to the working directory, one subdirectory per host:

cache/openstreetmap.org/  cache/wikidata.org/
cache/commons.wikimedia.org/  cache/wikipedia.org/

File names are SHA-256 digests of the request URL, the first 8 bytes of a file hold the fetch time. Only successful (HTTP 200) responses are stored; errors are always fetched fresh. A run served entirely from the cache performs no requests at all, which also means the delay never fires, because it is a pause before every real request. Pass -refresh to ignore and overwrite cached entries.

Verification of existing tags

Objects that already carry a wikimedia_commons tag are checked against Commons instead of being resolved from scratch. The status goes in the source column:

source	meaning
ok	value exists; HTML shows the tagged value, the TSV third column stays empty
missing	value gone, no replacement found (third column empty)
renamed	file was renamed; third column holds the new name
fix	value has no File:/Category: prefix; third column holds a fresh value
replace:P373, replace:wikipedia:de	third column holds the proposed replacement

Only the first component of a multi-value tag (File:X;Category:Y) is checked; the HTML value cell still lists all of them, each linked to its own page.

Build & test

make build
make test
make coverage   # fails below 88%