← back to all birds

How BirdView's data is built

Every number on a BirdView page comes from a published source through the commands shown below. Each source and table is a box listing its fields; each arrow says what a step does: a filter, a join, or a model.

How to read the fields. Each box lists one field per line, with one mark: ★ it reaches a page (and may also be used), · the pipeline uses it without showing it (a join key, a filter, a denominator), and ✗ nothing downstream uses it. Unused fields are listed so an auditor can see they were not forgotten. Italic text after a field is a note; italic text under a box's name says where the box lives.

Square boxes are data: a published source, a table in code, or a file a command writes. Rounded boxes are things that run: a command, or a numbered step inside birdview-ingest. The steps are not files; the ingest runs all eight in memory and writes species.json, and the counties apart in counties.gob.gz. Dotted arrows are facts copied into code rather than data read by a program. The diagram's corner repeats this legend.

Diagram: How BirdView's data is built

Select the diagram to open it full size.

GitHub draws the diagram from this source; BirdView serves a pre-rendered picture of it, so the page needs no network.

The steps

Scope comes from recurrence. birdview-status asks GBIF which names each land region records, then fetches the dates behind each species in its three regions with the most records. From how those dates fall into episodes and years it assigns a status: established, rare, vagrant, historical, or absent. Established and rare species are in scope. eBird's dates count everywhere; iNaturalist's count only in a region and month where eBird is confirmed to withhold the species. status.Declared, a short list in code, adds species the open records cannot speak for, each with its reason. See where the data comes from and the decision to count eBird first.

Open ocean or near shore comes from distance to shore. birdview-sea reads the offshore-cells download's eBird records, drops each 0.1° cell whose center lies on Natural Earth's land, and bands the rest by distance to shore, per species, into data/sea-distance.tsv. The ingest, after the regions, gives every bird with Marine the refinement that follows: open ocean when a quarter of its records on water lie more than 10 km out, sea near shore when fewer do or when 80% of all its records lie on land or within 10 km; see where the data comes from.

Proportions are reviewed by genus. The neck ratio of step 4 comes from the family table in the code, and the head and body from mass, unless data/proportions-review.tsv has a row for the bird's genus or family; a row may also scale a modeled wingspan in step 6, never a measured one. Rows are proposed, set by judgment, until a source other than photographs supports them; photographs foreshorten the parts the figure draws (decision).

Region counts come from the same source. birdview-regions asks GBIF for every name recorded in every region, for the whole year and for each month. For species mostly seen at sea, it also counts off-land records inside each of BirdView's sea boxes. A species is present in a region when it reaches 1 record per million of all bird records there. eBird's records count everywhere; iNaturalist's are added only where eBird's Sensitive Species list names the species and the records confirm that eBird is withholding it. Records where a species stays under the bar are kept, marked ~: the ingest counts them in the 1 in N and leaves the species off that region's list.

Names are normalized before anything is matched. Both commands map every published name onto the AviList name by the same rules: a subspecies is reduced to its species, genus moves and names a source files under another species are mapped through the tables in internal/taxonomy, and a hybrid matches no species. birdview-regions applies these rules through taxonomy.Occurrence; birdview-status applies the same tables in its own code and drops hybrids, which have no status. Where a name table applies to a species, its page says so under Recorded as.

The ingest joins everything into one file, in eight steps. It keeps the in-scope species (step 1), names them and their families from AviList (2), measures them from AVONET (3), models the segments no source measures (4), assigns habitat and diet (5), takes wingspan from Fox where it was measured and estimates it where it was not (6), records the species it does not draw and why (7), and attaches the region counts (8). Counties are compiled apart, into counties.gob.gz, which the server reads only with -counties and only for the counties chosen. Every metric carries its provenance (measured, near-match, or modeled), and the page shows which. See what is measured and what is modeled.

AVONET has three taxonomy sheets, and step 3 chooses between them. A species' own row comes from the eBird sheet unless OwnRowSheets names a narrower one: the eBird sheet's Larus argentatus contains every herring gull, so European Herring Gull is read from the BirdLife sheet. A species missing from the eBird sheet is looked up under its own name in the other two sheets, then under the older names in Renames; either way the row is its own, so it counts as measured. A species with no row of its own borrows a relative's, labeled near-match, from the sheet NearMatchSheets names where one holds a narrower concept of that relative. If a named sheet lacks the row, the build stops rather than fall back to a broader one.

Escapes are flagged, not dropped. status.Exotic lists in code the birds whose records here are escapes or birds kept by people. The ingest marks each with its reason; the server leaves them off the index and its controls, lists them at /excluded, and keeps their pages.

The server does the last arithmetic. The 1 in N on a tile and the month filter depend on the regions and months a reader selects, so they are computed per request from the counts and totals in species.json. The server also finds species drawn from the same AVONET row, from their provenance, and notes the shared measurements on both.

birdview-boundaries sits outside this pipeline. It writes data/region-boundaries.json, the region outlines the Here button tests a reader's position against, from Natural Earth's states and provinces and BirdView's own sea boxes. The server reads it directly; nothing in the dataset depends on it.

Where to go next

Rendered from BirdView's repository and served here, so citations do not depend on the network.