EcoVal Changelog v0.5.0

How the calculations are built, part 1: foundations and fitted models

This is the developer's view of how EcoVal calculates. docs/calculations.md (and its water and city siblings) explains what each number means and where its inputs come from. This document explains how the number is produced in code: the formula, then the C# that implements it, then how the pieces fit together.

Part 1 covers two things:

  1. The foundations every calculation stands on: turning an outline into pixels, summing rasters, and the types that carry a result (ranges, tiers, availability).
  2. Every place where a model is fitted to data: what kind of fitting it is, how it is trained, where the result is stored, and how the API uses it.

Parts 2 to 6 (forest, water, city, the landing calculator) are listed in README.md.

All file paths are relative to backend/. Code excerpts are copied from the source and shortened; // … marks what is left out.

Contents


1. The shape of every calculation

Every result follows the same path. The endpoint does the I/O (reads rasters and the database), then hands plain values to a calculator that is a pure function: the same inputs always give the same outputs, and nothing inside it touches a file or a database.

flowchart LR
    A["Browser<br/>GeoJSON polygon, WGS84"] --> B["PolygonReader<br/>validate, reproject to SWEREF 99 TM"]
    B --> C["Lookups and raster readers<br/>(I/O: PostGIS, GeoTIFF)"]
    O["Options classes<br/>(parameters with sources)"] --> D
    C --> D["Calculator<br/>(pure static methods)"]
    D --> E["Contract records<br/>ValueRange, tiers, codes"]
    E --> F["JSON response<br/>(OpenAPI contract)"]

The split matters for three reasons:

  • Testing. A calculator can be tested with hand-written inputs and hand-checked expected values, with no raster files on disk.
  • Reuse. EcoVal.Calibration calls the same calculator methods the API calls. A fitted parameter is therefore fitted against exactly the code that will later use it.
  • Traceability. Every parameter lives in an options class with a comment naming its source, so the answer to "where does 1.5 km come from?" is one "go to definition" away.

A typical calculator signature shows the pattern. All data arrives as arguments, parameters arrive as options:

// EcoVal.Api/Features/Forest/Recreation/RecreationCalculator.cs
public static double[] EverydayShares(
    double forestHectares,
    IReadOnlyDictionary<(double X, double Y), double> polygonForest,
    IReadOnlyList<ForestSquare> forest,
    IReadOnlyList<PopulationNear> population,
    double decayKm,
    RecreationOptions options)

2. From outline to pixels

2.1 Reading and reprojecting the polygon

The browser sends coordinates as longitude and latitude (WGS84). Areas and distances cannot be computed in degrees, so the first step is to reproject to SWEREF 99 TM (EPSG:3006), where one unit is one metre.

PolygonReader.Read (EcoVal.Api/Spatial/PolygonReader.cs) checks the input in order and returns the first problem as a stable error code, never as a sentence:

if (ring[0][0] != ring[^1][0] || ring[0][1] != ring[^1][1])
{
    return PolygonReadResult.Invalid(PolygonErrors.RingNotClosed, "Each ring must end at its starting position.");
}
if (ring.Any(p => p[0] is < MinLongitude or > MaxLongitude || p[1] is < MinLatitude or > MaxLatitude))
{
    return PolygonReadResult.Invalid(PolygonErrors.OutsideSweden, "The polygon must lie within Sweden.");
}
// …
var geometry = projection.ProjectPolygon(rings);
var validity = new IsValidOp(geometry).ValidationError;

The projection itself is a transverse Mercator transform done by the ProjNET library (EcoVal.Api/Spatial/Sweref99Projection.cs); EcoVal only supplies the projection's definition (central meridian 15°, scale factor 0.9996, false easting 500,000 m). Geometry operations (validity, intersection, area) come from NetTopologySuite.

2.2 Zonal statistics: summing a raster inside the polygon

Most forest numbers start as a raster: a grid of 10 m pixels where each pixel holds one value, for example the timber volume in m³sk per hectare. "Zonal statistics" means summarising those pixels inside a zone, here the polygon.

The formula. Let pixel pp have value vpv_p, and let sp∈[0,1]s_p \in [0, 1] be the share of the pixel's area that lies inside the polygon. Pixels without data are left out of the sums and counted separately.

S=∑p ∈ validsp vpN=∑p ∈ validspvˉ=SNS = \sum_{p \,\in\, \text{valid}} s_p \, v_p \qquad N = \sum_{p \,\in\, \text{valid}} s_p \qquad \bar{v} = \frac{S}{N}

For a raster that holds a per-hectare quantity, the total over the polygon is the sum times the area of one pixel in hectares, where ℓ\ell is the pixel size in metres:

total=S⋅ℓ210 000\text{total} = S \cdot \frac{\ell^2}{10\,000}

With 10 m pixels each pixel is 0.01 ha, so a pixel showing 200 m³sk/ha that lies fully inside contributes 2 m³sk.

How a boundary pixel is counted

The implementation is ZonalStatistics.Compute (EcoVal.Api/Spatial/Rasters/ZonalStatistics.cs). It has three steps:

var grid = PixelGrid.Of(tiles);                                  // the pixel grid all tiles share
var window = grid.WindowCovering(polygon.EnvelopeInternal);      // the pixel rectangle around the polygon
var pixels = Mosaic(tiles, grid, window, noData);                // read that rectangle into one short[]
return Summarise(pixels, grid, window, polygon, noData);         // weight and sum
  1. Mosaic. A raster comes as many files (tiles). ReadMosaic reads the window from each tile into one array. Where county files overlap, the first tile with data for a pixel wins, so nothing is counted twice.

  2. Coverage. CoverageCells works out sps_p for every pixel. Computing a geometric intersection for every pixel would be slow, so it uses two tricks:

    • It first marks the pixels the boundary passes through (BoundaryPixels). Only those get a real intersection: polygon.Intersection(box).Area / pixelArea.
    • Every other pixel is wholly inside or wholly outside. A scanline decides which: walking west to east along the row's centre line, each time the line crosses the boundary the state flips between outside and inside (the even–odd rule).

    The cost therefore grows with the length of the boundary, not with the polygon's area.

  3. Summarise. The loop is the formula above, almost symbol for symbol:

double sum = 0, valid = 0, noDataWeight = 0, zeroWeight = 0;
foreach (var cell in Coverage(grid, window, polygon))
{
    var value = pixels[cell.Index];
    if (value == noData)
    {
        noDataWeight += cell.Share;
        continue;
    }
    // …
    sum += value * cell.Share;      // S = Σ s_p · v_p
    valid += cell.Share;            // N = Σ s_p
}
return new ZonalResult(sum, valid, noDataWeight, zeroWeight, grid.PixelSize);

ZonalResult then exposes the derived numbers as properties, so callers never repeat the arithmetic:

public double PixelAreaHectares => PixelSize * PixelSize / 10_000;
public double TotalOfPerHectareValue => Sum * PixelAreaHectares;
public double Mean => ValidPixels > 0 ? Sum / ValidPixels : 0;
public double NoDataShare => ValidPixels + NoDataPixels > 0 ? NoDataPixels / (ValidPixels + NoDataPixels) : 1;

Sharing reads within a request. One valuation reads the same window for several layers and several steps. RasterReadScope (EcoVal.Api/Spatial/Rasters/RasterReadScope.cs) is an ambient cache held in an AsyncLocal: inside using var reads = RasterReadScope.Begin(); each layer's window is decoded once and the coverage is computed once. Outside a scope (tests, calibration) every read is fresh.

2.3 Class areas: counting by pixel centre, with sampling

Some rasters hold classes, not quantities (for example SLU's 2 m soil moisture map: dry, mesic, moist, wet). The question is "how many hectares of each class?", and a 10,000 ha polygon holds 25 million 2 m pixels. ClassAreas (EcoVal.Api/Spatial/Rasters/ClassAreas.cs) therefore does two things differently from ZonalStatistics: a pixel counts whole if its centre is inside the polygon, and above a limit only every nn-th row and column is read.

The formula. With a window of W×HW \times H pixels and a budget of MM samples, the step is

n=max⁡ ⁣(1, ⌈W⋅HM⌉)n = \max\!\left(1,\ \left\lceil \sqrt{\frac{W \cdot H}{M}} \right\rceil\right)

and each sample stands for n×nn \times n pixels, so the area of class kk is

Ak=∣{samples with class k}∣⋅(n ℓ)210 000 haA_k = \bigl|\{\text{samples with class } k\}\bigr| \cdot \frac{(n\,\ell)^2}{10\,000}\ \text{ha}
var pixels = (double)window.Width * window.Height;
var step = Math.Max(1, (int)Math.Ceiling(Math.Sqrt(pixels / maxSamples)));
var spacing = step * grid.PixelSize;
var sampleHectares = spacing * spacing / 10_000;

This is a systematic sample: the result is an estimate, and ClassCount.SpacingMetres tells the caller how coarse it was.

2.4 The 1 km grid

Population (SCB) is published per 1 km square, so the models about people (visits, fishing days) work on a 1 km grid aligned on SWEREF 99 TM's origin. SquareGrid.Split (EcoVal.Api/Spatial/SquareGrid.cs) cuts any geometry into its parts per square, each with its hectares and the square's centre. Distances in those models are then measured between square centres.


3. The types that carry a result

3.1 ValueRange: low, central, high

No result is a single number. Every computed value is three: a low, a central and a high case (EcoVal.Api/Ranges.cs):

public sealed record ValueRange(double Low, double Central, double High)
{
    public static ValueRange Spanning(double fromLow, double central, double fromHigh, int digits = 1) =>
        new(
            Math.Round(Math.Min(Math.Min(fromLow, fromHigh), central), digits),
            Math.Round(central, digits),
            Math.Round(Math.Max(Math.Max(fromLow, fromHigh), central), digits));
}

The range is not a statistical confidence interval. It is built by running the same formula three times, with the low, central and high value of each uncertain parameter (ParameterRange). When a formula multiplies several uncertain factors, the low case multiplies the lows and the high case the highs:

ylow=∏ixi,lowycentral=∏ixi,centralyhigh=∏ixi,highy_{\text{low}} = \prod_i x_{i,\text{low}} \qquad y_{\text{central}} = \prod_i x_{i,\text{central}} \qquad y_{\text{high}} = \prod_i x_{i,\text{high}}

Spanning exists because a parameter can push the result in either direction; it takes the smallest and largest of the three results as low and high, and leaves the central as computed.

3.2 Tier: how sure a number is

Each section of a result carries a confidence tier as a code (EcoVal.Api/Tiers.cs): measured (counted or mapped at the place), modelled (modelled from data about the place) or borrowed (a factor or price from a study elsewhere). A result is never more certain than its weakest input:

tier(result)=max⁡i rank(tieri),rank: measured=0, modelled=1, borrowed=2\text{tier}(\text{result}) = \max_i \ \text{rank}(\text{tier}_i), \qquad \text{rank: measured} = 0,\ \text{modelled} = 1,\ \text{borrowed} = 2
public static SectionTiers Priced(string quantity, string price) => new(quantity, Tier.Weakest(quantity, price));

3.3 Available<T>: a value, or the reason there is none

Missing data is not an error. A section either has a value or a code saying why not, modelled as a C# 15 union (EcoVal.Api/Available.cs):

public union Available<T>(T, Unavailable)
    where T : class
{
    public T? ValueOrNull => Value as T;
    public string? ReasonOrNull => (Value as Unavailable)?.Reason;
    public void Deconstruct(out T? value, out string? reason) => (value, reason) = (ValueOrNull, ReasonOrNull);
}

A calculator writes return new Unavailable(UnavailableReasons.NoForestData); or return value;, and the endpoint deconstructs it into the response's two fields: var (timber, timberUnavailable) = ….

3.4 Options: parameters are data

Model parameters are never constants in a formula. They are properties of an options class bound from configuration (builder.Services.AddOptions<ValuationOptions>().BindConfiguration(ValuationOptions.Section)), and the default value in the class is the documented value with its source in the XML comment:

// EcoVal.Api/Features/Forest/Recreation/RecreationOptions.cs
/// <summary>
/// Distance d₀ in w(d) = e^(−d/d₀), the weight a forest at distance d gets in where people go. In Friluftsliv 2018
/// (table 4.1) visits to the nearest nature area fall from about 180 a year within 100 m to about 55 at 0.5–1 km
/// and 33 at 1–3 km; an exponential through those points has d₀ ≈ 1.1–1.6 km. …
/// </summary>
public ParameterRange DecayKm { get; set; } = new(1, 1.5, 3);

This is also where fitted parameters end up, as section 6 shows.


4. Which kind of regression does EcoVal use?

Short answer: classical statistics and calibration, written by hand in C#. No machine-learning library, no neural network, no model file. The projects reference no ML.NET, Math.NET, ONNX or similar package; every fit is a few dozen lines of arithmetic you can read.

"Fitting a model" appears in four forms. They differ in how the best parameters are found:

Form How the parameters are found Where in EcoVal
Statistical regression, closed form A formula gives the best line directly (ordinary least squares) Stream width from catchment area
Statistical regression, iterative Repeated Newton steps until the estimate stops moving (logistic regression) Chanterelle habitat model
Calibration by grid search Try every combination on a grid of candidate values, keep the one with the smallest error Excursions, tourism, fishing days
Empirical ratios and quantiles Divide a counted total by a measured quantity; take medians and percentiles Salmon smolts per km, sea trout densities, form height

Is any of it machine learning? The chanterelle model is the closest: logistic regression is also a standard ML classifier, and it is evaluated the ML way, on data held out from training. But it is a 13-parameter statistical model with standard errors, not a learned black box, and the app does not use it (section 6.7). Everything else is calibration: a model whose form is decided by reasoning, with one to three free parameters tuned so the model matches counted data.

The fits also differ in when they run and where the result lives:

flowchart TB
    subgraph offline ["Offline: EcoVal.Calibration, started by hand"]
        C1["Counted data<br/>(visitor counts, surveys, GBIF finds)"] --> C2["Fit: grid search or regression"]
        C2 --> C3["Report in data/calibration/*.md"]
        C3 -->|"a developer copies the values"| C4["Options class defaults<br/>(source code, with a comment)"]
    end
    subgraph runtime ["Run time: EcoVal.Api, first request that needs it"]
        R1["Database tables<br/>(electrofishing sites, river network)"] --> R2["Fit: least squares, quantiles, ratios"]
        R2 --> R3["Singleton held in memory<br/>until the API restarts"]
    end
    C4 --> U["Calculators use the values on every request"]
    R3 --> U

Two consequences worth knowing:

  • Offline fits do not update themselves. New data changes nothing until someone runs the calibration again and copies the new values into the options class. The comment on the option records the date of the fit.
  • Run-time fits follow the data. A new ingest of electrofishing surveys changes the stream width model the next time the API starts. The current values are always visible at /api/water/sea-trout-calibration and /api/water/salmon-calibration.
Fit Kind When Stored in Used by
Stream width OLS on log–log API, first use SeaTroutModel singleton Dam: sea trout habitat
Sea trout density classes Quantiles per group API, first use SeaTroutModel singleton Dam: sea trout habitat
Smolts per young trout Ratios, median API, first use SeaTroutModel singleton Dam: sea trout smolts
Salmon smolts per km Ratios, regional quantiles API, first use SalmonCalibrationSource singleton Dam: salmon smolts
Excursions: s, d₁, γ Grid search Offline RecreationOptions.Excursion Forest: recreation
Tourism: α, d_w, weights Grid search + 1-D minimiser Offline TourismOptions Forest: recreation
Tourism: q Grid search Offline TourismOptions (kept as estimates) Forest: recreation
Fishing days: d₁, β Grid search, weighted least squares Offline LakeOptions.FishingByPeople Water: lake
Form height by age Binned ratio Offline CalculatorOptions…FormHeightByAge Landing calculator
Raw-water trend of a drinking-water source Sen's slope with the Mann–Kendall test (a non-parametric trend estimate and test) API, on each request Not stored Water: drinking water (part 5, section 5)
Growth model: production level, mortality, old-forest floor Grid search, least squares Offline, in a Python script (scripts/review/growth/fit-mortality.py) GrowthOptions Forest: growth and carbon (part 2, section 7.7)
Chanterelle habitat Ridge logistic regression Offline Report only Not used

5. Fits made by the API at run time

All three live in EcoVal.Api/Features/Water/Dam/. They are built once, on the first request that needs them, by a singleton "source" class that guards the build with a SemaphoreSlim so that two simultaneous first requests do not both build it:

// SeaTroutModel.cs
public async Task<SeaTroutModel> GetAsync(CancellationToken cancellationToken)
{
    if (_model is { } model)
    {
        return model;                        // the common path: already built
    }
    var network = await networks.GetAsync(cancellationToken);
    await _lock.WaitAsync(cancellationToken);
    try
    {
        if (_model is null)                  // checked again inside the lock
        {
            // … load the sites from the database …
            _model = new SeaTroutModel(network, sites, options.Value);
        }
        return _model;
    }
    finally
    {
        _lock.Release();
    }
}

5.1 Stream width: ordinary least squares on a log–log scale

What it is for. Young sea trout are counted per 100 m² of stream, so to know how many a stretch holds we need its width. Width is measured only at electrofishing sites; everywhere else it is predicted from how much land drains through the point.

Kind of fit: statistical regression (simple linear regression, ordinary least squares), trained by the API itself from the electrofishing_surveys table.

The model. Width grows with catchment area as a power law, which is a straight line once both are log-transformed:

log⁡10w=a+b log⁡10A⟺w=10a⋅Ab\log_{10} w = a + b \, \log_{10} A \qquad\Longleftrightarrow\qquad w = 10^{a} \cdot A^{b}

where ww is the wetted width in metres and AA the area draining through the point in km².

The fit. With xi=log⁡10Aix_i = \log_{10} A_i and yi=log⁡10wiy_i = \log_{10} w_i over the nn sites, least squares has a closed form:

Sxx=∑i(xi−xˉ)2Sxy=∑i(xi−xˉ)(yi−yˉ)Syy=∑i(yi−yˉ)2S_{xx} = \sum_i (x_i - \bar{x})^2 \qquad S_{xy} = \sum_i (x_i - \bar{x})(y_i - \bar{y}) \qquad S_{yy} = \sum_i (y_i - \bar{y})^2 b=SxySxxa=yˉ−b xˉR2=Sxy2Sxx Syyb = \frac{S_{xy}}{S_{xx}} \qquad a = \bar{y} - b\,\bar{x} \qquad R^2 = \frac{S_{xy}^2}{S_{xx}\,S_{yy}}

R2R^2 is the share of the variation in log⁡w\log w that the line explains.

The implementation is the private method SeaTroutModel.Fit, a direct transcription:

private static WidthModel? Fit(List<(double X, double Y)> points)
{
    if (points.Count < 10)
    {
        return null;                                    // too few sites: no model
    }
    var meanX = points.Average(point => point.X);
    var meanY = points.Average(point => point.Y);
    var sxx = points.Sum(point => (point.X - meanX) * (point.X - meanX));
    var sxy = points.Sum(point => (point.X - meanX) * (point.Y - meanY));
    var syy = points.Sum(point => (point.Y - meanY) * (point.Y - meanY));
    var slope = sxy / sxx;
    return new WidthModel(
        Math.Round(meanY - slope * meanX, 4),           // a
        Math.Round(slope, 4),                           // b
        points.Count,
        Math.Round(syy > 0 ? sxy * sxy / (sxx * syy) : 0, 3));   // R²
}

The training data is prepared in the constructor. Three choices are worth noticing:

Width = Fit(sites
    .Where(site => site.Width is > 0 && site.Snap <= options.WidthFitMaxSnapMetres)
    .Select(site => (Area: network.AreaAt(site.Position), Width: site.Width!.Value))
    .Where(pair => pair.Area is > 0)
    .Select(pair => (Math.Log10(pair.Area!.Value), Math.Log10(pair.Width)))
    .ToList());
  • A site surveyed several times enters once, with the median of its widths (computed just above this code), so heavily surveyed sites do not dominate.
  • Only sites within 30 m of the network's line are used (WidthFitMaxSnapMetres). A site on a small brook the network lacks would otherwise be attached to the big river nearby and teach the model that big rivers are narrow.
  • The logs are taken before fitting, so the regression minimises relative errors, not errors in metres.

Using the model. WidthModel.Width inverts the log, and SegmentValues clamps the result to a sensible range and caches it per river segment:

public double Width(double areaKm2) => Math.Pow(10, Intercept + Slope * Math.Log10(Math.Max(areaKm2, 0.01)));

var width = Math.Clamp(
    Width is { } model && network.AreaAtOutlet(id) is > 0 and var area ? model.Width(area) : options.MinWidthMetres,
    options.MinWidthMetres,      // 1 m
    options.MaxWidthMetres);     // 150 m
Stream width from the area draining through

At the time of writing the fit is log⁡10w=0.39+0.16log⁡10A\log_{10} w = 0.39 + 0.16 \log_{10} A from 4,126 sites, with R2=0.17R^2 = 0.17 (docs/calculations-water.md). An R2R^2 of 0.17 is weak: catchment area explains only about a sixth of the variation in width. The model gives a reasonable typical width, not a good width for one particular stream. That is acceptable here because widths are summed over many segments, where the errors partly cancel.

5.2 Sea trout densities and smolts per young trout: quantiles and ratios

Kind of fit: empirical statistics, not regression. Nothing is optimised; observed values are grouped and summarised.

Density classes. The sites that sea trout can reach freely today (no barrier on the way to the sea) are grouped by water district and by width class (bounds at 2, 4, 8, 16 and 32 m). Each group's density is the 25th percentile, median and 75th percentile of its sites, which become the low, central and high of a ValueRange.

The quantile used throughout interpolates linearly between neighbouring values. For sorted values x0≤⋯≤xn−1x_0 \le \dots \le x_{n-1} and a quantile qq:

t=q (n−1)Q(q)=x⌊t⌋+(t−⌊t⌋)(x⌊t⌋+1−x⌊t⌋)t = q\,(n-1) \qquad Q(q) = x_{\lfloor t \rfloor} + \bigl(t - \lfloor t \rfloor\bigr)\bigl(x_{\lfloor t \rfloor + 1} - x_{\lfloor t \rfloor}\bigr)
// SalmonCalibration.cs
public static double Quantile(IReadOnlyList<double> sorted, double q)
{
    var position = q * (sorted.Count - 1);
    var below = (int)Math.Floor(position);
    var above = Math.Min(below + 1, sorted.Count - 1);
    return sorted[below] + (sorted[above] - sorted[below]) * (position - below);
}

A group with fewer than MinSitesPerClass (10) sites is too thin to trust, so DensityFor falls back to the same width class over all of Sweden and reports which source it used as a code (district or national).

Young trout in a stretch. For a stretch made of pieces jj with length LjL_j (m), modelled width wjw_j (m) and class density ρj\rho_j (young trout per 100 m²):

Y=∑jLj⋅wj⋅ρj100Y = \sum_j L_j \cdot w_j \cdot \frac{\rho_j}{100}
public (double YoungTrout, double AreaM2) Habitat(HabitatStretch stretch)
{
    var (young, area) = (0.0, 0.0);
    foreach (var piece in stretch.Pieces)
    {
        var length = (piece.To - piece.From) * (network.Segment(piece.SegmentId)?.LengthMetres ?? 0);
        var (width, perMetre) = SegmentValues(piece.SegmentId);   // perMetre = width × density / 100
        young += length * perMetre;
        area += length * width;
    }
    return (young, area);
}

Smolts per young trout. In three rivers the smolts leaving for the sea are counted in a trap. For each such river rr the model computes YrY_r, the young trout its own method predicts above the trap, and divides each year's count by it:

kr,t=smoltsr,tYrk_{r,t} = \frac{\text{smolts}_{r,t}}{Y_r}

The value used everywhere is the median over rivers of each river's median, with the lowest and highest single year as the range:

kcentral=median⁡r(median⁡t kr,t)klow=min⁡r,tkr,tkhigh=max⁡r,tkr,tk_{\text{central}} = \operatorname{median}_r \bigl(\operatorname{median}_t\, k_{r,t}\bigr) \qquad k_{\text{low}} = \min_{r,t} k_{r,t} \qquad k_{\text{high}} = \max_{r,t} k_{r,t}
var ratios = young > 0 ? river.Smolts.Values.Select(smolts => smolts / young).Order().ToList() : [];

This is a calibration by ratio: the factor kk absorbs everything the density model gets wrong on average, so that the model reproduces the counted rivers by construction. It is honest about being three rivers only: the endpoint lists each river with its own ratio.

5.3 Salmon smolts per km: a ratio, with regional quantiles

Kind of fit: calibration by ratio. ICES publishes each wild salmon river's potential smolt production (PSPC). EcoVal walks the river in its own network, under the same habitat rule a dam's upstream stretch is walked with, and divides:

dr=PSPCrLrsmolts per kmd_r = \frac{\text{PSPC}_r}{L_r}\quad \text{smolts per km}
// SalmonCalibration.cs
var habitat = network.Habitat(mouth, new HashSet<int>(), options.MinUpstreamAreaKm2);
var km = habitat.LengthMetres / 1000;
var pspc = river.PspcThousands!.Value * 1000;
// …
new ValueRange(
    Math.Round((river.PspcLowThousands ?? river.PspcThousands.Value) * 1000 / km),
    Math.Round(pspc / km),
    Math.Round((river.PspcHighThousands ?? river.PspcThousands.Value) * 1000 / km))

The point of using the same walk is that any systematic error in how EcoVal measures habitat length appears in both the calibration and the prediction, and cancels.

A dam's river then gets a density by a two-step rule in SalmonCalibration.Density: its own river's value if the river is calibrated, otherwise the median of its region's rivers with the 10th and 90th percentiles as the range:

return (new ValueRange(
        Math.Round(Quantile(region, 0.1)),
        Math.Round(Quantile(region, 0.5)),
        Math.Round(Quantile(region, 0.9))),
    SalmonDensitySources.RegionMedian,
    region.Count);

6. Fits made offline by EcoVal.Calibration

EcoVal.Calibration is a console program listed in the AppHost with explicit start. Program.cs registers the API's own services (builder.AddForestry(), RecreationLookups, the options) and runs the models named in Calibration__Models, writing one Markdown report per model to data/calibration/.

Four of the six fits tune the same family of model, so that model comes first.

6.1 The model being calibrated: sharing visits by distance

The idea. People in each 1 km population square make a fixed number of nature visits a year. Those visits are shared among all the forest within reach: more to forest that is near, less to forest that is far, and the shares always add up to one. A forest surrounded by people and little other forest gets many visits; a forest among endless forest gets few. Models of this form are known as gravity or spatial interaction models.

The distance weight. Forest at distance dd from a home gets the weight

w(d)={exp⁡ ⁣(−max⁡(d, dfloor)d0)d≤dreach0d>dreachw(d) = \begin{cases} \exp\!\left(-\dfrac{\max(d,\ d_{\text{floor}})}{d_0}\right) & d \le d_{\text{reach}} \\[2ex] 0 & d > d_{\text{reach}} \end{cases}

d0d_0 is the distance at which the weight has fallen to 1/e≈37 %1/e \approx 37\,\%. The floor (0.5 km) is there because people live spread over their square, so a distance is never taken as shorter than that.

Distance weight for everyday visits

The polygon's visits. With aca_c the adults in population square cc, APA_P the forest land in the polygon, dcPd_{cP} the distance from the square's centre to the polygon, FkF_k the forest land in every other 1 km square kk (the polygon's own forest taken out) and VV the visits a year per adult:

visits=V⋅∑cac⋅AP w(dcP)AP w(dcP)+∑kFk w(dck)\text{visits} = V \cdot \sum_{c} a_c \cdot \frac{A_P \, w(d_{cP})}{A_P \, w(d_{cP}) + \sum_{k} F_k \, w(d_{ck})}

The fraction is the polygon's share of square cc's visits: its own pull over the pull of everything within reach.

The implementation is RecreationCalculator.EverydayShares. It returns the per-square terms ac×a_c \times share, so that callers (and the calibration) can sum, filter or group them:

double Weight(double distance) => distance > reach ? 0 : Math.Exp(-Math.Max(distance, floor) / decay);
// …
for (var i = 0; i < population.Count; i++)
{
    var square = population[i];
    var own = square.Adults > 0 ? forestHectares * Weight(square.DistanceMetres) : 0;   // A_P · w(d_cP)
    if (own <= 0)
    {
        continue;
    }
    var elsewhere = 0.0;                                                                 // Σ_k F_k · w(d_ck)
    foreach (var other in others)
    {
        var dx = other.CentreX - square.CentreX;
        var dy = other.CentreY - square.CentreY;
        if (Math.Abs(dx) > reach || Math.Abs(dy) > reach)
        {
            continue;                                   // a cheap box test before the square root
        }
        elsewhere += other.Hectares * Weight(Math.Sqrt(dx * dx + dy * dy));
    }
    shares[i] = square.Adults * own / (own + elsewhere);
}

Excursions use the same form with two changes: a longer distance d1d_1, and a size exponent γ\gamma that lets a large, well-known destination pull more per hectare than a small one. A destination of area DD hectares gets the pull DγD^{\gamma}, spread over its hectares as h⋅Dγ−1h \cdot D^{\gamma-1}. A share ss of each adult's visits are excursions:

excursions=s⋅V⋅∑cac⋅∑l∈Phl Dlγ−1 w1(dcP)∑l∈Phl Dlγ−1 w1(dcP)+pull of all other destination land\text{excursions} = s \cdot V \cdot \sum_c a_c \cdot \frac{\sum_{l \in P} h_l \, D_l^{\gamma-1}\, w_1(d_{cP})} {\sum_{l \in P} h_l \, D_l^{\gamma-1}\, w_1(d_{cP}) + \text{pull of all other destination land}}

RecreationCalculator.ExcursionShares implements it. The denominator's "all destination land within reach of this square" does not depend on the polygon, so DestinationGridData.Pull computes it once per population square and parameter set and caches it in a ConcurrentDictionary.

So the free parameters are: d0d_0 (set from the literature, not fitted), and ss, d1d_1 and γ\gamma, which have no Swedish source and are fitted.

6.2 Excursions: grid search for s, d₁ and γ

Kind of fit: calibration by grid search. Not regression: there is no formula for the best parameters, and the error surface is not smooth enough to trust a gradient method with so few data points. Instead every combination on a grid is evaluated and the errors are compared. File: EcoVal.Calibration/ExcursionCalibration.cs.

private static readonly double[] Shares = [0, 0.10, 0.15, 0.20, 0.25, 0.30, 0.35];
private static readonly double[] DecaysKm = [5, 7.5, 10, 15, 20, 30, 50];
private static readonly double[] SizeExponents = [0.7, 1.0, 1.3, 1.6, 2.0, 2.5];

The data is three independent kinds of evidence, each of which pins down a different aspect:

  1. Levels. Seven nature reserves in Huddinge with visits counted from mobile-phone data (2021).
  2. Origins. The same study's split of where visitors came from, by municipality.
  3. A guard. Gotland's least visited reserves (estimated at 1–100 visits a year): adding excursions must not inflate these.

The objective to minimise is the sum of three errors. For parameters θ=(s,d1,γ)\theta = (s, d_1, \gamma):

J(θ)=Elevel+Eorigin+EguardJ(\theta) = E_{\text{level}} + E_{\text{origin}} + E_{\text{guard}} Elevel=1n∑i∈Huddinge[ln⁡max⁡(m^i(θ), 1)ci]2E_{\text{level}} = \frac{1}{n} \sum_{i \in \text{Huddinge}} \left[\ln \frac{\max(\hat{m}_i(\theta),\,1)}{c_i}\right]^2 Eorigin=11000∑g[100 (o^g(θ)−og)]2Eguard=[ln⁡median⁡j∈Gotland, lowcjm^j(θ)]2E_{\text{origin}} = \frac{1}{1000} \sum_{g} \bigl[100\,(\hat{o}_g(\theta) - o_g)\bigr]^2 \qquad E_{\text{guard}} = \left[\ln \operatorname{median}_{j \in \text{Gotland, low}} \frac{c_j}{\hat{m}_j(\theta)}\right]^2

cc is a counted number, m^\hat{m} the model's, ogo_g the reported share of visits from origin gg and o^g\hat{o}_g the modelled one. The squared log ratio treats "twice too high" and "half as much" as equally wrong, which is right for counts that span several orders of magnitude. The divisor 1000 puts the origin error (in squared percentage points) on roughly the same scale as the other two, so each term is about 1 for a poor fit.

private static Fit Evaluate(/* … */)
{
    var huddinge = areas.Where(area => area.Area.Region == "Huddinge").ToList();
    var levelError = huddinge.Average(area =>
        Math.Pow(Math.Log(Math.Max(ModelledKept(area, share, combo, perAdult), 1) / area.Area.Visits), 2));

    var origins = OriginShares(huddinge, share, combo);
    var originError = OriginSplit.Reported.Sum(row => Math.Pow(100 * (origins.GetValueOrDefault(row.Code) - row.Share), 2)) / 1_000;

    var lowClass = areas.Where(area => area.Area.Region == "Gotland" && area.Area.VisitsHigh <= 100).ToList();
    var lowRatio = Median(lowClass.Select(area => area.Area.Visits / Math.Max(Modelled(area, share, combo, perAdult), 1)));
    // …
}

Comparing like with like. The phone data does not count everyone: it leaves out people living within 400 m and stays under 20 minutes. MobileDataFilter.Kept(distance, …) returns the share of visits from a home at a given distance that the phones would have counted, and the model's per-square terms are multiplied by it before comparing (ModelledKept). Fitting the unfiltered model to filtered counts would bias every parameter.

Why the search is fast. The expensive part is the spatial sums, and ss enters the model linearly:

m^(θ)=V[(1−s) E+s Xd1,γ]\hat{m}(\theta) = V \bigl[(1 - s)\,E + s\,X_{d_1,\gamma}\bigr]

So ComputeAsync calculates EE (everyday shares) once per area and XX (excursion shares) once per (d1,γ)(d_1, \gamma) pair, and the loop over ss is plain arithmetic:

private static double Modelled(AreaShares area, double share, int combo, double perAdult) =>
    perAdult * ((1 - share) * area.Everyday + share * area.Excursion[combo]);
flowchart LR
    A["For each counted area:<br/>everyday shares E<br/>excursion shares X per (d₁, γ)"] --> B["For each s, d₁, γ on the grid:<br/>model = V·((1−s)·E + s·X)"]
    B --> C["Objective J =<br/>level + origin + guard"]
    C --> D["Sort by J,<br/>write the best 15 to excursions.md"]
    D --> E["Developer chooses low, central, high<br/>and edits RecreationOptions.Excursion"]

The result and its use. The report's best set was s=0.35s = 0.35, d1=7.5d_1 = 7.5 km, γ=2\gamma = 2 (J=0.503J = 0.503), with several neighbours almost as good. The values stored are not simply the minimum; the options comment explains the judgement applied (the fit pushes ss to its bound, and γ=2\gamma = 2 would let the largest destinations dominate):

// RecreationOptions.cs
public List<ExcursionParameters> Excursion { get; set; } =
[
    new(0.25, 10, 1.6),     // low:     s, d₁ km, γ
    new(0.30, 10, 1.6),     // central
    new(0.35, 20, 1.6),     // high
];

RecreationCalculator.Excursions runs the model once per case and builds the ValueRange from the three results.

6.3 Tourism, where guests go: grid search with a fitted scale factor

File: EcoVal.Calibration/AttractionCalibration.cs. Kind of fit: grid search, with a one-dimensional numerical minimiser nested inside it.

The model. A municipality's guest nights are shared among its destination land. A share 1−α1 - \alpha goes by area, and a share α\alpha by attractiveness, where land near car parks, trailheads, viewpoints and beaches is more attractive:

shareP=(1−α) shareParea+α sharePattraction\text{share}_P = (1 - \alpha)\,\text{share}^{\text{area}}_P + \alpha\,\text{share}^{\text{attraction}}_P A(x)=∑points jωkind(j) exp⁡ ⁣(−d(x,j)dw)\mathcal{A}(x) = \sum_{\text{points } j} \omega_{\text{kind}(j)} \, \exp\!\left(-\frac{d(x, j)}{d_w}\right)

A(x)\mathcal{A}(x) is the attractiveness at a place xx: each access point adds its kind's weight ω\omega, fading with distance at the rate dwd_w. In code the blend is one line:

// RecreationContracts.cs
public double Combined(double attractionShare) => (1 - attractionShare) * Area + attractionShare * Attraction;

The grid covers α\alpha (9 values), dwd_w (4 values) and the weights of four groups of access point (each 0, 1, 3 or 10; only their ratios matter, so WeightSets() normalises and removes duplicates).

The nested fit. Gotland's 158 counted reserves are modelled as

m^i=Ri+k⋅Gi(α,dw,ω)\hat{m}_i = R_i + k \cdot G_i(\alpha, d_w, \omega)

with RiR_i the residents' visits (from the excursion model, fixed) and GiG_i the guest nights the candidate sends to reserve ii. The factor kk (visits per guest night) is unknown and of no interest here; what matters is whether the shape is right, that is, whether busy reserves come out busy. So for each candidate on the grid, the best kk is found first, and the candidate is scored at that kk:

E(α,dw,ω)=min⁡k 1n∑i[ln⁡max⁡(Ri+k Gi, 1)ci]2E(\alpha, d_w, \omega) = \min_{k}\ \frac{1}{n}\sum_i \left[\ln \frac{\max(R_i + k\,G_i,\ 1)}{c_i}\right]^2

The inner minimum over kk uses golden-section search, a method for finding the minimum of a function of one variable without derivatives. It keeps an interval that contains the minimum and shrinks it by the golden ratio φ=(5−1)/2≈0.618\varphi = (\sqrt{5}-1)/2 \approx 0.618 each step:

double Error(double logK) => SquaredLogError(gotland, guestNights, Math.Exp(logK));
double low = Math.Log(1e-3), high = Math.Log(30);
var ratio = (Math.Sqrt(5) - 1) / 2;
for (var i = 0; i < 60; i++)
{
    var a = high - ratio * (high - low);
    var b = low + ratio * (high - low);
    if (Error(a) < Error(b))
    {
        high = b;          // the minimum lies in [low, b]
    }
    else
    {
        low = a;           // the minimum lies in [a, high]
    }
}
var k = Math.Exp((low + high) / 2);

The search runs on ln⁡k\ln k, so it covers kk from 0.001 to 30 evenly in relative terms. Sixty steps shrink the interval by 0.61860≈10−130.618^{60} \approx 10^{-13}, far more than needed.

The result is stored in TourismOptions: AttractionShare = new(0.8, 0.9, 1), AttractionDecayKm = 0.5 and AccessPointWeights. The comments record what the fit could and could not tell: dwd_w sits at the edge of its grid, and several weight sets fit about equally well.

6.4 Tourism, visits per guest night: a grid search that could not decide

File: EcoVal.Calibration/TourismCalibration.cs. Kind of fit: grid search, over qq (nature visits per guest night) for city municipalities and for other municipalities, with the same three-part objective as the excursion fit (Huddinge's levels, Gotland's busy reserves, Gotland's quiet reserves as a guard):

private static readonly double[] OtherGrid = [0, 0.05, 0.1, 0.15, 0.2, 0.25, 0.3, 0.4, 0.5, 0.75, 1.0];
private static readonly double[] CityGrid = [0, 0.01, 0.02, 0.05, 0.1, 0.2];
// …
public double Objective => LevelError + TouristError + GuardError;

This is the instructive case of a calibration that ran but did not settle the parameter. The options comments say so plainly: the values are "a first estimate", because the counted data hardly moves with qq. The calibration is kept because it reports how the configured values score, and because it will decide the question once better counts exist. A fit is only as good as the data's power to tell candidate values apart.

6.5 Fishing days: weighted least squares on a grid

File: EcoVal.Calibration/FishingCalibration.cs. Kind of fit: grid search minimising a weighted sum of squares (χ²). This is the cleanest example of the four.

The problem. A national survey gives the fishing days a year for a few large areas. The page needs days for one lake. Sharing a region's days by water area alone is badly wrong: it gives Vänern 1.2 million days where the survey found 233,000.

The model. A share β\beta of the days follow the people living near each water, and the rest follow hectares. For area AA, with TT the total days, HH the total hectares and PAP_A the "pull" of area AA on people:

daysA=T[(1−β) haAH+β PA∑A′PA′]\text{days}_A = T \left[(1 - \beta)\,\frac{\text{ha}_A}{H} + \beta\,\frac{P_A}{\sum_{A'} P_{A'}}\right]

The pull uses the same distance-weighted sharing as section 6.1, now for water. Each population square cc splits its adults aca_c among all water squares within reach by hectares ×\times w(d)w(d):

PA=∑cac⋅∑l∈Ahl w(dcl)∑k∈all waterhk w(dck),w(d)=exp⁡ ⁣(−max⁡(d,dfloor)d1)P_A = \sum_c a_c \cdot \frac{\sum_{l \in A} h_l \, w(d_{cl})}{\sum_{k \in \text{all water}} h_k \, w(d_{ck})}, \qquad w(d) = \exp\!\left(-\frac{\max(d, d_{\text{floor}})}{d_1}\right)

FishingGridData.Compute (EcoVal.Api/Features/Water/Lake/FishingGrid.cs) evaluates it for about 95,000 population squares against 160,000 water squares. It indexes the water by 10 km blocks so each home only looks at nearby blocks, runs the homes in parallel with Parallel.For, and caches the result per distance setting:

Parallel.For(0, _homes.Length, i =>
{
    var home = _homes[i];
    var byArea = new double[_areas.Length];
    // … for each 10 km block within reach of the home …
    foreach (var (square, area) in block)
    {
        byArea[area] += square.Hectares * Weight(square.CentreX - home.X, square.CentreY - home.Y, distance);
    }
    total[i] = byArea.Sum();        // the denominator: this home's pull to all water
    byHome[i] = byArea;
});

The objective. Each survey figure comes with a margin of error. A figure the survey is unsure of should count for less, so each squared error is divided by that figure's variance. With σA\sigma_A = margin of error ÷ 1.96 (the margin is a 95 % interval, 1.96 standard deviations wide):

χ2(d1,β)=∑A(daysA(d1,β)−surveyAσA)2\chi^2(d_1, \beta) = \sum_A \left(\frac{\text{days}_A(d_1, \beta) - \text{survey}_A}{\sigma_A}\right)^2
foreach (var share in ShareGrid)       // β = 0, 0.05, …, 1
{
    var days = targets
        .Select((_, i) => total * ((1 - share) * hectares[i] / allHectares + share * pull[i] / allPull))
        .ToArray();
    var chi = targets.Select((target, i) => Math.Pow((days[i] - target.Days) / target.Sigma, 2)).Sum();
    fits.Add(new Fit(decay, reach, share, chi, days));
}
// …
var best = fits.MinBy(fit => fit.ChiSquare)!;

The pulls (expensive) are computed once per d1d_1 in the outer loop; the loop over β\beta inside is arithmetic, the same trick as in 6.2.

The fit's error for each distance

From the best fit to a range. χ² also gives a principled range. For a fit like this, parameter values whose χ² is within 1 of the minimum are the ones the data cannot reject at about one standard deviation. The report lists them, and they became the low, central and high cases:

// LakeOptions.cs
public List<FishingPeopleCase> FishingByPeople { get; set; } =
[
    new() { DecayKm = 0.75, PeopleShare = 0.85 },    // low
    new() { DecayKm = 1.5, PeopleShare = 0.90 },     // central: the best fit, χ² 0.10
    new() { DecayKm = 3, PeopleShare = 0.90 },       // high
];
Survey against the fitted model

How the API uses it. The fit was made on four areas, but the page needs any lake in any region. So the fitted β\beta is first turned into a quantity that travels: rr, the fishing days a year an adult makes on water near home.

r=β⋅T∑A∈fit areasPAr = \frac{\beta \cdot T}{\sum_{A \in \text{fit areas}} P_A}

A lake LL in region RR (with the region's days DRD_R from the survey) then gets r⋅PLr \cdot P_L days from the people near it, and its share by area of whatever the region has left:

βR=min⁡ ⁣(1, r⋅PRDR)daysL=DR[(1−βR) haLhaR+βR PLPR]\beta_R = \min\!\left(1,\ \frac{r \cdot P_R}{D_R}\right) \qquad \text{days}_L = D_R \left[(1 - \beta_R)\,\frac{\text{ha}_L}{\text{ha}_R} + \beta_R\,\frac{P_L}{P_R}\right]
// LakeCalculator.cs
private static double Rate(FishingPullCase pullCase, double fitDays, LakeOptions options) =>
    options.FishingFitAreas.Sum(code => pullCase.AreaPull.GetValueOrDefault(code)) is var fitPull and > 0
        ? pullCase.PeopleShare * fitDays / fitPull
        : 0;

// inside RegionShares:
peopleShare = Math.Min(1, rate * regionPull / regionDays);
var lakePull = pullCase.Lakes.GetValueOrDefault(lake.Id).Pull;
return regionDays * ((1 - peopleShare) * lake.Hectares / regionHectares + peopleShare * lakePull / regionPull);

Note what is and is not hard-coded: d1d_1 and β\beta are stored, but rr is recomputed from them and the current population and water data on every start, so it follows the data.

6.6 Form height by age: a binned ratio that becomes a lookup table

File: EcoVal.Calibration/FormHeightCalibration.cs. Kind of fit: an empirical table, not regression. No functional form is assumed at all.

What it is for. The landing calculator knows a typical stand's volume but its berry models need basal area. Form height is the bridge: F=V/GF = V / G (volume ÷ basal area, in metres).

The method. Draw random 30 m squares over Sweden until 20,000 have forest and data in all four rasters, sort them into 10-year age classes kk, and take the ratio of sums in each class:

Fk=∑i∈kVi∑i∈kGiF_k = \frac{\sum_{i \in k} V_i}{\sum_{i \in k} G_i}

The ratio of sums (not the mean of each square's ratio) weights each square by its basal area, so sparse young squares with unstable ratios do not distort the class.

var random = new Random(20261030);          // a fixed seed: the same squares on every run
// …
var age = (ageSpruce + agePine) / 2;
var key = Math.Min(LastClass, (int)(age / ClassYears) * ClassYears);
var entry = classes.GetValueOrDefault(key);
classes[key] = (entry.Volume + volume, entry.BasalArea + basalArea, entry.Count + 1);

Each square's values come from ZonalStatistics.Compute (section 2.2), the same code the API uses.

Using it. The table is stored in CalculatorOptions keyed by each class's middle age, and read with linear interpolation: straight lines between the ages, flat beyond the ends.

F(a)=Fi+a−aiai+1−ai (Fi+1−Fi)for ai≤a≤ai+1F(a) = F_i + \frac{a - a_i}{a_{i+1} - a_i}\,(F_{i+1} - F_i) \qquad \text{for } a_i \le a \le a_{i+1}
// ForestStandCalculator.cs
private static double Interpolated(Dictionary<double, double> byAge, double age)
{
    var table = byAge.OrderBy(entry => entry.Key).ToList();
    if (age <= table[0].Key)
    {
        return table[0].Value;
    }
    for (var i = 1; i < table.Count; i++)
    {
        if (age <= table[i].Key)
        {
            var (from, to) = (table[i - 1], table[i]);
            return from.Value + (age - from.Key) / (to.Key - from.Key) * (to.Value - from.Value);
        }
    }
    return table[^1].Value;
}
Form height by stand age

6.7 Chanterelle habitat: ridge logistic regression, tried and left out

Files: EcoVal.Calibration/ChanterelleCalibration.cs and ChanterelleModel.cs. Kind of fit: statistical regression (a generalised linear model), trained iteratively and validated on held-out data. This is the one model in the repository that a data scientist would recognise as "trained", and the one closest to machine learning. The API does not use it: it barely beat chance, so no score is shown. It stays in the repository so the fit can be repeated on better data, and it is the best worked example of a full training pipeline.

flowchart TB
    A["GBIF downloads:<br/>chanterelle finds (y = 1)<br/>21 other mushrooms (y = 0)"] --> B["Filter and thin<br/>one record per 100 m square"]
    B --> C["Read the stand in a 60 m square<br/>8 inputs per record"]
    R["12,000 random forest squares<br/>(reference sample)"] --> D
    C --> D["Clamp to 1st–99th percentile<br/>add squared terms (12 terms)<br/>standardise on the reference"]
    D --> E["Fit: weighted ridge logistic regression<br/>(Newton / IRLS)"]
    D --> F["Validate: 5 folds of whole 100 km blocks<br/>AUC on the held-out fold"]
    E --> G["chanterelle.md:<br/>coefficients, standard errors, AUC"]
    F --> G

The model. The probability that a record is a chanterelle find, given its site's terms xx:

p(x)=11+e−η(x)η(x)=β0+∑j=112βj zjzj=xj−μjσjp(x) = \frac{1}{1 + e^{-\eta(x)}} \qquad \eta(x) = \beta_0 + \sum_{j=1}^{12} \beta_j \, z_j \qquad z_j = \frac{x_j - \mu_j}{\sigma_j}

The twelve terms are eight inputs (age, basal area, pine share, spruce share, site index, dry share, wet share, northing) plus the squares of four of them, so the response can peak in the middle of a range (ChanterelleModel.Raw). Each term is standardised with the mean μj\mu_j and standard deviation σj\sigma_j of the random forest sample, so a coefficient reads as "per standard deviation of Swedish forest" and the ridge penalty treats all terms alike.

What is minimised. The penalised, weighted negative log-likelihood:

L(β)=−∑iωi[yiln⁡pi+(1−yi)ln⁡(1−pi)]+λ2∑j≥1βj2\mathcal{L}(\beta) = -\sum_i \omega_i \bigl[y_i \ln p_i + (1 - y_i)\ln(1 - p_i)\bigr] + \frac{\lambda}{2}\sum_{j \ge 1} \beta_j^2
  • ωi\omega_i is a class weight: 1 for finds and n1/n0n_1 / n_0 for background records, so both classes weigh the same in total even though there are many more background records than finds.
  • The last term is the ridge penalty (λ=1\lambda = 1). It pulls coefficients toward zero, which keeps correlated terms (age and age²) from taking huge opposite values. The intercept β0\beta_0 is not penalised.

How it is minimised. There is no closed form, so the fit uses Newton's method, which for logistic regression is known as iteratively reweighted least squares (IRLS). Each iteration computes the gradient gg and the Hessian HH at the current β\beta and takes a step:

g=∑iωi (yi−pi) xi−λ β(j≥1)H=∑iωi pi(1−pi) xixi⊤+λ I(j≥1)g = \sum_i \omega_i\,(y_i - p_i)\,x_i - \lambda\,\beta_{(j \ge 1)} \qquad H = \sum_i \omega_i\,p_i(1 - p_i)\,x_i x_i^{\top} + \lambda\,I_{(j \ge 1)} β←β+H−1g\beta \leftarrow \beta + H^{-1} g
for (; iterations < 50; iterations++)
{
    hessian = new double[k, k];
    var gradient = new double[k];
    foreach (var (x, y, _) in rows)
    {
        var row = Row(x);                                                    // [1, x₁, …]
        var p = 1 / (1 + Math.Exp(-row.Select((v, i) => v * beta[i]).Sum())); // p = 1 / (1 + e^(−η))
        var w = weight(y);
        for (var i = 0; i < k; i++)
        {
            gradient[i] += w * (y - p) * row[i];
            for (var j = 0; j < k; j++)
            {
                hessian[i, j] += w * p * (1 - p) * row[i] * row[j];
            }
        }
    }
    for (var i = 1; i < k; i++)                    // the ridge, skipping the intercept
    {
        gradient[i] -= Ridge * beta[i];
        hessian[i, i] += Ridge;
    }
    var step = Solve(hessian, gradient);           // solves H · step = g
    for (var i = 0; i < k; i++)
    {
        beta[i] += step[i];
    }
    if (step.Max(Math.Abs) < 1e-8)
    {
        break;                                     // converged
    }
}

Solve is Gaussian elimination with partial pivoting, about 30 lines, which is all the linear algebra a 13 × 13 system needs. The fit converged in 4 iterations. The standard error of each coefficient falls out of the same matrix:

se⁡(βj)=(H−1)jj\operatorname{se}(\beta_j) = \sqrt{\bigl(H^{-1}\bigr)_{jj}}
var errors = Enumerable.Range(0, k).Select(i => Math.Sqrt(Solve(hessian, Unit(k, i))[i])).ToArray();

Guarding against extrapolation. Squared terms explode outside the range the model has seen. So every input is clamped to the 1st–99th percentile of the training records, both when fitting and when predicting (ChanterelleModel.Clamped).

Validation: spatial block cross-validation. A model must be judged on data it was not fitted to. Ordinary cross-validation would cheat here, because finds from the same forest are near-duplicates: a model can score well by memorising places. So whole 100 km blocks are held out together:

var foldOf = blocks.OrderBy(_ => random.Next()).Select((block, i) => (block, fold: i % 5)).ToDictionary(/* … */);
for (var fold = 0; fold < 5; fold++)
{
    var train = rows.Where(row => foldOf[row.Block] != fold).ToList();
    var test = rows.Where(row => foldOf[row.Block] == fold).ToList();
    var beta = Fit(train, columns, weight).Beta;        // fitted without the held-out blocks
    // … score the held-out rows with beta …
}

The score: AUC. The area under the ROC curve is the chance that a randomly chosen find scores higher than a randomly chosen background record. 0.5 is a coin toss, 1 is perfect. It is computed from ranks (the Mann–Whitney statistic): sort all scores, add up the ranks R+R_+ of the n+n_+ finds, and

AUC=R+−n+(n++1)2n+ n−\text{AUC} = \frac{R_+ - \dfrac{n_+(n_+ + 1)}{2}}{n_+ \, n_-}
return positives == 0 || negatives == 0
    ? double.NaN
    : (rankSum - positives * (positives + 1) / 2.0) / ((double)positives * negatives);

Tied scores share the average of their ranks, which the loop above this line handles.

How well the chanterelle model tells finds from background

The outcome. The held-out AUC was 0.580, against 0.532 for a model that knows only how far north a place is. Leaving out any single group of terms changed the AUC by less than 0.01. In plain words: stand data tells chanterelle places from other mushroom places only slightly better than a coin, so the model was not put in the app. The calibration also shows why the choice of background matters: against all fungi the model mostly learned where conservation inventories go, which is why the background was narrowed to mushrooms that foragers report.

How it would be used. ChanterelleModel.LinearPredictor and Ratio are the prediction side, ready if a better fit arrives. They leave out the intercept (it only reflects how many background points were drawn) and report a ratio against average Swedish forest, not a probability:

ratio(x)=eη(x)mean⁡reference eη\text{ratio}(x) = \frac{e^{\eta(x)}}{\operatorname{mean}_{\text{reference}}\, e^{\eta}}
public static double Ratio(ChanterelleSite site, ChanterelleOptions options) =>
    Math.Exp(LinearPredictor(site, options)) / options.ReferenceMeanExp;

7. Running a calibration again

  1. Start the app with aspire run so PostGIS is up and the ingest has run.
  2. In the Aspire dashboard, start the calibration resource. By default it runs excursions, attraction, tourism and fishing. Set Calibration__Models=form-height or chanterelle for the two that are on request (they read the rasters at tens of thousands of points).
  3. Read the report in data/calibration/. Each report prints the configured values next to the best on the grid, so it shows at once whether the stored parameters still hold.
  4. To adopt new values, edit the defaults in the options class, update the XML comment with the date and the numbers, and update docs/calculations.md. Nothing reads the report automatically.

The run-time fits of section 5 need no step: they are rebuilt from the database when the API starts.

A checklist for adding a fitted parameter

  • Keep the model a pure function in the API project, and have the calibration call that function.
  • Compare the model with the data on the data's own terms (the phone-data filter in 6.2 is the example).
  • Hold something back as a guard or a check, so the fit cannot buy accuracy in one place by ruining another.
  • Store low, central and high, not just the best value, and write down in the option's comment what the data could and could not determine.