Does LiDAR-derived height improve Mask R-CNN tree-crown detection? This CATCH4D Upscaling study tested it on RGB orthophotos of Troisdorf, extending the original RGB-to-LiDAR crown workflow.
Two models used the same training data, settings and random seed: RGB only, and RGB plus relative height (height above nearby surroundings). Both were evaluated on ten test patches with 277 reference crowns.
For this dataset and workflow, adding the tested relative-height layer did not improve detection.
277 reference crowns. A found crown is a correct detection (true positive), a false alarm a detection without a matching crown (false positive), a missed crown a reference crown without a detection (false negative). F1 combines precision and recall (see Method); higher is better.
The models found different crowns. Each model was trained once; variation between training runs was not measured, so individual changes cannot be attributed to height alone.
Select a test patch, then click a crown or a detection box to inspect the recorded result. All ten test patches and 277 reference crowns are included.
Outlines are hand-drawn reference crowns; rectangles are detection boxes. Scoring used predicted pixel masks, so box overlap alone does not establish a correct detection.
The dataset contains 1,220 patches of 512 × 512 pixels, each covering 102.4 m × 102.4 m. Images, reference crowns and aligned height layers are included. The split is shown below.
tree).Mask R-CNN (Matterport implementation, ResNet-50 backbone) performs instance segmentation on 512 × 512 pixel patches. The comparison adds relative height as a fourth image channel, initialised from the mean of the RGB input filters.
The same scene can be represented by colour, height above terrain, or height above nearby surroundings. The panels below show one recorded evaluation patch.
The same evaluation patch in all three panels: 512 × 512 pixels, approximately 102.4 m × 102.4 m on the image grid. The height panels are colour displays of the existing input layers, not vegetation classifications or model predictions. Their colours represent different quantities and should not be compared as a shared numerical scale. Select an image to open it at full size, or inspect the layers and crown annotations in the tree-detection explorer.
A broad elevated surface can have low relative-height values even when it is high above ground. The transform changes which spatial differences stand out.
The normalised digital surface model is nDSM = max(DOM1 − DGM1, 0).
Let S = Gaussian(nDSM, σ = 1 pixel). The relative-height feature is the white top-hat:
Hrel = S − opening(S, disk radius = 12 pixels). On the model-aligned 0.2 m grid,
that radius is 2.4 m. The feature emphasises local height above nearby surroundings.
Raw nDSM is clipped to 0–35 m and relative height to 0–13 m; each is scaled to 0–255 for model input. The controlled comparison used relative height. The separately trained raw-nDSM model is historical context, not a third controlled arm.
An initial diagnostic examined height differences between crowns and nearby background. An exploratory raw-nDSM run followed, but its comparison with the older thesis detector was not controlled because their training conditions differed. A subsequent diagnostic examined whether height could help separate touching crowns and informed the choice of relative height. The matched RGB-only versus RGB + relative-height experiment is the controlled comparison reported here.
Published studies also report different outcomes: Lucena et al. (2022) found an improvement when combining RGB and elevation imagery for orange-tree crowns, whereas Ruschhaupt et al. (2025) reported no improvement from additional height input in their tree-crown experiments. Differences in vegetation, sensors and training design limit direct comparisons.
Precision is the share of detections that are correct, recall the share of reference crowns found, and F1 their harmonic mean. AP50 (average precision at IoU 0.50) summarises the trade-off between precision and recall as detections are added in order of confidence; here it is calculated per test patch from detections with confidence ≥ 0.70 and averaged over the ten patches. A confidence score is the model's own rating, not the probability that a detection is correct.
Only the two models of the controlled comparison may be compared directly. The thesis model was not retrained with the same recipe, and the height-above-ground model was trained separately without a fixed seed; they are shown for context, not as a ranking.