Frederico J.J.B. Santos, Jose Manuel Muñoz Contreras, Berfin Sakallioglu, Alberto Tonda, Leonardo Trujillo
Conference Paper, PPSN - The International Conference on Parallel Problem Solving From Nature, Trento, Italy, 2026 (to appear)
Ensembles of decision trees and Geometric Semantic Genetic Programming (GSGP) both build accurate regressors by aggregating weak learners, yet their exploration of the model’s output semantic space differs in ways that performance metrics cannot reveal. We pair interactive 2D and 3D dimensionality-reduction visualizations with a quantitative metric framework spanning two families: trajectory metrics (path length, straightness, target alignment, step-size schedule, effective dimensionality) and coverage metrics (cloud spread, centroid-to-target, total variance), and use it to compare six ensemble methods across five tabular regression benchmarks. We compare kPCA-cosine, t-SNE, and UMAP qualitatively and via trustworthiness and continuity scores, adopting kPCA-cosine for the visualizations; distance-sensitive metrics are computed in the original semantic space. Three contrasts emerge. Batch Random Forest and Extra Trees weak learners cluster tightly near the target; boosting weak learners (XGBoost, LightGBM, Gradient Boosting) cluster tightly far from it; the GSGP population spans an order-of-magnitude wider neighborhood than either. GSGP learning trajectories are roughly 3× longer than boosting’s, in essentially uncorrelated steps perpendicular to the toward-target direction, and this regime holds population-wide rather than only for the eventual elite. These contrasts identify GSGP-style exploration paired with boosted-tree exploitation as a promising hybrid direction.