Istanbul Technical University · arXiv:2606.24259 · June 2026

SURGeLLM

Rethinking multi-task evaluation through task-aware feature gating with class-balanced normalization

Noor Islam S. Mohammad¹ · Uluğ Bayazıt² Department of Computer Science, Istanbul Technical University · {islam23, ulugbayazit}@itu.edu.tr ¹Corresponding author ²Supervising author

A lexical feature gate that helps on three heterogeneous tasks and hurts on the fourth — until you fix the statistics it is fed. The architecture was never the problem.

D₄ authorship detection · macro-F₁ · RoBERTa backbone

0.762 Baseline encoder,
no gate

0.711 Add the gate on
globally standardized features

0.892 Same gate, class-balanced
statistics (IWN)

Paper (PDF) add file arXiv Code add URL Slides add URL Poster add URL BibTeX

The surgical feature vector s ∈ ℝ¹⁶ — ten lexical indicator groups, six surface statistics. Hover a cell.

Abstract

Fine-tuned encoders deployed across heterogeneous NLP tasks face three compounding problems: mismatched inductive biases, class-imbalance corruption of feature statistics, and no mechanism to condition attention on external lexical knowledge.

We introduce SURGeLLM, a unified transformer framework that addresses each with a dedicated lightweight module: a surgical feature gate (a learned per-dimension sigmoid over curated lexical indicators and [CLS], which provably degenerates to identity when features are uninformative), task-conditioned prefix tokens (quantized feature values and task identity prepended to every input), and Instance-Weighted Normalization (IWN, which removes class-prior bias from gate statistics). We prove an excess-risk bound linking gate benefit to surgical feature alignment.

Across four tasks — SST-2, multi-hop retrieval, LLM-prompt attribution, and authorship detection — covering 17,830 examples and eleven model variants over three seeds, the IWN variant achieves macro-F₁ 0.940 (+0.034 over the strongest non-IWN variant; +0.036 over the RoBERTa baseline; +0.130 on authorship detection). A random-vocabulary control (−0.028 avg. F₁) confirms gains are lexical, not parametric. Code, vocabularies, and a 99.5%-recovery auto-extraction recipe are released.

What this paper contributes

Framework

A unified multi-task encoder with per-dimension feature gates, task-conditioned prefix tokens, and IWN. Plug-compatible with any HuggingFace encoder; ~1.2M added parameters, about 1% of RoBERTa-base.

Theory

An excess-risk bound (Theorem 1) linking gate benefit to surgical feature alignment ρ_k, and a degeneracy result (Proposition 2) proving the gate is safe when features carry no signal.

Empirics

Eleven variants over four encoder backbones plus T5-base, three seeds, four-part vocabulary sensitivity study, and a random-vocabulary control that separates lexical content from added capacity.

Method

Four modules, one encoder

Let T = {t₁, …, t₄} be a fixed task set, each with label space Y_{t_k}. We fit a single model f_θ: X × T → ∪_k Y_{t_k} minimizing the multi-task empirical risk:

L(θ) = Σ_k=1..|T| w_k|D_k| Σ_{(x,y,t_k) ∈ D_k} ℓ(f_θ(x, t_k), y)

(1)

with cross-entropy loss ℓ and w_k = 1 throughout, relying on per-task batch sampling for balance. The encoder, the feature projection, the gate matrices, the task-embedding matrix, and the prefix embeddings are shared; only the four two-layer classification heads are task-specific. Shared parameters are over 99% of the total, which is what makes the multi-task framing meaningful in the MT-DNN sense.

Figure 1. The forward pass. The prefix injects feature values as in-context tokens visible to every attention layer; the gate re-weights the final [CLS] after all attention has resolved. They operate at different representational scales and are complements, not substitutes — removing either degrades performance.

Surgical feature extraction

For each of ten case-insensitive indicator groups v_j, the count feature is the number of group members present in the lowercased text, with prefix matching for inflectional families (oscillat* matches oscillation, oscillates, oscillating):

s_j = Σ_{w ∈ v_j} 1[ w ∈ x̃ ], j = 1, …, 10

(4)

Six surface features are appended: word count, mean word length, sentence count, question-mark count, exclamation-mark count, and a binary digit-presence indicator. Crucially, all sixteen are computed on the full untruncated text, while the encoder sees at most 96–128 tokens. Global statistics destroyed by truncation cannot be recovered from a partial view — this is the deterministic, lossless side channel the gate consumes.

The gate, in three steps

s′ = ReLU(W_s s + b_s), W_s ∈ ℝ^d×16

(5)

g = σ( W_g [ h̃ ; s′ ] + b_g ), W_g ∈ ℝ^d×2d

(6)

ĥ = LN( g ⊙ h̃ + (1 − g) ⊙ s′ )

(7)

Sigmoid, not softmax

Sigmoid lets each dimension take any value in (0,1) independently. Softmax would impose a unit budget across dimensions — a competitive constraint that makes no sense for dimension-wise modality fusion.

Per-dimension, not scalar

A scalar gate forces every hidden dimension to use the same modality mix. Too coarse when some dimensions encode lexical surface and others encode semantic content.

LayerNorm after fusion

Re-normalizes the fused vector back to the statistical regime of the unfused encoder output, so downstream layers are not surprised by a mean/variance shift.

The central result

Instance-Weighted Normalization

Before projection, the feature vector is standardized using statistics from the training partition. On a skewed corpus, those statistics belong almost entirely to the majority class.

Under global standardization, the mean is a prior-weighted mixture that collapses onto the majority profile as the skew grows:

ŝ(x) = ( s(x) − s̄_k ) / ( σ_k + ε )

(8)

s̄_k = Σ_c π_c s̄_c,k → s̄_c^⋆,k as π_c^⋆ → 1

(9)

The gate is then fed a measurement of deviation from the majority profile. Majority instances sit near the origin and look unremarkable; minority instances — the ones that decide macro-F₁ — sit far out and look like outliers. IWN replaces the marginal statistics with an unweighted average over per-class statistics:

s̄^bal_k = 1n_c,k Σ_{c=1..n_c,k} s̄_c,k, σ^bal_k = 1n_c,k Σ_{c=1..n_c,k} σ_c,k

(10)

s̃(x) = ( s(x) − s̄^bal_k ) / ( σ^bal_k + ε )

(11)

The bench: what class skew does to the gate's input

π = 0.50

Schematic illustration of Eq. 9 — not measured data. Two class-conditional feature profiles are placed at ∓1 in raw units with unit within-class scale. Drag the slider to increase the majority-class share and watch where the two classes land after standardization.

Majority-class share π

majority class minority class dashed line = zero of the gate's input space

Majority position−1.00

Minority position+1.00

Center offset0.00

Same under IWN−1.00 / +1.00

At π = 0.903 — the raw 9.3:1 skew of the HumLLM corpus — the majority class sits at −0.19 and the minority at +1.81. That asymmetry is what the gate learns to fuse against.

What IWN actually fixes

Without IWN, the gate's per-class precision and recall on D₄ come apart: LLM recall 0.63 against human recall 0.79. The model is biased toward predicting "human" because standardization has shifted the gate's input distribution toward the majority class. IWN symmetrizes the per-class statistics and both classes converge to ≈ 0.89.

SURGeLLM-FULL-RoBERTa (no IWN) SURGeLLM-IWN-RoBERTa

Figure 2. Per-class precision and recall on D₄ (HumLLM), RoBERTa backbone, before and after IWN. Measured values from Table 5. The largest single movement is LLM recall, +0.26.

Parameter-free

No new learnable parameters. Only the normalization constants change.

Test-time class-agnostic

The balanced statistics are computed once from training labels and applied at inference with no class information required.

Free on balanced data

When π_c = 1/n_c, the balanced and marginal estimators coincide up to the weighted/unweighted variance difference. IWN is a strict generalization that costs nothing in the balanced regime.

Compositional

Orthogonal to loss-side and sampling-side remedies — focal loss, class-balanced re-weighting, oversampling. We report IWN-only results for clarity.

Datasets

Four tasks chosen to interfere

Near-isotropic benchmarks do not expose destructive interference. We study the hard case: a single encoder handling movie-review sentiment, multi-hop retrieval QA, LLM-prompt attribution, and human/LLM authorship — tasks that share a backbone but draw on largely disjoint surface signals. All corpora use stratified 70/15/15 splits, label reindexing, and training-only computation of the normalization statistics.

**Table 1.** Corpus statistics after stratified capping. % min. = minority-class percentage in the capped subset.
Task	ID	n	n_c	% min.	Source	Raw skew
Sentiment	D₁	7,666	2	49.5	SST-2 (GLUE)	—
Retrieval	D₂	2,000	2	49.0	HotPotQA	—
Generation	D₃	3,164	2	50.0	LLM-7	≈ 11.8 : 1
Authorship	D₄	5,000	2	50.0	HumLLM	9.3 : 1
Total	—	17,830	—	—	—	—

Table 1. D₄ is capped to 50/50, yet IWN still helps: normalization statistics are computed on the full training data, and because P(s | y) differs in its moments across classes, residual imbalance effects survive the capping.

D₁ — SST-2

Binary movie-review sentiment. Standard GLUE training split (67,349), official validation set (872) as our test set, with a stratified 10% internal validation slice; capped at 7,666 for parity. Chosen because it is non-saturated on base-scale encoders (87–94% published accuracy) and exercises the polarity vocabulary the gate can act on.

D₂ — HotPotQA

Multi-hop QA. Input is [Q] q [CTX] c_:300; the binary label collapses the three-tier difficulty annotation (easy → 0, medium/hard → 1). Attribution phrases such as according to and the article reports are rare in questions and frequent in contexts, giving the retrieval group a clean discriminative signal.

D₃ — LLM-7

14,877 essays under seven prompt conditions, stratified-capped to 3,164. Probes the llm_stat, llm_formal, and llm_list groups on longer, prompt-structured text.

D₄ — HumLLM

5,000 balanced examples sampled from a 788,922-text corpus with 9.3:1 original skew. The hardest task in the suite — base models stay under 0.77 macro-F₁ without IWN — and where IWN yields its largest gain.

Experimental setup

How everything was trained

Backbones

DistilBERT-base-uncased (66M)
BERT-base-uncased (110M)
RoBERTa-base (125M)
ALBERT-base-v2 (11M)
T5-base (220M), as a text-to-text comparison

Optimization

AdamW (λ = 0.01, β₁ = 0.9, β₂ = 0.999, ε = 10⁻⁸), 6% linear warmup then decay, gradients clipped at 1.0. Learning rates: 2×10⁻⁵ for Baseline and SURGeLLM-S, 1.5×10⁻⁵ for SURGeLLM-G/FULL/IWN, 3×10⁻⁴ for T5.

Hardware and budget

2× NVIDIA T4 (16 GB), FP16 via Accelerate, effective batch size 32 through accumulation. Pre-tokenization caching (chunk size 2,048) gives roughly a 25% speedup. Early stopping with patience 2 on validation macro-F₁. Full reproduction is about 38 GPU-hours.

Statistics

All results are mean ± SD over seeds {0, 1, 2}. Significance is paired Welch t-tests with Benjamini–Hochberg correction at FDR = 0.05 over 4 × 4 task-variant comparisons, plus 95% bootstrap confidence intervals (B = 2,000). Reported metrics: accuracy, macro-F₁, precision, recall, ROC-AUC.

**Table 2.** Model variants. P = surgical prefix, G = gate, E = extended training, I = IWN.
Variant	P	G	E	I	What it isolates
Baseline	✗	✗	✗	✗	Fine-tuned encoder, task heads only
T5-base	—	—	—	—	Text-to-text framing of the same suite
SURGeLLM-G	✓	✗	✗	✗	Prefix alone
SURGeLLM-S	✓	✓	✗	✗	Prefix + gate, standard schedule
SURGeLLM-FULL	✓	✓	✓	✗	Adds extended training
SURGeLLM-IWN	✓	✓	✓	✓	Adds class-balanced statistics

Main results

Eleven variants, four tasks, three seeds

D₁ is non-saturated — the F₁ spread across variants is 0.901–0.937 — so aggregate averages reflect genuine differences rather than ceiling effects. SURGeLLM-IWN-RoBERTa is the top model at 0.940 average F₁, ahead of the strongest non-IWN variant by +0.034 and the RoBERTa baseline by +0.036. The improvement is driven almost entirely by D₄: +0.130 over baseline, which fully offsets the gate-induced drop seen without IWN.

**Table 3.** Macro-F₁, mean ± SD over three seeds. Bold = best per column. † = early stopping triggered. T(s) = mean wall-clock training time on 2×T4. ∆ = Avg F₁ against Baseline-RoBERTa.
Model	Par.	D₁ SST-2	D₂ HotPot	D₃ LLM-7	D₄ HumLLM	Avg F₁	∆	T(s)
Reference models
T5-base	220M	0.928 ± .005	0.939 ± .007	0.972 ± .004	0.748 ± .013	0.897	−0.007	412
Baseline-DistilBERT	66M	0.901 ± .006	0.940 ± .008	0.955 ± .006	0.749 ± .012	0.886	−0.018	82
Baseline-BERT	110M	0.918 ± .004	0.934 ± .007	0.963 ± .005	0.760 ± .011	0.894	−0.010	227
Baseline-RoBERTa	125M	0.929 ± .004	0.947 ± .006	0.978 ± .003	0.762 ± .010	0.904	—	233
SURGeLLM without IWN
SURGeLLM-S-DistilBERT	66M	0.911 ± .007	0.961 ± .006	0.925 ± .009	0.681 ± .013	0.870	−0.034	119
SURGeLLM-S-BERT	110M	0.926 ± .005	0.939 ± .007	0.965 ± .004	0.748 ± .011	0.894	−0.010	317
SURGeLLM-G-RoBERTa †	125M	0.937 ± .004	0.949 ± .005	0.977 ± .003	0.760 ± .010	0.906	+0.002	327
SURGeLLM-FULL-RoBERTa †	125M	0.932 ± .005	0.950 ± .006	0.961 ± .005	0.711 ± .012	0.889	−0.015	326
SURGeLLM-FULL-ALBERT	11M	0.918 ± .006	0.961 ± .005	0.957 ± .005	0.708 ± .013	0.886	−0.018	317
SURGeLLM with IWN
SURGeLLM-IWN-RoBERTa	125M	0.933 ± .004	0.954 ± .005	0.979 ± .003	0.892 ± .009	0.940	+0.036	332
SURGeLLM-IWN-BERT	110M	0.927 ± .005	0.946 ± .006	0.968 ± .004	0.866 ± .010	0.927	+0.023	322

Table 3. Retrieval gains are consistent — SURGeLLM-S-DistilBERT and SURGeLLM-FULL-ALBERT both reach 0.961 ± .006 on D₂, clearly above their baselines. T5-base is competitive at 0.897 but neither dominant nor efficient.

Statistical significance

**Table 4.** Benjamini–Hochberg-corrected p-values for selected comparisons. Bold = p < 0.05.
Comparison	Task	p (BH)
SURGeLLM-S-DistilBERT vs. Base-DistilBERT	D₂	0.008
SURGeLLM-FULL-ALBERT vs. Base-RoBERTa	D₂	0.011
SURGeLLM-IWN-RoBERTa vs. Base-RoBERTa	D₂	0.024
SURGeLLM-IWN-RoBERTa vs. Base-RoBERTa	D₄	< 0.001
SURGeLLM-IWN-RoBERTa vs. SURGeLLM-FULL	D₄	< 0.001
SURGeLLM-IWN-BERT vs. Base-BERT	D₄	< 0.001
SURGeLLM-G-RoBERTa vs. Base-RoBERTa	D₁	0.063
SURGeLLM-S-BERT vs. Base-BERT	D₁	0.082
All D₁/D₃ pairs (avg.)	—	> 0.05

Table 4. The IWN gains on D₄ are the strongest effects in the study. Differences on D₁ and D₃ mostly sit inside seed noise — which is what Proposition 2 predicts: where surgical alignment is moderate, the gate degenerates harmlessly toward a near-identity map.

Why text-to-text does not dominate

T5-base trains in 412s against 233s for Baseline-RoBERTa (a 1.77× wall-clock penalty), carries 220M parameters against 125M (a 1.76× parameter penalty), and still trails Baseline-RoBERTa by 0.007 avg. F₁ and SURGeLLM-IWN-RoBERTa by 0.043. Text-to-text framing is most powerful when tasks share a unifying linguistic structure; our four tasks are structurally heterogeneous, and T5's encoder-decoder must allocate capacity to a decoding side that classification does not need.

Ablations

Reading the components apart

The progression G → FULL → IWN on RoBERTa tells the cleanest story. The prefix alone is mildly beneficial (+0.002). Adding the gate without IWN is harmful (−0.015 average, dominated by D₄'s −0.051). Adding IWN reverses the regression and exceeds it (+0.036). The BERT rows show the same pattern.

**Table 5.** Component ablation across backbones. ∆ = Avg F₁ against the same-backbone baseline.
Model	P	G	E	I	D₁	D₂	D₃	D₄	Avg	∆
Baseline-RoBERTa	✗	✗	✗	✗	0.929	0.947	0.978	0.762	0.904	—
SURGeLLM-G-RoBERTa	✓	✗	✗	✗	0.937	0.949	0.977	0.760	0.906	+.002
SURGeLLM-FULL-RoBERTa	✓	✓	✓	✗	0.932	0.950	0.961	0.711	0.889	−.015
SURGeLLM-IWN-RoBERTa	✓	✓	✓	✓	0.933	0.954	0.979	0.892	0.940	+.036
Baseline-BERT	✗	✗	✗	✗	0.918	0.934	0.963	0.760	0.894	—
SURGeLLM-S-BERT	✓	✓	✗	✗	0.926	0.939	0.965	0.748	0.894	±.000
SURGeLLM-IWN-BERT	✓	✓	✓	✓	0.927	0.946	0.968	0.866	0.927	+.033
Baseline-DistilBERT	✗	✗	✗	✗	0.901	0.940	0.955	0.749	0.886	—
SURGeLLM-S-DistilBERT	✓	✓	✗	✗	0.911	0.961	0.925	0.681	0.870	−.016
SURGeLLM-FULL-ALBERT	✓	✓	✓	✗	0.918	0.961	0.957	0.708	0.886	—

Table 5. No ALBERT baseline was trained, so the ALBERT row carries no ∆.

Isolating IWN

**Table 6.** IWN ablation with the D₄ per-class breakdown. F₁ means over three seeds; ∆ is IWN against SURGeLLM-FULL on the same backbone.
Variant	D₁	D₂	D₃	D₄	Hum. P	Hum. R	LLM P	LLM R
SURGeLLM-FULL-RoBERTa	0.932	0.950	0.961	0.711	0.71	0.79	0.71	0.63
SURGeLLM-IWN-RoBERTa	0.933	0.954	0.979	0.892	0.89	0.89	0.89	0.89
∆ (RoBERTa)	+.001	+.004	+.018	+.181	+.18	+.10	+.18	+.26
∆ (BERT)	+.001	+.006	+.003	+.118	+.13	+.07	+.14	+.18

Training dynamics

SURGeLLM models start from a higher initial loss (≈ 1.7–2.1) because the encoder must simultaneously become useful for four heterogeneous tasks and coordinate with the gate and prefix. They converge to comparable validation F₁ within 4–5 epochs. Early stopping fires at epoch 4 for SURGeLLM-FULL-RoBERTa and SURGeLLM-G-RoBERTa, saving roughly one epoch (≈ 325s) with no test-F₁ regression.

**Table 7.** Training dynamics, seed-0 representative. ∆Loss = (epoch-1 loss) − (final loss).
Model	Init. loss	Final loss	Best ep.	∆Loss
Baseline-DistilBERT	0.583	0.179	3	0.404
Baseline-BERT	0.508	0.139	3	0.370
Baseline-RoBERTa	0.543	0.148	3	0.395
T5-base	1.234	0.412	4	0.822
SURGeLLM-S-DistilBERT	2.019	0.736	4	1.282
SURGeLLM-S-BERT	1.904	0.616	3	1.087
SURGeLLM-G-RoBERTa †	1.708	0.447	2	1.262
SURGeLLM-FULL-RoBERTa †	2.086	0.682	2	1.404
SURGeLLM-FULL-ALBERT	1.905	0.510	4	1.395
SURGeLLM-IWN-RoBERTa	1.812	0.421	3	1.391
SURGeLLM-IWN-BERT	1.847	0.503	3	1.344

Vocabulary sensitivity

Is the curated vocabulary doing the work?

Four complementary studies on SURGeLLM-G-RoBERTa, all over three seeds. The question behind them: are the gains lexical, or are they the extra parameters the feature channel happens to bring along?

**Table 8.** Sensitivity to the number of indicator groups |V|.
\|V\|	D₁	D₂	D₃	D₄	Avg
0 (none)	0.929	0.947	0.978	0.762	0.904
5	0.931	0.948	0.977	0.760	0.904
10 (ours)	0.937	0.949	0.977	0.760	0.906
15	0.935	0.950	0.976	0.755	0.904
20	0.933	0.949	0.974	0.748	0.901

Table 8. Performance plateaus around ten groups. Any value in {10, 15} is statistically indistinguishable, so the system is not sharply tuned to the exact count; twenty groups costs 0.012 on D₄ through noise from redundant thesaurus variants.

**Table 9.** Random-vocabulary control and auto-extracted alternative.
Vocabulary	D₁	D₂	D₃	D₄	Avg
None (baseline)	0.929	0.947	0.978	0.762	0.904
Random (BNC)	0.910	0.928	0.946	0.728	0.878
Auto-extracted	0.934	0.948	0.974	0.755	0.903
Curated	0.937	0.949	0.977	0.760	0.906
∆ Random	−.027	−.021	−.031	−.032	−.028
∆ Auto	−.003	−.001	−.003	−.005	−.003

Table 9. Random vocabulary of the same cardinality is worse than no vocabulary at all. The gate responds to semantic content, not added capacity.

**Table 10.** Surface-features ablation. G = lexical groups, S = surface statistics.
Config.	D₁	D₂	D₃	D₄	Avg
G + S (full)	0.937	0.949	0.977	0.760	0.906
G only	0.935	0.946	0.974	0.749	0.901
S only	0.928	0.945	0.974	0.755	0.901
∆ no-S	−.002	−.003	−.003	−.011	−.005
∆ no-G	−.009	−.004	−.003	−.005	−.005

Table 10. Neither half is redundant. Surface statistics matter most on D₄, where length and punctuation density separate human from LLM text; lexical groups matter most on D₁, where polarity vocabulary is decisive.

Why surface features survive the encoder

Truncation. The encoder receives at most 96–128 tokens. Total word count and total exclamation count are computed on the full document and carry information unavailable from a partial view.

Distributional shift. Even without truncation, the [CLS] representation is optimized for masked-token prediction and need not preserve precise count statistics. The surgical channel supplies them deterministically and losslessly.

Per-group leave-one-out

Figure 3. Macro-F₁ drop when each indicator group is removed and the model retrained (SURGeLLM-G-RoBERTa). Each task has a clearly dominant group, and cross-task leakage is minimal — removing a task-specific group rarely moves another task by more than 0.002. The leave-one-out pattern matches the design intent, which is what makes the gate interpretable.

Transfer without manual curation

For a new language or domain: extract candidate indicator words by class-conditional log-odds with an informative Dirichlet prior on each task's training set, rank by absolute log-odds and keep the top K = 50 per class, embed the union with SBERT, then k-means into ten clusters. Cluster membership becomes the indicator groups; surface features are unchanged. This recovers 0.903 against 0.906 for manual curation.

**Table 11.** Preliminary multilingual result. SURGeLLM-G-XLM-R-base with auto-extracted per-language vocabularies, 5,000 capped training examples, official test splits, three seeds.
Configuration	French (AlloCiné)	German
XLM-R-base baseline	0.917	0.872
SURGeLLM-G-XLM-R-base (auto)	0.926	0.881
∆	+.009	+.009

Table 11. Both gains land within 0.001 of the English-curated gain on D₁ (+0.008), suggesting the recipe transfers without per-language manual curation. A full-scale multilingual study remains future work.

Efficiency

What the gate costs

The gate adds 2d² + d ≈ 1.18M parameters, the feature projection ≈ 12.5K, the task embedding ≈ 3K — about 1.2M total, roughly 1% of RoBERTa-base. Asymptotically the surgical components are Θ(d²) against the encoder's Θ(L·d²), negligible for L ≫ 1. The real cost is wall-clock: about 100s on 2×T4 (233s → 332s), a 43% overhead driven mainly by extended training and prefix tokenization.

**Table 12.** Speed–accuracy trade-off. F₁/min = F₁ × 60 / T(s). Eff = F₁ × 10³ / log₁₀ P, with P in millions of parameters. ⋆ = on the Pareto frontier.
Model	Par.	T(s)	Overhead	Avg F₁	F₁/min	Eff
Baseline-DistilBERT ⋆	66M	82	1.0×	0.886	0.648	487.0
SURGeLLM-S-DistilBERT	66M	119	1.5×	0.870	0.439	478.2
Baseline-BERT ⋆	110M	227	2.8×	0.894	0.236	437.7
Baseline-RoBERTa ⋆	125M	233	2.8×	0.904	0.233	431.1
SURGeLLM-S-BERT	110M	317	3.9×	0.894	0.169	437.7
SURGeLLM-FULL-ALBERT	11M	317	3.9×	0.886	0.168	848.0
SURGeLLM-FULL-RoBERTa	125M	326	4.0×	0.889	0.164	423.9
SURGeLLM-G-RoBERTa ⋆	125M	327	4.0×	0.906	0.166	432.0
SURGeLLM-IWN-BERT	110M	322	3.9×	0.927	0.173	453.9
SURGeLLM-IWN-RoBERTa ⋆	125M	332	4.0×	0.940	0.170	448.3
T5-base	220M	412	5.0×	0.897	0.131	380.4

Table 12. SURGeLLM-FULL-ALBERT is the most parameter-efficient configuration, reaching 0.886 avg. F₁ with 11M parameters. T5-base is dominated on both axes.

Theory

When the gate helps, and why it cannot hurt

Definition 1 — Surgical feature alignment

For task t_k and input distribution P_{t_k}, the alignment ρ_k is the expected absolute inner product between the projected feature vector and the gradient of the conditional log-likelihood at the fused representation:

ρ_k = 𝔼_{(x,y) ~ P_{t_k}} [ | ⟨ s′(x), ∇_ĥ log p(y | ĥ) ⟩ | ]

(15)

When ρ_k is high, perturbing ĥ along s′ moves the log-likelihood a lot, so s′ encodes information about y. When ρ_k = 0, s′ is orthogonal in expectation to the score function and carries no task-relevant signal.

Theorem 1 — Gate approximation bound

Let f^⋆ be the Bayes-optimal classifier for task t_k and f_θ a SURGeLLM classifier from ERM on N_k examples. If the encoder is L_φ-Lipschitz, the head map L_head-Lipschitz, and the loss ρ-Lipschitz in its first argument, then with probability at least 1 − δ:

R(f_θ) − R(f^⋆) ≤ C√N_k + λ_max(W_g^⊤W_g)2 ‖ s′ − s^⋆ ‖²

(16)

The first term is standard: more data shrinks the generalization gap. The second is the novel piece — small when the projected feature vector is close to its oracle s^⋆, large when the gate matrix has high spectral norm. Highly aligned tasks benefit more, because s′ then carries useful signal that even a modest W_s approximates well.

Proposition 2 — Degeneracy under zero alignment

If ρ_k = 0, then at any local minimum of the regularized loss with weight decay λ > 0: ‖W_s‖ → 0, hence s′(x) → 0, hence g_i^⋆ → 1 for every dimension, and the fusion collapses to ĥ → LN(h̃). The expected gradient vanishes, the update reduces to pure exponential decay W_s ← (1 − λη)W_s, and to minimize loss the gate routes all signal through h̃.

Corollary 3 — Safety of adding the gate

For any task with ρ_k = 0, adding the surgical gate cannot increase the minimum achievable empirical risk. The gate either strictly improves (ρ_k > 0) or degenerates to identity (ρ_k = 0).

Empirical caveat — the imbalance loophole

Corollary 3 assumes ρ_k accurately captures gradient-feature alignment under the distribution the gate actually sees. Under severe class skew, Eq. 8 feeds the gate prior-contaminated features, and the effective ρ_k measured on that contaminated distribution can be misleadingly low even when the underlying feature signal is informative.

This is precisely the failure mode observed on D₄ without IWN — and it is why the safety guarantee did not save us. IWN restores the conditions of Proposition 2 on imbalanced data.

**Table 13.** Monte-Carlo estimates of ρ_k on the validation split, 1,000 examples per task, by backpropagating to obtain ∇_ĥ log p(y | ĥ).
Task	ρ_k (no IWN)	ρ_k (IWN)	∆F₁ from IWN
D₁ SST-2	1.42	1.39	+.001
D₂ HotPot	3.71	3.68	+.004
D₃ LLM-7	1.83	1.85	+.018
D₄ HumLLM	0.61	2.13	+.181

Table 13. The empirical gain ordering tracks the alignment ordering. D₄ is the only task whose alignment changes under IWN — from 0.61 to 2.13 — because it is the only task whose gate input was prior-contaminated. Alignment on the other three tasks is stable to within measurement noise, exactly as a normalization-only change should behave.

Failure analysis and limitations

Where this breaks

We manually inspected 50 misclassified examples per task on SURGeLLM-IWN-RoBERTa. The failures are diagnostic — they mark the boundary of the gate's utility rather than random error.

D₁ — SST-2

Negation scope (not bad), sarcasm, and mixed-sentiment reviews. The gate cannot help: polarity vocabulary fires on both sides of the decision.

D₂ — HotPotQA

Questions with implicit multi-hop chains and no explicit attribution cues. The retrieval group has nothing to fire on.

D₃ — LLM-7

Human essays that mimic LLM scaffolding in a formal academic register, and LLM essays edited by humans to strip enumerative markers.

D₄ — HumLLM

After IWN, the residual failures concentrate on short texts (< 30 words), where surgical-feature counts are unreliable. This motivates length-conditional gating.

Stated limitations

Experiments are English-only; the multilingual result in Table 11 is preliminary and small-scale.
Coverage is base-scale encoders, 11M–220M parameters. Extension to LLaMA-class encoders is explicit future work, and we expect absolute gains to shrink as encoder capacity saturates ρ_k — though the do-no-harm guarantee holds at all scales.
The bound is standard Rademacher complexity and may be loose for modern transformers. PAC-Bayes or NTK tightening is open.
Evaluation is four heterogeneous tasks by design, not the full GLUE/SuperGLUE suite. That is the point of the study, but it does limit external comparability.
IWN is reported alone, without composition against focal loss, class-balanced re-weighting, or oversampling. Whether the gains are additive is untested.

Revision record

What changed after review

This version responds to four reviewer reports and a meta-review. We publish the crosswalk because the most interesting result in the paper — IWN — exists only because reviewers refused to accept it as future work.

**Table 14.** Reviewer-to-revision crosswalk. ✓ = fully addressed; ∼ = partially addressed with a future-work note.
Raised by	Concern	Change	Status
Qs1u, EVkC, Meta	Class imbalance on D₄ corrupts gate statistics; IWN was deferred to future work	IWN fully implemented and evaluated	✓
Qs1u, idHo	No sensitivity analysis of the surgical vocabulary; unclear why exactly ten groups; surface features may be redundant	Four-part sensitivity study added, including a random-vocabulary control	✓
4Pvq, idHo, Meta	The original D₁ (synthetic physics oscillation) saturated at F₁ = 1.000 and inflated reported averages	D₁ replaced with SST-2; all aggregates recomputed	✓
idHo	No comparison to unified text-to-text models	T5-base added as an eleventh variant	✓
Qs1u, 4Pvq	Single-seed results weaken confidence in small F₁ differences	All results re-run over seeds {0,1,2}; mean ± SD throughout, per-seed breakdown published	✓
Qs1u	Abstract overclaimed "state-of-the-art performance"	Revised to "competitive parameter-efficient multi-task performance" with the bootstrap CI overlap stated explicitly	✓
idHo	No multilingual or cross-domain evaluation	Preliminary French/German experiment added; full study deferred	∼

Table 14. Removing the saturated D₁ narrowed bootstrap CI widths from ≈ 0.17 to ≈ 0.12 and turned a within-CI gap of ∆ = 0.001 into a separation of ∆ = +0.036 (p < 0.001). The headline comparison is now SURGeLLM-IWN-RoBERTa 0.940 ± .003, 95% CI [0.934, 0.946], against Baseline-RoBERTa 0.904 ± .003.

Reproducibility

Everything you need to re-run this

Code — released with preprocessing scripts that reproduce the stratified splits.
Data — all four corpora are publicly available.
Seeds — {0, 1, 2}; splits, weight initialization, dropout masks, and CUDA determinism are all seeded.
Software — PyTorch 2.1, Transformers 4.35, Accelerate 0.24, scikit-learn 1.3, sentence-transformers 2.2.
Hardware — 2× NVIDIA T4 (16 GB), FP16 mixed precision.
Statistics — bootstrap B = 2,000 (seed 0); paired Welch t-tests with BH FDR = 0.05.
Compute — ≈ 38 GPU-hours on T4 to reproduce all main and ablation results.

**Table 15.** Per-seed average F₁ for selected models.
Model	Seed 0	Seed 1	Seed 2	Mean
Baseline-RoBERTa	0.906	0.901	0.905	0.904
SURGeLLM-G-RoBERTa	0.908	0.902	0.908	0.906
SURGeLLM-FULL-RoBERTa	0.892	0.886	0.889	0.889
SURGeLLM-IWN-RoBERTa	0.943	0.937	0.940	0.940
SURGeLLM-IWN-BERT	0.929	0.924	0.928	0.927
T5-base	0.900	0.893	0.898	0.897

Hyperparameters

**Table 16.** Full configuration. LR = learning rate, EP = max epochs, BS = per-GPU batch size, GA = gradient accumulation, MaxL = max sequence length, WU = warmup fraction.
Model	LR	EP	BS	GA	MaxL	WU
Baseline-DistilBERT	2×10⁻⁵	3	32	1	96	0.06
Baseline-BERT	2×10⁻⁵	3	16	2	128	0.06
Baseline-RoBERTa	2×10⁻⁵	3	16	2	128	0.06
T5-base	3×10⁻⁴	5	8	4	128	0.06
SURGeLLM-S-DistilBERT	2×10⁻⁵	4	32	1	96	0.06
SURGeLLM-S-BERT	2×10⁻⁵	4	16	2	128	0.06
SURGeLLM-G-RoBERTa	1.5×10⁻⁵	4	16	2	128	0.06
SURGeLLM-FULL-RoBERTa	1.5×10⁻⁵	5	16	2	128	0.06
SURGeLLM-FULL-ALBERT	2×10⁻⁵	5	32	1	96	0.06
SURGeLLM-IWN-RoBERTa	1.5×10⁻⁵	5	16	2	128	0.06
SURGeLLM-IWN-BERT	2×10⁻⁵	5	16	2	128	0.06

The surgical vocabulary

Ten case-insensitive indicator groups; * marks prefix matching for inflectional families.

**Table 17.** Complete indicator groups, plus the six appended surface features.
Group	Members
sst_pos	great, excellent, brilliant, terrific, wonderful, masterpiece, captivat*, impressive, delightful, superb
sst_neg	terrible, awful, dreadful, unwatchable, boring, dull, mediocre, disappoint*, worst, painful
llm_stat	empirically, statistically, demonstrated, observed, evidenced, indicate, suggest, results show, data show
llm_formal	moreover, furthermore, additionally, consequently, therefore, in conclusion, in summary, to summarize
llm_list	firstly, secondly, thirdly, finally, in addition, on the other hand, (1), (2), (3)
human_pers	i, my, we, our, personally, i think, i believe, i feel
human_hedge	maybe, perhaps, possibly, kind of, sort of, i guess, probably, somewhat, arguably
human_emo	love, hate, amazing, awesome, terrible, awful, fantastic, horrible, sad, happy
retrieval	according to, as stated in, the article reports, the text states, multi-hop, supporting context, in the passage
prompt_cot	step by step, let us think, first, then, next, reasoning, the chain of thought, walk through
Surface features (s₁₁ – s₁₆)
s₁₁ – s₁₆	word count · mean word length · sentence count (split on .!?) · question-mark count · exclamation-mark count · binary digit presence

Citation

Cite this work

@inproceedings{mohammad-bayazit-2026-surgellm,
    title = "{SURGELLM}: Rethinking Multi-Task Evaluation through Task-Aware Feature Gating with Class-Balanced Normalization",
    author = "Mohammad, Noor Islam S.  and
      Bayazit, Ulug",
    editor = "Chang, Kai-Wei  and
      Mehrabi, Ninareh  and
      Krishna, Satyapriya  and
      Das, Anubrata  and
      Dhamala, Jwala  and
      Cao, Yang Trista  and
      Kumarage, Tharindu  and
      Ramakrishna, Anil  and
      Christodoulopoulos, Christos  and
      Wan, Yixin  and
      Galystan, Aram  and
      Kumar, Anoop  and
      Gupta, Rahul",
    booktitle = "Proceedings of the 6th Workshop on Trustworthy {NLP} ({T}rust{NLP} 2026)",
    month = jul,
    year = "2026",
    address = "San Diego, California",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.trustnlp-main.47/",
    doi = "10.18653/v1/2026.trustnlp-main.47",
    pages = "600--617",
    ISBN = "979-8-89176-418-7"
}

Published at the 6th Workshop on Trustworthy NLP (TrustNLP 2026), San Diego, California · ACL Anthology · doi:10.18653/v1/2026.trustnlp-main.47

SURGeLLM

What this paper contributes

Framework

Theory

Empirics

Four modules, one encoder

Surgical feature extraction

The gate, in three steps

Sigmoid, not softmax

Per-dimension, not scalar

LayerNorm after fusion

Instance-Weighted Normalization

What IWN actually fixes

Parameter-free

Test-time class-agnostic

Free on balanced data

Compositional

Four tasks chosen to interfere

D1 — SST-2

D2 — HotPotQA

D3 — LLM-7

D4 — HumLLM

How everything was trained

Backbones

Optimization

Hardware and budget

Statistics

Eleven variants, four tasks, three seeds

Statistical significance

Why text-to-text does not dominate

Reading the components apart

Isolating IWN

Training dynamics

Is the curated vocabulary doing the work?

Why surface features survive the encoder

Per-group leave-one-out

Transfer without manual curation

What the gate costs

When the gate helps, and why it cannot hurt

Where this breaks

D1 — SST-2

D2 — HotPotQA

D3 — LLM-7

D4 — HumLLM

Stated limitations

What changed after review

Everything you need to re-run this

Hyperparameters

The surgical vocabulary

Cite this work

D₁ — SST-2

D₂ — HotPotQA

D₃ — LLM-7

D₄ — HumLLM

D₁ — SST-2

D₂ — HotPotQA

D₃ — LLM-7

D₄ — HumLLM