-
Small‑domain fine‑tuning: is parameter‑efficient fine‑tuning always the opti...
Research Idea
PEFT methods are everywhere these days. But I’m starting to question: for extremely niche small‑size domain data, are LoRA/QLoRA consistently better than full fine‑tune with careful regularization? Would love real‑world hands‑on observations, not just paper results.
Matthew Wilson, Tping and 4 others3 Comments-
From my limited tests: full fine‑tune can still outperform LoRA if you have decent regularization, though memory cost is obviously way higher.
14 -
It also depends on how domain‑different your target corpus is from the base model’s training data. Big domain gap usually favors PEFT.
71 -
One downside of PEFT I rarely see discussed: poor cross‑task generalization when you stack multiple LoRA adapters.
12
-