Function vectors as a model diffing tool: 17 heads repair a bad fine-tune
Copying 17 attention heads from the base model into a medical fine-tune gone wrong undoes most of the harmful behavior without costing much capability.
LessWrong
Aniket Ghosh · Writing
Two things, mostly: what's actually going on inside neural networks, and whatever else I've been chewing on. It lands in a few different places. This page pulls it back together.
Copying 17 attention heads from the base model into a medical fine-tune gone wrong undoes most of the harmful behavior without costing much capability.
Probing where a capability lives in the residual stream predicts which layers survive low-bit quantization and which ones you can't touch.