@fchollet if I've trained 2+ identical Keras models on different training datasets, is there a simple way of merging the trained weights/gradients into one master model?
-
You can average either the gradients (during training), the weights (at the end -- see Polyak averaging for instance) or the final predictions (at inference time)
This assumes equivalent models (datasets drawn from same distribution). For fundamentally different models, only do inference time prediction averaging (possibly a weighted average)
-
-
I.e. only try to average weights or gradients if you have guarantees that every filter in every layer encodes the same thing across the models, which is the case if the models are offshoots of one another (e.g. Polyak averaging) or start with a reproducible seed training phase
-
Awesome, thank you! I will give Polyak averaging a shot. Averaging predictions at inference time doesn't work for my use case, but since I am only fine tuning off of a base model it seems like just averaging gradients/weights should work.
End of conversation
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.