@fchollet check out @bzamecnik 's perf. analysis of #Keras multi-GPU training speedups - many detailed measurements https://github.com/rossumai/keras-multi-gpu/blob/master/blog/docs/measurements.md …https://twitter.com/RossumAi/status/924985495851012096 …
-
Overally we focus on small batch sizes, as I see huge sample sizes in many real world applications, including internally at
@RossumAi.
We have significant overhead (shuffling the params from CPU to GPU at every step; split+concat). Need time(process(sub_batch)) >> overhead
-
-
Not doable to reduce the overhead, so to have a speedup you need to keep your per-sub-batch processing time high (large models or batches)
-
We have ideas to reduce the overhead - https://github.com/rossumai/keras-multi-gpu/blob/master/blog/docs/conclusion.md … Most of it is the samples, first step: StagingArea https://gist.github.com/bzamecnik/f76e480edf98e95ab263fd1a123af7a5 …pic.twitter.com/QdnICU4VAm
- Show replies
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.