Or just use a large value for steps_per_execution. With step fusing you can maintain high utilization while keeping your global batch size small.
-
-
i was looking for a good reference on the tuning of steps_per_execution recently but couldn't really find one. do you have a recommended reading?
-
Basically just fix your global batch size to something reasonable (each core should get at least a batch >8) then keep increasing steps_per_execution until you max out utilization (or minimize time per epoch). On a TPU pod steps_per_execution=32 or 64 is p good
- Show replies
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.