Distributed Training on Edge Devices. Large Batch vs. Federated Learning

(1/3) An Edgify Research Team Publication

Image for post
Image for post
Image for post
Image for post
Image for post
Image for post

The Basic Idea and Its Challenges

Image for post
Image for post

Large Batch

Federated Learning

The Two Approaches in Comparison

Image for post
Image for post
Image for post
Image for post
Federated learning has a hard time handling batches that are unevenly distributed

A Preliminary Empirical Comparison

Image for post
Image for post
Table 1: The parameters of the different methods
Image for post
Image for post
Figure 1: A graph of the three compared approaches — Large Batch, Federated Learning, and the classic single-server SGD training.
Image for post
Image for post
Table 2: Number of communication rounds required during the training in order to reach an accuracy of 80% in the experiment as above. For Federated Learning, synched once every epoch, this is simply the number of epochs for 80% accuracy. For Large Batch, this is the number of epochs for 80% accuracy, times the number of (3072) batches that go into a single epoch (of 50,000 samples).

Conclusion

Image for post
Image for post
From left to right: Nadav, Itay, Aviv, Neta, Daniel, Tomer, Liron
Image for post
Image for post

Written by

A foundational shift in the world of AI training. Deep Learning and Machine Learning training directly on edge devices.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store