Distributed training on edge devices:
Batch Normalization with Non-IID data

(3/3) an Edgify Research Team Publication

Image for post
Image for post

Non IID Data Distribution

The Impact on Batch Normalization

Image for post
Image for post

Group Normalization to the Rescue

Training Without Normalization — Fixup Initialization

Image for post
Image for post

The Experiments

Non-IID

Image for post
Image for post
Figure 2: Validation accuracy of the IID and Non IID case. The reference (iid BN) is Vanilla Resnet18 trained on IID data. The rest are Resnet18 and variants of Resnet18 (GN for group norm and Fixup for Resnet-Fixup) trained on non IID data. While training with Batch-Norm leads to random results (non iid BN) Group Norm and Fixup resnets achieve accuracy similar to the IID batch-Norm scenario with Group Norm a bit lower.

Large batch training

Image for post
Image for post
Validation accuracy for large batch training on CIFAR10 with Resnet18 after 100 epochs.
Image for post
Image for post
Figure 3: Validation accuracy for Resnet18 with Group Normalization on large batch training. Accuracy drops as batch size increases
Image for post
Image for post
Figure 4: Validation accuracy for Resnet18 Fixup on large batch training.

Federated Learning

Image for post
Image for post
figure 5: Validation accuracy of Resnet18 with group-normalization on Non-IID data. The results show that the validation accuracy drops as the frequency of the synchronization drops.

Conclusion

Image for post
Image for post
From left to right: Nadav, Itay, Aviv, Neta, Daniel, Tomer, Liron
Image for post
Image for post

References

A foundational shift in the world of AI training. Deep Learning and Machine Learning training directly on edge devices.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store