Bullet proofing Bayesian deep learning with particle flow

We derive a new theory of Bayesian deep learning with particle flow that bullet proofs the algorithm against stiffness. Bayesian tools, such as STAN, which uses Hamiltonian Monte Carlo, are plagued by extremely stiff flows. Moreover, if you look under the hood of the famous Adam algorithm for deep learning, you will see that Adam was carefully designed to mitigate stiffness. Nevertheless, it is common to use extremely small learning rates for deep learning (e.g., 0.00001) despite the use of Adam!!! Our new theory avoids such a waste of precious GPU resources. As a result, our new theory speeds up Bayesian deep learning by many orders of magnitude. Furthermore, the new theory allows us to avoid fancy stiff ODE solvers in STAN that require a large amount of computer run time, and which are not parallelizable. Our Bayesian particle flow is embarrassingly parallelizable. Our paper generalizes the recent work in Dai & Daum, IEEE AESS Transactions June 2023.