NeuroDiffEq: A Python package for solving differential equations with neural networks

Differential equations emerge in various scientific and engineering domains for modeling physical phenomena. Most differential equations of practical interest are analytically intractable. Traditionally, differential equations are solved by numerical methods. Sophisticated algorithms exist to integrate differential equations in time and space. Time integration techniques continue to be an active area of research and include backward difference formulas and Runge-Kutta methods (Conde, Gottlieb, Grant, & Shadid, 2017). Common spatial discretization approaches include the finite difference method (FDM), finite volume method (FVM), and finite element method (FEM) as well as spectral methods such as the Fourier-spectral method. These classical methods have been studied in detail and much is known about their convergence properties. Moreover, highly optimized codes exist for solving differential equations of practical interest with these techniques (Seefeldt et al., 2017; Smith & Abeysinghe, 2017). While these methods are efficient and well-studied, their expressibility is limited by their function representation. Artificial neural networks (ANN) are a framework of machine learning algorithms that use a collection of connected units to learn function mappings. The most basic form of ANNs, multilayer perceptrons, have been proven to be universal function approximators (Hornik, Stinchcombe, & White, 1989). This suggests the possibility of using ANNs to solve differential equations. Previous studies have demonstrated that ANNs have the potential to solve ordinary differential equations (ODEs) and partial differential equations (PDEs) with certain initial/boundary conditions (Lagaris, Likas, & Fotiadis, 1998). These methods show nice properties including (1) continuous and differentiable solutions, (2) good interpolation properties, and (3) less memory-intensive. By less memory-intensive we mean that only the weights of the neural network have to be stored. The solution can then be recovered anywhere in the solution domain because a trained neural network is a closed form solution. Given the interest in developing neural networks for solving differential equations, it would be extremely beneficial to have an easy-to-use software package that allows researchers to quickly set up and solve problems. NeuroDiffEq is a Python package built with PyTorch (Paszke et al., 2017) that uses ANNs to solve ordinary and partial differential equations (ODEs and PDEs). During the release of NeuroDiffEq we discovered that two other groups had almost simultaneously released their own software packages for solving differential equations with neural networks: DeepXDE (Lu, Meng, Mao, & Karniadakis, 2019) and PyDEns (Koryagin, Khudorozkov, & Tsimfer, 2019). Both DeepXDE and PyDEns are built on top of TensorFlow (Abadi et al., 2015). DeepXDE has an emphasis on the wide variety of problems it can solve. It supports mixing different boundary.