Convex Shape Representation Method with Fast Algorithms and Numerical Analysis for Solving PDEs by Deep Learning

This thesis contains two parts. A general introduction to the main contributions is given in Chapter 1. The first part is included in Chapter 2 and 3. The second part is included in Chapter 4 and 5.
In the first part, we present a new method for convex shape representation, which is regardless of the dimension of objects. To the best of our knowledge, the proposed prior is the first one which can work for high dimensional objects.
We first prove that the convexity of the considered object is equivalent to the convexity of the associated signed distance function. Then, the second order condition of convex functions is used to characterize the shape convexity equivalently.
We apply this new method to two applications: object segmentation with convexity prior and convex hull problem (especially with outliers). We also propose algorithms based on the alternating direction method of multipliers to solve these models. Numerical experiments are conducted to verify the effectiveness of proposed representation methods and algorithms. This part has been published in [38] and [39].
In the second part, we derive an a priori error estimate for the mixed residual method solving some elliptic PDEs by neural networks. Our work is the first theoretical study of this method. We prove that the neural network solutions will converge if we increase the training samples and network size.
Besides, our results suggest that the mixed residual method can recover high order derivatives better than the deep Ritz method. This part is included in the preprint [40] and [41].

[1] R. Acar and C. R. Vogel, Analysis of bounded variation penalty methodsfor ill-posed problems, Inverse Problems, 10 (1994), pp. 1217–1229.
[2] E. Bae, X.-C. Tai, and W. Zhu, Augmented lagrangian method for anEuler’s elastica based segmentation model that promotes convex contours,Inverse Problems & Imaging, 11 (2017), pp. 1–23.
[3] A. R. Barron, Universal approximation bounds for superpositions of asigmoidal function, IEEE Transactions on Information Theory, 39 (1993),pp. 930–945.
[4] Y. Boykov, O. Veksler, and R. Zabih, Markov random fieldswith efficient approximations, in Proceedings. 1998 IEEE Computer SocietyConference on Computer Vision and Pattern Recognition (Cat. No.98CB36231), IEEE, 1998, pp. 648–655.
[5] Y. Boykov, O. Veksler, and R. Zabih, Fast approximate energyminimization via graph cuts, IEEE Transactions on Pattern Analysis andMachine Intelligence, 23 (2001), pp. 1222 – 1239.
[6] Z. Cai, J. Chen, M. Liu, and X. Liu, Deep least-squares methods:An unsupervised learning-based numerical method for solving elliptic pdes,Journal of Computational Physics, 420 (2020), p. 109707.
[7] Z. Cai, R. Lazarov, T. A. Manteuffel, and S. F. McCormick,First-order system least squares for second-order partial differential equations:Part i, SIAM Journal on Numerical Analysis, 31 (1994), pp. 1785–1799.
[8] A. Chambolle, V. Caselles, D. Cremers, M. Novaga, andT. Pock, An introduction to total variation for image analysis, TheoreticalFoundations and Numerical Methods for Sparse Recovery, 9 (2010),p. 227.
[9] T. Chan and L. Vese, An active contour model without edges, inInternational Conference on Scale-Space Theories in Computer Vision,Springer, 1999, pp. 141–151.
[10] T. F. Chan and L. A. Vese, Active contours without edges, IEEETransactions on Image Processing, 10 (2001), pp. 266–277.
[11] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan,Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao,L. Yi, and F. Yu, ShapeNet: An Information-Rich 3D Model Repository,Tech. Rep. arXiv:1512.03012 [cs.GR], Stanford University — PrincetonUniversity — Toyota Technological Institute at Chicago, 2015.
[12] B. Chazelle, On the convex layers of a planar set, IEEE Transactionson Information Theory, 31 (1985), pp. 509–517.
[13] R. T. Chen, Y. Rubanova, J. Bettencourt, and D. Duvenaud,Neural ordinary differential equations, arXiv preprint arXiv:1806.07366,(2018).
[14] M. G. Crandall, H. Ishii, and P.-L. Lions, User’s guide to viscositysolutions of second order partial differential equations, Bulletin of theAmerican Mathematical Society, 27 (1992), pp. 1–67.
[15] G. Cybenko, Approximation by superpositions of a sigmoidal function,Mathematics of Control, Signals and Systems, 2 (1989), pp. 303–314.
[16] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn,X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer,G. Heigold, S. Gelly, et al., An image is worth 16x16 words: Transformersfor image recognition at scale, in International Conference onLearning Representations, 2020.
[17] J. Duchi, E. Hazan, and Y. Singer, Adaptive subgradient methods foronline learning and stochastic optimization., Journal of Machine LearningResearch, 12 (2011), pp. 2121–2159.
[18] S. Geman and D. Geman, Stochastic relaxation, gibbs distributions,and the bayesian restoration of images, IEEE Transactions on PatternAnalysis and Machine Intelligence, 6 (1984), pp. 721–741.
[19] R. A. Gingold and J. J. Monaghan, Smoothed particle hydrodynamics:theory and application to non-spherical stars, Monthly Notices of theRoyal Astronomical Society, 181 (1977), pp. 375–389.
[20] I. Goodfellow, Y. Bengio, A. Courville, and Y. Bengio, Deeplearning, MIT Press Cambridge, 2016.
[21] L. Gorelick, O. Vekslar, Y. Boykov, and C. Nieuwenhuis, Convexityshape prior for binary segmentation, IEEE Transactions on PatternAnalysis and Machine Intelligence, 39 (2017), pp. 258–270.
[22] L. Gorelick and O. Veksler, Multi-object convexity shape prior forsegmentation, in International Workshop on Energy Minimization Methodsin Computer Vision and Pattern Recognition, Springer, 2017, pp. 455–468.
[23] L. Gorelick, O. Veksler, Y. Boykov, and C. Nieuwenhuis, Convexityshape prior for segmentation, in European Conference on ComputerVision, Springer, 2014, pp. 675–690.
[24] J. Han, A. Jentzen, and E. Weinan, Solving high-dimensional partialdifferential equations using deep learning, Proceedings of the NationalAcademy of Sciences, 115 (2018), pp. 8505–8510.
[25] A. Harsh, J. E. Ball, and P. Wei, Onion-peeling outlier detectionin 2-d data sets, arXiv Preprint arXiv:1803.04964, (2018).
[26] K. He, X. Zhang, S. Ren, and J. Sun, Deep residual learning forimage recognition, in Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition, 2016, pp. 770–778.
[27] C. F. Higham and D. J. Higham, Deep learning: An introduction forapplied mathematicians, SIAM Review, 61 (2019), pp. 860–891.
[28] N. J. Higham, Matrix Nearness Problems and Applications, Citeseer,1988.
[29] G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, andR. R. Salakhutdinov, Improving neural networks by preventing coadaptationof feature detectors, arXiv preprint arXiv:1207.0580, (2012).
[30] S. Hochreiter and J. Schmidhuber, Long short-term memory, NeuralComputation, 9 (1997), pp. 1735–1780.
[31] K. Hornik, M. Stinchcombe, H. White, et al., Multilayer feedforwardnetworks are universal approximators., Neural Networks, 2 (1989),pp. 359–366.
[32] H. Isack, O. Veksler, M. Sonka, and Y. Boykov, Hedgehog shapepriors for multi-object segmentation, in IEEE Conference on ComputerVision and Pattern Recognition, 2016, pp. 2434–2442.
[33] F. Isensee, P. Kickingereder, W. Wick, M. Bendszus, andK. H. Maier-Hein, Brain tumor segmentation and radiomics survivalprediction: Contribution to the brats 2017 challenge, in InternationalMICCAI Brainlesion Workshop, Springer, 2017, pp. 287–297.
[34] T. Karras, S. Laine, and T. Aila, A style-based generator architecturefor generative adversarial networks, in Proceedings of the IEEE/CVFConference on Computer Vision and Pattern Recognition, 2019, pp. 4401–4410.
[35] D. P. Kingma and J. Ba, Adam: A method for stochastic optimization,in 3rd International Conference on Learning Representations, ICLR2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings,Y. Bengio and Y. LeCun, eds., 2015.
[36] A. Krizhevsky, I. Sutskever, and G. E. Hinton, Imagenet classificationwith deep convolutional neural networks, in Advances in NeuralInformation Processing Systems, vol. 25, 2012.
[37] Y. LeCun, B. Boser, J. Denker, D. Henderson, R. Howard,W. Hubbard, and L. Jackel, Handwritten digit recognition with aback-propagation network, Advances in Neural Information ProcessingSystems, 2 (1989).
[38] L. Li, S. Luo, X.-C. Tai, and J. Yang, Convex hull algorithms basedon some variational models, arXiv preprint arXiv:1908.03323, (2019).
[39] L. Li, S. Luo, X.-C. Tai, and J. Yang, A level set representationmethod for n-dimensional convex shape and applications, Communicationsin Mathematical Research, 37 (2021), pp. 180–208.
[40] L. Li, X.-C. Tai, and J. Yang, Generalization error analysisof neural networks with gradient based regularization, arXiv preprintarXiv:2107.02797, (2021).
[41] L. Li, X.-C. Tai, J. Yang, and Q. Zhu, Priori error analysisof deep mixed residual method for elliptic pdes, Preprint:http://dx.doi.org/10.13140/RG.2.2.21143.65446, (2022).
[42] Z. Liu, Multi-scale deep neural network (mscalednn) for solving poissonboltzmannequation in complex domains, Communications in ComputationalPhysics, 28 (2020), pp. 1970–2001.
[43] Z. Liu, D. W. Jacobs, and R. Basri, The role of convexity in perceptualcompletion: Beyond good continuation., Vision Research, 39 (1999),pp. 4244–4257.
[44] J. Lu, Y. Lu, and M. Wang, A priori generalization analysis of thedeep ritz method for solving high dimensional elliptic equations, arXivpreprint arXiv:2101.01708, (2021).
[45] S. Luo and X.-C. Tai, Convex shape priors for level set representation,arXiv preprint arXiv:1811.04715, (2018).
[46] S. Luo, X.-C. Tai, L. Huo, Y. Wang, and R. Glowinski, Convexshape prior for multi-object segmentation using a single level set function,in Proceedings of the IEEE International Conference on Computer Vision,2019, pp. 613–621.
[47] L. Lyu, K. Wu, R. Du, and J. Chen, Enforcing exact boundaryand initial conditions in the deep mixed residual method, arXiv preprintarXiv:2008.01491, (2020).
[48] L. Lyu, Z. Zhang, M. Chen, and J. Chen, Mim: A deep mixedresidual method for solving high-order partial differential equations, arXivpreprint arXiv:2006.04146, (2020).
[49] C. Ma, L. Wu, et al., Barron spaces and the compositional functionspaces for neural network models, arXiv preprint arXiv:1906.08039,(2019).
[50] Y. L. Ming et al., Deep nitsche method: Deep ritz method with essentialboundary conditions, Communications in Computational Physics, 29(2021), pp. 1365–1384.
[51] M. Minsky and S. Papert, Perceptrons., MIT press, 1969.
[52] S. Mishra and R. Molinaro, Estimates on the generalization error ofphysics informed neural networks (pinns) for approximating pdes, SAMResearch Report, 2020 (2020).
[53] , Estimates on the generalization error of physics informed neuralnetworks (pinns) for approximating pdes ii: A class of inverse problems,SAM Research Report, 2020 (2020).
[54] D. Mumford and J. Shah, Optimal approximations by piecewisesmooth functions and associated variational problems, Communicationson Pure and Applied Mathematics, 42 (1989), pp. 577–685.
[55] J. M¨uller and M. Zeinhofer, Error estimates for the variationaltraining of neural networks with boundary penalty, arXiv preprintarXiv:2103.01007, (2021).
[56] S. Osher and R. Fedkiw, Level Set Methods and Dynamic ImplicitSurfaces, vol. 153, Springer Science & Business Media, 2002.
[57] S. Osher and R. Fedkiw, Level set methods and dynamic implicitsurfaces, vol. 153 of Applied Mathematical Sciences, Springer-Verlag, NewYork, 2003.
[58] R. B. Potts, Some generalized order-disorder transformations, MathematicalProceedings of the Cambridge Philosophical Society, 48 (1952),pp. 106–109.
[59] M. Raissi, P. Perdikaris, and G. E. Karniadakis, Physicsinformedneural networks: A deep learning framework for solving forwardand inverse problems involving nonlinear partial differential equations,Journal of Computational Physics, 378 (2019), pp. 686–707.
[60] O. Ronneberger, P. Fischer, and T. Brox, U-net: Convolutionalnetworks for biomedical image segmentation, in International Conferenceon Medical image computing and computer-assisted intervention,Springer, 2015, pp. 234–241.
[61] F. Rosenblatt, The perceptron: a probabilistic model for informationstorage and organization in the brain., Psychological Review, 65 (1958),p. 386.
[62] G. Rosman, Y. Wang, X.-C. Tai, R. Kimmel, and A. M. Bruckstein,Fast regularization of matrix-valued images, in Efficient Algorithmsfor Global Optimization Methods in Computer Vision, Springer, 2014,pp. 19–43.
[63] L. A. Royer, D. L. Richmond, C. Rother, B. Andres, andD. Kainmueller, Convexity shape constraints for image segmentation,in Proceedings of the IEEE Conference on Computer Vision and PatternRecognition, 2016, pp. 402–410.
[64] L. I. Rudin, S. Osher, and E. Fatemi, Nonlinear total variation basednoise removal algorithms, Physica D: Nonlinear Phenomena, 60 (1992),pp. 259–268.
[65] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, Learningrepresentations by back-propagating errors, Nature, 323 (1986), pp. 533–536.
[66] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma,Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg,and L. Fei-Fei, ImageNet Large Scale Visual Recognition Challenge,International Journal of Computer Vision, 115 (2015), pp. 211–252.
[67] J. W. Siegel and J. Xu, Approximation rates for neural networks withgeneral activation functions, Neural Networks, 128 (2020), pp. 313–321.
[68] K. Simonyan and A. Zisserman, Very deep convolutional networks forlarge-scale image recognition, arXiv preprint arXiv:1409.1556, (2014).
[69] J. Sirignano and K. Spiliopoulos, Dgm: A deep learning algorithmfor solving partial differential equations, Journal of ComputationalPhysics, 375 (2018), pp. 1339–1364.
[70] E. Strekalovskiy and D. Cremers, Generalized ordering constraintsfor multilabel optimization, in 2011 International Conference on ComputerVision, IEEE, 2011, pp. 2619–2626.
[71] M. Sussman, P. Smereka, and S. Osher, A level set approach forcomputing solutions to incompressible two-phase flow, Journal of ComputationalPhysics, 114 (1994), pp. 146–159.
[72] E. Ukwatta, J. Yuan, W. Qiu, M. Rajchl, and A. Fenster,Efficient convex optimization-based curvature dependent contour evolutionapproach for medical image segmentation, in Medical Imaging 2013: ImageProcessing, vol. 8669, 2013, pp. 866–902.
[73] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones,A. N. Gomez, L. Kaiser, and I. Polosukhin, Attention is all youneed, in Advances in Neural Information Processing Systems, vol. 30,2017.
[74] O. Veksler, Star shape prior for graph-cut image segmentation, in EuropeanConference on Computer Vision, Springer, 2008, pp. 454–467.
[75] L. A. Vese and T. F. Chan, A multiphase level set framework forimage segmentation using the Mumford and Shah model, InternationalJournal of Computer Vision, 50 (2002), pp. 271–293.
[76] S. Vicente, V. Kolmogorov, and C. Rother, Graph cut basedimage segmentation with connectivity priors, in 2008 IEEE Conference onComputer Vision and Pattern Recognition, IEEE, 2008, pp. 1–8.
[77] B. Wang, Multi-scale deep neural network (mscalednn) methods for oscillatorystokes flows in complex domains, Communications in ComputationalPhysics, 28 (2020), pp. 2139–2157.
[78] E. Weinan, Machine learning and computational mathematics, Communicationsin Computational Physics, 28 (2020), pp. 1639–1670.
[79] E. Weinan and B. Yu, The deep Ritz method: A deep learning-basednumerical algorithm for solving variational problems, Communications inMathematics and Statistics, 6 (2018), pp. 1–12.
[80] S. Yan, X.-C. Tai, J. Liu, and H.-Y. Huang, Convexity shapeprior for level set based image segmentation method, arXiv PreprintarXiv:1805.08676, (2018).
[81] C. Yang, X. Shi, D. Yao, and C. Li, A level set method for convexitypreserving segmentation of cardiac left ventricle, in InternationalConference on Image Processing, 2017, pp. 2159–2163.
[82] J. Yuan, W. Qiu, E. Ukwatta, M. Rajchl, Y. Sun, and A. Fenster,An efficient convex optimization approach to 3d prostate mri segmentationwith generic star shape prior, Prostate MR Image SegmentationChallenge, MICCAI, 7512 (2012), pp. 82–89.
[83] D.-X. Zhou, Universality of deep convolutional neural networks, Appliedand Computational Harmonic Analysis, 48 (2020), pp. 787–794.
[84] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, Unpaired imageto-image translation using cycle-consistent adversarial networks, in Pro-ceedings of the IEEE International Conference on Computer Vision, 2017,pp. 2223–2232.
[85] Q. Zhu and J. Yang, A local deep learning method for solving high orderpartial differential equations, arXiv preprint arXiv:2103.08915, (2021).