PROJECT TITLE :

Analysis on the Number of Linear Regions of Piecewise Linear Neural Networks

ABSTRACT:

Deep Neural Networks, also known as DNNs, have been shown to be excellent solutions to some of the most complex and mind-boggling issues in Machine Learning. The robust expressive power of function representation is one of the primary factors contributing to their level of success. The number of linear regions in piecewise linear neural networks (PLNNs) is a natural measure of their expressive power because it characterizes the number of linear pieces that are available to model complex patterns. This makes the number of linear regions a useful metric. Counting and limiting the total number of linear regions is the method that we use in this piece to perform a theoretical analysis of the expressive power of PLNNs. We begin by revising the upper and lower bounds that have previously been established for the number of linear regions of PLNNs that contain rectified linear units (ReLU PLNNs). Next, we will derive the exact maximum number of linear regions that can be found in single-layer PLNNs by extending the analysis to PLNNs with general piecewise linear (PWL) activation functions. In addition, the upper and lower bounds on the number of linear regions of multilayer PLNNs have been obtained. Both of these bounds scale polynomially with the number of neurons at each layer and pieces of PWL activation function, but exponentially with the number of layers in the network. Because of this crucial property, deep PLNNs with complex activation functions are able to outperform their shallow counterparts when computing highly complex and structured functions. This helps to explain, to some extent, why deep PLNNs perform better than shallow counterparts when it comes to classification and function fitting.