President Trump loves Twitter. Its a direct streamofconsciousness rant about whatever pops into his mind or onto cable TV at any given second. But here at. InformationWeek. News, analysis and research for business technology professionals, plus peertopeer knowledge sharing. Engage with our community. Hot on the heels of last weeks study on the frightening prevalence of traumatic brain injury in footballand similar dangers that may lurk for players on the. License Plate Recognition Source Code Compare' title='License Plate Recognition Source Code Compare' />Neural networks and deep learning. When a golf player is first learning to play golf, they usually spend. Only gradually do they. In a similar way, up to. Its our basic swing, the foundation for learning in most work on. In this chapter I explain a suite of techniques. The techniques well develop in this chapter include a better choice. L1 and L2 regularization, dropout, and artificial. Ill also overview several other. The discussions are largely independent. Well also. many of the techniques in running code, and use them to improve the. Chapter 1. Of course, were only covering a few of the many, many techniques. The philosophy is. Mastering those. important techniques is not just useful in its own right, but will. That will leave you well prepared to quickly pick up. Most of us find it unpleasant to be wrong. Soon after beginning to. I gave my first performance before an audience. I was. nervous, and began playing the piece an octave too low. I got. confused, and couldnt continue until someone pointed out my error. I. was very embarrassed. Yet while unpleasant, we also learn quickly when. You can bet that the next time I played. I played in the correct octave By contrast, we. Ideally, we hope and expect that our neural networks will learn fast. Is this what happens in practice To answer this. The example involves a neuron. Well train this neuron to do something ridiculously easy take the. Book Of Love You Make Me Feel So Good Speed. Of course, this is such a trivial task. However, it turns out to be. So lets take a look at how the neuron learns. To make things definite, Ill pick the initial weight to be 0. These are generic choices used as a. I wasnt picking them to be special in any. The initial output from the neuron is 0. Click on Run in the bottom right corner below to. Note that. this isnt a pre recorded animation, your browser is actually. The learning rate is eta. The cost is the quadratic cost function, C. Chapter 1. Ill remind you of the exact form of. Note that you can run the animation multiple times by. Run again. As you can see, the neuron rapidly learns a weight and bias that. Thats not quite the desired output, 0. Suppose, however, that we instead choose both the starting. In this case the initial. Lets look at how the. Click on Run again Although this example uses the same learning rate eta 0. Indeed, for the. first 1. Then the learning kicks in and, much as in our first. This behaviour is strange when contrasted to human learning. As I. said at the beginning of this section, we often learn fastest when. But weve just seen that our. Whats more, it turns out that this behaviour occurs not just in this. Why is learning so slow And can we find a way of avoiding this slowdown To understand the origin of the problem, consider that our neuron. Cpartial w and. C partial b. So saying learning is slow is really. The. challenge is to understand why they are small. To understand that. Recall that were using the. Equation 6begineqnarray Cw,b equiv. C fracy a22. To write this. Using the chain rule to differentiate. Cpartial w a ysigmaz x a sigmaz tag5. Cpartial b a ysigmaz a sigmaz. I have substituted x 1 and y 0. To understand the. Recall the shape of the. We can see from this graph that when the neurons output is close to. Equations 5. 5begineqnarray. Cpartial w a ysigmaz x a sigmaz nonumberendeqnarray and 5. Cpartial b a ysigmaz a sigmaz nonumberendeqnarray then tell us that. C partial w and partial C partial b get very. This is the origin of the learning slowdown. Whats more, as. we shall see a little later, the learning slowdown occurs for. How can we address the learning slowdown It turns out that we can. To understand the. Well suppose instead that were trying to train a neuron with. The output from the neuron is, of course, a sigmaz, where z. We define the. cross entropy cost function for this neuron by. C frac1n sumx lefty ln a 1 y ln 1 a right. Its not obvious that the expression 5. C frac1n sumx lefty ln a 1 y ln 1 a right nonumberendeqnarray. In fact, frankly, its not even. Before. addressing the learning slowdown, lets see in what sense the. Two properties in particular make it reasonable to interpret the. First, its non negative, that is. C 0. To see this, notice that a all the individual terms in. C frac1n sumx lefty ln a 1 y ln 1 a right nonumberendeqnarray are negative, since both. Second, if the neurons actual output is close to the desired output. To prove this I will need to assume that the desired. This is usually the case. Boolean functions. To understand what happens when we dont make. To. see this, suppose for example that y 0 and a approx 0 for some. This is a case when the neuron is doing a good job on that. We see that the first term in the. C frac1n sumx lefty ln a 1 y ln 1 a right nonumberendeqnarray for the cost vanishes, since. A. similar analysis holds when y 1 and a approx 1. And so the. contribution to the cost will be low provided the actual output is. Summing up, the cross entropy is positive, and tends toward zero as. These are both properties wed intuitively. Indeed, both properties are also. So thats good news for the. But the cross entropy cost function has the benefit. To see this, lets compute the partial derivative of. We substitute a. C frac1n sumx lefty ln a 1 y ln 1 a right nonumberendeqnarray, and apply the chain. Cpartial wj frac1n sumx left. Putting everything over a common denominator and simplifying this. Cpartial wj frac1n. Using the definition of the sigmoid function, sigmaz. Ill ask you to verify this in an exercise. We see that the. sigmaz and sigmaz1 sigmaz terms cancel in the equation. Cpartial wj frac1n sumx xjsigmaz y. This is a beautiful expression. It tells us that the rate at which. The larger the error, the faster the neuron will. This is just what wed intuitively expect. In particular, it. Equation 5. 5begineqnarray. Cpartial w a ysigmaz x a sigmaz nonumberendeqnarray. When we use the cross entropy, the sigmaz term gets canceled. This. cancellation is the special miracle ensured by the cross entropy cost. Actually, its not really a miracle. As well see later. In a similar way, we can compute the partial derivative for the bias. I wont go through all the details again, but you can easily verify. Cpartial b frac1n sumx sigmaz y. Again, this avoids the learning slowdown caused by the sigmaz. Equation 5. 6begineqnarray. Cpartial b a ysigmaz a sigmaz nonumberendeqnarray. Verify that sigmaz sigmaz1 sigmaz. Lets return to the toy example we played with earlier, and explore. To re orient ourselves, well begin with the case where the. Press Run to see what happens when we replace the. Unsurprisingly, the neuron learns perfectly well in this instance. And now lets look at the case where our. Success This time the neuron learned quickly, just as we hoped. If. you observe closely you can see that the slope of the cost curve was. Its that steepness which. I didnt say what learning rate was used in the examples just. Earlier, with the quadratic cost, we used eta 0. Should we have used the same learning rate in the new examplesIn. For both cost functions I simply. If youre still curious, despite my disavowal, heres. I used eta 0. Meter Source Book by Federal Buyers Guide, inc. Published on Aug 3, 2. This comprehensive meter source book provides you with everything you need to know about products, services, and suppliers for meter equipme.