Hello. Welcome to the Coursera course on Neural Networks for Machine Learning. Before we get into the details of neural network learning algorithms, I want to talk a little bit about machine learning, why we need machine learning, the kinds of things we use it for, and show you some examples of what it can do. So the reason we need machine learning is that the sum problem, where it's very hard to write the programs, recognizing a three dimensional object for example. When it's from a novel viewpoint and new lighting additions in a cluttered scene is very hard to do. We don't know what program to write because we don't know how it's done in our brain. And even if we did know what program to write, it might be that it was a horrendously complicated program.

Another example is, detecting a fraudulent credit card transaction, where there may not be any nice, simple rules that will tell you it's fraudulent. You really need to combine, a very large number of, not very reliable rules. And also, those rules change every time because people change the tricks they use for fraud. So, we need a complicated program that combines unreliable rules, and that we can change easily.

The machine learning approach, is to say, instead of writing each program by hand for each specific task, for particular task, we collect a lot of examples, and specify the correct output for given input. A machine learning algorithm then takes these examples and produces a program that does the job. The program produced by the linear algorithm may look very different from the typical handwritten program. For example, it might contain millions of numbers about how you weight different kinds of evidence. If we do it right, the program should work for new cases just as well as the ones it's trained on. And if the data changes, we should be able to change the program runs very easily by retraining it on the new data. And now massive amounts for computation are cheaper that paying someone to write a program for a specific task, so we can afford big complicated machine learning programs to produce these stark task specific systems for us. Some examples of the things that are best done by using a learning algorithm are recognizing patterns, so for example objects in real scenes, or the identities or expressions of people's faces, or spoken words. There's also recognizing anomalies. So, an unusual sequence of credit card transactions would be an anomaly. Another example of an anomaly would be an unusual pattern of sensor readings in a nuclear power plant. And you wouldn't really want to have to deal with those by doing supervised learning. Where you look at the ones that blow up, and see what, what caused them to blow up. You'd really like to recognize that something funny is happening without having any supervision signal. It's just not behaving in its normal way. And then this prediction. So, typically, predicting future stock prices or currency exchange rates or predicting which movies a person will like from knowing which other movies they like. And which movies a lot of other people liked. So in this course I'm mean as a standard example for explaining a lot of the machine learning algorithms. This is done in a lot of science. In genetics for example, a lot of genetics is done on fruitflies. And the reason is they're convenient. They breed fast and a lot is already known about the genetics of fruit flies. The MNIST database of handwritten digits is the machine equivalent of fruitflies. It's publicly available. We can get machine learning algorithms to learn how to recognize these handwritten digits quite quickly, so it's easy to try lots of variations. And we know huge amounts about how well different machine learning methods do on MNIST. And in particular, the different machine learning methods were implemented by people who believed in them, so we can rely on those results. So for all those reasons, we're gonna use MNIST as our standard task. Here's an example of some of the digits in MNIST. These are ones that were correctly recognized by neural net the first time it saw them. But the ones within the neural net wasn't very confident. And you could see why. I've arranged these digits in standard scan line order. So zeros, then ones, then twos and so on. If you look at a bunch of tubes like the onces in the green rectangle. You can see that if you knew they were 100 in digit you'd probably guess they were twos. But it's very hard to say what it is that makes them twos. Theres nothing simple that they all have in common. In particular if you try and overlay one on another you'll see it doesn't fit. And even if you skew it a bit, it's very hard to make them overlay on each other. So a template isn't going to do the job. An in particular template is going to be very hard to find that will fit those twos in the green box and would also fit the things in the red boxes. So that's one thing that makes recognizing handwritten digits a good task for machine learning. Now, I don't want you to think that's the only thing we can do. It's a relatively simple for our machine learning system to do now. And to motivate the rest of the course, I want to show you some examples of much more difficult things. So we now have neural nets with approaching a hundred million parameters in them, that can recognize a thousand different object classes in 1.3 million high resolution training images got from the web. So, there was a competition in 2010, and the best system got 47 percent error rate if you look at its first choice, and 25 percent error rate if you say it got it right if it was in its top five choices, which isn't bad for 1,000 different objects. Jitendra Malik who's an eminent neural net skeptic, and a leading computer vision researcher, has said that this competition is a good test of whether deep neural networks can work well for object recognition. And a very deep neural network can now do considerably better than the thing that won the competition. It can get less than 40 percent error, for its first choice, and less than twenty percent error for its top five choices. I'll describe that in much more detail in lecture five. Here's some examples of the kinds of images you have to recognize. These images from the test set that he's never seen before. And below the examples, I'm showing you what the neural net thought the right answer was. Where the length of the horizontal bar is how confident it was, and the correct answer is in red. So if you look in the middle, it correctly identified that as a snow plow. But you can see that its other choices are fairly sensible. It does look a little bit like a drilling platform. And if you look at its third choice, a lifeboat, it actually looks very like a lifeboat. You can see the flag on the front of the boat and the bridge of the boat and the flag at the back, and the high surf in the background. So its, its errors tell you a lot about how it's doing it and they're very plausible errors. If you look on the left, it gets it wrong possibly because the beak of the bird is missing and cuz the feathers of the bird look very like the wet fur of an otter. But it gets it in its top five, and it does better than me. I wouldn't know if that was a quail or a ruffed grouse or a partridge. If you look on the right, it gets it completely wrong. It a guillotine, you can why it says that. You can possibly see why it says orangutan, because of the sort of jungle looking background and something orange in the middle. But it fails to get the right answer. It can, however, deal with a wide range of different objects. If you look on the left, I would have said microwave as my first answer. The labels aren't very systematic. So actually, the correct answer there is electric range. And it does get it in its top five. In the middle, it's getting a turnstile, which is a distributed object. It does, can't, it can do more than just recognize compact things. And it can also deal with pictures, as well as real scenes, like the bulletproof vest. And it makes some very cool errors. If you look at the image on the left, that's an earphone. It doesn't get anything, like an earphone. But if you look at this fourth batch, it thinks it's an ant. And for you to think that's crazy. But then if you look at it carefully, you can see it's a view of an ant from underneath. The eyes are looking down at you, and you can see the antennae behind it. It's not the kind of view of an ant you'd like to have if you were a green fly. If you look at the one on the right, it doesn't get the right answer. But all of its answers are, cylindrical objects. Another task that neural nets are now very good at, is speech recognition. Or at least part of a speech recognition system. So speech recognition systems have several stages. First they pre-process the sound wave, to get a vector of acoustic coefficients, for each ten milliseconds of sound wave. And so they get 100 of those actors per second. They then take a few adjacent vectors of acoustic coefficients, and they need to place bets on which part of which phoneme is being spoken. So they look at this little window and they say, in the middle of this window, what do I think the phoneme is, and which part of the phoneme is it? And a good speech recognition system will have many alternative models for a phoneme. And each model, it might have three different parts. So it might have many thousands of alternative fragments that it thinks this might be. And you have to place bets on all those thousands of alternatives. And then once you place those bets you have a decoding stage that does the best job it can of using plausible bets, but piecing them together into a sequence of bets that corresponds to the kinds of things that people say. Currently, deep neural networks pioneered by George Dahl and Abdel-rahman Mohammed of the University of Toronto are doing better than previous machine learning methods for the acoustic model, and they're now beginning to be used in practical systems. So, Dahl and Mohammed, developed a system, that uses many layers of, binary neurons, to, take some acoustic frames, and make bets about the labels. They were doing it on a fairly small database and then used 183 alternative labels. And to get their system to work well, they did some pre-training, which will be described in the second half of the course. After standard post processing, they got 20.7 percent error rate on a very standard benchmark, which is kind of like the NMIST for speech. The best previous result on that benchmark for speak independent recognition was 24.4%. And a very experienced speech researcher at Microsoft research realized that, that was a big enough improvement, that probably this would change the way speech recognition systems were done. And indeed, it has. So, if you look at recent results from several different leading speech groups, Microsoft showed that this kind of deep neural network, when used as the acoustic model in the speech system. Reduced the error rate from 27.4 percent to 18.5%, or alternatively, you could view it as reducing the amount of training data you needed from 2,000 hours down to 309 hours to get comparable performance. Ibm which has the best system for one of the standard speech recognition tasks for large recovery speech recognition, showed that even it's very highly tuned system that was getting 18.8 percent can be beaten by one of these deep neural networks. And Google, fairly recently, trained a deep neural network on a large amount of speech, 5,800 hours. That was still much less than they trained their mixture model on. But even with much less data, it did a lot better than the technology they had before. So it reduced the error rate from sixteen percent to 12.3 percent and the error rate is still falling. And in the latest Android, if you do voice search, it's using one of these deep neurall networks in order to do very good speech recognition.