"Skip the model, just output random garbage directly" subtitle: Your boss hasn’t read Vapnik and doesn’t know the difference between approximating the system and approximating the system’s output, and neither should you

Vapnik, statistical learning theory, page 21: two goals to pursue: • To imitate the supervisor's operator: Try to construct an operator which provides for a given generator G, the best prediction to the supervisor's outputs. • To identify the supervisor's operator: Try to construct an operator which is close to the supervisor's operator. There exists an essential difference in these two goals. In the first case. the goal is to achieve the best results in prediction of the supervisor's outputs for the environment given by the generator G. In the second case, to get good results in prediction is not enough; it is required to construct an operator which is close to the supervisor's one in a given metric. These two goals of the learning machine imply two different approaches to the learning problem. In this book we consider both approaches. We show that the problem of imitation of the target operator is easier to solve. For this problem, a nonasymptotic theory will be developed. The problem of identification is more difficult. It refers to the so-called ill-posed problems. For these problems, only an asymptotic theory can be developed. Nevertheless, we show that the solutions for both problems are based on the same general principles.

[vapnik page 48] 1.13 THE STRUCTURE OF THE LEARNING THEORY Thus. in this chapter we have considered two approaches to learning problems. The first approach (imitating the supervisor's operator) brought us to the problem of minimizing a risk functional on the basis of empirical data. The second approach (identifying the supervisor's operator) brought us to the problem of solving some integral equation when the elements of an equation arc known only approximately. It has been shown that the second approach gives more details on the solution of pattern recognition and regression estimation problems. Why in this case do we need both approaches? As we mentioned in the last section. the second approach. which is based on the solution of the in~ tegral equation. forms an ill-posed problem. For ill-posed problems. the best that can be done is to obtain a sequence of approximations to the solution which converges in probability to the desired function when the number of observations tends to infinity. For this approach, there exists no way to evaluate how well the problem can be solved if a finite number of observations is used. In the framework of this approach to the learning problem. any exact assertion is asymptotic.