Deep learning and neural networks

Cnn

Machine learning vs deep learning

Instead of writing a program that outputs a features space for the classifier, it’s better to learn also the feature creation algorithm, that is what deep learning is about

flowchart TD subgraph machine_learning direction LR A[Pixels] B[Program] C[learn] D[Output] A --> B --Features--> C --> D end subgraph deep_learning direction LR E[Pixels] F[learn] G[learn] H[Output] E --> F --Features--> G --> H end machine_learning ~~~~ deep_learning

The learning phase needs to introduce non-linear computation in the chain in order to produce usefull features for the classifier.

So the feature learn layer can be realized as a set of non linear functions called activation functions that take as input a linear combination of the input of the layer

flowchart TD subgraph neural_network direction LR A(input pixels) B[l2] C[....] D[l3] E(Output) A --> B --> C --> D --> E end

each layer contains $D$ nodes ($D$ is an hyperparameter) Where the function of the $l_i$ layer can be defined as:

$$ L_i= \phi(Wx+b) $$ where the $\phi()$ is the activation function and $W$ is a weight vector of size $D \times imageSize$

Activation function

non linear functions used in the layers to perform the features computation examples are:

with equations: $$ \begin{align} \phi(a) = \frac{1}{1 + exp(-a)}&& \phi(a)=\max(a,0)\ \end{align} $$

Fully connected layers

Networks where the input of the $L_i$ layer is the output of all the nodes of the $L_{i-1}$ layer

the output of the $y_i$ node can be described as :

$$ y_i = \phi(\sum_{j=1}^{D}{W_{ij}x_j +b_i}) $$

previous next