Neural architecture search (NAS)

• A problem with NNs is how to set the architecture and parameters.
• It is usually based the user's experience or on trial and error.
• Automatically setting architecture and parameters is notoriously difficult.
• This is a very challenging problem and active research topic.

NAS approaches

• Reinforcement learning
• Heuristic optimization algorithms
• Neuroevolution

Characteristics

• Neuroevolution investigates methods for automatic selection of neural network architectures and parameters.
• The neural architecture search is transformed into an optimization problem.
• Evolutionary algorithms are used to find the solution of the optimization problem.
• It is a very flexible approach that requires minimal human intervention.
• However, the methods may require a large number of evaluations.

Characteristics

• Each solution is represented using an individual or chromosome.
• A population of solutions is generated.
• Solutions are evaluated according to a fitness function .
• The solutions with the best fitness values are selected.
• New solutions are generated by applying crossover and mutation operators on the selected solutions.
• The population of individuals is evolved for a number of generations until a given stop criterion is satisfied.

How to do it?

1. Select a representation that encodes the NN in an appropriate way.
2. Design the crossover operator in such a way that combines components of the two (parent) neural networks.
3. Design a mutation operator that introduces potentially innovative variations in the generated solutions.
4. Use a fitness function that estimates the quality of the NN.
5. Decide on the rest of the EA components.

Characteristics

• NEAT is a neuroevolutionary algorithm.
• Combines the search for appropriate network weights with the complexification of the network structure.
• To avoid the creation of over-complex NNs, NEAT starts from a minimal structure and add nodes and connections incrementally.
• To avoid newly introduced NN structural changes to disappear from the population prematuraturely, it separates chromosomes in niches.

Characteristics

• SNNs are formed by neuron models that communicate by sequences of spikes.
• The spiking dynamics of the neurons are described by differential equations
• Neurons are connected by artificial sypnases.
• They are powerful tools for analysis of elementary processes in the brain.
• Used for fast signal-processing, event detection, classification, speech recognition, or motor control.
• Computationally more powerful than perceptrons and sigmoidal gates.

• The presence and timing of individual neurons is considered as the means of comunication and neural computation .
• It has \(n\) input neurons whose firing times are determined through some external mechanism
• Different types of neuron models and information processing strategies can be used.
• Learning is usually harder than in MLP perceptrons and other non-spiking neural models.

• The problem of neural code: How is information encoded in the neural signal?
• Two (not necessarily contradictory) explanations:
  1. Spike rate code: Neural information is encoded in the firing rate (number of spikes in a given time period).
  2. Spike timing code: Neural information encoded in the precise timing of spikes.
• Spiking neural networks can be designed to exploit different ways of coding information in the neuron spikes.

• The information is encoded in the latency between the beginning of the stimulus and the time of the first spike.
• Enables ultrafast information processing.

• Information is in the exact timing of a set of spikes relative to each other.
• Precise relative spike timing is one of the critical parameters that control many forms of synaptic plasticity
• Very efficient in terms of information capacity.

• The data-flow from input to output units is strictly one-directional.
• The data processing can extend over multiple layers of neurons.
• No feedback connections are present.
• Ussually applied to model low-level sensory systems (e.g. vision, olfaction, tactile sensing).