Project

I want to write a library that implements simple artificial neural network in C++. In this post I would like to discuss only a part of my project, namely connection between neurons: synapses.

The idea is to create an interface to implement ANN's. I decided to make neurons - elementary units of ANN, to be a substantive blocks from which one builds ANN. I think this approach provides big flexibility, though something becomes more complex or cumbersome.

For more information please visit this git repo. There you will find the list of definitions of the concepts I use below and test example that (I hope) will help understand what I have reached so far and what I want it to be.

Implementation

Communication between neurons. Description.

This section contains a description of how neurons communicate with each other. The idea is to have a mediator that performs such kind of communication which are Axon and Dendrite classes, so neurons do not know about each other explicitely. Axon and Dendrite classes are responsible for connecting of neurons. Also Axon class performs output operations for a neuron, Dendrite performs input operations for a neuron.

Connection is carried out by the following steps:

1. 
   Neuron neuronSender creates an Axon instance axon
   


2. 
   axon requests connection with Neuron neuronReceiver
   


3. If neuronReceiver agrees to connect then it asks its dendtite to register neuronSender as a sender, after that axon registers neuronReceiver as a receiver, assuming that neuronReceiver has Dendrite instance dendrite
   

Once connection is set communication (or signal propagation) is as follows

1. 
   Neuron neuronSender emits signal. That means that neuronSender tells to its Axon axon to transmit signal to all receivers that are connected to it


2. 
   axon in turn tries to update Neuron neuronReceiver with signal value from neuronSender
   


3. 
   neuronReceiver in turn asks its Dendrite dendrite to transmit weighted signal from neuronSender. dendrite checks if neuronSender in the list of senders and if so returns weighted signal and sets status of neuronSender as being received


4. Then neuronReceiver asks its dendrite if it is complete which means that signals from all senders have been received and if so emits itself

So signal propagation is kind of recursive process.

Communication between neurons. Code.

All stuff is placed in namespace sann.

Auxiliary objects

sann.h

#ifndef SANN_H
#define SANN_H

#include <iostream>
#include <iomanip>
#include <string>
#include <vector>

namespace sann

 enum NeuronType INPUT, HIDDEN, OUTPUT ;

 inline void sannwrn( std::string message )
 
 std::cout << "WARNING: " << message << "n";
 

#endif

Neuron

Neuron.h

#ifndef NEURON_H
#define NEURON_H

#include "sann.h"

namespace sann

 class Axon;
 class Dendrite;

 class Neuron
 
 private :
 Axon* _axon;//stores pointers to receivers
 Dendrite* _dendrite;//stores pointers to senders

 protected :
 std::string _name;
 NeuronType _type;//input, hidden or output

 int _value;

 public :
 Neuron( std::string name, NeuronType type, int value = 0 ) :
 _axon( nullptr ),
 _dendrite( nullptr ),
 _name( name ),
 _type( type ),
 _value( value )
 


 /*Ask _dendrite to update current value with a weighted
 *signal from a sender (fromWhich)*/
 void Update( int signal, const Neuron* fromWhich );
 Axon* CreateAxon();
 /*Decide to accept or not a synapse with a sender*/
 bool AcceptSynapse( Neuron* sender );
 /*Ask _axon to send signal to all receivers*/
 void Emit() const;

 inline std::string GetName() const return _name; 
 inline NeuronType GetType() const return _type; 
 inline int GetValue() const return _value; 
 size_t GetDendriteSize() const;
 size_t GetAxonSize() const;

 virtual int Activation() const = 0;//returns signal Act(x)
 virtual void Print() const = 0;
 ;


#endif

Neuron.cpp

#include "../inc/Neuron.h"
#include "../inc/Axon.h"
#include "../inc/Dendrite.h"

namespace sann

 void Neuron::Update( int signal, const Neuron* fromWhich )
 
 switch( _type )
 
 case( INPUT ):
 std::cout << "Update for INPUT neuron " << _name << "n";
 return;
 case( HIDDEN ):
 std::cout << "Update for HIDDEN neuron " << _name << "n";
 break;
 case( OUTPUT ):
 std::cout << "Update for OUTPUT neuron " << _name << "n";
 break;
 
 _value += _dendrite->Transmit( fromWhich, signal );

 if( _dendrite->IsComplete() )
 
 _dendrite->Reset();
 Emit();
 
 


 Axon* Neuron::CreateAxon()
 
 if( _axon == nullptr )
 
 _axon = new Axon( this );
 
 return _axon;
 


 bool Neuron::AcceptSynapse( Neuron* sender )
 
 if( this->_type == INPUT )
 
 sannwrn( "Input neuron cannot have a dendrite." );
 return false;
 
 else if( sender == this )
 
 sannwrn( "Neuron cannot be connected with itself." );
 return false;
 
 else if( sender->GetType() == OUTPUT )
 
 sannwrn( "Output neuron cannot have an axon" );
 return false;
 
 else
 
 if( _dendrite == nullptr )
 
 _dendrite = new Dendrite();
 
 _dendrite->CreateSynapse( sender );
 return true;
 
 


 void Neuron::Emit() const
 
 if( _axon == nullptr ) return; 
 _axon->Transmit( this->Activation() );
 


 size_t Neuron::GetDendriteSize() const
 
 if( _dendrite == nullptr ) return 0; 
 else
 
 return _dendrite->GetSize();
 
 


 size_t Neuron::GetAxonSize() const
 
 if( _axon == nullptr ) return 0; 
 else
 
 return _axon->GetSize();
 
 

Axon

Axon.h

#ifndef AXON_H
#define AXON_H

#include "Neuron.h"

namespace sann

 class Axon 
 
 protected :
 Neuron* _root;//master of this axon
 std::vector<Neuron* > _tree;//pointers to receivers

 void Transmit( int signal ) const;//send signal to all receivers

 /*Only Neuron can create an instance of this class*/
 Axon( Neuron* root ) : _root( root ) 
 Axon( const Axon& ) = delete;
 Axon& operator=( const Axon& ) = delete;

 public :
 void CreateSynapse( Neuron* receiver );//connect a receiver

 inline size_t GetSize() const return _tree.size(); 

 /*To have an access to ctor from Neuron*/
 friend class Neuron;
 ;


#endif

Axon.cpp

#include "../inc/Axon.h"

namespace sann

 void Axon::Transmit( int signal ) const
 
 for( size_t i = 0; i < _tree.size(); i++ )
 
 _tree[i]->Update( signal, _root );
 
 


 void Axon::CreateSynapse( Neuron* receiver )
 
 if( receiver->AcceptSynapse( _root ) )
 
 _tree.push_back( receiver );
 
 

Dendrite

Dendrite.h

#ifndef DENDRITE_H
#define DENDRITE_H

#include "Neuron.h"

namespace sann

 class Dendrite
 
 protected :
 /*Not sure about this.*/
 struct Synapse
 
 Synapse( const Neuron* neuron, bool state, int weight ) :
 _neuron( neuron ),
 _weight( weight ),
 _state( state )
 
 const Neuron* _neuron;
 int _weight;
 bool _state;
 ;
 /*stores pointer to senders with
 *their weights and statuses: either neuron 
 *has received signal from a sender or not.*/
 std::vector<Synapse> _tree;

 /*Returns weighted signal from sender (fromWhich)
 *if sender is in the sender's list*
 *and zero otherwise*/
 int Transmit( const Neuron* fromWhich, int signal );
 bool IsComplete() const;//check if neuron have received signals from all senders
 void Reset();//reset statuses of all senders

 /*Only Neuron can create an instance of this class*/
 Dendrite() 
 Dendrite( const Dendrite& ) = delete;
 Dendrite& operator=( const Dendrite& ) = delete;

 public :
 void CreateSynapse( const Neuron* sender );//connect a sender

 inline size_t GetSize() const return _tree.size(); 

 /*To have an access to ctor from Neuron*/
 friend class Neuron;
 ;

#endif

Dendrite.cpp

#include "../inc/Dendrite.h"

namespace sann

 int Dendrite::Transmit( const Neuron* fromWhich, int signal )
 
 for( size_t i = 0; i < _tree.size(); i++ )
 
 if( fromWhich == _tree[i]._neuron )
 
 _tree[i]._state = true;
 return signal * _tree[i]._weight;
 
 
 return 0;
 


 bool Dendrite::IsComplete() const
 
 for( size_t i = 0; i < _tree.size(); i++ )
 
 if( !_tree[i]._state ) return false; 
 
 return true;
 


 void Dendrite::Reset()
 
 for( size_t i = 0; i < _tree.size(); i++ )
 
 _tree[i]._state = false;
 
 


 void Dendrite::CreateSynapse( const Neuron* sender )
 
 _tree.push_back( Synapse( sender, false, 1 ) );
 

Discussion

Neuron class

Emit function

The main problem I see is that this function might be waiting for a long time before return. This is because of recursive process of signal propagation. The worst case is when all the receivers has almost full dendrite and get the last signal from a sender and emit theirselves and this process repeats until the last neuron in the propagation chain. is it a problem or I am missing something?

Creation of Dendrite and Axon

I have Axon and Dendrite as a pointers. So they are initialized as nullptr in the constructor. When they are needed they are created via new operator (note that Neuron instance can have only one instance of Axon and Dendrite). I suspect that it smells like potential memory leak. Would destructor resolve this problem?

Dendrite class

Struct Synapse

I am not sure about declaring struct Synapse inside a Dendrite class. Is it bad practice? The motivation is I do not need to use that struct outside of this class. And also I need to store Neuron's together with its status and weight.

IsComplete function

I see that I check if signals from all senders have been received by iterating over entire dendrite tree while having function Transmit that already responsible for setting statuses of senders. Is it actually a problem or kind of a design flaw?

Mediator class

As you could see Axon and Dendrite has a lot in common. I am thinking about create a MediatorComponent class to place all of similarities inside it and then inherit from this class. What do you think?

So let me know what do you think about it. As always any suggestion, critic or help would be appreciated. And sorry for a lot of material.