The nodes in ANNs are highly simplified biological neurons, and the networks with such nodes can perform learning by adjusting the synaptic weights of the edges [1]. While the ANNs partially mimic biological learning, usually their network architecture is artificially designed in regular patterns or searched by computer algorithms [2-5]. The architecture of ANNs matters because it is the streambed for neural information flow such as aggregation and distribution.

The purpose of this study is to explore ANN architectures inspired by biological neural networks, especially the connectome. A connectome is the complete wiring diagram of all the synaptic connections between the entire neurons of an animal [6] and thus it is the biological basis of any mental process and behavior. We construct an ANN using the C. elegans connectome, the only complete connectome available to date, as a basis for image classification tasks [7].

Many experiments have demonstrated that the ANN architecture is associated with accuracy, specifically in image classification tasks. The convolutional neural networks (CNNs) outperform the multilayer perceptrons [8]. The success of deep learning is attributed to the fact that accuracy improves for the networks with more hidden layers [9, 10]. In addition, skip connections between distant layers, which mimic biological long-range feedforward connections, improve accuracy further [11, 12].

Another plausible way of constructing a network architecture is to refer to the biological networks. In a recent study, a connectome-based modeling of ANN architecture was explored using hermaphrodite C. elegans model [7], but this experiment made significant changes to the connections of the network. It randomly changed edge directions in the network to construct a directed acyclic graph (DAG). Importantly, the neurons in the biological networks have specific directions of information flow. First, the sensory neurons turn the external stimuli into the neural information. Next, the interneurons process and relay the information from the sensory neurons to the next neurons. Finally, the motor neurons transform the responsive neural information into behavior. The DAG with randomized edge direction impairs such direction of information flow of the connectome.

As an expansion of the previous study, here we test whether more realistic biological network architecture yields higher accuracy. We model connectome-based ANN architectures preserving the neuronal types and edge directions. We also compare the two sexes of C. elegans to test how the sexual dimorphism in the network structures affect the computation. It is shown that the new DAG which preserves the direction of information flow yields higher accuracy and that the two sexes of C. elegans yield the similar accuracy. In conclusion, we argue that the biological networks have a great potential for high-performance ANN architecture.

Data source

We used the data from WormWiring, a website hosting the nematode connectome data mapped from electron microscope images [13]. We downloaded the latest versions of structural connectomes for both hermaphrodite and adult male C. elegans from the website [14]. The network structures of two sexes are chiefly identical with a few exceptions including the addition of roughly 80 mating-related neurons in the male tail compared to the hermaphrodite. Comparison of the networks with minor difference may reveal the importance of the difference. The connectome data include both chemical and electrical synapses, but we used only the chemical synapses because the mutual connections of electrical synapses couple the neurons which is not a commonly employed operation in the ANNs.

The chemical connectome is represented by a weighted directed graph, where the edge weights are the number of synapses between a pair of neurons and the edge directions are the direction of synapse pointing from pre- to postsynaptic neuron. The data are provided in MATLAB’s graph data structure together with metadata. In the metadata, each neuron was given a unique name and was classified into one of the five types (sensory, interneuron, motor, sex-specific, or other), where the sensory type information is utilized in our method as described later. After removal of electrical synapses and isolated neurons, 282 nodes of neurons and 3,567 edges of synaptic connections remained in the graph of hermaphrodite connectome, while 357 nodes and 3,889 edges remained in the graph of male connectome.

Extraction of DAG and network topology from connectome

The original C. elegans connectome contains cyclic connections that make feedback loops. The ANNs with feedback connections are generally called the recurrent neural networks (RNNs), which are generally advantageous in temporal computation, such as maintaining the short-term memory and generation of oscillatory patterns [15]. While the temporal computation is not essential in the single static image recognition, the training of RNNs takes a long time and is challenging mainly due to the issue known as vanishing and exploding gradient [16]. Therefore, the cyclic connections in the connectome need to be removed in the neural network for image classification.

A directed acyclic graph (DAG) can be obtained by removing cyclic connections from a graph (Fig. 1A~C). In the previous study, a random DAG is obtained from the connectome by randomly assigning indices to all the nodes and then setting the edges to point from smaller-index nodes to larger-index nodes (Fig. 1A, B). Then, the topology of the neural network is determined by adding two nodes to the DAG: the source node connecting to the input layer nodes and the sink node connected by the output layer nodes, which ensure singular input and output through the DAG graph (Fig. 1D). Here, 0-in-degree nodes are considered as the input layer and 0-out-degree nodes are considered as the output layer. The source node copies the input and distributes it to the input layer nodes and the sink node performs average on the output layer nodes of the DAG [7, 17]. However, this procedure severely disorients the information flow of the connectome because the determination of edge direction is arbitrary.

In this study, we aimed to construct a new network topology which preserves the flow of information of the connectome. A constrained DAG is obtained by the procedure which sets all and only the sensory neurons to the input layer and maximally retains the number and direction of the edges. We extract a DAG from the connectome G with the following steps. Let the set of all the neurons in G as N. Each node has a flag initially set as “unused”. The integer index i indicates the distance from sensory neurons, initially set as 0. The set X(i) is the set of nodes at the distance i, initially set as empty.

1. Get all the nodes of sensory neurons from N and add them to an empty set X(i). Set the flag of each node in X(i) as “used”.

2. Search G to find all the “unused” nodes which are connected by nodes in X(i), and add them to a new empty set X(i+1) setting the flags to “used”. Increase i by one.

3. Repeat step 2 until the flags of all the neurons are “used”.

4. Let D be a graph with nodes in N, initially having no edge. Copy the edges of G to D one after another in the decreasing order of weight if the edge satisfy all the three conditions below: 1) it does not connect two nodes in X(0), 2) it does not point from a X(s) node to a X(t) node when s>t, and 3) it does not make a cycle in D.

Contrary to the case of random DAG, the layer of X(0) nodes is considered as the input layer for the constrained DAG. Identically to the case of random DAG, 0-out-degree nodes are considered as the output layer. While all and only the sensory neurons are in the input layer, subsets of the motor and sex neurons become the output layer nodes. The constrained DAG obtained from the weighted directed graph in (Fig. 1A) is illustrated in (Fig. 1C). The difference in the structures of random DAG and constrained DAG is obvious from the figures (Fig. 1B, C). Similarly to the case of random DAG, the source and sink nodes are added respectively to input layer nodes and output layer nodes (Fig. 1E). Note that the constrained DAG preserves the major information flow of the connectome.

The hermaphrodite DAG has a total of 282 nodes and 2,260 edges and the male DAG has a total of 357 nodes and 2,494 edges, except the source and the sink nodes. These networks were used as the basis of the neural network architectures as in the following section.

Network architecture

We construct the neural network architecture from the network topology identically to the previous study (Fig. 2) [7]. The source node is a 2D convolution with kernel size of 3, stride size of 2 and padding of 1. All the DAG nodes are assigned with the identical processing units: rectified linear unit (ReLU) and convolution followed by batch normalization. The convolution operation is performed by 78 parallel kernels each of which is 2D convolution with kernel size of 3, stride size of 1 and padding of 1. The outputs of every node are passed onto next nodes following the edge directions in the DAG structure. Since all the nodes except the source nodes can receive inputs from multiple nodes, the inputs to each node were aggregated by taking a weighted sum. The outputs of output nodes are averaged and fed into the sink node. The sink node is a 2D convolution with kernel size of 1 and stride size of 1, followed by a fully connected layer for class decision. The networks were implemented and trained using Torch framework with its Python interface adopted from the previous study [7].

Training and evaluation

We trained and evaluated the two networks on MNIST and fashion-MNIST datasets [18, 19]. The MNIST dataset consists of 28×28-pixel grayscale images of hand-written digits. There are 60,000 images for training and 10,000 images for tests, and the number of images for each digit are the same. The fashion-MNIST dataset contains images of fashion items such as clothes, bags, and shoes, whose dimensions and numbers are the same as the MNIST images. Following the conventional preprocessing, each image was normalized with the mean value of 0.5 and standard deviation of 1. No data augmentation was performed. For consecutive training iterations, the network was fed with a batch of 100 images sequentially taken from the training set. The error measure was the negative loss likelihood. With the ADAM optimizer [20], the learning rate was initially set to 0.004 and was multiplied by 0.1 every 10,000 iterations. After each training epoch, the training dataset was reshuffled for upcoming iterations to prevent the network from learning particular patterns in the sequence of image feed.

The accuracy of the image classification of the two networks was measured for the test sets. For the MNIST classification task, the hermaphrodite network showed 99.6% and the male network showed 99.5% accuracy after roughly 15,000 iterations (Fig. 3A). For the fashion MNIST, the hermaphrodite network showed 92.7% and the male network showed 92.6% accuracy after roughly 15,000 iterations (Fig. 3B).

The error investigation reveals that most of the examples of misclassified MNIST digit images are even more difficult to recognize for a human compared to the correctly classified digit images (Fig. 3C, E). Similarly, the misclassified fashion images are less clear than the correctly classified images (Fig. 3D, F). However, not all the misclassified fashion images are unrecognizable to a human, which suggests that the classification of fashion MNIST is more difficult than that of digit MNIST to ANNs because of the complexity of the images.

The two networks exhibited similar accuracy, less than 0.1% point of accuracy difference, for both tasks. The difference in the connectomes of the two sexes is chiefly the number of sex neurons and network topologies are almost identical (Fig. 2) [14]. This may suggest that the role of the sex neurons is only minor in image classification because the sex neurons of male form an extra network added to the neural network of hermaphrodite and exist on the outside of the main path of information flow.

These networks achieved higher accuracy than the networks which were based on the random DAG did (Table 1). The accuracy of our networks is higher than the reported mean accuracy values of the random DAG network by 0.6% point on MNIST and by 2.7% point on fashion MNIST. Since there is no other significant difference in the training procedures, we attribute the higher accuracy to the method of constructing the DAG from the connectome, preserving the structure of information flow. In addition, the MNIST accuracy of our networks with 0.21 million adjustable parameters is comparable to the previous studies by a 2-layer convolutional neural network with 0.13 million parameters and by a 6-layer deep neural network with 12 million parameters [21, 22]. In the previous study, the accuracy can have a range of values due to the randomness of the DAG, but unfortunately the range was not given in the manuscript numerically but only an error bar is given on a plot. Except for the random DAG network, the rest of the networks applied to MNIST, including those in Table 1, present only a single value of accuracy because the tests are performed on a fixed set of data. Since the goal of these studies is to maximize the accuracy within the fixed set of test data, their performances can only be evaluated by the single numbers without error bars and the quality of those works are appreciated as given by the numbers.

We showed that the neural networks, constructed from C. elegans connectome preserving the information flow, are successful in image classification without data augmentation nor intensive fine-tuning during the training. The networks which exhibited higher accuracy than ours employed much greater number of nodes and adjustable parameters, or even a multitude of network models. Further studies are needed to test whether the networks from the visual system exhibit superior performance. This was briefly tested in the previous study with the regional network of mouse visual cortex [7], but more sophisticated investigations are required using the neuronal networks. Then, the crucial questions will be which properties of the biological networks are essential for the given task. The answers to such questions will provide the fundamental principle to design the network architectures for advanced AIs [6, 23].

The biologically-inspired ANN in this study is a variance of the conventional CNNs whose architecture is regular, as it utilizes the same convolution operations but with different network architecture which is irregular. The architecture of typical CNNs is composed of the layers consisting of similar numbers of nodes, and the nodes in adjacent layers are fully connected so that the number of connections of the nodes in a layer are the same. In recent studies, the “skip connections” are employed to connect nodes between distant layers which are also regular. On the other hand, the network in this study has highly varying number of nodes for different layers with heterogeneous numbers of connections. The skip connections, designed by nature, are also irregular and more frequent than the case of the typical CNNs. While the irregular architecture increases the computational cost on both running time and memory, the frequent skip connections can partly diminish the problem and help prevent the “vanishing gradient” problem in the learning algorithms [24]. Note that the computational cost may not be intrinsic to the biological networks, but it may be because the biological computation is performed on the machine with different framework.

We also note that the nodes used in this study are all identical and the edge weights of a single node can have arbitrary, either positive or negative values. This is the common setting for most of the ANNs. The positive and negative connection weights correspond to excitatory or inhibitory synaptic connections of biological networks. However, generally in biological networks, the neurons by themselves are excitatory or inhibitory, which means that all the output synaptic connections from a single neuron are either all excitatory or all inhibitory. It is not clear whether the natural selection of excitatory and inhibitory neurons over excitatory and inhibitory synapses was to improve computational efficiency or to compromise with biological limitations. The introduction of excitatory and inhibitory nodes to ANNs, whose output weights are all positive or all negative, may lead to interesting results. More generally, utilization of various types of neurons, such as the interneurons commonly found in the biological networks, may be critical in improving the performance of ANNs. For instance, a large part of the success of CNNs is attributed to the employment of the convolution and pooling operations, respectively performing filtering and down-sampling which imitate the simple and complex cells first discovered in the visual cortex of a cat [25]. Likewise, extensive surveillance is required to examine various computational components which mimics various functions of the biological neurons as a revolutionary advancement of ANNs.

The early AIs, particularly the ANNs, had advanced under the inspiration of neuroscience, mimicking the functional mechanisms of biological neurons and neural networks. Although such a relation appears to have become vague recently, the inspiration of neuroscience is still crucial for the advancement of AI. If the goal of an AI is to solve a specific engineering problem, any solution that serves the goal may be acceptable and it is not mandatory for AIs to imitate the mechanisms of biological neural systems. Nevertheless, finding the solution from the vast search space is so difficult that the hints from neuroscience will greatly reduce the search space and accelerate the search speed. The same argument can be made to the design of network architecture as suggested in this study [23].

This study also showcases that biological networks can be used for the architecture of ANNs. This may suggest that the biological networks, which have evolved specifically to serve the computational demand for the survival of an organism, have the plasticity that is capable of general neural computation.

This research was supported by KBRI Basic Research Program (21-BR-03-01) funded by the Korean Ministry of Science and ICT. C.P. acknowledges the support by the BK21 FOUR program through NRF funded by the Korean Ministry of Education.

C.P. and J.S.K. designed the project, interpreted the data, and wrote the paper. C.P. wrote the codes and performed the experiments. J.S.K. supervised the work.

Accuracy of different network types for MNIST image classification. The accuracy of the network in this study based on the constrained DAG is superior to the case of the random DAG and is comparable to a few deep neural networks with common architecture