Neuromorphic system using capacitor synapses

Abstract

Artificial intelligences are indispensable social infrastructures, neural networks are embodiment methodologies, and neuromorphic systems are promising solutions for compact size and low energy. Memristors were first prepared for the synapse devices but incur energy consumption, and memcapacitors were next prepared but have small dynamic ranges of capacitance. In this research, we have developed a neuromorphic system using capacitor synapses. Here, multiple capacitors have binary-weighted capacitances and are controlled to be connected to intermediate signals. They are discharged through transistors, and when they fall below the threshold voltage, the output signals are inverted. After all, electric charges in the multiple capacitances are summed and measured by the inverting intervals, which is the same as multiply–accumulate operation. A large-scale integration chip is actually fabricated. The working is confirmed by MNIST, and the circuit-aware rounding improves the accuracy to 96%, indicating a sufficient possibility for practical applications, and the energy efficiency is 163 GOPS/W even by the 180 nm technology, indicating a great potential for low energy consumption.

Introduction

Artificial intelligences are indispensable social infrastructures for smart information worlds^1,2, and neural networks are the most general embodiment methodologies with quite skillful biomimicries^3,4. However, conventional frameworks of neural networks are large and complicated software that runs on high-spec and energy-consuming hardware such as Neumann-type supercomputers, which are not customized for neural networks^5,6. Neuromorphic systems are promising customized solutions for compact system size and low energy consumption by device and hardware-level biomimetics^7,8, which consist of processing elements of neuron elements and synapse elements^9,10. Memristors^11,12,13, variable conductance devices, were first prepared for the synapse devices^{14,15,16,17,18,19,20,21,22,23}, but they are a kind of resistors and hence in principle incur energy consumption as Joule heating. Therefore, memcapacitors^24,25, variable capacitance devices, were next prepared^26,27,28, because memcapacitors are a kind of capacitors and hence in principle incur no energy consumption, but the dynamic ranges of the variable capacitance are not so large, and the operation voltages cannot be so large in order not to overwrite the memorized capacitances. Incidentally, although memcapacitors can be emulated by transistors^29,30 and circuits^31,32, some complicated structures and driving are needed. Even in recent years, many researchers are continuously publishing neuromorphic systems using memristors^33,34,35,36 and memcapacitors^{29,30,37,38,39}. However, in most cases, only the synapse elements are actually fabricated and the neuromorphic systems are just virtually simulated, the synapse elements are not integrated in the neuromorphic systems, only brief reports are released and detailed information are unknown, and so on.

In this research, we have developed a neuromorphic system using capacitor synapses. In this neuromorphic system, multiple capacitors have binary weighted capacitance values, and they are controlled to be connected to intermediate signals. The connected capacitors and intermediate signals are charged, and each signal is discharged through transistors. When they fall below the threshold voltage, the output signals are inverted. After all, electric charges charged in the multiple capacitances in one synapse element and all synapse elements in one row are summed, and they are measured by the inverting intervals, which is exactly the same as the multiply–accumulate operation used in neural networks. A large-scale integration chip of the neuromorphic system is physically designed and actually fabricated. The working is confirmed by MNIST recognition, and the circuit-aware rounding improves the accuracy to 96%, which indicates a sufficient possibility of this neuromorphic system for practical applications. Moreover, the energy efficiency is 106 GOPS/W even by the Si CMOS 180 nm technology, which indicates a great possibility for low energy consumption.

Neuromorphic system using capacitor synapses

The neuromorphic system using capacitor synapses is shown in Fig. 1. Synapse elements are aligned in a matrix array. Here, a synapse element at column i and row j is shown in detail. Multiple capacitors have binary-weighted capacitance values. Here, four capacitors have capacitance values of C₀, C₁ = 2 × C₀, C₂ = 4 × C₀, and C₃ = 8 × C₀. They are controlled by weight signals, $w_{i,j} = \left( { - 1} \right)^{s} \left( {\mathop \sum \limits_{k} 2^{k} w_{k} } \right)$, to be connected through the designated wiring to an intermediate signal, $y_{j}^{ + }$ or $y_{j}^{ - }$. Here, the weight signals are five bits of one sign bit for either positive sign or negative sign, s, and four bits for a binary number, w₀, w₁, w₂, and w₃. The $y_{j}^{ + }$ is for the positive signal to be connected, and $y_{j}^{ - }$ is for the negative signal to be connected. The connected capacitors, including those in other synapse elements, and $y_{j}^{ + }$ and $y_{j}^{ - }$, are preliminarily charged to V_dd and kept connected by an input signal, $x_{i}$, and each signal is discharged through discharging transistors, Td, to GND. Here, Td is composed of four serially connected n-type transistors to regulate the discharge current. When $y_{j}^{ + }$ and $y_{j}^{ - }$ gradually fall below the threshold voltage, the logic buffers invert the output signals, $z_{j}^{ + }$ and $z_{j}^{ - }$. The detailed architecture of the neuromorphic system is explained in the “Methods” chapter. After all, electric charges charged in the multiple capacitances selected by the weight signals and input signals in one synapse element are summed for each positive and negative sign, namely, $q_{i,j} = \left( {\mathop \sum \limits_{k} 2^{k} w_{k} } \right)C_{0} V_{dd} x_{i} = \left| {w_{i,j} } \right|x_{i} C_{0} V_{dd}$, and those in all synapse elements in one row are summed in $y_{j}^{ + }$ and $y_{j}^{ - }$ for each positive and negative sign, namely, $q_{j}^{ + } = \left( {\mathop \sum \limits_{i} \left| {w_{i,j} } \right|x_{i} } \right)C_{0} V_{dd} {\text{ for }} s = +$ and $q_{j}^{ - } = \left( {\mathop \sum \limits_{i} \left| {w_{i,j} } \right|x_{i} } \right)C_{0} V_{dd} {\text{ for }} s = -$. They are measured by the inverting intervals of $z_{j}^{ + }$ and $z_{j}^{ - }$, namely, by re-defining $z_{j}^{ + }$ and $z_{j}^{ - }$ as their own inverting intervals, $z_{j}^{ + } = \left( {\mathop \sum \limits_{i} \left| {w_{i,j} } \right|x_{i} } \right) {\text{ for }} s = +$ and $z_{j}^{ - } = \left( {\mathop \sum \limits_{i} \left| {w_{i,j} } \right|x_{i} } \right) {\text{ for }} s = -$, where the proportional coefficient is presumed to be 1, and from $z_{j} = z_{j}^{ + } - z_{j}^{ - }$, the following equation is obtained with correct handling of the positive and negative signs, which is exactly the same as multiply–accumulate (MAC) operation used as a common step in neural networks⁴⁰ and neuromorphic systems⁴¹.

$$z_{j} = \mathop \sum \limits_{i} w_{i,j} x_{i}$$

(1)

Fig. 1

Neuromorphic system using capacitor synapses.

The operation procedure of the capacitor synapse is shown in Fig. 2. First, during the weight memorizing period, four capacitors, C₀, C₁, C₂, and C₃, are controlled by weight signals, s, w₀, w₁, w₂, and w₃, to be connected through the designated wiring to an intermediate signal, $y_{j}^{ + }$ or $y_{j}^{ - }$. Here, as an example, C₀ and C₃ are connected by w₀ and w₃ to $y_{j}^{ + }$ by s. Subsequently, during the preliminary charging period, the connected capacitors, C₀ and C₃, and the intermediate signal, $y_{j}^{ + }$, are charged to V_dd by a charging signal, c, where the unconnected capacitors are not charged, which avoids unnecessary energy consumption. Next, during the input writing period, an input signal, $x_{i}$, is inputted, and only the selected capacitors, C₀ and C₃, are kept connected. Finally, during the output reading period, the intermediate signal, $y_{j}^{ + }$, is discharged through the discharging transistors, Td, switched on by the discharging signal, d, to GND. When the intermediate signal, $y_{j}^{ + }$, falls below the threshold voltage, the logic buffer inverts the output signal, $z_{j}^{ + }$. Both $z_{j}^{ + }$ and $z_{j}^{ - }$ are outputted for positive signs and negative signs, respectively, and $z_{j} = z_{j}^{ + } - z_{j}^{ - }$ is also obtained.

Fig. 2

Operation procedure of the capacitor synapse.

The working confirmation of the output reading is shown in Fig. 3. Here, circuit simulation is outcarried with a transistor model⁴² and circuit simulator HSPICE⁴³. Voltage waveforms of the intermediate signal, $y_{j}^{ + }$, and the output signal, $z_{j}^{ + }$, with variation of the ratio of the selected capacitors are shown in Fig. 3a, and the ratio of the selected capacitors vs the inverting interval, $z_{j}^{ + }$, is shown in Fig. 3b. It is found that the inversion interval, $z_{j}^{ + } ,$ linearly depends on the ratio of the selected capacitors, and the inversion intervals are less than 30 μs, It is confirmed from this result that the MAC operation can be correctly performed in practically possible time.

Fig. 3

Working confirmation the output reading.

LSI chip of the neuromorphic system

A large-scale integration (LSI) chip of the neuromorphic system is shown in Fig. 4. The LSI chip is physically designed and actually fabricated. Here, a silicon (Si) complementary metal–oxide–semiconductor (CMOS) 180 nm technology is used. The computer-aided design (CAD) layout of the synapse element is shown in Fig. 4a. The four capacitances are formed by parallel plate capacitances of a silicon oxide (SiO₂) thin film between two metal electrodes, whose capacitance density is 1 fF/μm², and therefore they have capacitance values and area values of C₀ = 20 fF = 20 μm², C₁ = 2 × C₀ = 40 fF = 40 μm², C₂ = 4 × C₀ = 80 fF = 80 μm², and C₃ = 8 × C₀ = 160 fF = 160 μm². Most of the transistors are CMOS field-effect transistors (FET) and have an experienced dimension of W = 1.25 μm and minimum dimension of L = 0.18 μm, except that the discharging transistors are n-type metal–oxide–semiconductor (NMOS) FETs and have deliberated dimensions of W = 0.22 μm and L = 10 μm, which prolongs the inverting intervals of the output signals. Vdd = 1.8 V, which is the standard voltage for the Si CMOS 180 nm technology.

Fig. 4

LSI chip of the neuromorphic system.

The microscope photograph is shown in Fig. 4b. It is of course found that the microscope photograph of the synapse element actually fabricated is exactly the same as the CAD layout of it. The overview photograph is shown in Fig. 4c. The LSI chip is packaged through wire bonding in a ceramic package. The peripheral control circuit boards are shown in Fig. 4d. The LSI chips are mounted in sockets and evaluated.

Working confirmation by MNIST recognition

The working confirmation by MNIST recognition.is shown in Fig. 5. The modified National Institute of Standards and Technology database (MNIST) is the most well-known database of handwritten digits⁴⁴ and is already outdated for practical purposes, but it is still very convenient to evaluate the potential feasibility of emerging systems. The network architecture is shown in Fig. 5a. First, the pre-process is conducted by a convolutional neural network (CNN)⁴⁵, which is not done by the neuromorphic system but by an external program^46,47,48. From the MNIST, 28 × 28 images are processed by a 3 × 3 kernel for edge detection, and they are reshaped to be 26 × 26 images. Moreover, they are processed by nine 3 × 3 kernels for direction detection, namely, kernels for the direction of 0, (1/8)π, (1/4)π, (3/8)π, (1/2)π, (5/8)π, (3/4)π, (7/8)π, and π, and they are reshaped and added with the original images to be nine 24 × 24 images. Furthermore, they are processed by a 2 × 2 kernel for max pooling, and they are reshaped to be nine 24 × 24 images. Next, the main process is conducted by a fully-connected neural network (FC)⁴⁹, which is realized by the aforementioned LSI chip of the neuromorphic system using the capacitor synapses. $x_{0}$–$x_{i}$–$x_{143}$ and $z_{0}$–$z_{j}$–$z_{9}$ of the detailed architecture in Fig. 1 correspond to $x_{0}$–$x_{143}$ and $z_{0}$–$z_{9}$ of the FC in Fig. 5a. The 24 × 24 images are inputted as input signals, $x_{0}$–$x_{143}$, and inverting intervals are outputted as output signals, $z_{0}$–$z_{9}$, which corresponds to the labels of the handwritten digits of 0–9.

Fig. 5

Working confirmation by MNIST recognition.

Training of the FC is executed to determine the weight signals, $w_{i,j}$, as follows. First, a back-propagation method is as usual employed to theoretically determine synaptic weights, $v_{i,j}$, by an external program of Python simulation⁵⁰. This is a merely normal method like the one described in textbooks⁵¹. Next, $v_{i,j}$ is converted to $w_{i,j}$ by the following three fashions. This is because $v_{i,j}$ is mathematically a real number, whereas $w_{i,j}$ is only four bits except for the sign bit and can express only 0–15, and therefore suitable conversion is required. The weight conversion is shown in Fig. 5b, and the conversion functions are shown in the left graph of Fig. 5b. “Simple rounding” is a simple conversion by the following equation. $w_{i,j}$ is proportional to $v_{i,j}$ by the proportionality factor $\alpha$, discretized to an integral number n in 0–$\beta$, and limited within the range ± $\beta$, where $\beta = 15$ here.

$$w_{i,j} = {\text{sign}}\left( {v_{i,j} } \right) \times \left\{ {\begin{array}{*{20}l} n \hfill & {{\text{if}}} \hfill & {n - 1 < \left| {\alpha v_{i,j} } \right| \le n} \hfill \\ \beta \hfill & {{\text{if}}} \hfill & {\beta < \left| {\alpha v_{i,j} } \right|} \hfill \\ \end{array} } \right.\quad 0 \le {\text{integer}}\;n \le \beta$$

(2)

In addition, “Stochastic rounding” randomly rounds up or down while retaining the tendency of rounding off to the nearest integral number. “Circuit-aware rounding” considers parasitic capacitances of the switching transistors for w₀, w₁, w₂, and w₃ by the following equation. It is assumed that $C_{p} = \gamma \delta_{n} C_{0}$ where $\delta_{n}$ is the number of the switching transistors in the off state when $w_{i,j} = n$.

$$w_{i,j} = {\text{sign}}\left( {v_{i,j} } \right) \times \left\{ {\begin{array}{*{20}l} n \hfill & {{\text{if}}} \hfill & {\left( {n - 1} \right) + \gamma \delta_{n - 1} < \left| {\alpha v_{i,j} } \right| \le n + \gamma \delta_{n} } \hfill \\ \beta \hfill & {{\text{if}}} \hfill & {\beta + \gamma \delta_{\beta } < \left| {\alpha v_{i,j} } \right|} \hfill \\ \end{array} } \right.\quad 0 \le {\text{integer}}\;n \le \beta$$

(3)

The histograms of $w_{i,j}$ are shown in the right graph of Fig. 5b. Incidentally, the average of $w_{i,j}$, $\overline{{w_{i,j} }}$, is 3.41, namely, $\overline{{w_{i,j} }} /\beta = 3.41/15 = 22.7\%$, for the circuit-aware rounding, which is used later as a pragmatical ratio for the analysis of the energy efficiency. Finally, the weight signals, $w_{i,j}$, namely, w₀, w₁, w₂, and w₃, are uploaded to the neuromorphic system.

The achieved accuracies are shown in the left table of Fig. 5c. The champion accuracies are shown for each weight conversion with parameter optimization. First, the Python simulation achieves an accuracy of 97%, which is the theoretical highest baseline, because it is assumed that all the processing elements of neuron elements and synapse elements work perfectly. Next, the neuromorphic chip using the simple rounding as the weight conversion achieves 94%, while the stochastic rounding improves it to 95%, and the circuit-aware rounding further improves it to 96%. It is surprising that the degradation of the accuracy from the Python simulation to the neuromorphic system is only 1%. It is indicated from this result that the neuromorphic system using the capacitor synapses has a sufficient possibility for practical applications.

The energy efficiency is also shown in the left table of Fig. 5c, and the ratio of the selected capacitors vs the energy consumption is shown in the right graph of Fig. 5c. First, the energy consumption for one inference for the pragmatical ratio of ${w}_{i,j}$ of 22.7% is 884pJ, which is calculated from the circuit simulation, the equivalent operation number for the MNIST recognition is 144 OP, and therefore the energy efficiency is 144 OP/884 pJ = 163 GOPS/W. It is surprising that this energy efficiency is achieved by the Si CMOS 180 nm technology. According to Dennard scaling, the energy efficiency will be 951 TOPS/W if it progresses to 10 nm technology. It is indicated from this result that the neuromorphic system using the capacitor synapses has a great possibility for low energy consumption.

The performance comparison with other state-of-the-art reports on neuromorphic systems implementing memristors and memcapacitors is shown in Table 1. First, it is found that the accuracy in this research is roughly the same as those in the other state-of-the-art reports. Next, it should be noted that this research is based only on the Si CMOS technology, and therefore miniaturization can be promoted according to Dennard scaling, which is different from the other reports. As a result, as aforementioned, the energy efficiency will be 951 TOPS/W if it progresses to 10 nm technology, which will be the highest among this table.

Table 1 Performance comparison with other state-of-the-art reports on neuromorphic systems implementing memristors and memcapacitors.

Conclusion

We have developed a neuromorphic system using capacitor synapses. In this neuromorphic system, multiple capacitors have binary weighted capacitance values of C₀, C₁ = 2 × C₀, C₂ = 4 × C₀, and C₃ = 8 × C₀. They are controlled by weight signals, ${w}_{i,j}={\left(-1\right)}^{s}\left(\sum_{k}{2}^{k}{w}_{k}\right)$, and an input signal, ${x}_{i}$, to be connected to intermediate signals, ${y}_{j}^{+}$ or ${y}_{j}^{-}$, for the positive and negative signals, respectively. The connected capacitors, ${y}_{j}^{+}$, and ${y}_{j}^{-}$ are preliminarily charged to V_dd, and each signal is discharged through discharging transistors, Td, to GND. When ${y}_{j}^{+}$ and ${y}_{j}^{-}$ gradually fall below the threshold voltage, the logic buffers invert the output signals, ${z}_{j}^{+}$ and ${z}_{j}^{-}$. After all, electric charges charged in the multiple capacitances in one synapse element and all synapse elements in one row are summed for each positive and negative sign, namely, ${q}_{j}^{+}=\left(\sum_{i}\left|{w}_{i,j}\right|{x}_{i}\right){C}_{0}{V}_{dd} \text{ for } s=+$ and ${q}_{j}^{-}=\left(\sum_{i}\left|{w}_{i,j}\right|{x}_{i}\right){C}_{0}{V}_{dd} \text{ for } s=-$. They are measured by the inverting intervals, ${z}_{j}^{+}$ and ${z}_{j}^{-}$, namely, ${z}_{j}^{+}=\left(\sum_{i}\left|{w}_{i,j}\right|{x}_{i}\right) \text{ for } s=+$ and ${z}_{j}^{-}=\left(\sum_{i}\left|{w}_{i,j}\right|{x}_{i}\right) \text{ for } s=-$, and from ${z}_{j}={z}_{j}^{+}-{z}_{j}^{-}$, ${z}_{j}=\sum_{i}{w}_{i,j}{x}_{i}$ is obtained, which is exactly the same as MAC operation used in neural networks. An LSI chip of the neuromorphic system is physically designed and actually fabricated. The working is confirmed by MNIST recognition, and the circuit-aware rounding improves the accuracy to 96%, which indicates a sufficient possibility of this neuromorphic system for practical applications. Moreover, the energy efficiency is 163 GOPS/W even by the Si CMOS 180 nm technology, which indicates a great possibility for low energy consumption.

Methods

Neuromorphic system using capacitor synapses

The detailed architecture of the neuromorphic system using the capacitor synapses is shown in Fig. 6. During the weight memorizing period, all the weight signals, $w_{i,j}$, are transmitted one by one through flip flops with a clock signal, $ck$, distributed all the synapse elements, and stored by the flip flops. Multiple capacitors having binary weighted capacitance values, C₀, C₁, C₂, and C₃, are controlled by weight signals, $s$, $w_{0}$, $w_{1}$, $w_{2}$, and $w_{3}$, using transmission gates, etc. to be connected to an intermediate signal $y_{j}^{ + }$ or $y_{j}^{ - }$. Subsequently, during the preliminary charging period, the connected capacitors and the intermediate signal are charged to V_dd by a charging signal, c. Next, during the input writing period, an input signal, $x_{i}$, is inputted, and only the selected capacitors are kept connected using a flip flop, NAND gates, etc. Finally, during the output reading period, the intermediate signals, $y_{j}^{ + }$ and $y_{j}^{ - }$, are discharged through the discharging transistors, Td, switched on by the discharging signal to GND. Here, two sets of the Td composed of four serially connected transistors are prepared to regulate the discharge current by the discharging signals, d₁ and d₂. When $y_{j}^{ + }$ and $y_{j}^{ - }$ gradually fall below the threshold voltage, the logic buffers invert the output signals, $z_{j}^{ + }$ and $z_{j}^{ - }$. The inversion intervals are evaluated by an outside circuit with sufficiently high operation frequency. It should be noted that additional logic buffers are inserted throughout to boost these signals.

Fig. 6

Detailed architecture of the neuromorphic system using the capacitor synapses.

In summary, the synapse element comprises the multiple capacitors controlled by the weight signals to be connected through the designated wiring. In this research, the weight signals are stored by the flip flops. Therefore, the synapse element functions as a volatile synapse element. Alternatively, the weight signals can be stored by replacing the flip flops with some non-volatile devices, such as, floating-gate transistors and ferroelectric-gate transistors. In these cases, the synapse element functions as a non-volatile synapse element.