Table 2 Comparison of key parameters of nonlinear activation devices and circuits
Function | Type | Energy (power) consumption | Area | Delay time | Refs. |
|---|---|---|---|---|---|
Sigmoid, Hyperbolic tangent, Softsign, Softplus, ELU, SeLU | CMOS | 3.7 mW (5 bit), 4.6 mW (8 bit) | 0.003 mm2@16 nm | N/A | |
Tanh | CMOS | 0.071 mW | 582 μm²@90 nm | 1.40 ns | |
ReLU, Leaky ReLU, Tanh | CMOS | 2.155 mW | 35.809 μm²@180 nm | 420 ps | |
Sigmoid | CMOS | 11.4 μW | 184 μm²@180 nm | N/A | |
Tanh, Sigmoid | CMOS | 0.07 mW | 1554 μm²@180 nm | 2.25 ns | |
ReLU, Sigmoid | CMOS | 55-88 mW | 0.028 mm2@45 nm | 570 ps | |
ReLU | Mott | 199.5 pJ | 0.64 μm² | 61.4 ns | |
ReLU | ADC | 29.6 fJ | 45 μm2 | N/A | |
ReLU, Tanh, Sigmoid | ADC | 4.75 nJ | 0.086 mm2@90 nm | 207 ns | |
ReLU | ADC | 87 μW | 7200 μm²@180 nm | 2.1 μs | |
Tanh, clipped-ReLU | ADC | 15 μW | 101.64 μm²@28 nm | N/A | |
ReLU | ADC | 3.08 TOPS/W | 3.91 TOPS/mm2@28 nm | N/A | |
Mish, GELU, SiLU | ADC | 168 μW | 1218 μm²@65 nm | N/A | |
Tanh, ReLU | Digital processors | 1982 mJ | 263 mm2 | N/A | |
Tanh | ASIC | N/A | 17864.24 μm2@180 nm | 2.45 ns | |
Sigmoid Tanh | ASIC | 2.16 mW 2.64 mW | 7973.2 μm2@65 nm 10180 μm2@65 nm | 3 ns, 3.16 ns | |
Sigmoid | ASIC | 2.66 mW | 8097.48 μm2@90 nm | 1.96 ns | |
ReLU | ASIC | 20.4 mW | 130 nm | 14 ns | |
Sigmoid | FPGA | 19 mW | N/A | 55 ns | |
Tanh | FPGA | N/A | N/A | 45.4 ns | |
Sigmoid, Tanh | ASIC FPGA | 0.31 mW, 0.28 mW 56 mW, 56 mW | 4678.78 μm2@180 nm 5582.5 μm2 | 1.6 ns, 1.7 ns 23.1 ns, 23 ns | |
Sigmoid | MRAM | 18.04 μW | 0.138 μm2 | 48 ns | |
ReLU, Sigmoid | Memory | 16 fJ for ReLU 80 aJ for Sigmoid | 15 μm² | <8 ns | This work |
