Fig. 6: Optimizing mutual information recovers one-hot encodings and improves with greater input multiplicity.
From: Limits on the computational expressivity of non-equilibrium biophysical processes
A Drawing of a fully connected graph with Nn = 6, with one (D = 1) input force Fa applied along the solid arrow for M = 1 and along both arrows for M = 2. B Top: For M = 1, plots of the components of the optimized π(Fa) colored as in the graph drawing in A. Insets schematically show the steady state of the graph for at the corresponding value of Fa. Note that several components are close to zero for all value of Fa. Middle: Same as top, but with M = 2. Bottom: Plot of the input distribution p(Fa), composed of three Gaussian peaks, used in this example. C Trajectories of the mutual information between the input distribution p(Fa) and the output distribution π(Fa) as it is optimized via updates to the Wij parameters using the Fletcher-Reeves conjugate gradient algorithm. Random initial conditions in the range [0, 1] for the Wij parameters are used for each trajectory. The dashed lines indicate theoretical upper bounds for the entropies of the discrete distributions {1/3, 2/3} (green) and {1/3, 1/3, 1/3} (purple). The final parameters of the trajectories which best optimized the mutual information for each value of M are used for the plots in panel B.
