Introduction

Physical and biological systems often exhibit relationships between their parts that cannot be reduced to dependencies in subsets of them1. The study of these high-order interdependencies has led to new insights in a wide range of physical systems2,3,4, as well as in studies involving genetics5,6 and neural systems (both biological7,8,9,10,11 and artificial12,13,14), to name a few. Overall, qualitatively different types of interdependence have been found to play complementary roles balancing needs for robustness and flexibility15,16.

There are different approaches to quantify high-order phenomena17, among which we focus on information-theoretic quantities based on Shannon entropy. While there is a rich literature offering various high-order information measures, their interpretation is highly non-trivial—being unclear whether these quantities are capturing the same effects or instead provide complementary perspectives. This lack of clarity makes it challenging for researchers to choose the right tools to carry out specific types of analyses, severely hindering the potential of these measures for practical analyses. Here we address this issue by introducing entropic conjugation as a theoretically grounded principle for exploring the space of possible high-order information measures. Entropic conjugation allows us to show that the existent high-order measures have a closer relationship than previously thought, while revealing gaps in the literature related to high-order interdependencies involving more than five variables. Moreover, our results show how the notions of symmetry and skew-symmetry with respect to entropic conjugation serve as critical indicators ensuring a balanced characterisation of high-order interdependence, which is key to provide a faithful delineation of the high-order profile of physical systems of interest. Our results establish that the O-information occupies a privileged position as the unique skew-symmetric measure whose computational complexity scales linearly with system size. Numerical investigations further reveal that the O-information provides a natural axis of variation among high-order quantities, highlighting its fundamental role in the information-theoretic characterisation of complex systems. The proofs of our results are included in the Methods section.

Results

Measures of multivariate interdependence

Consider a system whose state is specified by the random vector X = (X1, …, Xn) following a joint distribution pX and marginal distributions \({p}_{{X}_{i}}\). The literature presents various approaches to assess the dependencies between parts of X; here we focus on linear combinations of entropies of the form

$$\phi ({{{\boldsymbol{X}}}})= {\sum}_{{{{\boldsymbol{a}}}}\subseteq {I}_{n}}{\lambda }_{{{{\boldsymbol{a}}}}}H({{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{a}}}}}),$$
(1)

where In = {1, …, n}, Xa is the vector of variables whose indices are in a ⊆ In, H(Xa) is the Shannon entropy of Xa, and λa are scalars. In order to focus on metrics of interdependence, we further require these quantities to satisfy two key properties:

  1. (i)

    Labelling-symmetry: ϕ(X) should be invariant to permutations among X1, …, Xn.

  2. (ii)

    Dependency: ϕ(X)  = 0 if the variables are jointly independent (i.e., if \({p}_{{{{\boldsymbol{X}}}}}={\prod }_{k = 1}^{n}{p}_{{X}_{k}}\)).

In this way, property (i) guarantees that ϕ does not depend on how variables are named, and (ii) that it only captures effects of interactions between variables. There are multiple ‘directed’ high-order quantities that distinguish between predictor and target variables, which don’t satisfy labelling-symmetry18,19,20. These quantities will be studied in a follow-up work using a formalism that extends the one presented here.

There are several well-known quantities that satisfy these properties. The oldest of these is the interaction information21, which is defined as

$$\,{\mbox{II}}\,({{{\boldsymbol{X}}}}):= {\sum }_{k=1}^{n}{\left(-1\right)}^{k+1} \mathop{\sum}_{| {{{\boldsymbol{a}}}}| =k}H({{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{a}}}}}).$$
(2)

Other metrics of interdependence are the total correlation (TC)22 and the dual total correlation (DTC)23, which are given by

$${{{\rm{TC}}}}({{{\boldsymbol{X}}}}):= {\sum}_{j=1}^{n}H({X}_{j})-H({{{\boldsymbol{X}}}})\quad {{{\rm{and}}}}$$
(3)
$${{{\rm{DTC}}}}({{{\boldsymbol{X}}}}):= \,H({{{\boldsymbol{X}}}})-{\sum}_{j=1}^{n}H({X}_{j}| {{{{\boldsymbol{X}}}}}^{-\{j\}}),$$
(4)

where H(Xj∣X−a) is the conditional entropy and  − a denotes the set of indices of X that are not in a. Another such metric, well-known in computational neuroscience, is the Tononi-Sporns-Edelman complexity24, which is defined as

$${{{\rm{TSE}}}}({{{\boldsymbol{X}}}}):= {\sum}_{k=1}^{\lfloor n/2\rfloor }{\left(\begin{array}{l}n\\ k\end{array}\right)}^{-1}{\sum}_{| {{{\boldsymbol{a}}}}| =k}I({{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{a}}}}};{{{{\boldsymbol{X}}}}}^{-{{{\boldsymbol{a}}}}}),$$
(5)

where I(Xa; X−a) is Shannon’s mutual information. Finally, we also consider the more recently introduced O-information and S-information25:

$$\Omega ({{{\boldsymbol{X}}}}):= \,(n-2)H({{{\boldsymbol{X}}}})+{\sum}_{j=1}^{n}\left(H({X}_{j})-H({{{{\boldsymbol{X}}}}}^{-j})\right),$$
(6)
$$\Sigma ({{{\boldsymbol{X}}}}):= \,{\sum}_{j=1}^{n}I({X}_{j};{{{{\boldsymbol{X}}}}}^{-j}).$$
(7)

Of these, TC, DTC, TSE, and Σ are non-negative, while II and Ω can take positive and negative values. In the following we show how these seemingly unrelated metrics can be parsimoniously unified under the concept of entropic conjugation.

Characterising high-order interdependencies

We introduce two complementary approaches to assess high-order interdependencies.

The first approach comes from harnessing algebraic properties of linear combinations of entropies, as introduced in ref. 26. To explain this, let us first note that the quantities ϕ described by Eq. (1) form a vector space. Within this space, the condition of labelling-symmetry is satisfied only on an n-dimensional subspace spanned by the quantities

$${r}_{k}({{{\boldsymbol{X}}}}):= \frac{1}{\left(\begin{array}{l}n\\ k\end{array}\right)} {\sum}_{| {{{\boldsymbol{a}}}}| =k}H({{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{a}}}}})\quad {{{\rm{for}}}}\,k=0,1,\ldots ,n.$$
(8)

Informally, rk describes the average amount of information in groups of k variables, with rn = H(X), \({r}_{1}=1/n{\sum }_{j = 1}^{n}H({X}_{j})\), and r0 = 0. Further imposing the condition of dependency adds one linear constraint, reducing this subspace by one dimension. The resulting (n − 1)-dimensional subspace of metrics satisfying both labelling-symmetry and dependency can be shown to be spanned by the negative discrete Laplacian of the above quantities26:

$${u}_{k}:= 2{r}_{k}-{r}_{k+1}-{r}_{k-1}\quad {{{\rm{for}}}}\,k=1,\ldots ,n-1,$$
(9)

which can also be expressed as averages of conditional mutual informations of the form

$${u}_{k}({{{\boldsymbol{X}}}})=\frac{1}{\left(\begin{array}{l}n\\ k+1\end{array}\right)\,\left(\begin{array}{l}k+1\\ 2\end{array}\right)}{\sum}_{\begin{array}{c}i,j\in {I}_{n}\\ i < j\end{array}}\mathop{\sum}_{\begin{array}{c}| {{{\boldsymbol{a}}}}| =k-1\\ i,j\,\notin\,{{{\boldsymbol{a}}}}\end{array}}I({X}_{i};{X}_{j}| {{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{a}}}}}).$$
(10)

Moreover, it can be seen that each uk captures a specific order of interdependencies. To see this, let’s define

$${C}_{{X}_{i},{X}_{j},{{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{a}}}}}}=\left\{p({{{\boldsymbol{x}}}}):p({x}_{i},{x}_{j}| {{{{\boldsymbol{x}}}}}^{{{{\boldsymbol{a}}}}})=p({x}_{i}| {{{{\boldsymbol{x}}}}}^{{{{\boldsymbol{a}}}}})p({x}_{j}| {{{{\boldsymbol{x}}}}}^{{{{\boldsymbol{a}}}}})\right\},$$

so that \(p({{{\boldsymbol{x}}}})\in {C}_{{X}_{i},{X}_{j},{{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{a}}}}}}\) if and only if I(Xi; Xj∣Xa) = 0. Furthermore, define

$${C}_{k}={\bigcap}_{\begin{array}{c}i,j\in {I}_{n}\\ i < j\end{array}}{\bigcap}_{\begin{array}{c}| {{{\boldsymbol{a}}}}| =k-1\\ i,j\,\notin\,{{{\boldsymbol{a}}}}\end{array}}{C}_{{X}_{i},{X}_{j},{{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{a}}}}}},$$
(11)

which corresponds to distributions without interdependencies of order k. Then, it is direct to see that

$${u}_{k}({{{\boldsymbol{X}}}})=0\quad \iff \quad p({{{\boldsymbol{X}}}})\in {C}_{k+1}.$$
(12)

This implies, for instance, that if uk(X) = 0 for k = 1, …, j, then all subsets of j variables or less are statistically independent. Following this rationale, it can be shown that the set {u1, …, un−1} is the unique basis of the space of linear combinations of entropies satisfying labelling-symmetry and dependency that guarantees that, if \(\phi ={\sum }_{k = 1}^{n-1}{c}_{k}{u}_{k}\), then ϕ is non-negative if and only if ck ≥ 0 for k = 1, …, n − 126. In this sense, the resulting coefficient ck can be interpreted as capturing the relevance of (k + 1)-th order dependencies on ϕ.

A complementary and more fine-grained way of investigating high-order interdependence is enabled by partial information decomposition (PID)27, which addresses how information about a variable Y provided by X may be decomposed into the contributions of its different components28,29. PID reveals that while pairwise interdependence is quantified by its strength (measured e.g. by the mutual information), higher-order relationships can be of qualitatively different kinds—most notably redundant (multiple variables sharing the same information) or synergistic (a set of variables holding some information that cannot be seen from any subset). Moreover, PID recognises that synergy and redundancy can be mixed in non-trivial ways, and explores this thoroughly via a formal construction that leads to the decomposition

$$I({{{\boldsymbol{X}}}};{{{\boldsymbol{Y}}}})={\sum}_{{{{\boldsymbol{\alpha }}}}\in {{{{\mathscr{A}}}}}_{n}}{I}_{\partial }^{{{{\boldsymbol{\alpha }}}}}({{{\boldsymbol{X}}}};{{{\boldsymbol{Y}}}}),$$
(13)

where \({{{{\mathscr{A}}}}}_{n}\) is a collection of elements α that cover all possible combinations of redundancy and synergy (see subsection PID conjugation in Methods). For example, if n = 2 then \({{{{\mathscr{A}}}}}_{2}\) has four elements: α1 = {{1}{2}} corresponding to the redundancy between X1 and X2, α2 = {{1, 2}} corresponding to the synergy between them, and α3 = {{1}} and α4 = {{2}} corresponding to unique information in one but not the other. While outlining the relations between the different modes of interdependency, the PID framework itself does not prescribe a specific functional form to calculate these quantities — which has given rise to numerous proposals of how to best define these measures (see e.g. refs. 30,31,32,33,34,35,36,37,38,39,40). More technical details about PID can be found in the subsection PID conjugation in Methods.

A conjugation for Shannon quantities

The notion of ‘conjugation’ is pervasive in mathematics, with examples including the conjugate of a complex number turning z = a + ib into z* = a − ib, and the conjugation of matrices turning a matrix A with entries aij into its Hermitian transpose A* with entries \({a}_{ji}^{* }\). Formally, these examples of conjugation are linear involutions — i.e., linear operations that bring back the original element when applied twice. We now introduce one such conjugation for linear combinations of entropies.

Definition 1

The entropic conjugation is defined as

$${\left(H({{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{a}}}}})\right)}^{* }:= H({{{{\boldsymbol{X}}}}}^{-{{{\boldsymbol{a}}}}})-H({{{\boldsymbol{X}}}}),$$
(14)

with the conjugation of a linear combination of entropies \(\phi ({{{\boldsymbol{X}}}})={\sum}_{{{{\boldsymbol{a}}}}\subseteq {I}_{n}}{\lambda }_{{{{\boldsymbol{a}}}}}H({{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{a}}}}})\) being given by

$${\left(\phi ({{{\boldsymbol{X}}}})\right)}^{* }:= \mathop{\sum}_{{{{\boldsymbol{a}}}}\subseteq {I}_{n}}{\lambda }_{{{{\boldsymbol{a}}}}}{\left(H({{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{a}}}}})\right)}^{* }.$$
(15)

It can be seen that * is linear (by definition) and an involution, as \({({(H)}^{* })}^{* }=H\). Among the possible linear involutions (see the subsection Linear involutions of entropies in Methods), entropy conjugation has special properties related to high-order interdependencies which are outlined by our next results.

Lemma 1

Entropic conjugation acts on the mutual information as

$${\left(I({{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{a}}}}};{{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{b}}}}}| {{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{c}}}}})\right)}^{* }=I\left({{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{a}}}}};{{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{b}}}}}| {{{{\boldsymbol{X}}}}}^{-({{{\boldsymbol{a}}}}\cup {{{\boldsymbol{b}}}}\cup {{{\boldsymbol{c}}}})}\right),$$
(16)

where a, b, c are disjoint subsets of indices. Furthermore, Eq. (16) is equivalent to Def. 1.

The way in which entropic conjugation acts on the mutual information suggests that this operation turns low- into high-order interdependencies: for example, if a = {i}, b = {j}, and \({{{\boldsymbol{c}}}}=\varnothing\), then Lemma 1 states that \({(I({X}_{i};{X}_{j}))}^{* }=I({X}_{i};{X}_{j}| {{{{\boldsymbol{X}}}}}^{-\{i,j\}})\), turning a quantity that depends on the pairwise marginal p(xi, xj) into one that depends on the full distribution p(x). This observation can be generalised, setting the foundations for our analysis of high-order quantities in the next section.

Proposition 1

Entropic conjugation exchanges high- for low-order interdependencies in the following way:

$${\left({u}_{k}({{{\boldsymbol{X}}}})\right)}^{* }={u}_{n-k}({{{\boldsymbol{X}}}}).$$
(17)

A deeper insight on the effect of entropic conjugation can be attained by looking at it via the PID framework. Our next result shows that entropic conjugation is the unique operation that arises from applying the duality principle from order theory41 and category theory42 to PID, which results in a natural conjugation of atoms † that exchanges redundancies for synergies and vice-versa. For example, if α = {{1, 2}} then α† = {{1}, {2}}. For technical details about the conjugation of PID atoms, see the subsection PID conjugation in Methods.

Theorem 1

The natural conjugation of PID atoms † arising from the duality principle satisfies

$$I{\left({{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{a}}}}};{{{\boldsymbol{Y}}}}| {{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{b}}}}}\right)}^{{{\dagger}} }={\sum}_{{{{\boldsymbol{\alpha }}}}\in {{{{\mathscr{A}}}}}_{{{{\boldsymbol{a}}}}}^{{{{\boldsymbol{b}}}}}}{I}_{\partial }^{{{{{\boldsymbol{\alpha }}}}}^{{{\dagger}} }}({{{\boldsymbol{X}}}};{{{\boldsymbol{Y}}}})=I{\left({{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{a}}}}};{{{\boldsymbol{Y}}}}| {{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{b}}}}}\right)}^{* },$$
(18)

where \({{{{\mathscr{A}}}}}_{{{{\boldsymbol{a}}}}}^{{{{\boldsymbol{b}}}}}\) is a collection of atoms such that \(I({{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{a}}}}};{{{\boldsymbol{Y}}}}| {{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{b}}}}})={\sum}_{{{{\boldsymbol{\alpha }}}}\in {{{{\mathscr{A}}}}}_{{{{\boldsymbol{a}}}}}^{{{{\boldsymbol{b}}}}}}{I}_{\partial }^{{{{\boldsymbol{\alpha }}}}}({{{\boldsymbol{X}}}};{{{\boldsymbol{Y}}}})\) (see Lemma 3).

Crucially, this result holds for any operationalisation of synergy and redundancy that is consistent with the PID formalism. This has an important implication: entropic conjugation lets us exchange synergies and redundancies without having to specify a functional form for them.

Let’s illustrate this point with a basic example. Using the fact that the O-information is equal to redundancy minus synergy25, Th. 1 implies that

$${\left(\Omega ({X}_{1};{X}_{2};Y)\right)}^{* }= \, {I}_{\partial }^{{\{\{1\},\{2\}\}}^{{{\dagger}} }}({{{\boldsymbol{X}}}};{{{\boldsymbol{Y}}}})-{I}_{\partial }^{{\{\{1,2\}\}}^{{{\dagger}} }}({{{\boldsymbol{X}}}};{{{\boldsymbol{Y}}}})\\ = \, {I}_{\partial }^{\{\{1,2\}\}}({{{\boldsymbol{X}}}};{{{\boldsymbol{Y}}}})-{I}_{\partial }^{\{\{1\},\{2\}\}}({{{\boldsymbol{X}}}};{{{\boldsymbol{Y}}}})\\ = \, -\Omega ({X}_{1};{X}_{2};Y).$$
(19)

However, note that the same result could have been obtained by simply expressing Ω(X1; X2; Y) in terms of entropies using Eq. (6) and applying Def. 1. The fact that applying entropic conjugation to Ω(X1; X2; Y) gives a minus sign implies that this quantity displays a balanced number of synergy and redundancy terms with opposite signs. This observation leads to a general principle to analyse high-order measures that will be developed in the next section.

Symmetric and skew-symmetric quantities

One of the useful implications of conjugations is that they naturally define symmetric and skew-symmetric objects — e.g., Hermitian transposition defines symmetric and skew-symmetric matrices, and complex conjugation establishes purely real and imaginary numbers. We now use the same principle to introduce the notions of symmetric and skew-symmetric metrics of interdependence.

Definition 2

A linear combination of entropies \(\phi ={\sum }_{{{{\boldsymbol{a}}}}\subseteq {I}_{n}}{\lambda }_{{{{\boldsymbol{a}}}}}H({{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{a}}}}})\) is symmetric if (ϕ)* = ϕ and skew-symmetric if (ϕ)* = − ϕ.

This definition, combined with Prop. 1 and Th. 1, implies that symmetric and skew-symmetric quantities provide balanced accounts of low- and high-order interdependencies (alternatively, redundancies and synergies): symmetric quantities weight these equally, while skew-symmetric quantities weight them equally but with opposite signs. A practical way to recognise symmetric and skew-symmetric high-order metrics is via their weights in terms of the basis uk, as shown next.

Lemma 2

If \(\phi ({{{\boldsymbol{X}}}})={{\sum }_{k=1}^{c}{k}}{u}_{k}({{{\boldsymbol{X}}}})\), then

  • ϕ is symmetric ⇔ ck  = cn−k.

  • ϕ is skew-symmetric ⇔ ck  = − cn−k.

With these tools at hand, we now study the existent high-order quantities under the lens of conjugation.

Proposition 2

The mentioned multivariate measures can be decomposed as follows:

$${{{\rm{TC}}}}({{{\boldsymbol{X}}}})={\sum}_{k=1}^{n-1}(n-k){u}_{k}({{{\boldsymbol{X}}}}),$$
(20)
$${{{\rm{DTC}}}}({{{\boldsymbol{X}}}})= {\sum}_{k=1}^{n-1}k{u}_{k}({{{\boldsymbol{X}}}}),$$
(21)
$${{{\rm{TSE}}}}({{{\boldsymbol{X}}}})={\sum}_{k=1}^{n-1}\frac{k(n-k)}{2}{u}_{k}({{{\boldsymbol{X}}}}),$$
(22)
$$\Sigma ({{{\boldsymbol{X}}}})=n{\sum}_{k=1}^{n-1}{u}_{k}\left.{{{\boldsymbol{X}}}}\right),$$
(23)
$$\Omega ({{{\boldsymbol{X}}}})={\sum}_{k=1}^{n-1}(n-2k){u}_{k}({{{\boldsymbol{X}}}}),$$
(24)
$${{{\rm{II}}}}({{{\boldsymbol{X}}}})={\sum}_{k=1}^{n-1}{\left(-1\right)}^{k+1}\left(\begin{array}{l}n-2\\ k-1\end{array}\right){u}_{k}({{{\boldsymbol{X}}}}).$$
(25)

Therefore, the following relationships hold:

$${\left({{{\rm{TC}}}}({{{\boldsymbol{X}}}})\right)}^{* }={{{\rm{DTC}}}}({{{\boldsymbol{X}}}}),$$
(26)
$${\left(\Sigma ({{{\boldsymbol{X}}}})\right)}^{* }=\Sigma ({{{\boldsymbol{X}}}}),$$
(27)
$${\left({{{\rm{TSE}}}}({{{\boldsymbol{X}}}})\right)}^{* }={{{\rm{TSE}}}}({{{\boldsymbol{X}}}}),$$
(28)
$${\left(\Omega ({{{\boldsymbol{X}}}})\right)}^{* }=-\Omega ({{{\boldsymbol{X}}}}),$$
(29)
$${\left({{{\rm{II}}}}({{{\boldsymbol{X}}}})\right)}^{* }={(-1)}^{n}{{{\rm{II}}}}({{{\boldsymbol{X}}}}).$$
(30)

These results show that the S-information and TSE complexity are balanced metrics of overall interdependence strength, while the O-information provides a balanced opposition between high- and low-order interdependencies. In contrast, the interaction information alternates between being symmetric or skew-symmetric in a way that will be better understood in the next subsection. Additionally, this result also shows that the TC and DTC are not balanced, being duals to each other: the TC provides more weight to low-order effects while the DTC to high-order ones.

These results also reveal that this collection of measures is not as arbitrary as it may seem: when seen from their coefficients ck, they cover the constant (S-information), linear (TC, DTC, and O-information), quadratic (TSE), and binomial (interaction information) cases.

Spanning the possible metrics

Another useful implication of conjugations is that they define a natural decomposition of objects in terms of symmetric and skew-symmetric components. For instance, complex conjugation reveals that every complex number is a sum of a real and an imaginary part, and Hermitian transposition that every matrix is a sum of a symmetric and skew-symmetric components. Building on this rationale, we now show that entropic conjugation induces a decomposition of high-order quantities into symmetric and skew-symmetric components.

Theorem 2

Every information-theoretic metric of interdependence \(\phi ={\sum }_{{{{\boldsymbol{a}}}}\subseteq {I}_{n}}{\lambda }_{{{{\boldsymbol{a}}}}}H({{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{a}}}}})\) can be decomposed into unique symmetric and skew-symmetric components given by

$$\phi =\frac{1}{2}\underbrace{\left(\phi +{\phi }^{* }\right)}_{{{symmetric}}}+\frac{1}{2}\underbrace{\left(\phi -{\phi }^{* }\right)}_{{{skew}}-{{symmetric}}}.$$
(31)

Moreover, symmetric and skew-symmetric components are orthogonal under the inner product induced by \(\langle {u}_{k},{u}_{j}\rangle ={\delta }_{j}^{k}\).

For example, by taking ϕ = TC, Th. 2 states that the total correlation can be expressed as

$${{{\rm{TC}}}}=\frac{1}{2}\Sigma +\frac{1}{2}\Omega ,$$
(32)

where \(\frac{1}{2}\Sigma =\frac{1}{2}({{{\rm{TC}}}}+{{{{\rm{TC}}}}}^{* })\) is its symmetric component and \(\frac{1}{2}\Omega =\frac{1}{2}({{{\rm{TC}}}}-{{{{\rm{TC}}}}}^{* })\) its skew-symmetric component. Similarly, the dual total correlation can be expressed as

$${{{\rm{DTC}}}}=\frac{1}{2}\Sigma -\frac{1}{2}\Omega .$$
(33)

More generally, this result provides a guide to investigate the geometry of the (n − 1)-dimensional space of high-order quantities ϕ satisfying labelling-symmetry and dependency, which we denote by \({{{{\mathscr{I}}}}}_{n}\).

Corollary 1

If X has n variables, then the (n − 1) dimensions of \({{{\mathscr{I}}}}({{{\boldsymbol{X}}}})\) are divided in the following way:

$$\dim \left({{{{\mathscr{I}}}}}_{n}\right)=\underbrace{\lfloor n/2\rfloor }_{{{symmetric}}}+\underbrace{\lfloor (n-1)/2\rfloor }_{{{skew-symmetric}}}.$$
(34)

These results have the following consequences:- For n = 2 variables, Shannon’s mutual information is the only symmetric functional (up to scaling), as there are no skew-symmetric functionals.- For n = 3 variables, the S-information is the only symmetric functional and the O-information (equivalently, the interaction information) is the only skew-symmetric one (up to scaling). Therefore, if \(\phi \in {{{{\mathscr{I}}}}}_{3}\) then ϕ(X) = αΣ(X) + βΩ(X).- For n = 4 variables, the S-information and the interaction information span the subspace of symmetric quantities, while the O-information is the only skew-symmetric one (up to scaling). Therefore, if \(\phi \in {{{{\mathscr{I}}}}}_{4}\) then \(\phi ({{{\boldsymbol{X}}}})=\alpha \Sigma ({{{\boldsymbol{X}}}})+\alpha ^{\prime} {{{\rm{II}}}}({{{\boldsymbol{X}}}})+\beta \Omega ({{{\boldsymbol{X}}}})\).- For n = 5 variables, the space of symmetric quantities is spanned by the S-information and the TSE-complexity, and the space of skew-symmetric quantities is spanned by the O-information and the interaction information. Therefore, if \(\phi \in {{{{\mathscr{I}}}}}_{5}\) then \(\phi ({{{\boldsymbol{X}}}})=\alpha \Sigma ({{{\boldsymbol{X}}}})+\alpha ^{\prime} {{{\rm{TSE}}}}({{{\boldsymbol{X}}}})+\beta \Omega ({{{\boldsymbol{X}}}})+\beta ^{\prime} {{{\rm{II}}}}({{{\boldsymbol{X}}}})\). Larger systems can be analysed in a similar fashion, but the existing measures of high-order information do not cover all the dimensions.

Computational tractability

Most uk (and, therefore, most high-order measures) require estimating a large number of information-theoretic terms, and hence their computation becomes unfeasible when n grows. Our next result narrows down the space of computationally efficient symmetric and skew-symmetric metrics of high-order interdependence.

Proposition 3

The S-information and O-information are the only symmetric and skew-symmetric metrics of interdependence that can be computed via a linear number of entropy terms.

This result may explain, at least in part, the suitability of the O-information for practical analyses of high-order effects — as demonstrated by various empirical investigations43,44,45,46,47. Note that the TC and DTC also require a linear number of terms, but they are neither symmetric or skew-symmetric.

Empirical results

To illustrate the proposed framework, we showcase three examples of increasing complexity: simple multivariate distributions, spin systems, and empirical data from polyphonic music scores. All of these involve estimating multivariate discrete distributions and calculating the values of uk(X) according to Eq. (10). The details of the methods and datasets are described in subsection Numerical analyses in Methods.

We begin analysing a simple multivariate system whose distribution has one degree of freedom, which allows it to display different interdependencies ranging from highly redundant to highly synergistic. Taking inspiration in previous work25,48, we implement this via the so-called W-distribution:

$${p}_{w}({{{\boldsymbol{x}}}})=(1-w){p}_{{{{\rm{copy}}}}}({{{\boldsymbol{x}}}})+w{p}_{{{{\rm{xor}}}}}({{{\boldsymbol{x}}}}),$$
(35)

with x ∈ {0, 1}n being a binary vector of n components and 0 ≤ w ≤ 1 a mixture coefficient. Above, pcopy(x) is a distribution in which all binary components are copies of each other, and pcopy(x) encodes an n-th order XOR (see Numerical analyses in Methods). Based on previous work, it is expected that pw(x) for w ≈ 0 corresponds to a system dominated by redundant information, while for w ≈ 1 it corresponds to a system dominated by synergistic information.

To investigate the interdependencies elicited by the W-distribution by calculating the values of uk(X) (see Fig. 1). Results show that the effect of changes in the mixture coefficient are tracked by the terms corresponding to the lowest- and highest-order interactions, u1(X) and un−1(X), while the other uk(X) for intermediate values of k remain comparatively small. Furthermore, as expected, u1(X) (resp. un−1(X)) is highest when the system is close to a copy (resp. XOR) distribution. This confirms that the quantities uk effectively capture our intuitive notions of low- and high-order statistical structure in these simple systems.

Fig. 1: High-order interdependencies in the W-distribution.
figure 1

Description of the interdependencies within n = 7 binary variables considering a mixture of n identical bits (COPY) and an n-th order exclusive-or (XOR) given by the uk quantities. As expected, the effects of the mixture are tracked by the terms corresponding to the lowest- and highest-order interactions. In the figure, continuous and dotted lines are used for enhancing visual clarity.

Full size image

Next, we build on previous work49,50 and investigate the interdependencies exhibited by small spin systems under weak, ferromagnetic (positive), and frustrated (negative) types of interactions. We implemented these systems by considering Boltzmann distributions with fully-connected random Hamiltonians, and varying the sign of their average coupling (for details, see subsection Numerical analyses in Methods). The frustrated condition makes it impossible to simultaneously satisfy the tendency of all spins to be different from their neighbours, which is known to induce high-order interdependencies17,51.

We again calculated the values of uk, and this time investigated their principal axes of variability via principal component analysis (see Numerical analyses in Methods). Results show that two components explain almost all the variability: a first component of symmetric character similar to the S-information, and a second component of skew-symmetric character similar to the O-information (Fig. 2). This shows that an optimal information-theoretic analysis to characterise the high-order interdependence of these systems reduces to two key aspects: (i) their strength given by the symmetric component, and (ii) the balance between high- and low-order components given by the skew-symmetric component.

Fig. 2: High-order interdependencies in systems of n = 8 spins subject to positive (ferromagnetic), negative (frustrated), and weak interactions.
figure 2

a When calculating uk, each type of interactions exhibit distinct profiles of interdependence. b The variability among uk is captured by two principal components: one of symmetric character which accounts for the overall strength of the interdependence (PC1), and one of skew-symmetric character that accounts for the balance between high- and low-order interdependence (PC2). c The values of uk projected onto these PCs provide a simple characterisation of these three types of systems in terms of their overall interdependence strength (PC1) and the balance between high- and low-order effects (PC2).

Full size image

To further explore the applicability of this framework, we perform similar analyses to empirical data corresponding to polyphonic music scores. We leveraged a rich open dataset and selected a corpus consisting of 522 pieces of choral music from the early Renaissance employing n = 3, 4, or 5 simultaneous voices in a polyphonic manner (for details, see Numerical analyses in Methods). These datasets consider simultaneous singing voices that take discrete values (12 notes plus silence), leading to statistical relationships that often include complex high-order interdependencies25,52.

By repeating the analysis on the principal axes of variability of the values of uk(X), result shows that—similarly to the case of spin systems—most of the variability is captured by the first two components (Fig. 3a). These are again a symmetric and a skew-symmetric component, with the latter one closely resembling the O-information. This suggests that, despite the stark differences between spins and singing voices, an optimal information-theoretic analysis of these systems remains similar to previous case, including an assessment of the strength of interdependencies and a balance between high- and low-order components.

Fig. 3: High-order interdependencies in choral music scores employing n = 3, n = 4, or n = 5 voices.
figure 3

a Resembling the results presented in Fig. 2, the variability among uk is captured mainly by two principal components: one of skew-symmetric character that accounts for the balance between high- and low-order interdependence (PC1), and one of symmetric character that accounts for the overall strength of the interdependence (PC2). b Relationship between O-information, S-information, and interaction information. Results reveal a substantial correlation (∣ρ∣ > 0.5) between symmetric or skew-symmetric high-order measures, which is consistent with the switching nature of the interaction information.

Full size image

As a final analysis, we also calculated the O-information, S-information, and interaction information for each of these music pieces (Fig. 3b). Results show that while the O-information and S-information are largely uncorrelated, there exists a substantial correlation (∣ρ∣ > 0.5) between the interaction information and the S-information when the number of voices is n = 4 (ρ = − 0.52), and the interaction information and the O-information when the number of voices is n = 5 (ρ = − 0.87). Please note that the identity between the interaction information and O-information when n = 3 is expected, given that in that scenario their definitions coincide. These perhaps unexpected correlations can be elegantly explained by our theory, particularly by the fact that the interaction information is symmetric for even values of n and skew-symmetric for odd values of n (see Prop. 2).

Discussion

Here we investigated the space of possible metrics of high-order interdependence taking the form of linear combinations of Shannon entropies. We introduced the notion of entropic conjugation, the effect of which can be understood in two complementary ways: as exchanging how metrics account for high- and low-order interdependencies, or alternatively, how they account for redundancies and synergies. Crucially, while various alternative definitions of synergy and redundancy exist28, the properties of entropy conjugation hold for all approaches that are consistent with the PID formalism irrespective of the specific functional forms used to operationalise these notions.

When studying high-order quantities, non-negative measures such as the S-information and TSE complexity were found to be invariant (i.e. symmetric) under entropic conjugation, confirming that they provide a balanced account of overall interdependence strength. Similarly, applying entropic conjugation to a signed quantities such as the O-information results in a minus sign (i.e. skew-symmetric), guaranteeing that it provides a fair balance of the relative strength of redundancies and synergies. The interaction information was found to be either symmetric or skew-symmetric depending on the number of variables, providing a principled explanation to the observation (first made in Ref. 27) that interpreting this quantity from a high-order perspective requires nuance. This property was found to provide an elegant explanation to otherwise paradoxical empirical associations of the interaction information with other quantities that switched depending on the system’s size.

This framework also let us prove that the well-known high-order information measures cover all possibilities when considering systems of up to n = 5 variables. A similar analysis revealed that, in contrast, the space of possible high-order information measures for capturing interactions involving more than five variables remains largely unexplored. Additionally, the S-information and O-information were found to be the unique symmetric and skew-symmetric quantities whose estimation involves a number of terms that grow linearly with system size, making them ideally suited for practical analyses. Furthermore, numerical studies showed how symmetric and skew-symmetric quantities provide natural axes of variation among high-order quantities. In particular, the O-information was found to naturally arise in the analysis of high-order properties of multivariate systems as dissimilar as Ising spin models and Renaissance choral music. Together, these findings establish the O-information on a privileged position for empirical analyses due to its favourable analytical and phenomenological properties.

Overall, the framework put forward in this work establishes a solid foundation for a formal understanding of high-order information measures, while opening numerous directions for future work along theoretical and practical avenues. For example, while this study focuses on the analysis and interpretation of existent high-order measures, our ongoing work is exploring the potential of this framework to build new quantities tailored via specific desiderata. Additionally, future work may go beyond symmetric quantities to investigate the applicability of entropy conjugation to study directed high-order measures such as the redundancy-synergy index20,53, the dynamic O-information44, or the gradients of O-information50. Also, future work may explore the potential of entropy conjugation to unify pointwise high-order information theoretic quantities54, although in this case the requirement of non-negativity should be treated differently. Furthermore, an interesting challenge would be to extend entropy conjugation to non-Shannon entropies55,56,57 or even to topological notions of high-order phenomena58.

Methods

Numerical analyses

All numerical analyses were done using the Python package dit59. Code for calculating uk(X) for multivariate distributions p(x) is provided (see Data availability).

W-distribution

The W-distribution is a linear mixture of a copy and an xor distribution over a vector X = (X1, …, Xn) of n binary random variables, so that x ∈ {0, 1}n. To formally define it, let us first define the n-bit copy distribution as

$${{p}_{{{{\rm{copy}}}}}}({{{\boldsymbol{x}}}}) = \left\{\begin{array}{cc}0.5 &{{{\rm{if}}}}\,{{x}_{1}} = \cdots = {x}_{n}\,, \hfill \\ 0 \hfill& {{{\rm{otherwise}}}}.\hfill \end{array}\right.$$

Correspondingly, the n-bit XOR distribution is defined as

$${p}_{{{{\rm{xor}}}}}({{{\boldsymbol{x}}}}) = \left\{\begin{array}{cc}{2}^{-n+1} & {{{{\rm{if}}}}}\,{\bigoplus}_{i}{x}_{i}=0 \, \hfill \\ 0 \hfill & {{{\rm{otherwise}}}}.\hfill\end{array}\right.$$

With these, the W-distribution can be directly defined as the following mixture:

$${p}_{w}({{{\boldsymbol{x}}}})=(1-w){p}_{{{{\rm{copy}}}}}({{{\boldsymbol{x}}}})+w{p}_{{{{\rm{xor}}}}}({{{\boldsymbol{x}}}}).$$

Spin systems

Our experiments considered systems of n spins X = (X1, …, Xn) ∈ {−1, 1}n following a Boltzmann distribution pX(x) = e−βH(x)/Z with a Hamiltonian of the form

$$H({{{\boldsymbol{x}}}})=\frac{2}{n(n-1)} {\sum}_{i=1}^{n} {\sum}_{k=i+1}^{n}{x}_{i}{x}_{k}{J}_{ik}\,,$$
(36)

where the coupling coefficients Ji,k are i.i.d. sampled from a Gaussian distribution with mean μ and variance σ2. The results reported in Fig. 1 corresponds to systems of n = 8 spins with β = 1, σ2 = 2, and either μ = 5 (ferromagnetic), μ = 0 (weak), or μ = − 5 (frustrated). We first computed the joint distribution of 10 systems of each type, each with randomly sampled Jik, and calculated the value of uk for each of them.

Music scores

Our analysis is based on the electronic scores publicly available at http://kern.ccarh.org. The analyses focuses on choral repertoire from the early Renassaice (late XV and early XVI centuries). These pieces are characterised by an elaborate counterpoint between melodic lines, resulting in rich textures of balanced richness involving all voices — contrasting with the latter Classic (1730-1820) and Romantic (1780−1910) periods, where higher voices tend to take the lead.

We focused on scores with n = 3, n = 4, or n = 5 melodic lines from four contemporain composes: Josquin des Prez (1450−1521), Johannes Martini (1440−1498), Johannes Ockeghem (1410−1497), and Pierre la Rue (1452−1518). This let us obtain a total of 106 (n = 3), 344 (n = 4), and 72 (n = 5) pieces. The scores were pre-processed in Python using the Music21 package (http://web.mit.edu/music21). The melodic lines were transformed into time series of 13 possible discrete values (one for each note plus one for the silence), using the smallest rhythmic duration as time unit. In order to focus on polyphonic harmony, chords with 40% or more silent voices were dropped. With these data, the joint distribution of the values for the chords was estimated using their empirical frequency. We estimated a joint distribution for each piece, and then calculated the value of uk.

Principal component analysis

Our analysis pipeline for both the spin system and the music scores was structured as follows. In both cases, the resulting uk values were then used to perform a principal component analysis. From this, we obtained the loadings of the two first principal components, denoted as ξk and νj, respectively. For the analysis of spin systems, these loadings were then used to construct two high-order quantities: \({\phi }_{{{{\rm{PC}}}}1}({{{\boldsymbol{X}}}}):= {\sum }_{k = 1}^{n-1}{\xi }_{k}{u}_{k}({{{\boldsymbol{X}}}})\) and \({\phi }_{{{{\rm{PC}}}}2}({{{\boldsymbol{X}}}}):= {\sum }_{j = 1}^{n-1}{\nu }_{j}{u}_{j}({{{\boldsymbol{X}}}})\), which corresponds to projecting the value of the uk’s onto the directions given by the principal components. These two resulting metrics were used to characterise the systems of interest.

Short proofs

In the following, we present the proofs of our theoretical results.

Proof

(Proof of Lemma. 1) Let us first prove Eq. (16). For this, consider a, b, c to be disjoint subsets of indices. Then,

$${\left(I({{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{a}}}}};{{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{b}}}}}| {{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{c}}}}})\right)}^{* }=\, \left(H({{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{a}}}}},{{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{c}}}}})+H({{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{b}}}}},{{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{c}}}}})\right.\\ -{\left.H({{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{c}}}}})-H({{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{a}}}}\cup {{{\boldsymbol{b}}}}\cup {{{\boldsymbol{c}}}}})\right)}^{* }\\ =\, H({{{{\boldsymbol{X}}}}}^{-{{{\boldsymbol{a}}}}\cup {{{\boldsymbol{c}}}}})+H({{{{\boldsymbol{X}}}}}^{-{{{\boldsymbol{b}}}}\cup {{{\boldsymbol{c}}}}})\\ -H({{{{\boldsymbol{X}}}}}^{-{{{\boldsymbol{c}}}}})-H({{{{\boldsymbol{X}}}}}^{-{{{\boldsymbol{a}}}}\cup {{{\boldsymbol{b}}}}\cup {{{\boldsymbol{c}}}}})\\ =\, I({{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{a}}}}};{{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{b}}}}}| {{{{\boldsymbol{X}}}}}^{-{{{\boldsymbol{a}}}}\cup {{{\boldsymbol{b}}}}\cup {{{\boldsymbol{c}}}}}).$$

where the last equality uses the fact that I(A; B∣C)  = H(A, C) + H(B, C) − H(C) − H(A, B, C) with A = Xa, B = Xb, and C = X−a∪b∪c.

Let us now show that Eq. (16) is equivalent to Definition 1. For this, let’s assume that * is a linear operation defined on the mutual information as in Eq. (16). Then, using \({{{\boldsymbol{c}}}}=\varnothing\) one can find that

$${(I({{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{a}}}}};{{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{b}}}}}))}^{* }=I({{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{a}}}}};{{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{b}}}}}| {{{{\boldsymbol{X}}}}}^{-{{{\boldsymbol{a}}}}\cup {{{\boldsymbol{b}}}}}),$$
(37)

which by expanding both sides in terms of entropies yields

$$ {\left(H({{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{a}}}}})\right)}^{* }+\left({H}^{* }{\left({{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{b}}}}}\right)}^{* }-\left({H}^{* }{\left({{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{a}}}}\cup {{{\boldsymbol{b}}}}}\right)}^{* }\right.\right.\\ =H({{{{\boldsymbol{X}}}}}^{-{{{\boldsymbol{a}}}}})+H({{{{\boldsymbol{X}}}}}^{-{{{\boldsymbol{b}}}}})-H({{{{\boldsymbol{X}}}}}^{-{{{\boldsymbol{a}}}}\cup {{{\boldsymbol{b}}}}})-H({{{\boldsymbol{X}}}}).$$

Then, by fixing a and varying b one can note that this implies that

$${\left(H({{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{a}}}}})\right)}^{* } = \, H({{{{\boldsymbol{X}}}}}^{-{{{\boldsymbol{a}}}}})-C,\\ {\left(H({{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{b}}}}})\right)}^{* } = \, H({{{{\boldsymbol{X}}}}}^{-{{{\boldsymbol{b}}}}})-C,\\ {\left(H({{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{a}}}}\cup {{{\boldsymbol{b}}}}})\right)}^{* } = \, H({{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{a}}}}\cup {{{\boldsymbol{b}}}}})-C,$$

where C is a quantity that does not depend on a or b. Moreover, it is needed that C = H(X), proving the equivalence.

Proof

(Proof of Prop. 1) To prove Eq. (17), we first focus on rk as defined in Eq. (8) and note that

$${{r}_{k}}^{*}=H({{{\boldsymbol{X}}}})-\frac{1}{\left(\begin{array}{l}n\\ k\end{array}\right)}\mathop{\sum}_{| {{{\boldsymbol{a}}}}| =k}H({{{{\boldsymbol{X}}}}}^{-{{{\boldsymbol{a}}}}})={r}_{n}-{r}_{n-k}.$$
(38)

Then, using the definition in Eq. (9), one can find that

$${({{u}_{k}})}^{*} = \, {(2{{r}_{k}})}^{*}-{({{r}_{k+1}})}^{*}-{({r}_{k-1})}^{* }\\ = \, 2{{r}_{n-k}}-{{r}_{n-k+1}}-{{r}_{n-k-1}}={{u}_{n-k}}.$$

Proof

(Proof of Lemma 2) The proof follows directly from Definitions 1 and 2.

Proof

(Proof of Prop. 2) The expressions for each metric can be directly verified by applying the definition of uk in terms of rk from Eq. (9), and expressing rk in terms of entropies using Eq. (8). The second part of the proposition follows by using Lemma 2.

Proof

(Proof of Th. 2) Let’s denote as S = (ϕ + ϕ*)/2 and T = (ϕ − ϕ*)/2 the components proposed in the proposition, which can be directly shown to be symmetric and skew-symmetric and satisfying ϕ = S + T. Let’s assume there is another decomposition \(\phi ={S}^{{\prime} }+{T}^{{\prime} }\) where \({({S}^{{\prime} })}^{* }={S}^{{\prime} }\) and \({({T}^{{\prime} })}^{* }=-{T}^{* }\). However this would imply that \(\phi +{\phi }^{* }=2{S}^{{\prime} }\) and \(\phi -{\phi }^{* }=-2{T}^{{\prime} }\), which leads to \(S={S}^{{\prime} }\) and \(T={T}^{{\prime} }\), showing that the decomposition is unique. The orthogonality of these subspaces follows directly from Lemma 2.

Proof

(Proof of Prop. 3) Consider \(\phi ={\sum }_{k = 1}^{n-1}{c}_{k}{u}_{k}\). Using Eq. (9) one can re-write ϕ in terms of rk. Since every entropy term appears only in exactly one of the rk, there cannot be any term cancellations. Therefore, ϕ involves a linear number of (unconditional) entropy terms if and only if the resulting coefficients are non-zero only for r1, rn−1, and rn.

Let us first consider the case in which ck = αk + β is linear on k. Then, one can show that

$$\phi =-\mathop{\sum }_{k=1}^{n-1}{c}_{k}{\Delta }^{2}{r}_{k}=-\alpha \mathop{\sum}_{k=1}^{n-1}k{\Delta }^{2}{r}_{k}-\beta {\sum}_{k=1}^{n-1}{\Delta }^{2}{r}_{k}.$$
(39)

By using the fact that Δ2rk = Δrk+1 − Δrk where Δrk ≔ rk − rk−1, one can use discrete calculus to find that

$${\sum}_{k=1}^{n-1}{\Delta }^{2}{r}_{k}=\Delta {r}_{n}-\Delta {r}_{1}={r}_{n}-{r}_{n-1}-{r}_{1},$$
(40)
$${\sum}_{k=1}^{n-1}k{\Delta }^{2}{r}_{k} = \, (n-1)\Delta {r}_{n}- {\sum}_{k=1}^{n-1}\Delta {r}_{k}^{n}\\ = \, (n-1){r}_{n}-n{r}_{n-1}.$$
(41)

This calculation leads to \(\phi =\beta {r}_{1}^{n}+(\alpha n+\beta ){r}_{n-1}^{n}-(\alpha (n-1)+\beta ){r}_{n}^{n}\), showing that ϕ only includes a linear number of entropies.

To prove the converse statement, let’s consider a quantity ϕ = ∑kckuk, and let’s denote its coefficients under rk and Δrk as ak and bk respectively, so that ϕ = ∑iairi = ∑jbjΔrj hold. As the rk are linearly independent, one can see that the above equations imply that the following conditions hold for all k = 1, …, n − 2:

$${a}_{k}={b}_{k}-{b}_{k+1}\quad {{{\rm{and}}}}\quad {b}_{j}={c}_{j}-{c}_{j+1}.$$
(42)

Now, note that ak = 0 with k ∈ {2, …, n − 2} requires that bk = bk+1, and hence the above condition forces b1 = … = bn−1. Then, applying the same reasoning shows that ck − ck+1 has to be constant, which proves that ck depends linearly on k.

The above shows that the space of quantities that can be computed with a linear number of terms is two-dimensional, being spanned by DTC (α = 1, β = 0) and the S-information (α = 0, β = n). This space is also spanned by the S-information and the O-information, concluding the proof.

PID conjugation

Here we provide a more detailed introduction to PID, and then introduce the natural conjugation that takes place at the level of PID atoms.

Following Eq. (13), PID introduces a decomposition of the mutual information I(X; Y) in terms of information atoms of the form \({I}_{\partial }^{{{{\boldsymbol{\alpha }}}}}({{{\boldsymbol{X}}}};{{{\boldsymbol{Y}}}})\), where α = {α1, …, αl} with αj ⊆ In being sets of indices of the source variables X1, …, Xn such that no αj contains another αk — making α an ‘anti-chain’ of sets of sources. Conceptually, \({I}_{\partial }^{{{{\boldsymbol{\alpha }}}}}({{{\boldsymbol{X}}}};{{{\boldsymbol{Y}}}})\) quantifies the information about the target variable Y that is accessible via each collection of source variables α1, …, αl, while not being accessible via subsets of those collections or any other collections that do not include them. For example, if n = 2 then \({I}_{\partial }^{\{\{1,2\}\}}({{{\boldsymbol{X}}}};{{{\boldsymbol{Y}}}})\) corresponds to the information accessible in (X1, X2) but not accessible from X1 or X2 by themselves.

The accessibility relations denoted by PID antichains can be made explicit by ‘re-representing’ them in terms of Boolean functions \(f:{{{{\mathscr{P}}}}}_{n}\mapsto \{0,1\}\), where \({{{{\mathscr{P}}}}}_{n}\) is the powerset of {1, …, n}, taking a set of source indices as an input and returning 0 or 1 depending on whether the associated atom \({I}_{\partial }^{f}({{{\boldsymbol{X}}}};{{{\boldsymbol{Y}}}})\) is or isn’t accessible via the set of sources. For example, the atom α = {{1, 2}} corresponds to the Boolean function that gives \(f(\varnothing )=f(\{1\})=f(\{2\})=0\) and f({1, 2}) = 1. Crucially, it has been shown that there is a natural isomorphism between PID antichains and monotonic Boolean functions60,61, which implies that PID can be re-defined as follows.

Definition 3

A PID of the information provided by X = (X1, …, Xn) about Y is a set of quantities \({I}_{\partial }^{f}({{{\boldsymbol{X}}}};{{{\boldsymbol{Y}}}})\) that satisfy for all a ⊆ {1, …, n}

$$I({{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{a}}}}};{{{\boldsymbol{Y}}}})={\sum}_{\begin{array}{c}f\in {{{{\mathscr{B}}}}}_{n}\\ f({{{\boldsymbol{a}}}})=1\end{array}}{I}_{\partial }^{f}({{{\boldsymbol{X}}}};{{{\boldsymbol{Y}}}}),$$
(43)

where \({{{{\mathscr{B}}}}}_{n}\) is the set of all non-constant monotonic Boolean functions \(f:{{{{\mathscr{P}}}}}_{n}\mapsto \{0,1\}\).

Note that Eq. (43) is equivalent to Eq. (13), with the only difference being the way in which PID atoms are labelled (either as antichains or Boolean functions). That said, viewing PID in terms of Boolean functions lets us conveniently handle various expressions, as shown below.

Lemma 3

Given two disjoint sets of source variables a and b, we have

$$I({{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{a}}}}};{{{\boldsymbol{Y}}}}| {{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{b}}}}})= {\sum}_{f\in {{{{\mathscr{B}}}}}_{{{{\boldsymbol{a}}}}}^{{{{\boldsymbol{b}}}}}}{I}_{\partial }^{f}({{{\boldsymbol{X}}}};{{{\boldsymbol{Y}}}}),$$
(44)

where \({{{{\mathscr{B}}}}}_{{{{\boldsymbol{a}}}}}^{{{{\boldsymbol{b}}}}}=\{f\in {{{{\mathscr{B}}}}}_{n}:f({{{\boldsymbol{a}}}}\cup {{{\boldsymbol{b}}}})=1,f({{{\boldsymbol{b}}}})=0\}\).

Note that the set \({{{{\mathscr{A}}}}}_{{{{\boldsymbol{a}}}}}^{{{{\boldsymbol{b}}}}}\) used in Th. 1 corresponds to the same atoms in \({{{{\mathscr{B}}}}}_{{{{\boldsymbol{a}}}}}^{{{{\boldsymbol{b}}}}}\) but represented in antichain form instead of as Boolean functions.

Proof

$$ I({{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{a}}}}};{{{\boldsymbol{Y}}}}| {{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{b}}}}}) = I({{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{a}}}}},{{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{b}}}}};{{{\boldsymbol{Y}}}})-I({{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{b}}}}};{{{\boldsymbol{Y}}}})\\ = {\sum}_{\begin{array}{c}f\in {{{{\mathscr{B}}}}}_{n}\\ f({{{\boldsymbol{a}}}}\cup {{{\boldsymbol{b}}}})=1\end{array}}{I}_{\partial }^{\,f}({{{\boldsymbol{X}}}};{{{\boldsymbol{Y}}}})- {\sum}_{\begin{array}{c}f\in {{{{\mathscr{B}}}}}_{n}\\ f({{{\boldsymbol{b}}}})=1\end{array}}{I}_{\partial }^{\,f}({{{\boldsymbol{X}}}};{{{\boldsymbol{Y}}}}),$$
(45)

The information atoms have a natural order in terms of their accessibility: an atom \({I}_{\partial }^{f}({{{\boldsymbol{X}}}};{{{\boldsymbol{Y}}}})\) can be said to be ‘more accessible’ than an atom \({I}_{\partial }^{g}({{{\boldsymbol{X}}}};{{{\boldsymbol{Y}}}})\) if any set via which the latter is accessible is also a set via which the former is accessible. This property is elegantly captured by the Boolean function representation of atoms via the following partial ordering:

$$f \sqsubseteq g\;{{{\rm{if}}}}\,{{{\rm{and}}}}\,{{{\rm{only}}}}\,{{{\rm{if}}}}\;f({{{\boldsymbol{a}}}})\le g({{{\boldsymbol{a}}}})\,\,\forall {{{\boldsymbol{a}}}}\in {{{{\mathscr{P}}}}}_{n}.$$
(46)

Note that this partial ordering gives rise to a lattice of PID atoms denoted by \(({{{{\mathscr{B}}}}}_{n}, \sqsubseteq )\), which is isomorphic to the original PID lattice of antichains61.

Let us now introduce the notion of PID conjugation. For this, let us first note that, according to the order-theoretic principle of duality41, every lattice has a dual lattice in which all arrows are reverted. If we think of Boolean functions as bitstrings (with subsets ordered lexicographically), the dual of the PID lattice can be found by simply inverting the digits and reading the bitstring backwards, as shown by our next result.

Proposition 4

The mapping f ↦ f†, where f† is the Boolean function satisfying

$$\forall {{{\boldsymbol{a}}}}\subseteq \{1\ldots ,n\}:{f}^{{{\dagger}} }({{{\boldsymbol{a}}}})=1\iff f({{{{\boldsymbol{a}}}}}^{C})=0,$$
(47)

is an order-reversing involution on \(({{{{\mathscr{B}}}}}_{n}, \sqsubseteq )\).

Proof

Involution: f††(a) = 1 if and only if \(f({({{{{\boldsymbol{a}}}}}^{C})}^{C})=f({{{\boldsymbol{a}}}})=1\). Hence, f  †† = f. Order-reversing: Suppose first that f ⊑ g, meaning that g(a) = 1 → f(a) = 1. Then, if f †(a) = 1 we must have f(ac) = 0 and so g(ac);= 0 and hence g†(a) = 1. Therefore we have g† ⊑ f†. Conversely, if g† ⊑ f†, then f †(a) = 1 → g†(a) = 1. Hence, if g(a) = 1 it follows that g†(ac) = 0 and thus f †(ac) = 0 and thus f(a) = 1. Therefore, f ⊑ g.

The effect of † on the PID lattice can be understood as follows (see Fig. 4). Following Ref. 62, each atom can be expressed as concatenations of meet and join operations corresponding to redundancies and synergies. For example, the atom α = {{1, 2}, {1, 3}} can be constructed as (1 ∨ 2) ∧ (2 ∨ 3), where the join operation ( ∨ ) can be thought of as denoting the union between sources (i.e., synergy), and the meet operation ( ∧ ) as the intersection between them (i.e., redundancy). Then, the involution introduced in Proposition 4 can be understood as switching meets for joins and vice-versa. For example

$${\{\{1,2\},\{1,3\}\}}^{{{\dagger}} }\, = {((1\vee 2)\wedge (2\vee 3))}^{{{\dagger}} }\\ \, = (1\wedge 2)\vee (1\wedge 3)\\ {=}^{(a)} 1\vee (2\wedge 3)=\{\{1\},\{2,3\}\},$$

where equality (a) uses the distributivity between meets and joins62. This shows that the natural PID involution switches redundancy (i.e. easily accessible information, since it is contained in multiple sources) and synergy (i.e. difficult to access information, since one needs to observe multiple sources). Thus, the more easily an atom can be accessed, the more difficult it is to access its conjugate.

Fig. 4: Effect of conjugation on the PID lattice.
figure 4

The conjugation of PID atoms is based on the involution † applied to information atoms. The effect of this involution is to flip the lattice around its central axis.

Full size image

This PID involution leads to a natural conjugation between PID atoms, which we define next.

Definition 4

The conjugate of a PID atom is given by

$${\left({I}_{\partial }^{f}({{{\boldsymbol{X}}}};{{{\boldsymbol{Y}}}})\right)}^{{{\dagger}} }:= {I}_{\partial }^{{f}^{{{\dagger}} }}({{{\boldsymbol{X}}}};{{{\boldsymbol{Y}}}}),$$
(48)

where f† is as defined in Prop. 4. Additionally, the conjugate of a linear combination of PID atoms \(\psi ({{{\boldsymbol{X}}}};{{{\boldsymbol{Y}}}})={\sum}_{f\in {{{{\mathscr{B}}}}}_{n}}{c}_{f}{I}_{\partial }^{f}({{{\boldsymbol{X}}}};{{{\boldsymbol{Y}}}})\) is defined as

$${\left(\psi ({{{\boldsymbol{X}}}};{{{\boldsymbol{Y}}}})\right)}^{{{\dagger}} }:= {\sum}_{f\in {{{{\mathscr{B}}}}}_{n}}{c}_{f}{I}_{\partial }^{{f}^{{{\dagger}} }}({{{\boldsymbol{X}}}};{{{\boldsymbol{Y}}}}).$$
(49)

We now prove Theorem 1, which states that applying PID conjugation on a conditional mutual information I(Xa; Y∣Xb) leads to the same outcome as entropic conjugation does — namely, \(I({{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{a}}}}};{{{\boldsymbol{Y}}}}| {{{{\boldsymbol{X}}}}}^{{({{{\boldsymbol{a}}}}\cup {{{\boldsymbol{b}}}})}^{C}})\), where the complement is taken within the set of source variables.

Proof

(Proof of Th. 1) For a, b disjoint subsets of In, then

$$I \left({{\boldsymbol{X}}}^{{{\boldsymbol{a}}}};{{{\boldsymbol{Y}}}}|{{{\boldsymbol{X}}}}^{{{\boldsymbol{b}}}}\right)^{{\dagger}} {=}^{({\mbox{lemma }}\,3)} \left(\sum\limits_{f({{{{\mathbf{a}}}}}\cup {{{{\mathbf{b}}}}}) = 1 \& f({{{{\mathbf{b}}}}})=0} I_\partial^f({{{{\boldsymbol{X}}}}};{{{{\boldsymbol{Y}}}}}) \right)^{{\dagger}} \\ {=}^{({\mbox{def. }}\,4)} \sum\limits_{f({{{{\mathbf{a}}}}}\cup {{{{\mathbf{b}}}}}) = 1 \& f({{{{\mathbf{b}}}}})=0} I_{\partial}^{f^{{\dagger}}}({{{{\boldsymbol{X}}}}};{{{{\boldsymbol{Y}}}}}) \\ {=}^{({{\mbox{def. }}}\,f^{\dagger})} \sum\limits_{f(({{{{\mathbf{a}}}}}\cup {{{{\mathbf{b}}}}})^C) = 0 \& f({{{{\mathbf{b}}}}}^C)=1} I_\partial^f({{{{\boldsymbol{X}}}}};{{{{\boldsymbol{Y}}}}}) \\ {=} \sum\limits_{f({{{{\mathbf{a}}}}}\cup \left({{{{\mathbf{a}}}}} \cup {{{{\mathbf{b}}}}}\right)^C)=1 \& f(({{{{\mathbf{a}}}}}\cup {{{{\mathbf{b}}}}})^C) = 0} I_\partial^f({{{{\boldsymbol{X}}}}};{{{{\boldsymbol{Y}}}}}) \\ {=}^{({\mbox{lemma }}\,3)} I({{{{\boldsymbol{X}}}}}^{{{{{\boldsymbol{a}}}}}};{{{{\boldsymbol{Y}}}}}|{{{{\boldsymbol{X}}}}}^{\left({{{{\boldsymbol{a}}}}} \cup {{{{\boldsymbol{b}}}}}\right)^C}),$$

where the second to last equality follows because a ∪ (a∪b)C = bC if a and b are disjoint.

Please note that the proof uses no properties specific to particular instantiations of synergy or redundancy, and hence the result holds for any operationalisation of these quantities that are consistent with the PID framework.

Linear involutions of entropies

When considering all linear combinations of entropies of the form of Eq. (1) as a vector space spanned by the functions H(Xa), linear involutions can be characterised as linear operators T with the property T2 = I, with I being the identity operator. This implies that there are many linear involutions — for example, any operation exchanging between basis elements reciprocally is one such operation (e.g. \({(H({{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{a}}}}}))}^{{\prime} }=H({{{{\boldsymbol{X}}}}}^{-{{{\boldsymbol{a}}}}})\)). Other linear involutions include \({(H({{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{a}}}}}))}^{{\prime\prime} }=H({{{\boldsymbol{X}}}})-H({{{{\boldsymbol{X}}}}}^{-{{{\boldsymbol{a}}}}})\), and also the negentropy (\({(H({{{{\boldsymbol{X}}}}}^{{{{\boldsymbol{a}}}}}))}^{{\prime\prime} {\prime} }=\mathop{\sum }_{i = 1}^{n}\log | {{{{\mathscr{X}}}}}_{i}| -H({{{{\boldsymbol{X}}}}}^{-{{{\boldsymbol{a}}}}})\), defined for variables over finite alphabets \({{{{\mathscr{X}}}}}_{i}\) with i = 1, …, n) provides a linear involution.

Crucially, just as there are many linear involutions over complex numbers but complex conjugation is highlighted for its special properties, in this paper we focus on the proposed entropy conjugation for its unique properties pertaining high order effects. In particular, as outlined by Lemma 1, Prop. 1, and Th. 1, entropic conjugation is the only linear involution that yields a simple exchange between the quantities uk of different orders as in Eq. (17) (other variants would make the relationship more complicated, e.g. introducing a minus sign, etc) and also an exchange between redundancies and synergies in PID atoms.