Parametric action of homomorphic image of modular group and it’s application in image encryption

Abstract

In this article coset diagrams of the action of PSL(2, Z) on a $PL(F_p)$ are obtained, through parametrization, which yields one of the eight finite generalized triangle groups which are homomorphic images or quotients of PSL(2, Z). Other than this we analyzed the coset diagrams for the parameter for three finite generalized triangle groups. One of the most dependable methods for achieving data security has been the block cipher. S-Boxes constructed using algebraic structure have gained popularity recently because of their advantageous cryptographic properties and high non-linearity have been found in these structures, which attract researchers. With the help of these parametrized actions, a novel algebraic method to create $2^8$ S-Boxe was established. The S-Box provides strong cryptographic qualities of nonlinearity 112, differential uniformity 6, linear approximation probability of 0.0576 and differential attack probability of 0.0039. To assess the practical applicability of our S-box, we integrate it into an image encryption scheme and present experimental results to showcase its efficacy in real-world scenarios. When used with an image encryption framework, the following results were obtained: NPCR = 0.9959, UACI = 0.3348, and approx. Entropy 7.98. Therefore, GTG based parametrisation has been shown to be an effective and secure alternative to traditional algebraic construction method for S-Boxes.

Introduction

Information security has grown to be a major hot topic since network connectivity and big data applications have developed so quickly. The most crucial method of defense. Data encryption is information. As a result, the foundational technology for information security is cryptography. Block encryption methods are often employed in contemporary cryptographic systems. However, because to the huge quantity of data contained in photographs, previous encryption techniques have trouble encrypting them. The substitution box (S-box) is a critical component in various symmetric key encryption algorithms. It transforms a fixed-length input (usually a byte) into a different output, providing confusion and diffusion properties essential for secure encryption. The design of efficient and secure S-boxes is of great interest in the field of cryptography. S-Boxes (substitution boxes) have a significant role in the security performance of a block cipher system being the sole nonlinear component. AES is often regarded as being an efficient cryptosystem. The S-Box of AES, which is based on the inversion and affine translation of $\mathbb{G}\mathbb{F}(2^8)$ elements, is one of its key components. S-Box has caught the interest of authors of cryptosystems due to the widespread usage of AES in communication systems. Nonlinear modifications are used to change the input data, which increases the risk of cyberattacks by making the input more complicated and difficult to reverse. S-boxes are useful in a variety of cryptographic applications, such as symmetric key algorithms, block ciphers, and hashing algorithms. S-box design is a complex process that calls for careful evaluation regarding execution constraints, security, and efficiency. S-box development has been approached from a variety of approaches, including heuristic and mathematical methods. To fend against various cryptographic assaults, an S-box must exhibit strong cryptographic requirements. According to the SP network theory, all conventional current block ciphers, including AES, DES, and IDEA, incorporate nonlinear components to make the relationship between the original and encrypted text as complicated as feasible. In 1998, A. Yamamura introduced the $SL(2,\mathbb {C}[X])$ as the basis for a public key cryptosystem (see¹). Another public key cryptosystem employing the modular group was his new suggestion in 1999, where one complex number serves as the ciphertext representation (see²). R. Steinwandt demonstrated in³ that both of A. Yamamura’s proposed cryptosystems were susceptible to ciphertext-only attacks.⁴ offers a number of recommendations for creating cryptosystems that use linear groups and combinatorial group theory. A S-Box building approach based on group strutures was presented in^5,6,7.

Baumslag et al., first studied generalized triangle groups in⁸. Discrete generalized triangular groups were the focus of study by M. Hagelberg et al., in 1995. Beniash-Kryvets talked about several generalized triangle groups’ free subgroups. In 2000, Vinberg, E. et al., provided an explanation of the Pseudo-Finite generalized triangle groups. The high power relator of the finite generalized tetrahedron groups is provided by M. Edjvet et al. A very comprehensive classification of those finite groups is provided by J. Howie et al. For a finite generalized triangle group $\delta$ and special representation $\gamma$, the homomorphic image $\gamma (G)$ is a finite subgroup of $PSL(2,\mathbb {C})$ which is generated by two elements. The finite list of possible values for $tr(\gamma (ab))$, that is, roots for the trace polynomial of these subgroups have been studied in depth⁹. In¹⁰ the modular and Hecke groups have been parameterized by evaluating a subgroup of $SL(2,\mathbb {Z}[\xi ])$, providing parameterized modular group. The authors in¹¹ have taken it to a further step by introducing generalization of the parameterized modular group.

In the past few years, a number of new ways to design S-boxes have been developed by combining several different mathematical, chaos theory, heuristics, and structural approaches, with the goal of improving security in modern ciphers, particularly for images and multimedia. For example, a team of researchers led by Razaq created a fuzzy-logic-based S-box that increases the resiliency of the resulting encryption of medical images by providing a non-linear model of the nature of the image¹². In addition, Alali developed an S-box that is created dynamically using Mordell elliptic curves over finite fields, and it exhibits excellent cryptographic capabilities for protecting images¹³. Aydin created a high-security S-box, leveraging the use of FPGA implementation with random numbers based on SHA-256, which generate new S-boxes¹⁴. Safdar employed an algebraic structure in constructing S-boxes using nonlinear components over non-chain semi-local rings, further extending the techniques used for encrypting color images¹⁵. Furthermore, Shahzad produced S-boxes using the quotient of the modular group of modular group actions, exhibiting strong resistance against cryptanalysis¹⁶. Chaotic-algebraic hybrid methodologies are still evolving; for example, Haider utilized new pseudo-random number generators that exhibit strong diffusion and confusion properties for constructing S-boxes¹⁷, while Zahid and Arshad employed cubic polynomial mappings as models for non-linear substitution layers¹⁸. Designers of S-boxes have referred to a wide range of current literature and utilized chaotic dynamics (or degrees of chaotic behavior) in their work. For example, Lambic¹⁹ developed an algorithm for constructing S-boxes using discrete chaotic maps of integers, while Ge and Li²⁰ constructed a fractal-chaotic dynamic S-box for scrambling images. More specifically, some of the latest studies and research have focused on modifying pre-existing S-box designs and applying new algorithms, such as those developed by Alamsyah et al., who used a variety of affine matrices to generate an increase in strength for the AES S-boxes²¹. Additionally, researchers have combined both optimization techniques with hybrid chaotic systems to produce a more robust S-box. In this vein, Naz et al. developed a method that leverages the concepts of hypothesis testing and hybrid chaotic maps to increase the effectiveness of S-boxes²²; and Jawed and Sajid, in their proposal for COBLAH, present a chaotic initialization (OBL) heuristic method for the optimal creation of S-boxes through algebraic means²³. Similarly, Ullah et al. created an S-box that is constructed from a fractional-order hyperchaotic four-dimensional system and is used in conjunction with an RSA-based method for encryption of color images; they demonstrated that their S-box has a very high level of robustness against both statistical and differential attacks²⁴. The increasing trend to incorporate algebraic structural properties, as well as principles of chaotic behavior, into cryptographic S-box design provides a strong foundation for future efforts to develop dynamic, highly nonlinear, and resilient S-boxes for use in modern cryptography.

There are structural properties associated GTG’s (generalized triangle groups) over classical group structures give them advantages as the basis for constructing cryptographic substitution boxes (S-boxes). In contrast to classical groups (cyclic, dihedral, affine, etc.) that produce very symmetric and predictable pattern formations within their permutations, GTG’s have more complex group presentations because of the presence of higher-order relations and greater complexity in their subgroup structure. The end result is that the conjucted classes created by GTG’s will be greater and more diverse than those for the classical type of groups, so the permutation cycles they create will be of varying lengths, with sparce symmetrical elements, and without repeated methods of branching from one permutation to another. Since the GTG acts on projective lines, it creates many deeper orbits resulting in greater levels of confusion, while the parameterization of the GTG allows for the generation of many non-isomorphic coset diagrams due to very small changes in the algebraic structure of the GTG. This diverse structure creates a large space of S-boxes that can be created; therefore, it lowers the amount of exploitable linear structures and improves the S-boxes created to be more resistant to algebraic, linear, and differential attacks than S-boxes created from classical algebraic groups.

Problem statement

In the last few decades, more study has been done on image encryption. The disadvantages of conventional encryption systems include considerable computational and transmission costs. Moreover, data encryption may result in data loss. Many methods have been suggested and devised to safeguard sensitive data, including photos and data, but storage capacity, speed, and effectiveness are important considerations that must be given enough attention. Moreover, the dispersion effect, cascading effect, unpredictability, as well as sensitivity, must be taken into consideration by based on images encryption systems. These techniques offer lots of key space as well as high key sensitivity. Its security must be strengthened, though. While some researchers utilise the anticipated correlation coefficient to determine a secure image encryption technique, numerous others depend on entropy as a requirement for an effective encryption process. Nonetheless, for a method to encrypt images securely and effectively, both of the parameters are essential. There are key characteristics shared by algebraic structure and conventional chaotic structures. In certain schemes, the crucial design is irrational as well as the key needs to be changed each time an encrypted image is created. These approaches are impractical when many different photos need to be encrypted.

Contribution

The primary goals of this work are as follows:

• 1. We also obtain coset diagrams for the $\langle x$, y | $x^{2}$, $y^{3}$, $\left( xyxyxy^{2}xy^{2}\right) ^{2}\rangle ,$ $\langle x$, y | $x^{2}$, $y^{3}$, $\left( xyxy^{2}\right) ^{2}\rangle$ and $\langle x$, y | $x^{2}$, $y^{3}$, $\left( xyxyxy^{2}\right) ^{2}\rangle$ by taking $\theta =0$ as a parameter.

• 2. To suggest a new way of building S-boxes with generalized triangle groups.

• 3. To examine the suggested S-box’s cryptographic characteristics, such as bit independence, confusion, and diffusion criteria.

• 4. To evaluate the performance of the S-box in an image encryption scheme.

S-box construction generalizes to higher dimensional S-boxes. The coset action of PGL(2, q) on $PL(F_{pn})$ always contains $q+1$ points, so any choice of q ($2^{12}, 2^{16}$, etc.) will define $64\times 64$ or $256\times 256$ S-boxes. The parameterization process remains the same, but you must change the irreducible polynomial of the field.

Layout

This is how the remainder of the article is structured. The parametrization of the finite generalized triangle group are shown in Sect. 2. The S-box design technique based on generalized triangle groups is presented in Sect. 3. An extensive examination of the suggested S-box, including its cryptographic characteristics, is provided in Sect. 4. The integration of the S-box into an image encryption scheme is explained and experimental results are presented in Sect. 5. The article is finally concluded in Sect. 6, which also provides recommendations for further research and a summary of the results.

Parametrization of action for finite generalized triangle groups

In²⁵ factor groups of the abstract group $\delta ^{6,6,6}$ are studied through coset diagrams for an action on $PL(F_{q})$ by parametrizing its actions, the abstract group $\delta ^{p,q,r}$ may be defined for any positive integers p, q, r as follows:

$$\begin{aligned} \langle x,y,t\ |x^{2}=y^{k}=t^{2}=\left( xy\right) ^{p}=\left( xy\right) ^{2}=(xyt)^{q}=1\rangle . \end{aligned}$$

M. Ashiq, Q. Mushtaq and T. Maqsood also parametrized the actions of the subgroup $\left( u,v,t:u^{3}=v^{3}=t^{2}=(ut)^{2}=(vt)^{2}=1\right)$ of the modular group in²⁶.

Since there are actually 14 finite generalized triangle groups (G.T.G.)²⁷, out of these fourteen only eight groups are quotients of the modular groups. In this article, we have worked on three cases of G.T.G which are quotients of modular groups $\langle x$, y | $x^{2}$, $y^{3}$, $\left( xyxyxy^{2}xy^{2}\right) ^{2}\rangle ,$ $\langle x$, y | $x^{2}$, $y^{3}$, $\left( xyxy^{2}\right) ^{2}\rangle$ and $\langle x$, y | $x^{2}$, $y^{3}$, $\left( xyxyxy^{2}\right) ^{2}\rangle$ by taking $\theta =0$ as a parameter we get the coset graphs for each.

We take the terminologies and notations as used, that is, for x,  y and t of $PGL(2,\mathbb {Z} )$ the homomorphism $\xi :PGL(2,\mathbb {Z})\rightarrow PGL(2,q)$ is defined by $\overline{x}=\left( x\right) \xi$, $\overline{y}=\left( y\right) \xi$, $\overline{t}=\left( t\right) \xi$. It is known that PGL(2, q) has a natural permutation representation on $PL(F_{q})$, the homomorphism $\xi$ presents the action of $PGL(2, \mathbb {Z})$ on $PL(F_{q})$.

Let us consider that the field $F_{q}$ which is neither $F_{2^{n}}$ nor $F_{3^{n}}$, for a positive integer n.

Lemma 1

Homomorphism $\xi$ has two conjugacy classes and $\overline{x}\overline{y}\overline{x}\overline{y}^{2}$ have order 2, and two others in which $\overline{x}\overline{y}\overline{x}\overline{y}$ has order 2.

Proof

Let $w=\overline{x}\overline{y}\overline{x}\overline{y}^{2}$ be of order 2, having $\overline{x}^{2}=\overline{y}^{3}=(\overline{x}\overline{y} \overline{x}\overline{y}^{2})^{2}=1$ such that $\overline{x}$, $\overline{y}$ produce a group having order 24 and, of course, $\overline{t}$ normalizes $<\overline{y}>$, and in this case it is characteristic in $<\overline{x}\overline{y}\overline{x}\overline{y}^{2}>$. That is, $\xi$ actually maps $PGL(2,\mathbb {Z})$ into cyclic group order of 3 ’s normilzer. Here, normalizer of the group is order of 24. As , there is conjugation among the elements of order 3 in PGL(2, q), we can take $\overline{y}$ to be fixed element of order 3. Any more conjugation can occur within $N(<\overline{y}>)$. In this group there are two classes of involutions which are non-central, and we can select $\overline{x}$. Then $\overline{x}$ $\overline{t}$ is of order 2 and it centralizes $\overline{x}$ and $\overline{y}$. It is the particular non-trivial element of the center of $N(<\overline{y}>)$. Thus there are just two conjugacy classes of non-degenerate homomorphisms $\xi$ in which w is of order 2.

As the dual $\xi ^{^{\prime }}$ and $\xi$ maps x, y, t onto $\overline{x}$ $\overline{t}$, $\overline{y}$, $\overline{t}$, therefore $(\overline{x}\overline{y}\overline{x}\overline{y}^{2})\xi ^{^{\prime }}=\overline{x}$ $\overline{t}$ $\overline{y}$ $\overline{x}$ $\overline{t}$ $\overline{y} ^{2}=\overline{t}\left( \overline{x}\overline{y}\overline{x}\overline{y}\right) \overline{t}$. If $\overline{x}\overline{y}\overline{x} \overline{y}$ is of order 2, so is $(\overline{x}\overline{y}\overline{x}\overline{y}^{2})\xi ^{^{\prime }}$. Hence there are only two conjugacy classes of non-degenerate homomorphisms: in which one $\overline{x} \overline{y}\overline{x}\overline{y}^{2}$ is of order 2 and the other in which $\overline{x}\overline{y}\overline{x}\overline{y}$ is of order 2. $\square$

Lemma 2

The involution $\left( \overline{x}\overline{y}\overline{x} \overline{y}\right) ^{2}=1$ implies $w=\overline{x}\overline{y}\overline{x}\overline{y}^{2}$ is of order 3.

Proof

Let us consider an element $X=\left[ \begin{array}{cc} a & b\\ c & d \end{array} \right]$ of GL(2, q) which produce an element $\overline{x}$ of PGL(2, q). Since $\overline{x}^{2}=1$, therefore trace of X is zero. As each single element of GL(2, q) of trace zero has, up to scalar multiplication, a conjugate of the form $\left[ \begin{array}{cc} 0 & k\\ 1 & 0 \end{array} \right]$, we can therefore assume that X the form $\left[ \begin{array}{cc} 0 & k\\ 1 & 0 \end{array} \right]$. Let $Y=\left[ \begin{array}{cc} d & f\\ g & h \end{array} \right]$ be an element of GL(2, q) which yields the element $\overline{y}$ of PGL(2, q). Since $\overline{y}^{3}=1$, $Y^{3}$ is a scalar matrix given that the determinant of Y is a square in $F_{q}$. We assume that $\det (Y)=1$ such that $Y^{3}=I$. Then the characteristic equation of Y is $Y^{2} -trace(Y)Y+I=0$. Since $Y^{3}=I$, the polynomial $Y^{2}-trace(Y)Y+I$ must divide $Y^{3}-I$ whence $trace\left( Y\right) =-1$. That is, we may assume that $\det (Y)=1$, $trace(Y)=d+h=-1$; so that Y is $\left[ \begin{array}{cc} d & f\\ g & -d-1 \end{array} \right]$, where

$$\begin{aligned} fg+d^{2}+d+1=0. \end{aligned}$$

(1)

Proceeding further, the determinant of the matrix $W=Y^{2}XYX$ is $k^{2}$ and its trace is

$$\begin{aligned} -f^{2}-2fgk-g^{2}k^{2}-k=-\left( f+gk\right) ^{2}-k. \end{aligned}$$

(2)

Given that $\left( \overline{x}\overline{y}\overline{x}\overline{y}\right) ^{2}=1$, the trace of matrix YXYX is $f^{2}+g^{2}k^{2}-2d^{2}k=0.$ By using Eq. (1) the trace gives

$$\begin{aligned} \left( f+gk\right) ^{2}=-2k. \end{aligned}$$

(3)

Using Eq. (3) and Eq. (2), we get $-f^{2} -2fgk-g^{2}k^{2}-k=k$. This further implies $\left( -f^{2}-2fgk-g^{2} k^{2}-k\right) ^{2}=k^{2}$. Hence,

$$\begin{aligned} \left( traceW\right) ^{2}=\det W. \end{aligned}$$

Thus, the parameter of w is 1, so the order of w is 3, from the table (²⁸). $\square$

We now come to the following theorem.

Theorem 1

Any non trivial element g of PGL(2, q), and its order is other than 2 or 3, which is the image of $w=\overline{x}\overline{y}\overline{x}\overline{y}^{2}$ under the homomorphism $\xi$ of $PGL(2,\mathbb {Z})$.

Proof

We shall look for elements $\overline{x}$, $\overline{y}$, $\overline{t}$ of PGL(2, q) satisfying the relations

$$\begin{aligned} \overline{x}^{2}=\overline{y}^{3}=\overline{t}^{2}=\left( \overline{x}\overline{y}\overline{x}\overline{y}^{2}\right) ^{n}=(\overline{x}\overline{t})^{2}=\left( \overline{y}\overline{t}\right) ^{2}=1 \end{aligned}$$

(4)

in a given conjugacy class.

Take $\overline{x}$, $\overline{y}$ and $\overline{t}$ to be represented by $X=\left[ \begin{array}{cc} a & kc\\ c & -a \end{array} \right]$, $Y=\left[ \begin{array}{cc} d & kf\\ f & -d-1 \end{array} \right]$ and $T=\left[ \begin{array}{cc} 0 & -k\\ 1 & 0 \end{array} \right]$ where $a,b,c,d,k,f\in F_{q}$ with $k\ne 0$. We shall take $\Delta$ as the determinant of matrix X

$$\begin{aligned} \det \left( X\right) =-a^{2}-kc^{2}=\Delta \ne 0. \end{aligned}$$

(5)

Now we require the determinant of matrix Y to be 1, that is

$$\begin{aligned} d^{2}+d+kf^{2}+1=0. \end{aligned}$$

(6)

This definitely yields the elements which satisfies the relations $\overline{x}^{2}=\overline{y}^{3}=\overline{t}^{2}=(\overline{x}\overline{t})^{2}=\left( \overline{y}\overline{t}\right) ^{2}=1$. Therefore we just checked the conjugacy class of $w=\overline{x}\overline{y}\overline{x}\overline{y}^{2}$.Let $W=Y^{2}XYX$ be the matrix representing w, then W has the trace

$$\begin{aligned} r&=-3a^{2}d^{2}-3a^{2}d+a^{2}f^{2}k-a^{2}-8acdfk-\\&4acfk+c^{2}d^{2}k+c^{2}dk-3c^{2}f^{2}k^{2}. \end{aligned}$$

Since $\det (Y)=1$ and $\det \left( X\right) =-a^{2}-kc^{2}=\Delta$, we have identities $kf^{2}=-d^{2}-d-1$ and $kc^{2}=-\Delta -a^{2}$. By using these identities the determinant of the matrix W is $\Delta ^{2}$, and the trace of the matrix W reduces to

$$\begin{aligned} & r=2a^{2}-c^{2}k+4fk\left( a^{2}f-ac-c^{2}fk-2acd\right) \end{aligned}$$

(7)

$$\begin{aligned} & s=-\left( c-2af+2cd\right) \left( a+2ad+2cfk\right) . \end{aligned}$$

(8)

If the $trace(TW)=ks$, so that

$$\begin{aligned} r^{2}+ks^{2}&=16a^{4}d^{2}f^{2}k+16a^{4}df^{2}k+16a^{4}f^{4}k^{2} +20a^{4}f^{2}k+4a^{4}\nonumber \\&-32a^{3}cd^{3}fk-48a^{3}cd^{2}fk-32a^{3}cdf^{3}k^{2}-56a^{3}cdfk\nonumber \\&-16a^{3}cf^{3}k^{2}-20a^{3}cfk+16a^{2}c^{2}d^{4}k+32a^{2}c^{2} d^{3}k\nonumber \\&+24a^{2}c^{2}d^{2}k+8a^{2}c^{2}dk-16a^{2}c^{2}f^{4}k^{3}-24a^{2}c^{2} f^{2}k^{2}\nonumber \\&-3a^{2}c^{2}k+32ac^{3}d^{3}fk^{2}+48ac^{3}d^{2}fk^{2}+32ac^{3}df^{3} k^{3}\nonumber \\&+40ac^{3}dfk^{2}+16ac^{3}f^{3}k^{3}+12ac^{3}fk^{2}+16c^{4}d^{2}f^{2} k^{3}\nonumber \\&+16c^{4}df^{2}k^{3}+16c^{4}f^{4}k^{4}+12c^{4}f^{2}k^{3}+c^{4}k^{2} \end{aligned}$$

(9)

which reduces to

$$\begin{aligned} r^{2}+ks^{2}=\left( a^{2}+kc^{2}\right) \left( c^{2}k+4a^{2}-4c^{2} f^{2}k^{2}+4a^{2}f^{2}k-4acfk-8acdfk\right) . \end{aligned}$$

(10)

As $r=2a^{2}-c^{2}k-4c^{2}f^{2}k^{2}+4a^{2}f^{2}k-4acfk-8acdfk$ and $-a^{2}-kc^{2}=\Delta$, we get

$$\begin{aligned} r^{2}+ks^{2}=2\Delta ^{2}-\Delta r \end{aligned}$$

put $\Delta ^{2}=\psi$

$$\begin{aligned} r^{2}+ks^{2}=2\psi -\Delta r. \end{aligned}$$

(11)

$$\begin{aligned} \left( Y^{2}XYX\right) ^{n}&=\{ \left( {\begin{array}{c}n-1\\ 0\end{array}}\right) r^{n-1}-\left( {\begin{array}{c}n-2\\ 1\end{array}}\right) r^{n-3}\psi +\left( {\begin{array}{c}n-3\\ 2\end{array}}\right) r^{n-5}\psi ^{2}-\ldots \} \left( Y^{2}XYX\right) \nonumber \\&-\psi \{ \left( {\begin{array}{c}n-2\\ 0\end{array}}\right) r^{n-2}-\left( {\begin{array}{c}n-3\\ 1\end{array}}\right) r^{n-4}\psi +\left( {\begin{array}{c}n-4\\ 2\end{array}}\right) r^{n-6}\psi ^{2}-\ldots \}I \end{aligned}$$

(12)

and

$$\begin{aligned} f\left( r\right) =\left( {\begin{array}{c}n-1\\ 0\end{array}}\right) r^{n-1}-\left( {\begin{array}{c}n-2\\ 1\end{array}}\right) r^{n-3}\psi +\left( {\begin{array}{c}n-3\\ 2\end{array}}\right) r^{n-5}\psi ^{5}-...\text {.} \end{aligned}$$

(13)

By replacing $\theta$ for $r^{2}$ in $f\left( r\right)$ gives a polynomial $f\left( \theta \right)$. So that a minimal polynomial can be determined for any natural number n such that $q\equiv \pm 1\left( \operatorname {mod} p\right)$ and $\theta$ is a quadratic residue in $F_{p}$ (since $\theta =\left( r/\Delta \right) ^{2}$) by the equation:

$$\begin{aligned} g_{n}\left( \theta \right) =\frac{f_{n}\left( \theta \right) }{g_{d_{1} }\left( \theta \right) g_{d_{2}}\left( \theta \right) \ldots g_{d_{m} }\left( \theta \right) } \end{aligned}$$

(14)

$$\begin{aligned} X^{2}+\Delta I&=0\\ (W)^{2}-r(W)+\psi I&=0\\ Y^{2}+Y+I&=0. \end{aligned}$$

For the matrix in GL(2, q) which represents an involution PGL(2, q) the trace of the matrix is zero. This means that $r=0$, and by

$$\begin{aligned} \theta =\frac{r^{2}}{\Delta ^{2}} \end{aligned}$$

(15)

we get $\theta =0$. If $s=0$, then Eq. (11) implies $\theta ^{2} -5\theta +4=0$ which further gives $\theta =1,4$. By the table in Sect. 2, $\theta =1$ is related to the order 3 of w, that is $w=\left( xyxy^{2}\right) ^{3}=1$, and $\theta =4$ is related to the order 1 of w. Other than these three cases the matrix in GL(2, q) which do not represent order 1, 2 or 3 of w in PGL(2, q), has trace $r,s\ne 0$, as $ks=$trace(TW). Thus, the class of w is determined by $\theta =\frac{r^{2} }{\Delta ^{2}}\ne 0$,1, 4.

From the eqs. (6),( 7), (8), and (11), it is sufficient to show that we can choose a, c, d, f, k, s for $\theta =\frac{r^{2} }{\Delta ^{2}}$ that gives the matrices X, Y and T which satisfy the group $<\overline{x}^{2}=\overline{y}^{3}=\overline{t}^{2}=\left( \overline{x}\overline{y}\overline{x}\overline{y}^{2}\right) ^{n} =(\overline{x}\overline{t})^{2}=\left( \overline{y}\overline{t}\right) ^{2}=1>$. Hence, proved. $\square$

We now put together results (1) and (2) to obtain the main theorem.

Theorem 2

The conjugacy classes of the homomorphisms of $PGL(2,\mathbb {Z})$ into PGL(2, q) yielding $\overline{x}\overline{y}\overline{x}\overline{y} ^{2}$ are associated with the parameter $\theta \ne 0$, 1 of $PL\left( F_{q}\right)$.

Coset diagrams

The previous theorem conveys that, the non-degenerate homomorphisms of

$PGL(2,\mathbb {Z})$ into PGL(2, q) by the elements of $F_{q}$ can be parametrized. As this homomorphism generates the action of $PGL(2, \mathbb {Z})$ on $PL(F_{q})$. This action can be represented by a coset diagram $D(\theta ,q)$. A coset diagram (showing a conjugacy class of $\xi$) can be drawn corresponding to each parameter $\theta$ by determining $\overline{x}$, $\overline{y}$ with the help of theorem (2).

The method of obtaining the coset diagram for a particular prime p and $\theta$, by using equations (15), (6),( 7), (8), and (11), is explained with the help of an example.

Example 1

The action of $PGL(2,\mathbb {Z})$ on $PL\left( F_{19}\right)$ for $\theta =7$ of $F_{19}$ gives a coset diagram. By Eq. (15) $\theta =\frac{r^{2}}{\Delta ^{2}}$. Let $\Delta =4$ then $r=6$. By Eq. (11) $r^{2}+ks^{2}=2\Delta ^{2}-\Delta r$. Assuming $k=2$we have $s=9.$The Eq. (6) states $d^{2}+d+kf^{2}+1=0$. If we let $d=2,$ then $f=5.$From Eq. ( 7) $r=2a^{2}-c^{2}k+4fk\left( a^{2}f-ac-c^{2} fk-2acd\right)$ implies

$$\begin{aligned} 6=12a^{2}+9ac+16c^{2}. \end{aligned}$$

(a)

As (8) gives the value of $s=-\left( c-2af+2cd\right) \left( a+2ad+2cfk\right)$, we get

$$\begin{aligned} 9=12a^{2}+4ac+14c^{2}. \end{aligned}$$

(b)

Solving equations(a) and (b), we get

$$\begin{aligned} 0=8a^{2}+7c^{2}. \end{aligned}$$

(c)

The Eq. (5) $-a^{2}-kc^{2}=\Delta$ gives

$$\begin{aligned} a^{2}+2c^{2}=15. \end{aligned}$$

(d)

Once again solving equations (c) and (d) gives $c=8$ and $a=1.$We have finally obtained all the variables of the matrices X, Y, T which yields the linear fractional transformations;

$$\begin{aligned} \bar{x}:z\mapsto \frac{z+16}{8z-1}\text {, }\bar{y}:z\mapsto \frac{2z+10}{5z-3}\, \text {, }\overline{t}:z\mapsto \frac{-2}{z} \end{aligned}$$

For $z\in PL(F_{19})$ the permutations are:

$\bar{x}=(0$ 3)(1 16)(2 5)(4 8)(6 13)(7 17)(9 14)(10 15)(11 $18)\left( 12\,\infty \right)$

$\bar{y}=(0$ 3 14)(1 6 17)(2)(4 10 16)(5 13 11)(7 15 9)(8 12 $\infty )\left( 18\right)$

$\overline{t}=(0$ $\infty )(1$ 17)(2 18)(3 12)(4 9)(5 11)(6)(7 16)(8 14)(0 3)(15 10)(13).

The associated diagram $D\left( 7,19\right)$ is:

Theorem 3

If $w=\overline{x}\overline{y}\overline{x}\overline{y}^{2}$, then either the $\bar{t}$ exists or $w=1.$

Proof

Let $X=\left[ \begin{array}{cc} a & b\\ c & -a \end{array} \right] ,$ for all element of GL(2, q),  having trace equals to zero and it has up to scalar multiplication of a conjugate of the form $\left[ \begin{array}{cc} 0 & k\\ 1 & 0 \end{array} \right] .$ We can therefore assume that X has the form $\left[ \begin{array}{cc} 0 & k\\ 1 & 0 \end{array} \right] .$ Let $Y=\left[ \begin{array}{cc} e & f\\ g & h \end{array} \right] \in GL(2,q)$ which gives the element $\overline{y}$ of PGL(2, q). Thus replacing $Y^{3}=\pm 1$ and once more substitute $Y=-Y.$ According to requirement, we can assume that $Y^{3}=1.$ The characteristic equation of Y is $Y^{2}+rY+1=0$ where -r is the trace. Since $Y^{3}=1$ the polynomial $Y^{2}+rY+1=0$ should divide $Y^{3}=1$ where $r=1$. We can also consider that $\det \left( Y\right) =1,$ trace$\left( Y\right) =e+h=-1$ so that Y is $\left[ \begin{array}{cc} e & f\\ g & -e-1 \end{array} \right] ,$ where

$$\begin{aligned} fg+e^{2}+e+1=0 \end{aligned}$$

(a)

Now suppose there exist a transformation $\overline{t}$ in PSL(2, q) such that $\overline{t}^{2}=(\overline{x}\overline{t})^{2}=\left( \overline{y}\overline{t}\right) ^{2}=1.$ If $T=\left[ \begin{array}{cc} l & m\\ n & j \end{array} \right]$ is the element in GL(2, q) with correspondence to $\overline{t,}$ then $j=-l$ because t is of order 2. Similarly because $(\overline{x}\overline{t})^{2}=\left( \overline{y}\overline{t}\right) ^{2}=1,$ $m=-kn$ and

$$\begin{aligned} \left( 2e+1\right) l+\left( f-kg\right) n=0. \end{aligned}$$

(b)

The matrix T is not singular, that is

$$\begin{aligned} l^{2}-kn^{2}\ne 0 \end{aligned}$$

(c)

the compulsory and sufficient conditions for the presence of $\overline{t}$ in PSL(2, q) are Eq. (b) and Eq. (c) , the matrix T exists until $l^{2}-kn^{2}=0,$ if $\left( 2e+1\right) =0=\left( f-kg\right)$ the existence of t is trivial, otherwise from Eq. (b)

$$\begin{aligned} \left( 2e+1\right) l=-\left( f-kg\right) n \end{aligned}$$

then from Eq .(c),

$$\begin{aligned} \left( f-kg\right) ^{2}-k\left( 2e+1\right) ^{2}\ne 0 \end{aligned}$$

thus there exist $\overline{t}$ in PSL(2, q),  where $\overline{t} ^{2}=(\overline{x}\overline{t})^{2}=\left( \overline{y}\overline{t}\right) ^{2}=1$ unless $\left( f-kg\right) ^{2}=k\left( 2e+1\right) ^{2}$. But if $\left( f-kg\right) ^{2}=k\left( 2e+1\right) ^{2},$ then because of Eq. (a) , we get

$$\begin{aligned} \left( f-kg\right) ^{2}=-3k. \end{aligned}$$

(d)

As trace of $xyxy^{2}$ is denoted by r we can calculate r by using scientific workplace which is,

$$\begin{aligned} r=ke^{2}-g^{2}k^{2}+ke-f^{2}-fgk \end{aligned}$$

solving and using identity $fg+e^{2}+e=-1$ we get,

$$\begin{aligned} r=2k\Rightarrow r^{2}=4k^{2} \end{aligned}$$

As $\det \left( W\right) =$ $\Psi =k^{2},$

$$\begin{aligned} r^{2}=4\Psi . \end{aligned}$$

This yields $\theta =4$ and by the table²⁸, we have

$$\begin{aligned} \left( W\right) ^{1}=I \end{aligned}$$

Thus, either the matrix T exists or $W=I$ which completes the proof. $\square$

Parametric equations for $\theta =0$

We have considered the parametrization of the homomorphisms or actions for the group $\langle x,y\mid x^{2},y^{3},\left( xyxy^{2}\right) ^{2}\rangle$ in the previous section for all the elements $\theta$ of the field $F_{q}$. In this section we consider only that case of $\theta =0$ not only for the aforementioned group but also for two other finite generalized triangle groups $\langle x,y\mid x^{2},y^{3},\left( xyxyxy^{2}xy^{2}\right) ^{2}\rangle$ and $\langle x,y\mid x^{2},y^{3},\left( xyxyxy^{2}\right) ^{2}\rangle$ which are quotients of modular groups.

The group $\langle x,y\mid x^{2},y^{3},\left( xyxy^{2}\right) ^{2}\rangle$

Let $W_{1}=Y^{2}XYX$ , then the equations ( 5) ,( 6) ,(7), (8), and (9) are the parametric equations for $\theta =0.$

Example 2

Let us consider the action of $PGL\left( 2,\mathbb {Z}\right)$ on $PL\left( F_{17}\right)$ and correspondingly draw a coset diagram and let $\theta =0$ from $F_{17},$ then $\overline{x},\overline{y},$ and $\overline{t}$ respectively are:

$$\begin{aligned} \overline{x}&\longmapsto \frac{z+7}{7z-1}\\ \overline{y}&\longmapsto \frac{3z+2}{2z-4}\\ \overline{t}&\longmapsto \frac{-1}{z} \end{aligned}$$

where $z\in PL\left( F_{17}\right) .$ The associated diagram $D\left( 0,17\right)$ is given below.

The group $\langle x,y\mid x^{2},y^{3},\left( xyxyxy^{2} xy^{2}\right) ^{2}\rangle$

Let $W_{2}=Y^{2}XY^{2}XYXYX$ with the same matrices X and Y and determinant given by the equations ( 5) and ( 6) . Then the group $\langle x,y\mid x^{2},y^{3},\left( w_{2}\right) ^{2}\rangle$ represents a finite generalized triangle groups of order 576. Let $\left[ \begin{array}{cc} a_{11} & a_{12}\\ a_{21} & a_{22} \end{array} \right]$ represents the matrix $W_{2}$ and $r_{2}$ be its trace. The trace of $W_{2}$ is:

$$\begin{aligned} r_{2}&=\hspace{0in}2a^{4}d^{6}+6a^{4}d^{5}+2a^{4}d^{4}f^{2}k+15a^{4} d^{4}+4a^{4}d^{3}f^{2}k+20a^{4}d^{3}\\&-2a^{4}d^{2}f^{4}k^{2}-4a^{4}d^{2}f^{2}k+15a^{4}d^{2} -2a^{4}df^{4}k^{2}-6a^{4}df^{2}k+6a^{4}d\\&-2a^{4}f^{6}k^{3}+a^{4}f^{4}k^{2}-2a^{4}f^{2}k+a^{4} +8a^{3}cd^{5}fk+20a^{3}cd^{4}fk\\&+16a^{3}cd^{3}f^{3}k^{2}+64a^{3}cd^{3}fk+24a^{3}cd^{2}f^{3} k^{2}+76a^{3}cd^{2}fk\\&+8a^{3}cdf^{5}k^{3}-8a^{3}cdf^{3}k^{2}+40a^{3}cdfk+4a^{3} cf^{5}k^{3}-8a^{3}cf^{3}k^{2}\\&+8a^{3}cfk-7a^{2}c^{2}d^{4}k-14a^{2}c^{2}d^{3}k+82a^{2} c^{2}d^{2}f^{2}k^{2}-9a^{2}c^{2}d^{2}k\\&+82a^{2}c^{2}df^{2}k^{2}-2a^{2}c^{2}dk-7a^{2}c^{2}f^{4} k^{3}+22a^{2}c^{2}f^{2}k^{2}+8ac^{3}d^{5}fk^{2}\\&+20ac^{3}d^{4}fk^{2}+16ac^{3}d^{3}f^{3}k^{3}+24ac^{3}d^{2} f^{3}k^{3}-20ac^{3}d^{2}fk^{2}\\&+8ac^{3}df^{5}k^{4}+56ac^{3}df^{3}k^{3}-8ac^{3}dfk^{2} +4ac^{3}f^{5}k^{4}+24ac^{3}f^{3}k^{3}\\&-2c^{4}d^{6}k^{2}-6c^{4}d^{5}k^{2}-2c^{4}d^{4}f^{2}k^{3} -6c^{4}d^{4}k^{2}-4c^{4}d^{3}f^{2}k^{3}-2c^{4}d^{3}k^{2}\\&+2c^{4}d^{2}f^{4}k^{4}-10c^{4}d^{2}f^{2}k^{3}+2c^{4}df^{4} k^{4}-8c^{4}df^{2}k^{3}+2c^{4}f^{6}k^{5}+8c^{4}f^{4}k^{4}. \end{aligned}$$

Using the equations $kf^{2}=\left( -d^{2}-d-1\right)$ and $kc^{2} =-\Delta -a^{2}$, $r_{2}$ reduces to:

$$\begin{aligned} r_{2}&=2a^{4}\left( 4d+4d^{2}+3\right) \left( 2d+2d^{2}+1\right) +4acfk\left( 2d+1\right) \left( 8d+8d^{2}+5\right) (\Delta +2a^{2} )\nonumber \\&+4(\Delta +a^{2})d\left( d+1\right) \left( 4d+4d^{2}+5\right) (\Delta +7a^{2})+\left( \Delta +a^{2}\right) \left( 6\Delta +35a^{2}\right) . \end{aligned}$$

(16)

Let $T=\left[ \begin{array}{cc} 0 & -k\\ 1 & 0 \end{array} \right] ,$ then the matrix $W_{2}T$ is denoted by $\left[ \begin{array}{cc} b_{11} & b_{12}\\ b_{21} & b_{22} \end{array} \right]$. Similarly, the scientific workplace calculations shows that the trace of $W_{2}T$ is:

$$\begin{aligned} trace\left( W_{2}T\right)&=-4a^{4}d^{5}fk-10a^{4}d^{4}fk-8a^{4} d^{3}f^{3}k^{2}-20a^{4}d^{3}fk\\&-12a^{4}d^{2}f^{3}k^{2}-20a^{4}d^{2}fk-4a^{4}df^{5}k^{3}-10a^{4}dfk\\&-2a^{4}f^{5}k^{3}+2a^{4}f^{3}k^{2}-2a^{4}fk+4a^{3}cd^{6}k\\&+12a^{3}cd^{5}k+4a^{3}cd^{4}f^{2}k^{2}+25a^{3}cd^{4}k\\&+8a^{3}cd^{3}f^{2}k^{2}+30a^{3}cd^{3}k-4a^{3}cd^{2}f^{4}k^{3}\\&-42a^{3}cd^{2}f^{2}k^{2}+20a^{3}cd^{2}k-4a^{3}cdf^{4}k^{3}\\&-46a^{3}cdf^{2}k^{2}+7a^{3}cdk-4a^{3}cf^{6}k^{4}+5a^{3} cf^{4}k^{3}\\&-13a^{3}cf^{2}k^{2}+a^{3}ck+48a^{2}c^{2}d^{3}fk^{2} +72a^{2}c^{2}d^{2}fk^{2}\\&-48a^{2}c^{2}df^{3}k^{3}+36a^{2}c^{2}dfk^{2}-24a^{2}c^{2} f^{3}k^{3}\\&+6a^{2}c^{2}fk^{2}+4ac^{3}d^{6}k^{2}+12ac^{3}d^{5}k^{2} +4ac^{3}d^{4}f^{2}k^{3}\\&+9ac^{3}d^{4}k^{2}+8ac^{3}d^{3}f^{2}k^{3}-2ac^{3}d^{3} k^{2}-4ac^{3}d^{2}f^{4}k^{4}\\&+54ac^{3}d^{2}f^{2}k^{3}-4ac^{3}d^{2}k^{2}-4ac^{3}df^{4} k^{4}+50ac^{3}df^{2}k^{3}\\&-ac^{3}dk^{2}-4ac^{3}f^{6}k^{5}-11ac^{3}f^{4}k^{4}+11ac^{3} f^{2}k^{3}\\&+4c^{4}d^{5}fk^{3}+10c^{4}d^{4}fk^{3}+8c^{4}d^{3}f^{3} k^{4}+4c^{4}d^{3}fk^{3}\\&+12c^{4}d^{2}f^{3}k^{4}-4c^{4}d^{2}fk^{3}+4c^{4}df^{5}k^{5}+16c^{4} df^{3}k^{4}\\&-2c^{4}dfk^{3}+2c^{4}f^{5}k^{5}+6c^{4}f^{3}k^{4}. \end{aligned}$$

This simplifies into

$$\begin{aligned} ks_{2}&=k\left( c-2af+2cd\right) \left( a+2ad+2cfk\right) .\alpha \\ \text {where }\alpha&=\left( \begin{array}{c} 3a^{2}d+4a^{2}d^{2}+2a^{2}d^{3}+a^{2}d^{4}+a^{2}+a^{2}f^{4}k^{2}+3c^{2} f^{2}k^{2}\\ +c^{2}f^{4}k^{3}-c^{2}dk-a^{2}f^{2}k+2c^{2}d^{3}k+c^{2}d^{4}k+2a^{2}d^{2} f^{2}k\\ +2c^{2}df^{2}k^{2}+2c^{2}d^{2}f^{2}k^{2}+2a^{2}df^{2}k+4acfk+8acdfk \end{array} \right) . \end{aligned}$$

Now we consider $trace(W_{2}T)=ks_{2}$, using identities $kf^{2}=\left( -d^{2}-d-1\right)$ and $kc^{2}=-\Delta -a^{2},$ the value of $s_{2}$ reduces into:

$$\begin{aligned} s_{2}&=-2a^{4}f\left( 2d+1\right) \left( 4d+4d^{2}+3\right) +a^{3}c\left( 84d+148d^{2}+128d^{3}+64d^{4}+23\right) \\&-2f(\Delta +a^{2})(2d+1)\left( 28a^{2}d^{2}+28a^{2}d+17a^{2}+4\Delta d^{2}+4\Delta d+2\Delta \right) \\&+2ac(\Delta +a^{2})\left( 38d+70d^{2}+64d^{3}+32d^{4}+9\right) . \end{aligned}$$

Equate,

$$\begin{aligned} r_{2}^{2}=\left( \begin{array}{c} 36d^{2}\Delta ^{2}+32d^{3}\Delta ^{2}+16d^{4}\Delta ^{2}+6\Delta ^{2} +160a^{4}d+41a^{2}\Delta \\ +20d\Delta ^{2}+288a^{4}d^{2}+256a^{4}d^{3}+128a^{4}d^{4}+41a^{4}\\ +288a^{2}d^{2}\Delta +256a^{2}d^{3}\Delta +128a^{2}d^{4}\Delta +160a^{2} d\Delta +40a^{3}cfk\\ +20acfk\Delta +144a^{3}cdfk+192a^{3}cd^{2}fk+128a^{3}cd^{3}fk\\ +72acdfk\Delta +96acd^{2}fk\Delta +64acd^{3}fk\Delta \end{array} \right) ^{2} \end{aligned}$$

and

$$\begin{aligned} s_{2}^{2}=\left( \begin{array}{c} -41a^{3}c+40a^{4}f+4f\Delta ^{2}+24d^{2}f\Delta ^{2}+16d^{3}f\Delta ^{2}-18ac\Delta \\ -160a^{3}cd+144a^{4}df+38a^{2}f\Delta +16df\Delta ^{2}-288a^{3}cd^{2}\\ -256a^{3}cd^{3}-128a^{3}cd^{4}+192a^{4}d^{2}f+128a^{4}d^{3}f\\ +192a^{2}d^{2}f\Delta +128a^{2}d^{3}f\Delta -76acd\Delta -140acd^{2}\Delta \\ -128acd^{3}\Delta -64acd^{4}\Delta +140a^{2}df\Delta \end{array} \right) ^{2} \end{aligned}$$

so that,

$$\begin{aligned} r_{2}^{2}+ks_{2}^{2}&=a^{6}\Delta \left( 8d+8d^{2}+5\right) \left( 320d+576d^{2}+512d^{3}+256d^{4}+73\right) \\&+4a^{5}cfk\Delta \left( 2d+1\right) \left( 16d+16d^{2}+7\right) \left( 16d+16d^{2}+13\right) \\&+3a^{4}\Delta ^{2}\left( 1172d+3156d^{2}+4992d^{3}+5056d^{4}+3072d^{5} +1024d^{6}+203\right) \\&+8a^{3}cfk\Delta ^{2}\left( 2d+1\right) \left( 172d+300d^{2} +256d^{3}+128d^{4}+53\right) \\&+24a^{2}\Delta ^{3}\left( d+d^{2}+1\right) \left( 50d+98d^{2} +96d^{3}+48d^{4}+11\right) \\&+96acfk\Delta ^{3}\left( 2d+1\right) \left( d+d^{2}+1\right) \left( 2d+2d^{2}+1\right) \\&+4\Delta ^{4}\left( 4d+4d^{2}+5\right) \left( 2d+2d^{2}+1\right) ^{2} \end{aligned}$$

We analyze that it is not feasible to have a simplified relation of groups $\langle x,y\mid x^{2},y^{3},W_{1}^{n}\rangle .$ It is an open problem to find the simplified equation which will result in the complete parametrization of the homomorphisms for the group $\langle x,y\mid x^{2},y^{3},W_{2}^{n}\rangle$. For a given $\theta _{2}=\frac{r_{2}^{2}}{\Delta ^{4}}$ the parametric equations for $\theta _{2}=0$ when $W_{2}=Y^{2}XY^{2}XYXYX$ gives generalized triangle group $\langle x$, y | $x^{2}$, $y^{3}$, $\left( xyxyxy^{2} xy^{2}\right) ^{2}=1\rangle .$

Example 3

Consider the action of $PGL\left( 2,\mathbb {Z} \right)$ on $PL\left( F_{17}\right)$ for $\theta =0$. Then $\overline{x},\overline{y}$ and $\overline{t}$ respectively are:

$$\begin{aligned} \overline{x}&\longmapsto \frac{16z+10}{10z+1}\\ \overline{y}&\longmapsto \frac{3z+2}{2z+13}\\ \overline{t}&\longmapsto \frac{-1}{z} \end{aligned}$$

where z $\in PL\left( F_{17}\right)$. The associated coset diagram $D\left( 0,17\right)$ is:

The group $\langle x$, y | $x^{2}$, $y^{3}$, $\left( xyxyxy^{2}\right) ^{2}\rangle$

Considering the same above matrices of X and Y and get the equations ( 5) and ( 6) .

Then the group $W_{3}=Y^{2}XYXYX,$ of order 48 represents the finite generalized triangle groups. Let $\left[ \begin{array}{cc} c_{11} & c_{12}\\ c_{21} & c_{22} \end{array} \right]$ represents the matrix $W_{3}$ and Let $r_{3}=trace\left( W_{3}\right)$, $r_{3}$ is calculated by using scientific workplace.

$$\begin{aligned} r_{3}&=-4a^{3}d^{3}-6a^{3}d^{2}+4a^{3}df^{2}k-4a^{3}d+2a^{3}f^{2} k-a^{3}-20a^{2}cd^{2}fk-20a^{2}cdfk\\&+4a^{2}cf^{3}k^{2}-6a^{2}cfk+4ac^{2}d^{3}k+6ac^{2}d^{2}k-20ac^{2} df^{2}k^{2}+2ac^{2}dk\\&-10ac^{2}f^{2}k^{2}+4c^{3}d^{2}fk^{2}+4c^{3}dfk^{2}-4c^{3}f^{3}k^{3} \end{aligned}$$

using identities $kf^{2}=\left( -d^{2}-d-1\right)$ and $kc^{2} =-\Delta -a^{2}$

$$\begin{aligned} r_{3}&=-a^{3}\left( 2d+1\right) \left( 4d+4d^{2}+3\right) -2a^{2}cfk\left( 12d+12d^{2}+5\right) -2\left( \Delta +a^{2}\right) \\&\left[ a\left( 2d+1\right) \left( 6d+6d^{2}+5\right) +2fkc\left( 2d+2d^{2}+1\right) \right] \end{aligned}$$

The matrix $W_{3}T$ has trace: $\hbox{trace}=6a^{3}d^{2}fk+6a^{3}dfk-2a^{3}f^{3}k^{2}+2a^{3}fk-6a^{2}cd^{3}$$k-9a^{2}cd^{2}k+18a^{2}cdf^{2}k^{2}-5a^{2}cdk+9a^{2}cf^{2} k^{2}-a^{2}ck-18ac^{2}d^{2}fk^{2}$$-18ac^{2}dfk^{2}+6ac^{2}f^{3}k^{3} -4ac^{2}fk^{2}+2c^{3}d^{3}k^{2}$$+3c^{3}d^{2}k^{2}-6c^{3}df^{2}k^{3}+c^{3} dk^{2}-3c^{3}f^{2}k^{3}$

=$k\left( -c+2af-2cd\right) \omega$ where

$\omega =\left( 3a^{2}d+3a^{2}d^{2}+a^{2}+3c^{2}f^{2}k^{2}-c^{2} dk-a^{2}f^{2}k-c^{2}d^{2}k+4acfk+8acdfk\right)$

Now we consider $W_{3}T=ks_{3}$

$$\begin{aligned} s_{3}&=\left( -c+2af-2cd\right) \beta \text { where } \\ \beta&=\left( 3a^{2}d+3a^{2}d^{2}+a^{2}+3c^{2}f^{2} k^{2}-c^{2}dk-a^{2}f^{2}k-c^{2}d^{2}k+4acfk+8acdfk\right) \\ s_{3}&=4a^{3}f\left( 2d+2d^{2}+1\right) +-2a^{2}c\left( 2d+1\right) \left( 6d+6d^{2}+5\right) +(\Delta +a^{2})\\&\left[ 2af\left( 12d+12d^{2}+5\right) -c\left( 2d+1\right) \left( 4d+4d^{2}+3\right) \right] \end{aligned}$$

Now,

$$\begin{aligned} r_{3}^{2}=\left( \begin{array}{c} 10a\Delta +42a^{3}d+48a^{3}d^{2}+32a^{3}d^{3}+13a^{3}\\ +32ad\Delta +36ad^{2}\Delta +24ad^{3}\Delta +4cfk\Delta +14a^{2}cfk\\ +8cdfk\Delta +32a^{2}cdfk+8cd^{2}fk\Delta +32a^{2}cd^{2}fk \end{array} \right) ^{2} \end{aligned}$$

and

$$\begin{aligned} s_{3}^{2}=\left( \begin{array}{c} -3c\Delta -13a^{2}c+14a^{3}f+10af\Delta -10cd\Delta \\ -42a^{2}cd+32a^{3}df-12cd^{2}\Delta -8cd^{3}\Delta -48a^{2}cd^{2}\\ -32a^{2}cd^{3}+32a^{3}d^{2}f+24adf\Delta +24ad^{2}f\Delta \end{array} \right) ^{2} \end{aligned}$$

so that,

$$\begin{aligned} r_{3}^{2}+ks_{3}^{2}&=a^{6}\left( 196d+196d^{2}+27\right) \\&+a^{4}\Delta \left( 356d+484d^{2}+256d^{3}+128d^{4}+41\right) \\&+8a^{3}cfk\Delta \left( 2d+1\right) \left( 8d+8d^{2}+5\right) \\&+a^{2}\Delta ^{2}\left( 160d+288d^{2}+256d^{3}+128d^{4}+41\right) \\&+4acfk\Delta ^{2}\left( 2d+1\right) \left( 8d+8d^{2}+5\right) \\&+\Delta ^{3}\left( 20d+36d^{2}+32d^{3}+16d^{4}+7\right) \end{aligned}$$

We analyze that it is not feasible to have a simplified relation of group $\langle x,y\mid x^{2},y^{3},W_{1}^{n}\rangle .$ It is an open problem to find its simplified equation which will result in the complete parametrization of the homomorphisms for the group $\langle x,y\mid x^{2},y^{3},W_{2}^{n}\rangle$.

For given

$$\begin{aligned} \theta _{3}=\frac{r_{3}^{2}}{\Delta ^{3}} \end{aligned}$$

The parametric equations for $\theta =0,$ $W_{3}=Y^{2}XYXYX$ gives generalized triangle group $\langle x$, y | $x^{2}$, $y^{3}$, $\left( xyxyxy^{2} \right) ^{2}=1\rangle .$ (Fig. 1)

Example 4

Let us take the action of $PGL\left( 2,\mathbb {Z} \right)$ on $PL\left( F_{17}\right)$ and make a coset diagram. Assume that $\theta =0$ from $F_{17},$ then the linear fractional transformations are:

$$\begin{aligned} \overline{x}&\longmapsto \frac{7z+16}{16z+10},\\ \overline{y}&\longmapsto \frac{3z+2}{2z+13},\text { and}\\ \overline{t}&\longmapsto \frac{-1}{z} \end{aligned}$$

where $z\in PL\left( F_{17}\right) .$

Fig. 1

Fig.

S-box from GTG $\langle x,y\mid x^{2},y^{3},\left( xyxy^{2}\right) ^{n}\rangle$ and $GF(2^{8})$

The study of group actions and their application to various mathematical structures has garnered significant attention in recent years.²⁹ presents a new technique for constructing securely encrypted 8times8 S-boxes by employing an adjacency matrix based on the Galois field $GF(2^8)$. Similarly, an innovative construction strategy of the nonlinear part of the block cipher is provided in ?, based on the permutation triangle groups and the action of projective linear groups over the projective line.

A parameterized action is used in Sect. 2 to derive the coset diagrams illustrating the modular group’s action on the projective line over finite fields. This parametrization leads to the $\langle x, y \mid x^2 = y^3 = w(x, y)^k = 1 \rangle$, where $w(x, y) = xyxy^2$ and $k \in \mathbb {Z}^+$.

Our main objective is to explore the possibilities offered by the action of G.T.G in generation of a robust substitution box. We adopt the structural framework and nomenclature as presented in previous section. The matrices A and B, corresponding to the transformations $\bar{x}$ and $\bar{y}$, respectively, facilitate the construction of the coset diagram representing the action which are obtained from the following equations.

Let us consider the action of $PGL\left( 2, \mathbb {Z}\right)$ on $PL\left( F_{2^{8}}\right)$ for $\theta =\alpha ^{2\,}$ with $w(x,y)=xyxy^{2}$, then the parametric equations 15, 5, 6, 7, 8, and 11 transforms into.

$$\begin{aligned} & \det \left( X\right) =a^{2}+kc^{2}=\Delta \ne 0\text {, }d^{2}+d+kf^{2}+1=0. \\ & r=c^{2}k\text {, }s=ac\text {, }r^{2}+ks^{2}=\Delta r\text {, }\theta =\frac{ r^{2}}{\Delta ^{2}}. \end{aligned}$$

Take the characteristic polynomial $f\left( \alpha \right) =\alpha ^{8}+\alpha ^{4}+\alpha ^{3}+\alpha ^{2}+1$ of $F_{2^{8}}$, then parameterized action provides the values of $\bar{x}$, and $\bar{y}$ which are:

$$\begin{aligned} \bar{x}\longmapsto & \frac{\alpha ^{140}\left( z\right) +\alpha ^{128}}{ \alpha ^{128}\left( z\right) +\alpha ^{140}} \\ \bar{y}\longmapsto & \frac{\alpha \left( z\right) +\alpha ^{99}}{\alpha ^{99}\left( z\right) +\alpha ^{25}}. \\ \end{aligned}$$

where $\bar{x}$ have one fix point 1 and $\bar{y}$ have two fix points 105 and 141:

$$\begin{aligned} & \bar{x}= (0,125), (1,1), (2,208),\ldots , (233,253), (237,251) \\ & \bar{y}= (0,137,67), (1,205,125), (4,248,155), \ldots , (218,233,186), (230,255,237) \end{aligned}$$

The associated coset graph $D\left( \alpha ^{2},2^{8}\right)$ verifying the presentation $\langle x,y\mid x^{2},y^{3},\left( xyxy^{2}\right) ^{n}\rangle$ is given below (Fig. 2):

Fig. 2

Coset graph of GTG for $GF(2^8)\cup {\infty }$.

At this juncture, we have the flexibility to examine any exponent of w apart from n, provided that it adheres to the condition $w^n=1$. In particular, we will focus on $w^2$, which has the following form:

$$\begin{aligned} \left( z\right) \left( xyxy^{2}\right) ^{2}=\frac{\alpha ^{104}+z(\alpha ^{234})}{\alpha ^{27}+z(\alpha ^{232})} \end{aligned}$$

The permutation of $w=\left( xyxy^{2}\right) ^{2}$ on all elements of $\mathbb{G}\mathbb{F}(2^8 \cup \infty )$ is then $w=\left( xyxy^{2}\right) ^{2}: (\alpha ^{143},\alpha ^{159},\alpha ^{142},\ldots ,\alpha ^{103})(\alpha ,\alpha ^{240}, \alpha ^{223}, \ldots ,\alpha ^{138}) (\alpha ^{4},\alpha ^{163},\alpha ^{185},\ldots ,\alpha ^{238})$ $(\alpha ^{8},\alpha ^{104},\alpha ^{113},\ldots ,\alpha ^{250}) (\alpha ^{2},\alpha ^{187},\alpha ^{121},\ldots ,\alpha ^{50},\infty ) (\alpha ^{211})(\alpha ^{171})$ Which has five cycles of 51 length and two fixed points, making up a total of 257 elements. The singularity of the transformation $w^2$ occurs at $\alpha ^{50}$, which needs to be addressed in a system that does not allow $\{\infty \}$. Therefore, for $\mathbb{G}\mathbb{F}(2^8)$, the value of the function at $\alpha ^{50}$ is $\frac{\alpha ^{234}}{\alpha ^{232}}$, which is equivalent to $\alpha ^2$. Since there are two fixed elements in w, $\alpha ^{171}\equiv 179$ and $\alpha ^{211}\equiv 178$, we interchange both of them to eliminate fixed points in the proposed S-box. The use of a modular group action to parametrize the S-box results in a significant improvement in the S-box’s cryptographic properties as compared to classical approaches to S-box construction. In a classic setting (e.g., creating S-boxes based on affine or inversion functions), the S-box’s permutation structure is fixed, which means the permutation will have predictable algebraic symmetries. A large number of S-boxes can be created by varying the parameter $\theta$ in the parametrized action of PSL(2, Z) on $PL(F_p)$, leading to a family of coset diagrams indexed by the applied parameters ($\theta$). Each parameter produces a different associated generalized triangle group with different cycle lengths, branchings, and conjugacy classes, leading to an increased design space. Because of this increase, structural regularities that could otherwise be exploited do not exist in the resulting S-boxes. In contrast, the parametrized S-box construction can produce S-boxes with increased nonlinearity, decreased differential uniformity, and reduced likelihood of linear approximations compared to standard S-box constructions (classical methods). As a result of the use of parameters, additional entropy can be generated at the time of S-box construction, leading to confusion and a higher level of resistance to algebraic, differential, and linear cryptanalysis. The pseudo code for construction of S-Box is:

Input: Irreducible polynomial P(x) over $GF(2^8)$, parameter $\theta$

Output: $16 \times 16$ S-box S

1. Construct $GF(2^8)$ using polynomial P(x)

2. Choose matrices X, Y in PGL(2, q) satisfying $x^2 = y^3 = 1$

3. Computer $W=xyxy^2$

4. Apply parametrization using $\theta$ to compute r, s from equations (6-11)

5. Solve for transformation maps:

$\bar{x}(z)=\frac{az+b}{cz+d}$

$\bar{y}(z)=\frac{ez+f}{gz+h}$

6. Generate the coset diagram $D(\theta ,q)$

7. Compute permutation cycles of $w^2 = (xyxy^2)^2$

8. Replace any singular points ($\infty$) deterministically also eliminate fixed points

9. Map elements of $GF(2^8)$ to integer positions 0-255

10. Form the S-box by assigning $S[i] = w^2(i)$

Return S

Using these steps the transformation and final sbox are presented in Tables 1 and 2.

Table 1 Transformation $w^2$ on $GF(2^8)$ and S-box element mapping.

Table 2 Proposed S-box.

The mapping of GTG elements to S-box index is a deterministic function, where each $\alpha _i$ has a unique integer corresponding to its polynomial basis representation. The cycle structure of $w^2$ generates a permutation of $GF(2^8) \cup \{\infty \}$, where the invariance of singularities at $\alpha _{50}$ and fixed points ($\alpha _{171}$ & $\alpha _{211}$) creates a unique solution to reproduce results without any randomization.

Examination of S-Box

NonLinearity

The incorporation of an S-box is indispensable to introduce a controlled level of entropy within data, safeguarding it against potential security breaches orchestrated by unauthorized entities. Let $n$ and $m$ represent two positive integers, whereby the functions from $F_2^n$ and $F_2^m$ are denoted as $(n, m)$-functions. When equipped with function $F$, the corresponding Boolean functions $f_1, f_2, \ldots , f_m$ are defined for each $x \in F_2^n$ as $F(x) = (f_1(x), f_2(x), \ldots , f_m(x))$, and are referred to as the coordinate functions of $F$.

For a Boolean function $H: F_2^n \rightarrow F_2^m$ (alternatively referred to as an $(n, m)$-function or S-box), the nonlinearity indicator $NL(H)$ is formally expressed as

$$\begin{aligned} NL(H) = 2^{n-1} - \frac{1}{2} \max _{(v \in F_2^{m*}; w \in F_2^n)} \left| \sum _{x \in F_2^n} (-1)^{v \cdot H(x) + w \cdot x} \right| ,\end{aligned}$$

where $u \cdot v = \sum _{i} u_i v_i$ represents the conventional dot or inner product between $u$ and $v$, in accordance with the customary notation for dot products in $F_2^m$ and $F_2^n$. Nonlinearity serves as a metric to assess the degree to which functions resist linear attacks and, to some extent, fast correlation attacks when employed as filters or combiners within stream ciphers. Highly nonlinear S-boxes can effectively promote disorderliness in the data, as the nonlinearity indicator remains closely associated with the linear attack of block ciphers, thus representing a generalization of nonlinearity indicators for Boolean functions transitioning from $F_2^n$ to $F_2^m$. For further comprehensive insights, please refer to³⁰.

Strict avalanche criterion

The notion of the strict avalanche criterion (SAC) was initially introduced by Webster and Tavares in their work³¹. This concept was subsequently extended into the propagation criterion (PC) by Preneel et al. in their studies^32,33. The SAC and its generalizations are founded upon the examination of the derivatives of Boolean functions, which portray the behavior of a function when specific input coordinates are complemented. Consequently, they are closely related to the property of diffusion within cryptosystems utilizing the said function, with a particular emphasis on Boolean functions employed in block ciphers.

The avalanche outcome is observed when a singular alteration in the input leads to a sequence of changes throughout the entire substitution-permutation network, resulting in nearly 50% of the output bits undergoing transformation due to the variation in a single input. This behavior is significant for the evaluation of cryptographic functions in ensuring their effectiveness and security.

Bit independence criteria

It was Webster and Tavares³¹ who initially suggested the bit independence criterion (BIC). It checks whether the flip of one particular input bit affects any output bit independently. In other words, no single input bit must dominate or predict the changes of the output bits. A good BIC is supposed to make the S-box behave in a random manner, which is highly essential for security in encryption processes. For more information about BIC, refer to³⁴.

Differential uniformity

When appropriate circumstances are met, the Substitution box functions as a non-linear part of the encryption algorithm and displays differential uniformity. Changes in the input are detected at the intermediate phases at the output of the substitution-permutation system. The differential uniformity value is an indicator of how comparable the function appears for all input differences $\Delta \alpha$ as well as output differences $\Delta \beta$³⁵. A strong replacement box should have consistent differential properties. The following formula is used to determine the differential uniformity (DU):

$$\begin{aligned} DU = \max _{\Delta \alpha \ne 0, \Delta \beta } \left[ \# \{\alpha \in Y \mid S(\alpha ) \oplus S(\alpha \oplus \Delta \alpha ) = \Delta \beta \} \right] \end{aligned}$$

Note that an S-Box is more resistant to differential attacks if its differential uniformity score is lower.

Linear approximation probability

A minimal linear probability (LP) value for an S-Box is highly desirable in order to counteract linear crypt-analysis attacks. The tuple $(\mu , \gamma )$ defines a linear approximation $\phi$ for a function R(k, X), which for given X and k, satisfies the equation $\mu X \oplus \gamma R(k, X) = 0$. Probability p, which expresses the possibility that $\phi$ holds for a uniformly chosen X, is assigned to each linear approximation.

Differential attack probability

The DAP of an S-box measures the probability that a given input difference (input differential) will lead to a specific output difference (output differential) after passing through the S-box. In other words, it quantifies how likely it is for a specific input difference to produce a desired output difference. NL for nonlinearity, LAP for linear approximation probability and DAP for differentia approximation probability.

Table 3 Comparative analysis of S-Box.

The comparative evaluation of the proposed S-box has been completed by a comprehensive analysis across multiple attributes against other well-known S-boxes in Table 3 that include nonlinearity, strict avalanche criterion, bit-independent criterion, differential uniformity, linear approximation probability, and differential approximation probability. Based on an NL of 112, which was the highest value achieved in comparison to all other resulting values for the suggested S-box, the proposed S-box demonstrated the strongest resistance to linear cryptanalysis based on the current S-boxes reviewed by this researcher. The proposed S-box achieved a SAC value of 0.5234, which also mirrored a closer approximation to the optimal SAC value of 0.5 than many existing S-boxes, demonstrating a superior ability to generate the avalanche mechanism that is the basis for cryptography. The proposed S-box achieved a DU value of six, which was consistent with a number of state-of-the-art S-boxes, confirming its strength against differential attacks. The values for LAP and DAP achieved by the proposed S-box were significantly less than the similar achieved values reported for either of the S-boxes reviewed. Overall, the results of the comparative analysis indicate that the proposed S-box provides superior cryptographic properties across multiple evaluation criteria, making it suitable for use in secure image encryption and other lightweight cryptographic applications. We calculated all cryptographic metrics (NL, SAC, DU, LAP, DAP) using one S-Box generated at $\theta =\alpha ^2$; however, since parametrisation generates various S-Boxes, we plan to calculate average limitations over multiple $\theta$ values in future research.

Application of S-Box in image encryption

The encryption of images has become a crucial concern in the realm of information security, with the primary goal of preserving the confidentiality and legitimacy of image data. Strong and efficient encryption techniques are crucial in an era where digital images are used extensively. These techniques guard against unwanted access, modification, and interception of image data while it’s being transmitted and stored. For a variety of reasons, image encryption techniques have become more popular in recent years. For grayscale and RGB photos, there are four types of encryption: symmetric, asymmetric, chaotic, and transform-based. Grayscale images must be handled securely since they contain important data and are utilised in digital art, satellite observation, and medical imaging. A grayscale image can be made unreadable by applying a mathematical modification to it using a secret key. Using a secret key, grayscale encryption of images converts the original image into an unintelligible format. This section presents a pn-generated S-Box-based picture encryption technique. For this first columns of image based matrix are shifted by using S-Box. Then substitution of entries is performed by using S-Box as look up table. Finally a recursively mixing operation is performed that involves S-Box. The step wise procedure is given in the⁴⁰ algorithm.

In order to illustrate the suggested S-Box implementation strategy, we used Matlab R2018b to run the previously mentioned Algorithms 1 and 2 and applied them to the tree image, as seen in the figure below. Figure 3a shows the original tree image, Fig. 3b shows the encrypted tree image which is constructed using above mentioned algorithm; and Fig. 3c shows the decrypted tree image. The Fig. 3a was taken from google and it was created by Alicia Napierkowski. The permission was granted by Alicia to incorporate in this work.

Fig. 3

(a) The original tree image (b) encryption result of tree image (c) decrypted image.

Key space analysis

One important factor in a cryptographic system is the quantity of the set of possible keys. It should be big enough to keep the system from being broken by brute force attacks. A cipher’s functionality is managed by keys, which ensure that only those with the right key can decrypt data into unencrypted. All widely used ciphers are open access or are built on well-known algorithms, therefore, the only factor affecting system security-assuming there isn’t an analytical attack as well as the key isn’t widely available-is how hard it is to acquire. We have synthesised the S-Box in the suggested encryption technique using the generalized triangle. A user must attempt 256! distinct S-Boxes in order to decrypt data if they are unfamiliar with the employed S-Box. This results in an enormous key space. As a result, the suggested method offers a big enough key space to fend off brute force attacks.

Key sensitivity analysis

The secret keys of an image encryption scheme are very important and should affect the encrypted result completely even if a single-bit changes in any of them. Even a small variation in the encryption key should prevent the encrypted image from being properly decoded.

Majority logic criterion (MLC)

The majority logic criterion (MLC)⁴¹ combines five different analyses, namely, entropy, homogeneity, contrast, energy, and correlation. MLC investigates the statistical properties of substitution boxes in the encryption process. Encryption introduces distortion in the plain-text, and exploring the statistical properties through various tests. The MLC investigation states that these analyses are used on cypher images that come from various S-Box transformations.

Correlation

Correlation for am image can be defined as⁴²:

$$\begin{aligned} \text {Corr.} = \frac{(a - \mu _a)(b - \mu _b)}{\gamma (a, b) \phi _a \phi _b}, \end{aligned}$$

(17)

where $\mu$ as well as $\phi$ stand for the variance and standard deviation, respectively, and a and b designate the locations of the image pixels. $\gamma (a, b)$ indicates the pixel value at row a as well as column b of the image matrix. The degree of similarity between adjacent image pixels over the whole image is measured by the correlation analysis. Its range is [-1,1], where optimal correlation is indicated by a correlation value of 1.

Entropy

The definition of the image’s entropy is:

$$\begin{aligned} \text {Entropy} = -\sum _{a,b} \text {pr}(\gamma (a, b)) \log _2 \text {pr}(\gamma (a, b)), \end{aligned}$$

(18)

where $\text {pr}(\gamma (a, b))$ is the likelihood of the image pixel, a and b denote the positions of the image pixels, and $\gamma (a, b)$ is the value of a pixel at row a as well as column b of the image matrix. Entropy, which ranges from [0,8] for a picture with 256 gray-scale levels, represents the image’s degree of ambiguity. Uncertainty increases with increasing entropy values.

Contrast

The definition of the image’s contrast is:

$$\begin{aligned} \text {Contrast} = \sum _{a,b} |a - b|^2 \gamma (a, b), \end{aligned}$$

(19)

The amount of a pixels at row a as well as column b of the image matrix is represented by $\gamma (a, b)$, where a and b denote the relative positions of the image’s pixels. To locate items in an image’s appearance, contrast analysis is used.

$$[0,(size(Image)-1)^{2}]$$

is a spectrum of contrast levels. The contrast measurement for an image that is constant is 0.

Homogeneity

The definition of the image’s homogeneity as⁴²:

$$\begin{aligned} \text {Homo.} = \sum _{a,b} \frac{\gamma (a, b)}{1 + |a - b|}, \end{aligned}$$

(20)

where a and b denote the locations of the pixels in the image. The homogeneity analysis measures how close the pattern of distribution is to the GLCM diagonal in the GLCM. Homogeneity values fall within the range of [0,1].

Energy

The definition of the image’s energy is:

$$\begin{aligned} \text {Energy} = \sum _{a,b} \gamma (a, b)^2, \end{aligned}$$

(21)

where a and b denote the locations of the pixels in the image. The GLCM’s sum of squared elements is provided by the energy analysis. A constant in an image has an energy value of 1, with the other values ranging from 0 to 1.

The results of the aforementioned studies of proposed image encryption scheme is presented in Table 4.

Table 4 Comparative analysis.

Correlation analysis

Correlation analysis is used in image encryption to compare photos with ciphertext and plaintext. Determine the relationships that run horizontally, vertically, and diagonally between the images. The similar rows of the ciphertext and plaintext images are compared using horizontal correlation. Since higher horizontal correlation maintains the relationship between rows in the plaintext picture in the cipher text image, it is indicative of worse encryption. Columns in images with plaintext and ciphertext are compared using vertical correlation. More vertical correlation indicates worse encryption because the cipher text image retains the column connections from the plaintext picture. The analogy or distinction between the identical diagonal in the encrypted text image and the plaintext picture is measured by diagonal correlation. Since the cipher text image retains the diagonal relationships from the plaintext picture, a strong diagonal correlation indicates worse encryption. Safe encryption is not guaranteed by low correlation. To fully assess the encryption system, other attack types including differential assaults and linear cryptanalysis need to be looked at as well. To obtain the CC in the horizontal, vertical, as well as diagonal directions, we apply the following formula:

$$\begin{aligned} \ C_{r} = \dfrac{(n\sum _{t=1}^{n}\mu _{t}\nu _{t} - \sum _{t=1}^{n}\mu _{t} \sum _{t=1}^{n}\nu _{t})}{(n\sum _{t=1}^{n}(\mu _{t})^2 - (\sum _{t=1}^{n}\mu _{t})^2)((n\sum _{t=1}^{n}(\nu _{t})^2 - (\sum _{t=1}^{n}\nu _{t})^2)}. \end{aligned}$$

(22)

Here $\mu _{t}$ and $\nu _{t}$ are values of neighboring pixels in the image and the total amount of pixels used to calculate CC is n. The CC for our scheme with previous schemes are presented in Table 5 in horizontal, vertical and daigonal direction.

Table 5 Two adjacent pixels in the Plain and Cipher image’s CC.

Correlation is analyzed in the neighboring pixels of the plainimage in Fig. 4a and also in adjacent pixels of cipherimage through Fig. 4b. From the results given in Table 5 and Fig. 4, it is evident that cipherimage has a very low quantity of correlation among the adjacent pixels. Hence cipherimage do not reveal any information about the structure of plainimage.

Fig. 4

(a) Correlation among neighboring pixels in tree image (b) Correlation among neighboring pixels in cipherimage.

Histogram analysis

Using histogram analysis, one can determine an image’s statistical characteristics by examining the distribution of pixel intensity. It entails creating a histogram, a graph that illustrates the distribution of colour strength. By analysing the histograms between the unmodified and confidential photographs, histogram analysis can be utilised in the context of picture encryption to determine the security of an encryption technique. It is feasible to ascertain if the algorithm used for encryption maintains or modifies the pixel intensity range by looking at the histograms. The encryption method preserves the statistical characteristics of the image and is safe if the encoded image’s histogram resembles that of the original. Conversely, if the histogram of the protected image differs noticeably, then the security is compromised. Histogram of an image displays the dispersion of pixels in it. By plotting the pixels of image one can analyze the shape of histogram as dispersion of pixels (Fig. 5).

Fig. 5

Histogram of tree image (b) Histogram of the corresponding cipherimage.

Figure 3a displays the plain image histograms, while Fig. 3b shows the equivalent encrypted image components. It is evident from the examination of these histograms that it is not possible to launch an empirical attack on the image that has been encrypted.

Entropy analysis

An image encryption algorithm’s security and reliability are at risk if the cipherimage’s entropy value, which is derived from the formula above, is less than 8. This indicates that there is a possibility of plainimage predictability. This encryption technique computes the entropy for the cipherimage g. The Table 6 displays this cipherimage g’s entropy value.

Table 6 Entropy values comparison.

Differential analysis

Differential analysis contains two type of analysis: “number of pixel change rate (NPCR)”and “unified average changing intensity (UACI)”. For detail of NPCR and UACI refer to⁴⁰. In order to withstand divergent attacks, NPCR as well as UACI values should be high and close to optimal.

Table 7 NPCR and UACI of proposed scheme.

The experimental findings in Table 7 demonstrate that the suggested strategy achieves good performance for both UACI and NPCR. As such, it will provide strong defence against both “known plain text attacks”and “selected plain text attacks”. The forecasts made by the model are, on average, fairly close to the actual values, as seen by the peak at zero. This is encouraging since it shows that for a sizable chunk of the data, the model is producing precise forecasts. The short span of the histogram suggests that errors are concentrated in a narrow range of values. This indicates that the model’s predictions have relatively low variance, meaning it consistently performs well across most data points. The fact that the histogram tapers off quickly from the peak implies that extreme errors (both positive and negative) are infrequent. This indicates that the model is not making highly erroneous predictions and that it is robust in handling most cases. The total frequency or count of errors, which reaches a maximum of 16000, provides an understanding of the total number of errors present in the dataset.

Time complexity

Because all computations for the proposed S-box are conducted in the fixed field $GF(2^8)$, and because all operations consist of fixed-size matrix transformations that are mathematically created based on the action of the parameterized GTG, the performance of the proposed method is very efficient. Once the parameters have been computed, the most intensive operation is an evaluation of the transformation $w^2(z)$ applied to the 256 elements in the field. Each time $w^2(z)$ is evaluated it only requires a fixed number of finite field multiplications and additions. Thus, the total amount of time needed to create one complete S-box will increment linearly with the field size, i.e., O(256). The amount of space needed to store one full representation of 256 elements will also be O(256), because only a single 256 element permutation must be stored. During calculation, no extra large matrices, lookup tables, or coset diagrams need to be kept. As a result, the proposed parametrized GTG approach is lightweight and can be used in real-time and resource-limited environments for encryption development. The complete encryption process including S-box generation, and all rounds of substitution and diffusion take about 25 seconds on a standard desktop machine using MATLAB R2018b.

Conclusion

We extend the parametrization of the action of the modular group $PSL(2,\mathbb {Z})$ for the triangle groups $\Delta \left( 2,3,k\right)$ to the finite generalized triangle groups $\langle x,y\ |x^{2}=y^{3} =w^{k}=1\rangle$ where w is a word in x, y. Also, coset diagrams of the action of $PSL\left( 2,\mathbb {Z} \right)$ on a $PL\left( F_{p}\right)$ are obtained, through parametrization, which yields one of the eight finite generalized triangle groups which are homomorphic images or quotients of $PSL\left( 2,\mathbb {Z} \right)$, particularly the group $\langle x,y\ |x^{2}=y^{3}=w^{k}=1\rangle$ of order 24, where $w=xyxy^{2}$. Other than this we analyzed the coset diagrams for the parameter $\theta =0$ for three finite generalized triangle groups $\langle x$, y $\mid$ $x^{2}$, $y^{3}$, $\left( xyxyxy^{2}xy^{2}\right) ^{2}=1\rangle$, $\langle x$, y | $x^{2}$, $y^{3}$, $\left( xyxy^{2}\right) ^{2}=1\rangle ,$ and $\langle x$, y | $x^{2}$, $y^{3}$, $\left( xyxyxy^{2}\right) ^{2}=1\rangle$. The provided data shows that the S-box generated from the action of generalized triangle group on projective line $PL(F_{2^{8}})$ exhibits strong cryptographic properties. It is bijective, has high nonlinearity, and meets the avalanche effect and related-key security criteria. Additionally, it has low DU and LP, which contribute to its resistance against differential and linear cryptanalysis, respectively. The S-box’s Bent-NF value also indicates optimal non-linearity, enhancing its security further. According to empirical data, the GTG-based S-box compares favourably to S-boxes created using chaotic maps or polynomials. The S-box’s entropy value is close to the ideal 8, indicating a high level of randomisation, and the correlation factors for encrypted images (horizontal, vertical, diagonal) approach zero, providing better performance than a number of comparable methods documented in the academic literature. This enhanced statistical behaviour may be attributed to the increased algebraic complexity and non-linear permutation cycle structure created by the parametric group action, which provides a greater degree of confusion and decreases the ability to predict the resulting encrypted output. The choice of the Generalized Triangle Group in the S-box design demonstrates a sophisticated approach, leveraging the symmetries and transformations provided by this mathematical structure to ensure robust encryption and data protection. Overall, the S-box appears to be well-designed and suitable for cryptographic applications, providing a robust foundation for secure encryption and data protection.

Planned extensions with future applications of chaotic key scheduling and hybrid algebraic-chaotic permutation will have their own capabilities. For example, GTG-Based permuted elements will be the static nonlinear layer and the Chaotic Maps will be used to dynamically change the values of ($\theta$ and matrix entries), thereby increasing the unpredictability and sensitivity of the keys. Also, the proposed system is entirely algebraic however, it has sufficient flexibility to allow for future development towards hybrid cryptography. Chaotic maps or key-dependent dynamic perturbations can be implemented in the parametrization phase to alter the group parameters or linear fractional transform on-the-fly. This hybridization would create an even more unpredictable result and provide a greater key sensitivity, as well as better protect against algebraic, differential, and statistical attacks. Thus, further research should explore adding chaotic or hybrid methods to the GTG-based model.