Introduction

HMI have great potential in areas such as the Internet of Things, the metaverse, intelligent healthcare and robotics, due to their ability to collect, transmit and share human input across time and space1,2,3,4. Touch-based interaction5,6,7,8,9 has attracted particular attention for its intuitiveness, high information throughput, and strong sense of realism among various interaction modes, such as speech10,11, gesture12,13, and touch14,15,16,17. However, traditional HMI typically consist of M × N sensor arrays. As the number of sensing elements increases, this wiring approach requires a large number of electrodes, inevitably leading to complex circuit design, significant data redundancy, high power consumption and other related issues.

To resolve this issue, the existing research has explored innovative electrode designs18,19,20,21,22 and alignment strategies21,22,23,24. One approach defines sensing units as the intersecting nodes of row and column channels, reducing the number of electrodes from M × N to M + N. Another method uses combinations of lattice-like patterns, increasing the information capacity by modulating and outputting dynamic signals in space and time without increasing the number of channels. Meanwhile, simple structures with single functions are evolving into high-resolution arrays with multifunctional integration in the development of touch-based HMI. To achieve functional diversity while maintaining structural compactness, researchers have continued to simplify electrode designs through innovative approaches. For example, Kim et al. have proposed a soft sensor array in which each sensing unit consists of a capacitor placed next to an inductor and a resistor module. Regardless of the number of modules, only two external wires are required to achieve unambiguous detection25. Nevertheless, this approach does not fundamentally reduce the structural complexity. To further reduce the number of electrodes and eliminate the use of discrete components, Liao et al. developed an integrated bio-inspired tactile sensor to enhance hybrid human-machine perception for multifunctional interaction. By utilizing the separate electrical double-layers structure, this approach resolves the complexity of interconnected circuits26. Alternatively, Wei et al. designed an integrated epidermal electronic device based on an addressable electrical contact structure, which allowed the identification of the location of the mechanical stimuli using only two electrodes27. Whereas these approaches reduce component redundancy and interconnect complexity to some extent, resistive28,29 and capacitive30 HMI still depend on external power sources, posing challenges in terms of battery life and energy replenishment during long-term continuous operation31,32. To address this problem, self-powered tactile sensors based on piezoelectric33,34,35,36,37, triboelectric38,39,40,41,42 and electromagnetic43,44,45,46 inductive mechanisms become the focus of research. These sensors typically enable self-powered sensing functionality by converting mechanical energy, including human motion, into electrical signals. For example, Ding et al. proposed a parallel coil array based on electromagnetic induction and magnetized composite materials, which encodes the characteristic frequencies of the induced currents generated by mechanical stimuli, thus enhancing the interaction capabilities47. Guo et al. proposed a triboelectric electronic skin based on the design of an Archimedean spiral structure. Utilizing the triboelectric effect and electrostatic induction between the finger and the dielectric material, the system can achieve detection of 280 sliding directions with only four electrodes48. Unfortunately, these approaches typically rely on the limited energy generated by human movement49,50,51,52, making it difficult to maintain a stable energy output during subtle or rapid sliding actions. This in turn affects the accuracy of signal detection and the stability of system responses53.

With the advent of the electrification era, power-frequency electric and magnetic fields generated by the operation of household appliances, industrial equipment and power systems have become widespread in various environments. Traditionally, these power-frequency electric and magnetic fields have been considered as interference signals. Nevertheless, recent studies have shown that the human body, due to its high relative permittivity and conductivity, can serve as an efficient coupling medium54,55,56. The power-frequency electric and magnetic fields in the environment are converted into the available energy of the electronic equipment by means of body coupling57,58,59. This energy harvesting technique does not rely on mechanical motion and provides stable signal acquisition free from cross-interference effects. It presents a new approach to the power challenges of HMI.

In light of the above, this work proposes a body-coupled minimalist human-machine interface for multifunctional touch detection. The AC electric signal generated by coupling the human body with the power-frequency electric and magnetic field is used as the energy source of BM-HMI, thereby eliminating the need for battery power. Through the S-shaped arrangement of gradient resistive elements60,61 and a detection strategy based on the ratio of relative signal amplitudes, efficient detection can be achieved using only two sensing electrodes with significant signal discrimination across diverse touch and sliding operations. It also demonstrates strong adaptability to environmental changes and user variability. Sections “Design and working mechanism of BM-HMI” and “Body-coupled sensing mechanism utilizes power-frequency electric and magnetic field as energy source” elucidate the recognition and sensing mechanisms of finger mechanical stimuli by constructing a coupled circuit model involving the power-frequency electric and magnetic field, the human body, the BM-HMI, and the data acquisition (DAQ) device. Section “Performance and characteristic of the BM-HMI” evaluates the effects of factors such as resistance value, bending degree, touch frequency, sliding direction, and others on the output signals of the BM-HMI using a comprehensive performance testing platform. Section “Practical applications of BM-HMI” verifies the practical feasibility of the BM-HMI in controlling virtual vehicle, virtual UAV, and robotic leg. This work establishes an innovative and effective technological pathway for the subsequent generation of efficient, intelligent, and sustainable tactile sensing interaction systems.

Results and discussion

Design and working mechanism of BM-HMI

The working process of the BM-HMI is illustrated in Fig. 1a. A significant number of widely utilized electronic devices, including smartphones, computers, and printers, generate power-frequency electric and magnetic fields in their power lines. These low-frequency electric and magnetic fields have been shown to induce quasi-static induction in both air and the human body. In comparison to air, the human body demonstrates significantly higher relative permittivity and conductivity, resulting a substantial potential difference generated on the body, which is considerably greater than that observed in the air under equivalent environmental conditions. When BM-HMI is the sole component present in the electric and magnetic field, a small peak voltage signal is induced across its resistance due to the surrounding medium. Whereas, when a human hand comes into contact with the BM-HMI’s electrodes, the body facilitates the transfer of the induced potential difference from the power-frequency electric and magnetic field to the electrodes through quasi-static induction, thereby generating a larger peak voltage signal across its resistance. Upon contact between the finger and a designated point on the BM-HMI, current is conducted through the touch point to electrode 1 (E1) and electrode 2 (E2), thereby generating voltage signals V1 and V2, respectively. The location of the touch point can be determined by calculating the ratio of the peak voltages V1max and V2max of the voltage signals V1 and V2. The signals processed by the computer can then be used to generate control commands for various mechanical and electronic devices, such as virtual vehicle, UAV, and robotic leg.

Fig. 1: The concept, structure and working principle of BM-HMI.
figure 1

a The working process of BM-HMI. b The planar structure of BM-HMI. c The three-dimensional structure of BM-HMI, with microscope images of the touch point region (left) and region without touch point (right). d A radar plot comparing BM-HMI data with data reported in the literature. e The manufacturing process of BM-HMI. f A digital photograph of BM-HMI

Full size image

Figure 1b, c illustrate the structural composition of BM-HMI. The BM-HMI is composed of three distinct layers: the cover layer, the patterned electrode layer, and the substrate layer. The patterned electrode layer is engineered to the substrate layer, which is composed of two electrodes connected through the S-shaped pattern, SMD resistors, and the shield layer. The cover layer, featuring S-shaped apertures, is constructed to the patterned electrode layer, thus forming compose a HMI with nine touch points. The BM-HMI system utilizes clicking and sliding motions for interaction, exhibiting the capacity to recognize nine distinct touch positions and eight sliding directions. The geometric scalability of the S-shaped electrodes has been demonstrated to increase the number of touch positions and sliding directions by adding more inflection points. This enables BM-HMI to enhance information encoding capacity without expanding the number of channels. The precise dimensional parameters of BM-HMI are delineated in Figs. S1, S2 and Tables S1, S2. Figure 1d and Table S3 present a comparison of the performance of BM-HMI with existing devices, highlighting its advantages in terms of detection limits, environmental adaptability, self-powering capabilities, response time, accuracy, and cycle durability times.

The manufacturing process of BM-HMI is illustrated in Fig. 1e. The patterned electrode layer is created through a series of exposure, development, and etching steps, while the cover layer is formed by laser cutting to create openings. The substrate layer, patterned electrode layer, and cover layer are then thermally pressed and bonded to produce the flexible HMI. Gold (Au) is deposited on the touch points with a view to enhancing the corrosion resistance of the structure. The detailed manufacturing process is provided in the Experimental Section, and the microscopic morphology of the touch points is shown in Fig. S3. The utilization of lightweight materials and state-of-the-art manufacturing methodologies has resulted in BM-HMI exhibiting a thickness and weight of approximately 130 μm and 0.94 g, as illustrated in Figs. S4 and S5. Figure 1f presents a digital photograph of BM-HMI.

Body-coupled sensing mechanism utilizes power-frequency electric and magnetic field as energy source

The body-coupled sensing mechanism utilizes power-frequency electric and magnetic fields as energy source, can be discussed based on the generation process of these fields. These fields are generated by charges oscillating with a frequency of 50 Hz or 60 Hz. This phenomenon is derived from Faraday’s law of electromagnetic induction, which states that any AC in a conductor will generate an alternating magnetic field around the conductor. This can be expressed by the following formula:

$$\nabla \times E=-\frac{\partial B}{\partial t}$$
(1)
$$\nabla \times H=J+\frac{\partial D}{\partial t}$$
(2)

In the Eq. (1), E, B, H, J, and D denote the electric field intensity, magnetic flux density, magnetic field intensity, current density, and electric displacement field, respectively. The symbol ∇ × denotes the curl operator.

The body-coupled sensing mechanism utilizes power-frequency electric and magnetic field as energy, as illustrated in Fig. 2a. In the absence of contact between the body and BM-HMI, the electric and magnetic fields energy from the environment is transmitted through the air, BM-HMI, and the ground. Upon contact between the body and BM-HMI, the electric and magnetic fields energy is transmitted through the body, BM-HMI, and the ground. Following the disconnection of the body from BM-HMI, the transmission path reverts to its original state.

Fig. 2: Sensing mechanism of BM-HMI.
figure 2

a Schematic diagram of the body-coupled sensing mechanism using power-frequency electric and magnetic field as energy. b Equivalent circuit diagram of the body-coupled sensing mechanism using power-frequency electric and magnetic field as energy. c Schematic diagram of the microscale body-coupled sensing mechanism triggered by the finger contact with the BM-HMI. d Schematic diagram of signal transmission in the BM-HMI triggered by the finger contact. e Output signal when the finger slides on BM-HMI

Full size image

Figure 2b shows a schematic diagram of the coupling between power-frequency electric and magnetic field, the human body, BM-HMI, and the DAQ device. In the figure, Cp denotes the coupling capacitance between the power lines in the surrounding environment and the human body. C1 and C2 denote the coupling capacitances between the power lines and the external wires of the DAQ device. It is generally accepted that C1 and C2 are negligible due to their minute size. Cb is defined as the coupling capacitance between the human body and the ground, and Ciso is the coupling capacitance between the floating terminal of the DAQ device and the ground. These coupling capacitances allow weak displacement currents to flow. Upon contact between the human body and the BM-HMI, a displacement current is initiated, flowing into the human body through the coupling capacitance, Cp. A current flows unidirectionally through the BM-HMI into the floating ground end of the DAQ device, and subsequently into the ground through the coupling capacitance, Ciso. Concurrently, another current flows into the ground through the coupling capacitance, Cb, between the human body and the ground.

The microscale body-coupled sensing mechanism of BM-HMI is illustrated in Fig. 2c. In essence, the human body is composed of high-permittivity hydrated tissues (referred to as the body) and insulating keratinized epithelium (referred to as skin). When an ambient 50 Hz alternating electric field with a certain intensity is present, the human body generates an induced potential in the region of the skin in contact with the electrodes through a quasi-static induction mechanism. This process induces charge on the electrode, resulting in the flow of current along the external load to compensate for the potential difference, thereby generating an electrical signal across the external load.

As illustrated in Fig. 2d, SMD resistors R1 to R10 are connected in series to form the gradient resistive elements. E1 and E2 are set up at the positions of R1 and R10 to respectively measure the voltage signals. Nine touch points (P1 to P9) are sequentially distributed at the junctions between adjacent resistors. When there is human body contact with the BM-HMI, the displacement current flows through the body into the BM-HMI. The current is known to flow through the touch points towards E1 and E2, thereby generating voltage signals V1 and V2, respectively. The total voltage loaded on the BM-HMI, denoted as the VBM-HMI, can be expressed as follows (the derivation can be found in Figs. S6 and S7, Note S1 and Table S5).

$${V}_{{\rm{BM}}-{\rm{HMI}}}=\frac{{C}_{{\rm{p}}}{R}_{{\rm{BM}}-{\rm{HMI}}}}{({R}_{{\rm{BM}}-{\rm{HMI}}}+{R}_{{\rm{iso}}}+\frac{1}{2\pi f{C}_{{\rm{iso}}}})[{C}_{{\rm{p}}}+{C}_{{\rm{b}}}+\frac{{C}_{{\rm{iso}}}}{2\pi f({R}_{{\rm{iso}}}+{R}_{{\rm{BM}}-{\rm{HMI}}}){C}_{{\rm{iso}}}+1}]}\cdot {V}_{{\rm{a}}}$$
(3)

In Eq. (3), f is the frequency of the coupled AC source, and Va is the voltage of the power line. It can be observed that Cp, Cb, and Ciso exhibit variation in different environments. This phenomenon results in alterations to the VBM-HMI in response to environmental variations, thereby precluding the possibility of determining the touch position exclusively through measurement of the VBM-HMI value. V1 and V2 are voltage dividers of VBM-HMI, expressed as follows:

$${V}_{1}=\frac{{R}_{1}}{{R}_{1}+{R}_{2}+\ldots +{R}_{{\rm{x}}-1}+{R}_{{\rm{x}}}}{V}_{{\rm{BM}}-{\rm{HMI}}}$$
(4)
$${V}_{2}=\frac{{R}_{10}}{{R}_{{\rm{x}}+1}+{R}_{x+2}+\ldots +{R}_{10}}{V}_{{\rm{BM}}-{\rm{HMI}}}$$
(5)

As demonstrated in Eqs. (4) and (5), the output voltages V1 and V2 exhibit a variation in response to alterations in VBM-HMI. Consequently, it is not feasible to make a direct determination of the touch position from the measured values of V1 and V2. Nevertheless, this can be eliminated by calculating the ratio of V2 and V1, as indicated below:

$$\frac{{V}_{2}}{{V}_{1}}=\frac{x}{10-x}$$
(6)

As demonstrated in Eq. (6), it is evident that the magnitudes of V1 and V2 fluctuate in response to variations in VBM-HMI. Nevertheless, the sensing mechanism proposed in this paper is contingent on the ratio of the peak voltage amplitudes between the two electrodes, this proportional modulation characteristic of BM-HMI ensures the sensing signal is only related to the position of the finger in contact, thus avoiding the influence of variable electromagnetic fields on the output results. This significantly enhances the robustness and reliability of BM-HMI.

Figure 2e shows the signal generated in the electrodes when the finger slides over the touch points of BM-HMI. The blue arrows in the figure represent the direction of energy transfer. In the initial stage, when the finger is a considerable distance from the touch point, the potential difference induced in the body is canceled out by the potential generated by the shielding layer. This results in a small voltage signal with an average peak voltage of 0.025 V. As the finger approaches the touch point, a capacitive effect is generated between the body and BM-HMI. This capacitance facilitates the transfer of charge through the electromagnetic field, thereby inducing a gradual increase in the peak voltage signal. Subsequently, when the finger is placed directly onto the touch point, charge is transferred from the body to BM-HMI, causing the peak voltage signal to sharply increase to 0.1672 V and then stabilize. Finally, as the finger moves away from the touch point, the peak voltage signal rapidly decreases and then gradually stabilizes back to 0.025 V.

Performance and characteristic of the BM-HMI

In order to investigate the effect of different resistance values on the output performance of the BM-HMI, resistors of 1, 2, 5, and 10 MΩ are evaluated. The peak output voltages at E1 (V1max) and E2 (V2max), along with their peak ratio, are analysed. Figure 3a illustrates the experimental findings, which demonstrate an increase in both V1max and V2max with rising resistance values at the same touch position. The underlying rationale for this phenomenon can be elucidated through the application of Eqs. (3), (4), and (5). As the total resistance of the BM-HMI increases, it accounts for a larger proportion of the voltage in the overall body-coupled circuit. As the touch position moves from E1 towards E2, the peak value V2max gradually increases, while V1max decreases. This behavior is attributable to the alteration in the number of resistors in the two branches as the touch position is moved. It is evident that, in consideration of the touch position and resistance values, the peak voltage ratio between E2 and E1 exhibits distinct regional variations. This facilitates the BM-HMI in accurately identifying all touch positions.

Fig. 3: The output characteristics of the BM-HMI.
figure 3

a In the event of the BM-HMI utilizing resistors of differing values, the maximum output voltage V1max at E1, the maximum output voltage V2max at E2, and the ratio of the maximum output voltage V2max to V1max when a finger makes contact with each touch point. b Voltage response signal when a finger contacts the touch points of BM-HMI. (c) The BM-HMI stimuli corresponding to ln(V2max/V1max) for various bending radii at different touch points. d The maximum output voltages (V1max and V2max) at different touch points on the BM-HMI with sixteen points. e The output voltages (V1 at E1 and V2 at E1) when touch P2 on the BM-HMI is subjected to mechanical stimuli at different frequencies. f The output voltage (V1 at E1) when touch P2 and P8 on the BM-HMI are alternately contacted

Full size image

As illustrated in Fig. 3b, the voltage response signal of the BM-HMI is depicted in the 50 Hz power-frequency electric and magnetic field, and the spectral analysis of the voltage signals is shown in Fig. S8. The signal generated is of a sinusoidal nature, with a periodicity of approximately 20 ms. The time required for the voltage signal to rise from the average value to the peak value is approximately 5 ms, indicating that the BM-HMI meets the requirements for rapid detection. In order to investigate the output performance of the BM-HMI under bending deformation, the maximum voltage of the two electrodes was tested in the normal, inward-bent, and outward-bent states, as illustrated in Fig. 3c. The findings suggest that the ln(V2max/V1max) at all contact points of the BM-HMI remains predominantly constant, thereby evidencing the exceptional flexibility of the BM-HMI. To investigate the impact of repeated bending on the electrical output performance of BM-HMI, we performed bending tests using a stepper motor test rig. The results demonstrate that the voltage output of the BM-HMI under different bending cycles with no significant change, as shown in Fig. S9.

Furthermore, the BM-HMI demonstrates good applicability for different users in Fig. S10 and Table S4. The apparatus demonstrates notable robustness and consistent functionality across a range of environmental conditions, as illustrated in Fig. S11. A sliding test rig was constructed for the purpose of assessing the durability of the BM-HMI. This rig connected the human body to a slider via wires that were in contact with the BM-HMI. The speed of the slider was controlled by a stepper motor. The details of the experimental setup are shown in Fig. S12. The BM-HMI demonstrated excellent durability, with a recorded capacity of 400,000 cycles, as illustrated in Fig. S13.

The BM-HMI demonstrates high scalability and has been developed into a version with 16 touch points, as illustrated in Fig. 3d. As the touch position transitions from E1 to E2, the peak value of V1max experiences a gradual decline, while V2max undergoes an increase. The output voltages at different touch points exhibit clear spatial differentiation. By calculating the ratio of V2max to V1max, the output voltage of all 16 points can be distinctly distinguished. As demonstrated in Fig. 3e, the response characteristics of the BM-HMI are examined by applying stimuli at varying frequencies. Dynamic mechanical stimuli are applied to touch P2 of the BM-HMI at frequencies ranging from 1 to 5 Hz. The results demonstrate that the BM-HMI exhibits a continuous and stable dynamic voltage response, showing excellent frequency response consistency. As demonstrated in Fig. 3f, the peak values in the voltage response of the BM-HMI to mechanical stimuli at different positions exhibit highly region-specific characteristics. Brief mechanical stimuli are applied to touch P2 and P8 at one-second intervals, resulting in the rapid and stable generation of V1 with distinct regional differences within the BM-HMI. Furthermore, the touch pressure exerts negligible influence on the output voltage of the BM-HMI in Fig. S14. The BM-HMI exhibits an exceptionally low force detection threshold, operating effectively even at touch pressures as low as 0.04 N, as demonstrated in Fig. S15. The relevant experimental setup is shown in Fig. S16. At the same time, changes in contact area (Fig. S17), ambient temperature (Fig. S18), humidity (Fig. S19), and frequency of electric and magnetic field (Fig. S20) will cause changes in the voltage output characteristics of the BM-HMI, but will not affect the touch position recognition of the BM-HMI. As can be seen from Fig. S21, as the distance increases from 0.05 to 2 m, the peak output voltage of the BM-HMI gradually decrease. This phenomenon is attributed to the reduced coupling capacitance (Cp) between the human body and the BM-HMI as they move farther from the electromagnetic field source. According to Eq. (3), the output voltage (VBM-HMI) also decreases. However, distance between BM-HMI and the power cord does not affect touch localization accuracy in BM-HMI.

With regard to human-machine interaction on different interfaces, the two most common operations are clicking and sliding, as exemplified by those performed on smartphones, touchscreens, and writing boards. The BM-HMI developed in this study is capable of supporting both click-based and slide-based interactions. As demonstrated in Fig. 4a, the BM-HMI with nine touch points can recognize eight different sliding directions of the finger. When the finger is moved from the central touch point in a specific direction, it passes over two different touch points sequentially.

Fig. 4: Recognition of eight sliding directions of a finger on the BM-HMI.
figure 4

a The working principle and the corresponding generated electrical signals of the BM-HMI with eight sliding directions. b–i The output voltage V1 of E1, the output voltage V2 of E2, the ratio of maximum output voltage V2max of E2 and maximum output voltage V1max of E1 when the finger slides in the eight directions

Full size image

In its original state, when the finger approaches the central touch point, the BM-HMI generates a very small voltage signal. When the finger makes contact with the central touch point, charge is transferred from the human body to the BM-HMI, causing the peak voltage signal to sharply increase. It can be demonstrated that, since the number of resistors in the two branches connected to the central touch point is equal, the ratio of the maximum output voltages, V2max and V1max, generated between the two electrodes is approximately 1. Therefore, the central touch point can be used as a reference point for the voltage ratio signal. Upon the withdrawal of the finger from the central touch point, the peak voltage signal undergoes a rapid decline. Subsequent to this, as the finger makes contact with the subsequent touch point, the peak voltage signal undergoes an increase, ultimately it forms a two-waveform voltage signal. To illustrate this, consider the example of the finger sliding from the central touch point to the upper-left touch point, as depicted in Fig. 4b. At the point of contact with the central touch point, the peak average voltage signal generated by E1 is 0.1672 V, and the peak average voltage signal generated by E2 is 0.1658 V. The ratio of V2max to V1max is approximately 1.0088. As the finger is moved towards the upper-left touch point, the peak average voltage signal generated by E1 increases to 0.7132 V, while the peak average voltage signal generated by E2 decreases to 0.08034 V. This results in a V2max to V1max ratio of approximately 0.1126.

The configuration of the S-shaped SMD resistors results in a sequential alteration in the number of resistors within the two branches during the sliding process. Consequently, the BM-HMI generates distinct maximum output voltages, V2max and V1max, as well as their ratio. Therefore, by comparing the ratio of V2max and V1max generated when the finger slides over the touch points twice, the eight different sliding directions on the BM-HMI can be clearly distinguished, as shown in Fig. 4c–i.

Practical applications of BM-HMI

In order to demonstrate the concept, the BM-HMI can be used as a control interface for a variety of applications including entertainment, virtual reality and robotic control. As illustrated in Fig. 5, the BM-HMI has the capacity to function as a control interface for virtual vehicles and UAV. The complete control system consists of the BM-HMI, DAQ device, computer, and virtual vehicle or UAV, as demonstrated in Fig. 5a. Upon actuation of the BM-HMI via the clicking of a finger, the generation of output voltage signal is initiated on both E1 and E2. Subsequently, these output signals are collected by a DAQ device and transmitted to a computer. The computer then analyses and processes the signals in order to generate the necessary control commands. Subsequently, the commands are transmitted to the virtual vehicle and UAV, which then execute the corresponding actions.

Fig. 5: Demonstration of the BM-HMI as a control interface for the virtual vehicle.
figure 5

a Diagram of the BM-HMI control system and the corresponding signal flowchart. b Control interface of the BM-HMI and the corresponding control commands. c Output voltages of the BM-HMI for three motion states of the virtual vehicle (move forward, move left, and move right) and the corresponding output voltages of the BM-HMI

Full size image

As illustrated in Fig. 5b, the BM-HMI generates a series of voltage signals, each corresponding to a specific motion state of the vehicle. These include slow forward, normal forward, fast forward, move right, stop, move left, slow backward, normal backward, and fast backward. For instance, when a finger clicks on a touch point in the upper center, the V1 component generates a larger peak voltage signal, while the V2 component produces a smaller peak signal. Subsequent to processing by the control system, the input signal is matched with the stored signal for the virtual vehicle’s forward motion. Subsequently, the control system issues the forward motion command to the virtual vehicle, thus initiating its forward movement. As illustrated in Fig. 5c, the output voltages of the upper, left, and right touch points of the BM-HMI correspond to the three motion states of the virtual vehicle (move forward, move left, and move right). The complete range of motion control for the virtual vehicle is demonstrated in Video S1. The experimental results demonstrate the system’s capacity to facilitate real-time motion control of the virtual vehicle, thereby offering extensive potential applications in domains such as virtual reality, gaming, and smart home technologies.

The BM-HMI is distinguished by its adaptable design principles and a wide range of interaction methods. In addition to its capacity to control the motion of virtual vehicles through clicks, it can also function as a control interface for virtual UAV through a combination of clicking and sliding interactions. As illustrated schematically in Fig. 6a, the BM-HMI can be employed as a control interface for a virtual UAV. The BM-HMI is affixed to the dorsal aspect of the hand, and when the finger makes contact with the eight different touch points on the BM-HMI (top-left, top, top-right, left, right, bottom-left, bottom, and bottom-right), these correspond to the eight flight actions of the virtual UAV. The sequence of actions to be performed is as follows: first, roll left; second, pitch down; third, roll right; fourth, rotate left; fifth, rotate right; sixth, ascend; seventh, pitch upward; eighth, descend. As illustrated in Fig. 6b, the voltage signals generated by clicking on different touch points of the BM-HMI are shown, in addition to the representative charts of the virtual UAV’s flight actions (pitch down, roll right, and rotate right). Figure 6c presents a schematic of controlling the eight flight actions of a virtual UAV through eight sliding motions. When the finger is moved in eight different directions on the BM-HMI (top-left, top, top-right, left, right, bottom-left, bottom, and bottom-right), these correspond to the virtual UAV’s eight flight actions. The sequence of movements is as follows: left front, move forward, right front, move left, move right, left rear, move back, right rear. As illustrated in Fig. 6d, the voltage signals are displayed when the finger slides in different directions on the BM-HMI. A representative graph of the virtual UAV flight maneuvers (left front, right rear) is also presented.

Fig. 6: Demonstration of the BM-HMI as a control interface for a virtual UAV.
figure 6

a Schematic of the virtual UAV control interface based on BM-HMI, where the BM-HMI is worn on the back of the hand and controls the eight flight actions of the virtual UAV through eight clicking actions. b Representative charts of virtual UAV control by clicking on different positions on the BM-HMI. c Schematic of controlling the eight flight actions of the virtual UAV through eight sliding motions. d Representative charts of virtual UAV control by sliding in different directions on the BM-HMI

Full size image

The experimental findings demonstrate the efficacy of the BM-HMI in controlling the movement of the virtual UAV within a three-dimensional virtual space. Video S2 provides a comprehensive illustration of the utilization of the BM-HMI for the purpose of virtual UAV control. In contradistinction to conventional devices, which necessitate the incorporation of numerous distributed sensing components and a substantial quantity of electrodes, the BM-HMI boasts a minimal electrode configuration while concurrently furnishing comprehensive sensing data by means of a combination of clicking and sliding interactions.

The BM-HMI can be used as a control interface for robotic leg’s movement, as demonstrated in Fig. 7a. The dSPACE controller first collects the output voltage from the BM-HMI and converts it into a digital signal. These digital signals are then encoded by a computer and converted into various control commands for the robotic joint’s hydraulic drive system. Finally, the dSPACE converter converts the control commands into corresponding analog signals, which control the extension and retraction of the joint’s hydraulic drive system, thereby enabling the movement of the robotic leg62,63,64. The experimental setup is shown in Fig. 7b. When the finger slides in eight different directions on the BM-HMI (top-left, top, top-right, left, right, bottom-left, bottom, and bottom-right), eight distinct voltage signals are generated. These correspond to the eight control commands for the robotic leg’s movement. The following movements are defined: top-left movement, upward movement, top-right movement, leftward movement, rightward movement, bottom-left movement, downward movement, and bottom-right movement. Figure 7c shows the ratio of V2max to V1max generated by the BM-HMI as a control interface for robotic leg’s movement. The demonstration is available for viewing in Video S3.

Fig. 7: Demonstration of the BM-HMI as a control interface for the robotic leg.
figure 7

a Flowchart of the BM-HMI as a control interface for the robotic leg. b Scene of the BM-HMI controlling the robotic leg’s movement. c Demonstration of the BM-HMI controlling the robotic leg’s motion and the corresponding output signals of the BM-HMI

Full size image

Conclusion

In summary, this paper proposes a body-coupled minimalist HMI that uses the AC signal generated by coupling the human body with power-frequency electric and magnetic fields as an energy source, thus eliminating the need for battery power. The S-shaped arrangement of gradient resistive elements, in tandem with a detection method based on relative signal amplitude ratios, facilitates highly sensitive position detection and substantial signal differentiation in diverse operations, encompassing tapping and sliding, with a mere two sensing electrodes. This configuration has been exhibited to demonstrate effective adaptability to environmental changes and user differences. The experimental results demonstrate that the system exhibits an ultra-low detection threshold of 0.04 N, a rapid response time of 5 ms, and a durability of 400,000 cycles. The geometric scalability of the S-shaped electrodes has been demonstrated to surpass the wiring limitations of traditional sensor arrays by increasing the number of inflection points, thereby enhancing information capacity without expanding the number of channels. BM-HMI has been successfully applied to the control of virtual vehicle, UAV, and robotic leg. This work innovatively proposes a multi-dimensional input detection system that cohesively integrates environmental electric and magnetic fields energy harvesting, multi-dimensional signal detection, and mixed-reality interaction, establishing a new paradigm for HMI in smart home, IoT and assistive robotics applications.

Experimental section

Fabrication of BM-HMI

First, a 70 μm copper-coated PI film (30 μm thick) was cut into a 5 cm × 5 cm rectangle, and positioning holes were drilled into the PI film to serve as the substrate. The substrate surface was cleaned using an ultrasonic cleaner to remove the oxide layer on the copper surface. A photoresist layer was laminated onto the copper surface, and a film mask was aligned with the positioning holes on the PI film. The designed BM-HMI electrode pattern was transferred onto the photoresist layer through photolithography, with the exposure light activating the electrode pattern. Subsequently, a developer was used to remove the unexposed areas of the photoresist, leaving the exposed pattern intact. The exposed copper layer was etched using an etching solution, and the photoresist was stripped off with a stripping solution, revealing the desired electrode pattern. The patterned copper-coated PI film was then cut into a 5 cm × 5 cm rectangle using an automatic cutter, and an S-shaped pattern was engraved at the designated locations with a laser engraving machine. The processed PI film with holes was placed over the etched S-shaped electrode pattern, and the two layers were laminated under high temperature and pressure to prevent oxidation and short-circuiting. A gold layer was subsequently deposited onto the exposed copper areas to enhance corrosion resistance at the BM-HMI openings. Finally, surface-mount resistors were soldered at the reserved positions using low-temperature soldering, completing the fabrication of the BM-HMI.

Device and system characterization

A thickness gauge (model: 945, JURONG) was used to determine the thicknesses of both the PI film and the complete BM-HMI structure. A programmable stepper motor (model: 57HB56L4-30DB, Haijiejiachuang) was employed for sliding speed control and durability testing. The resistance values of surface-mounted resistors were measured using a digital multimeter (model: DLX-UA9233B, DELIXI). An analytical balance (model: FA2004, Shanghai Xiniu) was utilized to measure the weight of the BM-HMI devices. A NI data acquisition device (model: USB-6001, National Instruments) was applied to collect differential voltage and current signals from the two electrodes. In addition, a push-pull force gauge (model: HP-500, HANDPI) was used to measure the contact forces under different contact sizes. LabVIEW 2024 was employed for signal acquisition, feature extraction, signal conversion, and the development of the graphical user interface (GUI). The configuration of the virtual UAV flight control environment was carried out using MATLAB 2024b. Circuit modeling and simulation tasks were completed with NI Multisim 14.3. Experimental data analysis and image plotting were conducted using Origin 2024.