Introduction

Tactile skin is indispensable in robotic applications as it enables robots to perceive the location, timing, and manner of interactive contact with other objects1,2,3,4,5,6,7,8,9. Numerous tactile sensor designs have been proposed to measure contact on the sensing surface10,11,12,13,14. These designs often utilize various transduction methods such as resistive15,16,17,18,19, piezoresistive20,21,22,23, capacitive24,25,26,27, piezoelectric28,29,30, or triboelectric31,32. A common characteristic of surface tactile sensors is the integration of numerous small sensing units, forming a grid-like pattern on a flat or curved surface14,33. Each sensing unit, known as a taxel, is responsible for detecting interactions in its vicinity34. In typical applications, a high-resolution is desired, which necessitates a large number of taxels35. Increasing the density of smaller taxels and using thinner wires can enhance resolution, but it may lead to drawbacks such as mechanical fragility, electromagnetic noise, and cross-talk between taxels36,37. Technical challenges arise in terms of the physical size of taxels as well as the costs associated with manufacturing and wiring.

Efforts are being made to reduce the number of physical taxels while extracting more tactile information38. It is possible to reduce the number of taxels, by enabling each taxel to cover a larger area, and by combining information from multiple taxels39,40. From this perspective, the concept of super-resolution (SR) sensing is proposed, aiming to solve the inverse problem of inferring tactile information from a limited number of sensors and effectively creating high-resolution virtual taxels. The primary approach leverages properties of the tactile material and combining them with machine learning algorithms to extract meaningful tactile information41. A wide range of materials can be employed, such as, electrical resistance33,41, magnetic flux42, fluid dynamics43, etc.

Table 1 provides a comprehensive comparison of these sensors. M. Boutry et al. 44 and Ge et al. 45 both developed electrical perception methods to localize contact for capacitive tactile sensors with 25 taxels and 4 taxels respectively, both achieving a 1-fold SR factor. SR factor is defined as the effective sensing length per single sensor divided by the sensing accuracy as shown in Table 1. Their sensing accuracy is mainly limited by the physical size of taxels and the manufacturing difficulty. An SR factor of 15.5 is achieved by embedding five magnetometers underneath a soft magnetic skin with machine learning methods46. Sun et al. 34 made a barometer-based sensor array to achieve an SR factor of 32.5. Yan et al. 42 built a magnetic sensor with an SR factor of 42.3 by machine learning methods and corresponding sinusoidal magnetization theories. Park et al. 33 embedded both electrical and acoustic based-taxels to achieve an SR factor of 6 in a 40,000 mm2 area with data processing method. These different taxel arrays can achieve super-resolution sensing on varying sensing areas ranging from 324 mm2 to 40,000 mm2. However, the sensing area per taxel in the extant research is relatively low, meaning that a large amount of taxels are required to cover a large sensing area, and this will lead to a rise in the manufacturing and installation costs as well as maintenance difficulties.

Table 1 A comparison of state-of-the-art tactile sensors and our magnetic skin
Full size table

To address the above limitations, we previously present a wireless flexible magnetic tactile sensor consisting of a multi-direction magnetized film and contactless Hall sensors to achieve an SR factor of 29.6 with 3600 mm2 per taxel47. Here, based on our previous work, we expand the functional surface area from a 3600 mm2 discrete grid to a 48,400 mm2 continuous area by embedding magnetic films into the soft silicone layers (Fig. 1a). Sixteen magnetic taxels are contactless and placed under the film to perceive a 48-dimensional magnetic signal, from which the perception information is extracted with machine learning algorithms. Our work demonstrates a magnetic skin with a high SR factor and large sensing area per taxel, achieving large-area super-resolution sensing while keeping the manufacturing process simple. Benefiting from the high-dimensional magnetic field signals and machine learning methods, the average localization error is reduced by 42.86%, from 2.1 mm to 1.2 mm, and our magnetic skin achieves a sensing area of 3025 mm² per taxel with an SR factor of 45.8. Owing to the specific magnetization arrangement, more dimensions of the effective magnetic field signal, and neural network algorithms, our proposed magnetic skin has surpassed its previous limitations and now possesses the ability of multi-scale object perception and multi-point pressure sensing on the whole magnetic skin. The generalization of this technique allows for the recognition of multi-scale objects as well as the demonstration of real-time continuous pressure-sensing writing touchpad. Furthermore, we construct a human-machine interaction (HMI) system with the proposed magnetic skin, demonstrating its capability of simultaneously controlling two vehicles’ trajectory by pressing the skin with two different balls.

Fig. 1: Structure design and working mechanism of the magnetic skin.
figure 1

a The overall structure of the magnetic skin consists of four layers: upper silicone layer, magnetic film layer, lower silicone layer, and sensor array layer. The perception process of magnetic skin consists of two steps: b Deformation of the magnetic skin with pressure which results in a 48-dimensional magnetic field signals; c Real-time signal processing with machine learning algorithms to identify the contact force. d Illustration of the analogy between the human skin and magnetic skin. Three demonstrations of magnetic skin functionalities: e Multi-scale object recognition. f Real-time writing touchpad. g Intelligent car path mapping.

Full size image

Results

Magnetic Skin Design and Fabrication

Figure 1a shows the structure of the proposed magnetic skin, including four layers. The upper silicon layer (220 mm × 220 mm; thickness, 1 mm) is a flexible layer made of soft silicone elastomer to protect internal flexible magnetic films and integrate the dispersed magnetic films into an entirety. The middle magnetic film layer consists of twenty-five small magnetic films (37.5 mm × 37.5 mm; thickness, 1 mm) with specific magnetized arrangements. The lower flexible silicone layer (220 mm × 220 mm; thickness, 10 mm) is also made of flexible silicone material and consists of twenty-five small cavities, which can secure the twenty-five small magnetic films in place and provide a considerable range of deformation for the magnetic films. The bottom sensor array layer consists of sixteen magnetic Hall sensors. Each Hall sensor (MLX90393; Melexis) can detect a three-dimensional magnetic field signal, allowing for the real-time acquisition of 48-dimensional magnetic signal. The sensor array layer is placed 30 mm under the magnetic film layer in a contact-free manner. The fabrication process and material composition of each layer can be referenced in Supplementary Fig. 2 and Supplementary Note 2. When an external force is applied to the top layer of the magnetic skin, the magnetic field lines of the middle magnetic films will be displaced due to structural deformation, which can produce a certain magnetic signal pattern and can be detected by the Hall sensor array, as shown in Fig. 1b. Meanwhile, the real-time 48-dimensional magnetic signal is processed by the computer to extract contact information using machine learning methods, as shown in Fig. 1c.

As illustrated in Fig. 1d, the overall structure of the proposed magnetic skin resembles the human skin. The outermost silicone layer protects the internal sensory layer against the external contact, similar to the stratum corneum of human skin, by providing a barrier against external contact. The middle layer consists of specially arranged magnetic films that function like sensory neurons in human skin, perceiving external stimuli. The bottom sensor array detects the external contact through the wireless transmission of magnetic field signals, and the host computer performs real-time signal processing to extract the contact information, acting like the human brain.

The internal flexible magnetic films are primarily magnetized in opposite directions (Fig. 1d right), and the magnetic field simulation results are shown in Supplementary Fig. 7. Based on our previous work47, this arrangement allows us to determine the approximate region of contact by analyzing the signs of the three-dimensional magnetic signals. The detailed calculation process of location and magnitude of the contact force is presented in Supplementary Note 3 and the following section.

The capability and application of our magnetic skin is demonstrated with several tasks, as shown in Fig. 1e–g. First, the multi-scale sensing capability is illustrated with a multi-scale object recognition task (Fig. 1e). Then, the large-area full-range super-resolution continuous perception capability allows our proposed skin to serve as a writing touchpad for human-machine interaction (Fig. 1f). Moreover, with the multi-point and multi-scale perception capability, our magnetic skin can drive multiple trolleys simultaneously through trajectory mapping, providing much greater flexibility in control system compared with the conventional control with only four directions (forward, backward, left, and right) (Fig. 1g).

Super-resolution sensing ability

Figure 2a, b, c, and d illustrates the procedures of super-resolution perception of our magnetic skin. When the soft magnetic skin is pressed using a pressing head, the change in the 48-dimensional magnetic signal is collected by the 16 Hall sensors located at the bottom layer, and then transmitted to the host computer for localization and magnitude calculation (Fig. 2a).

Fig. 2: Principle of super-resolution pressure sensing and performance evaluation of magnetic skin.
figure 2

The principle of pressure perception in magnetic skin can be divided into four main parts: a The 48-dimensional pressure magnetic field signals corresponding to the sixteen sensors are obtained. b The magnitude of the magnetic field in the z-direction at the pressure location is used to determine the approximate position of the pressure, thereby identifying the rough pressure region. c The original dataset is expanded using the Kriging interpolation method to improve the perception accuracy. d The K-nearest neighbors (KNN) algorithm is used to determine the reconstructed pressure point based on the magnetic field signals, and the result is outputted. e–g Super-Resolution experimental error verification of the pressing position. A spherical head with 4 mm diameter is used to press the magnetic sample in a pattern of ZJU with three different conditions, and we show the localization error of each condition individually. h The relationships between loading force (F) and z-direction magnetic flux density (\({B}_{z}\)) of nearby left-up Hall sensor (Fig. 2h-inset). i The relationships between loading force (F) and z-direction magnetic flux density (Bz) after 500 cycles. j Magnetic skin has achieved a high-precision compression position sensing ability with large sensing area per sensing unit compared with the previous design33,34,42,44,45,46,47,48,49,50.

Full size image

The sixteen different magnetic sensors detect the changes in the magnetic strength (Bz) at corresponding position on the skin. By analyzing the Bz magnetic strength changes from these sensors, it becomes possible to estimate the approximate square location of the pressing point on the skin. This initial estimation helps in determining the general area where the pressure is applied.

Based on the location of the press, the pressing situations can be mainly classified into three categories. Firstly, when the press occurs on the four corners of the magnetic skin, it corresponds to three dimensions of valid magnetic field information generated by the nearby one sensor. When the press happens on the outer four edges of the magnetic skin, it corresponds to six dimensions of valid magnetic field information generated by the nearby two sensors. When the press occurs in the central position of the magnetic skin, the valid magnetic field information extends to twelve dimensions in four sensors as shown in Fig. 2b.

After roughly determining the position, the original pressing data of standard three-dimensional standard magnetic flux density signal (Bx0, By0, and Bz0) obtained from the corresponding small magnetic patches is expanded from a 10 × 10 dataset to a 100 × 100 dataset to improve the precision of super-resolution compressions using the Kriging interpolation method formula as shown:

$${\rm{Z}}(x)={\rm{m}}(x)+{\epsilon }^{{\prime} }(x)$$
(1)
$${\rm{m}}\left(x\right)=\mathop{\sum }\limits_{k=1}^{n}{\alpha }_{k}{p}_{k}(x)$$
(2)
$$\sqrt{{\sum }_{i=1}^{k=n}{({x}_{i}-{y}_{i})}^{2}}$$
(3)

where \({\rm{Z}}(x)\) is the linear universal Kriging result value we want, \({\epsilon }^{{\prime} }(x)\) is a stochastic, \({\rm{m}}(x)\) is the constant trend, \({\alpha }_{k}\) are the local trend or drift coefficients, \({p}_{k}(x)\) are functions of the site coordinates (trend equations) and x is the two-dimensional position vector\((x,y)\). (Fig. 2c).

The final step involves comparing and calculating the Euclidean distance between the input real-time signal and the generated 10,000-point dataset using the Kriging algorithm. This is done using the K-Nearest Neighbors (KNN) algorithm. The dataset with the closest Euclidean distance in the corresponding dimension \(\sqrt{{\sum }_{i=1}^{k=n}{({x}_{i}-{y}_{i})}^{2}}\) is selected, and the corresponding position is output as the reconstructed pressing location of the signal:

$$\Delta {\rm{E}}=\mathop{\sum}\limits_{i=1}^{k=n}{({B}_{{xi}}-{B}_{{xn}0})}^{2}+{({B}_{{yi}}-{B}_{{yn}0})}^{2}+{({B}_{{zi}}-{B}_{{zn}0})}^{2}$$
(4)

where Bxi, Byi, and Bzi are real-time acquired magnetic field signals, and Bxn0, Byn0, and Bzn0 are the premeasured standard magnetic signal datasets. The value of n depends on the number of effective sensor signals. For example, when the pressure is applied on the corners of the magnetic skin, there may be only one adjacent magnetic sensor that provides an effective signal, so n = 1. By setting the value of n, the computational dimension and time can be effectively reduced, thereby improving the real-time perception rate.

To validate the effectiveness of this method, we conducted accuracy measurements for the pressing positions using a 4 mm diameter spherical indenter. The three types of pressing scenarios mentioned earlier were individually evaluated to determine the precision of the pressing positions in the pattern of “Z”, “J”, and “U” as shown in Supplementary Note 5. As the number of effective sensors increases, the dimensionality of the effective magnetic field signals also increases. Consequently, the overall perception error decreases, which aligns with our expectations. The overall average localization error of the magnetic skin is shown in Supplementary Note 8 and Supplementary Figure 11.

In addition to accurately determining the position of the contact spot, it is crucial for tactile sensors to be able to ascertain the magnitude of the contact force. The force magnitude sensing performance of magnetic skin was assessed by recording the magnetic flux density in the z-direction (Bz) under different applied contact forces according to four different sensors. An electric drive XYZ three-dimensional platform (PMS-3), controlled by the platform itself, was used in conjunction with a digital force gauge (Zhiqu, DS2 − 50 N) to apply and measure the contact force as shown in Supplementary Fig. 15 and Supplementary Note 5. The relationship between the applied loading force and the amount of variation in Bz at the nearby left-up position of the pressing position was respectively shown in Fig. 2h. A linear relationship was observed between the loading force and pressing distance with an R-squared value of 0.9900, attributed to the material properties. Furthermore, Bz at the nearby right-up position of the pressing position, at the nearby left-lower position of the pressing position, and at the nearby right-lower position of the pressing position exhibited also strong linear relationship with R-squared values of 0.9871, 0.9752 and 0.9942 in Supplementary Fig. 14. Hence, based on our previous work and these results, the relationship between loading force and Bz changing also exhibits a linear relationship in the whole region on the magnetic skin.

Figure 2i and Supplementary Fig. 16 show the pressing force perception stability and pressing position perception stability of the magnetic skin to the loading force over 500 cycles. Supplementary Fig. 17 also presents the dynamic response of the magnetic skin to the periodically loaded depth of 4 mm over 2500 cycles. The stable and reproducible magnetic signals further confirm the excellent durability of the proposed magnetic skin toward long-term applications. As seen in Fig. 2j, the magnetic skin has a relatively low localization error and quite large sensing area per sensing unit compared with the previous design33,34,42,44,45,46,47,48,49,50.

Multi-scale pressing object recognition

As illustrated in Fig. 3a, the overall process of object recognition consists of two main parts. The first part is data preprocessing, which involves transforming the 48-dimensional magnetic field signals obtained from 16 magnetic field sensors into a 4 × 4 pixel-sized RGB image-like form. The different pixel positions in the image correspond to the locations of the respective sensors, while the RGB color information corresponds to the Bx, By, and Bz magnetic field signal information of each sensor. Before inputting the magnetic field signal data into the neural network model, the data should be standardized to align it with the data standards required for neural network training. By performing this data transformation, the magnetic field signal data can be better utilized and becomes more suitable for subsequent neural network model training. This process facilitates the effectiveness and efficiency of the training process51.

Fig. 3: Multi-scale object press recognition based on convolutional neural networks.
figure 3

a The overall process of object recognition mainly consists of two parts: data preprocessing and CNN classification. b Illustration of the five small-scale pressing heads with their press positions and corresponding effective sensors and c signals. The corresponding (d) 3D, e 2D PCA clustered results, f and the corresponding confusion map of five pressing heads. g Illustration of the thirteen large-scale pressing heads with corresponding forty-eight dimensions signals. The corresponding (h) PCA clustered results of thirteen pressing heads and i enlarged detail view of four pressing heads. and j The corresponding confusion map of thirteen large-scale pressing heads.

Full size image

Moving on to the second step, the transformed data is inputted into the convolutional neural network (CNN) model for forward propagation. The data undergoes a series of processing through layers such as convolutional layers, activation functions, and pooling layers. These layers gradually extract and combine features, ultimately resulting in the network’s output, which determines the shape of the object being pressed.

While considerable research has showcased the potential of leveraging magnetic sensors to develop highly accurate object recognition perception systems with AI analysis46,52, almost all of them have primarily focused on recognizing objects of a consistent size. The multi-scale object recognition ability has not been discussed so far.

Here we selected two different kinds of scale object shapes to test the multi-scale object recognition ability of the magnetic skin. First, we assessed the ability to recognize the pressing of small-scale objects. As depicted in Fig. 3b and Supplementary Fig. 18, we employed 3D printing to fabricate small-scale pressing heads in the shapes of square, triangle, pentagram, ellipse, and rhombus with corresponding same dimensions. The press head dimension is quite smaller than the small magnetic film of the magnetic skin, so the effective magnetic sensors are less than or equal to four. As shown in Fig. 3b, when the five different-shaped pressing heads are pressed onto the magnetic skin at the same position, the corresponding four effective magnetic sensors, as shown in Fig. 3b capture the corresponding signals. The intensity differences in the obtained signals from the four sensors in Fig. 3c allow us to distinguish between different pressing head shapes. A total of 25 pressing positions were considered, with two different pressing depths: 2 mm and 4 mm. For each pressing depth, 50 data points were collected and fed into the CNN where 80% of samples were used for training, 20% of samples were used for testing. After applying principal component analysis (PCA) for feature reduction, the preliminary classification results were visualized in Fig. 3d. The pressing signals from the five different pressing heads are evenly distributed along 25 lines. An example of the pressing signals from one of the lines is illustrated in the two-dimensional plot Fig. 3e. Each pressing head appears as two clusters due to the presence of two different pressing depths. However, the ellipse and rhombus pressing heads have a similar shape, leading to some overlap between the blue and red clusters. As shown in the confusion maps in Fig. 3f, the highest accuracy for small-scale shape recognition is 92.188% mainly because of the similarity of ellipse and rhombus pressing heads. The details of the CNN training process can be found in Supplementary Fig. 19.

For the large-scale shape recognition, thirteen different large pressing heads were used as shown in Fig. 3g and Supplementary Fig. 20. In contrast to the limited effective pressing signal information obtained from small pressing heads, the larger pressing heads, due to their larger contact area with the magnetic skin, provide 48-dimensional effective pressing magnetic field variation signals. These high-dimensional magnetic field variation signals are then inputted into the CNN model to obtain the recognition results. Upon implementing principal component analysis (PCA) to reduce the dimensionality of the features, the 2D preliminary classification results were depicted in Fig. 3h. Due to the increased dimensionality of the effective pressing signals, the data points cluster corresponding to each pressing shape are well separated and distinguishable. The magnified view of closely clustered points is illustrated in Fig. 3i, where the increased magnification reveals a more distinct separation between the clusters. According to Fig. 3j, the highest accuracy for large-scale shape recognition is 100% mainly because of the increase in the amount of effective pressing signal information available.

In summary, the magnetic skin system described can successfully recognize pressing signals from objects of different dimensions. This capability showcases the potential of magnetic skin technology in recognizing and interpreting pressing actions across a range of object dimensions.

High-precision real-time continuous trajectory perception

By combining specific magnetization arrangements and advanced signal processing methods, our magnetic skin achieves super-resolution perception across large areas. Unlike multi-point sensing capabilities achieved through sensor arrays which can only sense the pressing information upon the sensing unit49, our system utilizes real-time signal processing techniques with specific magnetization arrangements to achieve ultra-super-resolution real-time perception as shown in Supplementary Video 1. In the video, the actual finger pressing situation, when pressing downwards, the force exerted by the fingers cannot remain constant like on the three-dimensional platform, resulting in certain force variations, which may cause the pressing signal to have a slight movement.

With the magnetic skin’s wide-ranging, ultra-super-resolution perception capabilities, the continuous sensing of pressing points can be achieved. As demonstrated in Supplementary Video 2, a green ball is continuously moved and pushed on the magnetic skin. By processing the real-time pressing signals obtained, we can derive the real-time motion trajectory of the green ball.

Having demonstrated the ability of our magnetic skin to use specific magnetization arrangements and signal processing methods to achieve continuous ultra-super-resolution real-time perception, several representative daily tasks were subsequently conducted, which include continuous pressing handwriting board demonstration.

The overall process of pressing perception, as shown in Fig. 4a, involves pressing and rolling a ball (15 mm in diameter) on the magnetic skin. The real-time data changes in the magnetic field signals resulting from different pressing positions are transmitted to a computer for signal processing. By processing the corresponding magnetic field signals, real-time pressing points can be obtained. Connecting the obtained pressing points allows for the reconstruction and output of the ball’s movement trajectory on the computer. Figure 4d shows the trajectory paths formed by the movement of a ball on the magnetic skin, forming the letters “Z”, “J”, and “U” respectively. As the ball rolls, the real-time 48-dimensional signals are simultaneously captured by 16 magnetic sensors and processed on a computer. The real-time sensing signal positions and ball movement path comparison are shown in Fig. 4f. In the actual experimental test process, the movement force of the ball is formed by the vertical pressure and lateral push of the hand. There will be lateral thrust, which is different from our standard measurement (the data points are obtained by only vertical pressing), and the error point will be generated, because the pressing force of the hand cannot be kept constant when pressing, the noise point of the pressing signal will be produced and the trajectory of the writing board will be confused. To ensure a clear and comprehensible real-time handwriting effect, the stabilized point signals with three iterations are plotted on a drawing board interface. The consecutive points are then connected to form the trajectory of the ball’s movement. The reconstructed trajectory paths of the letters “Z”, “J”, and “U” based on the ball’s sliding motion are depicted in Supplementary Video 3.

Fig. 4: High-precision real-time continuous pressing perception and Real-time handwriting demonstration.
figure 4

The overall process of the real-time handwriting demonstration on the drawing board includes: a Pressing and sliding the ball on the magnetic skin to obtain the rolling trajectory, b real-time signal processing of the magnetic field signals corresponding to the pressing points to reconstruct the pressing point information, and c connecting the reconstructed points on the drawing board interface to generate the rolling trajectory of the ball. d The rolling path of the ball on the magnetic skin consists of three letters: “Z”, “J”, and “U.” The real-time 48-dimensional signals are processed to obtain reconstructed pressing positions e. The real-time sensing signal positions and ball movement path comparison f.

Full size image

Multi-point and multi-scale perception and intelligent control system

Conventional magnetic sensors typically rely on the magnetic field signal variations caused by the deformation of magnetic films to obtain information about the location and force of pressing47. The ability to perceive multi-point and multi-scale pressing information has not been discussed so far because of the integrity of magnetic materials and the low-dimensional information of magnetic sensors. Benefiting from the combination arrangement of multiple small magnetic pieces in our magnetic skin and the multi-dimensional magnetic field variations that can be obtained from the sixteen magnetic sensors, our magnetic skin enables the perception of multi-point and multi-scale pressing.

For example, when two points are pressed on the magnetic skin, the overall deformation can be considered as a combination of the individual deformations caused by the pressing of each point. Similarly, the magnetic field signal variations can be considered as a combination of the magnetic field signal variations obtained from the pressing of each individual point. By combining the signals obtained from single-point pressings in various configurations, we can generate the magnetic field signals corresponding to multi-point pressings. These signals can then be input into CNN for training to obtain the best model. This model enables real-time multi-point press sensing for subsequent applications. The press sensing capabilities of magnetic skin with two-point, three-point, and four-point presses are showcased in Fig. 5a, b, and c and Supplementary Video 4.

Fig. 5: Multi-point and multi-scale perception and real-time control of multi-cars trajectory demonstration.
figure 5

The demonstration of multi-point pressing, including (a) two-point pressing, b three-point pressing, and c four-point pressing. d Principle Illustration of controlling cars’ movement using magnetic skin. e The movement trajectories of the two balls on the magnetic skin, f the correspondence between the balls, reconstructed positions, and small cars, g the reconstructed positions of the two cars after signal processing, and h the actual routes of the two cars after being driven during dual-car control. i The variation of magnetic field signal strength in the x-direction for the sixteen sensors during the six stages of controlling the two cars.

Full size image

Similarly, by combining the signals of single-point presses of different scales to obtain the magnetic field signals of multi-point presses, and training a convolutional neural network with these signals, we can obtain the optimal model that enables real-time sensing of multi-point and multi-scale presses.

The magnetic skin enables the capability of large-scale, full-range sensing for multi-point and multi-scale presses. As a demonstration, real-time control of multiple small car trajectories has been successfully implemented, showcasing the versatility of the system.

Unlike traditional control methods that rely on four directional buttons (forward, backward, left, and right) to control the movement of the car53, our car control demonstration employs a mapping technique. By accurately mapping the positions of two balls on the magnetic skin, we are able to control the movement of the cars and achieve accurate trajectory mapping.

As shown in Fig. 5d, the magnetic field sensing signals obtained by pressing the ball are inputted to a computer for neural network recognition. The corresponding reconstructed press point information is then fed into an Arduino board, which controls the movement of the corresponding car’s left and right motors for a specified duration. This enables the precise navigation of the small car to reach the desired positions based on the recognized press points. Through this process, our magnetic skin enables the simultaneous and high-resolution control of multiple cars.

The plastic balls used in the experiments are inherently lightweight, and the weight of the balls itself does not cause them to perceptually change deformation on the magnetic skin. The gravity of the plastic ball itself is not within the perceptual range of our magnetic skin. So, if they roll on magnetic skin on their own, it will not cause the magnetic field signal to change, resulting in a press-sensing signal. In the experiment, we placed the plastic ball in the specified position by hand and then pressed it to obtain the magnetic field change signal for signal processing, after obtaining the perception signal, we used the Arduino to drive the left and right motors of the car to control the car to the specified position using the Bluetooth module.

The movement steps of the green ball (15 mm in diameter) and blue ball (10 mm in diameter) controlling the corresponding small cars to specific locations are illustrated in Fig. 5e. The correspondence between the balls, reconstructed positions, and small cars is depicted in Fig. 5f. Figure 5g illustrates the reconstructed positions of the two corresponding balls. The movement trajectories of the two cars are shown in Fig. 5h. The corresponding Bx, By, and Bz magnetic signals of the six-step process control for two cars are shown in Fig. 5i and Supplementary Figure 21. Supplementary Video 5 and Supplementary Video 6 demonstrate single-car control and simultaneous control of two cars, respectively.

This demonstration showcases the real-time, large-area, and multiscale signal perception and processing capabilities of magnetic skin. It can be widely applied in various fields such as human-computer interaction, virtual reality, etc., providing enhanced interactive experiences and control capabilities for users.

Discussion

In summary, we developed a magnetic skin that can acquire super-resolution X and Y coordinates, as well as magnitude information, for pressing actions. The 48400 mm2 continuous multi-point and multi-scale sensing areas and 45.8 SR factor can be achieved by 48-dimensional magnetic signal data from sixteen magnetic Hall sensors combing with machine learning methods.

By integrating a magnetization and arrangement method, expanding the datasets using a linear universal Kriging equation, employing the k-nearest neighbors (k-NN) algorithm to interpret three-dimensional magnetic flux density signals, and machine learning algorithm, our sensor surpasses current technologies. This combined approach enables us to achieve superior performance in determining the super-resolution magnitude and position of contact force. The average error of super-resolution sensing is reduced to 1.2 mm by utilizing higher-dimensional magnetic field signals.

The recognition of multi-scale objects can be achieved by magnetic skin through the combination of magnetic field signals and neural networks. By employing a neural network that utilizes low-dimensional magnetic field signals from neighboring sensors, we can recognize the pressing signals of small-scale objects. Additionally, by combining the overall 48-dimensional magnetic signal data with the neural network, we can identify the pressing of large-scale objects.

With its super-resolution recognition capability and a full range sensing area of 48,400 mm², magnetic skin can enable real-time handwriting applications. By continuously rolling a small ball (15 mm in diameter) on the magnetic skin, the corresponding rolling trajectory of “Z,” “J,” and “U” can be obtained by connecting the corresponding points of pressure perception.

Few discussions about magnetic tactile sensors have addressed the ability to perceive multi-point and multi-scale pressing information due to the integrity of magnetic materials and the low-dimensional information of magnetic sensors. Our magnetic skin overcomes these limitations by employing a combination arrangement of multiple small magnetic pieces and utilizing the multi-dimensional magnetic field variations obtained from sixteen magnetic sensors. As a result, our magnetic skin enables the perception of multi-point and multi-scale pressing. Here, we demonstrate this feature using an intelligent car control system. Unlike traditional control methods that rely on four directional buttons (forward, backward, left, and right) to control the movement of the car53, our car control demonstration introduces a mapping technique. This technique involves accurately mapping the positions of two balls on the magnetic skin, allowing us to precisely control the movement of two cars and achieve accurate trajectory mapping.

Moreover, the magnetic field has the ability to penetrate most materials, allowing for the separation of the magnetic films (perception module) and the Hall sensor array (signal-receiving module). This is unlike traditional wire-connected soft electronic sensors. This characteristic helps to address issues such as the difficulty of repairing damaged sensor arrays and the complexity of wiring.

By altering the ratio of the top and bottom silicone layers, the sensitivity of the sensor can be altered. For example, adding silicone oil during the molding of the bottom flexible silicone layer can adjust the elastic modulus of the substrate, thereby making the magnetic signal changes generated by pressing more sensitive. Due to the low-frequency magnetic field’s transparency through the human body and its non-invasive nature, our sensor has the potential to be applied to human-robot interaction applications. The only rigid components of magnetic skin are the Hall sensors which can be replaced by flexible printed circuits, enabling the development of continuous magnetic skin. This advancement would enhance the safety and reliability of human-machine interactions, as the flexible nature of the sensors would minimize the risk of injury or damage.

In summary, we have developed a large-area super-resolution magnetic skin with multi-point and multi-scale perception ability. The proposed design approach can be further applied to robot large-area tactile sensing skins, providing robots with the ability to perceive tactile information over a large surface area and allow for reliable and safe human-machine interactions.

Methods

Fabrication of magnetic skin

The fabrication method for the magnetic skin is relatively simple. According to our previous work47, the whole process of making magnetic films can be mainly divided into two steps. We first combine NdFeB microparticles (Jianghuai Ciye Corp) with the soft silicone elastomer (Ecoflex 00-30 Part B). The mixture is thoroughly mixed to form the magnetic materials. Using an aluminum mold with a cuboid shape measuring 37.5 mm * 37.5 mm * 1 mm, we stamp the magnetic pastes into consistent shapes with the desired thickness. Next, we take the aluminum mold containing magnetic gel and place it in a high-temperature heating cabinet. The mold is solidified at a temperature of 248 °F, resulting in a square solid magnetic sample. This sample can then be permanently magnetized in a magnetizer, aligning the magnetic particles in a specific direction.

Following the fix 16 magnetic Hall sensors (MLX90393; Melexis) in the specified positions on the base. Place the base fixture of the magnetic skin directly above the sensors and pour Ecoflex 00-30 AB resin. Use a mold with 25 compartments to allow the silicone to solidify, creating 25 square cavities. Attach pre-magnetized flexible magnetic films into the 25 cavities. Finally, pour Ecoflex 00-30 AB resin on top to integrate the components into a unified structure.

Material ratio selection of soft magnetic film

When an external force is applied to the sensor, it is observed that the degree of deformation of the magnetic film and the change in three-dimensional magnetic flux density signals (ΔB) are influenced by the proportion of magnetic powder in the film. According to our previous work47, considering the influence of sensing sensitivity, elastic modulus, stiffness, and the risk of fracture, 60% \({\rm{wt}}\) magnetic powder is chosen as the best composition of the soft magnetic film. The specific discussion can be found in Supplementary Note 1 and Supplementary Fig. 1.

Magnetic sensor selection

For this magnetic skin, we select the 3-axis magnetic Hall sensor (MLX90393; Melexis) as the unit of our sensor array. The main reason that we use the magnetic Hall sensor MLX90393 is that we need to make the sensor array and we want the unit to be small and easy to combine them together. The size of the MLX90393 is small, and they can use the SPI to connect and make the sensor array. On the other hand, the detecting point of MLX90393 is at the center of the sensor which can make the measurement more accurate. And we use the TCA9548A device to connect them together as an array.

Machine learning

For small-scale object recognition, we used a standard CNN with six fully connected hidden layers. The data consisted of 12500 samples that were split into datasets of training, validation, and test with a ratio of 8:1:1. The data were recorded from sixteen Hall sensors with 48-dimensional magnetic signals after being pressed of different heads at 25 different positions with two different strengths. The models were trained with the standard error loss. For the large-scale object recognition, the setting was the same as above, except that we have only 1950 data points.

Response time of the sensor

The instant response time of the sensor is measured by calculating the dropping time of the sensor signals when the external load is quickly changing. We do an experiment by using the Six axis force and torque sensor (KWR75 KUNWEI) to test the loading force and detecting the magnetic signal at the same time as shown in Supplementary Fig. 23. According to Supplementary Fig. 22a, the magnetic signal can catch the applied force very well. And the delay of the magnetic signal is about 20 ms in Supplementary Fig. 22b.

We also test the average response time of the 16-Hall sensor array to the computer and operating time of the super-resolution clustering algorithm is about 43.5 ms, which means the sensing frequency of the data transmitted from the Hall sensor to the computer is about 23 Hz as shown in Supplementary Fig. 24a. The computation time for the detect point is about 44.2 ms as shown in Supplementary Fig. 24b. In summary, the shortest time for the effective response is the combination of the magnetic signal delay and the computation time which is about 65 ms, which is mainly affected by the sensing frequency of the Hall sensor, and the operation time of the clustering algorithm.