rus
Issue #6/2024
E.V.Panfilova, A.R.Ibragimov, D.V.Frantsisin
NEURAL NETWORK MODEL FOR ADJUSTING THE PROCESS OF STUDYING COLLOIDAL NANO- AND MICROSTRUCTURES USING ATOMIC FORCE MICROSCOPY
NEURAL NETWORK MODEL FOR ADJUSTING THE PROCESS OF STUDYING COLLOIDAL NANO- AND MICROSTRUCTURES USING ATOMIC FORCE MICROSCOPY
INTRODUCTION
Colloidal nano- and microstructures find practical application in many actively developing areas of science and engineering. The use of silicon dioxide (SiO2), titanium dioxide (TiO2), polystyrene latex (PS), and a number of other materials for the formation of colloidal structures makes it possible to produce colloidal photonic crystals (CPCs), which are promising materials for photonics, optoelectronics, and laser devices [1]. In [2], a variety of reflective displays based on nature-like photonic crystal structures is described. In [3], a colorimetric strain sensor based on CFC integrated by a deformable graphene photoconverter is presented. In [4], a concept of all-optical logic gates based on CFCs modulated by photoluminescence of perovskite nanocrystals is proposed.
The variety of applications of colloidal nano- and microstructures poses many tasks and scientific challenges for scientists related to precision of development objects. The classical methods for their control and study are electron and probe microscopy. However, dielectric materials used in CFCs limit the use of electron microscopy. Therefore, for nondestructive control of morphology of colloidal nano- and microstructured films, probe atomic force microscopy (AFM) realised by amplitude-modulation semi-contact (intermittent contact) scanning is used [5]. The method allows rapid implementation of high-precision measurements in atmosphere without preliminary sample preparation. Despite a large number of advantages, this method is characterised by artefacts caused by various noises, probe and sample convolution, sticking of the probe when interacting with the sample, etc. The occurrence of many problems may result from incorrectly selected scanning parameters that differ from sample to sample. Occurrence of many problems can be a consequence of incorrectly selected scanning parameters, which differ from sample to sample. Current AFM systems allow automatic selection of parameters, however, this option is suitable mainly for trivial studies of surfaces without any features. Therefore, to study fragile samples with developed relief and to obtain the best results it is still necessary to select the correct scanning parameters "manually", which is a labour-intensive task requiring a high level of training and experience of the operator.
Currently, there are various methods to improve scanning efficiency, ranging from mathematical processing to development of fundamentally new elements of the system. In [6], an optimal set of scanning parameters is calculated for the probe-sample system function. In [7], the AFM probe dependences wear on scanning quality are identified experimentally, then optimal parameter settings that can alleviate probe wear are obtained. In [8], a method of increasing the scanning speed in the scanning range of several tens of micrometres by using a single-frequency control signal and implementing a fuzzy controller in the system is described. Neural network algorithms are becoming increasingly popular and are used to improve a wide range of manufacturing and research processes. The paper [9] describes application of neural network models and algorithms for AFM, which allow to increase the scanning process efficiency based on the characteristics of the studied samples. In some works it is proposed to perform neural network post-processing of the obtained control results [10], filtering of images and suppression of artefacts [11], compression of the number of points during scanning, with their subsequent reconstruction [12]. A neural network model combining both approaches was presented by the authors earlier in [13]; it allows selecting individual scanning parameters for CPC samples on the basis of parameters describing their properties and methods of formation. Neural networks that allow real-time adjustment of scanning parameters without any human presence are of great interest. Leading companies-developers of atomic force microscopes have already started to implement such neural networks (NN) in their software: NanoScope Analysis by BRUKER (USA) [14], intelligent software module SCANTRONIC™ by NT-MDT Spectrum Instruments (Russia) [5]. The above modules, being intended for a wide range of users, require for correct operation knowledge of properties and characteristics of the ыегвшув samples, which are not always known within the framework of research laboratories, which limits their applicability. Therefore, this paper presents a neural network model to optimise the process of imaging the surface of colloidal nano- and microstructures by atomic force microscopy by adjusting the scanning parameters directly during the examination of the sample. The principle of the model is to process the real-time AFM image and adjust the scanning parameters until an optimal image is obtained by a feedback system.
METHODS AND MATERIALS
Photonic crystalline films of spherical silica and polystyrene particles were used as studied colloidal nano- and microstructures. The colloidal solution of silica was prepared by a modified Stober method on a controlled colloidal solution synthesis bench. The colloidal polystyrene particles were commercially available standard samples. The films were prepared on silicon and polyethylene terephthalate substrates by spin-coating and vertical pulling methods using equipment included in the laboratory complex to form the colloidal photonic crystal structures [15].
The samples were controlled on the atomic force microscope Solver NEXT of NT-MDT Spectrum Instruments (Russia) in semi-contact mode. The main variable parameters during scanning were Set Point – the level of the probe-surface interaction parameter supported by feedback during scanning, Gain – gain of the synchronous amplifier, Rate – scanning speed, Amplitude – amplitude of cantilever oscillations in the semi-contact mode. The parameter Points – number of scanning points remained unchanged for obtaining and was equal to 512 × 512. The neural network was developed in the Visual Studio Code environment using the Python 3.11 language.
The data obtained by AFM method is usually an array of data that can be represented as a three-dimensional model. To represent their NN, all scan results were saved as 2D surface images using inbuilt software and were RGB images of 675 × 675 pixels (Fig.1). Each image was fragmented into 9 parts of 675 × 75 pixels to simulate the initial scanning step. In order to reduce the amount of data without loss of quality and informativeness, the images were converted to shades of grey. In this form, the NN received data about a pixel not as an RGB tuple, but as a single value corresponding to a shade of grey.
During the input data normalisation process, all pixel values were divided by 255 to normalise and produce a matrix of values from 0 to 1. As a result, regardless of the nature of the input number (pixel or process parameter value), its value was always within the same limits, resulting in a NN with more uniform weights. The scan parameter data were normalised using a similar principle, so that a value of 0 corresponded to the minimum parameter value from the training sample and 1 to the maximum.
The task of neural network design and implementation was complicated by the need to combine image and process parameter processing, which led to presence of two input layers in the neural network. The developed architecture contains 19 layers with the total number of varying parameters more than 43 million. Structurally, NN consists of several parts that perform defect extraction in the image, defect analysis in the image, defect analysis taking into account the process parameters in which the image was acquired, and correction.
Defect extraction is performed after the image passes alternating convolution and subsampling layers. To reduce the amount of information, speed up the learning process and simplify NN, the image passes through subsampling layers that compress the image by a factor of 4 after each layer. The first part of the developed NN contains an input layer, 4 convolution layers and 4 subsampling layers and a technical layer for matrix transformation (Fig.2). The result of passing the image of the convolution kernel is a feature, in the first convolution layer there are 8 features, i.e. for each two-dimensional matrix (image) coming to the input layer at the output will be 4 images, where the desired features are selected. In each next layer the number of features increases.
After feature extraction, the process of their analysis is implemented. In this part of NN the detected defects are classified and their influence on the process parameters is evaluated. For this purpose, the classical MLP (multi-layer perceptron) architecture is used, consisting of 6 layers with the number of neurons 512, 256, 64, 4, respectively, where the 2nd and 4th layers are DropOut nulling layers. A hyperbolic tangent was adopted as activation function of the neurons.
The last part of NN is fed with the process parameters at which the image was acquired. This part of the network is realised, like the previous one, on the basis of MLP architecture with the number of neurons in layers 32, 4, and the first two layers are the input layer and the technical concatenation layer.
The total number of links in the developed neural network is 43427108 units. All weights of the neural network model occupy about 498 MB of memory. The image of the neural network with indication of the types of layers and connections between them is shown in Fig.3.
In the process of training the neural network, the error back propagation algorithm was used. After each pass through the network, back propagation performs a backward pass and adjusts the model parameters (weights and biases). The parameters selected for training are given in Table 1.
To speed up the training, a mini-sampling method was used and the NN parameters were updated after passing an entire mini-sample. This reduced the training time from 8.5 hours to 20 minutes. The error on the training sample was about 8%, on the validation sample about 20%, and on the test sample 8.5%.
RESULTS AND DISCUSSION
The dependence of the error on the training sample and the error on the validation sample can be seen in Fig.4. The error decreases with increasing periods, taking into account the smoothness of the graph, we can say that the hyperparameters of NN are selected correctly, i.e. the neural network converges.
It is of scientific interest to understand how the developed NN perceives an image. Algorithmically extracted information containing images modified by the neural network (Fig.5). It can be seen that in the final convolution layer, it is the defects that stand out in the images, while the image without defects is less bright, indicating a lower degree of neuronal activation. The image in Fig.5 (and) does not contain elements of the image received as input. At the same time, the NN selected those areas that it considered the most important during the analysis and translated the information from conceptual-qualitative to analytical-quantitative format. Decrease in image resolution is associated with the passage of pooling layers.
To analyse quality of neural network performance, images with and without artefacts were analysed. In both cases the error percentage was less than 5% for each of the parameters. Table 2 presents the results of this NN testing.
CONCLUSIONS
To prevent artefacts in AFM images of nano- and microstructures, in particular colloidal photonic-crystalline films, it is rational to adjust the sample parameters scanning using artificial intelligence. A deep neural network containing convolution, subsampling and regularisation layers, supplemented by a network with multilayer perceptron architecture responsible for the analysis of input parameters (modes) of the process, has shown good results in analysing and detecting the features of the formed images. Such a model is able to produce qualitative adjustments of the scanning parameters in the required situations. The generalisation error in processing AFM images in this case does not exceed 5%. The obtained results and developments can be used in laboratory studies, as well as in the development of metrological documentation.
PEER REVIEW INFO
Editorial board thanks the anonymous reviewer(s) for their contribution to the peer review of this work. It is also grateful for their consent to publish papers on the journal’s website and SEL eLibrary eLIBRARY.RU.
Declaration of Competing Interest. The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Colloidal nano- and microstructures find practical application in many actively developing areas of science and engineering. The use of silicon dioxide (SiO2), titanium dioxide (TiO2), polystyrene latex (PS), and a number of other materials for the formation of colloidal structures makes it possible to produce colloidal photonic crystals (CPCs), which are promising materials for photonics, optoelectronics, and laser devices [1]. In [2], a variety of reflective displays based on nature-like photonic crystal structures is described. In [3], a colorimetric strain sensor based on CFC integrated by a deformable graphene photoconverter is presented. In [4], a concept of all-optical logic gates based on CFCs modulated by photoluminescence of perovskite nanocrystals is proposed.
The variety of applications of colloidal nano- and microstructures poses many tasks and scientific challenges for scientists related to precision of development objects. The classical methods for their control and study are electron and probe microscopy. However, dielectric materials used in CFCs limit the use of electron microscopy. Therefore, for nondestructive control of morphology of colloidal nano- and microstructured films, probe atomic force microscopy (AFM) realised by amplitude-modulation semi-contact (intermittent contact) scanning is used [5]. The method allows rapid implementation of high-precision measurements in atmosphere without preliminary sample preparation. Despite a large number of advantages, this method is characterised by artefacts caused by various noises, probe and sample convolution, sticking of the probe when interacting with the sample, etc. The occurrence of many problems may result from incorrectly selected scanning parameters that differ from sample to sample. Occurrence of many problems can be a consequence of incorrectly selected scanning parameters, which differ from sample to sample. Current AFM systems allow automatic selection of parameters, however, this option is suitable mainly for trivial studies of surfaces without any features. Therefore, to study fragile samples with developed relief and to obtain the best results it is still necessary to select the correct scanning parameters "manually", which is a labour-intensive task requiring a high level of training and experience of the operator.
Currently, there are various methods to improve scanning efficiency, ranging from mathematical processing to development of fundamentally new elements of the system. In [6], an optimal set of scanning parameters is calculated for the probe-sample system function. In [7], the AFM probe dependences wear on scanning quality are identified experimentally, then optimal parameter settings that can alleviate probe wear are obtained. In [8], a method of increasing the scanning speed in the scanning range of several tens of micrometres by using a single-frequency control signal and implementing a fuzzy controller in the system is described. Neural network algorithms are becoming increasingly popular and are used to improve a wide range of manufacturing and research processes. The paper [9] describes application of neural network models and algorithms for AFM, which allow to increase the scanning process efficiency based on the characteristics of the studied samples. In some works it is proposed to perform neural network post-processing of the obtained control results [10], filtering of images and suppression of artefacts [11], compression of the number of points during scanning, with their subsequent reconstruction [12]. A neural network model combining both approaches was presented by the authors earlier in [13]; it allows selecting individual scanning parameters for CPC samples on the basis of parameters describing their properties and methods of formation. Neural networks that allow real-time adjustment of scanning parameters without any human presence are of great interest. Leading companies-developers of atomic force microscopes have already started to implement such neural networks (NN) in their software: NanoScope Analysis by BRUKER (USA) [14], intelligent software module SCANTRONIC™ by NT-MDT Spectrum Instruments (Russia) [5]. The above modules, being intended for a wide range of users, require for correct operation knowledge of properties and characteristics of the ыегвшув samples, which are not always known within the framework of research laboratories, which limits their applicability. Therefore, this paper presents a neural network model to optimise the process of imaging the surface of colloidal nano- and microstructures by atomic force microscopy by adjusting the scanning parameters directly during the examination of the sample. The principle of the model is to process the real-time AFM image and adjust the scanning parameters until an optimal image is obtained by a feedback system.
METHODS AND MATERIALS
Photonic crystalline films of spherical silica and polystyrene particles were used as studied colloidal nano- and microstructures. The colloidal solution of silica was prepared by a modified Stober method on a controlled colloidal solution synthesis bench. The colloidal polystyrene particles were commercially available standard samples. The films were prepared on silicon and polyethylene terephthalate substrates by spin-coating and vertical pulling methods using equipment included in the laboratory complex to form the colloidal photonic crystal structures [15].
The samples were controlled on the atomic force microscope Solver NEXT of NT-MDT Spectrum Instruments (Russia) in semi-contact mode. The main variable parameters during scanning were Set Point – the level of the probe-surface interaction parameter supported by feedback during scanning, Gain – gain of the synchronous amplifier, Rate – scanning speed, Amplitude – amplitude of cantilever oscillations in the semi-contact mode. The parameter Points – number of scanning points remained unchanged for obtaining and was equal to 512 × 512. The neural network was developed in the Visual Studio Code environment using the Python 3.11 language.
The data obtained by AFM method is usually an array of data that can be represented as a three-dimensional model. To represent their NN, all scan results were saved as 2D surface images using inbuilt software and were RGB images of 675 × 675 pixels (Fig.1). Each image was fragmented into 9 parts of 675 × 75 pixels to simulate the initial scanning step. In order to reduce the amount of data without loss of quality and informativeness, the images were converted to shades of grey. In this form, the NN received data about a pixel not as an RGB tuple, but as a single value corresponding to a shade of grey.
During the input data normalisation process, all pixel values were divided by 255 to normalise and produce a matrix of values from 0 to 1. As a result, regardless of the nature of the input number (pixel or process parameter value), its value was always within the same limits, resulting in a NN with more uniform weights. The scan parameter data were normalised using a similar principle, so that a value of 0 corresponded to the minimum parameter value from the training sample and 1 to the maximum.
The task of neural network design and implementation was complicated by the need to combine image and process parameter processing, which led to presence of two input layers in the neural network. The developed architecture contains 19 layers with the total number of varying parameters more than 43 million. Structurally, NN consists of several parts that perform defect extraction in the image, defect analysis in the image, defect analysis taking into account the process parameters in which the image was acquired, and correction.
Defect extraction is performed after the image passes alternating convolution and subsampling layers. To reduce the amount of information, speed up the learning process and simplify NN, the image passes through subsampling layers that compress the image by a factor of 4 after each layer. The first part of the developed NN contains an input layer, 4 convolution layers and 4 subsampling layers and a technical layer for matrix transformation (Fig.2). The result of passing the image of the convolution kernel is a feature, in the first convolution layer there are 8 features, i.e. for each two-dimensional matrix (image) coming to the input layer at the output will be 4 images, where the desired features are selected. In each next layer the number of features increases.
After feature extraction, the process of their analysis is implemented. In this part of NN the detected defects are classified and their influence on the process parameters is evaluated. For this purpose, the classical MLP (multi-layer perceptron) architecture is used, consisting of 6 layers with the number of neurons 512, 256, 64, 4, respectively, where the 2nd and 4th layers are DropOut nulling layers. A hyperbolic tangent was adopted as activation function of the neurons.
The last part of NN is fed with the process parameters at which the image was acquired. This part of the network is realised, like the previous one, on the basis of MLP architecture with the number of neurons in layers 32, 4, and the first two layers are the input layer and the technical concatenation layer.
The total number of links in the developed neural network is 43427108 units. All weights of the neural network model occupy about 498 MB of memory. The image of the neural network with indication of the types of layers and connections between them is shown in Fig.3.
In the process of training the neural network, the error back propagation algorithm was used. After each pass through the network, back propagation performs a backward pass and adjusts the model parameters (weights and biases). The parameters selected for training are given in Table 1.
To speed up the training, a mini-sampling method was used and the NN parameters were updated after passing an entire mini-sample. This reduced the training time from 8.5 hours to 20 minutes. The error on the training sample was about 8%, on the validation sample about 20%, and on the test sample 8.5%.
RESULTS AND DISCUSSION
The dependence of the error on the training sample and the error on the validation sample can be seen in Fig.4. The error decreases with increasing periods, taking into account the smoothness of the graph, we can say that the hyperparameters of NN are selected correctly, i.e. the neural network converges.
It is of scientific interest to understand how the developed NN perceives an image. Algorithmically extracted information containing images modified by the neural network (Fig.5). It can be seen that in the final convolution layer, it is the defects that stand out in the images, while the image without defects is less bright, indicating a lower degree of neuronal activation. The image in Fig.5 (and) does not contain elements of the image received as input. At the same time, the NN selected those areas that it considered the most important during the analysis and translated the information from conceptual-qualitative to analytical-quantitative format. Decrease in image resolution is associated with the passage of pooling layers.
To analyse quality of neural network performance, images with and without artefacts were analysed. In both cases the error percentage was less than 5% for each of the parameters. Table 2 presents the results of this NN testing.
CONCLUSIONS
To prevent artefacts in AFM images of nano- and microstructures, in particular colloidal photonic-crystalline films, it is rational to adjust the sample parameters scanning using artificial intelligence. A deep neural network containing convolution, subsampling and regularisation layers, supplemented by a network with multilayer perceptron architecture responsible for the analysis of input parameters (modes) of the process, has shown good results in analysing and detecting the features of the formed images. Such a model is able to produce qualitative adjustments of the scanning parameters in the required situations. The generalisation error in processing AFM images in this case does not exceed 5%. The obtained results and developments can be used in laboratory studies, as well as in the development of metrological documentation.
PEER REVIEW INFO
Editorial board thanks the anonymous reviewer(s) for their contribution to the peer review of this work. It is also grateful for their consent to publish papers on the journal’s website and SEL eLibrary eLIBRARY.RU.
Declaration of Competing Interest. The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.
Readers feedback
