With the rapid growth of multi-modal data, the application requirements of cross-modal retrieval are brought, and the research on cross-modal retrieval methods is proposed. This paper traces the latest progress in this field, tracks and deeply studies the cross-modal retrieval methods based on representation learning at home and abroad, defines the cross-modal retrieval problems, and combs the common technical methods, mainstream models, common data sets, evaluation methods and main challenges in this field. This paper mainly introduces the cross-modal retrieval method based on representation learning from three aspects:statistical correlation analysis, graph regularization and metric learning, and analyzes its advantages and disadvantages. In order to analyze the advantages and disadvantages of the above methods, 14 methods are reproduced on four data sets for comparative evaluation. The experimental results show that the training method based on statistical correlation analysis is efficient and easy to implement; Based on graph regularization method, semantic association is realized by mining the similarity between and within modes; The metric-based learning method is to preserve the semantically similar / dissimilar information of data in the common subspace as much as possible. To sum up, this paper introduces the research status of cross-modal retrieval methods based on representation learning, which provided a reference for the research of cross-modal retrieval methods.

High utility complex pattern is a new topic in data mining. For the first time, the complex high utility pattern mining algorithms are reviewed from two perspectives: high utility integrated pattern and high utility derivative pattern. Firstly, high utility sequential pattern(HUSP) is classified and discussed according to different data structures; high utility episode pattern(HUEP) and periodic high utility pattern(PHUP) are summarized in chronological order. Secondlyly, for the derivative model, the high average utility pattern, the high-utility pattern with negative unit profits and high on-shelf utility pattern are summarized from the perspective of data structure; the concise high utility pattern is summarized from the perspective of concise types. Thirdly, the advantages and disadvantages and upper boundaries of the existing integrated pattern and derivative pattern mining algorithms are compared and analyzed. Finally, based on the current research deficiencies, further research direction is given, including high utility pattern mining in uncertain data, high utility on-shelf pattern mining over data stream, and parallel high utility pattern mining in a big data environment.

Multi-modal fusion aims to integrate multiple modal information to obtain a consistent and common model output, which is a basic problem in the multi-modal field. Through the fusion of multimodal information, more comprehensive features can be obtained and the robustness of the model can be improved. At present, multimodal fusion technology has become one of the core research topics in the field of multimodality. Based on Imagenet, HowNet and CCD, this paper constructs a new multimodal knowledge base through manual annotation. The calibration has completed the mapping of 21 455 noun concepts in ImageNet, effectively mapping the concepts in HowNet and CCD to ImageNet. The data set can be applied to natural language processing tasks and computer vision tasks, and improve the task effect through picture information and concept information. In image classification, by adding HowNet and ImageNet concepts, more image features can be integrated to assist classification. In semantic understanding, image information can be better understood by adding image information through mapping.

More time-effective and wider adverse drug reactions are concealed in tweets related to feelings of taking medication. However, it is difficult to extract adverse drug reaction (ADR) from these tweets due to relatively shortness and sparseness of tweets. Therefore, a neural network model is proposed in this paper, which employes the pretrained bidirectional language model and attention mechanism to identify ADR. Firstly, specific character-level features are extracted via a pretrained bidirectional character-level neural language model. Secondly, the attention mechanism is used to capture local and global semantic contexts while extracting ADRs. Thirdly, to improve the efficiency of the proposed method, Character-level features are combined with word-level features. Finally, co-training is replaced with the pretrained of the whole-word level and fine-tuned pretrained character embeddings. These optimizations contribute to improving the performance of identification. The proposed model achieves better performance on the PSB 2016 Social Media Mining Sharing Task Workshop-Task 2: ADR Extraction, obtaining the F₁-scores of 82.2% on official datasets. Character features are useful for distinguishing ADR and non-ADR in morphology. In addition, attention mechanism improves the performance of identifying ADR due to capturing local and global semantic contexts.

Short text semantic similarity measurement based on deep learning is the cornerstone of modern natural language processing, and its importance is self-evident. Text encoding model is proposed in this paper based on convolutional neural network and bidirectional gated circulation unit, by convolution important semantic extraction and through bidirectional gated circulation unit to ensure semantic sequence cycles. And the consistency of text encoding is ensured by Siamese neural network structure. In this paper, traditional convolution neural networl is compared with both short-term and long-term memory network and BERT model. Experimental results are done on Quora data set, Sick data set and MSRP data set. The verification results show that the accuracy and recall rate of the proposed model are excellent, and the comprehensive performance index F₁ value is the best compared with the traditional model.

In the field of natural language understanding, humorous computation has gradually become an important research content. Sitcom is a special form of humorous expression, which contains abundant humorous expressions. Chinese is so varied that it is a challenge for a computer to analyze the humor emotion. In order to solve the problem of Chinese humor calculation, the following work are done in this paper. First, a humor recognition algorithm, DISA-SE-GAT, based on segmentation enhancement and semantic enhancement, is proposed based on the graph attention network. Second, a humorous sitcom data set, ipartment, is constructed. Experimental results show that the model of word sense disambiguation and semantic enhancement, DISA-SE-GAT, performs well in the recognition of humorous expression in text.

Lung cancer is one of the cancers with the highest mortality in the world. As an important basis for early diagnosis of lung cancer, accurate segmentation of pulmonary nodules is particularly important. In order to help doctors diagnose lung lesions, an improved UNet lung nodule segmentation method is proposed. First, Efficient Channel Attention for Deep Convolutional Neural Networks(EcaNet) is introduced in the feature extraction part, which improves the UNet segmentation effect and makes it have good generalization ability. At the same time, in order to reduce the number of parameters of the model and improve the segmentation performance of the algorithm, a feature fusion model of depthwise separable convolution is proposed, which replaces the traditional convolution operation with depthwise separable convolution to complete feature fusion. According to the image characteristics of pulmonary nodules, the Dice Loss and the weighted cross entropy (WCE) are combined as a new loss function. To verify the effectiveness of the proposed algorithm Eca-UNet, our evaluated on the LIDC-IDRI public dataset of lung nodules. The results show that the DICE similarity coefficient and MIOU of the Eca-UNet algorithm are 10.47% and 7.34% higher than that of the UNet segmentation algorithm, respectively. At the same time, the training speed has increased by 10.10%, and the prediction speed has increased by 11.56%.

It is very important for diagnoses and treatments of lung tumors to segment lung nodules from CT images automatically and effectively. However, the lung nodules usually are very small, their shapes are irregular, and sometimes they are very similar to adjacent tissues and organs in vision, which brings difficulties to the segmentation task. This paper proposes a lung nodules segmentation network MRF-GMA based on multi-receptive field and grouped mixed attention mechanism. Firstly, the multi-receptive field feature aggregation module can capture nodules of different scales; secondly, the grouped mixed attention is used to improve the resolution of nodular pixels; finally, the hybrid loss function is used to optimize the training process to alleviate the class imbalance problem. In the experiment, MRF-GMA is respectively compared with FCN, SEGNET, R2U-NET and Attention U-NET, and the results show that MRF-GMA model has the best performance in DSC, Accuracy and Recall, and has increased by 2.25%, 1.19% and 2.98%, respectively, compared with the Attention U-Net model.

High-resolution remote sensing image classification is a current research hotspot. The high-resolution remote sensing image classification model (Deeplab) based on deep convolutional networks and fully connected conditional random fields is widely used in this field because of its efficient and accurate classification performance. The Deeplab model has the problem of insufficient information utilization of high-resolution remote sensing images by hole convolution, which limits the further improvement of classification accuracy. In view of this, this paper proposes a new high-resolution remote sensing image classification model (Dspp). The Dspp model adopts a dense convolution network connection structure, and replaces Deeplab′s hollow convolution pyramid structure with a dense connection structure to improve information utilization and enhance the generalization ability of the model. Compared with the FCN model, the FCN-8S model and the Deeplab model, the overall accuracy of the Dspp model has improved by 16.8%, 11.7%, and 7.7%, which verifies the effectiveness of the model.

To slove the problem that the face detection frame is difficult to learn, and the problems that complex process has high coupling and error accumulation serious in two-step series model, a fusion algorithm of face detection and head pose estimation based on YOLOv3 model is proposed. By using the K-means clustering method to cluster the size of the face area of the training dataset, 9 sets of results are obtained to simulate the size and scale of face areas under real conditions. By expanding the YOLOv3 model, face detection and head pose estimation are achieved simultaneously. Therefore, face detection and head pose estimation on three different levels, multi-scale detection for the feature map is realized. The new algorithm takes advantage of the information in the feature map and uses end-to-end mode training to simplify the processing flow of the head pose estimation task. In addition, an end-to-end model is completed to simplify the processing flow. The recognition accuracy rate of 99.23% is achieved on the pose subset of CAS-PEAL-R1, and the mean absolute error of 3.79° and 4.24° are achieved in the pitch and yaw directions on the Pointing′04 data set. The results show that the model can complete the task of face area detection and head pose estimation under the premise of meeting the real-time requirements, which proves the reliability and practicability of the algorithm in this paper.

Emotion recognition based on gait features is considered to have a wide range of applications in emotion computing, psychotherapy, robotics, surveillance and audience understanding. Existing methods show that combining the context information such as gesture position can significantly improve the performance of emotion recognition, and spatiotemporal information can significantly improve the accuracy of emotion recognition. However, the emotional information in gait can not be fully expressed only by using bone spatial information. In order to make good use of the gait features, an adaptive fusion method is proposed in this paper, which combines the spatiotemporal information of the skeleton with the rotation angle of the skeleton, and improves the emotion recognition accuracy of the existing models. The model uses the Autoencoder to learn the bone rotation information of human walking, uses the spatio-temporal convolution neural network to extract the spatio-temporal information of bone points, inputs the bone rotation information and spatio-temporal information into the adaptive fusion network, and obtains the final feature for classification. The model is tested on the Emotion-gait data set, and the experimental results show that the AP values of sadness, anger and neutral emotion have increased by 5, 8 and 5 percentage point respectively compared with the latest HAP method, and the average map value of the overall classification has increased by 5 percentage point.

Deep learning provides ideas for the intelligent development of bearing fault diagnosis. From the perspective of brain-like computing, this paper designs spiking neural network sensitive to bearing data to complete the task of fault data classification. First, signal decomposition is used to improve the feature extraction effect of the original signal, and then the fault signal is spiking-encoded, and the time step is filled with multi-channel chaotic input as the input of the neural network. Finally, the Spiking convolutional neural network (SCNN) is used. In order to verify the classification effect of the model, a classification experiment is done on the CWRU bearing data set, and the classification accuracy rate reaches 99.78%. The results show that the bearing data encoding scheme can give full play to the spatiotemporal dynamic characteristics of the SNN, and the SNN model has the characteristics of high precision and high efficiency in bearing fault diagnosis. This research is helpful to promote the research and application of SNN in the field of fault diagnosis.

In recent years, deep learning provides a new method for predicting the remaining useful life of bearings. In practice, due to the small number of bearing degradation data and the large difference in the distribution of bearing data under different working conditions, it is hard to realize that the remaining useful life prediction model trained on one bearing, which can be used for other bearings remaining useful life prediction under the same or different working conditions. In order to deal with the aforementioned shortcomings, a joint distribution adaptation based on cross domain mean approximation for bearing remaining useful life prediction method is proposed. Firstly, the original vibration signal data of the bearing is normalized. Then the source domain and target domain data are projected to a corresponding lowdimensional common feature subspace by projection matrix. In the subspace, a joint distribution adaptation based on cross domain mean approximation method is used to perform domain adaptation for source and target data. Finally, the gated recurrent unit is used for the bearing remaining useful life prediction. The validity of the proposed method is demonstrated by experiments on IEEE PHM Challenge 2012 dataset. The results show that the proposed method has good prediction accuracy under the same working condition or different working conditions for different bearings.

Aiming at the problems of low accuracy of most detection technologies for phishing websites, high consumption of computing resources and untimely detection, a phishing website detection method based on an improved Stacking strategy is proposed. This method integrates multiple base learners with excellent classification performance into a high-performance model through stacking strategy, and takes the input characteristics and prediction results of the first level of the stacking algorithm as the input characteristics of the second level at the same time, so as to give full play to the advantages of high precision and fast speed of each model, and further improve the performance of the model. Experimental results show that, compared with traditional machine learning phishing website detection technology, this integrated learning algorithm on a 100,000-level data set shows better performance on multiple indicators, with accuracy rate of 97.82% and F₁ value reach 97.54%, which can effectively detect phishing websites.

Aiming at the injury caused by the elderly who cannot receive timely assistance after falling, the study of fall detection algorithms and timely warnings can reduce the serious harm and consequences of falling of the elderly. In order to improve the accuracy and real-time performance of fall detection, a wearable fall detection algorithm based on bi-directional long and short-term memory neural network is proposed. This algorithm can automatically extract the deeper features from the input fall data (extracted from inertial sensors), and realize the processing from the pre-processed data to the detection result. The algorithm extracts the feature vectors of the acceleration sensor data through the neural network, and performs fall detection using bi-directional long and short-term memory neural network. The model is evaluated with SisFall dataset. The results show that the algorithm achieves high accuracy, while the latency also meets the requirements of real-time detection. The algorithm model has both good practicability and strong generalization ability.

Compared with traditional sequence modeling in session-based recommendation, modeling the session sequence as a graph structure performs better in this field. However, the existing research methods are limited in their ability to capture user’s current interest by only using the graph structure to mine the session characteristics between items. A current interest reinforced Graph neural network for session-based recommendation is proposed in the paper. By introducing position embedding and combining with graph neural network, the advantages of sequential perception model and graph perception model are complemented. The session sequence is modeled as a graph structure. Take the last click of the original sequence, and calculate its attention weight for graph node information through the multi-head attention mechanism, so as to obtain user’s current interest expression more accurately. Extensive experiments on two real-world datasets show that, the proposed method achieved the best performance of all methods, especially on the Diginetica data set, all evaluation indicators have increased by more than 7%, and the MRR@10 indicator has even increased by 9.52%. These results show the correctness and effectiveness of the proposed method for session-based recommendation.

The computational methods for predicting protein phosphorylation sites are usually used in the preliminary screening stage of the site identification. To further improve the prediction accuracy, a deep learning model called MAPhos is proposed. First, each residue is represented by the summation of the amino acid vector and the position vector. Second, a bidirectional GRU network is utilized to generate the hidden states of residues. Third, the multi-head attention mechanism is used for generating the context vector. Finally, the context vector and the sequence vectors are concatenated, and the concatenation vector is fed into a fully connected neural network for predicting the site. Experimental results on real-world datasets demonstrate that the MAPhos model can outperform the models based on feature extraction and the models based on convolutional neural network over several measures, and the new model has better interpretability than the convolutional neural network models.

Web traffic prediction has always been a hot issue in data center networks, which is of great significance for improving the quality of network services. Due to the complex characteristics of web traffic such as non-linearity, autocorrelation, and periodicity, it is very challenging to accurately predict it. In order to fully mine the predictable information of web traffic and make the prediction model fully interpretable and configurable, this paper proposes a combined prediction model based on Prophet and deep autoregression (DeepAR). Among them, Prophet is an additive model based on time series decomposition, which models the trend, seasonal period, and holiday information of Web traffic. At the same time, the autoregressive information implied by the Prophet residual is modeled using the DeepAR model based on probability prediction, and the long-term and short-term dependencies are captured to reduce the variance of the Prophet residual and fully capture the autoregressive information of web traffic. In this paper, the verification experiments are carried out on the real Web traffic dataset, and the results show that the evaluation indicators of RMSE and MAE are better than the comparative models, which verifies the effectiveness of the combined model.

Automatic Machine Learning (AutoML) is an important issue at the forefront of machine learning. Automatic machine learning tools compose machine learning primitives to construct pipelines based on datasets and task requirements, so that domain users can complete corresponding data analysis work without professional machine learning knowledge. However, current automatic machine learning tools generally suffer from the problems of long-time consumption and low precision. A data-driven method for automatic machine learning pipeline generation based on the principles of dataset similarity and reinforcement learning is proposed in this paper. This method uses the historical knowledge of similar datasets to guide the generation of machine learning pipelines. The experimental results show that the time-consumption of the method proposed in this paper is shortened to the minute level, and the pipeline performance is also improved.

The existence of no more than 1/3 of the total number of Byzantine nodes in the network can be tolerated by Practical Byzantine Fault Tolerance (PBFT) consensus algorithm. So PBFT consensus algorithm is often used as the consensus algorithm of the permissioned blockchains. However, the selection rule of the primary nodes is simple and the communication complexity is high in the PBFT consensus algorithm. A Byzantine fault tolerant consensus algorithm based on verifiable random function and BLS signature is proposed to improve PBFT consensus algorithm, which is called VBBFT consensus algorithm. In VBBFT consensus, Verifiable Random Functions is used to select primary nodes from the candidate set of consensus nodes. The master node is used as the coordinator of message collection and sending. At the same time , the process of information interaction between nodes is transformed into the process of BLS signature. This way can reduce the communication complexity and ensures the security of information interaction between nodes. The multi-node simulation results show that compared with Practical Byzantine Fault Tolerant consensus algorithm, the transaction throughput has increased by 62.3% and reduced the delay by 12% in VBBFT consensus algorithm.

In the cloud data storage system with multiple storage nodes, how to keep the load balance level of the cloud storage system to a reasonable value and minimize the time of data retrieval is a problem worth studying. To solve this problem, this paper proposes a balanced placement strategy of cloud data based on particle swarm optimization algorithm (BPCD). Firstly, a cloud storage system model is presented. Secondly, the Gini coefficient is introduced as the index to measure the load balancing level of the system, and a multi-objective constrained optimization model is constructed by combining the objective function of data retrieval time. Thirdly, the particle swarm optimization algorithm is used to solve the problem, which mainly includes four processes: data node coding and parameter setting, population initialization, particle swarm spatial search and algorithm iteration. Finally, the proposed algorithm is compared with the traditional cloud data placement algorithm. Simulation experiments show that the proposed cloud data balanced placement strategy has good effects in optimizing the load level and data retrieval time of the cloud storage system.