Journal of Guangxi Normal University(Natural Science Edition) ›› 2022, Vol. 40 ›› Issue (3): 49-56.doi: 10.16088/j.issn.1001-6600.2021071001

Previous Articles     Next Articles

Semantic Similarity Computing Model for Short Text Based on Deep Learning

ZHOU Shengkai1,2, FU Lizhen1,2*, SONG Wen’ai1,2   

  1. 1. School of Software, North University of China, Taiyuan Shanxi 030051, China;
    2. Shanxi Military and Civilian Intergration Software Engineering Technology Research Center, Taiyuan Shanxi 030051, China
  • Received:2021-07-10 Revised:2021-11-14 Online:2022-05-25 Published:2022-05-27

PDF (PC)

677

Abstract: Short text semantic similarity measurement based on deep learning is the cornerstone of modern natural language processing, and its importance is self-evident. Text encoding model is proposed in this paper based on convolutional neural network and bidirectional gated circulation unit, by convolution important semantic extraction and through bidirectional gated circulation unit to ensure semantic sequence cycles. And the consistency of text encoding is ensured by Siamese neural network structure. In this paper, traditional convolution neural networl is compared with both short-term and long-term memory network and BERT model. Experimental results are done on Quora data set, Sick data set and MSRP data set. The verification results show that the accuracy and recall rate of the proposed model are excellent, and the comprehensive performance index F1 value is the best compared with the traditional model.

Key words: natural language processing, semantic similarity, convolutional neural network, long short-term memory, gated recurrent unit

CLC Number: 

  • TP391.1
[1]殷美桂. 网络舆情系统的设计与实现[J]. 现代计算机, 2020(21): 104-108. DOI: 10.3969/j.issn.1007-1423.2020.21.022.
[2]张睿. 基于用户评价的强化学习推荐算法研究[D]. 西安: 西安电子科技大学, 2019.
[3]杨晨. 基于神经网络的短文本语义相似度计算方法研究[D]. 成都: 电子科技大学, 2020.
[4]DERIU J, MLYNCHYK K, SCHLÄPFER P, et al. A methodology for creating question answering corpora using inverse data annotation[C]// Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: Association for Computational Linguistics, 2020: 897-911. DOI: 10.18653/v1/2020.acl-main.84.
[5]WANG K, SHEN W Z, YANG Y Y, et al. Relational graph attention network for aspect-based sentiment analysis[C]// Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: Association for Computational Linguistics, 2020: 3229-3238. DOI: 10.18653/v1/2020.acl-main.295.
[6]周永称. 基于混合深度学习模型的临床医学文本分类研究[D]. 北京: 北京协和医学院, 2020.
[7]MIKOLOV T, SUTSKEVER I, CHEN K, et al. Distributed representations of words and phrases and their compositionality[C]// Proceedings of the 26th International Conference on Neural Information Processing Systems: Volume 2. Red Hook, NY: Curran Associates Inc., 2013: 3111-3119.
[8]LUHN H P. The automatic creation of literature abstracts[J]. IBM Journal of Research and Development, 1958, 2(2): 159-165.
[9]ZHOU Y, LIU C, PAN Y. Modelling sentence pairs with tree-structured attentive encoder[C]// Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers. Osaka, Japan: The COLING 2016 Organizing Committee, 2016: 2912-2922.
[10]石凤贵. 基于自然语言处理的Word2Vec词向量应用[J]. 黑河学院学报, 2020, 11(7): 173-177.
[11]MUELLER J, THYAGARAJAN A. Siamese recurrent architectures for learning sentence similarity[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2016, 30(1): 2786-2792.
[12]SEVERYN A, MOSCHITTI A. Learning to rank short text pairs with convolutional deep neural networks[C]// Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval. New York, NY: Association for Computing Machinery, 2015: 373-382. DOI: 10.1145/2766462.2767738.
[13]FUKUSHIMA K. Neocognitron: a self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position[J]. Biological Cybernetics, 1980, 36(4): 193-202.
[14]CHOPRA S, HADSELL R, LECUN Y. Learning a similarity metric discriminatively, with application to face verification[C]// 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition(CVPR’05). Piscataway, NJ: IEEE Press, 2005: 539-546. DOI: 10.1109/CVPR.2005.202.
[15]YIN W P, SCHÜTZE H. Learning word meta-embeddings[C]// Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Stroudsburg, PA: Association for Computational Linguistics, 2016: 1351-1360. DOI: 10.18653/v1/P16-1128.
[16]韩越, 艾山·吾买尔. 基于元嵌入的跨语言词嵌入方法研究[J]. 现代计算机, 2021(20): 20-25, 32.
[17]POERNER N, WALTINGER U, SCHÜTZE H. Sentence meta-embeddings for unsupervised semantic textual similarity[C]// Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: Association for Computational Linguistics, 2020: 7027-7034. DOI: 10.18653/v1/2020.acl-main.628.
[18]SANTUS E, WANG H M, CHERSONI E, et al. A rank-based similarity metric for word embeddings[C]// Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Stroudsburg, PA: Association for Computational Linguistics, 2018: 552-557. DOI: 10.18653/v1/P18-2088.
[19]VOR DER BRÜCK T, POULY M. Text similarity estimation based on word embeddings and matrix norms for targeted marketing[C]// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Stroudsburg, PA: Association for Computational Linguistics, 2019: 1827-1836. DOI: 10.18653/v1/N19-1181.
[20]李海林, 邹金串. 基于分类词典的文本相似性度量方法[J]. 智能系统学报, 2017, 12(4): 556-562.
[21]朱辉. 融合主题模型的文本语义表示方法研究[D]. 烟台: 山东工商学院, 2021.
[22]KUSNER M J, SUN Y, KOLKIN N I, et al. From word embeddings to document distances[C]// Proceedings of the 32nd International Conference on Machine Learning. Lille: PMLR, 2015: 957-966.
[23]GONG H Y, SAKAKINI T, BHAT S, et al. Document similarity for texts of varying lengths via hidden topics[C]// Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Stroudsburg, PA: Association for Computational Linguistics, 2018: 2341-2351. DOI: 10.18653/v1/P18-1218.
[24]MRABET Y, KILICOGLU H, DEMNER-FUSHMAN D. TextFlow: a text similarity measure based on continuous sequences[C]// Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Stroudsburg, PA: Association for Computational Linguistics, 2017: 763-772. DOI: 10.18653/v1/P17-1071.
[25]胡春光, 高燕, 李颖. 一种扩展滑动窗口算法[J]. 微电子学与计算机, 2007(8): 106-109, 112.
[26]VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[EB/OL].(2017-12-06)[2021-07-10]. https://arxiv.org/pdf/1706.03762. DOI: 10.48550/arXiv.1706.03762.
[27]DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[C]// Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). Stroudsburg, PA: Association for Computational Linguistics, 2019: 4171-4186. DOI: 10.18653/v1/N19-1423.
[28]PEINELT N, NGUYEN D, LIAKATA M. tBERT: topic models and BERT joining forces for semantic similarity detection[C]// Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Stroudsburg, PA: Association for Computational Linguistics, 2020: 7047-7055. DOI: 10.18653/v1/2020.acl-main.630.
[29]李雨楠. 基于智能问答系统的短文本语义相似度匹配[D]. 西安: 西安建筑科技大学, 2020.
[30]王培, 王亚文, 卢苗苗. 基于BERT模型的中医文本分类研究[J]. 电脑知识与技术, 2021, 17(27): 13-14, 20.
[31]CHUNG J Y, GULCEHRE C, CHO K H, et al. Empirical evaluation of gated recurrent neural networks on sequence modeling[EB/OL].(2014-12-11)[2021-07-10]. https://arxiv.org/pdf/1412.3555. DOI: 10.48550/arXiv.1412.3555.
[1] JIANG Rui, XU Juan, LI Qiang. A Prediction Method of Bearing Remaining Useful Life Based on Cross Domain Mean Approximation [J]. Journal of Guangxi Normal University(Natural Science Edition), 2022, 40(3): 121-131.
[2] DUAN Meiling, PAN Julong. Wearable Fall Detection Based on Bi-directional LSTM Neural Network [J]. Journal of Guangxi Normal University(Natural Science Edition), 2022, 40(3): 141-150.
[3] LI Bing, LI Zhi, YANG Yilong. Classification of Non-Functional Software Requirements Using Word Embeddings and Long Short-Term Memory [J]. Journal of Guangxi Normal University(Natural Science Edition), 2021, 39(5): 110-121.
[4] CHEN Wenkang, LU Shenglian, LIU Binghao, LI Guo, LIU Xiaoyu, CHEN Ming. Real-time Citrus Recognition under Orchard Environment by Improved YOLOv4 [J]. Journal of Guangxi Normal University(Natural Science Edition), 2021, 39(5): 134-146.
[5] YANG Zhou, FAN Yixing, ZHU Xiaofei, GUO Jiafeng, WANG Yue. Survey on Modeling Factors of Neural Information Retrieval Model [J]. Journal of Guangxi Normal University(Natural Science Edition), 2021, 39(2): 1-12.
[6] DENG Wenxuan, YANG Hang, JIN Ting. A Dimensionality-reduction Method Based on Attention Mechanismon Image Classification [J]. Journal of Guangxi Normal University(Natural Science Edition), 2021, 39(2): 32-40.
[7] YAN Hao, XU Hongbo, SHEN Yinghan, CHENG Xueqi. Research on Open Chinese Event Detection [J]. Journal of Guangxi Normal University(Natural Science Edition), 2020, 38(2): 64-71.
[8] FAN Rui, JIANG Pinqun, ZENG Shangyou, XIA Haiying, LIAO Zhixian, LI Peng. Design of Lightweight Convolution Neural Network Based on Multi-scale Parallel Fusion [J]. Journal of Guangxi Normal University(Natural Science Edition), 2019, 37(3): 50-59.
[9] ZHANG Jinlei, LUO Yuling, FU Qiang. Predicting Financial Time Series Based on Gated Recurrent Unit Neural Network [J]. Journal of Guangxi Normal University(Natural Science Edition), 2019, 37(2): 82-89.
[10] WU Wenya,CHEN Yufeng,XU Jin’an,ZHANG Yujie. High-level Semantic Attention-based Convolutional Neural Networks for Chinese Relation Extraction [J]. Journal of Guangxi Normal University(Natural Science Edition), 2019, 37(1): 32-41.
[11] XUE Yang,ZENG Qingke,XIA Haiying,WANG Wentao. Remote Sensing Image Fusion Based on Convolutional Neural Network Super-resolution Reconstruction [J]. Journal of Guangxi Normal University(Natural Science Edition), 2018, 36(2): 33-41.
[12] SHA Bei-bei, XIE Li-cong. Algorithm to Cluster Search Results Based on Frequent Itemsets [J]. Journal of Guangxi Normal University(Natural Science Edition), 2011, 29(2): 151-155.
[13] CHENG Xian-yi, PAN Yan, ZHU Qian, SUN Ping. Automatic Generating Algorithm of Event-oriented Multi-documentSummarization [J]. Journal of Guangxi Normal University(Natural Science Edition), 2011, 29(1): 147-150.
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed   
[1] AI Yan, JIA Nan, WANG Yuan, GUO Jing, PAN Dongdong. Review of Statistical Methods and Applications of Genetic Association Analysis for Multiple Traits and Multiple Locus[J]. Journal of Guangxi Normal University(Natural Science Edition), 2022, 40(1): 1 -14 .
[2] BAI Defa, XU Xin, WANG Guochang. Review of Generalized Linear Models and Classification for Functional Data[J]. Journal of Guangxi Normal University(Natural Science Edition), 2022, 40(1): 15 -29 .
[3] ZENG Qingfan, QIN Yongsong, LI Yufang. Empirical Likelihood Inference for a Class of Spatial Panel Data Models[J]. Journal of Guangxi Normal University(Natural Science Edition), 2022, 40(1): 30 -42 .
[4] ZHANG Zhifei, DUAN Qian, LIU Naijia, HUANG Lei. High-dimensional Nonlinear Regression Model Based on JMI[J]. Journal of Guangxi Normal University(Natural Science Edition), 2022, 40(1): 43 -56 .
[5] YANG Di, FANG Yangxin, ZHOU Yan. New Category Classification Research Based on MEB and SVM Methods[J]. Journal of Guangxi Normal University(Natural Science Edition), 2022, 40(1): 57 -67 .
[6] CHEN Zhongxiu, ZHANG Xingfa, XIONG Qiang, SONG Zefang. Estimation and Test for Asymmetric DAR Model[J]. Journal of Guangxi Normal University(Natural Science Edition), 2022, 40(1): 68 -81 .
[7] DU Jinfeng, WANG Hairong, LIANG Huan, WANG Dong. Progress of Cross-modal Retrieval Methods Based on Representation Learning[J]. Journal of Guangxi Normal University(Natural Science Edition), 2022, 40(3): 1 -12 .
[8] LI Muhang, HAN Meng, CHEN Zhiqiang, WU Hongxin, ZHANG Xilong. Survey of Algorithms Oriented to Complex High Utility Pattern Mining[J]. Journal of Guangxi Normal University(Natural Science Edition), 2022, 40(3): 13 -30 .
[9] CHAO Rui, ZHANG Kunli, WANG Jiajia, HU Bin, ZHANG Weicong, HAN Yingjie, ZAN Hongying. Construction of Chinese Multimodal Knowledge Base[J]. Journal of Guangxi Normal University(Natural Science Edition), 2022, 40(3): 31 -39 .
[10] LI Zhengguang, CHEN Heng, LIN Hongfei. Identification of Adverse Drug Reaction on Social Media Using Bi-directional Language Model[J]. Journal of Guangxi Normal University(Natural Science Edition), 2022, 40(3): 40 -48 .
Copyright © Journal of Guangxi Normal University(Natural Science Edition), All Rights Reserved.
Tel: 0773-5857325　E-mail: gxsdzkb@mailbox.gxnu.edu.cn
Powered by Beijing Magtech Co. Ltd