Journal of Guangxi Normal University(Natural Science Edition) ›› 2021, Vol. 39 ›› Issue (2): 13-20.doi: 10.16088/j.issn.1001-6600.2020082602

Previous Articles     Next Articles

Dynamic Learning Method of Neural Machine Translation Based on Sample Difficulty

WANG Su1,2, FAN Yixing1,2 , GUO Jiafeng1,2* , ZHANG Ruqing1,2 , CHENG Xueqi1,2   

  1. 1. Key Laboratory of Network Data Science & Technology, Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190, China;
    2. University of Chinese Academy of Sciences, Beijing 100049, China
  • Received:2020-08-26 Revised:2020-09-22 Online:2021-03-25 Published:2021-04-15

Abstract: In recent years, neural machine translation model has become the mainstream model in the field of machine translation. How to learn translation knowledge quickly and accurately from a large amount of training data is a problem worthy of discussion. Different training samples have different degrees of difficulty. Some training samples are simpler and easy for model to learn, while others are more difficult and not easy for model to learn. The difficulty of the samples has a great influence on the convergence of the model, but the traditional neural machine translation model does not consider this difference in the training process. Therefore, this paper explores the influence of the difficulty of the samples on the training process of the neural machine translation model. Considering the sample difficulty for the neural machine translation mode, a dynamic learning method is proposed based on the idea of “curriculum learning”. The difficulty degree of the training samples is quantified from the aspects of the translation effect of the neural machine translation model and the sentence length of the training samples, respectively, then, two learning strategies are designed from-easy-to-difficult and from-difficult-to-easy to train the model. Finally, the translation effects of the model are compared. The experimental results show that both from-easy-to-difficult and from-difficult-to-easy dynamic learning methods can improve the translation effect of the neural machine translation model.

Key words: neural machine translation, curriculum learning, sample difficulty, dynamic learning

CLC Number: 

  • TP391
[1] 叶绍林.基于注意力机制编解码框架的神经机器翻译方法研究[D].合肥:中国科学技术大学,2019.
[2] WANG R,UTIYAMA M,SUMITA E.Dynamic sentence sampling for efficient training of neural machine translation[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics(Volume 2:Short Papers).Stroudsburg,PA:Association for Computational Linguistics,2018:298-304.
[3] BENGIO Y,LOURADOUR J,COLLOBERT R,et al.Curriculum learning[C]//Proceedings of the 26th Annual International Conference on Machine Learning.New York,NY:Association for Computing Machinery,2009:41-48.
[4] KALCHBRENNER N,BLUNSOM P.Recurrent continuous translation models[C]//Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing.Stroudsburg,PA:Association for Computational Linguistics,2013:1700-1709.
[5] CHO K,Van MERRIËNBOER B,GULCEHRE C,et al.Learning phrase representations using RNN encoder-decoder for statistical machine translation[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing.Stroudsburg,PA:Association for Computational Linguistics,2014:1724-1734.
[6] SUTSKEVER I,VINYALS O,LE Q V.Sequence to sequence learning with neural networks[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems:Volume 2.Cambridge,MA:MIT Press,2014:3104-3112.
[7] BAHDANAU D,CHO K H,BENGIO Y.Neural machine translation by jointly learning to align and translate[EB/OL].(2016-05-19)[2020-08-26].https://arxiv.org/pdf/1409.0473.pdf.
[8] WU Y H,SCHUSTER M,CHEN Z F,et al.Google’s neural machine translation system:Bridging the gap between human and machine translation[EB/OL].(2016-10-08)[2020-08-26].https://arxiv.org/pdf/1609.08144.pdf.
[9] GEHRING J,AULI M,GRANGIER D,et al.Convolutional sequence to sequence learning[J].Proceedings of Machine Learning Research,2017,70:1243-1252.
[10] VASWANI A,SHAZEER N,PARMAR N,et al.Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems.Red Hook,NY:Curran Associates Inc.,2017:6000-6010.
[11] TSVETKOV Y,FARUQUI M,LING W,et al.Learning the curriculum with bayesian optimization for task-specific word representation learning[C]//Proceedings of the 54th Annual Meeting on Association for Computational Linguistics(Volume 1:Long Papers).Stroudsburg,PA:Association for Computational Linguistics,2016:130-139.DOI:10.18653/v1/P16-1013.
[12] CIRIK V,HOVY E,MORENCY L P.Visualizing and understanding curriculum learning for long short-term memory networks[EB/OL].(2016-11-18)[2020-08-26].https://arxiv.org/pdf/1611.06204.pdf.
[13] KOCMI T,BOJAR O.Curriculum learning and minibatch bucketing in neural machine translation[EB/OL].(2017-07-29)[2020-08-26].https://arxiv.org/pdf/1707.09533v1.pdf.
[14] ZHANG X,KUMAR G,KHAYRALLAH H,et al.An empirical exploration of curriculum learning for neural machine translation[EB/OL].(2018-11-02)[2020-08-26].https://arxiv.org/pdf/1811.00739.pdf.
[15] KUDO T,RICHARDSON J.SentencePiece:A simple and language independent subword tokenizer and detokenizer for neural text processing[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing:System Demonstrations.Stroudsburg,PA:Association for Computational Linguistics,2018:66-71.DOI:10.18653/v1/D18-2012.
[16] PAPINENI K,ROUKOS S,WARD T,et al.Bleu:a method for automatic evaluation of machine translation[C]//Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics.Stroudsburg,PA:Association for Computational Linguistics,2002:311-318.DOI:10.3115/1073083.1073135.
[1] YANG Zhou, FAN Yixing, ZHU Xiaofei, GUO Jiafeng, WANG Yue. Survey on Modeling Factors of Neural Information Retrieval Model [J]. Journal of Guangxi Normal University(Natural Science Edition), 2021, 39(2): 1-12.
[2] ZHUO Ming, LIU Leyuan, ZHOU Shijie, YANG Peng, WAN Simin. A New Method for Invulnerability Analysis of Spatial Information Networks [J]. Journal of Guangxi Normal University(Natural Science Edition), 2021, 39(2): 21-31.
[3] DENG Wenxuan, YANG Hang, JIN Ting. A Dimensionality-reduction Method Based on Attention Mechanismon Image Classification [J]. Journal of Guangxi Normal University(Natural Science Edition), 2021, 39(2): 32-40.
[4] XU Qingting, ZHANG Lanfang, ZHU Xinhua. An Automatic Scoring Method for Subjective Questions Using Semantic Technologies and LSTM [J]. Journal of Guangxi Normal University(Natural Science Edition), 2021, 39(2): 51-61.
[5] ZHU Yongjian, LUO Jian, QIN Yunbai, QIN Guofeng, TANG Chuliu. A Method for Detecting Metal Surface Defects Based on Photometric Stereo and Series Expansion Methods [J]. Journal of Guangxi Normal University(Natural Science Edition), 2020, 38(6): 21-31.
[6] TANG Rongchai, WU Xiru. Real-time Detection of Passion Fruit Based on Improved YOLO-V3 Network [J]. Journal of Guangxi Normal University(Natural Science Edition), 2020, 38(6): 32-39.
[7] ZHANG Canlong, LI Yanru, LI Zhixin, WANG Zhiwen. Block Target Tracking Based on Kernel Correlation Filter and Feature Fusion [J]. Journal of Guangxi Normal University(Natural Science Edition), 2020, 38(5): 12-23.
[8] WANG Jian, ZHENG Qifan, LI Chao, SHI Jing. Remote Supervision Relationship Extraction Based on Encoder and Attention Mechanism [J]. Journal of Guangxi Normal University(Natural Science Edition), 2019, 37(4): 53-60.
[9] XIAO Yiqun, SONG Shuxiang, XIA Haiying. Fast Pedestrian Detection Method Based on Multi-Features    and Implementation [J]. Journal of Guangxi Normal University(Natural Science Edition), 2019, 37(4): 61-67.
[10] WANG Xun, LI Tinghui, PAN Xiao, TIAN Yu. Image Segmentation Method Based on Improved Fuzzy C-means Clustering and Otsu Maximum Variance [J]. Journal of Guangxi Normal University(Natural Science Edition), 2019, 37(4): 68-73.
[11] CHEN Feng,MENG Zuqiang. Topic Discovery in Microblog Based on BTM and Weighting K-Means [J]. Journal of Guangxi Normal University(Natural Science Edition), 2019, 37(3): 71-78.
[12] ZHANG Suiyuan, XUE Yuanhai, YU Xiaoming, LIU Yue, CHENG Xueqi. Research on Short Summary Generation of Multi-Document [J]. Journal of Guangxi Normal University(Natural Science Edition), 2019, 37(2): 60-74.
[13] SUN Ronghai, SHI Linfu, HUANG Liyan, TANG Zhenjun, YU Chunqiang. Reversible Data Hiding Based on Image Interpolation and Reference Matrix [J]. Journal of Guangxi Normal University(Natural Science Edition), 2019, 37(2): 90-104.
[14] ZHU Yongjian, PENG Ke, QI Guangwen, XIA Haiying, SONG Shuxiang. Defect Detection of Solar Panel Based on Machine Vision [J]. Journal of Guangxi Normal University(Natural Science Edition), 2019, 37(2): 105-112.
[15] WANG Qi,QIU Jiahui,RUAN Tong,GAO Daqi,GAO Ju. Recurrent Capsule Network for Clinical Relation Extraction [J]. Journal of Guangxi Normal University(Natural Science Edition), 2019, 37(1): 80-88.
Viewed
Full text


Abstract

Cited

  Shared   
  Discussed   
[1] HU Jinming, WEI Duqu. Research on Generalized Sychronization of Fractional-order PMSM[J]. Journal of Guangxi Normal University(Natural Science Edition), 2020, 38(6): 14 -20 .
[2] ZHU Yongjian, LUO Jian, QIN Yunbai, QIN Guofeng, TANG Chuliu. A Method for Detecting Metal Surface Defects Based on Photometric Stereo and Series Expansion Methods[J]. Journal of Guangxi Normal University(Natural Science Edition), 2020, 38(6): 21 -31 .
[3] YANG Liting, LIU Xuecong, FAN Penglai, ZHOU Qihai. Research Progress in Vocal Communication of Nonhuman Primates in China[J]. Journal of Guangxi Normal University(Natural Science Edition), 2021, 39(1): 1 -9 .
[4] BIN Shiyu, LIAO Fang, DU Xuesong, XU Yilan, WANG Xin, WU Xia, LIN Yong. Research Progress on Cold Tolerance of Tilapia[J]. Journal of Guangxi Normal University(Natural Science Edition), 2021, 39(1): 10 -16 .
[5] LIU Jing, BIAN Xun. Characteristics of the Orthoptera Mitogenome and Its Application[J]. Journal of Guangxi Normal University(Natural Science Edition), 2021, 39(1): 17 -28 .
[6] LI Xingkang, ZHONG Enzhu, CUI Chunyan, ZHOU Jia, LI Xiaoping, GUAN Zhenhua. Monitoring Singing Behavior of Western Black Crested Gibbon (Nomascus concolor furvogaster)[J]. Journal of Guangxi Normal University(Natural Science Edition), 2021, 39(1): 29 -37 .
[7] HE Xinming, XIA Wancai, BA Sang, LONG Xiaobin, LAI Jiandong, YANG Chan, WANG Fan, LI Dayong. Grooming Strategies of Resident Males with Different Number of Mates in Yunnan Snub-nosed Monkeys (Rhinopithecus bieti)[J]. Journal of Guangxi Normal University(Natural Science Edition), 2021, 39(1): 38 -44 .
[8] FU Wen, REN Baoping, LIN Jianzhong, LUAN Ke, WANG Pengcheng, WANG Bing, LI Dayong, ZHOU Qihai. Jiyuan Taihang Mountain Macaque Population and Conservation Status[J]. Journal of Guangxi Normal University(Natural Science Edition), 2021, 39(1): 45 -52 .
[9] ZHENG Jingjin, LIANG Jipeng, ZHANG Kechu, HUANG Aimian, LU Qian, LI Youbang, HUANG Zhonghao. White-headed Langurs Select Foods Based on Woody Plants' Dominances[J]. Journal of Guangxi Normal University(Natural Science Edition), 2021, 39(1): 53 -64 .
[10] YANG Chan, WAN Yaqiong, HUANG Xiaofu, YUAN Xudong, ZHOU Hongyan, FANG Haocun, LI Dayong, LI Jiaqi. Activity Rhythm of Muntiacus reevesi Based on Infrared Camera Technology[J]. Journal of Guangxi Normal University(Natural Science Edition), 2021, 39(1): 65 -70 .