Teburin Abubuwan Ciki
- 1. Gabatarwa
- 2. Ayyukan da suka danganci
- 3. Hanyoyin Aiki
- 4. Gwaje-gwaje
- 5. Bincike
- 6. Kammalawa
- 7. Bincike na Asali
- 8. Cikakkun Bayanai na Fasaha
- 9. Sakamakon Gwaji
- 10. Nazarin Shari'a
- 11. Hanyoyi na gaba
- 12. Manazarta
1. Gabatarwa
Binciken koyon harshe na biyu (SLA) ya yi nazari mai zurfi kan canjin harshe, wato tasirin tsarin harshen asalin mai magana [L1] a kan nasarar koyon wani harshe na waje [L2]. Tasirin irin wannan canji na iya zama mai kyau (yana sauƙaƙe koyo) ko mara kyau (yana hana koyo). Mun gano cewa wallafe-wallafen NLP ba su ba da isasshen kulawa ga al'amarin canji mara kyau ba. Don fahimtar al'amuran canji mai kyau da mara kyau tsakanin L1 da L2, muna yin samfurin koyon harshe na biyu a jere a cikin LMs. Bugu da ƙari, mun gina tarin bayanai na MAO-CHILDES wanda ya ƙunshi harsuna biyar daban-daban, wato Jamusanci, Faransanci, Yaren mutanen Poland, Indonesiya, da Jafananci don fahimtar yadda maganganun yara na asali [L1] za su iya taimakawa ko kuma yin rikici da koyon harshen Ingilishi [L2].
2. Ayyukan da suka danganci
Canjin harshe ya sami kulawa sosai a binciken NLP (Wu da Dredze, 2019; Wu et al., 2019; Conneau et al., 2017, 2018; Artetxe et al., 2018; Ruder et al., 2017). Yawancin wannan binciken ya mayar da hankali kan abubuwan amfani kamar yadda madaidaicin tokenizer zai iya inganta canjin harshe, kuma bai kalli irin alakar canji a jere da ke tasowa a cikin koyon harshe na biyu na ɗan adam ba. Hanyoyi kamar Gwajin Inductive Bias ta hanyar Canjin Samfurin Harshe (TILT) (Papadimitriou da Jurafsky, 2020) suna mai da hankali kan canji mai kyau tare da nau'ikan horarwa daban-daban, kamar kiɗan MIDI da Sifaniyanci, don haskaka irin bayanan da ke haifar da siffofi na tsarin gama gari waɗanda bayanan harshe da marasa harshe ke rabawa.
3. Hanyoyin Aiki
3.1 Gina Tarin Bayanai
Mun gina tarin bayanai na MAO-CHILDES daga ma'ajiyar bayanai ta CHILDES, inda muka zaɓi maganganun da aka yi wa yara daga harsuna biyar: Jamusanci (Germanic), Faransanci (Romance), Yaren mutanen Poland (Slavic), Indonesiya (Austronesian), da Jafananci (Japonic). An tsara tarin bayanai bisa shekaru don kwaikwayi yanayin koyon harshe a jere. Kowane rukunin harshe yana ƙunshe da kusan maganganu 50,000 daga masu kula da yara masu shekaru 2-5.
3.2 Tsarin Samfurin
Tsarin SLABERT ɗinmu ya dogara ne akan tsarin BERT-base (Devlin et al., 2019) mai yadudduka na transformer 12, girma ɓoye 768, da kawunan hankali 12. Muna amfani da tsarin horarwa na mataki biyu: da farko, an horar da samfurin akan bayanan L1 CDS, sannan a sake gyara shi akan bayanan L2 (Ingilishi) CDS. Wannan horarwa a jere tana kwaikwayi tsarin SLA na ɗan adam inda ake koyon L1 kafin L2.
3.3 Hanyar Horarwa
Hanyar horarwa tana bin tsarin koyon canjin harshe na TILT. An fara horar da samfurin akan bayanan L1 ta amfani da manufar yin samfurin harshe mai ɓoye (MLM) tare da ƙimar ɓoye 15%. Bayan haka, an sake gyara samfurin akan bayanan CDS na Ingilishi tare da manufar MLM iri ɗaya. An ayyana aikin asara kamar haka:
$\mathcal{L}_{MLM} = -\sum_{i \in \mathcal{M}} \log P(x_i | x_{\backslash \mathcal{M}})$
inda $\mathcal{M}$ shine saitin wuraren da aka ɓoye kuma $x_{\backslash \mathcal{M}}$ yana wakiltar alamomin da ba a ɓoye ba.
4. Gwaje-gwaje
4.1 Saitin Gwaji
Muna kimanta samfuranmu akan jerin gwaje-gwajen nahawu na BLiMP (Benchmark of Linguistic Minimal Pairs for English) (Warstadt et al., 2020), wanda ya ƙunshi al'amuran nahawu 67 da aka tsara cikin rukuni 13. Muna kwatanta samfuran da aka horar da su akan harsuna daban-daban na L1 tare da samfurin tushe wanda aka horar da shi akan bayanan CDS na Ingilishi kawai. Ma'aunin kimantawa shine daidaito akan saitin gwajin BLiMP.
4.2 Sakamako
Tebur 1 yana nuna daidaiton BLiMP ga samfuran da aka horar da su tare da harsuna daban-daban na L1. Jamusanci L1 yana nuna mafi girman canji mai kyau (85.2%), yayin da Jafananci L1 ya nuna mafi ƙanƙanta (72.1%), wanda ya yi daidai da hasashen nisan iyalin harshe. Faransanci da Yaren mutanen Poland sun nuna sakamako na tsaka-tsaki (81.3% da 78.6% bi da bi). Indonesiya ta nuna daidaito 76.4%.
5. Bincike
5.1 Canji Mai Kyau da Mara Kyau
Mun lura cewa harsuna daga iyali ɗaya (Germanic) da Ingilishi suna nuna canji mai kyau galibi, yayin da harsuna daga iyalai masu nisa (Japonic) ke nuna canji mara kyau sosai. Wannan ya yi daidai da binciken SLA na ɗan adam wanda ke nuna cewa nisan nau'in harshe yana hasashen tasirin canji (Jarvis da Pavlenko, 2007).
5.2 Nisan Iyalin Harshe
Muna ƙididdige nisan iyalin harshe ta amfani da ma'aunin nisan phylogenetic. Alakar da ke tsakanin nisan iyalin harshe da canji mara kyau tana da mahimmanci a kididdiga (Pearson's r = -0.89, p < 0.05). Wannan yana nuna cewa tsarin SLABERT na iya zama samfurin kwamfuta don nazarin alakar nau'in harshe.
6. Kammalawa
Tsarin SLABERT ɗinmu ya yi nasarar yin samfurin duka canji mai kyau da mara kyau a cikin koyon harshe na biyu. Mun gano cewa nisan iyalin harshe yana hasashen canji mara kyau, kuma bayanan maganganun tattaunawa suna nuna sauƙaƙewa ga koyon harshe fiye da bayanan maganganun rubutacce. Sakamakonmu yana kira ga ƙarin bincike ta amfani da samfuran SLA na Transformer, kuma muna fitar da lambar mu, bayananmu, da samfuranmu don ƙarfafa hakan.
7. Bincike na Asali
Mahimmanci: SLABERT wani ƙoƙari ne mai ƙarfi na haɗa ilimin harshe na kwamfuta da binciken koyon harshe na biyu, amma yana da iyakancewa na asali: yana daidaita horarwar samfurin harshe da koyon harshe na ɗan adam, yana watsi da abubuwan jiki, zamantakewa, da fahimi na SLA. Babban gudunmawar takardar ita ce nuna cewa BERT na iya kwaikwayi tasirin canjin harshe, amma wannan nasara ce taƙaice.
Tsarin Hankali: Marubutan sun fara daga ingantaccen ra'ayi na SLA na canjin harshe, sannan suka gina tsarin kwamfuta don yin samfurinsa. Hankalin yana da kyau: idan LMs za su iya koyon tsarin harshe daga bayanai, to horarwa a jere akan L1 sannan L2 ya kamata ya bayyana tasirin canji. Gina tarin bayanai na MAO-CHILDES wani sabon abu ne na aiki, yana ba da bayanan maganganun yara masu inganci. Amfani da BLiMP don kimantawa ya dace, saboda yana gwada ilimin nahawu.
Ƙarfi da Rashi: Babban ƙarfin shine sabon amfani da koyon canji na TILT ga SLA, wanda ya buɗe sabon alkibla na bincike. Gano cewa nisan iyalin harshe yana hasashen canji mara kyau yana da ƙarfi kuma ya yi daidai da nazarin ɗan adam. Duk da haka, takardar tana da manyan rashi. Na farko, girman samfurin harsuna biyar ya yi ƙanƙanta don ƙarshe na nau'in harshe mai ƙarfi. Na biyu, samfurin bai yi la'akari da tasirin shekarun koyo ba, wanda ke da mahimmanci a SLA na ɗan adam (Lenneberg, 1967). Na uku, kimantawa ya iyakance ga nahawun Ingilishi; ba mu sani ba ko samfurin ya yi aiki ga wasu L2s. Na huɗu, takardar ba ta da kwatance da samfuran SLA na gargajiya kamar Samfurin Gasa (MacWhinney, 2005).
Abubuwan da za a iya amfani da su: Ga masu bincike, wannan aikin yana nuna cewa samfuran Transformer na iya zama kayan aiki masu amfani don binciken SLA, amma dole ne a haɗa su da samfuran fahimi. Ga masu aiki, gano cewa bayanan maganganun tattaunawa sun fi tasiri fiye da bayanan rubutacce yana da tasiri ga kayan koyar da harshe. Ayyukan gaba ya kamata su faɗaɗa samfurin harshe, su haɗa da shekarun koyo a matsayin mai canzawa, kuma su gwada akan L2s da yawa. Fitar da lamba da bayanai na takardar abin yabo ne kuma ya kamata ya sauƙaƙe maimaitawa da faɗaɗawa.
8. Cikakkun Bayanai na Fasaha
Samfurin SLABERT yana amfani da tsarin BERT-base tare da sigogi 110M. Ma'aunin horarwa sune: ƙimar koyo 2e-5, girman batch 32, matsakaicin tsayin jeri 128, da kuma lokutan horarwa 10 don horarwar L1 da 5 don gyaran L2. Ingantawa yana amfani da AdamW tare da raguwar nauyi 0.01. Manufar MLM tana ɓoye 15% na alamomi, tare da 80% maye gurbinsu da [MASK], 10% maye gurbinsu da alamomin bazuwar, kuma 10% ba a canza su ba.
Tsarin lissafi na manufar koyon canji shine:
$\mathcal{L}_{transfer} = \mathcal{L}_{MLM}^{L1} + \lambda \cdot \mathcal{L}_{MLM}^{L2}$
inda $\lambda$ shine ma'aunin sikeli wanda aka saita zuwa 0.5 a gwaje-gwajenmu.
9. Sakamakon Gwaji
Hoto 1 (ba a nuna shi ba) yana gabatar da ginshiƙi mai kwatanta daidaiton BLiMP a cikin harsunan L1. Tushen (Ingilishi kawai) ya sami daidaito 83.5%. Jamusanci L1 ya nuna mafi girman ingantawa (+1.7%), yayin da Jafananci L1 ya nuna mafi girman raguwa (-11.4%). Faransanci da Yaren mutanen Poland sun nuna sakamako na tsaka-tsaki. Sakamakon ya tabbatar da cewa nisan nau'in harshe yana da alaƙa da canji mara kyau.
Tebur 1: Daidaiton BLiMP ta Harshen L1
| Harshen L1 | Daidaito (%) | Canji daga Tushen |
|---|---|---|
| Ingilishi (Tushen) | 83.5 | - |
| Jamusanci | 85.2 | +1.7 |
| Faransanci | 81.3 | -2.2 |
| Yaren mutanen Poland | 78.6 | -4.9 |
| Indonesiya | 76.4 | -7.1 |
| Jafananci | 72.1 | -11.4 |
10. Nazarin Shari'a
Yi la'akari da al'amarin nahawu na Ingilishi na yarda da batun-fi'ili. A Jamusanci, wanda ke da irin wannan tsarin yarda, samfurin yana nuna babban daidaito (92%). A Jafananci, wanda ba shi da yarda da mutum-lamba, samfurin yana nuna ƙarancin daidaito (65%). Wannan yana nuna canji mara kyau: nahawun L1 yana tsoma baki cikin koyon L2. Misalin jimla biyu daga BLiMP:
Nahawu: "The dogs run fast."
Mara nahawu: "The dogs runs fast."
Samfurin Jamusanci L1 ya gano jimlar nahawu daidai 92% na lokaci, yayin da samfurin Jafananci L1 kawai 65% na lokaci.
11. Hanyoyi na gaba
Tsarin SLABERT yana buɗe hanyoyi da yawa don bincike na gaba. Na farko, faɗaɗa samfurin harshe don haɗa da ƙarin harsuna daban-daban (misali, Larabci, Mandarin, Swahili) zai ƙarfafa sakamakon. Na biyu, haɗa da shekarun koyo a matsayin mai canzawa zai iya yin samfurin tasirin lokaci mai mahimmanci a SLA (Lenneberg, 1967). Na uku, gwaji akan L2s da yawa (misali, Sifaniyanci, Faransanci) zai gwada yadda tsarin ke aiki. Na huɗu, haɗa SLABERT tare da samfuran fahimi kamar Samfurin Gasa (MacWhinney, 2005) na iya ba da kwaikwayo mafi gaskiya. Na biyar, amfani da tsarin don nazarin raguwar harshe (asarar L1 saboda rinjayen L2) wani faɗaɗa ne na halitta. A ƙarshe, ana iya amfani da tsarin don haɓaka kayan aikin koyon harshe na keɓaɓɓu waɗanda suka dace da L1 na mai koyo.
12. Manazarta
- Artetxe, M., Labaka, G., & Agirre, E. (2018). A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In Proceedings of ACL.
- Berzak, Y., Barbu, A., Harari, D., Katz, B., & Ullman, S. (2014). Do you see what I mean? Visual resolution of linguistic ambiguities. In Proceedings of EMNLP.
- Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2017). Word translation without parallel data. In Proceedings of ICLR.
- Conneau, A., Rinott, R., Lample, G., Williams, A., Bowman, S. R., Schwenk, H., & Stoyanov, V. (2018). XNLI: Evaluating cross-lingual sentence representations. In Proceedings of EMNLP.
- Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT.
- Jarvis, S., & Pavlenko, A. (2007). Crosslinguistic Influence in Language and Cognition. Routledge.
- Lenneberg, E. H. (1967). Biological Foundations of Language. Wiley.
- MacWhinney, B. (2005). A unified model of language acquisition. In Handbook of Bilingualism: Psycholinguistic Approaches.
- Papadimitriou, I., & Jurafsky, D. (2020). Learning Music Helps You Read: Using transfer to study linguistic structure in language models. In Proceedings of EMNLP.
- Ruder, S., Vulić, I., & Søgaard, A. (2017). A survey of cross-lingual word embedding models. Journal of Artificial Intelligence Research, 65, 569-631.
- Warstadt, A., Parrish, A., Liu, H., Mohananey, A., Peng, W., Wang, S.-F., & Bowman, S. R. (2020). BLiMP: The Benchmark of Linguistic Minimal Pairs for English. Transactions of the ACL, 8, 377-392.
- Wu, S., & Dredze, M. (2019). Beto, Bentz, Becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of EMNLP.
- Wu, S., Conneau, A., Li, H., Zettlemoyer, L., & Stoyanov, V. (2019). Emerging cross-lingual structure in pretrained language models. In Proceedings of ACL.