Table of Contents
- 1. Introduction
- 2. Related Work
- 3. Methodology
- 4. Experiments
- 5. Analysis
- 6. Conclusion
- 7. Original Analysis
- 8. Technical Details
- 9. Experimental Results
- 10. Case Study
- 11. Future Directions
- 12. References
1. Introduction
Second language acquisition (SLA) research has extensively studied cross-linguistic transfer, the influence of linguistic structure of a speaker's native language [L1] on the successful acquisition of a foreign language [L2]. Effects of such transfer can be positive (facilitating acquisition) or negative (impeding acquisition). We find that NLP literature has not given enough attention to the phenomenon of negative transfer. To understand patterns of both positive and negative transfer between L1 and L2, we model sequential second language acquisition in LMs. Further, we build a Multilingual Age Ordered CHILDES (MAO-CHILDES) dataset consisting of 5 typologically diverse languages, i.e., German, French, Polish, Indonesian, and Japanese to understand the degree to which native Child-Directed Speech (CDS) [L1] can help or conflict with English language acquisition [L2].
2. Related Work
Cross-lingual transfer has received considerable attention in NLP research (Wu and Dredze, 2019; Wu et al., 2019; Conneau et al., 2017, 2018; Artetxe et al., 2018; Ruder et al., 2017). Most of this research has concentrated on practical implications such as the degree to which the right tokenizer can optimize cross-lingual transfer, and has not looked at the kind of sequential transfer relationships that arise in human second language acquisition. Approaches like the Test for Inductive Bias via Language Model Transfer (TILT) (Papadimitriou and Jurafsky, 2020) focus on positive transfer with divergent pairs of training sets, such as MIDI music and Spanish, to shed light on which kinds of data induce generalizable structural features that linguistic and non-linguistic data share.
3. Methodology
3.1 Dataset Construction
We constructed the MAO-CHILDES dataset from the CHILDES database, selecting child-directed speech from five languages: German (Germanic), French (Romance), Polish (Slavic), Indonesian (Austronesian), and Japanese (Japonic). The dataset is age-ordered to simulate the sequential nature of language acquisition. Each language subset contains approximately 50,000 utterances from caregivers directed at children aged 2-5 years.
3.2 Model Architecture
Our SLABERT framework is based on the BERT-base architecture (Devlin et al., 2019) with 12 transformer layers, 768 hidden dimensions, and 12 attention heads. We employ a two-stage training process: first, the model is pre-trained on L1 CDS data, then fine-tuned on L2 (English) CDS data. This sequential training mirrors the human SLA process where L1 is acquired before L2.
3.3 Training Procedure
The training procedure follows the TILT-based cross-lingual transfer learning approach. The model is first trained on L1 data using masked language modeling (MLM) objective with a masking rate of 15%. Subsequently, the model is fine-tuned on English CDS data with the same MLM objective. The loss function is defined as:
$\mathcal{L}_{MLM} = -\sum_{i \in \mathcal{M}} \log P(x_i | x_{\backslash \mathcal{M}})$
where $\mathcal{M}$ is the set of masked positions and $x_{\backslash \mathcal{M}}$ represents the unmasked tokens.
4. Experiments
4.1 Experimental Setup
We evaluate our models on the BLiMP (Benchmark of Linguistic Minimal Pairs for English) grammar test suite (Warstadt et al., 2020), which contains 67 grammatical phenomena organized into 13 categories. We compare models trained on different L1 languages against a baseline model trained only on English CDS data. The evaluation metric is accuracy on the BLiMP test set.
4.2 Results
Table 1 shows the BLiMP accuracy for models trained with different L1 languages. German L1 shows the highest positive transfer (85.2%), while Japanese L1 shows the lowest (72.1%), consistent with language family distance predictions. French and Polish show intermediate results (81.3% and 78.6% respectively). Indonesian shows 76.4% accuracy.
5. Analysis
5.1 Positive vs Negative Transfer
We observe that languages from the same family (Germanic) as English show predominantly positive transfer, while languages from distant families (Japonic) show significant negative transfer. This aligns with human SLA research showing that typological distance predicts transfer effects (Jarvis and Pavlenko, 2007).
5.2 Language Family Distance
We quantify language family distance using phylogenetic distance metrics. The correlation between language family distance and negative transfer is statistically significant (Pearson's r = -0.89, p < 0.05). This suggests that the SLABERT framework can serve as a computational model for studying typological relationships.
6. Conclusion
Our SLABERT framework successfully models both positive and negative cross-linguistic transfer effects in second language acquisition. We find that language family distance predicts negative transfer, and conversational speech data shows greater facilitation for language acquisition than scripted speech data. Our findings call for further research using Transformer-based SLA models, and we release our code, data, and models to encourage this.
7. Original Analysis
Core Insight: SLABERT is a bold attempt to bridge computational linguistics and second language acquisition research, but it suffers from a fundamental limitation: it equates language model pre-training with human language acquisition, ignoring the embodied, social, and cognitive dimensions of SLA. The paper's key contribution is demonstrating that BERT can simulate cross-linguistic transfer effects, but this is a narrow victory.
Logical Flow: The authors start from the well-established SLA concept of cross-linguistic transfer, then build a computational framework to model it. The logic is sound: if LMs can learn linguistic structure from data, then sequential training on L1 then L2 should reveal transfer effects. The construction of the MAO-CHILDES dataset is a practical innovation, providing ecologically valid child-directed speech data. The use of BLiMP for evaluation is appropriate, as it tests grammatical knowledge.
Strengths & Flaws: The main strength is the novel application of TILT-based transfer learning to SLA, which opens a new research direction. The finding that language family distance predicts negative transfer is compelling and aligns with human studies. However, the paper has significant flaws. First, the sample size of five languages is too small for robust typological conclusions. Second, the model does not account for age of acquisition effects, which are crucial in human SLA (Lenneberg, 1967). Third, the evaluation is limited to English grammar; we don't know if the model generalizes to other L2s. Fourth, the paper lacks comparison with traditional SLA models like the Competition Model (MacWhinney, 2005).
Actionable Insights: For researchers, this work suggests that Transformer-based models can be useful tools for SLA research, but they must be combined with cognitive models. For practitioners, the finding that conversational speech data is more effective than scripted data has implications for language teaching materials. Future work should expand the language sample, include age of acquisition as a variable, and test on multiple L2s. The paper's release of code and data is commendable and should facilitate replication and extension.
8. Technical Details
The SLABERT model uses the BERT-base architecture with 110M parameters. The training hyperparameters are: learning rate 2e-5, batch size 32, maximum sequence length 128, and training epochs 10 for L1 pre-training and 5 for L2 fine-tuning. The optimization uses AdamW with weight decay 0.01. The MLM objective masks 15% of tokens, with 80% replaced by [MASK], 10% replaced by random tokens, and 10% unchanged.
The mathematical formulation of the transfer learning objective is:
$\mathcal{L}_{transfer} = \mathcal{L}_{MLM}^{L1} + \lambda \cdot \mathcal{L}_{MLM}^{L2}$
where $\lambda$ is a scaling factor set to 0.5 in our experiments.
9. Experimental Results
Figure 1 (not shown) presents a bar chart comparing BLiMP accuracy across L1 languages. The baseline (English-only) achieves 83.5% accuracy. German L1 shows the highest improvement (+1.7%), while Japanese L1 shows the largest drop (-11.4%). French and Polish show intermediate effects. The results confirm that typological distance correlates with negative transfer.
Table 1: BLiMP Accuracy by L1 Language
| L1 Language | Accuracy (%) | Change from Baseline |
|---|---|---|
| English (Baseline) | 83.5 | - |
| German | 85.2 | +1.7 |
| French | 81.3 | -2.2 |
| Polish | 78.6 | -4.9 |
| Indonesian | 76.4 | -7.1 |
| Japanese | 72.1 | -11.4 |
10. Case Study
Consider the English grammatical phenomenon of subject-verb agreement. In German, which has similar agreement patterns, the model shows high accuracy (92%). In Japanese, which lacks person-number agreement, the model shows low accuracy (65%). This demonstrates negative transfer: the L1 grammar interferes with L2 acquisition. A sample sentence pair from BLiMP:
Grammatical: "The dogs run fast."
Ungrammatical: "The dogs runs fast."
The German L1 model correctly identifies the grammatical sentence 92% of the time, while the Japanese L1 model only 65% of the time.
11. Future Directions
The SLABERT framework opens several avenues for future research. First, expanding the language sample to include more typologically diverse languages (e.g., Arabic, Mandarin, Swahili) would strengthen the findings. Second, incorporating age of acquisition as a variable could model critical period effects in SLA (Lenneberg, 1967). Third, testing on multiple L2s (e.g., Spanish, French) would test the generalizability of the framework. Fourth, combining SLABERT with cognitive models like the Competition Model (MacWhinney, 2005) could provide more realistic simulations. Fifth, applying the framework to study language attrition (loss of L1 due to L2 dominance) is a natural extension. Finally, the framework could be used to develop personalized language learning tools that adapt to the learner's L1.
12. References
- Artetxe, M., Labaka, G., & Agirre, E. (2018). A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In Proceedings of ACL.
- Berzak, Y., Barbu, A., Harari, D., Katz, B., & Ullman, S. (2014). Do you see what I mean? Visual resolution of linguistic ambiguities. In Proceedings of EMNLP.
- Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., & Stoyanov, V. (2017). Word translation without parallel data. In Proceedings of ICLR.
- Conneau, A., Rinott, R., Lample, G., Williams, A., Bowman, S. R., Schwenk, H., & Stoyanov, V. (2018). XNLI: Evaluating cross-lingual sentence representations. In Proceedings of EMNLP.
- Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT.
- Jarvis, S., & Pavlenko, A. (2007). Crosslinguistic Influence in Language and Cognition. Routledge.
- Lenneberg, E. H. (1967). Biological Foundations of Language. Wiley.
- MacWhinney, B. (2005). A unified model of language acquisition. In Handbook of Bilingualism: Psycholinguistic Approaches.
- Papadimitriou, I., & Jurafsky, D. (2020). Learning Music Helps You Read: Using transfer to study linguistic structure in language models. In Proceedings of EMNLP.
- Ruder, S., Vulić, I., & Søgaard, A. (2017). A survey of cross-lingual word embedding models. Journal of Artificial Intelligence Research, 65, 569-631.
- Warstadt, A., Parrish, A., Liu, H., Mohananey, A., Peng, W., Wang, S.-F., & Bowman, S. R. (2020). BLiMP: The Benchmark of Linguistic Minimal Pairs for English. Transactions of the ACL, 8, 377-392.
- Wu, S., & Dredze, M. (2019). Beto, Bentz, Becas: The surprising cross-lingual effectiveness of BERT. In Proceedings of EMNLP.
- Wu, S., Conneau, A., Li, H., Zettlemoyer, L., & Stoyanov, V. (2019). Emerging cross-lingual structure in pretrained language models. In Proceedings of ACL.