Prompting ChatGPT for Chinese Learning as L2: A CEFR and EBCL Level Study

1. Introduction

ChatGPT represents a significant advancement in natural language understanding and generation, offering versatile assistance for communication and learning tasks. Its widespread use raises central questions about the relevance of chatbots for language teaching, particularly for Chinese. This study explores how learners can use specific prompts to engage Large Language Models (LLMs) as personalized chatbots, aiming to target language levels based on the Common European Framework of Reference for Languages (CEFR) and the European Benchmarking Chinese Language (EBCL) project, specifically focusing on A1, A1+, and A2 levels.

2. Literature Review & Theoretical Framework

The integration of AI in education, especially for language learning, builds upon decades of chatbot evolution, from ELIZA to modern generative AI.

3. Methodology: Prompt Engineering for Level Targeting

The core methodology involves designing precise prompts to constrain LLM outputs to specific proficiency levels.

4. Experimental Setup & Results

A systematic series of experiments evaluated ChatGPT's adherence to prompt constraints.

5. Discussion: LLMs as Personalized Tutors

6. Core Insight & Analyst's Perspective

Core Insight: This research isn't just about using AI for language learning; it's a pioneering blueprint for constraining generative AI's boundless creativity to fit pedagogical frameworks. The real innovation is treating the prompt not as a simple query, but as a runtime pedagogical controller—a set of instructions that dynamically filters the LLM's vast knowledge to deliver grade-appropriate content. This moves beyond the chatbot as a conversation partner to the chatbot as a curriculum-aware tutor.

Logical Flow: The study correctly identifies the core problem: unfettered LLMs are terrible for beginners because they lack built-in pedagogical guardrails. Their solution is elegantly simple: inject those guardrails via prompt engineering. The logic flows from problem (uncontrolled output) to mechanism (EBCL lists as constraints) to validation (measuring adherence). It mirrors techniques in other AI domains, like using conditioning in generative models (e.g., guiding image generation in models like Stable Diffusion with specific descriptors) to steer output towards a desired distribution, formalized as learning a conditional probability $P(\text{output} | \text{prompt, EBCL constraint})$.

Strengths & Flaws: The strength is in its practical, immediately applicable methodology. Any teacher can replicate this. However, the flaw is its narrow focus on lexical compliance. It measures if the AI uses the right words, but not if it constructs pedagogically sound sequences, corrects errors effectively, or scaffolds complexity—key features of human tutoring. As noted in the seminal "Zone of Proximal Development" theory (Vygotsky), effective tutoring dynamically adjusts to the learner's edge of capability. Current prompt engineering is static; the next frontier is dynamic, AI-driven adjustment of these very prompts based on learner interaction.

Actionable Insights: For EdTech companies: The low-hanging fruit is building prompt libraries for each CEFR level and skill (listening, character recognition). For researchers: The priority must shift from constraint adherence to learning outcome validation. Conduct A/B tests comparing prompt-guided AI practice against traditional digital tools. For policymakers: This study provides a concrete argument for urgently developing standardized "pedagogical API" specifications for AI in education—common formats for communicating learning objectives and constraints to any LLM, akin to the SCORM standard for e-learning content.

7. Technical Details & Mathematical Framework

The prompting strategy can be framed as an optimization problem where the goal is to maximize the probability of the LLM generating pedagogically appropriate text ($T$) given a prompt ($P$) that encodes the EBCL constraints ($C$).

The core objective is to maximize $P(T | P, C)$, where $C$ represents the set of allowable characters/vocabulary for the target level (e.g., A1). The prompt $P$ acts as a conditioning context, akin to techniques in controlled text generation.

A simplified scoring function $S(T)$ to evaluate output adherence could be defined as:

$S(T) = \frac{1}{|T_c|} \sum_{c_i \in T_c} \mathbb{1}(c_i \in C)$

where $T_c$ is the set of unique characters in the generated text $T$, $\mathbb{1}$ is the indicator function, and $C$ is the EBCL constraint set. A score of 1.0 indicates perfect adherence. The study's effective prompts increase the expected value $E[S(T)]$.

This relates to the concept of probability masking in decoder-only transformers (the architecture behind models like GPT), where token probabilities for tokens not in $C$ are set to zero before sampling.

8. Results, Charts & Experimental Findings

Primary Result: The inclusion of explicit character list constraints in the prompt led to a statistically significant reduction in out-of-vocabulary (OOV) character usage in ChatGPT's generated dialogues and exercises.

Hypothetical Chart Description (Based on Findings): A bar chart comparing two conditions would show:

• Condition A (Generic Prompt): "Act as a Chinese tutor for a beginner." Results in high OOV rate (e.g., 25-40% of characters outside A1 list), as the model draws from its full vocabulary.
• Condition B (Constrained Prompt): "Act as a Chinese tutor for a CEFR A1 learner. Use only the following characters in your responses: [List of A1 characters]." Results in a dramatically lower OOV rate (e.g., 5-10%), demonstrating effective constraint adherence.

Key Insight from Results: The model's ability to follow complex, embedded instructions (the character list) validates the feasibility of using prompt engineering as a lightweight "API" for pedagogical control, without fine-tuning the model itself.

9. Analysis Framework: Example Prompting Case

Scenario: Generating a simple dialogue for an A1 learner practicing greetings and asking about well-being.

Weak Prompt (Leads to Uncontrolled Output):
"Generate a short dialogue in Chinese between two people meeting."
Risk: The model may use vocabulary and structures far beyond A1.

Strong, Pedagogically-Constrained Prompt (Based on Study Methodology):

You are an AI Chinese tutor specialized in teaching absolute beginners at the CEFR A1 level.

**TASK:** Generate a practice dialogue for a learner.

**STRICT CONSTRAINTS:**
1. **Vocabulary/Characters:** Use ONLY characters from the official EBCL A1 character list (provided below). Do not use any characters outside this list.
   [List: 你, 好, 我, 叫, 吗, 很, 呢, 什么, 名字, 是, 不, 人, 国, 哪, 里, 的, 了, 有, 在, 和, ...]
2. **Grammar:** Use only simple SVO sentences and A1-level grammar points (e.g., 是 sentence, 吗 questions).
3. **Topic:** The dialogue should be about "greetings and asking how someone is."
4. **Output Format:** First, provide the Chinese dialogue with Pinyin above each character. Then, provide an English translation.

**Begin the dialogue.**

This prompt exemplifies the study's approach by embedding the pedagogical framework (CEFR A1, EBCL list) directly into the instruction set, transforming the LLM from a general text generator into a targeted teaching assistant.

10. Future Applications & Research Directions

• Dynamic Prompt Adjustment: Developing systems where the AI itself modifies the constraint parameters (e.g., gradually introducing A2 characters) based on real-time assessment of learner performance, moving towards a true Zone of Proximal Development tutor.
• Multimodal Integration: Combining constrained text generation with image generation AI (e.g., DALL-E, Stable Diffusion) to create custom visual aids for the generated vocabulary and dialogues, enhancing comprehension for logographic characters.
• Error Correction & Feedback Loops: Engineering prompts that enable the LLM to not only generate content but also analyze learner input (e.g., typed sentences, spoken transcriptions) and provide corrective feedback tailored to the learner's level.
• Standardization & Interoperability: Creating open standards for "pedagogical prompts" or metadata that can be read by any educational AI tool, similar to the IMS Global Learning Consortium standards. This would allow seamless sharing of level-specific teaching activities across platforms.
• Longitudinal Efficacy Studies: The most critical direction is conducting long-term studies to measure if learning with prompt-constrained AI tutors leads to faster progression, better retention, and higher proficiency compared to traditional methods or unconstrained AI practice.

11. References

1. Adamopoulou, E., & Moussiades, L. (2020). An overview of chatbot technology. Artificial Intelligence Applications and Innovations, 584, 373-383.
2. Council of Europe. (2001). Common European Framework of Reference for Languages: Learning, teaching, assessment. Cambridge University Press.
3. European Benchmarking Chinese Language (EBCL) Project. (n.d.). Official documentation and character lists.
4. Glazer, K. (2023). AI in language education: A review of current tools and future potential. Journal of Educational Technology Systems, 51(4), 456-478.
5. Huang, W. (2022). The impact of generative AI on second language acquisition. Computer Assisted Language Learning, 35(8), 1125-1148.
6. Imran, M. (2023). Personalized learning paths through adaptive AI tutors. International Journal of Artificial Intelligence in Education.
7. Li, J., et al. (2024). ChatGPT and its applications in educational contexts: A systematic review. Computers & Education: Artificial Intelligence, 5, 100168.
8. Vygotsky, L. S. (1978). Mind in society: The development of higher psychological processes. Harvard University Press.
9. Wallace, R. S. (2009). The anatomy of A.L.I.C.E. In Parsing the Turing Test (pp. 181-210). Springer.
10. Wang, Y. (2024). A meta-analysis of the effectiveness of chatbots in language learning. Language Learning & Technology, 28(1), 1-25.
11. Weizenbaum, J. (1966). ELIZA—a computer program for the study of natural language communication between man and machine. Communications of the ACM, 9(1), 36-45.
12. Zhu, J. Y., Park, T., Isola, P., & Efros, A. A. (2017). Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE international conference on computer vision (pp. 2223-2232). (Cited as an example of a conditioning framework in generative AI).