The use of chatbots in language learning has evolved significantly since the 1960s, becoming more sophisticated platforms with the emergence of generative AI. These tools now simulate natural conversations, adapting to individual learners' needs. Our study explores how learners can use specific prompts to engage Large Language Models (LLMs) like ChatGPT as personalized chatbots, aiming to target their language level based on the Common European Framework of Reference for Languages (CEFR) and the European Benchmarking Chinese Language (EBCL) project. Focusing on A1, A1+ and A2 levels, we examine the teaching of Chinese, which presents unique challenges due to its logographic writing system. Our goal is to develop prompts that integrate oral and written skills, using high-frequency character lists and controlling oral lexical productions. The results indicate that incorporating level A1 and A1+ characters, along with the associated reference list, significantly enhances compliance with the EBCL character set. Properly prompted, LLMs can increase exposure to the target language and offer interactive exchanges to develop language skills.
Keywords: Chinese teaching, LLM, Prompting, CEFR, EBCL
ChatGPT is arguably the most advanced chatbot today in terms of natural language understanding and generation, offering versatile assistance for various communication and learning tasks (Li et al., 2024). It is used daily by millions worldwide, raising central questions about chatbots' relevance for language teaching, particularly for Chinese. These tools, thanks to their adaptability, could transform language pedagogy by facilitating personalized learning paths and offering immersive, interactive practice (Imran, 2023; Glazer, 2023). This study investigates the potential of prompt-engineered interactions with LLMs to serve as targeted, level-appropriate tutors for Chinese L2 learners, bridging the gap between AI capability and structured pedagogical frameworks like CEFR/EBCL.
The history begins with ELIZA (Weizenbaum, 1966), a rule-based program simulating conversation. The 1990s saw ALICE (Wallace, 2009), which used AIML for more natural interaction. The 2000s introduced scripted chatbots on platforms like Duolingo. The paradigm shift occurred post-2020 with Generative AI and LLMs like ChatGPT, enabling dynamic, context-aware conversations far beyond predefined rules.
The Common European Framework of Reference for Languages (CEFR) provides a standardized scale (A1-C2) for language proficiency. The European Benchmarking Chinese Language (EBCL) project adapts this framework specifically for Chinese, defining canonical vocabulary and character lists for each sub-level (e.g., A1, A1+). These frameworks provide the measurable targets for our prompt engineering.
Unlike alphabetic languages, Chinese requires mastery of thousands of unique characters (sinograms). This poses a distinct challenge for AI-assisted learning: ensuring character recurrence and controlled introduction aligns with learner level, avoiding cognitive overload from unseen, complex characters.
Core to the methodology is the design of precise prompts that instruct the LLM to act as a tutor constrained by specific EBCL levels. Example prompt structure: "You are a Chinese language tutor for a beginner at CEFR level A1. Use only vocabulary and characters from the official EBCL A1 list. Our topic is 'greetings and introductions'. Generate a simple dialogue and then ask me 2 comprehension questions."
The study uses high-frequency character lists from EBCL A1 and A1+ levels as a filter. Prompts explicitly include these lists or reference them, instructing the model to avoid characters outside the set. This aims to control both written output and suggested oral lexical productions.
A systematic series of experiments was conducted using different versions of ChatGPT (e.g., GPT-3.5-turbo, GPT-4). Each experiment involved sending batches of level-specific prompts and analyzing the responses for adherence to the character/vocabulary constraints, measuring the proportion of compliant characters used.
The primary finding is that explicitly incorporating the EBCL character list (e.g., A1 list of ~150 characters) into the prompt significantly improves the model's compliance. Responses generated with the list showed a marked reduction in out-of-level characters compared to baseline prompts that merely stated the level without the list.
Prompt designs that requested integrated tasks (e.g., "read this short dialogue, then answer verbally") were successfully generated by the LLM, providing a scaffold for combined skill practice. The controlled lexicon ensured the listening/reading input was comprehensible.
Quantitative analysis showed that compliance rates increased from an average of ~65% with simple level instructions to over ~92% when the prompt included the specific EBCL character list. This difference was statistically significant (p < 0.01), demonstrating the efficacy of detailed prompt engineering.
The study confirms the potential of LLMs as on-demand, personalized tutors. The key is not the model's raw capability, but the pedagogical intelligence encoded in the prompt. A well-crafted prompt can effectively "lock" the model into a specific pedagogical role and constraint set.
Limitations remain: 1) The model may still occasionally generate non-compliant content, requiring learner or teacher oversight. 2) The study focused on lexical control, not grammatical complexity or cultural nuance alignment with CEFR levels. 3) Long-term learning efficacy and motivation impacts were not measured.
This study demonstrates that strategic prompting can harness LLMs like ChatGPT for structured Chinese language learning aligned with CEFR/EBCL standards. By providing explicit character lists and task instructions, the AI's output can be effectively constrained to appropriate learner levels. This opens avenues for scalable, personalized practice. Future work should evaluate long-term learning outcomes, expand to higher CEFR levels (B1-C2), and integrate multimodal interactions (e.g., voice).
Core Insight: This paper isn't about AI's raw linguistic power; it's a masterclass in constraining that power for pedagogy. The real innovation is treating the LLM not as an oracle, but as a high-capability, low-reliability engine that requires a rigorous "pedagogical harness"—the prompt. The authors correctly identify that the value for L2 learning, especially in a complex language like Chinese, lies not in the model's ability to generate fluent Chinese, but in its ability to generate level-appropriate Chinese. This aligns with the core principle of Comprehensible Input (Krashen, 1985), which posits that acquisition occurs when learners understand messages just beyond their current competence (i+1). The prompt engineering described is essentially an algorithmic attempt to operationalize i+1 for an AI.
Logical Flow & Strengths: The methodology is sound and replicable. Starting from the historical context (ELIZA to ChatGPT) grounds the work. Leveraging established frameworks (CEFR/EBCL) provides external validity and practical utility for European educators. The focus on the logographic challenge is astute; controlling character recurrence is a far more concrete and measurable task for an AI than controlling vague grammatical "complexity." The experimental result—that providing the explicit character list boosts compliance from ~65% to ~92%—is the paper's killer finding. It quantitatively proves a point many AI-education papers only speculate on: specificity in prompting is everything. This echoes findings in other AI alignment research, such as the need for detailed "system prompts" to steer model behavior (OpenAI, 2023).
Flaws & Critical Gaps: The analysis, however, stops short of being truly groundbreaking. The major flaw is the lack of a learning outcome evaluation. Compliance is a proxy metric. Does practicing with a 92%-compliant AI tutor actually lead to better character retention, fluency, or confidence than a 65%-compliant one or a human tutor? Without this, it's a tool-efficacy study, not a learning-efficacy study. Secondly, the study seems to assume the EBCL lists are the optimal learning sequence—a debatable premise. AI could potentially personalize sequences better than a fixed list, but the paper doesn't explore dynamic adaptation. Furthermore, it ignores a critical weakness of LLMs: their propensity for "hallucination" or generating plausible but incorrect information about grammar or usage, which can be detrimental for beginners.
Actionable Insights & The Road Ahead: For educators and edtech developers, the takeaway is clear: The prompt is the product. Investing in prompt libraries and engineering is more crucial than chasing the latest model. The next step is to move from static lists to dynamic, adaptive prompting. Imagine a system that uses knowledge tracing (like the models behind Duolingo or Khan Academy) to estimate a learner's mastery of each character and then instructs the LLM to emphasize problematic items—a hybrid AI approach. Furthermore, the field must urgently develop robust evaluation frameworks that measure real learning gains, not just AI output compliance. The ultimate goal should be a seamless Human-AI collaborative tutoring system (HAICTS), where the AI handles drill, repetition, and scalable practice (as demonstrated here), freeing human teachers for higher-order mentorship, cultural instruction, and error correction that AI still cannot reliably provide. This paper provides a solid technical foundation for the AI half of that future.
The core technical concept is using the prompt to impose a filter on the LLM's output probability distribution. When generating the next token (character/word), the LLM samples from a probability distribution $P(x_t | x_{1:t-1}, \theta)$ over its entire vocabulary. An unconstrained prompt leads to sampling from this full distribution.
The prompt engineering in this study effectively creates a masked distribution. By specifying an allowed set of characters $C_{EBCL-A1}$, the desired model behavior is to sample only from the subset of the vocabulary where $x_t \in C_{EBCL-A1}$. This can be conceptualized as applying a mask $M$ where $M_i = 1$ if token $i$ is in the allowed set, else $0$. The adjusted probability for sampling becomes:
$P_{masked}(x_t = i | ...) = \frac{M_i \cdot P(x_t = i | ...)}{\sum_{j} M_j \cdot P(x_t = j | ...)}$
The prompt's effectiveness is measured by the compliance rate $\rho$:
$\rho = \frac{1}{N} \sum_{t=1}^{N} \mathbb{1}(x_t \in C_{EBCL-A1})$
where $N$ is the total number of characters in the response, and $\mathbb{1}$ is the indicator function. The experiment tests the hypothesis that explicitly stating $C_{EBCL-A1}$ in the prompt increases $\rho$ compared to a baseline prompt.
Chart Description (Imagined based on text): A grouped bar chart titled "LLM Output Compliance with EBCL-A1 Character Constraints." The x-axis has two conditions: "Baseline Prompt (Level Only)" and "Enhanced Prompt (Level + Character List)." The y-axis shows "Compliance Rate (%)" from 0% to 100%. For each condition, two bars represent different ChatGPT models (e.g., GPT-3.5 and GPT-4).
The chart visually underscores the paper's key finding: providing the explicit character list is the dominant factor in achieving compliant output, with model capability (GPT-3.5 vs. GPT-4) being a secondary factor. A p-value annotation (e.g., p < 0.01) would be placed between the two groups, indicating statistical significance.
Scenario: Designing a prompt for an A1 learner to practice buying food at a market.
Weak Prompt (Low Compliance Expected): "Act as a Chinese tutor. Have a conversation with me about buying fruit at a market."
Strong, Research-Informed Prompt (High Compliance Expected):
Role: You are a patient Chinese language tutor for a complete beginner (CEFR Level A1).
Constraint: Use ONLY the following characters and words from the EBCL A1 list in your responses: [苹果, 香蕉, 水, 多少, 钱, 一, 二, 三, 四, 五, 元, 吗, 谢谢, 再见, ... (full A1 list)]. Do not use any characters outside this list.
Task: Simulate a simple dialogue at a fruit stall. You play the vendor. I will play the customer.
Structure:
1. Start by greeting me (你好).
2. Ask me what I would like (你要什么？).
3. Respond to my simple request (e.g., "我要苹果").
4. Tell me the price using only numbers 1-5 and the word 元.
5. After I say "谢谢", say "再见".
Keep each of your turns very short (max 5 characters). After the dialogue, ask me one simple yes/no question in Chinese about the dialogue using 吗.Analysis: The strong prompt defines the role, provides the critical constraint (the explicit list), specifies the task and context, and even outlines the structure of the interaction. This massively reduces the LLM's creative latitude, funneling it towards the desired pedagogical goal and ensuring output stays within the A1 "comprehensible input" zone.