Deep Factorization Machines for Knowledge Tracing: Analysis of the 2018 Duolingo SLAM Solution

Table of Contents

---

1. Introduction & Overview

This paper presents the author's solution to the 2018 Duolingo Shared Task on Second Language Acquisition Modeling (SLAM). The core challenge was knowledge tracing at the word level: predicting whether a student would correctly write the words of a new sentence, given their historical attempt data on thousands of sentences annotated with lexical, morphological, and syntactic features.

The proposed solution utilizes Deep Factorization Machines (DeepFM), a hybrid model combining a wide component (a Factorization Machine) for learning pairwise feature interactions and a deep component (a Deep Neural Network) for learning higher-order feature interactions. The model achieved an AUC of 0.815, outperforming a logistic regression baseline (AUC 0.774) but falling short of the top-performing model (AUC 0.861). The work positions DeepFM as a flexible framework that can subsume traditional educational models like Item Response Theory (IRT).

2. Related Work & Theoretical Background

The paper situates its contribution within the broader landscape of student modeling and knowledge tracing.

2.1. Item Response Theory (IRT)

IRT is a classic psychometric framework that models the probability of a correct response as a function of a student's latent ability ($\theta$) and an item's parameters (e.g., difficulty $b$). A common model is the 2-parameter logistic (2PL) model: $P(\text{correct} | \theta) = \sigma(a(\theta - b))$, where $a$ is discrimination and $\sigma$ is the logistic function. The paper notes that IRT forms a strong, interpretable baseline but typically does not incorporate rich side information.

2.2. Knowledge Tracing Evolution

Knowledge tracing focuses on modeling the evolution of a student's knowledge over time.

• Bayesian Knowledge Tracing (BKT): Models the learner as a Hidden Markov Model with latent knowledge states.
• Deep Knowledge Tracing (DKT): Uses Recurrent Neural Networks (RNNs), like LSTMs, to model temporal sequences of student interactions. The paper cites work by Wilson et al. (2016) showing that IRT variants can outperform early DKT models, highlighting the need for robust, feature-aware architectures.

2.3. Wide & Deep Learning

The paper builds upon the Wide & Deep Learning paradigm introduced by Cheng et al. (2016) at Google. The "wide" linear model memorizes frequent feature co-occurrences, while the "deep" neural network generalizes to unseen feature combinations. Guo et al. (2017) proposed replacing the wide linear model with a Factorization Machine (FM), which efficiently models all pairwise interactions between features via factorized parameters, leading to the DeepFM architecture.

3. DeepFM for Knowledge Tracing

The paper adapts the DeepFM model for the knowledge tracing domain.

3.1. Model Architecture & Formulation

DeepFM consists of two parallel components whose outputs are combined:

1. FM Component: Models linear and pairwise feature interactions. For an input feature vector $\mathbf{x}$, the FM output is: $y_{FM} = w_0 + \sum_{i=1}^n w_i x_i + \sum_{i=1}^n \sum_{j=i+1}^n \langle \mathbf{v}_i, \mathbf{v}_j \rangle x_i x_j$, where $\mathbf{v}_i$ are latent factor vectors.
2. Deep Component: A standard feed-forward neural network that takes the dense feature embeddings as input and learns complex, high-order patterns.

The final prediction is: $p(\mathbf{x}) = \psi(y_{FM} + y_{DNN})$, where $\psi$ is a link function (e.g., sigmoid $\sigma$ or the normal CDF $\Phi$).

3.2. Feature Encoding & Embeddings

A key contribution is the treatment of features. The model considers C categories of features (e.g., user_id, item_id, skill, country, time). Each discrete value within a category (e.g., user=123, country='FR') or a continuous value itself is termed an entity. Each of the N possible entities is assigned a learnable embedding vector. An instance (e.g., a student answering a word) is encoded as a sparse vector $\mathbf{x}$ of size N, where components are set to 1 (for present discrete entities), the actual value (for continuous features), or 0.

4. Application to the SLAM Task

4.1. Data Preparation

For the Duolingo SLAM task, features included user ID, lexical item (word), its associated linguistic features (part-of-speech, morphology), sentence context, and temporal information. These were transformed into the entity-based sparse format required by DeepFM. This encoding allows the model to learn interactions between any pair of entities, such as (user=Alice, word="ser") and (word="ser", tense=past).

4.2. Experimental Setup

The model was trained to predict the binary outcome (correct/incorrect) for a student writing a specific word. The AUC (Area Under the ROC Curve) was used as the primary evaluation metric, standard for binary classification tasks with imbalanced data common in educational settings.

5. Results & Performance Analysis

The DeepFM model achieved a test AUC of 0.815. This represents a significant improvement over the logistic regression baseline (AUC 0.774), demonstrating the value of modeling feature interactions. However, it did not reach the top score of 0.861. The paper suggests this reveals "interesting strategies to build upon item response theory models," implying that while DeepFM provides a powerful, feature-rich framework, there is room for incorporating more nuanced educational theory or sequential modeling aspects that the top model might have captured.

6. Critical Analysis & Expert Insights

Core Insight: This paper isn't about a groundbreaking new algorithm, but a shrewd, pragmatic application of an existing industrial-strength recommendation system model (DeepFM) to a nascent problem space: granular, feature-rich knowledge tracing. The author's move is telling—they bypass the academic hype cycle around pure deep learning for education (like early DKT) and instead repurpose a model proven in e-commerce for capturing complex user-item-feature interactions. The real insight is framing knowledge tracing not just as a sequence prediction problem, but as a high-dimensional, sparse feature interaction problem, much like predicting a click in ads.

Logical Flow & Strategic Positioning: The logic is compelling. 1) Traditional models (IRT, BKT) are interpretable but limited to pre-defined, low-dimensional interactions. 2) Early deep learning models (DKT) capture sequences but can be data-hungry and opaque, sometimes underperforming simpler models as noted by Wilson et al. 3) The SLAM task provides a treasure trove of side information (linguistic features). 4) Therefore, use a model designed explicitly for this: DeepFM, which hybridizes the memorization of factorized pairwise interactions (the FM part, akin to IRT's student-item interaction) with the generalization power of a DNN. The paper cleverly shows how IRT can be seen as a special, simplistic case of this framework, thereby claiming the high ground of generality.

Strengths & Flaws: The primary strength is practicality and feature exploitation. DeepFM is a robust, off-the-shelf architecture for leveraging the rich feature set of the SLAM task. Its flaw, as revealed by the results, is that it was likely outperformed by models that better captured the temporal dynamics inherent in learning. An LSTM-based model or a transformer architecture (like those later used in KT, e.g., SAKT or AKT) might have integrated the sequential history more effectively. The paper's AUC of 0.815, while a solid improvement over the baseline, leaves a 0.046 gap to the winner—a gap that likely represents the price paid for not specializing in the temporal dimension. As research from the Riiid! AI Challenge and later works show, combining feature-aware architectures like DeepFM with sophisticated sequential models is the winning path.

Actionable Insights: For practitioners and researchers: 1) Don't overlook feature engineering. The success of applying DeepFM underscores that in educational data, the "side information" (skill tags, difficulty, response time, linguistic features) is often the main information. 2) Look to adjacent fields. Recommendation systems have spent a decade solving analogous problems of cold start, sparsity, and feature interaction; their toolkit (FM, DeepFM, DCN) is directly transferable. 3) The future is hybrid. The next step is clear: integrate the feature-interaction power of DeepFM with a state-of-the-art sequential module. Imagine a "Temporal DeepFM" where the deep component is an LSTM or Transformer that processes a sequence of these factorized interaction representations. This aligns with the trajectory seen in works like "Deep Interest Evolution Network" (DIEN) in ads, which combines feature interaction with sequential modeling of user interest evolution—a perfect analogue for knowledge evolution.

7. Technical Details & Mathematical Formulation

The core of DeepFM lies in its dual-component architecture. Let the input be a sparse feature vector $\mathbf{x} \in \mathbb{R}^n$.

Factorization Machine (FM) Component:
$y_{FM} = w_0 + \sum_{i=1}^{n} w_i x_i + \sum_{i=1}^{n} \sum_{j=i+1}^{n} \langle \mathbf{v}_i, \mathbf{v}_j \rangle x_i x_j$
Here, $w_0$ is the global bias, $w_i$ are weights for linear terms, and $\mathbf{v}_i \in \mathbb{R}^k$ is the latent factor vector for the i-th feature. The inner product $\langle \mathbf{v}_i, \mathbf{v}_j \rangle$ models the interaction between feature $i$ and $j$. This is computed efficiently in $O(kn)$ time.

Deep Component:
Let $\mathbf{a}^{(0)} = [\mathbf{e}_1, \mathbf{e}_2, ..., \mathbf{e}_m]$ be the concatenation of embedding vectors for the features present in $\mathbf{x}$, where $\mathbf{e}_i$ is looked up from an embedding matrix. This is fed through a series of fully connected layers:
$\mathbf{a}^{(l+1)} = \sigma(\mathbf{W}^{(l)} \mathbf{a}^{(l)} + \mathbf{b}^{(l)})$
The final layer's output is $y_{DNN}$.

Final Prediction:
$\hat{y} = \sigma(y_{FM} + y_{DNN})$
The model is trained end-to-end by minimizing the binary cross-entropy loss.

8. Analysis Framework & Conceptual Example

Scenario: Predicting if Student_42 will correctly translate the word "was" (lemma: "be", tense: past) in a Spanish exercise.

Feature Entities & Encoding:

• user_id=42 (Discrete)
• word_lemma="be" (Discrete)
• grammar_tense="past" (Discrete)
previous_accuracy=0.85 (Continuous, normalized)

The sparse input vector $\mathbf{x}$ would have 1s in the positions corresponding to the discrete entities, the value 0.85 for the continuous feature, and 0s elsewhere.

Model Interpretation:

• The FM part might learn that the interaction weight $\langle \mathbf{v}_{user42}, \mathbf{v}_{tense:past} \rangle$ is negative, suggesting Student_42 struggles with past tense generally.
• Simultaneously, it might learn $\langle \mathbf{v}_{lemma:be}, \mathbf{v}_{tense:past} \rangle$ is highly negative, indicating "be" in past tense is particularly difficult for all students.
• The Deep part might learn a more complex, non-linear pattern: e.g., a high previous_accuracy combined with a specific pattern of past errors on irregular verbs modulates the final prediction, capturing a higher-order interaction beyond pairwise.

This demonstrates how DeepFM can simultaneously capture simple, interpretable relationships (like IRT) and complex, non-linear patterns.

9. Future Applications & Research Directions

The application of DeepFM to knowledge tracing opens several promising avenues:

1. Integration with Sequential Models: The most direct extension is incorporating temporal dynamics. A DeepFM could serve as the feature interaction engine at each timestep, with its output fed into an RNN or Transformer to model knowledge state evolution over time, blending the strengths of feature-aware and sequence-aware models.
2. Personalized Content Recommendation: Beyond prediction, the learned embeddings for users, skills, and content items can power sophisticated recommendation systems within adaptive learning platforms, suggesting the next best exercise or learning resource.
3. Cross-Domain Transfer Learning: The entity embeddings learned from language learning data (e.g., embeddings for grammatical concepts) could potentially be transferred or fine-tuned for other domains like math or science tutoring, accelerating model development where data is scarcer.
4. Explainability & Intervention: While more interpretable than a pure DNN, DeepFM's explanations are still based on latent factors. Future work could focus on developing post-hoc explanation methods to translate factor interactions into actionable insights for teachers (e.g., "Student struggles specifically with the interaction between passive voice and past perfect tense").
5. Real-Time Adaptive Testing: The efficiency of the FM component makes it suitable for real-time systems. It could be deployed in computerized adaptive testing (CAT) environments to dynamically select the next question based on a continuously updated estimate of student ability and item-feature interactions.

10. References

1. Corbett, A. T., & Anderson, J. R. (1994). Knowledge tracing: Modeling the acquisition of procedural knowledge. User modeling and user-adapted interaction, 4(4), 253-278.
2. Piech, C., Bassen, J., Huang, J., Ganguli, S., Sahami, M., Guibas, L. J., & Sohl-Dickstein, J. (2015). Deep knowledge tracing. Advances in neural information processing systems, 28.
3. Wilson, K. H., Karklin, Y., Han, B., & Ekanadham, C. (2016). Back to the basics: Bayesian extensions of IRT outperform neural networks for proficiency estimation. In Educational Data Mining.
4. Cheng, H. T., Koc, L., Harmsen, J., Shaked, T., Chandra, T., Aradhye, H., ... & Shah, H. (2016, September). Wide & deep learning for recommender systems. In Proceedings of the 1st workshop on deep learning for recommender systems (pp. 7-10).
5. Guo, H., Tang, R., Ye, Y., Li, Z., & He, X. (2017). DeepFM: a factorization-machine based neural network for CTR prediction. arXiv preprint arXiv:1703.04247.
6. Vie, J. J., & Kashima, H. (2018). Knowledge tracing machines: Factorization machines for knowledge tracing. arXiv preprint arXiv:1811.03388.
7. Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory. Sage.
8. Settles, B., Brust, C., Gustafson, E., Hagiwara, M., & Madnani, N. (2018). Second language acquisition modeling. In Proceedings of the NAACL-HLT Workshop on Innovative Use of NLP for Building Educational Applications.