LegalDrill: Diagnosis-Driven Synthesis for Legal Reasoning in Small Language Models

Diagnosis-Driven Synthesis for Legal Reasoning in Small LLMs: The LegalDrill Framework

Motivation and Problem Setting

Deploying LLMs for automated legal reasoning in practical, security- and privacy-conscious applications faces structural constraints: cloud-based APIs are generally impermissible for sensitive content, and large-scale models (e.g., Qwen3-32B or Llama3-70B) are often infeasible for on-premise use due to resource demands. SLMs (sub-3B parameters) are attractive candidates, but their inherent limitations in multi-step reasoning, especially under the demands of statute interpretation and logical deduction, inhibit deployment in high-stakes legal contexts.

A crucial bottleneck is the data regime: SLMs require high-quality, compact reasoning trajectories tailored to their architectural capacity. Standard approaches relying on LLM-generated reasoning traces—either via naive rejection sampling or by direct imitation—perform poorly, as SLMs cannot effectively learn from verbose and deliberative chains preferred by RL-aligned LLMs. This phenomenon, demonstrated in recent studies (Li et al., 17 Feb 2025, Yeo et al., 5 Feb 2025), speaks to a fundamental teacher-student mismatch in legal domain reasoning transfer.

LegalDrill: Diagnosis-Driven Synthesis Framework

LegalDrill proposes an iterative, diagnosis-driven framework that bridges the reasoning gap between capable (teacher) LLMs and SLMs through diagnostics, synthetic data generation, and self-reflective verification.

Figure 1: The LegalDrill framework overview, illustrating exploration, diagnosis, targeted data generation, and self-reflective filtering within the iterative training loop.

Core Pipeline

LegalDrill decomposes the distillation process into three stages:

1. Exploration (Student Response Generation): The student SLM (e.g., Qwen3-0.6B/1.7B) is prompted with a legal context and query to produce a stepwise Chain-of-Thought (CoT) response. This exposes model-specific reasoning errors.
2. Diagnosis and Preference Pair Synthesis: An Audit Agent, instantiated as a strong LLM, analyzes each student response, identifying concrete reasoning failures (e.g., logical leaps, statute misinterpretations). The agent formulates error simulation instructions, context-agnostic and reusable, which populate a bank of typical SLM misconceptions.

   For each instruction, the teacher LLM generates:

   • a rejected response that intentionally embodies the error, and
   • a chosen response correcting the reasoning.

   This constructs preference pairs precisely targeting SLM blind spots, rather than compiling unstructured or globally optimal explanations.

3. Self-Reflective Verification: Since many teacher-generated pairs may be trivial given the SLM’s current capabilities, LegalDrill introduces a Difficulty Score which quantifies, using forced-choice prompts, the student's confusion margin between the rejected and chosen responses. Only pairs where the SLM explicitly prefers the error (i.e., Difficulty Score above threshold) are retained for training, focusing optimization on genuine reasoning deficiencies.

Optimization

Initial model parameters are warmed up via SFT on teacher-preferred responses. Subsequent updates employ DPO using the verified, confusion-targeted pairs, with the reference policy set to the current model at each iteration. This iterative process rapidly channels teacher signal into model weaknesses and avoids data inefficiency common in rejection-based filtering.

Qualitative Dynamics of Legal Reasoning Refinement

LegalDrill's framework not only corrects surface errors but also enables SLMs to acquire complex logical structures, illustrated in qualitative examples.

Figure 2: Example showing initial SLM flawed reasoning, agent diagnosis of error type, automatic generation of tailored preference pairs, and the subsequent student model correction after LegalDrill optimization.

The case study highlights:

• Subtle errors such as interpretive logical leaps, where the SLM’s argument is linguistically plausible but substantively flawed,
• The Audit Agent extracting the generalization, formulating context-agnostic reproduction instructions,
• The teacher LLM's ability to produce controlled error/correction pairs,
• Subsequent iteration yielding SLMs whose responses match the teacher's logical structure, even when distinct from verbatim teacher outputs.

Empirical Results

LegalDrill achieves substantial accuracy, F1, and judged reasoning gains across diverse legal tasks and datasets.

• On LegalBench datasets (Consumer QA, Contracts QA, Sara Entailment, Privacy Policy Entailment), LegalDrill-1.7B attains up to 96% accuracy, closely matching teacher (Qwen3-30B) performance, with consistent gains of 10–27 absolute accuracy points over base SLMs.
• On proprietary financial compliance QA datasets, LegalDrill distilled from GPT-4o produces SLMs whose performance is on par with the LLM teacher, demonstrating practical utility for on-device legal review workflows.

The scaling effect is monotonic: larger SLM backbones see larger absolute gains, indicating that diagnosis-driven distillation is especially effective for small-to-midsize, resource-constrained models.

Ablation Studies

Ablations demonstrate the distinctive value proposition of DPO. Using only SFT on the chosen responses produces significant gains, but the introduction of DPO with targeted preference pairs consistently yields further accuracy improvements and more robust legal reasoning (as judged by LLM critics).

Figure 3: Impact of DPO vs. SFT-only training; DPO brings consistent improvements in accuracy and reasoning robustness.

The effectiveness of LegalDrill saturates within 1–2 iterations, indicating high sample efficiency and fast convergence.

Figure 4: Study of the number of LegalDrill iterations; the largest gains occur early, with diminishing returns beyond two iterations.

Practical and Theoretical Implications

LegalDrill establishes that the gap between teacher and student reasoning capacity, especially in the legal domain, is not merely a matter of more data or parameter count, but of appropriately diagnosing and directly targeting student model blind spots. The diagnosis-driven approach of LegalDrill contrasts with recent global chain-of-thought compression or naive trajectory pruning methods (Zhao et al., 20 May 2025, Yeo et al., 5 Feb 2025), by explicitly constructing data that attacks errors where small models are confused, thus maximizing empirical risk minimization under SLM capacity constraints.

Practically, this enables the use of SLMs for legal tasks with strong accuracy guarantees, privacy, and cost advantages, applicable in compliance-heavy and confidential environments. Theoretically, the approach demonstrates how targeted error simulation and difficulty calibration can close behavioral teacher-student mismatches in distillation—a principle of broader relevance for alignment under constrained deployment topologies.

Future Directions

Further investigations should consider:

• Extension of the diagnosis-driven framework to general complex reasoning domains (e.g., STEM QA, math proof),
• Application to other model families (Dense vs. Mixture-of-Experts, quantized LLMs, etc.),
• Integration with process-based reward or rubric modeling for finer supervision granularity (Zheng et al., 9 Oct 2025),
• Exploration of online, continual diagnosis and distillation protocols for adaptive compliance systems.

Conclusion

LegalDrill provides a targeted, diagnosis-driven pathway for transferring robust legal reasoning capacity from capable LLMs to SLMs. By systematically analyzing student mistakes, synthesizing corrective data, and focusing optimization on genuine weaknesses via self-reflective mechanisms, LegalDrill delivers SLMs whose reasoning is accurate, reliable, and deployable in privacy-sensitive environments. The methodology offers both a practical solution for legal AI deployment and a theoretical contribution to teacher-student reasoning alignment (2604.23809).