Enriquecimento de documentos jurídicos por meio de citações de usuários para recuperação de jurisprudência

Abstract

The effectiveness of search systems in the legal domain directly impacts procedural speed and the quality of judicial decisions, but is hindered by the vocabulary problem and by the long, multi-topic nature of legal documents. This work evaluates the use of user queries and citations obtained through implicit copy feedback — as a source of information to better describe the content of legal documents (precedents, legislation, and legal doctrine). In partnership with Jusbrasil, the BM25 algorithm was used to retrieve the user queries (copy queries) associated with each document, aggregating them into a new indexable field named Matches. Strategies for building this field were compared raw insertion, filtering by a supervised classifier (Random Forest), LLM validation, LLM summarization (Llama 3.1, 8B and 70B versions), and human validation under different index field configurations, measuring NDCG@1, @5, and @10 in two relevance judgment regimes: grades assigned by the Sabiá-4 model (LLM-as-a-Judge) and a mixed regime that prioritizes human grades when available. The evaluation collection, with 465 queries, was expanded from 8,186 to 24,777 judged pairs, with mod- erate human–LLM agreement (quadratic Cohen’s κ of 0.52). Comparative metrics are reported on the test set, comprising the 465 queries of the evaluation collection. Adding the Matches field to the document body improved NDCG@10 across all strategies, with gains of up to +0.058 (≈ +7%) in the LLM regime; the gain is statistically significant when tags are summarized or validated (paired t-test and Wilcoxon, p < 0.001 for 70B summarization), with summarization being the most robust strategy across evaluation regimes.