Skip to content

3.0.0rc4

This is the fourth release candidate for version 3.0.0. It adds evaluation utilities and fixed-pair sampling. The 3.0.0rc1 notes describe version 3 as a whole, and the migration guide covers version 2 names.

New features

  • evaluation.select_threshold chooses a match/no-match threshold from validation scores, boolean labels, and an explicit metric, and the result works directly with MatchFinder. Built-in metrics are match_accuracy, match_balanced_accuracy, and match_f1. See threshold selection (#733).
  • samplers.FixedSetOfPairs draws a fixed, balanced set of positive and negative pairs and yields whole batches, so a pair is never split. The sampler documentation includes a complete DataLoader-to-loss loop (#259).
  • evaluation.ndcg_metric registers NDCG from a graded relevance function, normalized by the ideal ranking over the whole reference set. Multi-column labels can drive NDCG and binary metrics in one calculator, and each keeps its own rule for which queries count. See graded relevance and NDCG (#339).

Evaluation changes

  • RetrievalInputs gains the required fields reference_labels and ref_includes_query. Only code that constructs RetrievalInputs directly needs to change.
  • RetrievalMetric gains exclude_lone_queries, default True. A metric that sets it to False decides through NaN which queries count.

Fixes

  • AccuracyCalculator no longer raises TypeError with MPS inputs. The search stays on MPS and metric arithmetic runs on CPU, because MPS has no float64.

Documentation