Indexer

class lightning_ir.retrieve.base.indexer.Indexer(index_dir: Path, index_config: IndexConfig, module: BiEncoderModule, verbose: bool = False)[source]

Bases: ABC

Base class for indexers that create and manage indices for retrieval tasks.

__init__(index_dir: Path, index_config: IndexConfig, module: BiEncoderModule, verbose: bool = False) → None[source]

Initialize the Indexer.

Parameters:

  • index_dir (Path) – Directory where the index will be stored.

  • index_config (IndexConfig) – Configuration for the index.

  • module (BiEncoderModule) – The bi-encoder module used for encoding documents.

  • verbose (bool) – Whether to print verbose output. Defaults to False.

Methods

__init__(index_dir, index_config, module[, ...])

Initialize the Indexer.

add(index_batch, output)

Add a batch of documents to the index.

save()

Save the index configuration and document IDs to the index directory.
abstractmethod add(index_batch: IndexBatch, output: BiEncoderOutput) → None[source]

Add a batch of documents to the index.

Parameters:

  • index_batch (IndexBatch) – The batch of documents to add.

  • output (BiEncoderOutput) – The output from the bi-encoder module containing document embeddings.

save() → None[source]

Save the index configuration and document IDs to the index directory.