Batches and reranking
Run asynchronous provider batches and rank documents by relevance.
Batch requests
Batch operations normalize the common lifecycle while preserving the provider response in raw.
The input file format and accepted endpoint depend on the selected provider.
import { createBatch, retrieveBatch, retrieveBatchResults } from "any-llm-ts";
const batch = await createBatch({
provider: "openai",
endpoint: "/v1/chat/completions",
inputFilePath: "./requests.jsonl",
completionWindow: "24h",
});
const current = await retrieveBatch({
provider: "openai",
batchId: batch.id,
});
if (current.status === "completed") {
const result = await retrieveBatchResults({
provider: "openai",
batchId: current.id,
});
console.log(result.output, result.errors);
}Reusable clients expose the same lifecycle:
const batch = await llm.createBatch(params);
await llm.retrieveBatch(batch.id);
await llm.cancelBatch(batch.id);
await llm.listBatches();
await llm.retrieveBatchResults(batch.id);retrieveBatchResults() throws BatchNotCompleteError until output is available. Check
metadata.capabilities.batch before using the operation. Together uses its native batch upload and
job lifecycle while returning the same normalized batch types.
Reranking
Reranking scores candidate documents against a query. Cohere, Otari, Together, and Voyage
currently advertise the operation. Together serves no rerank model on its serverless tier, so a
dedicated endpoint is required. Voyage maps topN to topK and does not return a response id.
import { rerank } from "any-llm-ts";
const result = await rerank({
provider: "cohere",
model: "rerank-v3.5",
query: "How do I narrow a union in TypeScript?",
documents: [
"Use a discriminant property and a switch statement.",
"Use npm install to add a package.",
],
topN: 1,
});
console.log(result.results[0]?.index, result.results[0]?.relevanceScore);Documents may be strings or provider-compatible records. Use providerOptions for reranker fields
outside the common contract.