Data / RAG

Moving data from where it is produced to where a model can use it: connectors, change capture, batch and streaming loads. The transformation of documents into passages is document parsing; this is the transport.

Ordered by attributes answered — the column we hold for most of them. Blank where we hold none.

olake6 ATTRIBUTES ANSWEREDduckle5 ATTRIBUTES ANSWEREDAaddax4 ATTRIBUTES ANSWEREDjitsu4 ATTRIBUTES ANSWEREDsteampipe4 ATTRIBUTES ANSWEREDairbyte3 ATTRIBUTES ANSWEREDKkiba3 ATTRIBUTES ANSWEREDpgsync3 ATTRIBUTES ANSWEREDairy2 ATTRIBUTES ANSWEREDamphi-etl2 ATTRIBUTES ANSWEREDbroadway2 ATTRIBUTES ANSWEREDCcrawl4ai2 ATTRIBUTES ANSWERED

Human annotation: bounding boxes, spans, classifications and the review workflow around them. Generating examples rather than marking them up is synthetic data.

Ordered by attributes answered — the column we hold for most of them. Blank where we hold none.

kubric6 ATTRIBUTES ANSWEREDanylabeling4 ATTRIBUTES ANSWEREDCclapshot4 ATTRIBUTES ANSWEREDcvat4 ATTRIBUTES ANSWEREDDdrawpen4 ATTRIBUTES ANSWEREDannotorious3 ATTRIBUTES ANSWEREDdokieli3 ATTRIBUTES ANSWEREDGgromit-mpx3 ATTRIBUTES ANSWEREDinception3 ATTRIBUTES ANSWEREDksnip3 ATTRIBUTES ANSWEREDlabel-studio3 ATTRIBUTES ANSWEREDannif2 ATTRIBUTES ANSWERED

The store itself — where embeddings live, how they are indexed and what a similarity query costs. The search behaviour built over one belongs in retrieval and search; LanceDB, Qdrant and Redis sit in both because they genuinely are both.

Ordered by attributes answered — the column we hold for most of them. Blank where we hold none.

LanceDB8 ATTRIBUTES ANSWEREDQdrant7 ATTRIBUTES ANSWEREDparadedb6 ATTRIBUTES ANSWEREDPinecone5 ATTRIBUTES ANSWEREDRedis5 ATTRIBUTES ANSWEREDWeaviate5 ATTRIBUTES ANSWEREDlance4 ATTRIBUTES ANSWEREDredisearch4 ATTRIBUTES ANSWEREDTturbovec4 ATTRIBUTES ANSWEREDZilliz4 ATTRIBUTES ANSWEREDairbyte3 ATTRIBUTES ANSWEREDarcadedb3 ATTRIBUTES ANSWERED

Generating training or test data rather than collecting it, including privacy-preserving substitutes for real records. Marking up data that already exists is labelling and annotation.

Ordered by attributes answered — the column we hold for most of them. Blank where we hold none.

kubric6 ATTRIBUTES ANSWEREDedsl4 ATTRIBUTES ANSWEREDsdv4 ATTRIBUTES ANSWEREDdataarc-syndata-toolkit3 ATTRIBUTES ANSWEREDgen-fraud-graph3 ATTRIBUTES ANSWEREDgraphgen3 ATTRIBUTES ANSWEREDcosmos-drive-dreams2 ATTRIBUTES ANSWEREDgreenmask2 ATTRIBUTES ANSWEREDcopulas1 ATTRIBUTE ANSWEREDEeasy-dataset1 ATTRIBUTE ANSWEREDfg-data-synthetic1 ATTRIBUTE ANSWEREDMMimesis1 ATTRIBUTE ANSWERED

Turning documents into text a model can read — PDFs, scans, tables, HTML — including OCR and layout recovery. What happens to the passages afterwards is retrieval and search.

Ordered by attributes answered — the column we hold for most of them. Blank where we hold none.

Unstructured.io6 ATTRIBUTES ANSWEREDdocling5 ATTRIBUTES ANSWEREDopendataloader-pdf5 ATTRIBUTES ANSWEREDdocling-studio4 ATTRIBUTES ANSWEREDdocstrange4 ATTRIBUTES ANSWEREDfirecrawl4 ATTRIBUTES ANSWEREDgraphify4 ATTRIBUTES ANSWEREDLlamaIndex4 ATTRIBUTES ANSWEREDHhaiku.rag3 ATTRIBUTES ANSWEREDade-cli2 ATTRIBUTES ANSWEREDBbfscan2 ATTRIBUTES ANSWEREDNomic2 ATTRIBUTES ANSWERED

Versioning datasets and the artefacts derived from them, so a training run can be reproduced. Versioning the trained model itself is a model registry.

Ordered by attributes answered — the column we hold for most of them. Blank where we hold none.

DVC3 ATTRIBUTES ANSWEREDLakeFS3 ATTRIBUTES ANSWEREDdoltNO ANSWER ON RECORDlamindbNO ANSWER ON RECORD