Skip to content
InfratoApp
Layers / Data / RAG / ETL and ingestion

ETL and ingestion

COMPARE TWO OF THESE →

Moving data from where it is produced to where a model can use it: connectors, change capture, batch and streaming loads. The transformation of documents into passages is document parsing; this is the transport.

44 in this category
SORTED BY STARS

NARROW BY WHAT THIS CATEGORY ASKS

LICENCE · 8 NOT READ

OPEN SOURCE · 25 NOT READ

SCHEDULING · 34 NOT READ

PRODUCTUSED BYSTARSDOWNLOADS 30D
Ccrawl4aiOpen-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key.84,408pathwayPython ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.62,295kestraEvent Driven Orchestration & Scheduling Platform for Mission Critical Applications28,095vectorA high-performance observability data pipeline.22,544airbyteOpen-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.22,039cocoindexIncremental engine for long horizon agents 🌟 Star if you like it!11,534Apache SeaTunnelSeaTunnel is a multimodal, high-performance, distributed, massive data integration tool.9,663mage-ai🧙 Build, run, and manage data pipelines for integrating and transforming data.8,822connectFancy stream processing made operationally mundane8,743steampipeZero-ETL, infinite possibilities. Live query APIs, code & more with SQL. No DB required.7,957data-juicerData processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷7,037cloudqueryData pipelines for cloud config and security data. Build cloud asset inventory, CSPM, FinOps, and vulnerability management solutions. Extract from AWS, Azure, GCP, and 70+ cloud and SaaS sources.6,517daftHigh-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale5,757jitsuJitsu is an open-source Segment alternative. Fully-scriptable data ingestion engine for modern data teams. Set-up a real-time data pipeline in minutes, not days5,069RudderStackPrivacy and Security focused Segment-alternative, in Golang and React 4,485crateCrateDB is a distributed and scalable SQL database for storing and analyzing massive amounts of data in near real-time, even with complex queries. It is PostgreSQL-compatible, and based on Lucene.4,435docetlA system for agentic LLM-powered data processing and ETL4,087ingestringestr is a CLI tool to copy data between any databases with a single command seamlessly.3,956PeerDBFast, Simple and a cost effective tool to replicate data from Postgres to Data Warehouses, Queues and Storage3,271lakesoulLakeSoul is an end-to-end, realtime cloud-native Lakehouse framework for fast data ingestion, concurrent updates, incremental analytics, multimodal data processing and vector search — powering next-ge3,250ethereum-etlPython scripts for ETL (extract, transform and load) jobs for Ethereum blocks, transactions, ERC20 / ERC721 tokens, transfers, receipts, logs, contracts, internal transactions. Data is available in Go3,130broadwayConcurrent and multi-stage data ingestion and data processing with Elixir2,683instill-core🔮 Instill Core is a full-stack AI infrastructure tool for data, model and pipeline orchestration, designed to streamline every aspect of building versatile AI-first applications2,319TimeplusThe Fastest Unified Streaming SQL Engine in a Single C++ Binary. ⚡ Millisecond latency. 100+ GB/s throughput. Continuously compute real-time context from streams, logs, metrics, events, and CDC2,258go-streamsA lightweight stream processing library for Go2,172KkibaData processing & ETL framework for Ruby1,775bruinBuild data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.1,702multiwoven🔥🔥🔥 Open source Reverse ETL - alternative to hightouch and census.1,673Gglobal-stock-dataUS stock market data for AI coding assistants — zero-auth, official sources. CBOE options with full Greeks + 0DTE flow, FINRA market-wide short volume, SEC EDGAR filing stream, and a free market-wide 1,593Ssmarter_csvFastest end-to-end CSV ingestion for Ruby (with C acceleration). SmarterCSV auto-detects formats, applies smart defaults, and returns Rails-ready hashes for seamless use with ActiveRecord, Sidekiq, pa1,550olakeOLake - Fastest Databases, Kafka & S3 Replication to Apache Iceberg with Table optimization (Called OLake Fusion). ⚡ Efficient, quick and scalable data ingestion for real-time analytics. Supported so1,445AaddaxActively maintained successor to Alibaba DataX — a fast, versatile, open-source ETL tool for 30+ RDBMS and NoSQL data sources.1,434pgsyncPostgres, MySQL, or MariaDB to Elasticsearch/OpenSearch sync1,420amphi-etlvisual data prep powered by python1,402TISSupport agile Ontology DataOps Based on Flink, DataX and Flink-CDC with Web-UI1,393squid-sdkTypeScript ETL toolkit for indexing Ethereum, Solana, and Substrate data, sourced from SQD Network.1,339duckleOpen-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 385 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents.1,295Ggo-stashgo-stash is a high performance, free and open source server-side data processing pipeline that ingests data from Kafka, processes it, and then sends it to ElasticSearch.1,225datawaveDataWave is an ingest/query framework that leverages Apache Accumulo to provide fast, secure data access.728Crunchy DataCopy to/from Parquet in S3, Azure Blob Storage, Google Cloud Storage, http(s) stores, local files or standard inout stream from within PostgreSQL687clickhouse-etlGlassFlow ETL: Open source data ingestion and transformations for ClickHouse pipelines502airy💬 Open Source App Framework to build streaming apps with real-time data - 💎 Build real-time data pipelines and make real-time data universally accessible - 🤖 Join historical and real-time data i402OrbitalOrbital automates integration between data sources (APIs, Databases, Queues and Functions). BFF's, API Composition and ETL pipelines that adapt as your specs change.360OSuite.aiOnd ESG Intelligence Platform is a cloud-native solution that ingests ESG data, processes it with Azure Data Factory & Databricks, and applies AI/ML models on Azure ML for predictive insights. Power B277

SORTED BY ONE STATED COLUMN · USED BY = LISTED PRODUCTS DEPENDING ON IT, IN THIS DIRECTORY — NOT A POPULARITY FIGURE · NO OVERALL SCORE AND NO RANK COLUMN, BECAUSE BLENDING ADOPTION, STARS AND DOWNLOADS NEEDS WEIGHTS, AND WEIGHTS ARE AN OPINION PRESENTED AS ARITHMETIC

Cite this page

The count and its date sit inside the citation, because “ETL and ingestion” without a date is a claim that quietly goes wrong as the category fills.

infratoapp, “ETL and ingestion — Data / RAG”, read 27 October 2026. 44 listed products. https://www.infratoapp.com/layers/data-rag/etl-ingestion

No overall score, and no rank column

Blending adoption, stars and downloads needs weights, and weights are an opinion presented as arithmetic — so this table sorts by one stated column at a time and shows which. This view holds no billing column and joins no billing table: a sponsor cannot appear higher, and the signals meter is a row’s own confidence, never a score you can sort on.

Frequently asked

What is in the ETL and ingestion category?
44 products in the Data / RAG layer of the AI stack. Moving data from where it is produced to where a model can use it: connectors, change capture, batch and streaming loads. The transformation of documents into passages is document parsing; this is the transport. Listed members include crawl4ai, pathway and kestra. Every fact on each listing carries the page it was read from and the date it was read.
How are ETL and ingestion compared here?
By the 6 attributes this category declares, asked of every member and answered only where a source states it. A product with no answer on record shows a gap rather than a guess, and no blended score is published — weights would be an opinion presented as arithmetic.
crawl4ai or pathway?
Both are listed under ETL and ingestion. We publish no verdict between them — we hold no benchmark that would support one. What we do hold is the same attribute sheet for each, side by side with the source behind every value, at /compare/crawl4ai-vs-pathway.
What does a blank mean on a etl and ingestion row?
That nobody has told us, not that the answer is no. A hollow ring is "no answer on record", which is a different state from a recorded no — and the two look different on purpose. A listing with a blank has not failed the question; it has not been read for it.
How often are etl and ingestion figures re-read?
Every value carries the date it was read and a link to the page it came from. Prices move fastest and are re-read most often; an attribute read weeks ago says so rather than presenting itself as current. Anything past 21 days is hidden rather than shown stale.
Is this every etl and ingestion there is?
No. This is 44 we have read and can cite. A directory that claimed completeness would be making a promise it cannot keep, so the page states its own count instead and says when a category is too thin to shortlist from.
Can a vendor pay to rank higher here?
No. Nothing on this page can be bought: a sponsor cannot enter a table, reorder a row or change a mark, and there is no ranked position for sale. How the site is paid for, and the line each source of money does not cross, is set out under money and independence at /our-data#sec5.