ELT is a process for copying data from a source system into a target system. It stands for “Extract, Load, Transform” and starts with extracting a copy of data from the source location. It’s loaded into the target system like a data warehouse, and then it’s ready to be transformed into a usable format for things like modern cloud applications.
The company Meltano provides code that manages ELT pipelines through an open-source, self-hosted, CLI-first, debuggable, and extensible process. Meltano projects manage your Singer tap and target configurations to easily select which entities and attributes to extract. These pipelines track their own incremental replication state so they can pick up where the previous run left off. Once your raw data is in its target source, Meltano helps you transform it into a usable format. These pipelines can run on a schedule and be fed to supported orchestrators like Apache Airflow.
In this episode we talk to Douwe Maan, a Co-Founder of Meltano, about their product-market fit and delivery plans.
Sponsorship inquiries: sponsor@softwareengineeringdaily.com
Transcript
Transcript provided by We Edit Podcasts. Software Engineering Daily listeners can go to weeditpodcasts.com to get 15% off the first three months of audio editing and transcription services with code: SED. Thanks to We Edit Podcasts for partnering with SE Daily. Please click here to view this show’s transcript.
Are you bored writing scripts to move data into SaaS tools like Salesforce? Hightouch is the easiest way to sync data into the tools that your business teams rely on. It’s simple — connect your data warehouse, paste a SQL query, and use our visual mapper to specify how data should appear in downstream tools. No scripts, just SQL. Get started for free at hightouch.io/sedaily.
This article is purposely trimmed, please visit the source to read the full article.
The post Meltano: ELT for DataOps with Douwe Maan appeared first on Software Engineering Daily.