Very cool - we could potentially use this in my product for several data sources. a couple questions - 1. does this work with private s3 buckets? 2. how often can i update the data? some of our files come in hourly, some daily, and some on demand (we don't know beforehand)? 3. does the data append to the destination or does it replace all the data at the destination table? we have situations for both. Thanks for building this!

Report

4yr ago

Kirat Pandya

Osmos Pipelines

Maker

@ashish_singal1 1) Yes, we support S3, GCS, Google Drive etc. and can read CSVs, Excel Files and Zips (containing CSVs and Excel files). More file formats being added every week. 2) You can set up the pipelines to look for new source data on an frequence from every minute to yearly, and anything in between. We have a fine grained scheduler that lets you do things like “Check every Tuesday at 2 PM”. The pipeline will automatically deduplicate source data and only pick up data that has changed since the last time it ran. The type of deduplication can also be configured to be “across all historical rows”, “Across this run”, “Across each file” etc. 3) The default mode is to append data, but we are able to overwrite or upsert (update or insert) based on the destination system’s abilities. Thank you for giving Osmos Pipelines a shot! Please don’t hesitate to reach out - we would love to hear more about your usecase and help make it successful!

Report

4yr ago

Kirat Pandya

Osmos Pipelines

Maker

Hello Product Hunt, and thanks @jdprater for the hunt. 👋 Kirat here, I’m the CEO and co-founder of Osmos.io. We are thrilled to launch Osmos Pipelines to provide your onboarding teams with a no-code ETL to bring in clean customer data into your product in a few clicks. Osmos Pipelines allows you to clean, map, and send data into your operation systems in minutes. 📍TL;DR 1. Accelerate your customer data onboarding from weeks to minutes with pre-built connectors and AI-powered data transformations. 2. Eliminate manual entry and custom scripts. Validate and clean up messy data with a few examples and simple formulas. No coding required. 3. Automate recurring data onboarding by turning them into scheduled data pipelines. ⌛ How it started My co-founder @naresh_venkat and I were at Google, leading AI/ML solutions and partnerships for Google Cloud prior to starting Osmos. While working with companies of all sizes (startups to enterprises) at Google, we realized that every single one of them was struggling with the challenges of dealing with external data - data from customers, partners, data providers etc. When it came to working with external data/systems, the solution was almost always duct-taped using ETL, RPA + OCR, some API and a LOT of human time. >60% of the companies rely on CSVs to share data, which creates a big “data onboarding” pain for the receiving party. There was no end-to-end solution to simplify data collaboration for the modern era. We started Osmos in late 2019 to solve the pain of cross-company data collaboration. Data sharing across companies is on the rise—but still takes way too long. ‍Companies have increasing numbers of “data relationships” where they need to share data with customers, suppliers, and partners. Nowhere is this data silo problem more painful than when onboarding customers. One of the biggest challenges is populating your product with your customer's live data as quickly as possible, so that they can start using it, have that “aha” moment and see the value. We’ve solved this. We built Osmos Pipelines to eliminate these data silos so applications talk directly to each other across company boundaries 👤Who is using it Here are a few examples of how customers use Osmos: 1. An eCommerce company automates ingestion of product catalog data from multiple distributors and vendors. 2. A martech company automates ingestion of campaign, online and offline transactions data, from their retail customers’ systems, into their data warehouse. 3. A manufacturing customer ingests email attachments into their EDI system. Let us know what you think! Thanks for checking us out, ask us all the questions you have, @naresh_venkat and I love to chat about all things data. We can’t wait to see what you will do with Osmos!

Report

4yr ago

Naresh Venkat

Osmos Pipelines

Maker

Hello Product Hunt, and thanks @jdprater for the hunt. 👋 Naresh here, I’m the COO and co-founder of Osmos.io. We have been very hard at work for almost 2 years, building the product with our early customers. Just came out of stealth and are super excited to get your feedback. I’d like to add a couple of points to @kirat 's post: 1. We have an ALWAYS FREE tier - You can process upto 1 million records a month for free. Please find more details at osmos.io/pricing. 2. The transformation engine is truly magical - no coding required; just a few examples and our engine will figure out the program needed to perform the transformations. Even someone like me, who hasn’t written code in years can clean up and move data in minutes. Give it a try! 3. We handle any data errors in subsequent runs through an active feedback loop to fix errors and retrain the transforms with ease. Check out the product and give us your candid feedback! - osmos.io

Report

4yr ago

Osmos Pipelines

No-code, ETL data pipelines for external data onboarding ⚡️

Do you use Osmos Pipelines?

Review Osmos Pipelines?

Reviews of Osmos Pipelines