What are your preferred methods for integrating data from different sources?

Question

# Direct data transfers: Do you prefer tools that move data directly from source to destination (like ETL pipelines), or do you rely more on virtualizing access to data without moving it?

# Real-time vs. batch processing: Is real-time data integration critical for your applications, or are batch processes sufficient? What tools do you use to achieve this?

Simon🍋 · Accepted Answer

ETL tools like Talend or custom Python scripts, depending on data complexity. SQL for joining disparate sources once loaded.

Tatiana Vdovychenko · Answer

At JetSoftPro, our teams pick methods based on the project. One of our engineers said, "ETL pipelines are perfect for big data jobs – like building a custom track for data." But when clients need live access without copying data, we go with data virtualization. Real-time? Great when speed matters, but risky if something breaks. Batch processing is simpler and works for non-urgent tasks. I highly recommend JetSoftPro's article about data analyst and their role in the startups: https://jetsoftpro.com/blog/the-...

Tatiana Vdovychenko · Answer

For integrating data, I usually prefer a mix depending on the project:
- ETL pipelines for moving data directly from source to destination, especially when dealing with large datasets that need transformation.
- Data virtualization when I need quick access to data without actually moving it, like for analytics or reporting.

As for real-time vs. batch:
- Real-time is crucial for applications that require up-to-date info (like monitoring systems), and I use tools like Apache Kafka or Fivetran.
- For less urgent data, batch processing works fine, and I use tools like AWS Glue or Airflow for that.

Aria Madeline Sinclair · Answer

Fivetran for pulling data from various SaaS tools into a data warehouse. dbt for transformation and modeling once the raw data is loaded. Then good old SQL for analysis across tables.

Benjamin Thomas Clark · Answer

Depends on data volume and refresh needs. Cloud ETL like Fivetran or Stitch for scheduled extracts, transformations in the warehouse. Python for advanced pipelines or real-time streaming. Looker or Tableau for visual joining once loaded.

Answer

Combination of tools for me - Fivetran for extracting data from SaaS apps, dbt for transforming & modeling the data, and Looker to join sources for analysis. Python when I need more customization. Curious what others are using?

Answer

Using APIs allows for real-time data integration by connecting applications and enabling data exchange between them.

What are your preferred methods for integrating data from different sources?

Replies