 |
Apache Spark vs Presto
May 18, 2023 | Author: Michael Stromann
17
Apache Spark is a fast and general engine for large-scale data processing. Run programs up to 100x faster than Hadoop MapReduce in memory, or 10x faster on disk. Write applications quickly in Java, Scala or Python. Combine SQL, streaming, and complex analytics.
10
Presto is a highly parallel and distributed query engine for big data, that is built from the ground up for efficient, low latency analytics.
Apache Spark and Presto are both powerful distributed query engines used for processing and analyzing large datasets, but they have different design principles and use cases. Apache Spark is a general-purpose data processing framework that emphasizes speed and versatility. It provides in-memory processing and a wide range of libraries and APIs for tasks like batch processing, real-time streaming, machine learning, and graph processing. Spark's unified programming model allows developers to write complex data pipelines with ease. On the other hand, Presto is designed for fast, interactive SQL queries across various data sources. It offers a federated architecture that allows querying data from different systems with minimal data movement. Presto excels in ad-hoc analysis, exploratory queries, and interactive analytics. It provides excellent performance for interactive queries but may have limitations for complex batch processing compared to Spark.
See also: Top 10 Big Data platforms
See also: Top 10 Big Data platforms
Apache Spark vs Presto in our news:
2019. Starburst raises $22M to modernize data analytics with Presto
Starburst, the company seeking to commercialize the open-source Presto distributed query engine for big data (originally developed at Facebook), has announced a successful funding round, raising $22 million. The primary objective of Presto is to enable anyone to utilize the standard SQL query language for executing interactive queries on vast amounts of data stored across diverse sources. Starburst intends to monetize Presto by introducing several enterprise-oriented features. These additions will primarily focus on enhancing security, such as role-based access control, and integrating connectors to enterprise systems like Teradata, Snowflake, and DB2. Additionally, Starburst plans to provide a management console that empowers users to configure the cluster for automatic scaling, among other functionalities.
2015. IBM bets on big data Apache Spark project
IBM has made a significant announcement regarding its involvement in the open source big data project Apache Spark. The company plans to allocate a team of 3,500 researchers to this initiative. Additionally, IBM has unveiled its decision to open source its own IBM SystemML machine learning technology. These strategic moves are aimed at positioning IBM as a frontrunner in the domains of big data and machine learning. Cloud, big data, analytics, and security form the pillars of IBM's transformation strategy. In conjunction with this announcement, IBM has committed to integrating Spark into its core analytics products and partnering with Databricks, the commercial entity established to support the open source Spark project. IBM's participation in these endeavors goes beyond mere altruism. By actively engaging with the open source community, IBM aims to establish itself as a trusted contributor in the realm of big data. This, in turn, enhances its credibility among companies working on big data and machine learning projects using open source tools. The collaborative involvement with the community opens doors for IBM to offer consulting services and seize other business opportunities in this space.
2015. Google partners with Cloudera to bring Cloud Dataflow to Apache Spark
Google has announced a collaboration with Cloudera, the Hadoop specialists, to integrate its Cloud Dataflow programming model into Apache's Spark data processing engine. By bringing Cloud Dataflow to Spark, developers gain the ability to create and monitor data processing pipelines without the need to manage the underlying data processing cluster. This service originated from Google's internal tools for processing large datasets at a massive scale on the internet. However, not all data processing tasks are identical, and sometimes it becomes necessary to run tasks in different environments such as the cloud, on-premises, or on various processing engines. With Cloud Dataflow, data analysts can utilize the same system to create pipelines, regardless of the underlying architecture they choose to deploy them on.
