Skip to content

A kernel that enables applications to interact with Apache Spark.

License

Notifications You must be signed in to change notification settings

virtualirfan/spark-kernel

 
 

Repository files navigation

Spark Kernel

Build Status Coverage Status Scaladoc License

The Spark Kernel has one main goal: provide the foundation for interactive applications to connect and use Apache Spark.

Overview

Spark Kernel Overview

The kernel provides several key features for applications:

  1. Define and run Spark Tasks

    • Executing Scala code dynamically in a similar fashion to the Scala REPL and Spark Shell
  2. Collect Results without a Datastore

    • Send execution results and streaming data back via the Spark Kernel to your applications

    • Use the Comm API - an abstraction of the IPython protocol - for more detailed data communication and synchronization between your applications and the Spark Kernel

  3. Host and Manage Applications Separately from Apache Spark

    • The Spark Kernel serves as a proxy for requests to the Apache Spark cluster

The project intends to provide applications with the ability to send both packaged jars and code snippets. As it implements the latest IPython message protocol (5.0), the Spark Kernel can easily plug into the 3.x branch of IPython for quick, interactive data exploration. The Spark Kernel strives to be extensible, providing a pluggable interface for developers to add their own functionality.

If you are new to the Spark Kernel, please see the Getting Started section.

For more information, please visit the Spark Kernel wiki.

For bug reporting and feature requests, please visit the Spark Kernel issue list.

About

A kernel that enables applications to interact with Apache Spark.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Scala 99.4%
  • Other 0.6%