Apache Zeppelin, the missing component for your BigData ecosystem

Transcription

1 Apache Zeppelin, the missing component for your BigData ecosystem DuyHai DOAN, Cassandra Technical Advocate

2 Who Am I?! Duy Hai DOAN Cassandra technical advocate talks, meetups, confs open-source devs (Achilles, ) OSS Cassandra point of contact duy_hai.doan@datastax.com 2

3 Datastax! Founded in April 2010 We contribute a lot to Apache Cassandra 400+ customers (25 of the Fortune 100), 400+ employees Headquarter in San Francisco Bay area EU headquarter in London, offices in France and Germany Datastax Enterprise = OSS Cassandra + extra features 3

4 What is Apache Zeppelin?! Presentation! Architecture!!

5 Zeppelin Presentation! 5

6 Zeppelin Architecture! JVM Spark Interpreter Group JVM Spark SparkSQL R E S T WebSocket Zeppelin Server Zeppelin Engine Zeppelin Interpreter Factory JVM Tajo Interpreter JVM Flink Interpreter JVM Cassandra Interpreter 6

7 What does Zeppelin provide?! Front-end & display system for free Generic back-end with REST APIs & WebSocket Pluggable interpreters system Task scheduler (à la CRON) 7

8 Zeppelin UI Layout! Notebook! Paragraph! UI elements!

9 Demo

10 Zeppelin Display System!! Raw, Table, HTML! Available graphs! View modes! Dynamic form! Iframe export!

11 Demo

12 Interpreter system!! Core interpreters! Third-parties interpreters! Interpreters conf & usage!

13 Interpreter processing lifecycle! 1 Receive input commands/data as raw text from form data Process the input commands/data by the external back-end Format the response using Zeppelin display system Send response back to the Zeppelin engine 13

14 Core interpreters!! Spark (Spark core, SparkSQL/DataFrame, PySpark) Spark core = default (or %spark) SparkSQL = %sql Shell (%sh) Markdown (%md)! AngularJS (%angular) 14

15 Third-parties interpreters! Hive Phoenix Tajo Flink Ignite Lens Cassandra Geode PostgreSQL Kylin 15

16 Interpreter conf & usage

17 Writing An Interpreter!! How To! Simple interpreter example (AsciiDoc)! Complex interpreter example (Cassandra)!

18 Steps to write your own interpreter! Create a class that extends Interpreter base class Register it in a static block static { Interpreter.register("MyInterpreterName", MyClassName.class.getName()); } Optionnally define default config params static { Interpreter.register("MyInterpreterName", MyClassName.class.getName(), new InterpreterPropertyBuilder().add("property1", "default value", "Description of property1").build()); } 18

19 To register your interpreter as default! Edit the enum ZeppelinConfiguration.ConfVars Add your interpreter FQCN in the property ZEPPELIN_INTERPRETERS 19

20 To register your interpreter in config files! Create conf/zeppelin-site.xml from conf/zeppelin-site.xml.template Add your interpreter FQCN in the property zeppelin.interpreters <property> <name>zeppelin.interpreters</name> <value>org.apache.zeppelin.spark.sparkinterpreter,org.apache.zeppelin.spark.pysparkinterpreter, org.apache.zeppelin.spark.sparksqlinterpreter,org.apache.zeppelin.spark.depinterpreter, org.apache.zeppelin.markdown.markdown,org.apache.zeppelin.shell.shellinterpreter, org.apache.zeppelin.hive.hiveinterpreter,com.me.mynewinterpreter </value> </property> 20

21 Simple AsciiDoc Interpreter! 1 Raw Text Block JVM Zeppelin Server 2 Raw Text Block Zeppelin Engine JVM AsciiDoc Interpreter 4 HTML Output 3 Converted To HTML 21

22 Simple interpreter (AsciiDoc)

23 Cassandra Interpreter Architecture! 1 Raw Text Block JVM Zeppelin Server 2 Raw Text Block JVM Cassandra Interpreter 3 Async CQL statements Cassandra Java Driver Cassandra 6 Render HTML 5 Display Results as HTML 4 23

24 Cassandra Interpreter Commands! Native CQL statements SELECT * FROM ; INSERT INTO ; Schema commands DESCRIBE TABLE ; DESCRIBE KEYSPACE ; Options ; Prepared statements ; Help command HELP; 24

25 Complex interpreter (Cassandra)

26 Zeppelin future!! Roadmap!

27 Roadmap & future! More graph options (Map viz ZEPPELIN-157) Helium project, packaging Zeppelin view, logic (code) & resource into Applications Interpreters packaging re-design ship & compile core interpreters only third-parties interpreters can be pulled from repository which interpreter is core? Who will maintain? Community. Integrate security (Apache Shiro, ZEPPELIN-53 ) 27

28 Roadmap & future! Out of incubation state to become 1 st class Apache project 28

29 ! "! Q & R

30 Thank You