Apache Zeppelin, the missing component for your BigData ecosystem DuyHai DOAN, Cassandra Technical Advocate
Who Am I?! Duy Hai DOAN Cassandra technical advocate talks, meetups, confs open-source devs (Achilles, ) OSS Cassandra point of contact duy_hai.doan@datastax.com 2
Datastax! Founded in April 2010 We contribute a lot to Apache Cassandra 400+ customers (25 of the Fortune 100), 400+ employees Headquarter in San Francisco Bay area EU headquarter in London, offices in France and Germany Datastax Enterprise = OSS Cassandra + extra features 3
What is Apache Zeppelin?! Presentation! Architecture!!
Zeppelin Presentation! 5
Zeppelin Architecture! JVM Spark Interpreter Group JVM Spark SparkSQL R E S T WebSocket Zeppelin Server Zeppelin Engine Zeppelin Interpreter Factory JVM Tajo Interpreter JVM Flink Interpreter JVM Cassandra Interpreter 6
What does Zeppelin provide?! Front-end & display system for free Generic back-end with REST APIs & WebSocket Pluggable interpreters system Task scheduler (à la CRON) 7
Zeppelin UI Layout! Notebook! Paragraph! UI elements!
Demo https://github.com/doanduyhai/incubator-zeppelin/tree/zeppelinpresentation
Zeppelin Display System!! Raw, Table, HTML! Available graphs! View modes! Dynamic form! Iframe export!
Demo https://github.com/doanduyhai/incubator-zeppelin/tree/zeppelinpresentation
Interpreter system!! Core interpreters! Third-parties interpreters! Interpreters conf & usage!
Interpreter processing lifecycle! 1 Receive input commands/data as raw text from form data 2 3 4 Process the input commands/data by the external back-end Format the response using Zeppelin display system Send response back to the Zeppelin engine 13
Core interpreters!! Spark (Spark core, SparkSQL/DataFrame, PySpark) Spark core = default (or %spark) SparkSQL = %sql Shell (%sh) Markdown (%md)! AngularJS (%angular) 14
Third-parties interpreters! Hive Phoenix Tajo Flink Ignite Lens Cassandra Geode PostgreSQL Kylin 15
Interpreter conf & usage https://github.com/doanduyhai/incubator-zeppelin/tree/zeppelinpresentation
Writing An Interpreter!! How To! Simple interpreter example (AsciiDoc)! Complex interpreter example (Cassandra)!
Steps to write your own interpreter! Create a class that extends Interpreter base class Register it in a static block static { Interpreter.register("MyInterpreterName", MyClassName.class.getName()); } Optionnally define default config params static { Interpreter.register("MyInterpreterName", MyClassName.class.getName(), new InterpreterPropertyBuilder().add("property1", "default value", "Description of property1").build()); } 18
To register your interpreter as default! Edit the enum ZeppelinConfiguration.ConfVars Add your interpreter FQCN in the property ZEPPELIN_INTERPRETERS 19
To register your interpreter in config files! Create conf/zeppelin-site.xml from conf/zeppelin-site.xml.template Add your interpreter FQCN in the property zeppelin.interpreters <property> <name>zeppelin.interpreters</name> <value>org.apache.zeppelin.spark.sparkinterpreter,org.apache.zeppelin.spark.pysparkinterpreter, org.apache.zeppelin.spark.sparksqlinterpreter,org.apache.zeppelin.spark.depinterpreter, org.apache.zeppelin.markdown.markdown,org.apache.zeppelin.shell.shellinterpreter, org.apache.zeppelin.hive.hiveinterpreter,com.me.mynewinterpreter </value> </property> 20
Simple AsciiDoc Interpreter! 1 Raw Text Block JVM Zeppelin Server 2 Raw Text Block Zeppelin Engine JVM AsciiDoc Interpreter 4 HTML Output 3 Converted To HTML 21
Simple interpreter (AsciiDoc) https://github.com/doanduyhai/incubator-zeppelin/tree/zeppelinpresentation
Cassandra Interpreter Architecture! 1 Raw Text Block JVM Zeppelin Server 2 Raw Text Block JVM Cassandra Interpreter 3 Async CQL statements Cassandra Java Driver Cassandra 6 Render HTML 5 Display Results as HTML 4 23
Cassandra Interpreter Commands! Native CQL statements SELECT * FROM ; INSERT INTO ; Schema commands DESCRIBE TABLE ; DESCRIBE KEYSPACE ; Options Commands @consistency ; @retrypolicy ; @fetchsize ; Prepared statements Commands @prepare ; @bind ; @remove_prepared ; Help command HELP; 24
Complex interpreter (Cassandra) https://github.com/doanduyhai/incubator-zeppelin/tree/zeppelinpresentation
Zeppelin future!! Roadmap!
Roadmap & future! More graph options (Map viz ZEPPELIN-157) Helium project, packaging Zeppelin view, logic (code) & resource into Applications Interpreters packaging re-design ship & compile core interpreters only third-parties interpreters can be pulled from repository which interpreter is core? Who will maintain? Community. Integrate security (Apache Shiro, ZEPPELIN-53 ) 27
Roadmap & future! Out of incubation state to become 1 st class Apache project 28
! "! Q & R
Thank You duy_hai.doan@datastax.com http://zeppelin.incubator.apache.org/