Building a platform from open source at Yahoo! Dus$n Whi*le. Dustin Whittle Yahoo! Developer Network

Transcription

1 Building a platform from open source at Yahoo! Dustin Whittle Yahoo! Developer Network

2

3 Overview Why symfony? symfony vs ysymfony What does scaling really mean? Social Search: Delicious and Answers Yahoo! Open Strategy What is the Yahoo! Open Stack? Application Platform + Developer Tool Developer Tools YUI, Design Patterns, Tutorials Data (YQL) & Social APIs YQL, Geo, Profiles, Connections, Updates, YOS SDK for PHP Building an open application with symfony and YOS OAuth / YQL / OpenSocial

4 Who am I? Working with symfony since open source symfony Core Team Member Responsible for the development and support of symfony at Yahoo! Worked with Y! Answers, Delicious, Y! Widgets, Y! Bookmarks, Yahoo! Application Platform Consultant Commercial symfony support + training (USA)

5 EXAMPLES TUTORIALS CODE SAMPLES DEVELOPER.YAHOO.COM

6 FREEBSD LINUX APACHE PHP MYSQL BUGZILLA HADOOP SYMFONY YAHOO! IS POWERED BY OPEN SOURCE TECHNOLOGIES

7 W3C MICROFORMATS OAUTH OPENID OPENSOCIAL YAHOO! EMBRACES OPEN STANDARDS

8 RASMUS LERDORF DOUG CROCKFORD DOUG CUTTING CHRISTIAN HEILMANN YAHOO! HIRES OPEN SOURCE DEVELOPERS

9 YUI BROWSER PLUS DESIGN PATTERNS R3 YSLOW + PERFORMANCE RULES YAHOO! GIVES BACK TO OPEN SOURCE

10 YQL PIPES BOSS CONTACTS UPDATES MAIL DELICIOUS FLICKR UPCOMING HOTJOBS MAPS FIREEAGLE GEOLOCATION LOCAL TRAFFIC WEATHER MUSIC ANSWERS SHOPPING FINANCE TRAVEL YAHOO! SHARES ITS DATA THROUGH OPEN APIS AND WEB SERVICES

11 Conferences Hack Days HackU Tech Talks YDN Theater YAHOO! ENGAGES COMMUNITIES WITH OPEN HACK EVENTS AROUND THE WORLD

12

13 Apache Custom Modules Users Load Balancers Frontend PHP APC, PEAR, PECL, Custom Extensions ysymfony / YUI Backend MySQL/Oracle Web Services Ad API User API FreeBSD 4.x/6.x, Linux 2.6.x

14 Y! needs from a frontend platfrom Fit existing environment (RHEL/PHP5/Apache) Development Cycle How easy to develop, test, and deploy? Clean separation between data, logic, and display (MVC) Independent model layer to fit service oriented architecture Extensible and pluggable Internationalization and localization support Detailed documentation and active community of support Open source and ability to contribute back

15 Why a frontend presentation platform? Rasmus says frameworks are not well suited for Y! Build applications to requirements Do exactly what you need: no more, no less Understand that frameworks add a lot of overhead Choosing functional components is a better fit Despite choosing open source or building your own Everyone uses a framework If you use open source, use only the pieces you need

16 Why a framework at all? Another software layer (ysymfony, yphp, yapache) Factors out common patterns Code Layout Configuration URL Routing Authentication / Security Form Validation / Repopulation Internationalization / Localization Encourages good design Abstraction > Consistency > Maintainability

17 The choice to adopt symfony? Philosophy Full-stack framework for building complex web applications Adopt best ideas from anywhere, using existing code if available (Mojavi, Prado, Rails, Django) Design Clean separation between Model, View, and Controller Controller using modules and actions Views using templates in straight PHP with helpers Easy to reuse view modules to compose a page Layouts, Components, Partials, Slots

18 The choice to use symfony Configurability / Flexibility Features we do not want are easily disabled Use of factories for easy customization Documentation / Support Community The Definitive Guide to symfony (free online) Excellent tutorials and example applications Askeet & Jobeet Active community with wiki, mailing lists, forums, irc channel

19 Why ysymfony for Yahoo! teams? Eliminate common patterns by adding a layer on PHP Code layout/structure (MVC) Configuration Internationalization ysymfony is just a toolkit Learn one set of tools Shift between multiple projects Consistency Long term maintainability through platform

20

21 How Yahoo! uses symfony Full Stack Yahoo! Bookmarks Yahoo! Answers Delicious Yahoo! Widgets Yahoo! Application Platform Developer Tool Corporate Reporting Tools Components Forms Caching Routing

22 symfony MVC framework

23

24

25 A look at Yahoo! Answers Yahoo! Answers is the largest collection of human knowledge on the Web with more than 135 million users and 515 million answers worldwide (Yahoo! Internal Data, March 2008). Yahoo! Answers is the 2nd ranked education & reference site on the web (comscore) Available in 26 markets and 12 languages

26 Yahoo! Answers at the beginning Started as a small development team on PHP4 from a fork of Yahoo! Taiwan Knowledge+ Launched December 2005 by December 2006 there were 60 million users and 65 million answers The code base eventually became difficult to maintain and iterate new features Large distributed development teams (US / UK)

27

28

29

30

31

32

33

34 The big picture A complete platform for building web applications from frameworks PHP Framework JavaScript Framework CSS Framework UI Design Patterns + Best Practices Development Tools (logger, profiler, debugger, docs) Unit + Functional Testing Frameworks (LIME / YUI Test) Deployment Tools (rsync deployment system)

35 What does Yahoo! change? Minor changes to fit our environment (bsd/php/apache) Most of our changes are easily implemented via factories Added dimensions to configurations (ysfdimensionsplugin) Integrate R3 translation/template management (ysfr3plugin) R3 - Dropped the ORM and pushed down the stack (SOA) Added a parallel API Dispatcher (ysfapiclientplugin) Created a build and deployment solution (ysfbuildplugin) Integrate support for Y! User Interface libraries (ysfyuiplugin)

36 Propel or Doctrine or??? No ORM for large projects Propel or Doctrine for large projects Doctrine for internal projects (best supported) Service Oriented Architecture Platforms as services (reusable to all) No heavy lifting, push down the stack Thin Controller/Fat Model (where model == services) Java/C++/Erlang + JSON/XML

37 Localizing the experience with dimensions Cascading Configuration based on YAML Framework -> Project -> Application -> Module Extending the cascade to be based on dimensions Dimensions can be anything (and can be chained together) Caching Data Center + Environment for customizing configurations Culture for localizing user interface + data Theme for customizing look and feel User info (is user on corporate intranet?)

38 symfony I18n + L10n Translation + Template Management Integrated with symfony i18n + view layers <r3:trans>hello</r3:trans> vs ( hello ) Dictionary Management (YALA) SQLite/XLIFF Backend Added tasks Importing/Exporting XLIFF dictionary Generating translations for projects Find missing strings

39 A build and deployment system Aggregate and minify stylesheets and javascripts Rewrite templates, css, js for CDN (Akamai, S3, ) Generate translations for configurations + templates Generate configuration cache Aggregate core classes + remove debug statements Run lint, unit, functional tests Package applications as.tgz Deployment via rsync

40 What does it mean to scale? A system whose performance improves after adding hardware, proportionally to the capacity added, is said to be a scalable system. High Availability + Scalability + Performance Bigger dataset, more traffic, maintainable Not about performance PHP is slow, but it is not your bottleneck Languages do not scale, architectures do. Planning to grow and planning to fail Capacity Planning Business Continuity Planning

41 Scaling Planning Planning hardware purchases and hosting options to have as much as you need without breaking your wallet Partitioning and distributing databases to support large datasets and simultaneous transactions Monitoring your applications to find and clear bottlenecks Providing services APIs and using services from other providers to increase your site's reach and capabilities Think Minimal, Plan to grow, Plan to fail.

42 Scaling The basics in PHP PHP is rarely the bottleneck Most performance comes not from the language, but from application design - Rasmus Share Nothing Architecture Independent, self-sufficient, no single point of contention No local storage = No PHP Sessions Use a database (works for distributed) Use a small signed cookie (ideal) Important data in database Individual expiration on session objects Small data items Use a distributed cache Memcache Forget about small efficiencies -- Premature optimization is the root of all evil.

43 Scaling Databases The basics Master/Slave Replication First steps Helps with reads, writes are still bottleneck Partitioning Segmenting data Sharding (horizontal partitioning) Segmenting data onto different physical machines Make problems smaller, easier to grow

44 Improving latency with Caching Always use PHP opcode cache (APC, Xcache, etc) Use for routing and i18n cache Memcache (distributed cache) Use for view cache Distributed invalidation can be a pain sfviewcachemanager makes this easy! Be intelligent about cache_keys (uri, user, state) There is a fine line to caching At what point do you spend more time managing the cache, than reading from it?

45 Tweaking Performance Don t use features you do not need settings.yml / factories.yml Use core_compile (aggregate classes) Remove debug statements (sfoptimizerplugin) Do not use.htaccess (move to real apache config) Set a minimal include path Increase realpath_cache_size + realpath_cache_ttl Use apc.stat=0 Do not use components in loop

46 Do it yourself for cheap Open source software = Free Apache PHP MySQL Memcache / Perlbal / MogileFS / Squid / Gearman symfony / Doctrine / Propel / Swift Nagios Amazon Shared Infrastructure = Cheap EC2 Cloud Computing S3 Distributed Storage SimpleDB

47 Yahoo! Open Strategy

48 OPENID XRDS OAUTH PORTABLE CONTACTS OPEN SOCIAL OPEN PLATFORMS + COLLABORATION

49 The Open Web

50 Y! OS The Open Stack

51 What does Yahoo! offer developers? Yahoo! Developer Network Developer Tools (YUI, etc) Social APIs Profiles ConnecWons Updates Data APIs OAuth Yahoo! Query Language Yahoo! ApplicaWon PlaYorm OpenSocial

52

53 developer.yahoo.com

54 What is Yahoo! Developer Network? The Yahoo! Developer Network offers open source tools and open data APIs to make it easy for developers to build applications and mashups. 50+ APIs / Web Services Developer Dashboard Tutorials + Code Samples API Documentation Yahoo! User Interface libraries + ASTRA Design Patterns Library PHP / Python / Ruby Dev Centers Evangelism: Conferences / Theater / Blogs / Events

55

56 Y! Developer Network YUI Javascript JavaScript Framework Utilities - YAHOO, Dom, Event, Animation, Browser History Manager, Connection Manager, Cookie, DataSource, Drag and Drop, Element, Get, ImageLoader, JSON, Resize, Selector, Loader Controls / Widgets - AutoComplete, Button, Calendar, Charts, Color Picker, DataTable, ImageCropper, Rich Text Editor, Slider, Uploader Container (Module, Overlay, Panel, Tooltip, Dialog), Layout Manager, Menu, TabView, TreeView Debug Logger, Profiler, Test

57 Y! Developer Network YUI CSS CSS Foundation Reset - Neutralizes browser CSS styles Base - Applies consistent style foundation Fonts - Foundation for typography and font-sizing Grids - Thousands of wireframe layouts User Interface Design Patterns Library Proven solutions to common interfaces Grade Browser Support / Progressive Enhancement

58 Y! Developer Network Documentation More than 275 functional examples YSlow + Performance Rules YUI Blog Mailing Yahoo! Groups

59 A Query Language for the Web SELECT * FROM INTERNET

60 Before YQL Thousands of Web Services that provide valuable data Require developers to read documentation and form URLs/queries. Data is isolated Needs combining, tweaking, shaping even after it gets to the developer.

61 Y! Open Stack YQL SQL-Like Language Synonymous with Data access Familiar to developers Expressive enough to get the right data Self Describing - show, desc table Allows you to query, filter and join data across Web Services.

62

63 YQL Open Tables Twitter Weather Wesabe Whitepages Zillow. Delicious Dopplr Friendfeed Github New York Times Shopping Available on github h[p://github.com/spullara/yql tables/

64 YQL Open Tables

65 YQL Open Tables

66 YQL Open Tables

67 YQL - Examples select * from social.connections select * from delicious.feeds.popular select * from flickr.photos.interestingness select * from friendfeed.status select * from github.checkins

68 YQL Javascript Execute Allows executing of javascript on the server side to mashup (join, filter, etc) Makes combining many web services very simple Support for Oauth

69 YQL API End Points OAuth Endpoint Public Endpoint YQL Console

70 Y! Open Stack Application Platform Allows developers to deploy their own web based applications on Yahoo! Multiple Views: Small and Canvas Social Context: the new Yahoo! Social Directory OpenSocial 0.8 Javascript APIs

71 Y! Open Stack Open Applications

72

73

74

75 Y! Open Stack SDKs PHP SDK Available Open and OAuth Applica$ons Ac$onScript 3 SDK Available Open Applica$ons Objec$veC SDK Available Open and OAuth Applica$ons

76 YOSSDK Methods 3 Legged OAuth getsessioneduser (session) getowner (session) getuser (session) query (session) getpresence (user) setpresence (user) listupdates (user) listconnecwonupdates (user) insertupdate (user) deleteupdate (user) loadprofile (user) getconnecwons (user) getcontacts (user) setsmallview (user) 2 Legged OAuth setsmallview (applicawon) query (applicawon)

77 YOSSDK 2-Legged OAuth Used For: Public user data and open APIs

78 YOSSDK 3-Legged OAuth Used For: Private data access

79 Building an Open App : CommonGround Experience: Basic Web Dev Knowledge PHP/HTML/CSS/ Javascript What we are building: Common ground Find out what you have in common with your social graph: music, movies, books, hobbies. What we will use: YOSSDK, YQL, YAP

80

81

82 h*p://developer.yahoo.com/dashboard

83

84

85

86

87 CommonGround available on GitHub

88 QUESTIONS? (I will be around for the whole event, so please do say hello and ask questions)

89 WANT TO JOIN YAHOO? WE ARE HIRING AND HAVE INTERNSHIPS!

90 ENJOY THE REST OF SYMFONYLIVE 2009 EXAMPLES TUTORIALS CODE SAMPLES DEVELOPER.YAHOO.COM

91 Books worth reading

92