visual interface dependency or relationship model of systems workflow engine comparator function poller

Transcription

1 discovery dependency or relationship model of systems Metric Store visual interface workflow comparator function poller

2 Model of system polling/threshold/state system state state state state data) state data) data) data) data) http (REST) interface http (REST) interface management system web application interface web interface notification ticket creation

3 The system model is expected to be reasonable complex, but not have significant needs for scale. It is the object database of all "interesting" elements in our computing system that we're paying attention too - from operating system metrics to application performance. Since other components of our overall system are expected to need to scale horizontally, we reference those components through a REST style interface. entities in the ststem model have addressable references to pollers and state data state state state state data) state data) data) data) data) http (REST) interface Model of system (ical and physical connectivity represented as relationships) http (REST) interface Entities modeled include multiple relationships including depends-on, contained-with, etc. Entities also have references to anything that we're interested in storing externally - typically that data will be set to be gathered by a poller (pull data) or just stored (push data). The polling system should be set up to be able to dynamically create, delete, and update pollers as desired. Templates of pollers for a common configuration may able be invoked and set up on an entity in the system model. The system model would include things like operating system image, java application, apache, mysql, network interface, disk file system, etc. Ideally the model should be discoverable and build-able automatically.

4 A poller and it's associated state data should be an addressable entity (REST interface) that is referenced and resolved from a system that manages the architecture of the environment being monitored. The pollers may also have identified thresholds or other means of generating s that are expected to flow external to this monitoring system. The polling system is expect to exhaust internal resources at some point and therefore needs to be expandable to multiple instances that can all be equally referenced. Bonus would be to have the polling systems have knowledge of each other and be able to forward references/data as needed, migrating pollers and associated state repositories to balance load across multiple systems. entities in the ststem model have addressable references to pollers and state data http (REST) interface system object (application, service, etc) state data) state data) operating system (container) http (REST) interface pass-through state data) The pollers may have identified thresholds or other calculations which trigger s - or perhaps every request triggers an. In either case, there will be an expected stream leaving the polling systems. Potential protocols for sending include XMPP or open source message queues. flow The pass-through mechanism is a special case enabling the basic storage of values/state data without having this polling system request it.

5 state/persistence The poller needs to inquire to the state of all "monitors" that need to be fulfilled, which will grow over time (i.e. the expectation that you update a given monitor every 5 minutes). mon req When the poller is complete with a given "monitor" request, it should return the state to the persistence element. TIME state/persistence mon mon reqmon reqmon req req mon mon mon req mon req req req If the poller goes offline, or dies, then any elements that it's working on won't get fulfilled, and the number of outstanding items to be updated will grow. If we populate a queue with all elements to be updated, then we'll need to manage that queue so that we don't end up with duplicate elements in that queue in the of poller failure. We need something more like a common "ticket board" than a queue structure. state/persistence mon req Most queue systems are a push model: beanstalkd, celery, gearmand. What's needed for this use case is a pull model. Celery, although the most complex, comes closest to providing that basic mechanism.

6 a poller (monitor) is a block of code that runs concurrently with other components within the system, iterating on an expected cycle time. Common poller interfaces should be expected to include SNMP, JMX. Taking advantage of Nagios plugins as an implementation component would speed adoption and provide a quick set of flexible plugins. The poller should ideally be able invoke and run any arbitrary script - ruby, python, perl, bash, java, etc. in order to get its results. SNMP JMX HTTP script state data) the state should be a fast dumping ground for the results of the poller - whether it's an enumerated state or a value. The values should be able to be very quickly updated and localized to be associated with the poller to minimize latency and potential failure. The overall interface of the polling systems should be expected to create, delete, and modify aspects of the poller as well as request data from the state, including potentially a graph of the state. Being able to export state data in bulk for a predefined period of time would allow the export into a batch-processing environment for further correlation and detailing (i.e. Hadoop/Hive/Pig) and potential correlation with other data sources (ging). Otherwise a polling system should be treated as a sharded database of time information, with the system requesting data asking all relevant systems what it needs to know and fusing the resulting data appropriately. The REST interface for the polling subsystem should include a basic web interface sufficient to interact with the system through a web browser, including functionally testing the components.

7 The management system is the least fleshed out, but where the most opportunity exists from a value standpoint outside of convenience and scale. In existing solutions, the management system is where s are displayed with some priority, typically tagged in various ways to be more useful. Events other than polling/threshold results can be processed, including system ging (sys). Many management systems include a mechanism to create external notifications (send , page, etc) and potentially manage escalation related to that. In the that the system was expected to handle escalations, the management system would need to also manage state. Advanced monitoring would enable a triggering of a bit of specialized code (specific and customized to any given instance) that would enable the checking of state of other related components (using the system model and associated state referenced from it) to produce more actionable s, including supressing duplicate s and the generation of tickets into an existing ticketing system for resolution. flow notification notification notification notification? ticket creation The potential exists to have the management system either ride atop or simple be a big-data style blackboard (NoSQL - HBase/Hadoop, Voldemort, etc) enabling interesting correlations and system analysis capabilities with another component external to the management system. For basic purposes, simple external notification and basic deduplication/alert supression based on system dependencies would be sufficient.

8 comparison poller raw data comparison function poller raw data flap detection simple dependency analysis escelation environment model comparison function poller raw data correlation dependency analysis flap detection deduplication escelation

9 poller raw data comparison escelation

10 comparison function poller raw data flap detection simple dependency analysis escelation

11 environment model comparison function poller raw data correlation dependency analysis flap detection deduplication escelation

12 MySQL monitor result state (cron) get monitors needing to be polled Event Processor django application polling work requests gearmand beanstalkd polling result data performance information & time data messages ( , sms, etc) This arrangement adds a bit more complication - the work and result queues - but enables the poller code to be anywhere and agnostic. The benefit is scaling out pollers onto other systems requires no state infrastructure so the pollers can be completely agnostic and exist only when needed. The queue mechanism allows the state to be stored off separately and dealt with as desired. The core application A few potential optimizations: 1) enable work request queue's for specific/custom pollers. The first to come immediately to mind would be something like "fping", which deals with ping results in a far more efficient manner, but would need a different input & output structure than your stock "monitor/poller". 2) the process also enables a modular interface that can be distributed and use whatever data state mechanisms that are effective. You could store time data into files, shove everything into HBase, or stash some components in one data store, other in another. The interface is also perfect for accepting the raw values and doing additional computational ic - either predictive functions based on earlier state, simple (stateless) thresholds, or even correlation with the additional of an external model and information.

13 poller control django project (cron) get monitors needing to be polled work 'queue'board cache (memcache) control/api Event Processor performance information & time data MySQL messages ( , sms, etc) Another take on the application structure, setting up django "applications" (subcomponents) as the elements and combining them into a single virtual machine that exports all the relevant API's as RESTful services. The main components include: * Work "queue" board - a view into the monitors and state stored within the application that shows what's pending an update. The queue board is set up something like an amazon SQS server or ghetto queue where an item can be reserved for something external to process, hiding it from view until an expiration period has elapsed. * - the API is a very lightweight control and information system to the status of the poller, the main task of which is offline processing outside of the scope of user requests. The poller will check the work-queue board for anything pending, pull down a set and process away, returning the results through the processor. Note that the poller itself is an independent process from the Django application. * Event processor - the API by which s returned through the work-queue board can interact ically. The processor can be a worker process for an internal queue from the work-queue board to process the results as they're received by a poller.