What Counts? Using IPDR to accurately measure subscriber usage in DOCSIS networks

What Counts? Using IPDR to accurately measure subscriber usage in DOCSIS networks [ Know Your Network ] Executive Summary Service Providers worldwide are becoming increasingly concerned over the rising tide of network resources consumed by their subscribers. New Internet applications and richer media content drive a voracious appetite for capacity while every subscriber s expectation for an enhanced service experience continues to increase. This demand for resources increases the provider s costs while revenues remain constant. In Cable Broadband, rising use of capacity within the access network impacts the Operator s DOCSIS infrastructure. With the rollout of DOCSIS 3.0, IPDR (Internet Protocol Detail Record) is now mandatory and provides a powerful new tool for the visibility and management of subscriber service flows. With Cable operators deploying subscriber management solutions and resource capacity models based on information gathered using IPDR, it is critical that accurate usage measurements are made. A number of complex factors influence the behavior of IPDR-based measurements when applied to subscriber usage metering scenarios in DOCSIS networks. In this whitepaper, Applied Broadband Architect, Andrew Sundelin, examines these diverse factors and suggests best practices for accurate subscriber usage monitoring in Cable networks using IPDR.

[ Know Your Network ] Contents Introduction 3 DOCSIS Background 3 Implications of Deploying IPDR in Cable 8 Best Practice for IPDR and DOCSIS 9 Conclusion 11 About the Author 12 About Applied Broadband 12 Figures Input Source to Classifier to Output Service Flow 5 Address Range Options for Simple Classifiers 7 Pipeline s Deployment in Cable OSS/BSS 8 2

What Counts? Accurately measuring subscriber usage in DOCSIS networks with IPDR Introduction To the chagrin of Homer Simpsons everywhere the days of all you can to eat high-speed data service are coming to an end. Just recently Verizon s CTO was quoted as saying that the broadband industry will see a pricing paradigm shift because broadband service providers cannot continue to grow the Internet without passing the cost on to someone. 1 If the usage accounting flood hasn t happened yet it is most likely on the verge. Cable Operators have long viewed the IPDR Subscriber Accounting Management Interface Specification (SAMIS) as the best yardstick for measuring subscriber usage in DOCSIS networks. The problem, however, is that without careful design from the classifier and service flow level all the way to the back office the potential exists to not get the numbers right. Early adopters of IPDR-based usage accounting have quickly found an extremely observant user community. When things happen such as discrepancies in reported usage between an operator web server and an email from the same operator usage accounting quickly gets a black eye. Introducing usage accounting also causes subscribers to start monitoring their own usage more carefully technologies such as tomato (open source router software for many Linksys wireless access points) make this easier than ever before. 2 These sophisticated users seem likely to be the same users who are above average consumers of data. Another area where operators need to be careful is in which packets they count against any subscriber usage caps. While the majority of packets sent to or from an MTA or CM are likely to be from subscriber PCs or calls, the amount of traffic that is used to manage and provide service to that device is not insignificant. Traffic for ARP and call signaling are essential to the functioning of DOCSIS and PacketCable services, but they are generally not considered end-user data traffic and most customers would likely agree that they should not be counted against their data service usage caps. Given the sensitive nature of usage accounting in the eyes of subscribers it is imperative that operators get it right. This white paper takes a holistic approach to accounting management in Cable networks using DOCSIS and IPDR. We focus on best practices for designing classifiers and service flows so that data can be categorized, counted, and collected appropriately and accurately so that usage accounting can meet the operator s overall capacity engineering, business, and subscriber experience goals. DOCSIS Background Service Flows Service flows are the fundamental mechanism for providing QoS to subscribers as bandwidth is allocated in DOCSIS networks on a service-flow-by-service-flow basis. The entirety of all IP traffic to or from a given subscriber is carried in a set of operator-configured service flows. However, service flows are more than that. Since service flows are primarily used for QoS, all DOCSIS usage metering is performed on a service flow basis. This means that to determine the total usage of a subscriber the byte count from all of their CM s service flows needs to be totaled together. The design and implementation of service flows in DOCSIS networks is fundamental to the behavior of subscriber usage monitoring. Service flows are unidirectional so there are always at least two service flows per CM a downstream primary service flow and an upstream primary service flow. Additional service flows can either be statically provisioned in the CM configuration file or can be added dynamically through PacketCable voice or PacketCable Multimedia mechanisms. 1 http://telephonyonline.com/residential_services/news/verizon-cto-metering-092909/ 3

Classifiers Since there is more than one service flow per CM how does the CM determine which service flow to transmit an upstream packet on? 3 That s where the classifier comes into play. Each classifier has an associated match criteria, priority and service flow. The highest priority classifier whose match criterion (aka rule) matches the classifier determines which service flow the CM uses to transmit the packet. The operator is responsible for designing and provisioning a collection of DOCSIS classifiers that meshes with their service flow architecture to realize their service delivery goals. Match criteria or classifier rules are layer 3 or layer 2 fields that need to match a field in the packet. Here is a list of packet attributes that can be matched against in a classifier: IP ToS/DSCP bits IP Protocol (e.g. TCP, UDP, ARP) IP Source and Destination Address TCP/UDP Source and Destination Port Range MAC Source and Destination Address MAC Management Message Type VLAN Priority and Id CM Interface (DOCSIS 3.0 only) While the set of classifier parameter types is rich it is important to note that there are some significant limitations to some of these types. In particular, it is not possible to create classifiers for a number of MAC Management Messages (MMMs). 4 Therefore, these MMMs always traverse the primary service flow. Thus, if an operator wants to avoid including MAC Management Messages in their subscriber usage accounting it is a requirement that they classify CPE traffic off of the primary service flow. In a broader view, it is also important to note that the CM or MTA only has one set of classifiers that apply to all traffic destined for the upstream regardless of whether the packet originates from the MTA stack (e.g. SNMP, MGCP call signaling, voice traffic, etc) or from a subscriber PC. The following figure illustrates the way a set of classifiers is applied to inbound packets by the MAC layer. In this scenario, there is an active POTS call on the MTA voice jack that has created a mixture of RTP bearer, RTCP traffic control, MGCP signaling traffic on the MTA s internal interface to the DOCSIS CM MAC. The CM s own internal interfaces are generating an SNMP response and performing periodic ranging. All while a user on a PC in a different room is audio chatting while web surfing. Finally, there is some layer 2 traffic both the CPE and the MTA require an ARP refresh on their default gateway s IP address. In this relatively simple case, there is a very precise classifier for the MTA voice call s RTP traffic. All other traffic from the MTA IP address block is considered signaling traffic and is sent on the Signaling Service Flow. 5 The other interesting traffic in this example the CPE Ethernet traffic the PC in a different room which also has RTP traffic. This RTP traffic doesn t match a port classifier; rather the classifier design assumes that any traffic from the CM or MTA IP addresses have already been handled and any remaining IP traffic is subscriber traffic. By explicitly 2 Refer to http://www.dslreports.com/shownews/cogeco-still-struggling-with-accurate-meters-104077 3 This same problem exists for the CMTS in determining which downstream service flow to send packets on, but for clarity this discussion will focus on the upstream direction. 4 Refer to Section C.2.1.8.3 of CM-SP-MULPIv3.0-I11-091002, which explicitly specifies that ranging and registration MAC Management Messages cannot be classified. 5 Note, that since PacketCable telephony service flows are dynamic their classifiers are also dynamic. So, the first classifier in the example is not from the CM configuration file and has a classifier priority from the dynamic range. 4

mapping this traffic to the Subscriber Traffic SF it avoids the catch-all nature of the default/primary service flow which could inadvertently increase the amount of subscriber traffic. This is an important feature of a classifier design since it is impossible to classify all DOCSIS MAC Management Messages off of the primary/default service flow. Thus, some MMM traffic would always count against the subscriber s volume cap if the default service flow were utilized for the Subscriber Traffic Service Flow. MTA Internal Input Queue CM Internal Input Queue CPE Ethernet Input Queue RTP (voice bearer) RTCP (signaling) MGCP (signaling) RTP (voice bearer) ARP (management) SNMP (management) Ranging Resp. (MMM) ARP (management) RTP (voice bearer) RTCP (signaling) HTTP (web browsing) HTTP (web browsing) CM MAC Classifier Rule by Priority 80 RTP from MTA IP to Voice Bearer SF 40 Everything from MTA IP Address Range to Signaling SF 20 Everything from CM IP Address Range to Management SF 10 All IP traffic to Subscriber Traffic Service Flow Signaling Service Flow Voice Bearer Dynamic Service Flow Management/Default Service Flow Subscriber Traffic Service Flow RTP (voice bearer) ARP (management) RTCP (signaling) SNMP (management) RTP (voice bearer) MGCP (signaling) RTCP (signaling) RTP (voice bearer) Ranging Resp. (MMM) HTTP (web browsing) ARP (management) HTTP (web browsing) Figure 1. Input Source to Classifier Table to Output Service Flow Finally, this example has a little bit of layer 2 traffic both the CPE and the MTA are refreshing their ARP cache. Since all of our classifiers are IP-layer classifiers these packets do not match any of them and default to the primary service flow. 5

QoS Parameters & Service Classes As mentioned above, the service flow is the fundamental mechanism for delivering QoS in DOCSIS. How does DOCSIS quantify Quality of Service? QoS is quantified by large set of parameters like maximum sustained rate, minimum reserved rate, latency and jitter tolerance, traffic priority, etc. Generally only a subset of the total possible QoS parameters are assigned to a given service flow, but this subset can still be large. The number of parameters describing QoS on just a single service flow can be substantial and inferring the intent of those parameters (e.g. what service they support) is not particularly human friendly. To ease the management burden of configuring QoS, DOCSIS also provides the concept of a Service Class. A service class has an associated set of relevant QoS parameters and a Service Class Name (which is a string that is configured to identify the service class). The Service Class Name (SCN) is shorthand is a way to reduce the amount of data required to uniquely identify a QoS parameter set (QPS) and provides a mechanism to make a QPS more human-readable. Service class names are important from a usage accounting perspective because QPSes are not represented in IPDR SAMIS records, but service class names are. Thus, if an operator wants to separate the wheat (subscriber traffic) from the chaff (management traffic) and wants to quantify each of them independently then use of Service Class Names is essentially mandatory. Part of the reason that service class names are essentially mandatory has to do with the fact that PacketCable 1.x telephony service flows do not (and cannot) use them. This means that if the default service flow also doesn t have a SCN it s traffic can t be differentiated (on an SCN basis, anyway) from telephony traffic. This can be a big problem when customers are already paying separately for voice service they surely don t want their voice-related traffic counting against the usage cap for their data service. 6 IPDR The most important take home messages about IPDRs have already been mentioned but are worth repeating: IPDR SAMIS records are provided on a Service Flow basis so multiple SAMIS records are provided per CM per reporting interval The best way to differentiate between these SAMIS records for a given CM is through the Service Class Name as the SAMIS record contains no information about the Classifiers or the QoS associated with the service flow IP Traffic Taxonomy & Addressing What packets are end-user packets and what packets are management packets? Some operators attempt to argue that any packet that reaches a CM can be counted against a usage cap because it wouldn t have reached the CM if it wasn t required to run the network. This argument is a tough sell to customers who only get benefit from packets transmitted from or received by CPE devices. At the very least, it would appear that operators need to avoid counting voice-related traffic (not only bearer channel itself, but also signaling channel and related traffic) against data service caps since voice service is already an added-cost service separate from data. Customers might also consider network monitoring an operator requirement for keeping their data service usable, but not part of the service itself. Thus, it would seem that most customers would consider SNMP, ToD, TFTP, MMMs and other network management traffic separate from their own use of the network. It is important to note, however that some of these protocols required to manage or register a CM, for example TFTP download of the CM configuration file, are also legitimate end-user traffic types. Thus, a Classifier that naively matched all TFTP traffic (regardless of source address) and put it into a management service flow would not account for end user TFTP traffic properly as user traffic. 6 Having learned from PacketCable telephony s mistakes the PacketCable Multimedia specification does allow for the use of Service Class Names 6

The following traffic types typically flow to/from the CM as part of its DOCSIS management: DHCP, TFTP and ToD SNMP and SNMP Traps SYSLOG ARP DOCSIS MAC Management messages All of these protocols could legitimately come from a subscriber CPE as well as being sent from a CM. In addition to all the CM traffic types listed above, the MTA portion of an emta can also send the following traffic types as part of supporting PacketCable telephony service: DNS HTTP MGCP (call signaling traffic) RTP (bearer traffic) and RTCP (bearer control traffic) SIP (PacketCable 2.0) To reiterate a point mentioned previously: It is critical to observe that all of these protocols are legitimate end-user protocols as well. Thus, knowing the type of traffic is not sufficient one also has to know where that traffic comes from. Therefore an operator s IP addressing architecture can be critical to simplifying classifier design to ensure that only traffic to/from subscriber CPEs is counted. DOCSIS 3.0 makes this somewhat easier since Classifiers can be associated with specific internal and/or external interfaces, but the vast majority of operators are going to be required to determine a solution for a massive installed base of both DOCSIS 1.1 and 2.0 CMs. From a Classifier perspective, the ideal addressing architecture is one where CM IP addresses come from one address space, MTA IP addresses come from another address space and CPE IP addresses come from a third separate address space (or, more likely, a set of separate address spaces). Realistically, if public IP addresses are utilized for CPE then an operator is likely to have tons of smaller address blocks to manage for CPE addresses. Thus, classifiers based on CPE IP addresses are frequently impractical. Therefore, classifiers for MTA and CM IP addresses are critical as one can assume that the only other source of IP traffic behind the CM is the CPE. MTA 172.16.0.0/12 or 10.192.0.0/10 or Routable Block Separate from CPEs CM 10.0.0.0/8 (if MTA in 172 or routable ) 10.0.0.0/10, 10.64.0.0/10 & 10.128.0.0/10 (if MTA in 10.192.0.0/10) CPE Anything as long as CM and MTA IPs allow traffic originated there to be readily differentiated from CPE traffic Figure 2. Address Range Options for Simple Classifiers 7

Implications of Deploying IPDR in Cable Having examined the implications of DOCSIS network configuration on the behavior of usage counters, what happens to the IPDR flows containing usage data after they leave the CMTS? Between the network layer and the presentation of metered data to the subscriber there is a critical system of data collection and management that must be considered in order for a reliable and robust subscriber metering solution to be fully realized. Figure 3. Pipeline s Deployment in a Cable OSS/BSS Scalability & Storage IPDR was added to DOCSIS 1.1 for one reason and one reason only to address the inability of SNMP to scale to the level needed to collect per service flow data from every CM in an operator network at a frequency necessary to support accounting management processes. The data volumes from IPDR are orders of magnitude higher than all other sources of management data typically utilized in an operator network combined. For example, say a CMTS is configured to output IPDR records every 15 minutes and has 20,000 CMs each with 6 provisioned service flows. At 200B per IPDR SAMIS record that is 96MB per hour and 2.3GB per day. If we take 50 CMTSs like this (which is what is needed for a 1M CM network relatively small by major MSO standards), this adds up to 4.8GB per hour and 115.2GB per day. Do that every day for a year and suddenly you ve got 42TB of data bigger than all the storage in some operator s back offices combined. 8

Looked at another way, in this same scenario, an IPDR Collector for this million CM network needs to be able to average almost 7,000 IPDR transactions per second with much higher bursts to keep up with this real-time data. Thus, deploying IPDR not only requires substantial expertise to get the address, classifier and service flow architectures right, it also requires a massively scalable IPDR collection infrastructure. Practical Limitations of the IPDR Protocol Another consideration when deploying IPDR has to do with the design of the protocol itself. IPDR/SP only allows a single primary Collector for a given session and generally CMTS implementations only allow a given Service Definition (like SAMIS) to be transmitted on a single session. What does this mean in plain English? If you want to have multiple downstream systems consume IPDR data from a single CMTS they can t all get it from that CMTS directly. Only a single Collector can get it from the CMTS directly. This limitation of IPDR/SP calls for a layered architecture. A collection layer that gathers data from multiple CMTSs and then provides the data to multiple downstream consumers can proxy IPDR data for the network layer either via IPDR/SP re-exporting or via other protocols (e.g. SOAP). Diversity of CMTS implementations Though CableLabs has worked diligently to standardize the implementation of IPDR in DOCSIS, the definition of what fields make up a SAMIS record has evolved over time and there is significant variability in CMTS implementation of these records (from Cisco s almost completely proprietary XML over TCP version to minor variability in the data types used for a given field). A smart collection layer that removes this variability by standardizing the data can make this complexity disappear before downstream systems even receive the information. Best Practice for IPDR and DOCSIS Service Flow Design Best practice requires separate service flows for each type of traffic that need to be accounted for separately. A fairly simple standard solution for residential high-speed data and voice service would have three provisioned service flows in each direction for a total of six: Call Signaling Traffic Service Flows: The reason for the Call Signaling Service flows has less to do with usage accounting and more to do with signaling traffic needing better QoS than subscriber or management traffic to avoid dial tone delay and other call quality problems. 7 These flows might have a low maximum sustained rate, but a high priority or a different service flow scheduling type than the Subscriber Traffic Service Flows on the network. Subscriber Traffic Service Flows: These are the flows that CPE traffic gets classified to and that are used for usage accounting. These service flows would typically have the maximum sustained rates (US and DS) for the subscriber s service tier as part of their associated QoS parameters. Management Traffic Service Flows: Using the Primary or default Service Flow for Management traffic allows an operator to err on the side of caution when it comes to over-counting end-user traffic. This SF could have a low priority and a low maximum sustained rate, thus if subscribers were able to find a way to intentionally get their traffic classified to this flow (to avoid usage accounting) they would get a reduced quality of service. Of course, situations vary by operator. Some operators utilize Service Flows to map traffic into post-cmts MPLS VPNs or other more advanced uses of Service Flows. Other operators want more granularity for QoS or usage 7 Full discussion if this topic is outside the scope of this whitepaper. Contact Applied Broadband for more details. 9

accounting reasons and, thus, separate out additional types of traffic. On the other hand, some operators may want to simplify things further and combine the Management Traffic and Call Signaling service flows. Dynamic service flows such as telephony bearer channel flows or PacketCable Multimedia flows take care of themselves for the most part. The service flows and classifiers for these types of traffic are handled via the signaling for each of the individual service types. As mentioned previously, PacketCable telephony flows cannot have an associated Service Class Name 8 making them problematic from a usage accounting perspective the easiest way to keep them from looking like other service flows in a set of SAMIS records to have all other Service Flows utilize Service Class Names. 9 Service Class Naming To allow segregation of an individual CM s accounting records by service flow or aggregation of usage data by service class, the best practice is to have all provisioned service flows utilize Service Class Names. Generally, service class names are associated with a high-speed data tier or a class of traffic (e.g. signaling). An operator might have SCNs like LiteUp, LiteDown, ExtremeUp, ExtremeDown for HSD tiers; VoiceSigUp, VoiceSigDown for signaling; and SIPVoiceUp, SIPVoiceDown for PacketCable Multimedia dynamic flows. As mentioned this allows the operator to readily identify which SAMIS counters from a CM apply to which type of traffic. This is useful for usage accounting on an individual subscriber level, but it can also be useful for traffic management. In particular, the total data volume by service tier can also be trended over time and utilized to infer the contribution of individual tiers to interface utilization and other capacity management metrics. Classifier Design, Address Architecture and CM Configuration Files The simplest classifier design is based on a complementary IP addressing scheme. In particular, the best practice is to have CM internal IP addresses, emta IP addresses and CPE IP addresses all come from different address spaces. Any other approach to IP addressing can add substantial complexity to the classifier design or the provisioning of the classifiers themselves. A simple, direct approach to classifiers would be to utilize the CM and MTA MAC addresses for the classifiers, as this would clearly separate CM management-related traffic and MTA voice-related traffic from subscriber traffic. The downside to this approach is that instead of having a different CM configuration file per CM/MTA IP address block an operator would now need to have CM configuration files unique per subscriber. This essentially requires a provisioning infrastructure capable of dynamic configuration files, which not all operators have. A smart IPv4 addressing architecture can alleviate or minimize this problem, but for many operators their IPv4 addressing architecture is long established and a more complex classifier design or dynamic configuration files may be required in lieu of wholesale changes. That said, operators moving to IPv6 would be well advised to include ease of classifier management into their IPv6 address allocation strategy. Hand and glove with classifiers are packet filters. Another best practice is to configure the CM with drop filters on its CPE-side interfaces for the emta and CM internal IP address range. That way, if a subscriber tries to spoof IP addresses in either of these address ranges to avoid usage accounting the packets will be dropped by the CM. Along with having the non-subscriber Traffic SFs have relatively low rate limits, these drop filters provide additional assurance that subscribers won t be able to bypass usage accounting. 8 At least one CMTS vendor s equipment can be configured to insert an SCN into bearer channel IPDR records, and while this is a non-standard feature it could prove useful for differentiating traffic types when performing usage accounting. 9 There are other ways to differentiate telephony flows, but the SCN method is the only way that is consistent across the multiple versions of SAMIS as it has evolved from DOCSIS 1.1 to 2.0 and now to DOCSIS 3.0. 10

IPDR Infrastructure Design In addition to the DOCSIS complexity of Service Flow, Service Class and Classifier design, IPDR/SP adds the quirks of a new network management protocol to the mix. Best practice is to have an IPDR/SP collection and mediation layer between the OSS/BSS systems interested in IPDR data like SAMIS. This allows for distribution of this critical data to more than one back-office consumer. Given the quantity of data and its mission-critical nature, this collection layer needs to be high-speed, massively scalable and extremely robust. Ideally, it allows access to the IPDR data through a variety of interfaces. Conclusion The buffet is closing. The smörgåsbord is over. The days of all you-can-eat high-speed data service are finis. It is simply not fair to other subscribers or to operators bottom lines when 5% of the subscribers consume 50% of the bandwidth. This simple fact can no longer be ignored. Call them acceptable use policies, fair use policies or whatever, usage accounting is coming to DOCSIS networks everywhere. However, the downside of not accounting for subscriber usage accurately is at best a public relations challenge and at worst a legal issue. Cogeco and other first-movers are no doubt still smarting from the tongue lashing they received in both the blogosphere and more mainstream media when their usage reporting proved inaccurate. As we ve seen throughout this whitepaper, enabling IPDR in an operator network is more complicated than just buying an IPDR Collector and turning on IPDR at the CMTS. To truly be able to take advantage of the power of IPDR and to accurately account for subscriber usage requires a comprehensive design that extends all the way from the DOCSISlayer to the back-office accounting systems using this data. www.appliedbroadband.com 11

About the Author Andrew W. Sundelin, Senior Architect Prior to Applied Broadband, Andrew was an early employee and Principal Architect at WildBlue Communications, a provider of high speed networking services over satellite. At WildBlue, he worked on adapting the DOCSIS MAC layer for satellite use and protocol optimization (e.g. HTTP and TCP) in a high delay environment. As an industry expert Andrew has worked extensively in network policy design, Quality of Service (QoS) implementation, capacity management, service quality assurance, and abuse mitigation systems and technologies. Andrew was a contributing author to the PacketCable Multimedia specification. Prior to these roles, Andrew was a system architect at CableLabs from 1996-1999. At CableLabs, Andrew lead the specification team that wrote DOCSIS 1.1 as well as leading both the DOCSIS 1.0 MAC and the PacketCable QoS groups. Andrew has served on the SCTE High-Speed-Data MAC working group, as technical advisor for the CableLabs DOCSIS Certification Board, and has worked within both the IETF and IEEE on topics related to both wired and wireless DOCSIS. Andrew holds a B.Sc. in Computer Science and Russian Studies from Brown University and has completed all the course work for an MS in Telecommunications at the University of Colorado, Boulder.. About the Applied Broadband Applied Broadband provides powerful tools and expertise for broadband service intelligence and network management in Cable networks. We specialize in software implementation of policy-based networking, multi-service networks, and subscriber management, and service monitoring. Our customers are the world s leading service providers of voice, video, and data in the cable, satellite, and wireless marketplaces. Applied Broadband is the leading provider of IPDR collection and mediation technology to the Cable industry. The Applied Broadband s Pipeline addresses Cable operators need to collect, process, normalize and distribute massive amounts of IPDR data for visibility into their networks, devices, services and subscribers overall experience. Pipeline provides a unified IPDR collection layer serving as an enabling foundation of visibility into the cable operators subscriber devices, services, and networks. The use cases for Pipeline information based on IPDR data within a cable service delivery network are numerous. Applied Broadband is uniquely positioned to help its customers remain agile in the increasingly competitive broadband marketplace. We have worked with several companies, both large and small, in developing and launching numerous voice, video, and data broadband services and technologies. Our commitment to research and development along with innovative and practical software engineering has helped us to evolve several components, methodologies and models to address different customer scenarios. These reduce the cost, time-to-market, and risk of deployment, while enhancing the end subscriber s service experience. Applied Broadband offers a depth of services across voice, video, and data technology in three key expertise domains: Network Engineering, Software Engineering, and Systems Architecture. Applied Broadband, Inc. 1909 Broadway, 2nd Floor Boulder, Colorado 80302 p 303.449.2033 f 303.449.0119 e sales@appliedbroadband.com www.appliedbroadband.com 12