Cloud Cruiser and Microsoft Azure Usage API Integration

Transcription

1 Cloud Cruiser and Microsoft Azure Usage API Integration In this article: Introduction Microsoft Azure Tags Creating a Resource Group with tags on Microsoft Azure Import data from the Usage API into Cloud Cruiser Processing the tag information from the Usage API Adding the tag information to the consumption data Introduction Microsoft recently introduced the Azure Usage API, allowing subscribers to programmatically pull in usage data to gain insights into their consumption. This is great news for Cloud Cruiser customers that can take advantage of the richer dataset available through this API. Cloud Cruiser can leverage the integration with the Usage API in several ways. The granularity (hourly usage information) and resource metadata information available through the API provides the necessary dataset to support flexible Showback or Chargeback models. In this tutorial, we will present one example of how Cloud Cruiser can benefit from the Usage API information. More specifically, we will create a new Resource Group on Azure, associate tags for the account structure, and then describe the process of pulling and processing the tag information into Cloud Cruiser. The final goal is to be able to create reports like the one below, and be able to analyze cost and consumption based on the account structure populated by the tags.

2 Figure 1 - Report with breakdowns using tags Microsoft Azure Tags The data available through the Azure Usage API includes not only consumption information, but also resource metadata including any tags associated with it. Tags provide an easy way to organize your resources, but in order to be effective, you need to ensure that: - Tags are correctly applied to the resources at provision time; - Tags are properly used on the Showback/Chargeback process to tie the usage to the organization s account structure. Both of these requirements can be challenging, especially when there is some sort of manual process on the provision or charging side. Mistyped, incorrect or even missing tags are common complaints from customers when using tags and these errors can make life on the charging side very difficult. With the new Azure Usage API, Cloud Cruiser can pull resource tagging information, and through a very sophisticated ETL tool called workbooks, fix these common tagging errors. Through transformation steps leveraging regular expressions and data correlation, Cloud Cruiser is able to identify incorrectly tagged

3 resources and apply the correct tags, filling the gaps and ensuring the correct association of the resources with the consumer. On the charging side, Cloud Cruiser automates the Showback/Chargeback process, and can leverage the tag information to tie the usage to the appropriate consumer (Department, Division, Project, etc.). This automation provides a huge improvement and can ensure a consistent and auditable charging process. Creating a Resource Group with tags on Microsoft Azure The first step in this tutorial is to create a new Resource Group on the Azure Portal and then create new tags to associate to the resources. For this example, we will be creating the following tags: Department Environment Owner Project The screenshot below of the Azure Portal shows a sample Resource Group with the associated tags. Figure 2 - Resource Group with associated tags on Azure Portal

4 The next step is to pull the information from the Usage API into Cloud Cruiser. The Usage API currently provides responses in JSON format. Here is a sample of the data retrieved: { "id": "/subscriptions/bb678b04-0e48-4b44-xxxx- XXXXXXX/providers/Microsoft.Commerce/UsageAggregates/Daily_BRSDT_ _0000", "name": "Daily_BRSDT_ _0000", "type": "Microsoft.Commerce/UsageAggregate", "properties": { "subscriptionid": "bb678b04-0e48-4b44-xxxx-xxxxxxxxx", "usagestarttime": " T00:00:00+00:00", "usageendtime": " T00:00:00+00:00", "metername": "Compute Hours", "meterregion": "", "metercategory": "Virtual Machines", "metersubcategory": "Standard_D1 VM (Non-Windows)", "unit": "Hours", "instancedata": "{\"Microsoft.Resources\":{\"resourceUri\":\"/subscriptions/bb678b04-0e48-4b44-XXXX- XXXXXXXX/resourceGroups/DEMOUSAGEAPI/providers/Microsoft.Compute/virtualMachines/MyDockerVM\",\"locati on\":\"eastus\",\"tags\":{\"department\":\"sales\",\"project\":\"demo Usage API\",\"Environment\":\"Test\",\"Owner\":\"RSE\"},\"additionalInfo\":{\"ImageType\":\"Canonical\",\"ServiceType\":\"Sta ndard_d1\"}}}", "meterid": "e60caf26-9ba0-413d-a f58081d6", "infofields": {}, "quantity": 8 } },

5 Import data from the Usage API into Cloud Cruiser Cloud Cruiser workbooks provide an automated way to collect and process information from the Usage API. An ETL (extract-transform-load) workbook allows you to configure the collection, transformation, and publishing of data into the Cloud Cruiser database. Each workbook can have one or multiple collections. This allows you to correlate information from different sources to complement or augment the usage data. For this example, we will create a new sheet in the Azure template workbook (UsageAPI) and set a new collection to import information from the Usage API. Figure 3 - Usage API data imported into the UsageAPI sheet Notice that this workbook already has other sheets to import services from Azure (ImportServices), and process the consumption information from the Billing API (PublishData). We are going to extract and process the information from the Usage API on the UsageAPI sheet, and correlate the information with the consumption data from the Billing API on the PublishData sheet.

6 Processing the tag information from the Usage API After importing the data into the workbook, we will create transformation steps in the UsageAPI sheet in order to process the information from the API. First step is to use a JSON split processor to extract the tags from a single field (as they are imported from the API) and create new fields for each one of them (Department, Project, Owner and Environment). Figure 4 - Create new fields for the tag information Notice that the Networking service is missing the tag information (yellow box), but we can tell that this service is part of the same Resource Group by looking at the ResourceGroupName field. Since we have the tags for the other resources from this same Resource Group, we can use this information to apply the missing tags to this resource later in the process. Next step is to create a lookup table associating the information from the tags to the ResourceGroupName. This lookup table will be used on the next step to enrich the consumption data with tag information.

7 Adding the tag information to the consumption data Now we can jump to the PublishData sheet, which processes the consumption information from the Billing API, and add the fields extracted from the tags. This process is performed by looking at the lookup table created on the previous step, using the ResourceGroupName as the key for the lookups. Figure 5 - Populating the account structure with the information from the lookups Notice that the appropriate account structure fields for the Networking service were applied, fixing the issue with the missing tags. We also populated the account structure fields for resources other than our target Resource Group with Other, in order to differentiate them on the reports. Now we just need to add another step to publish the usage data. During this step, the appropriate rates for each service defined on our Rate Plan will be applied to the usage information, and the resulting charge is then loaded into the database. The best part is that you only have to go through this process once. When the workbook is completed, you just need to add it to the scheduler and it will run hourly or daily at the scheduled time. Then it s just a matter of creating new reports, or customizing existing ones, in order to visualize and analyze the data to get meaningful insights from your cloud usage.

8 Next Steps For more information about Cloud Cruiser, please contact For detailed instructions on creating Cloud Cruiser workbooks and reports, please reference Cloud Cruiser s online documentation (valid login required). To learn how to integrate with the Azure Rate Card API, refer to the online tutorial Cloud Cruiser and Azure Public Rate Card API Integration