Renderbot Tutorial. Intro to AWS

Transcription

1 Renderbot Tutorial Thanks for choosing to render your Blender projects in the cloud using Renderbot. This guide will introduce Amazon AWS, walk you through the setup process, and help you render your first project in the cloud. Intro to AWS Renderbot allows you to create a personal render farm and that makes sure that you keep full control of your project files and allows you to render your projects affordably. Everything you do with Renderbot will happen within your personal Amazon Web Services account. Amazon Web Services is a group of cloud computing services run by Amazon.com. Many sites you use every day, including Netflix, Expedia, Airbnb, and Amazon.com itself are hosted on AWS. AWS is composed of many services, and caters to nearly all aspects of web hosting and management. You, however, will only be using two of these services: S3, the Simple Storage Service, is a cloud file storage service where you will place your Blender projects to be rendered and retrieve your finished images.

2 EC2, the Elastic Compute Cloud, is a cloud computing resource where you will launch Renderbot, which in turn will allow you to create a render farm in the cloud. How Renderbot Works Renderbot follows the basic scheme of a traditional render farm. You will first start and access one computer, the Renderbot Client. This system will provide you with an interface from which you can launch various Renderbot Nodes. These are the slave machines that actually complete the render jobs. AWS uses a certain vocabulary for these components: An instance is a computer running in the cloud. The Renderbot Client is an instance, so are all of your Renderbot Nodes. Instances run within Amazon EC2. An image is a sort of snapshot of an EC2 instance. Renderbot provides you with readymade snapshots for the Client and the Nodes. An image can be launched on an instance. An instance type is a certain computer configuration that can be launched within EC2. Just like a home computer, different types of instances have different processing power, different graphics cards, and different prices. An instance can be started or stopped much in the same way as your home computer. A stopped instance retains all of its data, just like your computer s files remain intact after a shutdown. You pay only for the storage used by a stopped instance, not for the instance hardware. To terminate an instance is to completely end its operation. All data is lost when an instance is terminated. An on-demand instance is launched per these hourly prices: Instances launched on-demand are guaranteed to be available 24/7. If you pay those prices, your instances will start immediately and will not be terminated or stopped unless you expressly perform these actions.

3 A spot instance is launched via a different mechanism: the spot market. The spot market, like a stock market, prices instances based on supply and demand. Prices are generally significantly lower than for on-demand instances. However, spot instances can be terminated based on price movements. A spot market bid, like a bid on Ebay, is the maximum price you are willing to pay for one or more instances. If the price you offer is greater than the current market price, your instances will launch within a few minutes of your bid. You will always pay the market price, even if your bid is higher. If the market price goes above your bid price, your instances will be terminated. You can also bid below market price, in which case your instances will start when prices drop below that level. If the market price moves above your bid, you will not be charged for the billing hour during which your instances are terminated. The AWS free tier is an introductory program that allows you a certain level of AWS services free of charge for 12 months following the creation of your account. IAM refers to Identity Access Management, the scheme by which AWS handles permissions and security. SQS refers to another AWS service, the Simple Queue Service. This service houses the list of tasks that you create via Renderbot. You will only deal indirectly with this service through the Renderbot interface. An S3 Client is a program that allows easier and faster access to S3. Good clients include Cyberduck and S3 Browser. These make uploading and downloading files faster and easier thanks to multipart upload and other features. The use of these clients is fully compatible with Renderbot, but their use is not covered in this tutorial. Renderbot itself uses several terms with which you should be familiar. A work queue is the list of the tasks you want the system to complete. The Renderbot Client is an AWS image (a readymade snapshot which can be launched on EC2) which provides the interface from which you will access Renderbot. This computer will be launched on a t2.micro instance, which is often free due to free tier benefits. You should stop (not terminate) this instance whenever you are finished rendering a project. You can then start it up again whenever you need Renderbot.

4 Renderbot Node is another image, this one launched by the Renderbot Client on whatever instances you specify to complete your rendering tasks. Renderbot Nodes will terminate themselves when your tasks are complete, unless you specify otherwise. The system is designed to tolerate the disruptions that are inherent in the spot market, in that the Renderbot Nodes return tasks they cannot complete (if they are terminated) to the work queue for future completion. Initial Setup Head over to AWS (aws.amazon.com) and create a free account. You will need a valid credit card number to pay for the rendering services you use, as well as a phone number to complete a verification step. Select the Basic (Free) support plan. You, as a first time user, will qualify for the Free Tier that will make Renderbot even cheaper. With your account created, you should be able to log in and see the full list of AWS services. You must select the N. Virginia (also called US Standard or useast-1) region at the top left. While S3 as a service is global, be sure to create your buckets within this region to avoid data transfer charges.

5 Whenever you are on AWS, you can return to this home screen by clicking the orange box on the upper left. Take a moment to locate the S3 and EC2 links (under Compute and Storage). These will be the services you use. You have now completed your initial AWS setup. You will shortly be able to start rendering in the cloud.

6 Handling Permissions You are going to need to give your render farm access to your project and allow it to launch instances on your behalf. Amazon handles permissions through pairs formed of one Access Key and one Secret Key. Go over to the Security Credentials manager as shown. When prompted, click Get Started with IAM Users.

7 Click Create New Users. You do not need to fill in all five names. Fill one in and click Create.

8 Download the CSV file containing your new user s credentials and save it somewhere where you will be able to find it. Then click Close. As of now, your new user has no permissions. Click the username to change that. Now click Attach Policy.

9 Check Administrator Access and then click Attach Policy. You should now be able to see the policy you attached under Managed Policies.

10 You have now successfully configured an IAM user for Renderbot. Be sure to remember where you saved your access/secret keys. Click the orange box in the upper left hand corner of the screen to go to the homescreen.

11 Uploading Your Project You should now upload your project to Amazon s Simple Storage Service in order for Renderbot to be able to access it. Click over to S3 and open your project in Blender. Be sure to make sure all your settings are final. Renderbot executes precisely the settings you save to your file. If you want Blender to use Cycles, set it to Cycles. Make sure your render output settings are accurate. Renderbot will render your files in PNG format. IMPORTANT: If you are planning on using subframe rendering (dividing frames into pieces for the render farm to render, as opposed to rendering each frame as a unit), be sure to set your render output settings (located near the output format in the Properties panel, bottom right in this picture) to RGBA. The alpha channel is important to allow easy recombining of split frames.

12 You now should pack all external resources into the.blend file as shown below. This will allow Renderbot to access all the textures/images/resources you use in your project.

13 Your Blender file is now ready for upload to AWS S3. Head over there now. You will probably be greeted by a Welcome screen that encourages you to create a bucket. Create two buckets (similar to folders). One will be used to host the blender project that you will be rendering. The other will contain the rendered images that Renderbot uploads. Create these buckets within, as usual, the US Standard AKA us-east-1 AKA N. Virginia region. Bucket names must be globally unique, so don t expect extremely generic names (e.g. images ) to be available. You will now want to upload your project to the bucket you have created to hold your input file. You can upload your project as a straight.blend, or can archive it as a.zip or.tar.gz file if it is large. Click on the bucket you created to hold your Blender file.

14 Click Upload. Click Add Files, select your prepared Blender file and press Start Upload. When the upload is complete, your file has been uploaded and is ready for Renderbot to render it. Click the box in the upper left corner to go home.

15 Launching Renderbot We will now create a key pair for you to use to access the Renderbot Client. Go to the EC2 page (available from the home screen), click on Key Pairs in the column on the left. Create a key by clicking the blue button shown below. Name the key whatever you want. Save the resulting.pem file where you will be able to find it.

16 Go to the AWS Marketplace Sign in if you are not already logged in. Search for Renderbot and select the latest version of the Renderbot NODE. Click the large orange Continue button. Accept the terms to subscribe to this software.

17 Search for Renderbot and select the latest version of the Renderbot CLIENT. Click the large orange Continue button. Please note that you only need to launch the Renderbot Client, NOT the Renderbot Node. The Client will launch the render nodes for you. You want to launch the Client on a very small instance, at this time a t2.micro is ideal. Select the key pair that you just created and saved. Then hit Launch with 1 Click.

18 Head back to the EC2 manager. Click the Instances tab. You should now see your instance running in the Instance manager. This will be the brain of your Renderbot render cluster. When you are finished rendering your project and have terminated all your render nodes, right click and stop (do NOT terminate) the instance in order to save on EC2 fees. When the instance is stopped, you will only be charged for storage (~$0.80 cents per month). Start the instance whenever you want to use your personal render farm. This will cost you about one cent an hour.

19 Connecting to your Instance You will need to connect to your Renderbot instance via SSH. This should not be a problem if you are on Linux/Mac, as Unix includes a built-in SSH client easily accessible through the terminal. If you are on a Unix system (Linux/Mac), simply use the following command in the Terminal to connect to your instance: ssh i [path to the key file we just downloaded] ubuntu@yourinstancepublicip For example: ssh i ~/Downloads/mykey.pem ubuntu@ On Unix, press ENTER followed by ~ (tilde) followed by. (period) to quit Renderbot. That IP address can be found here:

20 If, however, you are on Windows, we recommend Putty: Download putty.exe and puttygen.exe in order to handle all of your Windows SSH needs. These would be within the green area on the website. Open puttygen.exe and load the key file you downloaded from Amazon. Save it as a private key file. You will need to set your explorer window to read All Files, as the Amazon key is not in Putty format yet. You can add a password if you wish, which will be required every time you connect to your instance.

21 Now launch putty.exe. Head over to the AWS EC2 management console and copy your instance s public IP to Putty.

22

23 Make this change to the Putty Data settings;

24 And now, for the last Putty setting, add the path of your key file (.ppk) that you created with puttygen.exe.

25 Be sure to save your settings to make sure you never have to do this configuration again. You can now load these settings easily every time you launch Putty by clicking your settings name and hitting Load. Every time you use Renderbot, you will need to grab the Public IP from the EC2 instance management page, as it changes every time you start the instance.

26 You can now click Open at the bottom of the screen and connect to your instance. Add the server to your registry. Provide the password that you provided when you created the key (if any) and the login ubuntu. You are now connected to Renderbot. With the one-time setup now complete, your project will be rendered very soon. Using Renderbot You should now be able to see the Renderbot homescreen. With the setup behind us, you will have to deal mostly with this simple interface in the future for your rendering activities.

27 The following guide explains the various functionalities that are possible with Renderbot. You do not need be doing all of the various steps. In fact you should not. The only activity you really must do right at the start is to input all of the Setup information. Press 1 to enter the Setup dialog. Tip: right click in Putty to paste. Press 1 to configure your settings. Press Enter to move down. And press 1 to edit again.

28 o AWS Access Key: The Access Key for the IAM User we created earlier. This is in the CSV file we downloaded earlier when we created the user. o AWS Secret Key: Secret Key for the IAM User. Again in the CSV file. o Blender Project: S3 path to your project. E.g. s3://mybucket/project.blend o Output Location: the S3 path to where you want your render saved. o SQS work queue: Any name you want to identify your task list. o Default instance type: the default type of server you want Renderbot to launch ( o When finished: Renderbot can either terminate your render nodes ( shutdown ), or poll the task list for new tasks ( poll ). o AWS region: MUST be us-east-1 Be sure to maintain the format of the addresses. E.g. s3://, sqs://, s3:// Pressing 2 allows you to change your settings mid-render. This is useful if you want to render two projects on the same farm without restarting any instances. Pressing 3 takes you back to the homescreen. Press 2 to check instance prices. AWS allows you to launch several types of instances ( For rendering purposes, you will want to stick with Compute and GPU instances. (cx.xxlarge or gx.xlarge). o Press 1 to input an instance type, press Enter and Renderbot will show you the latest market (bid) price for that instance. AWS also allows you to request instances on-demand ( o 2 refreshes prices. o 3 takes you home.

29 Press 3 to add Renderbot tasks. o Press 1 to add whole frames. Just add the frame range you want rendered. o Press 2 to add subframes, specifying horizontal and vertical subdivisions. o 3 to refresh remaining tasks o 4 to reset tasks to 0. o Press 5 to go home.

30 Finally, press 4 to start Renderbot instances o 1 creates spot requests. o 2 launches on-demand render nodes at on-demand prices but without risk of termination (see intro for details.) o 3 cancels all requests o 4 stops all running instances

31 o 5 allows you to reduce the number of running instances to a specified number. You can tell Renderbot to only stop instances that have gone more than X minutes into the current billing hour. o 6 opens a spot request viewer. o 7 opens a running instance viewer. o 8 allows you to go home. Once you have completed the setup, you should probably go about rendering your project by: Setting the project path to your project file. Adding the tasks you want completed Checking the spot prices for the instance types you want to be using.

32 Retrieving Renders Downloading your renders is probably the easiest part of using Renderbot. You can either grab them from S3, accessible through the AWS Console (console.aws.amazon.com), or can use a special S3 client to streamline the process (Cyberduck, S3 Browser, etc ). The use of these clients is not covered in this tutorial. If you used subframe rendering (dividing each frame into pieces to be rendered), be sure to put your frames back together by importing all of the pieces of each frame into a single image file using your favorite image editing program (GIMP, Paint.net, Photoshop ). You will probably want to import each subframe as its own layer. Because you enabled the alpha channel, the pieces will merge into a single image for each frame you rendered. Conclusion And there you have it. You can now check prices, add tasks, and bid on instances with Renderbot. Launch instances to render your project, keeping EC2 instance limits in mind (

33 When you are done using Renderbot, Stop (do NOT Terminate ) your client instance via the EC2 management page. Your settings will be preserved and you will only be billed for the instance storage (pennies per month). Start your instance whenever you want and connect to it with Putty, loading the settings you created and updating the IP with your Client instance s Public IP available in the EC2 manager. Thank you for choosing Renderbot.