Don t Break the Glass! In case of emergency

Transcription

1 Don t Break the Glass! In case of emergency

2 Intro Apache CloudStack committer, PMC member Security Guy Founder of Stratosec hacker, #CCCNA14

3 Goal Provide bread crumbs for you to gain expertise on CloudStack s data model so you can react quickly in an emergency situation.

4 TOC What Should Production Look Like? My Nightmare My response How to prepare How we re trying to make things better Homework

5 What Should Production Look Like? Briefly: Redundant management servers Redundant databases Redundant storage (primary and secondary) Redundant infrastructure Data backups encrypted Pro tip: store backups outside CloudStack.

6 One Painful February Day Made a change in control panel prior to giving customer access Control panel interpreted this as Delete the Customer. More precisely: Delete the Customer, VMs and all storage. Now.

7 Wait, What?? Took about 15 seconds to realize what happened Looking in CloudStack Management UI, saw VM shutting down, then being expunged.

8 Periodic Table

9 Emergency Steps Shut down ACS management server immediately Fire up your favorite SQL tool (I recommend Squirrel SQL)

10 Immediately Establish Contact Establish Contact With Data Shrine With your Data Recovery Shrine of Choice

11 Know how CloudStack Works Steps to delete a VM: Shut down Destroy Expunge volumes Expunge VM

12 The SQL API DB Tables: vm_instance volumes nics user account Common Fields: account_id instance_id created/updated/removed state removed BACK UP DB BEFORE MAKING CHANGES!

13 Actual SQL

14 Make change in SQL.

15 Bonus: changing VM s IP UPDATE user_ip_address SET allocated=(select allocated FROM (SELECT allocated, public_ip_address FROM user_ip_address) AS oldip WHERE public_ip_address=' ') WHERE public_ip_address=' '; UPDATE user_ip_address SET account_id=(select account_id FROM (SELECT account_id, public_ip_address FROM user_ip_address) AS oldip WHERE public_ip_address=' ') WHERE public_ip_address=' '; UPDATE user_ip_address SET domain_id=(select domain_id FROM (SELECT allocated, public_ip_address FROM user_ip_address) AS oldip WHERE public_ip_address=' ') WHERE public_ip_address=' '; UPDATE user_ip_address SET state='allocated' WHERE public_ip_address=' '; UPDATE user_ip_address SET network_id=(select network_id FROM (SELECT network_id, public_ip_address FROM user_ip_address) AS oldip WHERE public_ip_address=' ') WHERE public_ip_address=' '; UPDATE nics SET ip4_address=' ' WHERE instance_id=1797; UPDATE vm_instance SET private_ip_address=' ' WHERE instance_name='i VM';

16 Move VM between accounts UPDATE user_ip_address SET account_id=10 WHERE id=29; UPDATE user_ip_address SET domain_id=8 WHERE id=29; UPDATE user_ip_address SET network_id=217 WHERE id=29; UPDATE vm_network_map SET network_id=217 WHERE vm_id=144; UPDATE vm_instance SET domain_id=8 WHERE id=29; UPDATE op_networks SET nics_count=nics_count+1 WHERE id=217; UPDATE op_networks SET mac_address_seq=mac_address_seq+1 WHERE id=217; UPDATE nics SET broadcast_uri='vlan://230' WHERE id=179; UPDATE nics SET network_id=217 WHERE id=179; UPDATE nics SET isolation_uri='vlan://230' WHERE id=179; UPDATE firewall_rules SET account_id=10 WHERE ip_address_id=29; UPDATE firewall_rules SET domain_id=8 WHERE ip_address_id=29; UPDATE firewall_rules SET network_id=217 WHERE ip_address_id=29;

17 How you can prepare Create written cloud policy, audit production to match Test your backups Become familiar with CloudStack internals Do you really need frequent data expunge processes?

18 Next: Production Lock Feature: Allow our customers to ensure that no automations will modify or delete their compute or storage instances Working for VM Adding for storage, networks Need to add to UI

19 Homework Pick a user VM in your Cloud. Via the database: Find the VM instance record Find the VM s nics in the database Find the VM s storage in the database Try deleting a VM via normal means, then before expunge, restore via SQL

20 kthxbye!