Small Python Tools for Software Release Engineering

Small Python Tools for Software Release Engineering

• 1.
  Small Python Toolsfor Software Release Engineering Scott Wang a.k.a. lunastorm
• 2.
  Self Introduction • 學C++起家 •偶爾慣Ｃ • 結果工作都用Java • 常常偷懶只寫shell script • Python??
• 3.
  講一朵開發雲的故事
• 4.
  Release Engineering • “asub-discipline in software engineering concerned with the compilation, assembly, and delivery of source code into finished products or other software components.” -- Wikipedia • 對我來說 • 把code寫好放到production上跑
• 5.
  Version Control 出build 寫code 測試
• 6.
  米國 慢。。。 台灣
• 7.
  $$$$$ http://aws.amazon.com/
• 9.
  Local LAB installed withXenServer
• 10.
  XenCenter
• 11.
  開VM 砍VM 關VM 複製 搬VM VM XenCenter
• 12.
  Yum
 Repo Local LAB installed with XenServer
• 13.
  Admin 順利交接
• 14.
  路遙知馬力 日久見人辛 Admin becomes the bottleneck!
• 15.
  外商
 外商
 菜比八
 小Leader
• 16.
  包RPM試試 東西做好了
• 17.
  注意 尛 dependency
• 18.
  找個乾淨機 Production 器裝看看 VM嗎?
• 19.
  對 找誰要?
• 20.
  Admin 好
• 21.
  Admin 開會⋯
• 22.
  我包的RPM 等了一天終 你測一下 於搞定
• 23.
  要再開一台 對 VM?
• 24.
  Admin 教召⋯
• 25.
  I WANT MY F**KINGVM!!!
• 26.
  什麼不是雲端運算?
• 27.
  “Cloud Computing” Definition by NIST • On-demand self-service • A consumer can unilaterally provision computing capabilities, such as server time and network storage, as needed automatically without requiring human interaction with each service’s provider.
• 28.
  “Cloud Computing” Definition by NIST • Rapid elasticity • Capabilities can be rapidly and elastically provisioned, in some cases automatically, to scale rapidly outward and inward commensurate with demand.
• 29.
  「真男人 / 女人不需 要桌面環境！！」 Howto automate this?
• 30.
  Xen Management API •Java and Python binding • Using Java binding • 要compile，麻煩 • Using Python binding • Trial and error in the interpreter first (勝)
• 31.
  http://docs.vmd.citrix.com/XenServer/6.0.0/1.0/en_gb/api/
• 32.
  Give Me VM! •Objective • Create a temporary VM for testing by self service • Login into it automatically • Destroy it when the testing is finished
• 49.
  X
• 50.
  準備作業 • Install aPara-Virtualized VM • Install Xen Tools in the VM • Will report IP via XenBus to XenServer • Convert the VM to a template
• 51.
  Report
 IP https://community.emc.com/servlet/JiveServlet/showImage/38-3466-30315/Xen.png
• 52.
  動作分解 • 第一動 • importXenAPI • XenAPI.py can be downloaded from XenServer SDKs • Actually a XML-RPC wrapper inside
• 53.
  第二動 • Create XenAPIsession • session = XenAPI.Session("http://master") • session.xenapi.login_with_password("user name", "password")
• 54.
  第三動 • Create aVM from a template • template = session.xenapi.VM.get_by_name_label(vm_label) [0] • name = "spot-" + str(time.time()).replace(".","") • new = session.xenapi.VM.clone(template, name)
• 55.
  第四動 • Provision andstart VM • session.xenapi.VM.provision(new) • session.xenapi.VM.start(new, False, False)
• 56.
  Waiting for IP retry_count= 0
 while retry_count < MAX_RETRIES:
 try:
 retry_count = retry_count + 1
 metric = session.xenapi.VM_guest_metrics.
 get_record(session.xenapi.VM.get_record
 (new)['guest_metrics'])
 ip = metric['networks']['0/ip']
 break
 except:
 print "Waiting for IP information..."
 time.sleep(5)
• 57.
  Waiting for SSHd retry_count= 0
 while retry_count < MAX_RETRIES:
 try:
 retry_count = retry_count + 1
 sock = socket.socket(socket.AF_INET, 
 socket.SOCK_STREAM);
 sock.connect((ip, 22))
 sock.close()
 break
 except:
 print "Waiting for sshd to come up..."
 time.sleep(5)
• 58.
  Time to Login! Returns after logout os.system("ssh -i spot_key -o UserKnownHostsFile=/ dev/null -o StrictHostKeyChecking=no root@" + ip)
• 59.
  Garbage Collection session.xenapi.VM.hard_shutdown(new)
 for vbdin session.xenapi.VM.get_record(new)['VBDs']:
 if session.xenapi.VBD.get_record(vbd)['type'] == 'Disk':
 vdi = session.xenapi.VBD.get_record(vbd)['VDI']
 session.xenapi.VBD.destroy(vbd)
 session.xenapi.VDI.destroy(vdi)
 else:
 session.xenapi.VBD.destroy(vbd) session.xenapi.VM.destroy(new)
• 60.
  Admin 山不轉，路轉
• 61.
  Evolution! by @jeffhung IMAGE_TYPES= [ { 'key': 'spn-centos53', 'name': 'CentOS 5.3 (Production VM)', 'label': 'SPN- Production-VM-CentOS-5.3-spot' }, { 'key': 'spn-centos62', 'name': 'CentOS 6.2 (Production VM)', 'label': 'SPN- Production-VM-CentOS-6.2-spot' }, { 'key': 'lucid', 'name': 'Ubuntu 10.04 (Lucid)', 'label': 'Ubuntu-10.04- spot' }, { 'key': 'myspn', 'name': 'MySPN Dev VM (CentOS 6.2)', 'label': 'tw- MySPN-devvm' }, ]
• 62.
  Supports Non- interactive Mode parser = optparse.OptionParser(description="Give me a temporary VM that volatile when I'm done.") parser.add_option('-l', dest='list', action='store_true', help='list available VM image types') parser.add_option('-t', dest='type', help='VM image type') parser.add_option('-f', dest='file', action='append', help='Preload file to VM instance in / folder') parser.add_option('-i', dest='init', help='Script for initialize VM instance, default to init.sh if file exist') parser.add_option('-I', dest='interactive', action='store_true', help='Run interactively, default enabled if -i not specified')
• 63.
  Automatically Running Scripts print 'Preloading file to newly created VM instance: ', file
 
 os.system("scp -i spot_key -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no " + file + " root@" + ip + ":/")
 
 os.system("ssh -i spot_key -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no root@" + ip + " “ + file)
• 64.
  Automatic Daily Regression Test • CI System triggers daily build job • Daily build artifacts will be sent to Yum repository • Trigger regression test job • Automatically creates a new VM • Execute the test scripts • Destroy the VM
• 65.
  Live Demo 不打假球
• 66.
  References • XenServer SDKs • http://community.citrix.com/display/xs/ Download+SDKs • XenAPI Documentation • http://docs.vmd.citrix.com/XenServer/ 6.0.0/1.0/en_gb/api/ • http://downloads.xen.org/Wiki/XenAPI/ xenapi-1.0.6.pdf
• 67.
  Thank You! Questions?
• 68.
  # ./give_me_vm.py AvailableImage Types:
 1) spn-centos53 : CentOS 5.3 (Production VM)
 2) spn-centos62 : CentOS 6.2 (Production VM)
 3) lucid : Ubuntu 10.04 (Lucid)
 4) myspn : MySPN Dev VM (CentOS 6.2)
 Please choose one of the above: 3 Using image type: Ubuntu-10.04-spot
 Will preload file to VM instance: init.sh
 Creating VM spot-133826046314 from Ubuntu-10.04-spot... Done!
 Provisioning VM... Done!
 Starting VM... Done!
 Waiting for IP information...
 Waiting for IP information... IP obtained: 10.1.112.84 Preloading file to newly created VM instance: init.sh
 Warning: Permanently added '10.1.112.84' (RSA) to the list of known hosts.
 init.sh 100% 443 0.4KB/s 00:00 
 Done!

• 69.
  Running init.sh...
 Warning: Permanentlyadded '10.1.112.84' (RSA) to the list of known hosts.
 Running init.sh hello!
 Done!
 Opening SSH connection...
 Warning: Permanently added '10.1.112.84' (RSA) to the list of known hosts. Linux localhost 2.6.32-33-server #70-Ubuntu SMP Thu Jul 7 22:28:30 UTC 2011 x86_64 GNU/Linux
 Ubuntu 10.04.3 LTS
 
 Welcome to the Ubuntu Server!
 * Documentation: http://www.ubuntu.com/server/doc
 
 System information as of Tue May 29 11:02:21 CST 2012
 
 System load: 0.55 Processes: 86
 Usage of /: 11.0% of 7.23GB Users logged in: 0
 Memory usage: 9% IP address for eth0: 10.1.112.84
 Swap usage: 0%
 
 Graph this data and manage this system at https://landscape.canonical.com/
 
 15 packages can be updated.
 9 updates are security updates.
 
 Last login: Fri Feb 24 18:19:24 2012 from 10.1.112.190
 root@localhost:~#