Troubleshooting Puppet

Troubleshooting Puppet

• 1.
  Troubleshooting Puppet ThomasUphill thomas@narrabilis.com
• 2.
  Latest version http://goo.gl/b2NISc If you see something, say something! Google Comments enabled
• 3.
  Watch in PresentationMode! Seriously, Trust Me ...Animations Are Good things
• 4.
  Me PuppetConf 2013 Mastering Puppet / Puppet Cookbook 4* PUGS /r/dadjokes
• 5.
  Outline network restapi certificates catalog problem workers compiling printing scope
• 7.
  Failures - can'tfind puppet - can't connect to puppet - can't get certificate - can't get/compile a catalog - can't apply a catalog - can't upgrade puppet
• 8.
  can't connect topuppet...
• 9.
  can't find puppet puppet puppet.domain [main/agent] server ca_server DNS nsswitch.conf /etc/hosts
• 10.
  can't connect topuppet node puppet master network pixies 8140 masterport
• 11.
  can't connect topuppet ● ping ● mtr (--port 8140) ● netcat (nc)
• 12.
  can't connect topuppet $ ping puppet ping: unknown host puppet $ mtr puppet My traceroute [v0.75] cookbook.example.com (0.0.0.0) Sat Oct 25 01:26:31 2014 Keys: Help Display mode Restart statistics Order of fields quit $ nc -v puppet 8140 Packets Pings Ncat: Version 6.45 ( http://nmap.org/ncat ) Ncat: Connected to 192.168.122.100:8140. Host Loss% Snt Last Avg Best Wrst StDev 1. puppet.example.com 0.0% 157 0.7 0.5 0.2 1.6 0.2
• 13.
  can't get certificate ● already signed (clean) ● dates off - expired CA, expired cert puppet cert clean host.example.com rm /var/lib/puppet/ssl/*/hostname* openssl x509 -in cert.pem -text ntpq -p
• 14.
  can't get certificate ● don't know your own name ● basic unix permissions ● something else... root@puppet:~# sudo -iu puppet puppet@puppet:~$ cd /etc/puppet/environments/ -bash: cd: /etc/puppet/environments/: Permission denied SELinux
• 15.
  REST API
• 16.
  Rest API node puppet master 8140 GET https://puppet:8140/production/certificate/ca HTTP/1.1 -----BEGIN CERTIFICATE ----- MIIFXjCCA0agAwIBAgIBATANBgkqhkiG9w0BAqsFADAcMRowGAYDVQQDDBFQdXdb w FiyvryAxxuETs9KORMwoThDIMd4bHLDbqtAz0q0cJ7W/8w== -----END CERTIFICATE-----
• 17.
  Rest API ca_server/server environment GET https://puppet:8140/production/certificate/ca HTTP/1.1 ca_port/masterport resource key
• 18.
  Rest API GEThttps://puppet:8140/production/certificate/ca HTTP/1.1 resource key resource certificate file_metadata file_content node catalog
• 19.
  Rest API ●wget ● curl # diff puppet-ca.pem /var/lib/puppet/ssl/ca.pem # echo $? 0 # curl --insecure https://puppet:8140/production/certificate/ca >puppet-ca.pem % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 101 1923 101 1923 0 0 25506 0 --:--:-- --:--:-- --:--:-- 312k # openssl x509 -in puppet-ca.pem -fingerprint SHA1 Fingerprint=5A:C2:03:7B:40:44:2C:81:45:81:07:11:D3:AC:29:FB:A5:EC:E3:55
• 20.
  Rest API #curl --cacert puppet-ca.pem https://puppet:8140/production/certificate/lisa.example.com Not Found: Could not find certificate lisa.example.com # curl --cacert puppet-ca.pem https://puppet:8140/production/certificate/cookbook.example.com -----BEGIN CERTIFICATE----- MIIFcTCCA1mgAwIBAgIBCjANBgkqhkiG9w0BAQsFADAcMRowGAYDVQQDDBFQdXBw ZXQgQ0E6IHB1cHBldDAeFw0xNDA5MzAwNDM1MDdaFw0xOTA5MzAwNDM1MDdaMB8x # openssl x5.0.9. -in cookbook.pem -text … Issuer: CN=Puppet CA: puppet Validity Not Before: Sep 30 04:35:07 2014 GMT Not After : Sep 30 04:35:07 2019 GMT Subject: CN=cookbook.example.com …
• 21.
  Catalog # curl--cacert puppet-ca.pem --cert /var/lib/puppet/ssl/certs/cookbook.example.com.pem --key /var/lib/puppet/ssl/private_keys/cookbook.example.com.pem -H 'Accept: yaml' https://puppet:8140/production/catalog/cookbook.example.com > cookbook.yaml % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 26099 100 26099 0 0 17434 0 0:00:01 0:00:01 --:--:-- 18642
• 22.
  Tools (gnutls-cli oropenssl s_client) $ gnutls-cli --insecure --port 8140 puppet.example.com GET /production/certificate/ca HTTP/1.0 Accept: s HTTP/1.1 200 Server: nginx/1.6.1 Date: Thu, 23 Oct 2014 21:03:58 GMT Content-Type: text/plain; charset=UTF-8 Content-Length: 875 Connection: close X-Powered-By: Phusion Passenger (mod_rails/mod_rack) 2.2.11 Status: 200
• 23.
  can't get/compile a catalog
• 24.
  Problem workers node puppet worker 8140 puppet worker 8140 mod_proxy_balancer puppet master 8140 devel puppet worker 8140 problem production
• 25.
  problem/bugfixes branches gitbranches ● per user ● per problem ● per ticket
• 26.
  Problem workers puppet worker 8140 --logdest /var/log/puppet/problem.log --debug --profile logrotate
• 27.
  Compiling $ sudopuppet master Thu Oct 23 14:34:24 -0700 2014 Puppet (debug): Using settings: adding file resource 'bucketdir': 'File[/var/lib/puppet/bucket]{:loglevel=>:debug, :group=>"puppet", :ensure=>:directory, :links=>:follow, :owner=>"puppet", :backup=>false, :mode=>"750", :path=>"/var/lib/puppet/bucket"}' Thu Oct 23 14:34:24 -0700 2014 Puppet (debug): Using settings: adding file resource 'publickeydir': 'File[/var/lib/puppet/ssl/public_keys]{:loglevel=>:debug, :group=>"puppet", :ensure=>:directory, :links=>:follow, :owner=>"puppet", :backup=>false, :mode=>"755", :path=>"/var/lib/puppet/ssl/public_keys"}' Thu Oct 23 14:34:24 -0700 2014 Puppet (debug): Using settings: adding file resource 'plugindest': 'File[/var/lib/puppet/lib]{:loglevel=>:debug, :ensure=>:directory, :links=>:follow, :backup=>false, :path=>"/var/lib/puppet/lib"}' Thu Oct 23 14:34:24 -0700 2014 Puppet (debug): Using settings: adding file resource 'fileserverconfig': 'File[/etc/puppet/fileserver.conf]{:loglevel=>:debug, :ensure=>:file, :links=>:follow, :backup=>false, :path=>"/etc/puppet/fileserver.conf"}' --compile problem.example.com --debug --trace --logdest /tmp/problem.puppet.log --environment sandbox { "data": { "resources": [ { "title": "main", "exported": false, "tags": ["stage"], "type": "Stage", "parameters": { JSON "before": "Stage[post]",
• 28.
  Apply $ sudopuppet apply --debug --trace --environment sandbox code.pp JSON $date = "+%S" exec {'epoch': command => "echo $((`date $date` / 86400))", path => '/bin:/usr/bin:/sbin:/usr/sbin', } Info: Applying configuration version '1415729233' Debug: Exec[epoch](provider=posix): Executing 'echo $((`date +%S` / 86400))' Debug: Executing 'echo $((`date +%S` / 86400))' Notice: /Stage[main]/Main/Exec[epoch]/returns: executed successfully
• 29.
  Catalog YAML -read it JSON - jq
• 30.
  Catalog catalog classes.txt jq '.data.classes[]' < catalog
• 31.
  Catalog fails tocompile ● duplicate resource ● modulepath/bad module name fails to apply ● unpredicable exec ● bad/broken service ● bad/missing variable
• 32.
  fails to compile
• 33.
  duplicate resource separateinto subclass ( package {'httpd'} ) virtual resources ( @user, @package, @service) modulepath puppet config print modulepath root@puppet:~# puppet config print modulepath --environment production /etc/puppet/environments/production/public:/etc/puppet/environments/producti on/modules root@puppet:~# puppet config print modulepath --environment master /etc/puppet/modules:/usr/share/puppet/modules
• 34.
  fails to apply
• 35.
  Unpredictable exec #!/bin/bash echo $JAVA_HOME /home/javadev/.bashrc JAVA_HOME=/your/face
• 36.
  Broken Service serviceprovider hasstatus => true /sbin/service $service status /etc/init.d/$service status /usr/bin/systemctl is-active $service
• 37.
  Bad/Missing Variable $one= "1" file {"lisaone": Info: Caching catalog for node1.example.com Error: path Failed => "/to tmp/apply lisa$catalog: one", Cannot alias File[lisa1] to ["/ensure tmp/lisa1"] => 'directory', at /} etc/puppet/environments/production/manifests/site.pp:34; resource file {"lisa1": ["File", "/tmp/lisa1"] already declared at /etc/puppet/environments/production/manifests/site.pp:30 path => "/tmp/lisa1", ensure => 'file', }
• 38.
  Bad/Missing Variable lisa{'one': place => "/tmp/$LISA", type => "directory", Info: Caching catalog for node1.example.com Error: Failed to apply catalog: Cannot alias File[two] to ["/tmp"] at /etc/puppet/environments/production/modules/lisa/manifests/i nit.pp:5; resource ["File", "/tmp"] already declared at /etc/puppet/environments/production/modules/lisa/manifests/i nit.pp:5 } lisa {'two': place => "/tmp/$LISA", type => "file", } define lisa ($place,$type) { file {"$title": path => $place, ensure => $type, } }
• 39.
  Printing - Notify notify {"$variable": }
• 40.
  chaining notify {'something': }->exec{'thingthatfails': }->notify{'after': }
• 41.
  Checking exec{'before resolv.conf': command => '/usr/local/bin/puppet-debug before resolv.conf', require => Class['debug'] } -> file { '/etc/resolv.conf': source => template("dns/resolv.conf"), noop => true, } class debug { file {'puppet-debug': path => '/usr/local/bin/puppet-debug', source => 'puppet:///modules/debug/puppet-debug', mode => 0755, } }
• 42.
  Debug Script… justan example #!/bin/bash LOG=$(mktemp /tmp/puppet-debug.XXXXXX) echo Puppet Debug -- $@ -- $(date) | tee $LOG echo "-- Disk --" | tee -a $LOG df -h |tee -a $LOG df -i |tee -a $LOG echo "-- Mem --" | tee -a $LOG free | tee -a $LOG echo "-- Files --" | tee -a $LOG PUPPET=$(pgrep puppet) for proc in $PUPPET do lsof -p $proc |tee -a $LOG done Puppet Debug -- before resolv.conf -- Fri Oct 24 01:13:34 EDT 2014 -- Disk -- Filesystem Size Used Avail Use% Mounted on /dev/mapper/VolGroup-lv_root 6.7G 2.5G 3.9G 39% / tmpfs 246M 0 246M 0% /dev/shm /dev/vda1 485M 80M 380M 18% /boot Filesystem Inodes IUsed IFree IUse% Mounted on /dev/mapper/VolGroup-lv_root 440640 79253 361387 18% / tmpfs 62783 1 62782 1% /dev/shm /dev/vda1 128016 50 127966 1% /boot -- Mem -- total used free shared buffers cached Mem: 502268 415488 86780 0 22176 172036 -/+ buffers/cache: 221276 280992 Swap: 835580 0 835580 -- Files -- COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME puppet 2058 root cwd DIR 253,0 4096 14 /root puppet 2058 root rtd DIR 253,0 4096 2 / puppet 2058 root txt REG 253,0 10600 36617 /usr/bin/ruby puppet 2058 root mem REG 253,0 156928 4134 /lib64/ld-2.12.so puppet 2058 root mem REG 253,0 1926680 6282 /lib64/libc-2.12.so
• 43.
  Printing - Template - scope.to_hash - reject a few file { "/tmp/puppet-debug.txt": content => inline_template("<% vars = - sort scope.to_hash.reject { |k,v| !( k.is_a?(String) && v.is_a?(String) ) }; vars.sort.each do |k,v| %><%= k %>=<%= v %>n<% end %>"), - print, one per line } vars = scope.to_hash.reject { |k,v| !( k.is_a?(String) && v.is_a?(String) ) }; vars.sort.each do |k,v| k=vn end
• 44.
  _timestamp=2014-10-23 22:29:52 -0700 architecture=x86_64 augeasversion=1.0.0 bios_release_date=01/01/2011 bios_vendor=Bochs bios_version=Bochs blockdevice_vda_size=8589934592 blockdevice_vda_vendor=6900 blockdevices=vda caller_module_name= clientcert=cookbook.example.com clientnoop=false clientversion=3.7.1 concat_basedir=/var/lib/puppet/concat domain=example.com environment=production facterversion=2.2.0 filesystems=ext4,iso9660 fqdn=cookbook.example.com gid=root hardwareisa=x86_64 hardwaremodel=x86_64 hostname=cookbook id=root interfaces=eth0,lo Printing - Template - scope.to_hash - reject a few file { "/tmp/puppet-debug.txt": content => inline_template("<% vars = - sort scope.to_hash.reject { |k,v| !( k.is_a?(String) && v.is_a?(String) ) }; vars.sort.each do |k,v| %><%= k %>=<%= v %>n<% end %>"), - print, one per line } vars = scope.to_hash.reject { |k,v| !( k.is_a?(String) && v.is_a?(String) ) }; vars.sort.each do |k,v| k=vn end
• 45.
  Scope The scene: class ntp { include ntp::server ● roles and profiles ● ntp server class role::ntp { include ntp } }
• 46.
  Scope The solution: class ntp { include ntp::server ● fully scope everything ● remember scope class role::ntp { include ::ntp } }
• 47.
  Summary learn somenetworking remember the REST api read up on SSL / x509 use --trace make a debug class remember scope
• 48.
  Questions/Comments? http://goo.gl/b2NISc http://ramblings.narrabilis.com