Advanced

Bash Scripting
Guide
Advanced Bash-Scripting Guide

An in-depth exploration of the art of shell scripting

Mendel Cooper

<[Link]@[Link]>

10

10 Mar 2014

Revision History
Revision 6.5 05 Apr 2012 Revised by: mc
'TUNGSTENBERRY' release
Revision 6.6 27 Nov 2012 Revised by: mc
'YTTERBIUMBERRY' release
Revision 10 10 Mar 2014 Revised by: mc
'PUBLICDOMAIN' release

This tutorial assumes no previous knowledge of scripting or programming, yet progresses rapidly toward an
intermediate/advanced level of instruction . . . all the while sneaking in little nuggets of UNIX® wisdom and
lore. It serves as a textbook, a manual for self-study, and as a reference and source of knowledge on shell
scripting techniques. The exercises and heavily-commented examples invite active reader participation, under
the premise that the only way to really learn scripting is to write scripts.

This book is suitable for classroom use as a general introduction to programming concepts.

This document is herewith granted to the Public Domain. No copyright!

Dedication
For Anita, the source of all the magic
 Advanced Bash-Scripting Guide

Table of Contents
Chapter 1. Shell Programming!.........................................................................................................................1

Chapter 2. Starting Off With a Sha-Bang........................................................................................................3

2.1. Invoking the script............................................................................................................................6
2.2. Preliminary Exercises.......................................................................................................................6

Part 2. Basics.......................................................................................................................................................7

Chapter 3. Special Characters...........................................................................................................................8

Chapter 4. Introduction to Variables and Parameters..................................................................................30

4.1. Variable Substitution......................................................................................................................30
4.2. Variable Assignment.......................................................................................................................33
4.3. Bash Variables Are Untyped..........................................................................................................34
4.4. Special Variable Types...................................................................................................................35

Chapter 5. Quoting...........................................................................................................................................41
5.1. Quoting Variables...........................................................................................................................41
5.2. Escaping..........................................................................................................................................43

Chapter 6. Exit and Exit Status.......................................................................................................................51

Chapter 7. Tests................................................................................................................................................54
7.1. Test Constructs...............................................................................................................................54
7.2. File test operators............................................................................................................................62
7.3. Other Comparison Operators..........................................................................................................65
7.4. Nested if/then Condition Tests.......................................................................................................70
7.5. Testing Your Knowledge of Tests..................................................................................................71

Chapter 8. Operations and Related Topics....................................................................................................72

8.1. Operators.........................................................................................................................................72
8.2. Numerical Constants.......................................................................................................................78
8.3. The Double-Parentheses Construct.................................................................................................80
8.4. Operator Precedence.......................................................................................................................81

Part 3. Beyond the Basics.................................................................................................................................84

Chapter 9. Another Look at Variables...........................................................................................................85

9.1. Internal Variables............................................................................................................................85
9.2. Typing variables: declare or typeset.............................................................................................104
9.2.1. Another use for declare.......................................................................................................107
9.3. $RANDOM: generate random integer..........................................................................................107

Chapter 10. Manipulating Variables.............................................................................................................119

10.1. Manipulating Strings...................................................................................................................119
10.1.1. Manipulating strings using awk........................................................................................127
10.1.2. Further Reference..............................................................................................................127
10.2. Parameter Substitution................................................................................................................128

i
 Advanced Bash-Scripting Guide

Table of Contents
Chapter 11. Loops and Branches..................................................................................................................138
11.1. Loops..........................................................................................................................................138
11.2. Nested Loops..............................................................................................................................152
11.3. Loop Control...............................................................................................................................153
11.4. Testing and Branching................................................................................................................156

Chapter 12. Command Substitution.............................................................................................................165

Chapter 13. Arithmetic Expansion................................................................................................................171

Chapter 14. Recess Time................................................................................................................................172

Part 4. Commands..........................................................................................................................................173

Chapter 15. Internal Commands and Builtins.............................................................................................181

15.1. Job Control Commands..............................................................................................................210

Chapter 16. External Filters, Programs and Commands...........................................................................215

16.1. Basic Commands........................................................................................................................215
16.2. Complex Commands...................................................................................................................221
16.3. Time / Date Commands..............................................................................................................231
16.4. Text Processing Commands........................................................................................................235
16.5. File and Archiving Commands...................................................................................................258
16.6. Communications Commands......................................................................................................276
16.7. Terminal Control Commands.....................................................................................................291
16.8. Math Commands.........................................................................................................................292
16.9. Miscellaneous Commands..........................................................................................................303

Chapter 17. System and Administrative Commands..................................................................................318

17.1. Analyzing a System Script..........................................................................................................349

Part 5. Advanced Topics.................................................................................................................................351

Chapter 18. Regular Expressions..................................................................................................................353

18.1. A Brief Introduction to Regular Expressions..............................................................................353
18.2. Globbing.....................................................................................................................................357

Chapter 19. Here Documents.........................................................................................................................359

19.1. Here Strings................................................................................................................................369

Chapter 20. I/O Redirection...........................................................................................................................373

20.1. Using exec...................................................................................................................................376
20.2. Redirecting Code Blocks............................................................................................................379
20.3. Applications................................................................................................................................384

Chapter 21. Subshells.....................................................................................................................................386

ii
 Advanced Bash-Scripting Guide

Table of Contents
Chapter 22. Restricted Shells.........................................................................................................................391

Chapter 23. Process Substitution...................................................................................................................393

Chapter 24. Functions....................................................................................................................................398

24.1. Complex Functions and Function Complexities.........................................................................402
24.2. Local Variables...........................................................................................................................413
24.2.1. Local variables and recursion............................................................................................414
24.3. Recursion Without Local Variables............................................................................................417

Chapter 25. Aliases.........................................................................................................................................420

Chapter 26. List Constructs...........................................................................................................................423

Chapter 27. Arrays.........................................................................................................................................427

Chapter 28. Indirect References....................................................................................................................456

Chapter 29. /dev and /proc.............................................................................................................................460

29.1. /dev..............................................................................................................................................460
29.2. /proc............................................................................................................................................463

Chapter 30. Network Programming.............................................................................................................469

Chapter 31. Of Zeros and Nulls.....................................................................................................................472

Chapter 32. Debugging...................................................................................................................................476

Chapter 33. Options........................................................................................................................................487

Chapter 34. Gotchas.......................................................................................................................................490

Chapter 35. Scripting With Style..................................................................................................................499

35.1. Unofficial Shell Scripting Stylesheet..........................................................................................499

Chapter 36. Miscellany...................................................................................................................................502

36.1. Interactive and non-interactive shells and scripts.......................................................................502
36.2. Shell Wrappers............................................................................................................................503
36.3. Tests and Comparisons: Alternatives..........................................................................................509
36.4. Recursion: a script calling itself..................................................................................................509
36.5. "Colorizing" Scripts....................................................................................................................512
36.6. Optimizations..............................................................................................................................525
36.7. Assorted Tips..............................................................................................................................528
36.7.1. Ideas for more powerful scripts.........................................................................................528
36.7.2. Widgets..............................................................................................................................539
36.8. Security Issues............................................................................................................................541
36.8.1. Infected Shell Scripts.........................................................................................................541
36.8.2. Hiding Shell Script Source................................................................................................541

iii
 Advanced Bash-Scripting Guide

Table of Contents
Chapter 36. Miscellany
36.8.3. Writing Secure Shell Scripts.............................................................................................541
36.9. Portability Issues.........................................................................................................................541
36.9.1. A Test Suite.......................................................................................................................542
36.10. Shell Scripting Under Windows...............................................................................................543

Chapter 37. Bash, versions 2, 3, and 4..........................................................................................................544

37.1. Bash, version 2............................................................................................................................544
37.2. Bash, version 3............................................................................................................................548
37.2.1. Bash, version 3.1...............................................................................................................551
37.2.2. Bash, version 3.2...............................................................................................................552
37.3. Bash, version 4............................................................................................................................552
37.3.1. Bash, version 4.1...............................................................................................................559
37.3.2. Bash, version 4.2...............................................................................................................560

Chapter 38. Endnotes.....................................................................................................................................564

38.1. Author's Note..............................................................................................................................564
38.2. About the Author........................................................................................................................564
38.3. Where to Go For Help.................................................................................................................565
38.4. Tools Used to Produce This Book..............................................................................................565
38.4.1. Hardware...........................................................................................................................565
38.4.2. Software and Printware.....................................................................................................565
38.5. Credits.........................................................................................................................................566
38.6. Disclaimer...................................................................................................................................567

Bibliography....................................................................................................................................................569

Appendix A. Contributed Scripts..................................................................................................................577

Appendix B. Reference Cards........................................................................................................................787

Appendix C. A Sed and Awk Micro-Primer................................................................................................792

C.1. Sed................................................................................................................................................792
C.2. Awk..............................................................................................................................................795

Appendix D. Parsing and Managing Pathnames.........................................................................................798

Appendix E. Exit Codes With Special Meanings.........................................................................................802

Appendix F. A Detailed Introduction to I/O and I/O Redirection.............................................................803

Appendix G. Command-Line Options..........................................................................................................805

G.1. Standard Command-Line Options...............................................................................................805
G.2. Bash Command-Line Options......................................................................................................806

Appendix H. Important Files.........................................................................................................................808

iv
 Advanced Bash-Scripting Guide

Table of Contents
Appendix I. Important System Directories..................................................................................................809

Appendix J. An Introduction to Programmable Completion.....................................................................811

Appendix K. Localization...............................................................................................................................814

Appendix L. History Commands...................................................................................................................818

Appendix M. Sample .bashrc and .bash_profile Files.................................................................................820

Appendix N. Converting DOS Batch Files to Shell Scripts.........................................................................837

Appendix O. Exercises....................................................................................................................................841
O.1. Analyzing Scripts.........................................................................................................................841
O.2. Writing Scripts.............................................................................................................................843

Appendix P. Revision History........................................................................................................................853

Appendix Q. Download and Mirror Sites.....................................................................................................856

Appendix R. To Do List..................................................................................................................................857

Appendix S. Copyright...................................................................................................................................858

Appendix T. ASCII Table..............................................................................................................................860

Index....................................................................................................................................................862
Notes..............................................................................................................................................899