Defensive C++ Arduino Programming by

Michèle Delsol
Proper names, trademarks, and designations used by the author are
capitalized to distinguish them from ordinary text. They are the
property of their respective owners. The author and publisher of this
book have no intent at establishing any relationship whatsoever with
the owners of these names, trademarks, and designations.
The author and publisher have exercised due diligence as to the
exactitude of the book’s content and issue no explicit or implied
warranty of any kind as to the suitability of content presented for any
purposes whatsoever and assume no responsibility for errors or
omissions. The author and publisher assume no liability for incidental
or consequential damages resulting from the use of information,
code snippets, and programs presented in this book.
First published via Amazon August 2023.
Delsol, Michèle
Defensive C++ Arduino Programming / Michèle Delsol, first edition
Copyright © 2023 by Michèle Delsol
All rights reserved. Printed via Amazon "Print-on-demand" in the
United States and in other countries where Amazon distributes this
book. This book is protected by United States and international
copyright laws. No parts of this book may be copied in any form
whatsoever; permission must be obtained to reproduce parts and the
entirety of this book by any means whatsoever: electronic,
photocopying, mechanical, or other.
Indexing was undertaken via JavaScript scripts applied on manually
created tags. The indexing system was created by the author.
The warthog illustration on the front cover is an original pencil and
China ink drawing by the author.
E Kindle edition : ISBN 978-2-9585628-3-0
First edition published via Amazon August 2023.
This book is published via Amazon in paperback and electronic
(Kindle) editions.
 Table of contents
List of tables and figures x
Acknowledgements xi
Preface 1
Introduction 4
Chapter 1 Common sense and new tools 8
Chapter 2 Why choose Arduino 11
Chapter 3 The build toolchain 13
3.1 Intelligent C++ editor 14
3.2 Make 14
3.3 Preprocessor 15
3.4 Compiler 15
3.5 Linker 15
3.6 Uploader 15
3.7 Bootloader 15
3.8 Serial terminal 16
Chapter 4 Hardware setup 17
4.1 Breadboards 18
4.2 Prototype board 19
4.3 Final PCB 19
Chapter 5 Interoperability 21
5.1 Directory structure 21
5.2 IDE specific entry files 22
5.3 Render setup/loop independent of code changes 24
5.4 Single-level directory structure interoperability scenario 24
5.5 Two-level directory structure interoperability scenario 25
Chapter 6 Which IDE to work with? 27
6.1 Dedicated IDEs 27
6.2 Plugins 27
Chapter 7 Arduino IDE 30
7.1 Arduino distribution 31
7.2 Arduino Basic features 32
7.3 Arduino IDE V2 specifics 34
7.4 Arduino caveats (legacy version 1.8.19) 36
Chapter 8 AtmelStudio 37
8.1 Why adopt it for your Arduino projects 37
8.2 Missing features 39
8.3 Managing directories 39
8.4 Importing an Arduino project (AtmelStudio) 40
8.5 Upload code into the microcontroller (create external tool) 44
8.6 Interface with Arduino applications 48
8.7 Editor features 49
8.7.1 Workspace 51
8.7.2 Intelligent window scrollbars 52
8.7.3 Quick access to functions 53
8.7.4 Go to implementation 53
8.7.5 Inline function related tasks 53
8.7.6 Color coding 54
8.7.7 Inline syntax checking 55
8.7.8 Find/Replace 55
8.7.9 Name completion 56
8.7.10 Code collapsing and indentation 57
8.7.11 Collapse /*...*/ group 59
8.7.12 Group comment/uncomment 59
8.7.13 Refactoring 59
8.7.14 Compiler error reporting 60
8.7.15 Code navigation 61
8.7.16 Bookmarks 61
8.7.17 Spell-checker 61
8.7.18 Go to line number 61
8.8 AtmelStudio hardware-based debugging 62
8.9 AtmelStudio caveats 62
8.9.1 Workspace font size 65
8.9.2 Application code size (AtmelStudio) 65
8.9.3 Failure to upload code 66
8.9.4 Code changes 67
8.9.5 Second set of header files import 68
8.9.6 Quick access to functions pulldown selection box failure 68
8.9.7 Improper indenting 69
8.9.8 Failed code formatting 69
8.9.9 Segmentation faults 70
8.9.10 Errors not listed in the error list 71
8.9.11 Make utility errors not listed 72
8.9.12 Inexistant file in project not listed in error messages window 73
8.9.13 Search and replace 73
8.9.14 AtmelStudio code formatting and code collapsing bug 74
8.9.15 Auto format if...else problem 74
8.9.16 Refusal to uncollapse a section of code 75
8.9.17 Unable to generate code for new microcontroller 75
8.10 Keyboard shortcuts 77
8.11 Documentation 78
Chapter 9 Visual Studio 79
9.1 Visual Studio features 79
9.2 Visual Studio download 80
9.3 Visual Studio 2019 vs. Visual Studio 2022 80
9.4 Visual Studio for Arduino developers 81
9.5 Visual Studio (Arduino project template) 82
9.6 Visual Studio (Visual Micro) 82
9.7 Visual Studio (debugging) 83
Chapter 10 Visual Micro 84
10.1 Visual Micro for AtmelStudio (MicrochipStudio) 84
10.2 Visual Micro for Visual Studio 85
Chapter 11 VS Code 86
11.1 VS Code features 87
11.2 VS Code installation 87
11.3 VS Code navigation 88
11.4 VS Code configuration 89
11.5 VS Code - folder vs. project 91
11.6 VS Code extensions 91
11.6.1 Arduino CLI 92
11.6.2 VS Code Arduino extension 93
11.6.3 VS Code C++ extension 94
11.6.4 VS Code serial terminal extension 95
11.6.5 VS Code PlatformIO extension 95
11.6.6 VS Code Awk extensions 96
11.6.7 VS Code Perl extensions 96
11.6.8 VS Code regular expressions (regex) extensions 96
11.7 .json files 96
11.8 VS Code caveats 97
Chapter 12 PlatformIO 98
12.1 PlatformIO Installation 100
12.2 PlatformIO features 101
12.3 PlatformIO managing projects 102
12.4 PlatformIO creating projects 103
12.5 PlatformIO project configuration ([Link]) 106
12.6 PlatformIO interoperability 107
12.7 PlatformIO hardware-based debugging 108
12.8 PlatformIO help and documentation 108
12.9 PlatformIO gotchas and caveats 108
Chapter 13 Other IDEs (Code::Blocks and
MPLAB) 111
13.1 Code::Blocks 111
13.2 MPLAB 112
Chapter 14 Debugging 113
14.1 Avoiding bugs 114
14.2 Types of problems 114
14.2.1 Glitches 114
14.2.2 Code thrashing 115
14.2.3 Misdoings 117
14.3 Print-based debugging 118
14.4 Hardware-based debugging 119
14.4.1 ATmega328P-Xmini 120
14.4.2 ATmega2560RFR2 Xplained Pro 121
14.4.3 Atmel ICE 122
14.5 Serial debugging 125
Chapter 15 Good programming practices 129
15.1 Be consistent 132
15.2 Naming conventions 132
15.3 Code formatting 133
15.3.1 Readability 133
15.3.2 Indentation 133
15.3.3 Collapsing 135
15.4 Think 135
15.5 Document code 136
15.6 Plan your work offline 137
15.7 Never assume anything 137
15.8 Error handling 138
15.9 Project organization 139
15.10 Monitor memory use 139
15.11 Careful with lenient compiler type checking 140
15.12 Abide by the KISS principle 140
15.13 Task wrap-up phase 141
15.14 Mental condition 143
15.15 Use C++ macros 143
15.16 Define constants only once 144
15.17 Use enum lists 145
15.18 Parameter default initialization 146
15.19 Initialize using curly braces 146
15.20 Comment closing curly braces and #endif 147
15.21 Use the auto-indent feature of the editor 147
15.22 Always start with code skeletons 147
15.23 Exploit C++ features sparingly 147
15.24 Do your homework 148
Chapter 16 Frameworks 149
16.1 Organizational Frameworks 151
16.1.1 Project Files Framework 152
16.1.2 Program Documentation Framework 155
16.1.3 Function Creation Framework 157
16.2 Data Handling Frameworks 158
16.2.1 Class Data Framework 159
16.2.2 Data Packets Framework 161
16.2.3 Format Driven float to byte Conversion Framework 162
16.2.4 DataGroup Framework 164
16.2.5 Bitfield Storage Framework 166
16.2.6 Event Storage Framework 171
16.2.7 Linked List Framework 172
16.3 Specialized Frameworks 173
16.3.1 Algorithm Test Framework 174
16.3.2 Class and Function Names Referencing Framework 176
16.3.3 Class and Function Names Referencing Framework
instrumentation 178
16.3.4 Memory Management Framework 179
16.3.5 Pseudo Exception Handling Framework 183
16.3.6 Error Reporting Framework 185
16.3.7 Operator overloading 186
16.3.8 Print-based Debugging Framework 188
Chapter 17 Should know tools 190
17.1 Regular expressions (regex) 191
17.1.1 Regex terminology 193
17.1.2 Regex primer 194
17.1.3 Metacharacters and literal characters 195
17.1.4 Regex tidbits 196
 17.1.5 Regex example - search for enums 197
17.1.6 Regex look ahead/behind 198
17.1.7 Regex groupings 198
17.1.8 Regex greediness 199
17.2 Awk vs. Perl 201
17.3 Awk 204
17.3.1 Awk terminology 206
17.3.2 Awk program structure 206
17.3.3 Awk conditions 209
17.3.4 Awk action-blocks 209
17.3.5 Awk rules 210
17.3.6 Awk BEGIN and END 214
17.3.7 Awk functions and variables 214
17.3.8 Awk vs. C 214
17.3.9 Awk gotchas 216
17.4 Perl 217
17.4.1 Perl terminology 218
17.4.2 Perl primer 219
17.4.3 Perl program structure 220
17.4.4 Perl quick overview 221
17.4.5 Simulate Awk using Perl 223
17.4.6 Perl vs. C 224
17.4.7 Perl caveats 224
Chapter 18 From back of the envelope to final
product 225
18.1 Tinkercad 226
18.2 Eagle 227
18.3 Fusion 360 227
18.4 3D printing 228
18.5 Fused deposition modeling FDM 229
Chapter 19 Appendix 231
Chapter 20 RAM requirements 232
20.1 Code size (IDE comparisons) 232
20.2 Code size (build options) 233
Chapter 21 DOS box (appendix) 237
21.1 DOS box - What is it? 237
21.2 DOS box launch 238
21.3 DOS commands 240
21.4 DOS batch files 241
21.5 DOS box - other info 242
Chapter 22 Frameworks (appendix) 243
22.1 Organizational Frameworks (appendix) 243
22.1.1 Project Files Framework (appendix) 243
22.1.2 Program Documentation Framework (appendix) 244
22.1.3 Function Creation Framework (appendix) 249
22.1.4 Class and Function Names Referencing Framework
instrumentation (appendix) 251
22.2 Data Handling Frameworks (appendix) 253
22.2.1 Class Data Framework (appendix) 254
22.2.2 Data Packets Framework (appendix) 257
22.2.3 Format Driven float to byte Conversion Framework (appendix) 258
22.2.4 Bitfield Storage Framework (appendix) 261
22.2.5 DataGroup Framework (appendix) 265
22.2.6 Event Storage Framework (appendix) 269
22.2.7 Linked List Framework (appendix) 270
22.3 Specialized Frameworks (appendix) 272
22.3.1 Algorithm Test Framework (appendix) 273
22.3.2 Class and Function Names Referencing Framework (appendix) 275
22.3.3 Memory Management Framework (appendix) 278
22.3.4 Pseudo Exception Handling Framework (appendix) 281
22.3.5 Error Reporting Framework (appendix) 284
22.3.6 Print-based Debugging Framework (appendix) 285
Chapter 23 Misdoings (appendix) 292
23.1 = instead of == in if/while 292
23.2 == instead of = in assignment 293
23.3 switch statement empty or missing default case 294
23.4 F() macro in [Link] missing 294
23.5 Work to do 294
Chapter 24 Memory structure (appendix) 295
24.1 Memory use macros 295
24.2 Determining memory usage 296
24.3 Highly fragmented heap 298
24.4 How do free and delete know how much memory to deallocate 299
Chapter 25 Awk (appendix) 300
25.1 Download Awk 300
25.2 Invoking Awk from a DOS box 300
25.3 Awk short example 302
25.4 Awk some features illustrated 303
25.4.1 Awk scope 303
25.4.2 Awk arrays 304
25.4.3 Awk reference 308
25.4.4 Awk command line 309
25.4.5 Awk comments 309
25.4.6 Awk rules (appendix) 309
25.4.7 Awk condition only and action-block only rules 310
25.4.8 Awk interrupt processing lines 311
25.4.9 Awk BEGIN and END (appendix) 311
25.4.10 Awk functions and variables (appendix) 311
25.4.11 Awk built-in variables 312
25.4.12 Awk program flow control 313
25.4.13 Awk operators 313
25.4.14 Awk strings 314
25.4.15 Awk print 314
25.4.16 Awk mathematical functions 315
Chapter 26 Perl (appendix) 317
26.1 Download Perl 318
26.2 Invoking Perl from DOS box 318
26.3 Perl simulate Awk 319
26.4 Perl file handling 320
26.5 Perl subroutines (functions) 321
26.6 Perl strings 321
26.7 Perl string interpolation - single or double quotes 322
26.8 Perl lists 322
26.9 Perl program flow control 323
26.10 Perl arrays 324
26.11 Perl multidimensional arrays 326
26.12 Perl pass by reference 328
26.13 Perl built-in variables 328
26.14 Perl string functions 329
26.15 Perl importing packages 329
26.15.1 Perl variable types 330
26.15.2 Perl scope resolution operator :: and my 330
26.16 Perl text match operator 330
Chapter 27 Bibliography 331
27.1 Bibliography - C/C++ programming 331
27.2 Bibliography - PROGMEM framework 333
27.3 Bibliography - Software Engineering 333
27.4 Bibliography - Regular Expressions (regex) 333
27.5 Bibliography - Awk 333
27.6 Bibliography - Perl 333
27.7 Bibliography - Arduino 334
27.8 Bibliography - AtmelStudio (now MicrochipStudio) 334
27.9 Bibliography - Visual Micro 335
27.10 Bibliography - PlatformIO 335
27.11 Bibliography - Espruino and JavaScript 335
27.12 Bibliography - Hardware-based debugging 335
27.13 Bibliography - Programming psychology 336
A note on the book's source code 337
About the author 338
Beehive weighing system 340
Index table 343
 List of tables and figures
Table 16.1 - Job class - <x>Tables[AUX]Bitfield Storage Framework - Table of bitfield based variables of Job class</x><x>Bitfield Storage
Framework[AUX]Job class inherits from the Bitfield class</x>Table of bitfield based variables. 167

Table 16.2 - <x>Memory (use)[AUX]Memory used as the application runs</x>Memory used as the application
runs. 180
Table 20.1 - RAM memory <x>Tables[AUX]RAM memory and flash memory requirements</x><x>Memory (use)[AUX]RAM memory and
flash memory requirements</x>and flash memory required by

the three IDEs to run the MemMgt application with the -flto option. 233
Figure 22.1 - Test Arduino-based beehive weighing system execution paths
with Algorithm Test Framework. 273
Table 25.1 - Weekly grocery sales 303
 Acknowledgements
This book, Defensive C++ Arduino Programming and its companion,
Pragmatic C++ Arduino Programming, are the result of chance
encounters which led me to beekeeping and to create Arduino-based
gadgets. Put the two together and, aha! why not create an Arduino-
based beehive weighing system. That is how it all got started. And
one thing leading to another, I got into writing two books which
address the needs of C++ savvy DIY Arduino makers.
I must thank the many who contributed to my getting started on
writing these books and continuing it to its ultimate conclusion.
And I thank Christine C., my psychoanalyst, whom I see regularly to
express my little travails. She has been an unconditional supporter of
my endeavor.
First in line is Daniel T. I am particularly grateful to him since he
made me discover Arduino. His electronics advice, despite his being
a practicing pediatrics surgeon, hence electronics not being his field
at all, contributed immensely towards getting me started with
Arduino.
And then there is Charlie H., a practicing physician and fellow
airplane builder (RV8). He introduced me to beekeeping - a few visits
to his bee yard and I was hooked.
There is of course Xavier M., who purchased my company years
back and who has since become a friend. His continued support has
contributed to my persevering in this book's endeavor.
My neighbors Martine and Olivier B. continuously supported my
endeavors. My special thanks to them. I must say that as the project
advanced from milestone to milestone, we celebrated by opening
one or two bottles of Champagne - by now, a few cases have gone
down our respective esophagus.
As luck would have it, Allison (Olivier's daughter) is an InDesign
professional consultant. I am thankful to her as she accepted to
create the print and digital ready document. She was patient as she
suffered through my unorthodox approach which consisted in
creating tags in Word which would be used by an InDesign
JavaScript script to produce the finished book (layout, cross
references, indices, table of contents, etc.).
I must also thank daughter #1 Giselle, also an author, for her
continued support.
Finally, but not least, I owe daughter #2 Pascale special thanks as
she patiently proofread both manuscripts, a total of four hundred plus
A4 pages of tight letter size text. Since I am an engineer, my thought
process, hence sentence construction, tends to be a bit linear
(somewhat tedious to read). She managed to smooth things out and
put some pep into many of my phrases.
And I thank all those others who manifested their support as they
patiently heard me out as I described my project.
 Preface
How did I come to write Pragmatic C++ Arduino Programming and
Defensive C++ Arduino Programming? <x>Pragmatic C++ Arduino Programming book[AUX]Explains why C++ can
lead to excessive debugging time and how to reduce it</x><x>Defensive Programming[AUX]Good programming practices, plan and implement procedures and frameworks, use

The journey
professional tools</x><x>Beehive weighing system[AUX]How the beehive weighing system got me into writing these two books</x>

started due to my being an amateur beekeeper. In order to check the

health of my hives, I weigh them regularly, weekly, or daily,
depending on the season. I use an ordinary luggage scale to lift the
rear of the hive. I naturally asked myself "How about hacking an
electronic Arduino-based beehive weighing system to replace my
trips to the beehives. Monitoring the weight provides an indication on
the beehive's health and how much honey is being produced. It also
helps determine whether the beehive has swarmed. Swarming
means that the old queen and half the bees leave the beehive; a 6
lbs. loss in 10 minutes.
<x>Good programming practices (GPP)[AUX]A hobbyist mindset leads to extensive debugging time</x>The back of the envelope

design of the Arduino beehive weighing system seemed simple.

Building such a system was the type of challenge I would enjoy. Little
did I know that the electronics would prove easy, while the software
to control the electronics would be a major endeavor which would
take two+ years to complete! As I progressed on my project, I was
forced into transitioning from a hobbyist mindset to a professional
programmer mindset. These two books are the concepts,
procedures, and tools which enabled me to complete the beehive
weighing system software – its size is now 35 files/15000 lines of
code.
<x>Microcontrollers[AUX]Choosing microcontroller is compromise; why Arduino?</x>I quickly defined the general
characteristics of the project: ordinary bathroom scale strain gauges
amplified by opamps, star shaped network radio communications,
clocks to synchronize the modules, a GSM board to send data to my
smartphone. The deciding factors on choosing components were
cost, availability, suitability, and ease of development. The
inexpensive Arduino board, its free easy to use development
environment, and C++ being the best language for producing
compact, fast, executables were the deciding factors. What's more, I
already had experience programming C++.
I then wrote code to try out the subsystems one at a time.
Throughout this early dabbling, I was impressed by the ease
provided by the Arduino IDE. After experimenting with the examples,
I started my beehive weighing system with one .ino file, then, slowly
but surely, the application grew. I added .h/.cpp files, created
classes, and defined macros.
I started the project by validating my choice of electronic
components and experimenting with each subsystem. And this is
where the fun began. Getting subsystems to work one at a time was
easy. Getting them to work together got to be particularly time
consuming. I confess that I had unknowingly embarked on a major
project with a hobbyist mindset. I soon spent much more time
debugging than coding. It got to be a painful experience.
As I progressed and the application was getting more complex, I was
at the mercy of poor planning, and I no longer controlled
development. I could not continue with my undisciplined hobbyist
approach hacking out an Arduino-based gadget by coding directly,
straight from mind to code. I had to modify my ways because the
application was getting unwieldy. It had reached a size and
complexity which required that I adopt formal design and
implementation techniques. If I were to bring my project to
completion, I had to step back, rethink my approach, and get
organized. Furthermore, I had to update my C++ knowhow since I
had become a little rusty and was making way too many mistakes.
<x>Good programming practices (GPP)[AUX]Efficient programming starts with good programming practices</x>As luck would have it, I

had already developed applications at a professional level and had

been project manager of a large industrial plant project, but that was
years ago. I thought back on that work and reviewed literature on the
subject. My past experience incited me, slowly but surely, to adopt
good programming practices and to structure development. This
proved to be difficult and challenging. It demanded hard work, but
the fun returned.
<x>Object-oriented programming (OOP)[AUX]Design your system the way you think - outside-in, aka top-down</x><x>Object-oriented programming (OOP)[AUX]Bottom-up

The foremost change in my approach was to formalize

design is fill in the details</x>

the parts and pieces of the project, define them as individual

modules, and lay them down on paper. Understanding the system's
structure from both the electronic and software standpoint led to
partitioning the project into modules. There were subsystems at the
electronic level (GSM, clock, radio, EEPROM) and there were
subsystems at the software level (data storage, data validation, data
conversion, data transmission, etc.). Thanks to C++'s object-oriented
features, I started the application's design top-down. The details
were to be filled-in bottom-up later on.
Informal solutions to manage certain tasks morphed into formal
frameworks. Getting these done required drawing flow charts,
creating algorithms, defining functionalities, components, and
procedures. Working offline on paper proved to be a critical first step.
This planning turned out to be vital to the application's robustness,
as well as to my own productivity. After a while, the application
began to take shape as pieces fell into place.
<x>Algorithm Test Framework[AUX]Two-pronged process: methodology (plan on paper and then code), and thinking (top down how one thinks and bottom up fill-in the details)

Development became a two-pronged process: design and

</x>

implement the big picture (i.e., top-down via classes and wrapper
functions) and design and implement low level helper functions and
functional frameworks - bottom up. The process was iterative: go to
the drawing board; work on a module; change things; go to the
computer; implement new code; test and examine results; then go
back to the drawing board. I did this until I got it right. Documenting
each step helped me learn and clarify concepts. Many hours later, I
had laid the groundwork and developed standards, procedures, and
frameworks. In the process of doing this, I documented code as
much as possible for future reference.
I took notes on the causes of the various bugs I was getting (array
overruns, forgetting the end null in char strings, etc.). Then I created
frameworks (file organization, register storage of small values,
extract documentation out of source files). I jotted down items of
interest as I progressed to hone my understanding of C/C++
features: adopt practical how-to procedures; create frameworks to
accomplish specific tasks; learn how to use AtmelStudio; use regular
expressions to facilitate searches; use a Perl program to produce
product documentation, one of many tasks Perl could help me with.
It then dawned on me that there were amateur Arduino developers,
not professional C/C++ programmers, who could benefit from this
work. I could write a pragmatic presentation of the C/C++ know-how
I had learned. This morphed into Pragmatic C++ Arduino
Programming. The defensive procedures and frameworks I had
developed became Defensive C++ Arduino Programming.
Writing these two books turned out to be as much of a challenge as
designing and creating my Arduino-based beehive weighing system.
Little did I suspect, when I started putting together the electronic
components of my system, that it would take me two years to get the
software done, and that I would write two books to convey these
newly acquired skills.
Knowhow, good practices, good tools, good frameworks are the keys
to success. I should add hard work to this list; success does not
come free. Adopting these concepts and tools will make you more
efficient: avoid C/C++ gotchas, manage memory, make the program
efficient (small and fast), facilitate debugging, understand what you
wrote months past because you commented your code correctly,
build in safeguards to protect the application as it unravels, modify
your code painlessly, etc.
 Introduction
I wrote this book with the intent that it should help you, a DIY C/C++
Arduino programmer, evolve from an amateur mindset condition into
a professional one, so that you could undertake large complex
projects. Time spent adopting good practices, learning how to use
professional tools, and implementing frameworks will be amply
compensated by less debugging time, much improved adaptability to
change, faster more compact code, more robust applications, and a
more enjoyable programming experience.

Productivity - Get
<x>Programming objectives[AUX]Enhance productivity, maintainability, robustness, compactness, and speed</x>

the project out faster with less pain. Programming should be

fun.
Maintainability - Modifications should be easy. Logic should be
clear and simple. Complexity should be reduced via
encapsulation and frameworks.
Robustness - The application should be free of crashes. This
means not using bad pointers, avoiding runaway memory
condition leading to stack overflow or heap exhaustion, and
staying clear of other such C/C++ mishaps. You will need to
hone your C++ skills. The companion book, which dwells
deeply into the why's and how's of C++, Pragmatic C++
Arduino Programming, could help you with this.
<x>Programming objectives[AUX]Monitoring code size important since premium on RAM although speed might be a

Compactness and speed - Monitoring code size is

requirement</x>

important because there is a premium on RAM. You may also

need to monitor application speed if this were to become
critical.

<x>Think[AUX]incremental programming, i.e. mind to keyboard can be a costly time wise trap</x><x>Defensive Programming[AUX]Good programming practices, plan and

Writing small code which fits in a single

implement procedures and frameworks, use professional tools</x>

sketch file differs from writing large code which needs to be spread
across files. The latter requires planning, workflow, tools, and a
corresponding mindset. If the application is small (one .ino sketch
file) you could hack it out directly - mind to keyboard: incremental
programming. However, if the application is large, you should adopt
good programming practices, plan and implement procedures and
frameworks, and use professional tools. I refer to this as defensive
programming. If you do not evolve from a hobbyist mindset to a
professional mindset, you will spend ages trying to get your
application to work, the danger being that you might never get it
finished.
This book is a compilation of lessons learned and frameworks I
developed through two years of experience developing my Arduino-
based beehive weighing system application (35 files/15000 lines of
code). It covers the Arduino IDE (1 and 2), AtmelStudio, Visual
Studio, Visual Micro, PlatformIO, VS Code, Code::Blocks, and
MPLAB (the last two superficially). It also covers debugging,
program organization, program documentation, frameworks, etc.
Whichever IDE you use, you can benefit from being disciplined as
you work on your project and use frameworks to manage various
functional domains.
The chapter Common sense and new tools (page 8) should be read
first. It presents a concise summary of the content of this book. The
chapters Good programming practices (page 129), Frameworks
(page 149) and Debugging (page 113) should be read next, they
build on each other. The remaining chapters and the Appendix (page
231) are self-explanatory. They can be read in random order. They
will most certainly influence your development choices.
The main chapters of this book are:

Chapter 1 - Common sense and new tools (page 8)

summarizes the main subjects covered in this book.
Chapter 2 - Why choose Arduino (page 11) - You chose Arduino
as your core hardware, as opposed to ESP32, RaspberryPi, or another
microcontroller. For my beehive weighing system, Arduino with
C++ was the best compromise.
Chapter 3 - The The build toolchain (page 13) - You have been
using the Arduino IDE since you first discovered Arduino. It is a
great tool. As this book is going to print, Arduino just released
version 2 of its IDE, an improvement over the one we have
been using (latest 1.8.19). Version 2 did manifest a few
problems; I feel that it is not solid enough to fully replace the
old version (referred to as legacy Arduino). I dedicated a section to
version 2 (Arduino IDE V2 specifics page 34) and covered version
1 more extensively (Arduino IDE page 30) .
Chapter 4 - Hardware setup (page 17) - Programming your
chip is one thing. Getting it to drive the electronics is another
thing. Initial experimentation is usually done on breadboards.
You ultimately reach a point where the jumble of wires proves
unsatisfactory. This is when it is time to get the soldering iron
out and migrate to prototype boards. When the application
runs properly and the design has been validated, go for your
own PCB with its own Atmel microcontroller. You may
ultimately develop a final professional grade SMT (surface-
mount technology) PCB.
Chapter 5 - Interoperability (page 21) - The Arduino IDE is a
fantastically easy to use tool. But, you will find that you will be
much more productive if you adopt a professional grade tool
such as AtmelStudio (page 37), Visual Studio (page 79), Visual
Micro (page 84), or PlatformIO (page 98). And there will be
times when you will want to switch to using one or the other on
a given project. Interoperability means seamlessly switching
from one tool to the other on the same set of files.
Chapter 6 - Which IDE to work with? (page 27) - We have all
started with the Arduino IDE and are probably still using it for
our daily Arduino work. But you would be considerably more
productive, and your application would be more robust, if you
were to adopt a professional grade tools such as AtmelStudio
(page 37), Visual Studio (page 79), or PlatformIO (page 98).
Chapter 7 - Arduino IDE (page 30) - You probably know the
Arduino IDE well. This chapter presents its strengths and
weaknesses.
Chapter 8 - AtmelStudio (page 37) - If you are serious about
programming Arduino, you owe it to yourself to look into how
much more you can do with AtmelStudio as compared with the
Arduino IDE, particularly when doing hardware-based
debugging.
Chapter 9 - Visual Studio (page 79) with Visual Micro (page
84) is practically the same as AtmelStudio with Visual Micro.
So, if you have switched from the Arduino IDE to AtmelStudio,
why not do your programming with Visual Studio/Visual Micro,
it is far more versatile and futureproofs your knowhow.
Chapter 10 - Visual Micro (page 84) - This plugin is available
for both AtmelStudio (MicrochipStudio) and Visual Studio. It
adds a serial monitor, an upload tool, serial-based debugging,
and more. As for Visual Studio, it renders Microsoft's flagship
IDE Arduino compatible.
Chapter 11 - VS Code (page 86) is a Microsoft foundation tool
which is designed to be enhanced via extensions. PlatformIO,
for example, is such an extension. It is well worth knowing how
to use it as it could meet a diversity of needs (Python
programming, Perl...).
Chapter 12 - PlatformIO (page 98), a VS Code extension, is a
formidable tool which could prove to be your choice if you
intend to migrate to ESP32 or other microcontrollers.
Chapter 13 - Other IDEs (Code::Blocks and MPLAB) (page
111) - Code::Blocks is a good free IDE for C++ development
but which, unfortunately, does not seem to be Arduino
compatible. MPLAB is a professional grade tool for general
Microchip microcontrollers embedded development.
Chapter 14 - Debugging (page 113) - Three debugging
techniques are presented: print-based (Print-based Debugging
Framework page 188), Hardware-based debugging (page
119), and Serial debugging (page 125) provided by Visual
Micro.
Chapter 15 - Good programming practices (page 129) -
Programming is a craft. A set of simple guidelines, easy to
implement but which require a little discipline, should
contribute to your improving your performance.
Chapter 16 - Frameworks (page 149) are application
independent modules designed to accomplish specific tasks
(data transfer, data storage, etc.). Taken together they
constitute a toolbox which enhances reuse, robustness, and
productivity.
Chapter 17 - Three Should know tools (page 190) are
presented: regular expressions, Awk, and Perl. Regular
expressions (regex) (page 191) is a tool which does text
searches using wildcards. Do yourself a favor, learn the
fundamentals of regular expressions. It will provide you with
unsuspected possibilities when doing text analyses, such as
search/replace, syntax analysis, etc. Awk (page 204) and Perl
(page 217) are text processing programming languages
designed to implement complete programs. Awk is basically
used for search and replace. Perl is Awk with extended
programming capabilities.
Chapter 18 - From back of the envelope to final product (page
225) - Once you have gotten the electronics to work, i.e. you
have created a soldered prototype of your gizmo, you may
have to do a lot more work such as creating a PCB, fabricating
enclosures, sensor supports, and battery holders, whatnot.
These require tools for creating PCB files (Gerber files with
Eagle page 227), a CAD tool to create 3D parts (Fusion 360
page 227), and get them 3D printed (Zortrax 3D printing page
228). You will find descriptions of the tools I use and of other
tools I did some research on.
Chapter 19 - The Appendix (page 231) expands on certain
subjects: application code size, DOS box, memory use,
frameworks, misdoings, Awk, and Perl.
Chapter 20 - The Bibliography (page 332) lists a few books which
might be helpful to deepen your C++ know-how. It also includes Web
links and YouTube videos.
A note on the book's source code (page 338) - This book and its
companion Pragmatic C++ Arduino Programming contain C++
code, C++ frameworks, and Awk and Perl programs. These
are provided on an as is basis, free for noncommercial use, no
 guarantees whatsoever as to quality and suitability, under an
MIT type Open-Source Licensing basis (go to [Link]
to download the code). The initial release is incomplete, I shall
improve on it progressively.
About the author (page 339) - I, the author, have led a rich
multi-national, multi-discipline life. My current interests are
beekeeping, flying the airplane I built (RV8), creating electronic
devices, 3D printing (Zortrax + Fusion 360), writing these
books and possibly more to come, and much more a retiree
indulges in.
Beehive weighing system (page 341) - This two-book set,
Pragmatic C++ Arduino Programming and Defensive C++
Arduino Programming, would not exist had I not embarked on
creating an Arduino-based beehive weighing system.
The Index table (page 344) was designed so that you could find in it
just about all of the book's content meant to be accessed by pertinent
keywords.

In a nutshell, the contents of these two books cover C++ knowhow,

good programming practices, IDEs and other tools, methodologies
and procedures, and frameworks to help you be more productive
when creating large, fast, compact, robust, and maintainable
Arduino-based C++ applications.
 1.
Common sense and new tools

Common sense and new tools

Developing the software to drive my Arduino-based beehive
weighing system turned out to be an unanticipated <x>Good programming practices (GPP)
[AUX]Common sense improves productivity</x><x>Golden rules[AUX]The right tool for the right job crucial</x>adventure. I developed

methodologies which I qualify as common sense and discovered

new tools (AtmelStudio, Visual Studio, Visual Micro, VS Code,
PlatformIO, Perl, regular expressions) which enabled me to be
considerably more productive.
What follows is a set of common-sense directives and advice on
tools to improve your programming and the application's quality.

Plan ahead - If a module gets a

<x>Think[AUX]Planning offline before coding can save loads of time</x>

little complex, step away from the computer for a spell. Sit
down at a table with pencil, paper, and eraser to draw flow
charts; list objectives, requirements, parameters. Brainstorm
the module, how it works, what it is meant to accomplish. This
phase is crucial. Assiduous attention to detail at this stage will
save you tons of time later on. Furthermore, when coding
directly on the computer, your hands being tied to the
keyboards and your eyes glued to the screen, you are subject
to tunnel vision and to getting lost in details. A large table with
paper and books spread out liberates your mind. You are given
free rein to enhance your creativity and imagination. You are
freer to express itself. You will do a better job.
<x>Object-oriented programming (OOP)[AUX]Design your system the way you think - outside-in, aka top-down</x><x>Object-oriented programming (OOP)

Top-down design, bottom-up details - Use

[AUX]Bottom-up design is fill in the details</x>

wrapper functions and C++ classes to define the big picture,

top-down or outside-in. Use functions to handle details bottom-
up or inside-out.
<x>AtmelStudio (editor)[AUX]Extensive user assist editor features</x>Use a smart C/C++ editor - Imagine
writing letters with Notepad as opposed to using Word. Word
will format text, correct spelling, do grammatical error checks.
The same holds with coding. A good programmer's editor will
color text based on syntax, indent code, do code completion,
correct and/or signal errors, and much more. It is important to
use the correct tool for the job. Notepad++ or the embedded
editors in AtmelStudio, Visual Studio/Visual Micro, or
PlatformIO are alternatives to the Arduino IDE.
<x>Arduino (build)[AUX]Arduino, AtmelStudio, PlatformIO, Visual Micro use the GNU C++ compiler</x><x>AtmelStudio (build)[AUX]Arduino, AtmelStudio,

A
Visual Studio/Visual Micro, PlatformIOuse the GNU C++ compiler</x><x>PlatformIO (build)[AUX]Uses Arduino distribution's GNU C++ compiler</x>

good C/C++ compiler will generate good code and will error
check or issue warnings. Fortunately, all four IDEs covered will
do this (Arduino IDE, AtmelStudio, Visual Studio with Visual
Micro, PlatformIO) since they incorporate the excellent GNU
compiler.
<x>Visual Micro[AUX]Improve productivity by orders of magnitude</x><x>Visual Studio[AUX]Improve productivity by orders of magnitude</x><x>AtmelStudio

Migrate to Visual Micro for Visual

(general)[AUX]Improve productivity by orders of magnitude</x>

Studio (page 85) - The Arduino IDE is great; without it, you
most probably would never have adopted Arduino. Visual
Studio/Visual Micro is more complete; the learning curve is
short; and the Arduino sketch creation process is easy and
works well. Your productivity and programming comfort will be
immensely improved. They provide several other important
features such as grammar checking as you type, goto
implementation, direct from error reporting to source code, and
more. Adopting them is a no brainer. What is more, it is
possible to establish interactivity between IDEs, switch from
one IDE to the another in less than 30 seconds, by activating
an IDE specific #define macro. You could also envision using
PlatformIO (page 98), but I find Visual Studio/Visual Micro
more user friendly. Program with Visual Studio/Visual Micro;
hardware-debug with AtmelStudio (see Hardware-based
debugging page 119).
<x>Good programming practices (GPP)[AUX]Common sense improves productivity</x>Adopt Good programming

practices (page 129) - There are resources (papers, videos,

seminars, etc.) which advise on procedures and good
programming practices (upper/lower-case, error checking,
naming conventions, file organization). The sheer quantity is
overwhelming and generally addresses professional
programming teams. Having looked at many of these, I
compiled a subset of simple rules and procedures which
should help you reduce errors and render programs more
readable: Adopt naming conventions, break big functions up
into several smaller functions thus easier to manage, comment
your code, etc.
<x>Frameworks[AUX]Do not reinvent the wheel - use frameworks</x> Frameworks (page 149) - Source

file organization, data storage, memory management, error

reporting, etc. are tasks an application needs to implement.
Frameworks respond to such needs; these ready-made tools
enhance the programmer's productivity and the application's
robustness.
<x>Memory Management Framework[AUX]Memory allocations verify heap availability and reveal memory leaks</x>Memory

management - Monitor RAM to avoid being out of heap or

stack space. Bring things out of scope early to release
memory. Optimize data storage. Whenever possible,
encapsulate code in curly braces. Managing memory is one of
the keys to robust, fast, compact, efficient code. See Memory
Management Framework (page 179) and Memory structure
(appendix) (page 296).
<x>Comments[AUX]Countless reasons for commenting code</x>Documenting your code is often an

afterthought. My Perl based Program Documentation

Framework (page 155) meets two objectives: document the
code in a highly organized way (this cannot be
overemphasized) and extract the corresponding
documentation. The golden rule being Comment! Comment!
Comment! Why not use a little discipline to structure these
comments so that Perl could extract program documentation?
<x>Misdoings[AUX]Dumb mistakes not detected by compiler; found by Perl [Link] program</x><x>Perl (general)[AUX]Perl

program looks for dumb mistakes: [Link]</x>Use Perl (page 217) as a static

checker of vicious bugs - Simple mistakes like = instead of == in

 an if statement, or the reverse in an assignment statement, can
remain unnoticed for ages. Attempting to discover a problem's
origin can eat up precious time. A Perl program will extract all
pertinent lines of code to enable you to peruse through them
thereby detecting potential problems. I used it on my 15000
lines application and found three such mistakes. It saved me
hours of debugging.
<x>Regex (general)[AUX]Regex engine is a generalized intelligent wildcard protocol to find/replace text</x> Regular expressions

(regex) (page 191) - If you are to use Perl, or profit from the
fact that professional grade IDEs (AtmelStudio, Visual Studio,
PlatformIO) search functions support regular expressions,
then, by all means, learn how to use regexes. They can get
cryptic however you only need a basic understanding to
quickly undertake tasks you would have otherwise thought
impossible. The learning curve is short.
<x>Debugging hardware-based[AUX]ATmega328P-Xmini Uno pin compatible supported by AtmelStudio, success using it</x><x>Debugging hardware-

based[AUX]ATmega328P-Xmini - watch variables to find glitches</x><x>Bugs (avoiding them)[AUX]ATmega328P-Xmini - watch variables to find glitches</x>

Hardware-based debugging (page 119) is a desirable activity

since it should save on debugging time. Hardware-based
debugging is based on interfacing the microcontroller with a
hardware tool which enables step-by-step program execution
(set breakpoints, display variable values). Atmel proposes
many tools, 3 of which seem to respond to Arduino hardware-
based debugging needs. These require that AtmelStudio be
used.

Serial debugging (page 125), provided by Visual Micro, is an

additional debugging method. It lies halfway between print-based
and hardware-based debugging. See Debugging (page 113).
 2.
Why choose Arduino

Why choose Arduino

This book addresses the needs of Arduino developers who embark
on major projects. By this I mean a project <x>Arduino (general)[AUX]Arduino tools are user-friendly and free,
components cheap, community huge</x><x>Which IDE to work with?[AUX]Arduino readily available, inexpensive, particularly easy to work with, required by other IDEs</x>

that
<x>Which IDE to work with?[AUX]GitHub search on Arduino yields more than 100,000 results</x><ALSO>Github[AUX]Why choose Arduino</ALSO>

contains subsystems which interact with each other: many modules,

many conceptual entities, many files. The question remains: Why
choose Arduino for a particular project? There are lots of other
microcontrollers. Why Arduino?
Most of us started using Arduino because it is readily available,
inexpensive, and particularly easy to work with. There are thousands
of developers, possibly millions, which explains why there are
libraries for just about every electronic component. A GitHub search
on Arduino yields more than 100,000 results. And, it is inexpensive,
but it carries a cost. Its main programming language is C++, a
professional grade language which can lead the inexperienced to
excessive debugging time. There is the alternative of using Python to
program Arduino, but this carries important costs, greater memory
requirements, slower application, and reduced programming
possibilities. But programming is easier and faster.
It is a sure thing that, as with most things in life, a choice is a
compromise. Certain criteria weigh more than others in a particular
context thereby tipping the scales. Here are reasons for my having
chosen Arduino and C++ for my Arduino-based beehive weighing
system project:

Cost - I developed my Arduino-based beehive weighing

system for personal use, cost was not an issue as long as it
remained reasonable. But, lurking in the back of my mind, if I
succeeded in developing a sufficiently low-priced useful
gadget, it could be turned into a money-making venture. Low
 component costs thereby became high priority. This implies the
smallest possible program to minimize microcontroller size,
thereby imposing C++ right away. All other development
languages take up considerably more space and are slower.
Hardware cost was another issue. My own PCB stand-alone
based Atmel microcontroller would be less expensive than
designs using board-based microcontrollers such as the
ESP٣٢ or RaspberryPi.
Development tools - Most cross-development toolchains are
costly from an Arduino developer's perspective. They are
reserved for professional teams, one main provider being the
Swedish company IAR Systems (check them out -
[Link] they propose a rich set of tools for just about
every microcontroller. Luckily, Arduino developers may use
one of several free development environments, the Arduino
IDE (page 30) developed and maintained by the Arduino
organization (htpp://[Link]), AtmelStudio (page 37), a
Microsoft Visual Studio specifically designed for Arduino
microcontrollers, PlatformIO (page 98), a plugin for Microsoft's
VS Code targets embedded applications on numerous
microcontrollers, including the ESP32. Finally, you may opt for
Visual Studio (Visual Micro) (page 82), an almost identical twin
of Visual Micro for AtmelStudio (MicrochipStudio) (page 84).

Other IDEs (Code::Blocks and MPLAB) (page 111) are either not
suited for Arduino development (Code::Blocks) or entail an
extensive learning curve (MPLAB).

Sensors - There are

<x>Sensors[AUX]Thousands of Arduino compatible sensors, boards, devices available</x>

thousands of Arduino compatible sensors, boards, and

devices. By Arduino compatible, I mean devices for which
someone has developed an Arduino interface in C/C++. This
alone can tilt the scale in favor of Arduino. There is no point in
choosing a particular microcontroller if the desired device
interface is not available.
Community - When a problem arises, it is comforting to rely on
 a community of savvy developers who can respond
knowledgeably to an issue. The Arduino community is huge.
Just about every imaginable question has been posed.

Arduino with C++ was an obvious choice. I could have chosen

RaspeberryPi because it is powerful, can be programmed in Python,
and sits on top of an operating system, which makes it a
minicomputer - keyboard, screen, and all. It is, however, more costly
since it comes as a board as opposed to Arduino whose Atmel
microcontrollers can be bought as stand-alone items.
 3.
The build toolchain

The build toolchain

There is a lot more to getting a microcontroller to do what you intend
it to do than downloading the Arduino IDE, <x>Build toolchain[AUX]C++ editor, make, preprocessor, compiler,
linker, upload (avrdude)</x>creating a program, followed by a click on the
build/upload button. This is what you would do when trying out the
Arduino IDE with the Blink program. But, if you are to create an
application which does substantial work, you will necessarily end up
with many functions, many classes, i.e., many lines of code in many
files. Upon attaining this level, you shall need to understand what is
happening behind the scenes to remain productive, i.e., what the
Arduino IDE is doing. You shall also need to look at debugging
options other than putting occasional prints in your source code. And
finally, you should examine professional grade IDE's; they will
enhance your productivity and their learning curve is usually not
steep.
This chapter covers the behind-the-scenes tools driven by the IDEs
(compiler, linker, etc.). The chapters which follow cover the IDEs
individually, starting with the Arduino IDE.
When you create a program with the Arduino IDE and click the
build/upload button, a great deal of work gets undertaken behind the
scenes. Understanding what is going on is essential to your being
able to figure out what is happening when things get a little awry.
The only tool you see is the Arduino C++ editor along with the IDE's
toolbar. The behind-the-scenes tools (compiler, linker, etc.) take your
C++ code and magically do what it takes for you to witness what
your electronic gizmo does.
The build toolchain was extensively covered in the companion book
Pragmatic C++ Arduino Programming. Here is a recap:

Intelligent C++ editor (next section) - Creating a program relies

on a good editor with which to write code. The source code is
then passed to the preprocessor to handle macros, if any.
Make (page 14) - C++ being multi-module, make handles
which modules the IDE is to work on depending on their
timestamp compared against the last build's timestamp. It
controls work so that it should only process files which have
been modified since the last build.
Preprocessor (page 15) - C++ offers programmers a unique
tool, the preprocessor. It enables doing text substitution,
conditional inclusions, and file assembly. These tasks are
controlled by macros which the preprocessor recognizes.
Please note that this is an extremely useful C++ feature not
available in most other mainstream languages although some
may have third-party preprocessors available as plugins.
Compiler (page 15) - Human readable programs (source code)
need to be converted into machine code for the target
microcontroller.
The Linker (page 15) assembles modules and library
components into an executable.
The Uploader (page 15) loads the executable into the
microprocessor so that it be stored inside one of its permanent
memory, flash memory in Arduino's case.
The Bootloader (page 15) is a program which resides in the
microprocessor; it converts it into being its own ISP (In System
Programmer). The ISP is electronic hardware whose role is to
load a program into a microcontroller. The bootloader makes it
so that the microcontroller becomes its own ISP.
Serial terminal (page 16) - If you want to send information to
the outside word, you can do so by displaying it on a screen
connected to the microcontroller via a serial port. Your
application might not require your doing so; but, during the
development phase, you might want to investigate what it is
doing. Your only recourse might be to print values as the
application runs; this is when a serial terminal comes in handy.
Hardware setup (page 17) - Developing your gizmo may start
out by piecemeal breadboard proof-of-concept work gradually
 evolving to a final SMT (surface mount technology). There are
different types of hardware you may use at each stage of the
development process (breadboard, prototype board...).
Debugging (page 113) - It is a fact of life that as you program,
you make mistakes; your program fails to work properly - it has
bugs. Finding bugs is what debugging is all about. You can do
so by placing prints to get to know what your program is doing;
or, you can use a hardware-based debugger to stop the
application as it runs, and look at the program's state.

These are summarily described below. Debugging (page 113) is

described in its own chapter.

1.
Intelligent C++ editor
<x>Build toolchain[AUX]Intelligent If you are to create a C++
C++ editor - AtmelStudio/Visual Studio best</x>

application for your gizmo, you must write the code into a file and
feed it to the build toolchain. The editor should do a lot more than
just indent code according to its scope level. It should color code
according to syntax; do name completion to help you avoid
misspelling already defined variables; functions, etc.; detect
programming errors from improperly written constructs, (missing
parentheses, missing items,...); report undefined entities...
Dedicated programmer editors such as Notepad++ and Arduino IDE
V1.8 are basic editors. Arduino IDE V2 begins to provide some
editing enhancements. AtmelStudio (page 37), Visual Studio (page
79), and PlatformIO (page 98) incorporate intelligent editors which
vastly improve programmer productivity.

2.
Make
 If your program is small, one or two
<x>Build toolchain[AUX]Make - timestamp defines what to work on</x>

small modules, you may resort to recompiling everything and create

the executable. But what if your program is large, 40 modules,
100,000 lines of code. You should only recompile modules which
have been modified since the last build, one or two instead of forty.
The IDE should create a make file which establishes which modules
to recompile and which modules to assemble to create an
executable. Fortunately, all IDEs covered in this book use make to
control the build process.

3.
Preprocessor
C++ has a unique feature
<x>Build toolchain[AUX]Preprocessor enables text replacement, conditional inclusions, pasting files</x>

referred to as macros. These are special variables which begin with

a # (pound sign) which the preprocessor recognizes. They enable
accomplishing three fundamental tasks:

Text substitution - If you write #define BUFFER_SIZE 256, the

preprocessor replaces every occurrence of BUFFER_SIZE with 256.
Conditional inclusions - You may want to differentiate targeting
an ATmega328P from an ATmega2560. You can do this by
placing code inside an conditional inclusion such as
#ifdef ATMEGA328P...#else...#endif.
File assembly - Files can be pasted inside other files via a
macro such as #include <Arduino.h>.

C++ is unique in that no other language except possibly Fortran

formally offers this feature. You may however find third-party
preprocessors for other programming languages (Python in
particular).

4.
 Compiler
<x>Build The compiler transforms human
toolchain[AUX]Compiler --> source code to machine code</x>

readable source code (.h/.cpp file) into machine code (.o file).
Because C++ applications can be very large, they are multi-module.
The compiler works on one module at a time.

5.
Linker
The linker creates your
<x>Build toolchain[AUX]Linker assembles machine code files + library components into executable file</x>

program. Once the compiler has finished compiling all of a program's

modules, the linker assembles them, along with library components,
into a whole, an executable, also referred to as image (usually a .hex
file).

6.
Uploader
Once the executable has been
<x>Build toolchain[AUX]Uploader (avrdude) loads executable into microcontroller via an ISP</x>

created, it must be uploaded into the microcontroller. A specialized

program, avrdude, handles this task. It sends the program to an ISP,
electronic hardware whose job is to load (burn) the program into the
microcontroller. One of the neat features of Arduino boards resides
in their becoming their own ISP via installed software, referred to as
a bootloader (next paragraph).

7.
Bootloader
<x>Build toolchain[AUX]Bootloader transforms board into self ISP</x> The ISP (In System Programmer) is a
piece of electronic hardware which receives the program via its com
port and transfers it into the microcontroller's memory. An ISP must
consequently be placed between the development computer and the
microcontroller board. The bootloader is a program installed in an
Arduino board which enables it to become its own ISP; an external
ISP is no longer needed. The price to pay is little less flash memory.

8.
Serial terminal
It is all good and well to create a gizmo
<x>Build toolchain[AUX]Serial terminal - visualize program output</x>

which opens a chicken coop door at sunrise and closes it at sunset.

But, during the development process, you may want to display info to
understand what the program is doing. You do this by writing to one
of your computer's serial terminal applications. The terminal could
also be a LED screen connected to your board.
 4.
Hardware setup

Hardware setup
This book is not about electronics. It is not about how to design an
Arduino-based board to respond to some <x>Hardware setup[AUX]Soldered prototype boards atop ATmega2560
board replace breadboards</x><x>Hardware setup[AUX]Breadboards enables rapid prototyping but has drawbacks (unreliable serial communications, spaghetti error prone wiring)

need such as my Arduino-based beehive weighing system.

</x>

However, the electronics setup (breadboard, prototype board, etc.)

you choose does have an impact on the development process.
When designing your gizmo, you must choose an Arduino board to
run it in. Which one can be a dilemma as there are many to choose
from: the ubiquitous Uno based on the ATmega328P, or a large one
such as the ATmega2560, or one of the many others such as the
Micro, Mini, Nano, and others. I suggest you should choose two
boards:

Choose the largest possible

<x>Hardware setup[AUX]ATmega2560 for development, smaller board for final release</x>

board to develop your application with - The ATmega2560 has

lots of RAM and flash memory which means you can insert
print-based debugging code to your hearts content and
encapsulate them in #ifdef DEBUG ... #endif pairs so that you might
choose what to include when debugging.
Production board: a Mini, Standard, Mega, whatever - Once
your code is fully debugged and running properly, you can
upload it into the smallest board which can handle the job.
PCB board - Once you have gotten your application to run
properly on the prototype board you may opt to create your
own PCB with a standalone microcontroller plugged into a
socket such as some of the Arduino Uno boards are equipped
with. Or, you may opt for a surface mounted board. You may
even opt for a smaller chip (ATmega١٢٨ or other) to reduce
costs.
 Having chosen your board, you need to
<x>Hardware setup[AUX]Breadboards, prototype board, PCB, SMT</x>

build your circuit and create a program which will run it. The first step
is to interface your board with electronics components which need to
be interconnected into a working circuit (opamps, sensors, radios,
timers, etc.). The immediate solution is breadboards. These are little
plastic blocks containing interconnected rows of holes into which you
can plug wires and components. You can assemble an electronic
circuit in no time without soldering.
When I began experimenting with Arduino, breadboards were a
godsend. I tried several devices such as displays, opto-couplers,
opamps, and others. I then moved up to more complex components
such as clocks, radios, GSM boards which required serial
communications. That is when things started going awry. The code
would run, and occasionally fail. Was it stack overflow, a glitch in my
program? Whatever the case, I was faced with glitches, the worst of
all bugs. Breadboard development provides flexibility because the
circuitry may be modified at a moment's notice, but it has its
drawbacks. These are detailed in the breadboards section below.
As soon as the circuitry had been firmed up and everything in place,
I built a soldered prototype board which replaced all the breadboard
circuitry with soldered circuitry; it plugged directly on top of an
Arduino ATmega2560 board. It did take me a few days to design and
assemble the prototype board; the time spent was well worth it since
all connection related glitches disappeared. In hindsight, I realize
that I could never have gotten the application to run had I stuck with
breadboards.
At this stage, I could have designed my final PCB on top of a Uno;
however, I preferred to design and fabricate my own PCB based on
a stand-alone microcontroller. A first PCB would be based on
through-hole technology (individual components soldered into
interconnected holes on the PCB). Once this production version was
fully tested, I could migrate to SMT (surface mount technology) to
bring the cost down. This of course supposes that my initial product
morphed from being a hobbyist's creation into a commercial product.

1.
 Breadboards
A
<x>Hardware setup[AUX]Breadboards enables rapid prototyping but has drawbacks (unreliable serial communications, spaghetti error prone wiring)</x>

breadboard circuit is the first step when designing a gizmo. You can
experiment with connections, validate a design, and get results
quickly. Breadboards enable putting together small circuits without
soldering. They are practical since they enable rapid prototyping of
small electronic circuits. They are approximately 6" x 2", or half the
length, have on their outer long edges two lone rows of
interconnected pins to provide voltage and ground, and two rows of
interconnected 5 pin wide columns to connect your components.
Good quality boards can usually be physically attached side by side
to each other to increase the total working surface. You can then
plug in your components and use jumper wires to connect them
together and to your Arduino board. This solution, although easy to
implement, has several drawbacks: large size, errors from wiring
mistakes which may damage components, poor connections. I
recommend color wires to reflect their function. It may save you grief.
Breadboard wiring can get unwieldy. You may need a second
breadboard, and a third, lots of wires. You end up with spaghetti
wiring! And then connections may become a little iffy! Poor contacts
generate disturbances which lead to runtime errors. Serial
communications in particular may prove unstable since connections
may undergo capacitive effects which can distort signals.
Notwithstanding these hardships, there are good reasons for using
breadboards; however, you should recognize their limitations and at
some point, you should migrate to soldered prototype boards.
Once you have assembled your circuit onto your breadboard and
checked, double checked, triple checked wiring, you can feed power
to your breadboard from some external source to check for eventual
short circuits by progressively connecting your components. If there
is a short circuit, you will quickly find out. When you are reasonably
sure that the circuit has no shorts, disconnect the external power
source and connect your Arduino to your computer via a USB. Use
the Arduino's 5V and ground pins to supply power to your
breadboard or connect a separate stabilized 5v power source if the
load requirements exceed the USB's 5V power capacity. If all seems
well, connect your Arduino ports to the breadboard progressively.
Upload your code into your Arduino and run. It is as simple as that.
Throughout the process of creating the circuitry on the breadboard,
always double check your work and use color coded wires.
Furthermore, remember that components are quite delicate,
astonishingly easy to damage.
Once your circuitry has been stabilized and firmed up, you may want
to consider building a soldered prototype board. Connections will be
more reliable and serial communications considerably more stable.
Read on.

2.
Prototype board
I mentioned that I develop on
<x>Hardware setup[AUX]Soldered prototype boards atop ATmega2560 board replace breadboards</x>

an ATmega2560 regardless of the final board the application will run

on. Using a prototyping board which piggybacks onto an
ATmega2560 is the way to go to continue development after the
proof-of-concept breadboard phase. Soldered prototype boards
ensure solid connections and hassle-free serial communications. Its
role is to interconnect your main components with the underlying
Arduino, in my case a clock, an XBee radio module, a temp/humidity
module, and a GSM board. I also provided pins for measuring 5V
and 8V voltage supplied from the outside, pinouts for turning power
on and off, and the means to connect Arduino ports to some other
board, be it a breadboard or other.
There are ATmega2560 compatible prototyping boards available on
eBay. The center is filled with holes just as if it were a breadboard,
except that the holes' top and bottom surfaces are interconnected so
that you can solder leads and interconnect holes via solder bridges.
Before doing any soldering, you should design the layout of your
components on paper - you are then ready to solder your
components. Since the ATmega2560 prototyping board piggybacks
onto the ATmega2560, you have access to all of its ports. You might
then envision placing the main serial communications components
on the prototype board and place other components on a
breadboard right next to it if need be. This solution has proven to be
practical for me, with the added benefit that serial communications
are clean, and connections are solid.

3.
Final PCB
You are probably using
<x>Hardware setup[AUX]PCB should have serial comm pins to interface board with computer</x>

through-hole technology (ex., resistors are little cylinders with a wire

on each end). You could also envision surface mount technology
(SMT - tiny little components soldered onto tiny pads). You would
have to provide a pinout from your board to connect to a serial
interface so that you could upload your application and visualize
serial output on your PC's serial terminal.
<x>Hardware setup[AUX]Eagle (now in Fusion ٣٦٠) for the schematics and Gerber files</x><x>Eagle software[AUX]Electronic schematic and Gerber files; now in Fusion

٣٦٠</x>Once the electronics are finalized, it is time to create the final

PCB. This final product should benefit from your experience with the
prototype board. Whatever your design, it should have pins for
battery power and ground, spare Arduino port pins just in case, pins
for LEDs to signal various conditions. I use Eagle (page ٢٢٧) (now in
Fusion ٣٦٠) for the schematic and for Gerber files to get the PCB
manufactured,
<x>Hardware setup[AUX]PCB should have serial comm pins to interface board with computer</x>You should also incorporate

four pins to connect to a serial interface; it enables serial

communications to upload code into the Arduino microcontroller and
visualize the application's prints. The following four Arduino pins are
needed:
TX/RX - transmit and receive pins.
RST- the reset pin
GRD - a ground pin
To interface with the computer, I use a standard Arduino Uno board
with the ATmega328P microcontroller removed. This assumes that
the microcontroller can be removed, as there are Arduino Uno's on
which the Arduino microcontroller is permanently surface mounted. I
connect the production board PCB pins to the Arduino board pins:
TX to TX, RX to RX, RST to RST and GRD to GRD. I then connect
the Arduino board to the computer via a USB cable. Having done
this, I develop, upload code, and monitor serial output on the
production PCB just as if it were an Arduino board.
 5.
Interoperability

Interoperability
When working with several IDEs, interoperability is desirable, i.e.,
the capacity to easily switch from one IDE <x>Interoperability[AUX]PlatformIO creates [Link] with empty
setup and loop</x><x>Interoperability[AUX]Seamlessly switch between AtmelStudio, Visual Studio, PlatformIO, Arduino IDE</x><x>PlatformIO (interoperability)[AUX]Creates

[Link] with empty setup and loop</x><x>PlatformIO (interoperability)[AUX]Seamlessly switch between AtmelStudio, PlatformIO, Arduino IDE, Visual Studio</x>

to another, hassle free. Interoperability

<x>Interoperability[AUX]Common set of files independent of IDE</x>

means one set of project files which you can be worked on via one of
several IDEs. Six single and dual Arduino development tools are
covered in this book: Arduino IDE (page 30), AtmelStudio (page 37),
Visual Micro for AtmelStudio (MicrochipStudio) (page 84), VS Code
Arduino extension (page 93), VS Code PlatformIO extension (page
95), and Visual Micro for Visual Studio (page 85). The question is,
should you work with one IDE only, with two, or with more.
This chapter covers the issues which interoperability raises:

Directory structure (next section) - Some scenarios require a

single-level directory structure (Visual Micro), others require a
two-level one (AtmelStudio), and others can be adapted to one
or the other.
IDE specific entry files (page 22) - Arduino and Visual Micro
place setup and loop in a .ino file, AtmelStudio and Visual
Studio/Visual Micro in a [Link] file, PlatformIO in a
[Link] file. These files coexist in the same source code
directory and will consequently be compiled/linked into the
executable, thereby raising a multiple definitions problem.
Appropriate IDE #define macros solve this problem.
Render setup/loop independent of code changes (page 24) -
The Arduino sketch file (.ino file) contains setup and loop, each of
which will contain source code. Furthermore, the .ino file may
also contain function definitions, includes, global variables, and
 more. Interoperability raises the issue that changes in the .ino
file should be identically undertaken in the other IDE entry files.
The solution to this dilemma relies on creating a function,
called by setup, which contains all the code of the original .ino
file.
The Single-level directory structure interoperability scenario
(page 24) applies when using Visual Micro (page 84) as a
plugin in AtmelStudio or in Visual Studio.
The Two-level directory structure interoperability scenario
(page 25) applies when working with AtmelStudio (page 37)
along with PlatformIO (page 98).

These subjects are covered in the sections which follow.

1.
Directory structure
Which interoperability combination you will want to work with dictates
whether you are to work in a single-level project source code
directory structure, or in a two-level one.
Defining a common directory's location and how to set it up requires
understanding how the IDEs structure their directories. The Arduino
IDE and AtmelStudio differ when placing code in directories. The
Arduino IDE's directory structure could not be simpler. It boils down
to creating a directory anywhere in your computer, whose name is
identical to your sketch's name without the ending .ino.
<x>Interoperability[AUX]How to establish interoperability</x><x>AtmelStudio (interoperability)[AUX]Importing Arduino project procedure</x><x>PlatformIO (interoperability)

The AtmelStudio and PlatformIO

[AUX]PlatformIO Arduino project located in AtmelStudio project base directory</x>

directory structures are a little more complex than the Arduino IDE's.
They both have a directory and subdirectory (source files) structure.
AtmelStudio looks for source files in a subdirectory which has the
same name as the main project directory, the project's name.
PlatformIO looks for source files in a subdirectory whose name is
src, but this can be easily changed to the project's name, which
opens the way for interoperability between AtmelStudio and
PlatformIO. Interoperability does have its limitations. I recommend
developing with Visual Micro for Visual Studio (page 85) and debug
with AtmelStudio (Hardware-based debugging page 119). I have,
unfortunately, not succeeded in doing this within the same directory
structure.
When deciding upon a directory structure here are some givens you
need to consider:

Visual Micro (page 84) - Whether atop AtmelStudio or Visual

Studio, Visual Micro functions on a single-level directory
structure basis. PlatformIO may be used within this single
directory structure (see PlatformIO interoperability page 107).
AtmelStudio and PlatformIO - A two-level directory structure
makes for a clearer directory structure. Start with creating a
two-level directory AtmelStudio project via an import (Importing
an Arduino project (AtmelStudio) page 40) and create a new
PlatformIO Arduino project (PlatformIO creating projects page
103) inside the Arduino project directory.
Arduino IDE - Interoperability is simple, just move the .ino file
to wherever the source files are and click on the .ino file to
launch the Arduino IDE. Whatever interoperability combination
you want to work with, Arduino IDE interoperability is
established by naming the source code directory the same as
the .ino file's name without the .ino extension.

The above leads to two possible interoperability scenarios:

The single-level classical Arduino IDE directory structure.

The two-level classical AtmelStudio one project/one solution
directory structure (see Managing directories page 39). This
two level directory structure may be used for Hardware-based
debugging (page 119) with AtmelStudio.

Combining the two setups above allows interoperability between the

IDEs (Arduino IDE, AtmelStudio (Visual Micro), Visual Studio (Visual
Micro), VS Code/PlatformIO, VS Code/Arduino extension).
 2.
IDE specific entry files
There are three project entry files to deal with - The first and
foremost project entry file is the ArduinoIDE's .ino file. This file
contains code (setup and loop) which gets the application going. Both of
these functions may call functions, classes, and variables defined in
other files (.h/.cpp). If you are to work with AtmelStudio and
PlatformIO, you will have to contend with other files which contain
setup and loop:

Arduino IDE .ino file - This file, referred to as a sketch file, will
have code at global level, #includes, macro definitions, and code
inside both setup and loop.
AtmelStudio and Visual Studio/Visual Micro [Link] file -
When you do an AtmelStudio Arduino project import,
AtmelStudio automatically copies the contents of the .ino file
into [Link]. The build uses this file as the application entry
file, the .ino file is ignored.
Visual Studio/Visual Micro create empty setup/loop functions
inside a [Link] file.
PlatformIO creates a [Link] file which contains nothing save
empty setup and loop function definitions. Visual Studio/Visual
Micro create a [Link] file with empty setup/loop functions.
The .ino file is ignored.

<x>Interoperability[AUX]Create If you try to compile the

ATMEL_STUDIO, ARDUINO_IDE, PLATFORM_IO macros</x>

project, no matter which IDE you do it with, you will get multiple
definition compiler errors because it will find setup and loop, and
possibly other stuff, defined more than once due to their being
several entry files. The solution to this problem lies in wrapping each
entry file's content with an IDE specific macro #ifdef...#endif pair, as
follows:
 //#define ATMEL_STUDIO
<x>Interoperability[AUX]Create ATMEL_STUDIO, ARDUINO_IDE, PLATFORM_IO macros</x>

//#define ARDUINO_IDE
//#define VISUAL_STUDIO_IDE // Visual Micro for Visual Studio
#define PLATFORMIO_IDE
You should verify that one and only one macro gets defined by
logically testing for the existence of one and only one active #define.
The simplest way is to find all possible valid combinations and check
whether any has been defined, such as:
#if ( /* if any return true, we are OK */ \
(#if defined(AA) && !defined(BB) && ! defined(CC) && ...) /* AA defined */ || \
(#if defined(AA) && !defined(BB) && ! defined(CC) && ...) /* BB defined */ || \
(#if defined(AA) && !defined(BB) && ! defined(CC) && ...) /* CC defined */ || \
...
)
#elif /* None of the above returned true hence error */ \
#error (F("You must define one IDE, only one"))
#endif // Define IDE
AA, BB, etc. above are shorthand replacements for the individual IDE
specific #defines (AA stands for ATMEL_STUDIO, etc.). The macro looks for
a valid IDE selection - 3 IDEs require 3 macros, 4 IDEs require 4. If
none is found, the overall #if logic returns false thereby triggering #error.
When compiling the project, the macros will exclude code save the
one that is specific to the IDE you are working with. If you were to
compile your code in an IDE different from the active macro, you
would get an undefined reference on setup and loop.
The coexistence of three project entry files raises another issue. If
you make changes in one of the entry files, you have to transpose
those changes into the other entry files. This is both time consuming
and error prone, not good. The solution consists in rendering the
entry files independent of code changes (see next section).

3.
Render setup/loop independent of
code changes
As indicated in the previous section, making changes in the .ino file
requires that the changes be also brought to the [Link]
(AtmelStudio) and [Link] (PlatformIO) entry files because they
both contain setup and loop. To remove this error prone and time-
consuming burden, make it so that the entry files should never need
to be modified. The solution is threefold.

Displace all global definitions and includes into a

Globals.h/[Link] file pair. The three entry files will have a
#include "Globals.h" only.
Remove all code from the loop function and place it inside a
while(true) loop at the bottom of setup. I recommend not using loop.
<x>Interoperability[AUX]Aardvark() contains original setup code, key to interoperability</x><x>Aardvark[AUX]Aardvark() contains original setup code, key to

interoperability; in Aardvark.h/.cpp</x><x>AtmelStudio (interoperability)[AUX]Aardvark() contains original setup code, key to

Create a function (I named it Aardvark) into which you

interoperability</x>

move all of setup's code. Place this function in a file pair named
Aardvark.h/.cpp. The [Link], the [Link] and the .ino
files are now reduced to a minimum and need never be
altered. These three files will contain a call to Aardvark from
within setup. Each of the three entry file's content will be
contained within an #ifdef...#endif macro, thus:
#include "Globals.h"
#ifdef XXX // where XXX is the platform, ARDUINO_IDE or other.
setup(){
Aardvark(); // In Aardvark.h and [Link] files
} // setup
loop(){} // Always empty
#endif // #ifdef XXX

Commenting/uncommenting the IDE specific #defines in Globals.h

suffice to switch from one IDE to the other. All the code which
would normally be in the entry file (setup and other code) is now in
Aardvark. Changes to Aardvark and to other .h/.cpp files are now IDE
independent. The code is guaranteed to be identical between IDEs.
Interoperability has now become a simple macro choice in
Globals.h. All program logic lies in Aardvark.h/.cpp and other
.h/.cpp files common to the IDEs. The .ino, [Link] and
[Link] entry files are relegated to being service files whose
content never changes.
Warning: It is possible to open two or three IDEs on the same project
simultaneously. There is nothing to prevent you from undertaking
such a dangerous initiative. Save yourself pain and grief, do not do
this, even momentarily.

4.
Single-level directory structure
interoperability scenario
If you are to work with Visual Micro, particularly if Visual Studio is
your main development tool, a single-level directory structure is
mandatory because Visual Micro imposes it. What follows is an
overview of tasks required to achieve a single-level directory
structure interoperability:
Arduino IDE, Visual Micro, and PlatformIO can all work within a
single-level directory structure. But you must make some
adjustments.

Arduino IDE - Wherever you decide to place your Arduino IDE

project, you must respond to one criterium. The project's
directory name must be the same as the .ino file's name
without the .ino extension. Easy!
Visual micro with AtmelStudio uses the [Link] file within
the Arduino directory structure. So does Visual Studio with
Visual Micro.
PlatformIO interoperability requires that you do some work as
the PlatformIO entry file is [Link], created by PlatformIO
with empty setup and loop functions. Adapting PlatformIO for
single-level directory structure limits itself to:
Adding a call to Aardvark from the [Link] setup function.
Wrapping setup and loop in [Link] with the #ifdef PLATFORMIO
#endif macro.
Placing an #include "Globals.h" in the top [Link].
Putting src_dir = . in the platformio section of the [Link] file.
Voilà - Very simple. The single -level directory structure establishes
interoperability between four IDE's (Arduino IDE, AtmelStudio +
Visual Micro, Visual Studio + Visual Micro, PlatformIO).

5.
Two-level directory structure
interoperability scenario
What follows is an overview of tasks required to achieve a two-level
directory structure interoperability scenario. This scenario is the one
you would adopt when working with the Arduino IDE, AtmelStudio
(stand-alone without the Visual Micro plugin), and PlatformIO - you
could still work with Visual Micro directly from the source code
directory. Before continuing, note that you will have to create a new
project directory since the two-level scenario does not support using
the original Arduino project directory; this is explained further down.
To establish interoperability, begin with AtmelStudio. When you
import an Arduino sketch with AtmelStudio, it creates a directory and
source subdirectory structure, both with the same name, the
application's name. Note that AtmelStudio will not allow importing a
sketch into a directory which already exists. This means that you
cannot use the original Arduino IDE project directory for your
common interoperability one. You must choose another location.
That is why you must begin with AtmelStudio.

<x>AtmelStudio (interoperability)[AUX]Importing Arduino project procedure</x><x>AtmelStudio (file management)[AUX]AtmelStudio directories key to

Start by creating an AtmelStudio Arduino project from

interoperability</x>

an existing Arduino IDE project. When doing so, choose the

AtmelStudio one solution per project scenario (see Managing
directories page 39) in which case do not tick the Create
directory for solution checkbox when importing the Arduino
sketch. The solution's name should be that of the Arduino
project: the Arduino sketch without the .ino extension. Then do
a complete Arduino sketch import as detailed in Importing an
 Arduino project (AtmelStudio) (page 40). AtmelStudio is ready
to go.
<x>AtmelStudio (interoperability)[AUX]AtmelStudio Arduino project entry file [Link] contains setup and loop</x><x>AtmelStudio (interoperability)

Copy the Arduino sketch

[AUX]Copy the Arduino sketch file (.ino file) and .h/.cpp files into the AtmelStudio project files</x>

file (.ino file) and .h/.cpp files into the AtmelStudio project files
subdirectory, you have now achieved interoperability between
AtmelStudio and the Arduino IDE.
AtmelStudio Arduino entry file - AtmelStudio creates
[Link] and copies the contents of the .ino file into it. The
Arduino .ino file and the AtmelStudio's [Link] are identical
C++ files. The only difference is the name. This file is placed in
the MySketch subdirectory, the project's source file directory.
You should modify this file as explained in Render setup/loop
independent of code changes (page 24).
<x>Aardvark[AUX]Aardvark() contains original setup code, key to interoperability; in Aardvark.h/.cpp</x><x>PlatformIO (interoperability)[AUX][Link] is

PlatformIO's Arduino app entry file (.ino replacement)</x><x>PlatformIO (interoperability)[AUX]Operate in single- or two-level directory structure; code in src_dir

PlatformIO - Having created an Arduino project and

[Link] variable</x>

established interoperability with AtmelStudio, continue with

PlatformIO. Start by creating a new PlatformIO project. When
specifying its name and location, choose the AtmelStudio
project directory (not the source file subdirectory). PlatformIO
will not touch any of the files already there, it will just add its
own files in the main project directory; they do not interfere
with AtmelStudio. It will create a src subdirectory for the source
files. You do not want them in the PlatformIO src subdirectory,
you want them in the AtmelStudio subdirectory named
MySketch. At this point all you need to do is add the following
two lines in the [Link] file located in the main project
directory to specify where source files should be located:
[platformio]
src_dir = MySketch

Transfer and modify [Link] from the src subdirectory to the

MySketch subdirectory; and have setup call the Aardvark function as
per Render setup/loop independent of code changes (page 24)
 above. You can now delete the src subdirectory.

<x>AtmelStudio (interoperability)[AUX]ATMEL_STUDIO macro to enable AtmelStudio</x><x>Arduino (interoperability)[AUX]ARDUINO_IDE macro enables

Arduino IDE</x><x>PlatformIO (interoperability)[AUX]Creates [Link] with empty setup and loop</x><x>AtmelStudio (interoperability)[AUX]Aardvark()

contains original setup code, key to interoperability</x><x>Interoperability[AUX]Aardvark() contains original setup code, key to interoperability</x>

Other work - For the

<x>Aardvark[AUX]Aardvark() contains original setup code, key to interoperability; in Aardvark.h/.cpp</x>

rest do the same as indicated in the single-level directory

structure above, namely regarding creating IDE macros and
creating the Aardvark function..

As to interfacing with the board via the consoles (serial terminals),

please note that PlatformIO has its own serial terminal. AtmelStudio
does not - you can use an Arduino IDE's serial terminal to visualize
your AtmelStudio program's output (see Interface with Arduino
applications page 48). Please note that if you opt to work in an
AtmelStudio/Visual Micro or Visual Studio/Visual Micro combination,
you will have a serial terminal, but you will also be in a single-level
directory scenario, hence outside this section's context (see previous
section).
You are all set to switch from one IDE to another via
comment/uncomment IDE specific #defines in Globals.h.
 6.
Which IDE to work with?

Which IDE to work with?

The year 2005 marks Arduino's birth, the beginning of highly
accessible, ubiquitous and cheap microcontroller development.
<x>Interoperability[AUX]Seamlessly switch between AtmelStudio, Visual Studio, PlatformIO, Arduino IDE</x><x>Arduino (interoperability)[AUX]Interoperability between

Arduino IDE, AtmelStudio, PlatformIO, Visual Micro, hassle free</x><x>AtmelStudio (interoperability)[AUX]Interoperability between Arduino IDE, AtmelStudio, PlatformIO,

Visual Micro</x><x>PlatformIO (interoperability)[AUX]Interoperability between Arduino IDE, AtmelStudio, PlatformIO, Visual Micro</x><x>Visual Micro[AUX]Interoperability

between Arduino IDE, AtmelStudio, PlatformIO, Visual Micro</x><x>Which IDE to work with?[AUX]Interoperability possible between Arduino IDE, AtmelStudio, Visual Micro,

PlatformIO</x><x>Which IDE to work with?[AUX]Short list: Arduino IDE, AtmelStudio, Visual Studio, PlatformIO, VS Code</x><x>Which IDE to work with?[AUX]PlatformIO -

free, professional grade, easy import of Arduino projects</x><x>PlatformIO (general)[AUX]IDE supports cross-platform, cross-architecture, multiple frameworks</x>

You
<x>PlatformIO (general)[AUX]Plugin for Microsoft VS Code and CLion</x><x>PlatformIO (general)[AUX]Free, professional grade, easy import of Arduino projects</x>

and I, ordinary electronic tinkerers, have access to ever increasingly

powerful microcontrollers (from 8-bit Atmel to 32-bit ESP32) and to
ever more user-friendly IDEs (from the early Arduino IDE to today's
Visual Studio/Visual Micro combination).
My first encounter with embedded systems development was
Arduino - it got me to create a beehive weighing system. That is
because I raise bees. The program got big (35 files - 15,000 lines of
code). The result was that debugging became a major time-
consuming chore. To make a long story short, my search as to why I
was spending so much time debugging, and how to reduce it, got me
to write this book and its companion Pragmatic C++ Arduino
Programming. In the process of writing these books, I researched
alternate Arduino development tools.
You will find below a section which summarizes the main Arduino
compatible IDEs I came up with and a section containing various
plugins which enhance some IDEs.

1.
Dedicated IDEs
Dedicated chapters summarize the main free (or very cheap)
Arduino compatible IDEs I came across.

<x>Which IDE to work with?[AUX]Short list: Arduino IDE, AtmelStudio, Visual Studio, PlatformIO, VS Code</x><x>Which IDE to work with?[AUX]Arduino

readily available, inexpensive, particularly easy to work with, required by other IDEs</x><x>Which IDE to work with?[AUX]AtmelStudio best: solid, professional

The Arduino IDE (page 30) is the entry point

tool, supports hardware-based debugging</x>

into embedded systems programming - an extremely easy to

use IDE along with cheap boards and microcontrollers. It is
probably the tool you are currently using to program Arduino
boards with.
AtmelStudio (page 37) is a customized version of Visual Studio
designed for work on Atmel microcontrollers (Microchip).

These two tools are standalone IDEs. They are out-of-the-box self-
contained tools which enable Arduino development.

2.
Plugins
The tools described in this section are foundation tools (Visual
Studio and VS Code) which can be enhanced by plugins. They are
foundation tools in that they meet certain basic needs, such as code
editing, file/folder navigation... Other tools (Visual Micro and
PlatformIO) are plugins to build/upload the application, undertake
hardware-based debugging, etc.

Visual Studio (page 79) is Microsoft's flagship

multiplatform/multilanguage development tool. You can
develop Arduino applications with it by installing the Visual
Micro (page 84) plugin. If you already use AtmelStudio with
Visual Micro, you will find Visual Studio's learning curve to be
practically nil. Using Visual Studio futureproofs your knowhow.
IMHO, it the best tool for Arduino work.
PlatformIO (page 98) is a VS Code extension. Very powerful, it
 rivals with AtmelStudio and Visual Studio, but its learning curve
is steeper.
Visual Micro (page 84) is an plugin for AtmelStudio and Visual
studio which provides them with missing features. It supports
Serial debugging (page 125), a debugging technique I qualify
as halfway between print-based debugging and hardware-
based debugging.
VS Code (page 86) is Microsoft's development tools foundation
product with which powerful, sophisticated tools can be
created via extensions. PlatformIO (page 98) is one of many
VS Code extensions.

You will find detailed info on the IDEs presented above in IDE
specific chapters. You will also find details on achieving
interoperability between them and how to switch from one IDE to
another, a click away, while working on a unique set of files. See
Interoperability (page 21).
The AtmelStudio (page 37) section contains considerably more
details than sections on other IDEs because I found it to be the
easiest, most powerful IDE to use for Arduino development. Visual
Studio may be used as an alternative if you want to work on
microcontrollers other than Atmel and ESP32 ones. Note that its
coverage is short because Visual Studio is practically the same as
AtmelStudio. The PlatformIO section is fairly detailed as this platform
is unique and a potential alternative to AtmelStudio and Visual
Studio.
Most Arduino newbies continue along the Arduino IDE route
because it is complete, easy to use, and has practically no learning
curve - they never make the jump to professional grade IDEs such
as AtmelStudio (page 37), Visual Studio (page 79), or PlatformIO
(page 98). This was my case. I had been developing with the
Arduino IDE for three years before switching to AtmelStudio. I must
mention that I have excuses for not using AtmelStudio earlier.
Importing an Arduino sketch into AtmelStudio seemed to be a hassle
thereby dampening my enthusiasm. Furthermore, AtmelStudio's
interface, with its many toolbars, icons, and menus was intimidating,
presaging a steep learning curve. It is only since version 7 that
AtmelStudio made importing a sketch easy, and the learning curve,
contrary to my first impression, turned out to be gentle. As for
PlatformIO, it is a professional grade IDE alternative unfortunately
beset by a not-negligible learning curve.
The Arduino programmer today is lucky in that he/she can choose
amongst one of several IDEs. The Arduino IDE will certainly be an
initial choice, just to get going. Even if you do not intend to use the
Arduino IDE, other IDEs require the Arduino distribution; the Arduino
IDE will consequently get installed. Having tested the Arduino IDE,
take a look at AtmelStudio (+Visual Micro), Visual Studio (+Visual
Micro), and PlatformIO. Whatever your IDE choice, interoperability
enables you to work on a unique set of files and seamlessly switch
from one IDE to another; this is Interoperability (page 21).
My advice is that you may be well heeded to work with AtmelStudio
plus the Visual Micro plugin; it is powerful, reliable, easy to use, and
almost free (AtmelStudio is free, Visual Micro is $25/yr) - it is a win-
win proposition. Or, you might opt for Visual Studio with the Visual
Micro plugin. On the other hand, although not perfect, you may opt to
continue with the Arduino IDE since it has proven to be immensely
useful and simple to use.
 7.
Arduino IDE

Arduino IDE
The Arduino team created an extremely user-friendly, ready-to-use,
out-of-the-box Arduino programming <x>Arduino (general)[AUX]IDEs: Arduino, AtmelStudio, Visual Studio, VS Code,
PlatformIO, Visual Micro</x><x>Arduino (general)[AUX]For new Arduino IDE version see Arduino (version 2)</x><x>Arduino (general)[AUX]IDE for Atmel 8-bit and ESP32

boards, no hardware-based debugging</x><x>Arduino (general)[AUX]Arduino tools are user-friendly and free, components cheap, community huge</x><x>Arduino (general)

[AUX]Distribution includes: tools to create program, libraries, upload, serial terminal</x><x>Arduino (general)[AUX]Two versions: 1.8.19 (legacy) and V2 (new features)</x>

environment: create code, compile

<x>ESP32[AUX]Supported by Arduino IDE and PlatformIO, not by AtmelStudio</x>

and link to generate an executable; upload the executable into the

microcontroller; and interface with the microcontroller via the built-in
serial terminal . All this may be achieved by just clicking on one
button.
The hardware is an 8-bit CPU based board; the software is an as
user-friendly development environment as you can get. The result is
that millions have successfully used this system, hobbyists and
professionals. Before Arduino, working with embedded systems was
the domain of professionals, and the tools were expensive. Today,
$25 and a practically zero learning curve gets you going.
The Arduino boards may be summarily described as being boards
with input/output ports controlled by a CPU (an 8-bit Atmel/Microchip
microcontroller). The Arduino software is designed so that you, the
programmer, could create programs which drive the CPU. The initial
Arduino board was named a Uno, it was created in 2005 by a team
of Italian students and researchers. It seems that the original Arduino
name was the name of the cafe where the team usually met. Since
2005 new boards have been created, tiny ones and large ones, to
respond to a spectrum of needs. As for the Arduino software, initially
limited to 8-bit Atmel microcontrollers, it can now be used for ESP32
work.
If you intend to work on embedded systems as a hobbyist or at a
professional level but are just getting started with such systems, the
Arduino system is a perfect starting point.
It is a great all-in-one tool but remains a little limited when compared
with professional grade tools presented in this book; I say this
despite the new version 2.x release which begins to narrow down the
features gap. For rapid testing of small code programs, the Arduino
IDE is unbeatable.
Get an Arduino Uno; download/install the Arduino system
([Link] open the Arduino IDE; connect the board to your
PC via a USB port; run the Blink program (Hello World Arduino
equivalent); watch the LED on your board blink; and finally, examine
the code and experiment. You will marvel at how easy this process
is; yet, a lot of technology hides behind this simplicity. And don't be
deterred by the fact that you are constrained to programming with
C++. It is the best language to achieve compact, fast programs.
Other languages, Python for example, are easier to program with but
they consume much more RAM and are a lot slower. So, futureproof
your development skills by using C++, and be glad that you are
given a sound justification (fast, compact executable) for learning
this language.
I diligently used the Arduino IDE to develop my beehive weighing
system. I started with a back of the envelope concept and started
coding: my first .ino file. I progressively added features: GSM board
support, radio transmission, sensor interfaces, etc., and I created
frameworks along the way. The Arduino IDE proved to be a complete
integrated IDE for creating, compiling, and uploading code into the
microcontroller, and for visualizing output in a serial monitor. Its main
drawbacks are its limited program editing features, a lack of
hardware-based debugging, and two bad glitches as the application
gets large (ctrl-Z which messes up your code and fragmentation
faults - these may have been fixed in version 2.x). See Arduino
caveats (legacy version 1.8.19) (page 36).
It is well worth your while to explore what the Arduino IDE can do
before considering installing AtmelStudio (page 37). I mention this
because AtmelStudio has lots of features the Arduino IDE is missing;
you might have already installed it just for this reason. I suggest that
you should look at the Arduino IDE. For creating/testing small
programs or C++ constructs, it is unbeatable. So, get started by
trying out Arduino IDE example programs, the C HelloWorld
equivalent being the Blink program.
Before continuing, you should be aware that there are today two
Arduino software versions: the original one, whose last version is
1.8.19, referred to as legacy code, and the recent one, version 2
(2.1.1 May 2023), which addresses some limitations of the legacy
version. The material which follows covers both versions. You will
find a dedicated section on the V2 version further down (Arduino IDE
V2 specifics page 34).

1.
Arduino distribution
The Arduino distribution
<x>Arduino (general)[AUX]Arduino distribution seamless installation; IDE, libraries, toolchain, serial monitor</x>

contains the Arduino IDE (a C++ program editor and application

build manager), a GNU build toolchain (compiler, linker, etc.), and a
set of libraries destined to create applications for microcontroller
boards based on Atmel 8-bit microcontrollers. Its ease of use has led
many to enthusiastically create all sorts of devices which do small
yet important jobs: orient sun panels, bead counter, open/close a
chicken coop door at sunrise/sunset, weigh beehives and send an
SMS, etc. There are literally hundreds of thousands Arduino gizmos
out there. This success is accounted for by the simplicity of using the
Arduino board/Arduino programming environment combination.
One download from the Arduino Web site
[Link] and you are all set to go. After
undertaking a seamless installation, here is what you get:

A complete Arduino
<x>Arduino (general)[AUX]Distribution includes: tools to create program, libraries, upload, serial terminal</x>

installation in C:\Program Files (x86)\Arduino - IDE, libraries,

executables, examples.
Arduino IDE - The Arduino environment has an IDE which is
basic but useful enough for small programs.
<x>Arduino (build)[AUX]Arduino, AtmelStudio, PlatformIO, Visual Micro use the GNU C++ compiler</x>The GNU compiler

and linker are installed, ready to use. The IDE drives them in
the background.
 avrdude uploads the
<x>Arduino (build)[AUX]avrdude uploads the executable into the microcontroller</x>

executable into the microcontroller. The IDE drives it in the

background.
<x>Arduino (build)[AUX]Installs basic and third-party libraries</x>Basic libraries - The Arduino

distribution is shipped with libraries which include just about all

of The C Standard Library, a String class, and Arduino specific
functions to manage the microcontrollers. Note that neither the
Standard C++ Library nor the Standard Template Library (STL)
are included.
Third-party libraries - The Arduino distribution also includes a
set of useful third-party nonessential libraries. It is just a
sample. You will find more such libraries deposited in GitHub.

Click on a shortcut to the

<x>Arduino (general)[AUX]Arduino distribution seamless installation; IDE, libraries, toolchain, serial monitor</x>

Arduino IDE and you are set to start your first program. You will need
to connect an Arduino board to your computer via an USB port
though, since the program runs on a board, not in your computer.
Once you get your program running, you can visualize its output and
communicate with it via the Arduino IDE's built-in serial monitor, or
you may blink LEDs, or you may turn on a motor via a relay, or you
may undertake countless other endeavors.
Now that you have installed the Arduino programming environment
for your microcontroller, you are all set to go.

2.
Arduino Basic features
Just a reminder: this book is not about using Arduino boards; i.e.,
how to connect them to hardware, how to use the pins to manage
electronics. It is about being more proficient programming Arduino
microcontrollers with C++. The companion book Pragmatic C++
Arduino Programming details what C++ features an Arduino
programmer should know. This book is a sequel; it presents
frameworks and tools which should significantly enhance your
productivity and improve your application's robustness.
Programming an Arduino board requires your writing code and
compiling/linking/uploading the executable. It is entirely possible to
program with notepad, to compile and link code directly with the
GNU toolchain, and to upload an executable with avrdude, all this as
commands in a DOS box (appendix) (page 237). You will still need to
download Arduino libraries; they contain essential components
(serial communications, math functions, etc.).
<x>Arduino (general)[AUX]Wraps tools inside a clean, practical interface</x>You would be much better off using the

free Arduino IDE than doing command line work; it takes the hassle
out of creating embedded applications by wrapping it all inside a
clean, practical interface between yourself and the tools which do the
work. This is where the Arduino team did a tremendous job. They
created a practical C++ IDE workspace which includes a fairly
decent C++ editor and whose toolbar contains commands to
compile, link, upload your program into your microcontroller, and run
it.
Having downloaded and installed the Arduino distribution, you
should be aware of a few details before you start programming:

Project
<x>Arduino (build)[AUX]Arduino project's name is that of .ino file</x><x>Interoperability[AUX]Arduino project's name is that of .ino file</x>

directory - The project's files are located in the project directory

(any convenient directory in your computer). There are no
other directories specific to an Arduino project such as one
would find with AtmelStudio and PlatformIO. The project
directory's name must be the same as the application's name:
the .ino file name.
<x>Sketch[AUX]Name used to refer to Arduino's .ino file (program entry file)</x><x>Arduino (build)[AUX].ino file is Arduino's entry file (C++ source code)

The Arduino entry file is an .ino file located in the project

</x>

directory whose name is the .ino file's name without the .ino
extension. It differs from other C++ files in that it has a .ino
extension as opposed to a .cpp extension, it is otherwise an
ordinary .cpp file. Clicking on the .ino file opens the Arduino
IDE with the contents of this file. This .ino file is referred to by
Arduino as being a sketch. When you save a new project, you
give it a name. The Arduino IDE creates a directory whose
name is the project's name and saves your code in a file
 whose name is the project's name with an .ino extension,
instead of a .cpp extension, inside the just created project
directory .
<x>.h/.cpp files[AUX]Arduino loads all project directory .h/.cpp file into the editor</x><x>Arduino (build)[AUX]Loads all project directory .h/.cpp file into the

.h/.cpp files located in the Arduino project directory are

editor</x>

loaded into the IDE along with the .ino file. They show up as
tabs in the Arduino IDE workspace.
<x>avrdude[AUX]Arduino compile/upload uses avrdude to send .hex file to board, bootloader uploads it</x><x>Arduino (editor)[AUX]Button compiles and uploads

Compile and upload - A button builds the executable. A

application</x>

second button builds the executable and uploads it into the

microcontroller. Note that when doing a build, the Arduino IDE
saves all files.
<x>Arduino (editor)[AUX]Serial monitor interfaces with board</x>Interface with the board - A button

opens the Arduino IDE's serial monitor to interface with the

microcontroller. It will display your program's [Link]. When
the serial monitor is launched or when you reload your
program, the board receives a reset command which restarts
the application.

An Arduino application's code and file structure must respond to

specific requirements you should be aware of, namely:

<x>Arduino (build)[AUX]main is generated behind the scenes, calls setup and loop</x><x>Arduino (build)[AUX]setup and loop - mandatory functions for Arduino

The .ino file is the entry point into the application; it

programs; leave loop blank</x>

contains two functions, named setup and loop; there is no visible

main function. It is supplied in the background when you click on
the build button. For whatever reason, the Arduino team
decided that every Arduino program should be structured
around two functions: setup and loop.
void setup() is called once, at startup, hence the name setup.
void loop() is called repeatedly once setup has exited. It does what
its name implies; it loops forever. A word of advice: forget loop,
do everything in setup. And, if you do need an infinite loop such
as in the Blink example, just do a while(true) { ... } inside setup. Doing
this enables you to interrupt the while and do something else
 whereas using the Arduino IDE's loop means that it will run
indefinitely. This is fine for the Blink program, but useless and
a bother for most other programs.

Why did the Arduino team structure an Arduino program around

these two functions? I have not the slightest idea. But that is the way
it is. The .ino file concept is useful though; it enables Windows to tie
it to the Arduino IDE at the Windows Explorer level. Clicking on an
.ino file brings up the Arduino IDE with the .ino file.
<x>Arduino (build)[AUX]main is generated behind the scenes, calls setup and loop</x><x>Arduino (build)[AUX]Suggestion: do not use loop, do all inside

Remember that all C++ programs start by calling a function

setup</x>

named main. Arduino programs being C++, they are not exempt from
this requirement.
Behind the scenes, main is indeed generated; it calls setup. When setup
returns, a while(true){ ... } or for(;;){ ... } runs loop indefinitely. The following is
a complete Arduino program, nothing is missing, though it is useless
since it does nothing.
int main() { // Added by Arduino
setup(); // your code in setup
while(true) { loop(); } // your code in loop
} // main()
I advise that you should not use loop. Leave it empty and do
everything from within setup unless you are planning to do event-
based programming. By event-based programming I mean that loop
would contain a switch, each case representing an event: an alarm, a
change in a port's voltage, button press, etc.
I shall not cover details on using the Arduino IDE version 1 (legacy)
as practically all you readers of this book have been using it for
years. You may however not have tried the new version 2; this is why
I have written a short section on my first experience using it (next
section).

3.
Arduino IDE V2 specifics
I should
<x>Arduino (version 2)[AUX]New features and minor problems</x><x>Arduino (general)[AUX]Two versions: 1.8.19 (legacy) and V2 (new features)</x>
mention that Arduino recently released its new IDE version 2.1.1
(May 2023). It is no longer in beta mode. I downloaded it and started
putting it through its paces. It is a major advance relative to version
1.8.19, the latest prior version, now referred to as legacy Arduino.
The discussion which follows is specific to Arduino 2.1.1. New
versions may contain additional features and may have fixed some
of the problems raised.
<x>Arduino (version 2)[AUX]Look and feel similar to legacy version</x>The main new significant non cosmetic

features of version 2.1.1 are:

Installation was successful although there were some error

messages which required reinstalling some features. It was not
as seamless as installing the old version.
The look and feel is very similar to version 1, although there is
the possibility of changing the background color, from white to
black and intermediate shades.
A tool bar has been added in the left margin . Question is
"Why?" since it has only five icons. These could have been
placed in the horizontal toolbar which is mostly empty.
The version 2.1.1 icon is the same, although square instead of
round, which enables the two versions to sit side by side in a
toolbar yet remain differentiated.

Arduino V2 begins to support hardware-based debugging, but it is

limited to SAMD boards, neither the Uno nor the Mega boards. See
Hardware-based debugging (page 119).
On the positive side, it has new programmer assist features:

Go to or peek definition (right

<x>Arduino (version 2)[AUX]Right click to visualize function declaration</x>

click on a function or variable). This is a very useful feature

which enables you to verify a function's declaration. It will
however not recognize library functions such as strcpy.
<x>Arduino (version 2)[AUX]Provides name completion on defined items</x>Name completion has been

implemented, not activated by default, Arduino calls it Editor

Quick Suggestions.
<x>Arduino (version 2)[AUX]Find/replace supports regular expressions</x>Search supports regular
 expressions - it does do regex captures.
It has a long list of keyboard shortcuts. The comment toggle
keyboard shortcut on selected text works on a French
keyboard.
<x>Arduino (version 2)[AUX]New output window: serial plotter</x>It has a new output window: serial

plotter - very easy to use - just do a [Link](myVar). It will put

myVar's values as a line graph into the serial plotter each time
you do a print on myVar. It beats printing values into the serial
monitor and copy/pasting them into Excel to visualize them as
a graph.

On the negative side:

<x>Arduino (version 2)[AUX]Slow to open</x> Slow IDE open - Loading the V2 IDE is a lot
slower than the legacy version (15 to 30 secs vs. 5 secs with
V1), not good. Visual Studio takes 5 secs., AtmelStudio 12
secs. There is definite room for improvement.
<x>Arduino (version 2)[AUX]Copying text from serial terminal problematic</x>Serial monitor copy problem. It

is possible to copy text from the serial monitor by highlighting

it. The caveat is that a select all (ctrl-A) in the serial monitor
gets the source code above it, not the monitor's content.
Getting text is iffy; I selected all of a serial monitor's content by
highlighting it all and did a ctrl-C; upon pasting, I only got the
contents of the visible part of the text.
<x>Arduino (version 2)[AUX]Editor widows not undockable</x><x>Arduino (version 2)[AUX]Serial terminal is fixed bottom pane, not separate

window</x> The user interface design is rigid; by this I mean that

source code editing windows which should be undockable or
closable. The current setup is a top toolbar which occupies the
entire width of the IDE's window. There are two vertical left-
hand panes to the left of the workspace, the leftmost one
contains 5 icons which select one of 5 panes - both of these
are permanent, yet one needs them very occasionally. Why not
place these 5 icons on top and have the panes open as
temporary panes down in the bottom panel?
<x>Arduino (version 2)[AUX]Problem loading code into ATmega328P-Xmini</x>Loops forever when uploading
 into an ATmega328P-Xmini - Debugging does not work on an
Arduino Uno but that is normal as the Uno does not support
hardware-based debugging. However, I tried the
ATmega328P-Xmini, I did get Arduino V2 to recognize the
board and compile however it got into an infinite loop trying to
upload the executable into the board. Upon shutting down and
reloading the IDE, it asked me for updates on the board. I
responded yes. I then proceeded to compile/upload; result
was:
avrdude: ser_open(): can't open device "\\.\COM14": Access
is denied.

Yet access is not denied to avrdude on a Uno on com 14. Tools

reports Port "Com 14", board info is correct. This debugger board
works perfectly in AtmelStudio.
On other occasions I got:
exec:
"C:\\Users\\agir\\AppData\\Local\\Arduino15\\packages\\arduino\\t
ools\\avr-gcc\\7.3.0-atmel3.6.1-arduino5/bin/avr-g++": file does not
exist
Compilation error: exec:
"C:\\Users\\agir\\AppData\\Local\\Arduino15\\packages\\arduino\\t
ools\\avr-gcc\\7.3.0-atmel3.6.1-arduino5/bin/avr-g++": file does not
exist
And the list of problems goes on and on. You get the gist of it, there
is a problem.

Refactoring disappeared -
<x>Arduino (version 2)[AUX]Refactoring does not seem to be supported any more</x>

Refactoring was in V2.0 but did not work. I could not find it in
version 2.1.1. It seems to have been replaced with Change all
occurrences, but all this does is a replace all, no matter where.
I conclude that there is no longer a refactoring feature. Along
the same line, there was a rename feature, I could not find it
anymore.
<x>Arduino (version 2)[AUX]C-Style /*...*/ comments collapsing inoperant </x>C-Style /*...*/ comments

collapsing inoperant - Version 1.8.19 does collapse C-Style

 /*...*/ comments. For some reason, version 2 does not.
<x>Arduino (version 2)[AUX]#ifdef…#endif pairs collapsible but renders function not collapsible</x>#ifdef…#endif pairs

collapsing - In legacy version (1.8.19) #ifdef…#endif pairs not

collapsible thereby rendering finding unbalanced pairs difficult.
IMHO, since the Arduino IDE does an excellent job collapsing
code by using closing curly braces, it should go a step further
and do the same with #ifdef...#endif pairs.

In new version ( 2.1.1), #ifdef…#endif pairs collapsible but renders

functions and other code blocks not collapsible.

Compilation errors -
<x>Arduino (version 2)[AUX]Click on compilation error in output window does not take you to file/line</x>

The IDE does not allow you to go directly to source code by clicking
on the error in the error window; you still need to get there manually
as in the old version.
Major legacy version problems - As for the two major caveats
of version 1.8.19, fragmentation faults on large applications
and ctrl-Z undo messing up your code on small applications as
well, I cannot say. I would need to work with version 2 for a
while to report on my experience.

There could be a lot more to say on

<x>Arduino (version 2)[AUX]Major advance over version 1, long way to go</x>

Arduino IDE 2.1.1 however, as things stand, it somewhat bridges the

gap between it and AtmelStudio and Visual Studio/Visual Micro, but
it has a long way to go; they are far more programmer friendly tools
as they bring in many more productivity features. The Arduino team
should progressively add new features and hopefully iron out the
problems itemized above. I shall test V2 on a project I am working on
and shall report on it in a future version of this book.

4.
Arduino caveats (legacy version
1.8.19)
On the whole, the Arduino IDE is quite reliable; but, as my
application got big, the Arduino editor bugged seriously, requiring
restarts or code changes.
The legacy Arduino IDE 1.8.19 has three major flaws. I do not know
whether Arduino 2.1.1 is free of these as verifying for them requires
long programming sessions with medium to large programs, which I
have yet to undertake with V2. These flaws are:

The undo
<x>Arduino (caveats)[AUX]After a while, the Arduino IDE gives up, undo ctrl-Z messes up code; fragmentation fault blocking</x>

(ctrl-Z) would, after a while, mess-up my code. I would do a

ctrl-Z and wham, holes would appear in my code and pastes
would show up in odd places. Most of the time, impossible to
unravel. The only solution was to close the IDE not saving
code and reopen it. Of course, I lost all the changes since my
last save.
When doing a build, I would occasionally get a fragmentation
fault which could not be overcome by redoing the build. I
discovered that modifying code, such as adding a useless
macro, could get around the problem; this is cumbersome. And
then, one day, I was really stuck - I just could not get past the
fragmentation fault despite all my efforts. I did not find a cause-
and-effect relationship. This problem triggered my migrating to
AtmelStudio, and later, to Visual Studio/Visual Micro.
The Arduino IDE is tired and gives up - A long list of Java
errors suddenly appears in the output window; you cannot
continue using the IDE. The only recourse is to close the IDE
and restart. Unfortunately, by doing this, you lose all your work
since your last save.

The ctrl-Z undo problem described above is specifically an Arduino

editor problem; fragmentation fault problem seems to be a
compiler/linker issue, hence not an Arduino editor issue; and the
Arduino IDE is tired problem is an IDE issue.
 8.
AtmelStudio

AtmelStudio
AtmelStudio (MicrochipStudio) is a free professional grade IDE; it is
a customized version of Visual Studio for AVR <x>AtmelStudio (general)[AUX]MicrochipStudio is new
name - continue referring to AtmelStudio in book</x><x>AtmelStudio (general)[AUX]Free, professional grade, short learning curve, easy import of Arduino projects</x>

<x>AtmelStudio (general)[AUX]AtmelStudio Arduino compatible in two versions, without and with Visual Micro</x><x>Visual Micro[AUX]AtmelStudio Arduino compatible in

two versions, without and with Visual Micro</x><x>AtmelStudio (general)[AUX]AtmelStudio Arduino compatible in two versions, without and with Visual Micro</x>

<x>AtmelStudio (general)[AUX]IDEs: Arduino, AtmelStudio, Visual Studio, VS Code, PlatformIO, Visual Micro</x><x>AtmelStudio (general)[AUX]Supports hardware-based

debugging</x><x>AtmelStudio (editor)[AUX]Extensive user assist editor features</x><x>Debugging hardware-based[AUX]Atmel ICE - Not successful debugging Uno or

Mega</x><x>Debugging hardware-based[AUX]ATmega328P-Xmini Uno pin compatible supported by AtmelStudio, success using it</x><x>Visual Studio[AUX]AtmelStudio is

Atmel specific Visual Studio</x><x>AtmelStudio (general)[AUX]AtmelStudio is Atmel specific Visual Studio</x><x>Which IDE to work with?[AUX]AtmelStudio best: solid,

professional tool, supports hardware-based debugging</x><x>Debugging hardware-based[AUX]ATmega328P-Xmini Uno pin compatible supported by AtmelStudio, success using

and SAM Atmel microcontrollers; this includes the ones used on

it</x>

Arduino boards (AVR). Atmel has since been acquired by

Microchip who since renamed it MicrochipStudio; however, I shall
continue referring to it as AtmelStudio because most of the YouTube
documentation and forum discussions refer to AtmelStudio. At time
of proofreading this book (May 2023) the latest version is 2020
7.0.2594.
Both AtmelStudio and the Arduino IDE have a large workspace split
horizontally in two: source code above and compiler, linker, upload
messages below. They also have toolbars which contain menus and
icons. This is where the similarity ends. AtmelStudio excels in
intelligent, practical programmer assistance, and is more robust than
the Arduino IDE. No more ctrl-Z undo problems. No more lost time
and aggravation with fragmentation faults. To be fair, I should
mention that Arduino V2 may have solved these problems; verifying
this is in my TODO list.
The AtmelStudio installation lasts approximately 5 minutes. It is
hassle free. Since I did most of my work with Build 1391, some
problems may have been fixed since, but they are still listed in
AtmelStudio caveats (page 62) for reference. Although I have tried to
update this book's content on AtmelStudio, some differences
between info in this book and the current Build 2594 or later may
remain.

1.
Why adopt it for your Arduino
projects
AtmelStudio boosts productivity, reduces programmer stress, and
helps build robust code. Its learning curve is gentle; it is extremely
programmer friendly; and it has lots of intelligent productivity
features. It displays programming errors which you can click on to
navigate to the faulty line. Furthermore, it has a powerful find/replace
mechanism which supports regular expressions. AtmelStudio's
characteristics are described in the Editor features (page 49).
Importing Arduino projects with AtmelStudio is a breeze. It is
unbeatable if you are limiting your work to Arduino. It does have
several minor drawbacks though: it does not have an integrated
serial monitor such as the Arduino IDE has; and there is no upload
facility - you must create one (easy but needs to be done - see
Missing features page 39). This being said, the Visual Micro (page
84) plugin adds these missing features to it, plus Serial debugging
(page 125).
If you choose to adopt AtmelStudio, please note that it requires the
Arduino libraries; this is why, irrespective as to whether you program
with the Arduino IDE or not, you need to download/install the Arduino
distribution (page 31).
I easily managed to import, compile/link, and upload sketches with
AtmelStudio. However, after working with the Arduino IDE, I found
AtmelStudio daunting - lots of icons in the toolbar, many of which
were mysterious, and windows galore, plus a vocabulary which
proved rather confusing (solution, project, task, build, rebuild, clean).
But, after a while, I found that I was using just a few. The ones I do
not use are still there; I just ignore them.
As I dug into AtmelStudio, I began to discover features which made
life easier than working with the Arduino IDE. I became far more
productive due to its programming aids. Furthermore, it was more
robust: I was free from the Arduino IDE's fragmentation faults, and
from the mess the undo (ctrl-Z) could cause (Arduino caveats
(legacy version 1.8.19) page 36).
One important feature which incited me into adopting AtmelStudio is
that it provides hardware-based debugging (see ATmega328P-Xmini
page 120 - works extremely well). Icing on the cake, if you use the
Visual Micro plugin, you get a serial monitor, seamless upload into
the microcontroller, and serial debugging (Serial debugging page
125).
<x>Interoperability[AUX]AtmelStudio import hassle free</x><x>AtmelStudio (interoperability)[AUX]Interoperability between Arduino IDE, AtmelStudio, PlatformIO, Visual

Micro</x><x>AtmelStudio (editor)[AUX]Extensive user assist editor features</x><x>AtmelStudio (editor)[AUX]Syntax checking - color goes black when there is a programming

error</x><x>AtmelStudio (find/replace)[AUX]Supports whole word, regular expressions, past searches</x><x>Regex (general)[AUX]Many tools contain a regex engine -

The learning curve proved to be gentle,

AtmelStudio and PlatformIO find/replace, Perl, Awk, Word (wildcards)</x>

far less than I had anticipated. AtmelStudio can seamlessly import

Arduino sketch files to create Arduino projects. Granted, the import
procedure requires a little bit of attention (Importing an Arduino
project (AtmelStudio) page 40); but, once you have gotten over its
quirks, working with AtmelStudio is a treat. I imported my beehive
weighing system project. To my great surprise, despite its being
rather large (35+ files, 15000 lines), it compiled, linked, uploaded
into the ATmega2560, and ran flawlessly. To top it all, I managed
interoperability with the Arduino IDE thereby enabling me to use a
single set of source code files. I could then switch seamlessly from
one IDE to the other.
This being said, code size and code quality must be verified on a
case-by-case basis. If there were such issues, you could always opt
to compile/link your final release with the Arduino IDE. See Code
size (IDE comparisons) (page 232).
To be sure, the Arduino IDE remains a fantastic tool. It takes all the
hassle out of compiling, linking, and uploading code into Atmel
microcontrollers. It just cannot get simpler than that. Had it not
existed, I do not think I would have gotten into developing apps for
Arduino.
It has now been two years. AtmelStudio has boosted my productivity
and made coding less stressful by signaling errors of all sorts before
compiling. These benefits helped me concentrate on the algorithm
rather than on the mechanics of C++.
AtmelStudio's features are described in the following sections:

Managing directories (page 39) - Projects may be managed

individually or as a group (this is what solution refers to).
Importing an Arduino project (AtmelStudio) (page 40) is easy
but there are a few AtmelStudio quirks you should be aware of.
<x>Interoperability[AUX]Seamlessly switch between AtmelStudio, Visual Studio, PlatformIO, Arduino IDE</x> Interoperability

(page 21) - The objective is to seamlessly switch between

AtmelStudio and the Arduino IDE to work on the same set of
source files. Achieving interoperability between these two IDEs
turns out to be unexpectedly easy but there are constraints.
<x>AtmelStudio (editor)[AUX]Inline tasks provide multiple-choice text box to rename, create, declare, etc.</x> Editor features

(page ٤٩) is where AtmelStudio excels. It really improves

programmer productivity by an order of magnitude over that of
the Arduino IDE.
<x>Code size[AUX]AtmelStudio code size by default ٤٠٪ too large, needs -flto flag</x><x>-flto flag[AUX]Code size depends on -flto flag; reduce by ٤٠٪</x>

Code size (build options)

<x>AtmelStudio (caveats)[AUX]Code size by default ٤٠٪ too large, needs -flto flag</x>

(page ٢٣٣) is an important issue. Since the three

environments, AtmelStudio, the Arduino IDE, and PlatformIO
use the GNU compiler, code size should be independent of the
tool you use. I ran tests on a small application; the Arduino IDE
and PlatformIO produced equivalent RAM code sizes;
AtmelStudio however produces considerably larger code sizes.
It suffices to include the -flto compiler option to bring it down to
the other IDEs code sizes. However, there were significant
differences relating to flash memory use.
Hardware-based debugging (page 119) - This is a domain in
which AtmelStudio excels. Just plug in an ATmega328P-Xmini
and do a build (the upload is automatic). Place breakpoints in
your code and run the application. When it stops because of a
breakpoint, look at the variables, change their values if you
need to. A world of debugging possibilities opens up.
 AtmelStudio caveats (page 62) - AtmelStudio, like all tools, has
flaws, bugs, and inadequacies. You will find a few, some more
serious than others. I discovered these using Build ١٣٩١.
Some may have been fixed with Build ٢٥٩٤; I have not
checked which bugs have lingered on.
<x>AtmelStudio (documentation)[AUX]Atmel (Microchip) Web site two inline doc packages: Getting started and

Documentation - Atmel (Microchip) proposes two inline

videos</x>

documentation packages: Getting Started with AtmelStudio

and AtmelStudio Documentation video tutorials. See
Bibliography - AtmelStudio (now MicrochipStudio) (page 335).

If you are to work on large C/C++ programs for Arduino, you should
use AtmelStudio, Visual Studio, or PlatformIO. I prefer AtmelStudio +
Visual Micro or Visual Studio + Visual Micro (see Visual Micro page
84). They enhance your productivity and programming will be a lot
more comfortable. Their supposedly steep learning curve is the one
factor which has dissuaded programmers from adopting them - this
is unfortunate. Based on my experience using AtmelStudio, I confirm
a gentle learning curve and an extremely easy import of Arduino
projects. These two factors alone make it an excellent choice. Icing
on the cake, AtmelStudio supports hardware-based debugging; I
tested it with the ATmega328P Xplained Mini board, works like a
charm. See Debugging (page 113).
So, roll up your sleeves, and get started with both AtmelStudio and
Visual Studio + Visual Micro.

2.
Missing features
<x>Visual Micro[AUX]Three identical Arduino plugins: for AtmelStudio and for Visual Studio (2019 and 2022)</x><x>AtmelStudio (general)[AUX]Visual Micro brings

AtmelStudio lacks a couple of features

AtmelStudio missing features: upload tool, serial monitor, serial debugging</x>

which the Arduino IDE provides, namely: an upload tool and a serial
monitor. These force the programmer into creating an external tool to
upload code into the microcontroller and to use an external serial
monitor to visualize program output. These missing features can be
supplied by incorporating the Visual Micro (page 84) plugin into
AtmelStudio (note that an identical plugin is available for Visual
Studio).

3.
Managing directories
The key to
<x>AtmelStudio (file management)[AUX]Differentiate projects (managed individually) from solutions (managed as group of projects)</x>

understanding AtmelStudio's project directories lies in knowing the

difference between projects and solutions. When creating a new
project, AtmelStudio opens a text box containing three fields: name,
location, and solution name.

<x>AtmelStudio (interoperability)[AUX]AtmelStudio's project name is Arduino project name</x> Name is the project's
name. You should name the project the same as the Arduino
sketch you are importing, without the .ino extension.
Location - This is your root directory into which you will
presumably locate all your AtmelStudio Arduino projects. This
directory must already exist, it will not be created. I use a
directory named AtmelDev.
<x>AtmelStudio (file management)[AUX]Differentiate projects (managed individually) from solutions (managed as group of projects)</x>Solution

name - If you want to group your projects into one folder, give
the projects you are importing a unique solution name. You
may want to group projects according to the category they
belong to. For example, in my beehive weighing system, I may
have several categories: frameworks, devices (GSM, DS3231
clocks...), application domain (beehive, Asian wasp, etc.).

When naming projects, two possible scenarios present themselves.

Let us assume you have defined a root directory named AtmelDev
(location) and that you shall be importing two Arduino sketches:
[Link] and [Link]. In the first scenario, projects are
not grouped into a solution. Another way of saying this is that each
project is in its own solution, hence solution name and project
name are the same. This is the directory structure you would get:
 AtmelDev
MemMgt <-- solution
ArduinoCore
MemMgt <-- project
DataGroup <-- solution
ArduinoCore
DataGroup <-- project
In the second scenario (multiple projects, one solution), you decide
to group the projects into one solution named Frameworks - to do
this, click on the Create Project from Arduino Sketch checkbox.
This is the directory structure you will get:
AtmelDev
Frameworks <-- solution
ArduinoCore
MemMgt <-- project
DataGroup <-- project
<x>Interoperability[AUX]PlatformIO/AtmelStudio/Visual Micro interoperability requires AtmelStudio one solution per project scenario</x>The second

scenario is a lot cleaner, but it prevents interoperability with

PlatformIO. The one project, one solution scenario (first one above)
is preferable in that it enables interoperability with PlatformIO (page
98).

4.
Importing an Arduino project
(AtmelStudio)
<x>AtmelStudio AtmelStudio provides a Create
(interoperability)[AUX]Importing Arduino project procedure</x>

project from Arduino sketch option. When choosing this option, the
infrastructure required to compile/link Arduino sketches is created,
including precompiled libraries. Proceed cautiously; a little
experimentation with AtmelStudio will enable you to understand the
distinction between projects and solutions. These determine the
directory structure. See the Managing Directories above.
When importing an Arduino project with AtmelStudio, a two-directory
structure gets created: a root directory (MyProject directory) contains
some AtmelStudio files and a source file subdirectory (MyProject
subdirectory) which contains the .ino file and .h/.cpp files. If you
desire to extend AtmelStudio with Visual Micro to get a serial
monitor, embedded upload, and serial debugging, you need to resort
to the classical one directory structure. AtmelStudio with Visual Micro
works directly from the subdirectory described above. It so happens
that Visual Studio with Visual Micro works the same way (Visual
Studio page 79).
An AtmelStudio non Visual Micro Arduino import will create the
following directory structure, MySketch being the name of the
Arduino IDE project being imported. AtmelDev is a convenience
directory I have defined to group all AtmelStudio Arduino work.
AtmelDev
...
MySketch
ArduinoCore
MySketch
...
The main MySketch directory contains the ArduinoCore subdirectory
(the AtmelStudio Arduino core build). The MySketch subdirectory
contains the [Link] file (the .ino equivalent). After having done
an AtmelStudio import of your Arduino application, you need to
copy/paste all the .h/.cpp files from the original Arduino IDE directory
to the AtmelStudio source files subdirectory, and then load them into
your AtmelStudio project via the AtmelStudio View --> Solution
Explorer window. AtmelStudio does not automatically add files to a
project. They must be explicitly added to the project which allows
.h/.cpp files to be physically in the source files directory and not
included when doing a build. After having copied the .h/.cpp files into
the source file directory, you need to handle a few more details to
upload your program into the microcontroller.
Details concerning importing Arduino sketches into AtmelStudio are
grouped in the following sections:

<x>AtmelStudio (file management)[AUX]Differentiate projects (managed individually) from solutions (managed as group of projects)</x>

Managing directories (page 39) - It is important to understand

how AtmelStudio handles project directories when importing an
 Arduino sketch. Projects are managed individually (project), or
as a group (solution).
<x>Interoperability[AUX]AtmelStudio import hassle free</x>Importing a sketch is easy, there are

subtleties though - read on. See Importing an Arduino project

(AtmelStudio) (page 40).
<x>AtmelStudio (build)[AUX]External tool for upload not needed with Visual Micro</x><x>AtmelStudio (interoperability)[AUX]External upload avrdude tool

Uploading code -
needed</x><x>avrdude[AUX]AtmelStudio requires external custom avrdude upload tool , not with Visual Micro</x>

Unlike the Arduino IDE, uploading an executable is handled by

an external command which needs to be created (AtmelStudio
calls them Tools). This is done by copying, modifying, and
pasting an Arduino IDE avrdude command to create an
external tool. See Upload code into the microcontroller (create
external tool) (page 44).
<x>AtmelStudio (editor)[AUX]Arduino monitor displays output</x> Interface with Arduino applications

(page 48) - AtmelStudio does not provide a serial monitor such

as the one provided by the Arduino IDE. An external monitor is
required. Opening an Arduino sketch and using its serial
monitor is an easy practical solution.
Visual Micro for AtmelStudio (MicrochipStudio) (page ٨٤) -
Note that if you decide to use AtmelStudio with a Visual Micro
plugin, you do not need to create an upload tool, you get an
external monitor, and you may try Serial debugging (page ١٢٥).

The AtmelStudio import Arduino sketch process starts by opening

AtmelStudio and selecting File-->New-->Project. A window pops up
with a selection asking you for the type of project you may want to
create, one of which is Import an Arduino sketch. This New Project
window contains three fields, name, location, and solution name, as
indicated in the Managing directories (page 39).
Suppose you want to import an Arduino sketch named MemMgt into
AtmelStudio following the one project, one solution scenario. Inside
the new project window, define the Name and Location fields - do not
tick the Solution Name checkbox since you want the solution name
to be the same as the project's name, which is the default. After
doing this, select Create Project from an Arduino sketch and click on
OK. A new popup text box opens with four fields: sketch file, Arduino
IDE path, board, and device.

Sketch File - When

<x>AtmelStudio (caveats)[AUX]Importing Arduino project - directory navigation bug on searching .ino file</x>

importing an Arduino project, you are asked for the .ino file
path. There is a button to browse the directories; do not use it;
there is a bug. See AtmelStudio caveats (page ٦٢).
The Arduino IDE installation is C:\Program Files (x٨٦)\Arduino
by default.
<x>AtmelStudio (caveats)[AUX]Select device board does not seem relevant when creating Arduino project</x>Board - Choose

your board from the popup options. This choice does not
appear to be modifiable. If the app is to be later uploaded to
another board, a provision for specifying a new board does not
seem to exist. The text field and board choice are there,
however, I have not found it to be relevant during development.
If you switch boards, from an Arduino ATmega٢٥٦٠ to a Uno
for example, there is a device choice in the project properties
window Project --> Properties --> Device. This enables you to
change from ATmega2560 to ATmega328P, but you will not
find an option to define a new board. This leads me to
conclude that setting the board in the Create Project from an
Arduino sketch window is a carryover from an earlier
AtmelStudio or a provision for future enhancements.
Device - Select your device from the list. The device target can
be reset via the Project Properties menu. But this poses a
major problem - see Unable to generate code for new
microcontroller (page 75).

You are done. Click OK. AtmelStudio will run for a while, create the
directories, and finally it will create a [Link] file in the project
source files directory. It will copy into this file the contents of the
original [Link] file. Next, look at the directory structure. This is
what you will find:
 ATmelDev or whatever directory you chose to put your projects in.
MemMgt
ArduinoCore
MemMgt
...
AtmelStudio will have created an ArduinoCore directory which
contains all the pre-compiled libraries you need, as well as an
Arduino project files directory inside the project directory whose
name is the same as the original MemMgt Arduino IDE project. It will
have created a file named [Link] in which you will find the
contents of your [Link] file with a comment block and a #include
as shown below. This file will be displayed in the main work window
of AtmelStudio. You will find it is identical to your [Link] file,
with a few extra lines at the top:
/*Beginning of Auto generated code by AtmelStudio */
#include <Arduino.h>
/*End of auto generated code by AtmelStudio */
You should replace the <Arduino.h> include with a "Globals.h"
include, assuming that your Globals.h file contains #include <Arduino.h>.
You have not quite finished yet as two more tasks need to be
completed:

Transfer your .h/.cpp files - Copy all the Arduino IDE project
source files to the AtmelStudio project source files subdirectory
MemMgt/MemMgt.
<x>AtmelStudio (build)[AUX]Solution Explorer - add/choose existing Item</x>Specify which .h/.cpp files

define your AtmelStudio project - open the AtmelStudio

Solution Explorer (View-->Solution Explorer). Right click on the
bold MemMgt caption. Go down to Add, then choose Existing
Item. A file selection window opens, choose the files you want
to add to the project - all your .cpp files and .h files. Unlike the
Arduino IDE and PlatformIO, which automatically add to the
project all existing .h/.cpp files in the project files folder,
AtmelStudio requires that you define which ones you want in
your project.

You are now ready to go. Compile and link! You have done this with
a small sketch to get a feel of things. AtmelStudio should report no
errors assuming the Arduino application you just imported compiled
error free. Chances are you will get lots of error messages of the
sort:
Invalid conversion from 'void*' to 'DateTime*'[-fpermissive]
The problem at this stage of the import process lies in passing a
DateTime pointer to a function which expects a void pointer. This is
acceptable but not quite legal. AtmelStudio considers that it is
dangerous and reports it as an error. For better or for worse, the
Arduino libraries are a little lax, requiring that the compiler be
somewhat lenient.
<x>AtmelStudio (build)[AUX]Add -fpermissive and -flto flags compiler flags</x><x>-fpermissive[AUX]AtmelStudio (import Arduino project) - add flag to C/C++ compiler

options</x><x>-fpermissive[AUX]Converts compiler errors into warnings, build completes</x><x>-fpermissive[AUX]Included by default in Arduino IDE, PlatformIO, not

By default, AtmelStudio adopts strict compiler rules. The

AtmelStudio</x>

solution lies in having the AtmelStudio compiler be a little lenient.

The -fpermissive option must be added to the C and C++ compiler
options via Project--> Properties. Clicking on this menu item displays
a window with a list of tools and sections for each. Go to AVR/GNU
C Compiler, choose Miscellaneous and add the -fpermissive option
in the Other flags box. Do the same for the C++ compiler, the net
effect being that instead of reporting the void* pointer problem as an
error, hence a failed compilation, it reports it as a warning in the
Warnings window, hence a successful compilation.
In passing, you might want to look at the warnings options, one of
which in particular is interesting:
Warn undefined identifier in an #if directive (-Wundef).
<x>AtmelStudio (build)[AUX]Add -Wundef to detect unused macros</x>Activating this option may render the

output excessively verbose, since you may have commented out

debugging print macros. It could, however, prove useful since an
undetected undefined macro could be the source of an elusive bug.
If you are swamped by the number of messages, most of which
begin with _DEBUG_, copy/paste the output into a file and use Excel to
sort them so that you could focus on those which do not start with
_DEBUG_.
AtmelStudio will have accomplished a number of tasks for you, the
three main ones being:
 It has created a directory hierarchy to contain your project
inside an AtmelStudio compatible structure according to
scenario # 1 described in the Managing Directories above: 1
project, 1 solution.
It has created an ArduinoCore directory which contains
precompiled libraries according to the project's settings.
It has copied the code inside your Arduino sketch [Link]
file into the [Link] file it has created.

The next and final step is to

<x>avrdude[AUX]AtmelStudio requires external custom avrdude upload tool , not with Visual Micro</x>

build and upload the compiled code into your board. AtmelStudio
does provide a build but does not provide an upload facility such as
the Arduino IDE does. Instead, you need to create an avrdude
command to add to External Tools. This is described in the next
section. But, if you installed Visual Micro for AtmelStudio
(MicrochipStudio) (page 84), you need not create such an external
tool.

5.
Upload code into the microcontroller
(create external tool)
<x>AtmelStudio (interoperability)[AUX]External custom avrdude upload tool needed, not with Visual Micro</x><x>avrdude[AUX]AtmelStudio requires external custom avrdude

Having successfully
upload tool , not with Visual Micro</x><x>AtmelStudio (build)[AUX]External tool for upload not needed with Visual Micro</x>

compiled and linked your project, you need to upload the executable
into your microcontroller. The Arduino IDE does this automatically
when you click on the build and upload button after selecting the
comm port, the board, and the microcontroller. AtmelStudio has no
provision for doing this, it requires that you create an external
command for you to invoke. You must create one via Tools--
>External Tools. In a nutshell, you must create an avrdude command
(AtmelStudio calls it a tool) for you to invoke when you decide to
upload the executable into the microcontroller. This being said, if you
installed Visual Micro for AtmelStudio (MicrochipStudio) (page 84),
you need not create such an external tool.
Creating an avrdude command is rather simple. Just copy whatever
the Arduino IDE is doing when uploading, undertake modifications in
notepad or the like, and add the modified command to the
AtmelStudio External Tools list. Now for the details.
<x>Arduino (upload)[AUX]Validate verbose checkbox in options to show details of avrdude upload command</x><x>avrdude[AUX]Arduino verbose details upload

Start by getting Arduino to display what it is doing. Behind the

command</x>

scenes, when uploading, the Arduino IDE calls avrdude with the
appropriate parameters. You can see this happening by choosing
verbose in the Arduino IDE's preferences window. Go to the Arduino
IDE's File-->Preferences. There is a checkbox called Show verbose
output during - click on it to turn this particular option on. Upload your
sketch.
Verbose means lots of text. While avrdude is uploading your
executable into your microcontroller, you will indeed get lots of text in
the bottom section of the Arduino IDE workspace. Scroll down to the
end, you will see a Thank you. Isn't it being polite? Scroll back to the
top of the messages section - the third line starts with something like
C:\Program Files (x86)\Arduino..., it is quite long. This is the avrdude
command you want. Copy it into notepad. Here is what I got:
C:\Program Files (x86)\Arduino\hardware\tools\avr/bin/avrdude -
CC:\Program Files (x86)\Ardui\hardware\tools\avr/etc/[Link]
-v -patmega2560 - cwiring -PCOM5 -b115200 -D -
Uflash:w:C:\Users\agir\AppData\Local\Temp\arduino_build_981991
/[Link]:i
It is well worth the effort to understand what it contains. Notice that
all the options start with a dash followed by a case sensitive letter
and the option's parameter, such as -patmega2560. I have broken
up the line into its individual components and have added spaces to
render the options more readable, as follows:

<x>avrdude[AUX]Full path of [Link] and [Link]</x><x>Arduino (upload)[AUX]Full paths of [Link] and

C:\Program Files
[Link]</x>

(x86)\Arduino\hardware\tools\avr/bin/avrdude is the full path of

the avrdude program.
-C C:\Program Files
 (x86)\Arduino\hardware\tools\avr/etc/[Link] is the full
path of the avrdude configuration file [Link]. Careful
with -CC:\pro... The first -C is an avrdude option. The second
C (in C:\pro...) is the beginning of the file's path, which begins
with disk C:.
-v - Disable verify/check when uploading.
-p atmega٢٥٦٠ - the target microcontroller.
-c wiring - whatever that means.
-P COM٥ - the serial port the board is connected to.
-b ١١٥٢٠٠ - baud rate for serial communications with the
board.
-D - Disable auto verify/check when uploading data. I do not
know what it does, but it needs to be there.
<x>Arduino (upload)[AUX]Arduino build output</x><x>Arduino (build)[AUX]Compile/upload uses bootloader to upload program</x>-U

flash:w:C:\Users\agir\AppData\Local\Temp\arduino_build_٩٩٣٢
٥٦/[Link]:i
File to be uploaded, the compile/link output.

The avrdude command above contains items which never change;

however, three of these need to be customized for each AtmelStudio
upload tool as follows:

<x>AtmelStudio (build)[AUX]AtmelStudio upload tool parameters: identify microcontroller, define COM port, baud rate</x><x>AtmelStudio (interoperability)

Microcontroller name: -p
[AUX]AtmelStudio upload tool parameters: identify microcontroller, define COM port, baud rate</x>

atmega2560. You will find microcontroller names by googling

avrdude options.
Comm port: -P COM5. Simple: COM1, COM2…
Baud rate: -b 115200. Simple 9600, 19200, etc.

Arduino adjusts these automatically according to the settings you

have defined. AtmelStudio will need one distinct external command
for each context combination: microcontroller name, comm port,
baud rate. There are ways of avoiding this constraint - they are
explained a little further on.
The Arduino IDE's avrdude command you copied in notepad needs
to be modified because AtmelStudio requires that file names be
enclosed in double quotes. The avrdude command above contains
two file names, the avrdude configuration file's full path
([Link]) and the .hex file's full path (your program to upload:
the -C and the -U options):

The avrdude configuration file full path for AtmelStudio should

be -C "conf_file". The following Arduino IDE's output needs to
be modified:
-CC:\Program Files
(x86)\Arduino\hardware\tools\avr/etc/[Link]

Careful with the -CC above: the first C is an avrdude option -C, the
second C is the beginning of the full path of the configuration file
"C:\Program... ". This -C option requires that the full file path be
enclosed in double quotes, change the line as follows:
-C "C:\Program Files
(x86)\Arduino\hardware\tools\avr/etc/[Link]"

Likewise, the compile/link output command full path should

be -Uflash:w:"full file path":i. The following Arduino IDE's
output needs to be modified:
-
Uflash:w:C:\Users\agir\AppData\Local\Temp\arduino_build_9
93256/[Link]:i

Insert a quote just before C:\ thus "C:\ and a quote just after hex
thus hex":i.
The -U option becomes:
-U
flash:w:"C:\Users\agir\AppData\Local\Temp\arduino_build_99325
6/[Link]":i
The file path in the -U option defines where the Arduino IDE's puts
its .hex file. AtmelStudio puts it in a different place, which requires
additional work.
<x>AtmelStudio (build)[AUX]$ProjectDir, $TargetDir, $TargetName define .hex file location and name</x><x>AtmelStudio (file management)[AUX]$ProjectDir, $TargetDir,
 We are not
$TargetName define .hex file location and name</x><x>AtmelStudio (file management)[AUX]AtmelStudio target release: Debug or Release</x>

quite finished. The [Link] file is always in the same place, and
therefore independent of the project's location. This is not the case
with AtmelStudio's [Link] file, the compile/link executable.
Theoretically you would need one command for each project since
the name will differ from project to project. Fortunately, AtmelStudio
provides environment variables which you may use. Three of these
are particularly pertinent: ProjectDir, TargetDir, and TargetName.
Note that the .hex file can be located in two locations: Debug and
Release.

The Arduino IDE uses a temporary location created on the fly:

"C:\Users\agir\Appdata\...\[Link]"
AtmelStudio uses the project's location: a Debug or Release
directory depending on the configuration setting:
"D:\ArduinoDev\AtmelDev\MemMgt\MemMgt\Debug"
"D:\ArduinoDev\AtmelDev\MemMgt\MemMgt\Release"

The .hex file's file path may now be created via the two environment
variables as follows:

$ProjectDir - Full path name of the project:

"D:\ArduinoDev\AtmelDev\MemMgt"
$TargetDir - Combines the project directory ($ProjectDir ) with
the target directory (Debug or Release).
$TargetName - Name of the target: "MemMgt"

The full .hex file path name, based on environmental variables, is

now generic:
$( TargetDir)\$( TargetName).hex
hence the -U option of the upload command becomes
-Uflash:w:"$( TargetDir)\$( TargetName).hex":i
For those not in the know, environmental variables are system wide
macros which applications may define and use. Thus, AtmelStudio
has defined three such macros: $ProjectDir, $TargetDir, and
$TargeName. They facilitate defining what to upload.
Allow me one last word while you are still in notepad. The command
must be a one-line command. If you, like I, have decomposed the
command you copied from the Arduino IDE's output into several
lines to make the changes manageable, you must reassemble the
pieces into one line after having made the changes, thereby
completing modifications to the avrdude command, but remember,
you are still in notepad.
Having modified the avrdude command in notepad, the next step is
to create an AtmelStudio external tool. Open Tools-->External Tools;
it contains three sections:

List of existing tools - This is

<x>AtmelStudio (build)[AUX]External tool for upload not needed with Visual Micro</x>

a pane which displays existing external tools, it is useful in that

existing commands may be used as a model when creating the
same command with minor differences, such as upload with a
different device, a different comm port, a different baud rate,
etc.
Four text boxes - Title, Command, Arguments, Initial Folder
Four check boxes - Use Output Window, Prompt for
Arguments, Treat output as Unicode, Close on
exit(prechecked).

That is a lot of content, but it is simpler than it looks. Here is the info
you will need to complete to create an external tool:

Title - You need to give your new command a name, such as

Load to ATmega2560 Com5 or Load to ATmega328P Com3.
Prompt for Arguments - An alternative, provided you have
checked the Prompt for Arguments check box, consists in
manually defining options such as defining the port (-PCOM5)
or the baud rate (-b115200) or the microcontroller (-p
ATmega328P) when invoking the command. This would allow
you to create commands which are more generic, hence the
command's name would be generic. It is up to whatever works
best for you.
Command contains the program which the external tool
 triggers. You want to use avrdude. It needs to be invoked via
its full path. Thus:
C:\Program Files
(x86)\Arduino\hardware\tools\avr\bin\avrdude

If you placed your Arduino somewhere else, the path to avrdude

would be different.

Arguments - This line contains everything that avrdude needs.

It is the command you modified in notepad - copy/paste the
arguments list.
Initial Folder - Leave blank. It is possibly a useful option;
however, I have not found any need for it.
Use Output Window - Activate it to redirect the output into the
AtmelStudio workspace. Leave the other check boxes with
their default values unchanged.

Click OK. You have just created an external command to upload your
program into the microcontroller. You are all set to develop with
AtmelStudio. There is an exception though. If you are working with
the ATmega328P-Xmini (page 120) hardware debugger board
instead of an Arduino board, the upload process is automatic. You
do not need an external tool such as described above. There is a
second exception, if you are using AtmelStudio with Visual Micro,
you will have an upload possibility from within the vMicro choices
(see Visual Micro for AtmelStudio (MicrochipStudio) page 84).
You can now modify your code to get it to compile and link by doing
a Build Solution; after which, upload the executable via the
appropriate external command amongst the commands you have
created. Make sure that the target comm port is not being used by a
serial monitor. If so, close it. The upload messages should tell you
whether the upload was successful or not. Open the serial monitor.
Define the baud rate. Run. Examine the output and modify your code
accordingly. Repeat the process. But be careful, there are a couple
of caveats:
 The serial monitor must be closed when uploading so that
avrdude can use it. Otherwise, you will get the following cryptic
message:
[Link]: stk500v2_getsync(): timeout communicating
with programmer
[Link] done. Thank you.
<x>AtmelStudio (caveats)[AUX]avrdude fails to upload code into chip</x>When uploading code into the

microcontroller via your external command, avrdude will

sometimes fail to upload your code into it. Yes, avrdude fails
occasionally, and no particular reason is given. It is a little bit
like the occasional segmentation faults. The problem is that
after uploading, a quick cursory look at the avrdude message
leads you into believing that all went well. But no, there could
have been an error. The problem is that the error message, if
present, is not readily visible because it has scrolled up and
out of sight. See AtmelStudio caveats for a more detailed
description of this problem.

That is it. You have completed the import of an Arduino sketch. You
have compiled, linked, uploaded it into the microcontroller, and
visualized its output. You are all set to develop Arduino applications
with AtmelStudio. What is more, you are about to establish
interoperability between AtmelStudio and the Arduino IDE so that
you could seamlessly switch from one to the other. See
Interoperability (page 21).
I shall add a final note concerning AtmelStudio's External Tools. An
external tool is any executable (along with its parameters) you might
want to run from AtmelStudio. You can put any executable in the
command text box. Try this by creating a tool which you shall name
Notepad. Put notepad's full file path in the command box
(C:\Windows\System32\[Link]), click in the prompt for
arguments check box, and click OK - you are all set. You can now
call notepad directly from the AtmelStudio's tools menu.

6.
 Interface with Arduino applications
AtmelStudio does not provide an external monitor; you must use an
external one. You can use the Arduino serial monitor, unless you
have incorporated the Visual Micro for AtmelStudio
(MicrochipStudio) (page 84) plugin.
<x>AtmelStudio (editor)[AUX]Arduino monitor displays output</x>To visualize AtmelStudio program output,

open any sketch with the Arduino IDE; set it to the correct board,
microcontroller and comm port, and use its console (serial terminal).
Do not worry about what sketch happens to lie in the Arduino IDE
you just opened. The console and the sketch are unrelated to each
other. When you open the console or do a restart, the console's
output is independent of the Arduino IDE's sketch. All it does is reset
the microcontroller and get it to run whatever program it contains.
That is why you can build and upload a program with AtmelStudio
and visualize the output with any Arduino IDE's serial monitor. One
detail merits attention though: when uploading the application's
executable from AtmelStudio into the microcontroller, the comm port
has to be free, which means that the serial terminal you are using to
visualize output must be closed. When working with AtmelStudio, the
sequence is: close the console; modify code; build and upload into
the microcontroller; open the console; visualize output. You need not
do this with the Arduino IDE and PlatformIO because the console or
terminal can remain open when you do a build/upload. If you are
using AtmelStudio with Visual Micro, the above does not apply since
Visual Micro provides a built-in monitor (see Visual Micro for
AtmelStudio (MicrochipStudio) page 84).

7.
Editor features
Both AtmelStudio and the Arduino IDE
<x>AtmelStudio (editor)[AUX]Extensive user assist editor features</x>

are classic C/C++ programming tools. By classic, I mean that they

can be characterized by a workspace containing files to be worked
on, above which sits a toolbar containing icons and menus. The
similarity between the two IDEs ends here in that AtmelStudio can
do just about everything the Arduino IDE can, and much, much
more. Its multi-window capabilities and programming assistance are
where AtmelStudio excels. These programming aids are contained in
two embedded packages: VAssist and Intellisense.

<x>AtmelStudio (editor)[AUX]VAssist has many help mechanisms</x> VAssist, short for Visual Assist, is
accessed via a top-level menu item. It contains features which
make programming more comfortable and productive. It
provides shortcuts such as go to implementation, find all
locations where a variable is used, go to member function
when the cursor is inside a class function, etc. You should
review the VAssist menu items one by one, so that you know
that they exist and know what they can do for you.
<x>AtmelStudio (editor)[AUX]Intellisense complementary to VAssist</x>Intellisense is similar to VAssist in

that it brings diverse aids. It is accessed via the bottom of the

Edit top level menu. One example of Intellisense's features is
adding customizable documentation to a function via an
enclosed set of comments /*...*/ just above a function's
declaration. The result is that when, somewhere in your code,
you place the cursor on the function and do ctrl-alt-space, a
pop-up opens the function declaration and the accompanying
help you may have defined. You may also display such help
with library functions. Try it on rand().

What follows is a non-exhaustive list of AtmelStudio smart editor

features not implemented by the Arduino IDE. The new Arduino IDE
V2 does provide some of these features but it falls far short as to
what AtmelStudio does. These features are not differentiated as
being VAssist or Intellisense related; they are the ones I use often:

Workspace
<x>AtmelStudio (editor)[AUX]Workspace contains two splittable unpinnable work windows and Solution Explorer</x>

(page 51) - Whereas the Arduino IDE workspace has only one
window to display code with fixed tabs to select the file to be
displayed, AtmelStudio's workspace may be split into two
workspaces, side by side or one above the other, each one
containing several source code files you have selected. Each
workspace has movable tabs to display one file or another.
Thus, you can position two workspaces side by side, one with
.cpp files, the other with .h files. You could be coding functions
in a .cpp file with the corresponding .h file sitting nearby for
reference. Furthermore, windows can be split in two so that
you can work in one part of a file as you are looking at another
part. Files can be unpinned so that they are displayed
independently in their own window. This is particularly useful if
you are using large screens or multiple screens.
<x>AtmelStudio (editor)[AUX]Intelligent window scrollbars provide quick access to various tasks</x> Intelligent window

scrollbars (page 52) - The vertical bars in the source code

windows contain several useful tabs: gray ones to display the
window's visible area relative to the total code, a blue one to
indicate the cursor's position for quick access when it is
outside visible code, orange ones which locate text being
searched for via the search/replace text box, as well as a few
others.
<x>AtmelStudio (editor)[AUX]Multiple-choice quick access to functions</x> Quick access to functions (page

53) - When a file is displayed in a window, a small horizontal

selection box located in the top left corner of the window
enables accessing global variables and functions in the file.
Selecting one of these positions the cursor directly onto the
selected variable or function. This is useful to quickly get to
where you want to go. There is no need to use the find/replace
dialog box to go to the function you wish to work on.
Go to implementation (page 53) - AtmelStudio provides
another quick access to variables and to function declarations
and implementations. A right click on a function's name opens
a small text box containing the function's declarations (.h file)
and implementations (.cpp file). This enables you to visualize
the function's parameter list. If you so desire, click on the one
you want to access.
<x>AtmelStudio (editor)[AUX]Inline tasks provide multiple-choice text box to rename, create, declare, etc.</x> Inline function
related tasks (page 53) - When the cursor is on a function, a
minuscule arrow appears, pointing down below the function's
name. Click on it - it will display a multiple-choice text box
containing tasks: rename, create declaration, show references,
and more.
<x>AtmelStudio (editor)[AUX]Color coding highly customizable</x> Color coding (page 54) is more

than a way to differentiate items from each other, for example:

variables are blue, text is green, strings are red. It also alerts
you that something is wrong by changing an item's usual color
to black.
<x>AtmelStudio (editor)[AUX]Syntax checking - color goes black when there is a programming error</x> Inline syntax checking

(page 55) - As you type code in AtmelStudio, syntax errors

such as forgetting a semicolon, an unfinished for statement, or
an undeclared variable, are underlined in red and their color
will change to black. This allows for immediate correction. You
do not need to wait for the compiler to catch the error. It is as
though a compiler were sitting on your shoulder, ready to point
out that you just made a mistake.
<x>Regex (general)[AUX]Many tools contain a regex engine - AtmelStudio and PlatformIO find/replace, Perl, Awk, Word (wildcards)</x><x>AtmelStudio

The Find/Replace (page 55)

(find/replace)[AUX]Supports whole word, regular expressions, past searches</x>

dialog box supports the whole word concept. It also supports

regular expressions, a powerful and flexible tool not particularly
difficult to master. I strongly suggest you get familiar with it as it
is the key to using Awk and Perl. See Regular expressions
(regex) (page 191). If you happen to use the Adobe tools
InDesign, Photoshop, etc., you may have used the Grep
utility - it is based on regular expressions. Note that the
Arduino IDE V2 find/replace does support regular expressions;
the old legacy version 1.8.19 does not.
<x>AtmelStudio (editor)[AUX]Name completion contextual popup opens</x> Name completion (page 56) -

When typing macros, global variables, functions, and classes,

a text box opens which lists existing items so that you may
complete typing by selecting the one desired. This feature not
only saves on typing, but it also ensures that the item is
spelled correctly. The new Arduino IDE V2 does support name
completion.
<x>AtmelStudio (editor)[AUX]Collapses classes, functions, enums, ifs, whiles, etc. and #ifdef...#endif pairs</x><x>AtmelStudio (caveats)[AUX]Does not support

Code collapsing and indentation (page 57) -

curly braces collapsing</x>

#ifdef...#endif collapsing, which the Arduino IDE does not support,

can be particularly useful if you lose track of #endifs. However, to
be fair, the Arduino IDE is better at code collapsing than
AtmelStudio, since it collapses curly braces which AtmelStudio
does not do. Both IDEs have their code collapsing weaknesses
which can be circumvented. To summarize: both IDEs collapse
high level aggregates such as functions, structs, enums...
AtmelStudio does collapse #ifdef...#endif pairs but does not
collapse curly braces. The Arduino IDE is good at collapsing
curly braces but does not collapse #ifdef...#endif.
Collapse /*...*/ group (page 59) - Arduino 1.18.9 collapses c-
style comments, not version 2.1.1.
<x>AtmelStudio (editor)[AUX]Keyboard shortcut to quickly comment/uncomment highlighted sections</x> Group

comment/uncomment (page 59) highlighted code - Highlight a

group of lines of code and click on an icon to comment or
uncomment them all. (C++ type comment //). Other shortcuts
expand the possibilities to comment bits of code. The Arduino
IDE and PlatformIO also support this.
<x>AtmelStudio (editor)[AUX]Refactoring - rename items throughout app</x> Refactoring (page ٥٩) means

change the name of a variable. You can do this with a

find/replace all but there could be side effects since you are
not controlling each find/replace. With refactoring, you select a
variable, a macro, or a function and give it a new name. It will
replace all occurrences of the old variable with the new name.
Furthermore, AtmelStudio is smart enough to differentiate local
variables from global variables with the same name; syntax
and code logic are preserved - no side effects.
<x>AtmelStudio (editor)[AUX]Compiler one-click access to faulty line</x> Compiler error reporting (page

60) - You will often get a list of errors, each one identified by
file and line number. Double clicking on the error takes you
directly to the location (file and line).
<x>AtmelStudio (editor)[AUX]Navigate to places recently worked on</x> Code navigation (page 61) - A go
 forward and go back to set of buttons enables you to navigate
forward and backward to places you recently were working in.
<x>AtmelStudio (editor)[AUX]Bookmark to navigate back and forth</x> Bookmarks (page 61)- You may, at

times, want to return to specific places in your code. You can

tag these - they are called bookmarks. You can navigate from
one bookmark to another.
<x>AtmelStudio (editor)[AUX]Spell checks strings and comments</x> Spell-checker (page 61) -

AtmelStudio checks spelling inside comments and strings. This

useful feature helps you rectify spelling mistakes.
<x>AtmelStudio (editor)[AUX]Ctrl-G takes you to line # in file</x> Go to line number (page ٦١) - You

may sometimes want to go to a specific line number of a file -

just do a ctrl-G.

The above are just a few of the main productivity aids of

AtmelStudio; they are expanded upon in the pages which follow.

1.
Workspace
AtmelStudio's workspace is truly what the name implies, a space in
which work is to be undertaken. The Arduino IDE's workspace may
be described as a window which enables access to a project's
source files, one at a time, via fixed tabs. AtmelStudio's workspace is
considerably richer. Some of its more salient features include:

The overall
<x>AtmelStudio (editor)[AUX]Workspace contains two splittable unpinnable work windows and Solution Explorer</x>

workspace can be split into two distinct work areas arranged

horizontally or vertically, each of which can contain files,
accessed via movable tabs at the top of the window. Once files
have been added to workspaces from the Solution Explorer,
AtmelStudio allows reorganizing them. This is useful when
coding since the programmer can put .h files in one
workspace, .cpp files in another, and thereby work on a .cpp
file and view the corresponding .h file next to it.
 <x>AtmelStudio (file management)[AUX]Solution Explorer access project components</x> The Solution Explorer may
be added as an additional component of the overall
workspace. It enables managing the project's contents by
adding/removing files. It is also used to load files into one of
the two workspace windows for viewing and editing.
Individual files can be unpinned from their workspace so that
they may be edited independently. This is particularly useful
when working with a large screen or with multiple screens.
Splitting a window allows working on part of a file while looking
at another part of the same file.
<x>AtmelStudio (editor)[AUX]Intelligent window scrollbars provide quick access to various tasks</x>Intelligent scrollbars

(next section) include markers to define the zone being

displayed relative to the entire file, to locate the cursor's
position relative to the code being viewed, to locate text found
via the find/replace text box, to display where errors are
located after doing a build, and to reveal the changes that have
been undertaken on the file since last loading the file into the
workspace. These scrollbars may be displayed in tab mode or
map mode.

2.
Intelligent window scrollbars
<x>AtmelStudio (editor)[AUX]Intelligent AtmelStudio provides
window scrollbars provide quick access to various tasks</x>

windows with vertical and horizontal scrollbars to navigate vertically

or horizontally inside a window's source code. These contain a
grayed-out area which indicates the position and size of the view
window relative to the window's content. This is standard practice
with just about every text editor - nothing new here; drag the slider to
shift the viewing area. Note that AtmelStudio uses the vertical bar to
provide the programmer with additional info, notably:

A horizontal dark blue line marks the cursor's position relative

to the source code. This means that the programmer can
determine where the cursor is relative to what is being looked
 at. He/She can slide the gray viewing area slider up or down to
the blue line to bring the viewing area to the cursor's position
or right click on the cursor to move the viewing area directly to
the cursor's position.
After undertaking a text search via the find text box (ctrl-F),
orange patches in the scrollbar indicate where text has been
found. The programmer can then go directly to found text by
clicking on one of the orange patches. These patches will also
appear in other files for as long as the find text box remains
open.
Yellow patches will appear in the scrollbar as the programmer
makes changes to files. Upon doing a save, these patches turn
green. Upon closing AtmelStudio or closing the file and
reopening it, they disappear.
Small red patches appear in the scrollbar. These indicate
spelling mistakes in text strings and comments.
After doing a build which contains errors, green patches will
appear in the scrollbar. They reveal the locations of errors.

The vertical scrollbar may be displayed in one of two modes: Tab

mode and Map mode. By default, AtmelStudio opens a window with
Tab mode scrollbars. They may be changed to Map mode to display
miniature source code. They also display a text window containing
source code as the cursor is moved up or down the scrollbar. This
feature enables viewing code without shifting the window's viewing
area or the cursor's position. Shifting between tab mode and map
mode may be undertaken by right clicking inside the scrollbar. It
opens a text box with menu choices. Choose the bottom item
Scrollbar options; you will find choices to configure the horizontal
and vertical scrollbars.

3.
Quick access to functions
Some files may contain many items
<x>AtmelStudio (editor)[AUX]Multiple-choice quick access to functions</x>
(global variables, functions, classes, macros...). If you know which
file the function is implemented in, select the file's tab, then go to the
upper left corner of the file's window. There you will find a scroll
down multiple-choice text box which contains all the items declared
or implemented in the file. Select the one you want to work on. No
need to do a find to go to a variable or function.

4.
Go to implementation
When working on a function it is often useful to look at what the
function does, how it gets to the desired result. You may want to look
at its parameter list and maybe its code. Of course, it is always
possible to do a search via the function's name, however, there is a
much quicker solution. Click on the function you are viewing. A
window pops up which reads Go to implementation. It contains
function declarations and implementations. Select the one you want.
AtmelStudio will open the corresponding file and position the cursor
on the declaration or implementation you selected. This Go to
implementation facility enables you to quickly verify parameters and
what the function does. To go back to what you were working on use
AtmelStudio' code navigation feature by clicking on the go back to
button enough times to reach your desired location. This navigation
feature is described below in Code navigation (page 61).

5.
Inline function related tasks
AtmelStudio provides two quick access
<x>AtmelStudio (editor)[AUX]Right-click on function for quick access</x>

popups to handle diverse tasks otherwise accessible via top toolbar

icons and menu items. These multiple-choice text boxes provide
shortcuts to tasks. I refer to these as the function related quick
access method and as the right-click anywhere quick access
method.
The function related quick access method is based on clicking a
small down arrow beneath a function call. It opens a multiple-choice
text box which contains access to several tasks:

<x>AtmelStudio (editor)[AUX]Create declaration in corresponding .h file</x> Create declaration - Having

implemented a function in a .cpp file, you need to create the
corresponding declaration in the .h file. This can be done
automatically by way of the create declaration option.
Rename (refactor) changes the name of a variable, macro,
class, function, etc. in the entire project.
<x>AtmelStudio (editor)[AUX]Change signature changes the name and the type of a declaration and refactors</x>Change signature

changes the name and the type of a declaration and refactors.

<x>AtmelStudio (editor)[AUX]Documentation above a function call</x>Document methods brings up

documentation of the form /*...*/ when hovering above function

call.
<x>AtmelStudio (editor)[AUX]Find reference: all uses of variable/function</x>Find reference lists all uses of a

variable or function. Clicking on one of the listed uses brings

the cursor to the corresponding location. I use this feature
constantly.

The right-click anywhere quick access method is triggered by right

clicking on a called function. It brings up the quick access multiple-
choice box, a shortcut to other tasks, namely:

Go to
<x>AtmelStudio (editor)[AUX]Clicking on function opens a goto multiple-choice window for declarations</x>

implementation opens a multiple-choice window to quickly

access a function's declarations or implementations.
<x>AtmelStudio (editor)[AUX]Refactoring - rename items throughout app</x>Refactoring - This is a quick

access to renaming items. It saves going up to the toolbar to

invoke it.
<x>AtmelStudio (editor)[AUX]Insert snippet such as if, while, for, etc.</x>Insert snippet enables inserting

skeleton code for if, while, for, etc. Here is the switch snippet:
 switch (key) {
case value:
/* Your code here */
break;
default:
/* Your code here */
break;
}
Surround with - Select a set of
<x>AtmelStudio (editor)[AUX]Surround selected code with a for, if, etc.</x>

contiguous lines and choose a control statement or other to

surround them with. Here is the for loop:
for (i = 0; i < length; i++)
{
lines of code you have selected
}

Not forgetting the closing brace and placing it in the correct location
are two benefits derived from using this code shortcut.

Outlining means
<x>AtmelStudio (editor)[AUX]Code collapsing in AtmelStudio is outlining; indenting is formatting</x>

collapse in AtmelStudio parlance. It grants access to the

various collapse options.
<x>AtmelStudio (hardware-based debugging)[AUX]Hardware-based debugging with ATmega328P-Xmini successful, with Atmel ICE not

Debugging commands - Access debugging commands

so</x>

directly when doing hardware-based debugging - stop,

resume, and others.

6.
Color coding
Color coding is a means to
<x>AtmelStudio (editor)[AUX]Syntax checking - color goes black when there is a programming error</x>

differentiate program types (variables, macros, text, etc.). This can

prove useful to detect unsuspected massive commented out text via
/*...*/ pairs. The Arduino IDE does do color coding, but AtmelStudio
carries color coding a giant step further. It uses it as an aid in syntax
checking. If something is wrong, the culprit's color turns black. Here
is a non-exhaustive list of color-coding features:
 Macros are purple -
<x>AtmelStudio (editor)[AUX]Editor displays colors: functions red, variables blue, comments pale green, etc.</x>

If a specific macro is commented out, the color changes to

black to signal that it is undefined. This can be extremely
useful when looking at #ifdefs.
Function declaration errors are black - You have just created a
function declaration with parameters. In the process of doing
so, you have forgotten to define a parameter's type. The
function will be black with a red wiggly underline under the bad
parameter.
Functions are red - If a function is undefined, its color changes
to black and is underlined by a wiggly red line to indicate
where the problem lies (undeclared, bad parameter, parens
problem).
Variables are blue - If a variable is undefined, the color
changes to black.
Bad parens turn functions black - If there are unbalanced
parens in a function's declaration, the function's name turns
black, underlined by a wiggly line.
Comments /*...*/ are pale green - An unbalanced closing
comment turns black */.
Strings are red - Omitting a start or end double quote changes
the string's color to black.

The eye gets used to these color conventions. A wrong color alerts
you. You know that something is wrong. Just look for anything
colored black. You have just saved yourself time; no need for you to
wait until the compiler tells you that something is amiss.

7.
Inline syntax checking
In-line syntax checking is one of AtmelStudio's powerful productivity
features. While you are writing code, if AtmelStudio encounters a
syntax error, it will underline it with a wiggly red line. For example,
imagine that you have just typed a function declaration and forgot
the ending semicolon, this is what you will get:
void MyFunction()
~~~~~~~~~~
the function's name is black, not blue, and is underlined by a wiggly
red line. Add the semicolon, the variable's color turns blue, and the
red wiggly underline disappears.
void MyFunction();
Or, if you forget a parameter's type in a declaration:
void MyFunction( _param);
~~~~~~
you get the same results except that the wiggly line is shorter and
under the parameter, not under the function's name. Add the
parameter's type:
void MyFunction( char* _param);
and things revert to normal.
An undefined variable or function is another example. It changes its
color to black and gets underlined with a wiggly line. Furthermore,
out of scope variables are detected. So, if you see wiggly lines
anywhere, look carefully at the code, there is an error somewhere.
These small errors, semicolon forgotten, one too many parens,
missing or extraneous quote, etc. are caught early as you type. They
are a significant time saver.

8.
Find/Replace
<x>AtmelStudio (find/replace)[AUX]Supports whole word, regular expressions, past searches</x><x>Regex (general)[AUX]Many tools contain a regex engine - AtmelStudio and

The Arduino IDE's find/replace works well but

PlatformIO find/replace, Perl, Awk, Word (wildcards)</x>

is somewhat limited although version 2's find/replace does support

regular expressions. AtmelStudio's is more powerful. It can handle
whole word, past searches, regular expressions, and where to do the
search.
But there is more to AtmelStudio's find function. As soon as you
have typed the text you wish to find, two mechanisms are activated.

<x>AtmelStudio (editor)[AUX]Intelligent window scrollbars provide quick access to various tasks</x><x>AtmelStudio (find/replace)[AUX]Little orange patches in
 You can immediately, just by glancing at the
vertical scrollbars show find location</x>

window's content, see where your text has been found

because the found text's background color has changed to
dark gray, or scroll up and down to get to found text.
The file may be quite large. Most of the text being searched for
may be out of sight. The vertical scrollbar in the window's right
edge displays little orange patches indicating the location of
found text. You can click on these little patches to go directly to
text found. These little patches will also appear in other files
which contain the text assuming you did a project wide search.
You can also navigate from found text to found text forward
and backward. See Intelligent window scrollbars (page 52).

The colored found text with an orange background reverts to normal

when the find box gets closed.

9.
Name completion
AtmelStudio anticipates what you will
<x>AtmelStudio (editor)[AUX]Name completion contextual popup opens</x>

be doing. When typing anything, it will open a contextual popup

suggesting name completion. By name completion, understand that
it means displaying suggestions as you begin typing a name (list of
variables, functions, macros). Most IDEs do this but there is a lot
more to name completion. Read on.

Variable name completion - The popup will list the available

classes, enums, macros and variables (global and local) after
you have typed the first letter. For example, suppose your
application contains, amongst others, two variables whose
name begins with temp such as:
float tempFloat{_floatValue};
uint32_t tempUint32{0};

When you type t, the popup will contain time, true, other stuff and your
two temp variables. After you have typed te the popup will contain
tempFloat and tempUint32 and other variables which start with te. Scroll
down and select the one you want.

Function name completion - When adding a function call in

your code, you will usually not remember the details of
required parameters. You can do a copy/paste from the header
file but that means go to the header file, find the function
declaration, select/copy, come back and paste. Too many
steps. You could, instead, start typing the function then select
the function using the name completion feature. This will insert
the function into your code with open/close parens. The cursor
will be positioned inside the parens pair along with a popup of
functions parameters. Finish coding by filling in the required
parameters.
<x>AtmelStudio (editor)[AUX]Name completion contextual popup opens</x>enum list access - Choosing an

enum in a list with AtmelStudio is idiot proof. Suppose you

defined the following enum list:
enum class EMyEnum: uint8_t { first, second... };

If, somewhere in your code, you want to assign an enum value to a

variable such as EMyEnum var =, as soon as you type the = sign, a
popup opens with the enum choices first, second, etc. - just select the
one you want. There is no need to chase for valid enum spellings in
the .h files.
Here is a variant on the above. A function passes the enum as a
variable and tests the variable's value, such as:
void MyFunction (... EMyEnum _eChore, ...) {
...
if ( _eChore == EMyEnum::second ) { // test the enum's value
... do something
} // if
...
} // MyFunction
As you are creating MyFunction, at some point you will be typing the if
statement:
if ( _eChore ==
and here is the neat part. As soon as you finished typing ==, a
 popup opens with all the EChore enums enabling you to select the
appropriate one. No need to go to the header file where EChore has
been defined.

Macro name completion - I use macros extensively, mainly for

print-based debugging. With AtmelStudio, as soon as you start
typing the beginning of a macro, a small popup window opens
with the macros which match whatever you have typed. What
is more, as you scroll down the list of macros, you can see the
code for each macro, thereby knowing right away what it does
and what parameters the macro takes, if any.

10.
Code collapsing and indentation
Both the Arduino IDE and
<x>AtmelStudio (editor)[AUX]Collapses classes, functions, enums, ifs, whiles, etc. and #ifdef...#endif pairs</x>

AtmelStudio provide a code collapsing facility. When a file gets large,

1000+ lines of code, collapsing code is a great help to visualize the
file's structure and overall content.
<x>#ifdef...#endif[AUX]Collapsing supported by AtmelStudio</x>The Arduino IDE uses brace pairs { ... } to

collapse and indent code. It thereby enables code to be collapsed at

any level, as long as there is a pair of curly braces. It could be an if's
content, a while's content, a case if you enclosed its content in curly
braces. It also collapses content enclosed in /*...*/ comments. It relies
on little + and - square boxes in the left margin to collapse selectively.
One of the Arduino IDE's big failings (1.8.19 & V2) is that it does not
have a global collapse. This is unfortunate since a global picture of a
file's content requires that all functions be collapsed. As a sideline,
please note that I use the Arduino IDE to indent and/or collapse
JavaScript and Perl code. The Arduino IDE does not look at syntax
when collapsing; it merely uses curly braces to indent and collapse
code.
<x>AtmelStudio (editor)[AUX]Code collapsing in AtmelStudio is outlining; indenting is formatting</x>AtmelStudio does not rely on

curly braces to collapse code. It collapses structures, classes, functions,

enums, /*...*/ comment blocks, and #ifdef ... #endif pairs. It can
collapse/uncollapse these one at a time via little + and - square
boxes in the left margin, or the entire file's content via the Outline all
option. Collapsing everything gives you an immediate view of a file's
content. This is particularly useful for .cpp files. AtmelStudio calls it
Outlining. To collapse all, select Toggle All Outlining; to un-collapse
all, select Stop Outlining.
AtmelStudio supports the following code collapsing capabilities:

#ifdef
<x>#ifdef...#endif[AUX]Collapsing supported by AtmelStudio</x> - It enables collapsing #ifdef...#endif
pairs. The legacy Arduino IDE 1.8.19 does not do this. The
new Arduino IDE 2.1.1 does collapse #ifdef...#endif pairs however
there is a catch. If the #ifdef is inside a function, it will not
collapse the function. If the function has no #ifdef...#endif pair, the
Arduino IDE V 2.1.1 will collapse the function.
<x>/*...*/[AUX]C-style comments collapse supported by AtmelStudio and by the Arduino IDE</x>/*...*/ - Commented out

blocks of code via C-style comment pairs may be collapsed at

will. The Arduino IDE ١.٨.١٩ does this, not the new V٢ version.
Structures, classes, functions, enums, unions may be
individually collapsed. Both IDEs do this.
<x>AtmelStudio (editor)[AUX]Supports one-click all code collapse</x>One-click all code collapse -

Collapse all code at the highest level via one click to get a
bird's eye view of your file's content and organization; then,
uncollapse selectively. (toggle All Outlining).
Keyboard shortcuts - AtmelStudio provides collapsing via
keyboard shortcuts (CTRL+m twice).
Collapsed code visualization tool - The Arduino IDE,
AtmelStudio, and Visual Studio/Visual Micro provide collapsed
code visualization content without having to expand/contract
the collapsed code.
Copy/paste code - If there is a need to copy or move the
collapsed code somewhere else, you can copy/paste the
collapsed code. It is not necessary to un-collapse. When
copying/pasting a block of code, it gets un-collapsed when
pasted. The other items remain collapsed. Both the Arduino
IDE and AtmelStudio do this.
The Arduino IDE and AtmelStudio differ so much from one another
that it is sometimes useful to close one IDE and open the other to
visualize code. I systematically develop with AtmelStudio but
sometimes, when doing major reorganizing, the Arduino IDE's code
indenting and collapsing proves to be more flexible.
<x>AtmelStudio (editor)[AUX]Unnamed code blocks improve readability, decrease RAM</x>The Arduino IDE's code
collapsing, based on using open/close curly brace pairs { ... }, opens
interesting possibilities. The programmer may place braces to isolate
code: collapse individual cases in switches and collapse sections of
code, the result being controlled visualization of source code. This
has the added benefit of limiting the scope of variables declared
inside these additional curly brace pairs, hence early release of
memory used by these as the execution leaves the curly braced
group. I refer to code inside these extra curly braces as unnamed
code blocks.
<x>AtmelStudio (editor)[AUX]Establish Arduino IDE code collapsing #ifdef interoperability by adding curly braces</x>Unfortunately, you

cannot use these extra brace pairs to collapse with AtmelStudio.

Fortunately, AtmelStudio does recognize #ifdef...#endif pairs. It is
possible to achieve code collapsing interoperability by combining
Arduino IDE brace pairs {..} with AtmelStudio #ifdef...#endif pairs.
Imagine a switch statement in which there are many cases, each with
considerable code. It makes for pages of listing. You can selectively
collapse individual cases with either IDE by using combined
braces/macro pairs as follows:
#define ISOLATE_CODE // NEVER COMMENT THIS OUT
...
switch( val ) {
case 10:
#ifdef ISOLATE_CODE // #ifdef...#endif pairs for AtmelStudio collapsing
{ // brace pair to isolate code for Arduino collapsing
...
break;
} // brace pair to isolate code for Arduino collapsing
#endif // ISOLATE_CODE #ifdef...#endif pairs for AtmelStudio collapsing
case 11:
#ifdef ISOLATE_CODE // #ifdef...#endif pairs for AtmelStudio collapsing
{ // brace pair to isolate code for Arduino collapsing
...
Enclosing each case's code inside #ifdef...#endif/{ ... } combinations such
as shown above may prove useful to declutter the screen while
working on a particular case, whether it be with the Arduino IDE or with
AtmelStudio. An alternative to long cases would rely on functions to
encapsulate individual case's code.
In a nutshell, when working with the Arduino IDE, collapsing is based
on curly brace pairs. With AtmelStudio, collapsing is based on
definitions (functions, classes, enums...) and #ifdef...#endif pairs.

11.
Collapse /*...*/ group
AtmelStudio collapses a group of lines commented out by the C-style
/*...*/ style comment pairs. Legacy Arduino 1.8.19 also does this, not
version 2.

12.
Group comment/uncomment
It is easy to comment a
<x>AtmelStudio (editor)[AUX]Keyboard shortcut to quickly comment/uncomment highlighted sections</x>

group of lines: select a group of lines and comment or uncomment

them as a group all in one click. AtmelStudio has both buttons and
keyboard shortcuts to do this. The Arduino IDE does this via menu
choices. It does have a keyboard shortcut (ctrl-/) but it works on US
keyboards only. The new Arduino V2 version's comment/uncomment
keyboard shortcut does work on French keyboards. Another way of
doing this in AtmelStudio is to select a group of lines and type /, it will
comment/uncomment the group, much faster than going for the
comment or uncomment buttons.
The C++ // comment is good for commenting an entire line or its tail.
But it is sometimes necessary to comment out a couple of words
inside a line via the /*...*/ pair. For example, suppose you want to
comment out param2, from the following function definition:
void MyFunction(int param1, int param2, int param3);
With AtmelStudio, highlight int param2, and type the forward slash /. The
result will be:
void MyFunction(int param1, /*int param2,*/ int param3);
This is much faster than typing laboriously the /*...*/ pair.

13.
Refactoring
<x>AtmelStudio Refactoring means rename all
(editor)[AUX]Refactoring - rename items throughout app</x>

instances of a variable. Suppose that you originally defined the

following enum list
enum EAllocMode { eCreate, eRemove };
which is quite primitive given that you were less C++ savvy and had
not quite learned that there was more to enums than the above. You
named the enums eCreate and eRemove to render the names unique. You
have indeed accomplished uniqueness, but at the expense of
readability. There must be a better way.
Some time later, your application is getting to be sizable, and you
have improved your C++ know-how. You have discovered the enum
class keyword and size qualifier which may be used in enum definitions.
You have rewritten the enum definition as follows:
enum class EAllocMode: uint_8t {
start = 0,
create = 0, remove,
end
}; // enum class EAllocMode
Notice how the enums are more readable; eCreate has been replaced
with create. When using the enum, you now need to qualify it with the
scope operator::. Instead of eCreate, write EAllocMode::create.
You removed the individual items' enum prefix to improve readability.
You need to replace all eCreate by EAllocMode::create. Of course, you can
do this via the IDE's find/replace all or via a one by one find/replace
to avoid unwanted changes. There is a better way. You can modify
the individual enums by refactoring. This enables you to change the
name of all cases of a given instance, whatever it might be: enum,
function, variable, class, or other. It is an intelligent text find/replace.
You can thus refactor eCreate to EAllocMode::create.
Here is an additional refactoring example. How often does a
programmer decide to rename a variable or function to render the
code more readable or less prone to confusion? The programmer
goes to the declaration (.h file) and modifies it, the editor then
proposes to change all references to the modified variable/function.
Upon accepting to do this, all references are changed. In effect the
programmer has done a refactoring without explicitly asking for it.
When you want to change a variable's name, there is another good
reason for using refactoring as opposed to doing a global
find/replace; refactoring is syntax based. This means that if you want
to change the name of a variable, it will do so within its scope.
Imagine the following example. It has two variables named buffer, a
global one and a local one. Granted, it is poor programming practice,
but it was late, you were not careful, so here is what you did.
char* buffer; // global buffer
void MyFunction() {
char * buffer; // local buffer
...
} // MyFunction
Having noticed your mistake, you want to fix this source of
confusion. If you refactor the global variable buffer to globalBuffer, only
the global variable will be renamed, not the local one. After
refactoring, you will get
char* globalBuffer; // global buffer
void MyFunction() {
char * buffer; // local buffer
}
Had you used a global find/replace, you would have changed all buffer
instances to globalBuffer - both the global and the local variable -
gotcha!
As you program, you create variables, classes, functions, enums, and
other entities which you name on the fly. You will often want to
rename some to make the names more meaningful and less prone
to misinterpretation. Refactoring enables you to do this risk free. If
the program had compiled correctly before refactoring, it compiles
correctly after refactoring. Extremely useful! I use it frequently.

14.
Compiler error reporting
<x>AtmelStudio (editor)[AUX]Compiler one-click access to faulty line</x><x>Arduino (caveats)[AUX]Compiler error reporting - must manually go to file/line # to access

AtmelStudio supports a direct clickable link between the error

code</x>

message and the corresponding file/line number location. Navigating

to an error's location could not be simpler. Click on a specific error
message. The cursor will go directly to the error's location. Fix it -
then click on the next error message and continue.
The Arduino IDE also does a good job reporting compilation errors.
The problem is that the programmer, having decided upon the error
to work on, must then search for the file in the tabs list, select it, and
go to the offending line, a time-consuming way of doing things.

15.
Code navigation
As you continue working on your code,
<x>AtmelStudio (editor)[AUX]Navigate to places recently worked on</x>

AtmelStudio keeps track of where you have been. A couple of

buttons enable you to navigate backward and forward along your
path - these are similar to Bookmarks (page 61). This feature can
prove useful when you ask yourself: Where is it that I just modified
such and such a parameter? It saves having to do finds to locate a
particular spot you recently worked in. Here is an example: You
undertake a one by one find/replace. The last replace takes you far
from your original location. Having finished doing find/replace, you
ask yourself: Where was I when I started doing this find/replace?
With AtmelStudio, just click on navigate back repeatedly, it will
ultimately take you to the location you started the find/replace search
from. Note that this is not ctrl-Z (undo); you can navigate backward
and forward to places you recently worked in.

16.
Bookmarks
Bookmarks can be useful. To bookmark
<x>AtmelStudio (editor)[AUX]Bookmark to navigate back and forth</x>

means to tag a line of code. Tagging enables you to go from one

tagged line to another tagged line by clicking on next bookmark or on
previous bookmark. This feature proves useful when doing
experimental development. You may want to tag a line of code so
that you could come back to it later on.

17.
Spell-checker
When writing text inside a string or
<x>AtmelStudio (editor)[AUX]Spell checks strings and comments</x>

comment, AtmelStudio considers that it is prose, not code - it will

verify spelling for standard English. It signals a spelling problem by
underlying the offender with little red wiggles. It is not as good as a
full-fledged text editing tool such as Word, yet it remains useful. So
far, I have only used it with English. It might support other
languages - this may depend on the environment AtmelStudio is
being used in. Spell-checking is not a fundamental programmer
assistance feature; however, it is convenient. It contributes to
productivity. Comments and text become cleaner, practically no
spelling mistakes.
Furthermore, AtmelStudio's spell-checker seems to be blessed with
intelligence. If a text string contains code, such as radius =, it will not
signal it as a spelling mistake. There are however borderline
situations such as malloc returns. It will signal malloc as being a spelling
mistake.

18.
Go to line number
If you want to go
<x>AtmelStudio (editor)[AUX]Ctrl-G takes you to line # in file</x><x>Arduino (editor)[AUX]Ctrl-L takes you to line # in file</x>

to a specific line number, type ctrl-G. A popup box opens. It will

enable you to go to a specific line in the current file. The window will
display the current line number and the total number of lines in the
file.
 8.
AtmelStudio hardware-based
debugging
There are three basic ways to hardware-based debug with Atmel
studio.

Use a hardware-based debugger board such as Atmel ICE

(page 122). The debugger board interfaces between
AtmelStudio and the target board (the board your application
will run in). It receives stop/resume commands from the
computer and collects information from the target board so that
the programmer could visualize variables and memory. The
problem with Atmel ICE is that despite hours spent trying to
debug with it, all I succeeded in doing was ruin boards and
microcontrollers and experience lots of frustration. I was
comforted that it was not my incompetence since I discovered
on the Internet that there are plenty of others who have had
the same experience.
Use integrated debugger boards which combine the hardware-
based debugger and the target microcontroller such as the
ATmega328P-Xmini board. Fortunately, I succeeded with the
ATmega328P-Xmini board - do yourself a favor, get one of
these - it will open new horizons. What's more, it is pure plug &
play. You will find more details in Hardware-based debugging
(page ١١٩).
Serial-based debugging lies somewhere in between hardware-
based debugging and print-based debugging. It does not
require an external board; debugging is undertaken directly on
the target microcontroller. It enables the programmer to define
breakpoints to stop the application, inspect variables, assign
new values to variables, and resume. It is however limited by
constraints. See Serial debugging (page 125).
 9.
AtmelStudio caveats
Nobody is perfect, not even AtmelStudio. It has its share of bugs,
inadequacies, and hard to find reasons why things fail. The following
caveats summarize problems I encountered. Their workarounds are
described further on:

Sketch File - There

<x>AtmelStudio (caveats)[AUX]Importing Arduino project - directory navigation bug on searching .ino file</x>

is a bug in AtmelStudio in the browse for the .ino file when

creating an Arduino project; do not use it. Instead, copy/paste
the full path name of your sketch file from Windows Explorer.
You will get the path only:
D:\ArduinoDev\Frameworks\MemMgt. You will need to
manually add the sketch's name: \[Link].
<x>AtmelStudio (caveats)[AUX]Workspace font size not documented; - '>' and '<' become ',' and ';' on French keyboard - tied to physical key, not to key mapping</x>

<x>AtmelStudio (keyboard shortcuts)[AUX]Workspace font size not documented - '>' and '<' become ',' and ';' on French keyboard - tied to physical key, not to key

Workspace font size (page 65) - One day, as I

mapping</x>

accidentally did a ctrl-alt or ctrl-shift something or other, not

remembering the particular combination, the font size in my
workspace got smaller. Since the Arduino IDE has a menu
item to control workspace font size, I naturally looked for the
same in the AtmelStudio's icons and menus, but to no avail. I
should mention that the AtmelStudio V7 User Guide (see
Bibliography) provides a list of keyboard shortcuts, but the list
extends over more than 14 pages. I did search the list; I could
not find the increase/decrease workspace font size in it (I shall
get into Keyboard Shortcuts later).
<x>-flto flag[AUX]Code size depends on -flto flag; reduce by 40%</x><x>AtmelStudio (caveats)[AUX]Code size by default 40% too large, needs -flto

AtmelStudio generates much larger code size than the

flag</x>

Arduino IDE (Code size (build options) page 233) - The

inexistent -flto compiler option leads to builds much larger than
had the flag been applied (40% greater on a medium sized
application). Unfortunately, this option is not proposed in the
AtmelStudio Project Properties dialog box, and by default, not
implemented. One would think that AtmelStudio's Project
Properties dialog box would cover all important compiler/linker
flags. Unfortunately, the important -flto one, whose absence is
responsible for 40% larger build sizes, is missing. I consider
this missing compiler option to be a flaw. It leaves the
programmer with no clue as to why his/her application
consumes more RAM than it should. Had I not compared
codes sizes from AtmelStudio, from the Arduino IDE, and from
PlatformIO, and examined individual compiler/linker options, I
would not have realized that I could have saved considerable
RAM space. Furthermore, a newbie trying out AtmelStudio
might decide to systematically edit code with it and compile
upload with the Arduino IDE because AtmelStudio does not
provide a clue as to why its code is 40% larger.
<x>AtmelStudio (caveats)[AUX]avrdude fails to upload code into chip</x> Failure to upload code (page 66)

into the microcontroller - At times, avrdude fails to upload the

.hex file into the microcontroller for no apparent reason. This is
not really a problem since trying again generally succeeds,
although on one occasion it has taken doing. The problem is
that the message in the output window looks as though the
upload succeeded, misleading the programmer into thinking all
is well. He/She got used to just glancing at the output window
not really examining the messages. This led to wasted time
trying to understand why the program's behavior remained the
same despite code changes.
<x>AtmelStudio (caveats)[AUX]Failure to reckon with Debug vs. Release output causes confusion</x> Code changes (page

67) do not affect the application's runtime behavior - I made

configuration changes and continued development not
remembering details on what changes I had made. At some
point, I did a build and uploaded the executable into the
microcontroller. I was disconcerted since the changes in my
program did not modify my application's behavior yet a .hex file
did get uploaded into the microcontroller. It was not the failure
to upload code into the microcontroller described above.
Something escaped me! But what?
<x>AtmelStudio (caveats)[AUX]Mysteriously imports second set of header files into ArduinoCore directory</x> Second set of
header files import (page 68) - When you import an Arduino
IDE project to create an AtmelStudio project, the project's
header files (.h) get copied into an inconvenient directory,
ArduinoCore/include, for no apparent reason. It is necessary to
remove them since the project's .h files should be in the same
directory as the project's .cpp files. Having two sets of files in
the same workspace is a recipe for disaster.
<x>AtmelStudio (caveats)[AUX]File has 20+ functions, only two showed up in the quick access selection box</x>When working on a

file, the little Quick access to functions pull-down multiple-

choice box in the top left-hand corner of a file's window is
handy. It enables you to quickly access a function deep down
inside the file without having to do a find. One day, I opened
the little box and found that only two functions were listed. It
was missing the 20+ other functions defined in the file - yet, the
build was successful. What is going on??? See Quick access
to functions pulldown selection box failure (page 68).
<x>AtmelStudio (caveats)[AUX]Indenting of if...else statement depends on where opening curly brace is placed</x><x>AtmelStudio (editor)[AUX]Code collapsing

Improper indenting (page 69) - The

in AtmelStudio is outlining; indenting is formatting</x>

Arduino IDE calls it indenting; AtmelStudio calls it formatting,

no matter. Correct indenting is a valued help when checking
that braces and parentheses are correctly balanced and
positioned. There are times when AtmelStudio fails to indent
properly.
<x>AtmelStudio (caveats)[AUX]Extra */ causes indentation failure</x> Failed code formatting (page 69) -

AtmelStudio, like the Arduino IDE, allows an entire file to be

formatted (indented) via a single click: Edit --> Advanced -->
Format document. On one occasion, after doing a Format
document, I found that all the code located after a large block
of C-style /*...*/ commented code was stuck against the left
margin, not a single indented line. The code before the
commented block was formatted correctly. Why? In spite of
this, the application compiled and ran correctly.
<x>AtmelStudio (caveats)[AUX]Occasional segmentation faults are benign; no clues on causes; do rebuild to fix</x> Segmentation
faults (page 70) when using the Arduino IDE became more
than a nuisance; this problem prompted me to switch to
AtmelStudio. Having made the switch, I found that AtmelStudio
did manifest occasional segmentation faults generally without
error messages; but, on one occasion, it did provide an error
message. The fix was quite unexpected. AtmelStudio
segmentation faults have never been a problem, they would go
away after a rebuild or two.
<x>AtmelStudio (caveats)[AUX]Build fail due to inexistant file in project not in error messages</x> Errors not listed in the

error list (page 71) and output windows - Having tucked away
the error message window down to get more space to work in,
I found that it contained no errors after a build failure. It is as
though the error list window had been rendered inactive.
<x>AtmelStudio (caveats)[AUX]Make error from missing .cpp file drowned in reams of messages</x> Make utility errors not

listed (page ٧٢) - Compiler and linker errors are displayed in

the error list window however, if you get an undefined
reference because you simply forgot to include a file in your
project, you will get a build failure (missing file) drowned in
reams of build output, but no error message in the error
message list. This makes it difficult to find the cause of the
build failure.
<x>AtmelStudio (caveats)[AUX]Build fail due to inexistant file in project not in error messages</x>Build fails - NO errors on

compile output - When physically removing files from an

AtmelStudio project directory because you no longer need
them, forgetting to remove them from the project's
configuration, via the Solution Explorer, will cause a build
failure. This can be disconcerting because the error message
which indicates that a file is missing is hidden in reams of
output. See Inexistant file in project not listed in error
messages window (page 73).
<x>AtmelStudio (caveats)[AUX]Careful with find/replace Opened Documents</x> Search and replace (page 73)

- Beware of the search/replace Opened Documents option.

You may fail to make changes in some files.
<x>AtmelStudio (caveats)[AUX]Code remains obstinately collapsed</x> Refusal to uncollapse a section of

code (page 75) - The Format Document tool can sometimes

 prove to be obstinate when you ask it to uncollapse a section
of code. You click in the little square icon to uncollapse code,
nothing happens.
Auto format if...else problem (page 74) - When doing an if...else,
indentation depends upon where you place the else opening
curly brace, in the same line as the else, or in the next line. The
compiler does not care but the code formatter does
differentiate the two.
Unable to generate code for new microcontroller (page 75) -
When doing an Arduino IDE sketch import into AtmelStudio, it
seems that AtmelStudio hard codes the device into the
ArduinoCore build (ATmega328P or ATmega2560). This
means that if you develop on an ATmega2560 and later want
to rebuild for the ATmega328P, you must create a new
ATmega328P specific project; not good.

You will find details on these caveats, with the corresponding fixes,
in the following pages.

1.
Workspace font size
<x>Arduino (editor)[AUX]Edit menu item changes workspace font size</x><x>AtmelStudio (caveats)[AUX]Workspace font size not documented; - '>' and '<' become ',' and ';' on

French keyboard - tied to physical key, not to key mapping</x><x>AtmelStudio (keyboard shortcuts)[AUX]Workspace font size not documented - '>' and '<' become ',' and ';' on

The Edit menu of the Arduino IDE has a

French keyboard - tied to physical key, not to key mapping</x>

couple of options: increase font size and decrease font size, easy to
access and useful when the font size accidentally changes. This can
happen when you mistype keyboard shortcuts. AtmelStudio does
provide font type, size, and color for all sorts of items, but there does
not seem to be a provision for the workspace as a whole. Here is
what happened to me.
<x>AtmelStudio (editor)[AUX]Keyboard shortcuts tied to physical keys or to symbol mapping</x>I was trying out keyboard

shortcuts (navigation forward and backward amongst others) which

did not seem to work. I have a French keyboard (AZERTY).
AtmelStudio is installed with English as the language. While trying
out the keyboard, the font size of my .h/.cpp files suddenly
increased. I persisted, it kept increasing until the letters were much
too big. I had found the shortcut to increase the font size, but not the
one to decrease it. Question: How does one get the font size back
down to a reasonable size? AtmelStudio's help was of no use, none
of the choices in the Tools-->Options-->Fonts and Colors seemed to
work.
I looked at the keyboard shortcuts int the AtmelStudio 7 User Guide
to no avail (see Bibliography page 332). It became a trial-and-error
process. If a keyboard shortcut increases the font size, another
shortcut should decrease it. I finally found it. Sure enough, on a
French keyboard ctrl-shift comma increases the font size, ctrl-shift
semicolon decreases it. On a US keyboard (QWERTY) these keys
map to ctrl-shift > and ctrl-shift < respectively, which make sense.
Finally, the key to understanding what caused the problem lies in
understanding that AtmelStudio is an American designed program.
The developers chose symbols that related to their use. The problem
is that once the key combination has been defined, it is the key itself
which is the shortcut, not the character the key maps to when
representing the shortcut. This is different from Window's ctrl-A,
which works on the letter A whether you are using a QWERTY or an
AZERTY keyboard. It seems that some keyboard shortcuts are hard
coded onto the physical key, no matter what letter it maps to, while
other keyboard shortcuts are soft coded to the letter, not to the
physical key. With AtmelStudio, when moving to a different keyboard
layout, the shortcut's ID becomes whatever letter happens to be on
the physical key. On a French keyboard, '<' and '>' become ',' and ';'
respectively.

2.
Application code size (AtmelStudio)
<x>AtmelStudio (caveats)[AUX]Code size by default 40% too large, needs -flto flag</x><x>-flto flag[AUX]Code size depends on -flto flag; reduce by 40%

Interoperability having been achieved amongst the three IDEs

</x>

(Arduino IDE, AtmelStudio and PlatformIO), I compared build code

size (RAM and flash memory) to determine whether one of the
environments was clearly better than the others. Upon doing a code
size comparison on one of my programs (MemMgt, default options,
Release build) I found that AtmelStudio generated code 40% larger
than the other two IDEs. This did not make sense since they use the
same compiler (GNU C++) hence should produce essentially the
same output. Furthermore, the Arduino IDE and PlatformIO
generated nearly identical code sizes.
Faced with this dilemma, my next step was to test AtmelStudio's
numerous compiler and linker options as provided via the Project
Properties dialog box. I tested them all, changed an option and
recompiled. None of the AtmelStudio supplied options had a
significant impact on code size. The mystery got to be more obscure
since I incorrectly assumed that all significant compiler/linker options
would be accessible via the Project Properties dialog box. The
permissive option (which is important to use otherwise your code will
not compile) is not present but that is not a problem since there is a
message in the compiler's output suggesting that you use it to
transform an error into a warning.
Luckily for me, I had accomplished interoperability amongst the three
environments. I copy/pasted the entire build command of each
environment to examine the differences. To make a long story short,
the -flto (link time optimization) was absent from the AtmelStudio list
of compiler options. I added -flto to the build options, recompiled the
program, and all fell into place. AtmelStudio produced code about
the same size as the other two compilers. You will find full details in
Code size (build options) (page 233).
In my opinion, the absence of the -flto option as one of the Project
Properties dialog box options is a serious flaw. Had I not established
interoperability amongst the three IDEs, which enabled me to
compare build options, I would have had to spend lots of time on
forums trying to get an answer, or worse, I would have naively
continued development, ignorant of the application's unnecessarily
larger size.

3.
Failure to upload code
 There seems to be a slight upload bug
<x>AtmelStudio (caveats)[AUX]avrdude fails to upload code into chip</x>

which pops up on occasion, but which can sometimes be a real

nuisance, requiring restarting AtmelStudio in order to successfully
upload code into the microcontroller. The problem is that when doing
an upload into the microcontroller, avrdude will read the code from
flash memory and occasionally terminate cleanly without having
uploaded the .hex file into the microcontroller. It finishes by saying
Thank you. If you do not see that there is an error, you will carry on
normally and will be surprised when the changes you have
implemented in your code have no effect upon your application's
behavior. At this point, you may waste a lot of time searching for the
cause, not realizing that avrdude did not do its job. The solution is to
restart AtmelStudio, but that is not all there is to it.
Normally, AtmelStudio will first read the contents of the Microchip's
flash memory, it then writes the .hex file into it and finally reads it
again. The typical normal behavior output is:
[Link]: AVR device initialized and ready to accept
instructions
Reading | ################################## | 100% 0.01s
...
[Link]: writing flash (25744 bytes):
Writing | ################################### | 100%
.14s
Reading | ################################## | 100% 3.31s
[Link]: verifying ...
[Link]: 25744 bytes of flash verified
...
[Link] done. Thank you.
Every once in a while, you will get abnormal behavior which is not
readily apparent, as follows:
 Reading |
################################################## |
00% 0.01s
...
[Link]: reading input file
"D:\...ProjecDir\ArduinoCore\Debug\[Link]"
[Link]: cannot open input file <-- THIS IS THE HARD TO
SEE LINE
D:\..ProjecDir\PeseRucheBigGuyV11\ArduinoCore\Debug\libArduin
[Link]:
No such file or directory
[Link]: read from file 'D:\
..ProjecDir\ArduinoCore\Debug\[Link]'
failed
...
[Link] done. Thank you.
<x>AtmelStudio (caveats)[AUX]avrdude - cannot open input file error message hard to see</x>In the above there is indeed a

message which alerts you that there is a failure, namely cannot open
input file. However, it is muffled by a lot of output and what is more,
the polite Thank you is deceptive. There is no strong visual cue to
signal that things did not work as expected. Thank you indeed.
Conclusion: if you see three lines of multiple # symbols, the upload
was successful. If there is only one line of multiple #, avrdude did not
upload the .hex file into the microcontroller. Close AtmelStudio;
reopen it; and try again. This work around has worked for me every
time.

4.
Code changes
After modifying compiler, linker, and
<x>AtmelStudio (caveats)[AUX]Code changes no effect on runtime</x>

active configuration options to determine memory requirements, I

found that none of the changes in my code affected the application's
runtime behavior. Essentially, when the Arduino IDE uploads a .hex
file into the microcontroller, it uses the one contained in a temporary
build directory. Not so with AtmelStudio, when it does a build, it puts
the .hex file inside the project Debug subdirectory or inside the
Release subdirectory.
<x>AtmelStudio (build)[AUX]$ProjectDir, $TargetDir, $TargetName define .hex file location and name</x><x>AtmelStudio (caveats)[AUX]Failure to reckon with Debug vs.

When I started using AtmelStudio, I learned that

Release output causes confusion</x>

uploading an executable into a microcontroller required an external

command. Creating such an external command was easy, just do an
upload with the Arduino IDE in verbose mode, look at the messages,
copy paste the upload command, modify it somewhat in notepad,
and use it to define the upload external tool (Upload code into the
microcontroller (create external tool) page 44). Info on uploading the
file required that its full path be defined. All along I had been
developing in a Debug configuration, so naturally I set the path as
$(ProjectDir)Debug/[Link]
I had inadvertently hard coded the Debug directory into the file's
path. Of course, if I switched the configuration to Release, the .hex
file would be put inside the Release subdirectory, not the Debug one.
It was absolutely normal that after making the changes to my
application, building, and updating it, I should not witness the results.
All along I had been uploading the old .hex file from the Debug
subdirectory, not the new one from the Release folder I was working
on. The problem was solved by redefining, the file path above to:
$(TargetDir)/[Link]
TargetDir automatically includes Debug or Release according to the
configuration. Problem solved. See Failure to upload code (page 66).
Note that if you are using AtmelStudio with Visual Micro (page 84),
you avoid this problem because Visual Micro provides an upload
feature.

5.
Second set of header files import
<x>AtmelStudio (caveats)[AUX]Mysteriously imports second set of header files into ArduinoCore directory</x><x>Header files (.h)[AUX]AtmelStudio mysteriously imports second

AtmelStudio copies the project's header files .h

set of header files into ArduinoCore directory</x>

into the ArduinoCore/include directory when importing an Arduino

sketch. This is most inconvenient because you will copy the .h and
.cpp files into a different directory, the project's source files directory.
AtmelStudio's copying the header files into the ArduinoCore/include
directory poses several issues:

Since, as you develop your application, you will be working on

both .h and .cpp files, they should be together in the same
source files subdirectory.
If you want interoperability between AtmelStudio and the
Arduino IDE, you must locate both .h and .cpp files in one and
the same directory. This is pure common sense.
AtmelStudio does not care where the .h files are located, as
long as the #include statements find them.

If you leave project header files .h files in the ArduinoCore/include

directory, there is the danger of a mix up since two sets of the same
files coexist. Some changes could be undertaken in one set, other
changes in the other set. There should be only one set of header
files. I recommend that you delete the project .h files which
AtmelStudio placed in the ArduinoCore/include directory and stick to
the header files .h files alongside the .cpp files inside AtmelStudio's
source file subdirectory.

6.
Quick access to functions pulldown
selection box failure
Remember that useful
<x>AtmelStudio (caveats)[AUX]File has 20+ functions, only two showed up in the quick access selection box</x>

little text box in the upper left-hand corner of a window, the one
which enables you to quickly access any of the functions defined in
the file? Well, while working on a file, I clicked on the little text box to
access a function in the file. I was surprised to find only two functions
listed out of a total of 20+ functions. Yet the application compiled
correctly and formatted correctly - indentations were all correct.
Argh!!! What was going on?
 To make a long story short, the problem
<x>AtmelStudio (caveats)[AUX]Extra */ causes indentation failure</x>

was an unbalanced opening brace { inside an inactive #ifdef...#endif pair,

as follows:
//#define DO_SOMETHING
#ifdef DO_SOMETHING
...
DoSomething( ... ) { // Extra brace
...
#endif // DO_SOMETHING
I had copied/pasted the first line of the function definition into my
code, forgetting to replace the opening brace with a semicolon.
Since the preprocessor excluded this bit of code, the compiler
reported all is well. Had I uncommented the //#define DO_SOMETHING, the
compiler would have reported an error and then forced me into doing
something about it.
The conclusion is that #defines will isolate unwanted code during
compile time; it will not isolate code during source code formatting. It
took me quite a while to find the cause underlying the missing
functions displayed.

7.
Improper indenting
<x>AtmelStudio (editor)[AUX]Indenting in AtmelStudio called formatting</x><x>AtmelStudio (caveats)[AUX]Indenting of if...else statement depends on where opening curly brace

AtmelStudio has a slight problem when indenting code in an

is placed</x>

if...else statement. If the opening brace of an if...else statement is located

in the same line as the if and then the else, it does not indent the else
part statement correctly; as follows:
if ( ... ) {
some code
} else {
some code
} // if...else
Notice above how the if's closing curly brace is indented inwards - it
should be at the same level as the if keyword.
If the opening brace had been put on a separate line, you will get
correct indentation:
 if( ... )
{
some code
} else
{
some code
} // if...else
Getting opening and closing braces to locate at the same indent
level is an important rule when indenting. Putting the opening brace
in the same line as the if/else or in the following line is a question of
style. I consider failure to indent the else properly under certain
circumstances, despite the fact that the application compiles error
free, to be an AtmelStudio flaw.

8.
Failed code formatting
Indentations
<x>AtmelStudio (caveats)[AUX]Extra */ causes indentation failure</x><x>AtmelStudio (editor)[AUX]Format document indents code entire file</x>

(code formatting) improve readability; render program logic more

understandable; and signal that parens and brace pairs are correct.
If lines of code seem to be too far in or too far out; i.e., they do not
seem to be aligned properly; something is wrong; there is a missing
something: comma instead of semicolon, parens, brace, or other.
The programmer can quickly locate the offending line by tracing back
to where the indentation problem starts.
Occasionally, AtmelStudio's Format Document function loses it (Edit-
-> Advanced--> Format Document). On one occasion, after doing a
Format Document, code got aligned flat against the left edge of the
page; indentations were lost. This was most unnerving since
readability was tremendously impaired and missing or excessive
braces were next to impossible to find.
It took me quite a while to discover that this behavior is caused by
chunks of /*...*/ commented out code. Everything after the
commented code got aligned flat against the left margin, but
everything before was fine.
I first thought that the cause was the size of the /*...*/ block. But no!
This was not the cause. The cause relates to what I usually do when
I comment large sections of code which I might want to uncomment
later on. To render code more visible, I added a bunch of * with an
ending */ as the last line. This is where I made the mistake. The last
line is legal, there are two */ end of C style comment markers. The
first of the two */ (end of comment) is ignored; the second one pairs
with the start comment /* of the first line. Thus:
/****************************************
This is a large
block of code
which I might want to uncomment
....
*/**************************************/
If, having done the above, you do a Format Document, all the code
below the commented block will be stuck to the left margin. Yet, the
application compiles error free and runs. I should originally have
defined my two comment lines as follows:
/****************************************
...
*****************************************/
The fact that there is only one end of comment marker */ in the last
line makes all the difference. Problem solved.
This bug took me a while to resolve. The rule should be, if the
application compiles error free and runs, programmer assistance
tools should also work as expected.

9.
Segmentation faults
The Arduino IDE's
<x>AtmelStudio (caveats)[AUX]Occasional segmentation faults are benign; no clues on causes; do rebuild to fix</x>

segmentation faults problem has proven to be painful. Usually, the

problem could be fixed by doing a build again after modifying #defines.
But there were occasions which really stumped me, resulting in
aggravation and lots of wasted time. Ultimately the segmentation
fault problem prompted me to switch to AtmelStudio as my main
Arduino development tool. But AtmelStudio also fails on occasion
with a segmentation fault message. The AtmelStudio solution has
systematically been to re-do a Build. This has always solved the
problem, yet it would sometimes require several tries.
The problem with segmentation faults is that the accompanying error
messages give you no clues on causes; however, on occasion, there
are useful messages. What follows is an example whereby the
AtmelStudio segmentation fault error message did point to a specific
file/line location. The closing brace of a function was at fault;
apparently, the closing brace in the code below caused a
segmentation fault.
void MyFunction() {
...
}
Adding a comment to the closing brace
void MyFunction() {
...
} // MyFunction()
solved the problem.
<x>AtmelStudio (caveats)[AUX]Occasional segmentation faults are benign; no clues on causes; do rebuild to fix</x>

Why? Just as with the Arduino IDE, shifting things around by

adding/removing comments solves the problem. This being said, I
still favor AtmelStudio. The Arduino IDE failed too often and became
a real nuisance. AtmelStudio, after two years of working with it daily,
did fail occasionally, but one or several rebuilds overcame the
problem. AtmelStudio segmentation faults are non-blocking such as
they were with the Arduino IDE (version 1.8.9. I have not checked
version 2.1.1).

10.
Errors not listed in the error list
AtmelStudio's Error List and Output
<x>AtmelStudio (caveats)[AUX]Error and warning messages not displayed</x>

windows open automatically when doing a Build or Rebuild solution.

This is the default behavior, but it does not always work. I did
encounter a couple of issues:

No messages get displayed in either window - When doing a

Build or Rebuild solution, if the Error list and/or Output
windows are tucked way down to maximize workspace, you
will discover that no messages will be displayed in these
widows when you pull the window up to look at their content. If
 you need extra space to work, the best solution is to close
these windows. Upon doing a Build or Rebuild solution, they
will open automatically.
The Error list Window refuses to open - I sometimes decide to
close the Error list window, the one which displays compilation
errors. I would undertake changes in my code, then do a build.
All being well and good, the compiler did its job, it opened the
Output window, telling me that there are errors in my code.
Build: 1 succeeded or up-to-date, 1 failed, 0 skipped

Despite their being an error, the

<x>AtmelStudio (caveats)[AUX]Error and warning messages not displayed</x>

Error list window did not open - I could not see the compilation
errors. I have no idea why! So, I went to the Menu-->Windows to
open the Error list window, but in the displayed list there is no such
window. The question then was: How do I get the Error list window
to open and display my programming errors? After searching for a
while with lots of trials and errors and still not finding a solution, I
accidentally discovered that resetting windows to default via Menu-
->Windows-->Reset prompts the Error list window to show up
again. This is not the least bit intuitive.

A make error does not

<x>AtmelStudio (caveats)[AUX]Make error from missing .cpp file drowned in reams of messages</x>

get displayed in the error list window - Having successfully

displayed the Error List window, I proceeded to rebuild the
application. Unfortunately, I seemed to have hit another
obstacle. The Build and Rebuild Solution both yielded a failure
in the output window. However, the Error list window had 0
errors, 0 messages, and 0 warnings. I deliberately introduced
an error into one of the source files and it did report the error in
the Error list window which indicates the compiler was doing its
job. Having fixed the deliberate error, a rebuild yielded the
following in the end of the output window:
Done building target "CoreBuild" in project
"[Link]" -- FAILED.
Done building project "[Link]" -- FAILED.
The above is at the end of 15 pages of output.
What is going on? Since I am doing a parent/child application, I
loaded and built the child. This enabled me to double check that
AtmelStudio was functioning correctly. I thereby confirmed that
AtmelStudio was not at fault, something in the parent application was
causing the problem. This particular experience led me to discover
an additional AtmelStudio caveat: make utility errors do not get
displayed in the error list window. See next section Make utility errors
not listed.

11.
Make utility errors not listed
The previous section, errors
<x>AtmelStudio (caveats)[AUX]Make error from missing .cpp file drowned in reams of messages</x>

not listed in the errors list, illustrated how it can be difficult to find
what caused a build failure. The problem gets worse in that, although
the compiler and linker errors are displayed in the error list window,
there is nothing to lead you to suppose that make errors are not
listed. On the contrary, one would suppose that any error which
causes a build to fail would be listed in the error window.
This problem came up as I was trying to find out why a build failed. I
looked at project dependencies and project build order; I even
copied the Arduino Core directory from the child application to the
parent application. Still, I got the FAILED message. I compared the
parent and child Tools-->Options settings, they were identical. I then
tried to upload the code into the microcontroller, NO GO, the
message was no hex file to upload. AtmelStudio was telling me it
could not do the build, but it failed to tell me why.
<x>AtmelStudio (caveats)[AUX]Make error from missing .cpp file drowned in reams of messages</x>Since my development

environment enables me to work with AtmelStudio or the Arduino

IDE on an identical set of files with just a change in a #define, I thought
that perhaps the Arduino IDE might shed some light on the problem.
Compiling with the Arduino IDE posed problems but I got undefined
references error messages. This meant that functions or symbols
were missing. Could a file be missing? I finally identified the cause of
the problem. Thank you, Arduino IDE. In the process of transferring
files common to both the parent and the child apps into AtmelStudio,
I had left out a .cpp file. Copying the missing .cpp file back into the
parent .h/.cpp directory and adding the missing file to the
AtmelStudio project solved the problem.
<x>AtmelStudio (caveats)[AUX]Make error from missing .cpp file drowned in reams of messages</x>This is one big thumbs up for

the Arduino IDE. It saved me hours of AtmelStudio work trying to

figure out the failure's cause. Despite its power and sophistication,
AtmelStudio failed to deliver a clear error message which would
have directed me to the cause of the problem. AtmelStudio did
indeed do its job, however, unlike the Arduino IDE, it failed miserably
by not telling me clearly what was wrong. I verified the 15 pages of
AtmelStudio's Rebuild Solution output, there is indeed mention that
something is missing, as follows:
"EEPROMstorage.o" ".././[Link]"
Finished building: .././ [Link]
make: *** No rule to make target '.././ [Link]', needed by
MemoryManagement.o'. Stop.
The above is on page 12 of a 15-page output - I supplied the bold
type. What it really means is that [Link] does not exist. There,
deal with it. AtmelStudio should have listed the undefined references
in the Error list window, such as the Arduino IDE did. That would
have immediately led me to the problem's cause. The reason
AtmelStudio did not list the undefined references error is due to its
not doing the link phase, which is where the undefined references
message would come from. The make utility, having found a
problem, i.e., a missing file, returned a FAIL error code, thereby
aborting the rest of the build process and preventing the linker from
reporting missing references. Had the file existed but the function not
been defined, AtmelStudio's linker would have reported the error, not
the make utility. This AtmelStudio errors list problem stems from
AtmelStudio's only listing compiler and linker errors in the errors list.
It does not list errors from the make utility. It should.

12.
Inexistant file in project not listed in
error messages window
 For one reason or another,
<x>AtmelStudio (caveats)[AUX]Build fail due to inexistant file in project not in error messages</x>

while working with the Arduino IDE, I decided I no longer needed to

use a particular class. I removed/modified all references to this class
and to its functions and removed the corresponding #include. I rebuilt
and ran the application. Perfect? Not quite, because upon switching
to AtmelStudio, the build failed. It was not a compiler error. It had to
be something else.
Upon analyzing the contents of the output window (copy/pasted into
notepad for easier reading), I found:
make: *** No rule to make target '.././[Link]', needed by
MyFile.o'. Stop.
It turns out that the project properties still referred to [Link] and
MyFile.h as being part of the project. The last step consisted in
opening the Solution Explorer to remove the two files I had physically
removed from the project's file list. Removing these fixed the
problem. To summarize: when removing a MyFile.h/[Link] file
pair from the project, do the following:

Remove or comment out all calls from [Link] and MyFile.h

content.
Remove or comment out the #include "MyFile" from the files in the
project.
Remove the [Link] and MyFile.h from the project
properties via the Solution Explorer.

After doing the above, things got back to normal. This error is the
inverse of the make utility errors not being listed: the file does indeed
exist, in the correct folder, but it has not been incorporated into the
project files list.

13.
Search and replace
<x>AtmelStudio (find/replace)[AUX]Careful with find/replace Opened Documents</x><x>AtmelStudio (find/replace)[AUX]To change application wise item name, consider

I should issue a word of

refactoring instead of find/replace</x><x>AtmelStudio (editor)[AUX]Refactoring - rename items throughout app</x>
warning concerning AtmelStudio's search/replace. One of the
options when doing a search/replace is to specify that this should be
undertaken in Opened documents; choosing this option can be a
little dangerous. When doing a search/replace with the Opened
documents setting, only documents which are open will be
processed. Unopened files containing items which should have been
modified are left as is. These unmodified items will cause errors easy
to find if they trigger compiler/linker errors, otherwise, the application
will mysteriously bug.
If you intend to use find/replace to change a variable's or function's
name throughout the application, you should consider Refactoring
instead.

14.
AtmelStudio code formatting and code
collapsing bug
Code formatting and code collapsing does not always work. You can
collapse functions with AtmelStudio by clicking on the little minus box
located to the left of the functions name (outlining in AtmelStudio
parlance). You can also format code by doing an Edit --> Advanced -
-> Format (document or selection). When there is no such little box,
neither of these AtmelStudio features work.
As I was doing work with AtmelStudio on some functions, there were
occasional functions which were not collapsible (no little box with the
minus sign) nor could they be formatted properly. After some trial
and errors, I came across two causes:

C-style comment after opening brace - Legacy code will often

contain a C-style comment after a function's opening curly
brace. Here is an example of some legacy code:
void SomeFunction() { /* this function needs to be rewritten */
...
} // SomeFunction
 The function was not collapsible, no little minus box in the left edge.
Changing the comment to C style, as follows:
void SomeFunction() { // this function needs to be expanded
rendered the function collapsible. This bug does not prevent the
function from being indented properly.

Constructor inline initialization - Inline initialization is not a

necessity. The programmer could well initialize class variables
with separate code, but this clutters constructor code
unnecessarily. A typical inline constructor initialization would
look as follows;
SomeClass::SomeClas( uint8_t _var1, uint8_t _var2)
: var1(_var1), var2(_var2) {
…
} // SomeClass( uint8_t _var1, uint8_t _var2)

You will find that the function above will neither be collapsible, nor
can it be indented properly. Removing the initialization list removes
this problem; but, doing this is not a solution.

15.
Auto format if...else problem
When I write if...else statements, this is how I like to see the code:
if ( 1st ) {
...
} else {
if ( 2nd ) {
...
} else {
...
} // 2nd if
} // 1st if
It is highly readable; embedded ifs are clearly displayed.
If you auto format the above with the Arduino IDE, it will do a good
job, not AtmelStudio, it will indent the else, thus:
 if ( 1st ) {
...
} else {
if ( 2nd ) {
...
} else {
...
} // 2nd if
} // 1st if
The code becomes difficult to read. The only work around is to place
the else in the line after the if's closing curly brace. This is what you
will get:
if ( 1st ) {
...
}
else {
if ( 2nd ) {
...
}
else {
...
} // 2nd if
} // 1st if
It is more readable but not as good as what you get with the Arduino
IDE.

16.
Refusal to uncollapse a section of code
When #ifdefs are numerous and several
<x>AtmelStudio (caveats)[AUX]Code remains obstinately collapsed</x>

levels deep, it sometimes happens that after collapsing code,

AtmelStudio refuses to uncollapse it via the little minus icon on the
left margin. The collapsed section remains obstinately collapsed.
The solution is to format the entire document (Edit-->Advanced-->
Format Document). This will uncollapse everything in the file.
Nevertheless, before doing this you may try to get the code to
uncollapse by doing some editing in the visible lines of the collapsed
code, but this does not always lead to success.
 17.
Unable to generate code for new
microcontroller
<x>AtmelStudio (caveats)[AUX]Arduino core When creating an
content seems locked into device first compiled for</x>

AtmelStudio project from an Arduino project, a device needs to be

specified. I generally specify the ATmega2560 (Arduino ATmega
board) because it is my development board. Note that I occasionally
want to run the application MemMgt on a ATmega328P (Arduino
Uno board). When doing a rebuild, I specify the new microcontroller
in the ArduinoCore and MemMgt project properties and do a rebuild
solution. Because of the dependencies, Arduino Core gets rebuilt
first, and then my project (MemMgt). I am dismayed by the result -
many error messages on ArduinoCore such as:
Error recipe for target 'src/core/wiring_digital.o' failed
Error 'DDRA' undeclared here (not in a function)
Error 'DDRE' undeclared here (not in a function)
Error 'DDRF' undeclared here (not in a function)
... 40+ such errors
If I reset the device back to ATmega2560, the device of the original
Arduino sketch import, things go back to normal. It appears that if I
wanted to recompile successfully for a new microcontroller, I would
have to do so in a separate Arduino sketch import. This would
require that I define two distinct project directories, one for the
ATmega328P and one for the ATmega 2560, thereby requiring two
sets of files and all the problems this could engender. I may be
missing something.
<x>AtmelStudio (caveats)[AUX]Arduino core content seems locked into device first compiled for</x><x>Arduino (build)[AUX]Specify microcontroller and board, changing them

If you wanted to change the board/controller combination

transparent to programmer</x>

in the two environments, you would have to do the following:

The Arduino IDE allows you to specify microcontroller and

board, sometimes the board only. Go to Tools-->Board and
choose. Arduino will use the corresponding core libraries and
will generate the corresponding output to be uploaded into the
 microcontroller. If the code were to be compiled for a different
board/controller combination, just specify the new choices,
compile, and upload into the microcontroller.
AtmelStudio - When importing a sketch and defining the
board/controller combination, AtmelStudio generates the
appropriate Arduino core files from the info supplied to it. Later
on, you might want to modify the target microcontroller via the
project properties and recompile. The board choice does not
seem to come into play and trouble ensues.

Here is an AtmelStudio situation I ran into quite unexpectedly. My

project requires serial communications between ZigBee radio
modules. During the development phase, I use an ATmega2560
board because it has lots of memory and because it provides several
hardware serial ports. I use hardware serial to display [Link] in the
console and hardware serial1 to communicate with the ZigBee radio
modules.
When I did the Arduino IDE's sketch import, I naturally specified the
ATmega2560 board and microcontroller since that is where I would
be doing development. The process was simple, the application
worked well. The release version was meant to run on the smaller
ATmega328P which provides only one hardware serial port which I
use for print-based debugging. Since the ATmega328P does not
have a second hardware serial port, my recourse was to use the
software serial library to handle ZigBee communications. This means
that somewhere in my code:
#ifdef MEGA328P
#include <SoftwareSerial.h>
extern SoftwareSerial XBeeSerial; //RX, TX
#endif // MEGA328P
I modified the device setting in the Project-->Properties window from
ATmega2560 to ATmega328P and did a Build. Here is what I got:
SoftwareSerial.h: No such file or directory
This is not what I was hoping for. The explanation is simple. When I
originally imported my project into AtmelStudio, I had specified
Arduino_2560 in the project properties device options thereby not
requiring the SoftwareSerial library - the Arduino core was built
without it. So, I figured that setting AtmelStudio's device option to
ATmega328P and doing a Rebuild Solution would rebuild the
Arduino Core content and integrate the Software Serial library. This
did not happen. I found only one solution: create an entirely new
project specifically for the ATmega328P microcontroller - not ideal.
When you do a Rebuild Solution, it passes the microcontroller
specification to the compiler, but it does not redefine files to rebuild.
It maintains the files initially chosen upon the project's creation.
Doing a Rebuild all did not solve the problem.

10.
Keyboard shortcuts
You will find below sources of information on AtmelStudio's keyboard
shortcuts:

AtmelStudio V7
<x>AtmelStudio (keyboard shortcuts)[AUX]AtmelStudio V7 User Guide - 14 pages which itemize keyboard shortcuts</x>

User Guide - It has 14 pages which itemize predefined

keyboard shortcuts (see the Documentation section next).
Note that I found one listed shortcut which was not active;
CTRL+SHIFT+SPACE to list parameters which a highlighted
function takes.
Menu item - Some menu items propose keyboard shortcuts.
AtmelStudio's VAssist help - You will find help at the bottom of
the VAssist menu item. It is a must read if you are to use
VisualStudio as it presents shortcuts which will prove to be
valuable time savers. Open the VAssist shortcuts dialog box - it
displays the predefined VAssist keyboard shortcuts. The Print
All button displays all VAssist commands, the left column items
which have a thumbs up icon are predefined , all others are
customizable.
<x>AtmelStudio (keyboard shortcuts)[AUX]Standard Windows shortcuts supported</x>Standard Windows

shortcuts - AtmelStudio supports the standard Windows

shortcuts such as ctrl-F for find, ctrl-H for find/replace, ctrl-C for
 copy, ctrl-V for paste, ctrl-X for cut/copy.
Search the Internet - If you Google AtmelStudio V7 keyboard
shortcuts, you will find references including a Microchip
documentation site.

After having to deal with the workspace font size, I blindly tested
random shortcuts. You will find below shortcuts I have found by trial
and error.
ctrl+space - operates in one of three ways:

Cursor positioned on a non-highlighted function opens a text

box with functions. Selecting one opens a second text box; it
displays the function's parameters and help on using it.
Cursor positioned in empty space - same as above except that
the list narrows down as a function's name gets typed.
Cursor positioned on a highlighted word - opens a text box with
snippets to plug in.

Here are a few useful keyboard shortcuts:

ctrl+mouse wheel roll - Increases/decreases workspace font size.
ctrl+shift+< and ctrl+shift+> - Same as the mouse wheel, it
increases/decreases workspace font size.
ctrl+left mouse click highlights a word. As you slide the cursor, it will
continue highlighting on a one word at a time basis as opposed to
highlighting a character at a time.
ctrl+l - Removes a line.
ctrl+m -- toggles collapse code - need to do this twice.
ctrl+u and ctrl+shift+u - To lower-case/to upper-case.
ctrl+t - inverts current character with next character and advances
one character.
ctrl+$ - Opens solution explorer window on French keyboard
(AZERTY); ctrl+l on QWERTY keyboard.

11.
Documentation
 The main source for
<x>AtmelStudio (documentation)[AUX]Atmel (Microchip) Web site two inline doc packages: Getting started and videos</x>

information on AtmelStudio was the AtmelStudio 7 User Guide.

Search for MicrochipStudio in the Microchip Web site. You will come
to a page which contains a downloads and a documents section.
The user guide is located in the documents section. When inside
AtmelStudio studio, you will find two sets of help documentation:

Inline
<x>AtmelStudio (documentation)[AUX]Inline help under the Help menu; VAssist help under top level pull-down VAssist menu </x>

documentation located under the Help menu item. It seems to

be the same documentation as the AtmelSudio's User Guide
above.
VAssist help, located under the top level pull-down VAssist
menu item. This particular help posed a problem with Build
1391 in that the URL it points to ([Link]) no longer exists.
This help has since been updated in Build 2594. It contains
one particularly useful feature, a list of keyboard shortcuts. The
other help link, labeled Documentation, points to a Web page
containing a very extensive list of Microchip documentation,
several hundreds. Fortunately, there is a search textbox;
unfortunately, searching for the VAssist documentation did not
yield any results.
See Bibliography - AtmelStudio (now MicrochipStudio) (page
335).
 9.
Visual Studio

Visual Studio
Visual Studio is Microsoft's flagship development tool. It is a
professional tool, highly reliable (Word, Excel, etc. are <x>Which IDE to work with?
[AUX]Visual Studio - highly professional multi-language development tool</x><x>Visual Studio[AUX]IDEs: Arduino, AtmelStudio, Visual Studio, VS Code, PlatformIO, Visual

Micro</x><x>Visual Studio[AUX]Microsoft's flagship development tool - two versions, 2019 and 2022</x><x>Visual Studio[AUX]Multilanguage: C++, C#, Python... and multi-

platform: Windows, MacOS, Linux</x><x>Visual Studio[AUX]Extensions for embedded development: Arduino, ESP32, RaspberryPi, ...</x><x>AtmelStudio (general)

probably created with it), multilanguage (C++,

[AUX]AtmelStudio is Atmel specific Visual Studio</x>

C#, Python...), and multi-platform (Windows, MacOS, Linux). There

are extensions for embedded development (Arduino, ESP32,
RaspberryPi...). The learning curve is not steep at all. Furthermore,
the community variant which provides more than enough features for
Arduino development, is free.

1.
Visual Studio features
The question is: What can one do with Visual Studio? The answer is:
Plenty! Upon opening Visual Studio and clicking on new project, a
multiple-choice window opens. It is striking as it illustrates just how
much you can do with it. It offers many possibilities which depend on
the configuration you are working with.
Here are three Visual Studio Community programming scenarios you
may experiment with:

The out-of-the-box Visual Studio you just downloaded is bare

bones. When you first open Visual Studio, you get a window
which offers the possibility of opening a recent project, do
other things, and a Create a new project.
The 2022 version includes remote Linux work and much more; but,
surprisingly, you cannot create a simple print Hello world program
to run from a DOS box; you can however import a console app
plugin later on. And I did discover an Arduino related option:
something called Arduino project template. It looked promising (see
next section).
The 2019 version includes a console app (create a simple print
Hello world program to run from a DOS box) but does not include
any Arduino plugins as the 2022 version does; and none seems
available save the Visual Micro plugin described further down.

Console app (programs

<x>Visual Studio[AUX]Create Arduino apps via Arduino project template or via Visual Micro</x>

which run from a DOS box) - If you want to create simple

command line programs, i.e. programs which run as .exe files
from a DOS box, you need to install the Console App plugin.
When you do a Create New Project, a list of possible project
types opens. If you do not find the one you want, click on the
message at the bottom of the list which reads Not finding what
you're looking for? Install more tools and features. Do this and
you will find the Console App extension and more. This
extension having been installed, create a simple program, do a
build, open, and run the .exe file from a DOS box. You might
not need to create such programs since your focus is doing
Arduino work; but, having acquired this new skill, you may in
the future find it useful. As you are looking for the Console App
extension, you will discover that there are many other
extensions you might be interested in.
Visual Studio (Visual Micro) (page 82) for Arduino plugin - The
alternate way of creating an Arduino application with Visual
Studio, the first way being using the Arduino project template
with the 2022 version (see next section), is to install the Visual
Micro plugin . This plugin, available for both the 2019 and 2022
versions and for AtmelStudio, handles the build, upload, and
program output via a serial terminal. Visual Studio is relegated
to being a tool to handle source code editing and project
management. Visual Studio 2019 with the Visual Micro plugin
 becomes a twin of AtmelStudio with the Visual Micro plugin.
Note that Visual Micro also provides Serial debugging (page
125).

Icing on the cake if you get

<x>Visual Studio[AUX]Expand your horizons with Visual Studio (Python, Raspberry Pi...</x>

proficient using Visual Studio for Arduino applications, there are

hundreds of other plugins which you may find useful now or later
(Python, Raspberry Pi, Console App introduced above...).
<x>AtmelStudio (general)[AUX]Develop with Visual Studio 2022 + Visual Micro, hardware debug with AtmelStudio + Xplained boards</x><x>AtmelStudio (hardware-based

debugging)[AUX]Develop with Visual Studio 2022 + Visual Micro, hardware debug with AtmelStudio + Xplained boards</x><x>Visual Micro[AUX]Develop with Visual Studio

2022 + Visual Micro, hardware debug with AtmelStudio + Xplained boards</x><x>Visual Studio[AUX]Develop with Visual Studio 2022 + Visual Micro, hardware debug with

AtmelStudio + Xplained boards</x><x>Which IDE to work with?[AUX]Develop with Visual Studio 2022 + Visual Micro, hardware debug with AtmelStudio + Xplained

I recommend using Visual Studio 2022 with Visual Micro as your

boards</x>

default development environment, particularly since it is possible to

establish interoperability with the Arduino IDE and AtmelStudio. You
would thereby be using Microsoft's latest dev-tools development
environment. Having said this, I currently (May 2023) know of only
one way to undertake practical Arduino hardware-based debugging,
i.e., by using the Xplained boards (see Hardware-based debugging
page 119) with AtmelStudio. Maybe I am wrong, but I have not found
a way of using these boards with Visual Micro.

2.
Visual Studio download
The term Visual Studio refers to a family of products, both version-
wise and variant-wise. Upon googling Visual Studio download, you
are directed to a Microsoft Web site
([Link] which proposes
Visual Studio 2022 in four variants (Community, Professional,
Enterprise, and Preview). Community is the one you want; it is a free
download; the other variants require a yearly subscription, although
a free trial is proposed.
But notice that the variant proposed is the 2022 one. Atmel Studio
being based on the 2019 version, you might want to download the
2019 version. Note that the two versions, 2019 and 2022, can be
installed on the same computer and used side by side
simultaneously. They do not seem to interfere with each other.
To download the 2019 version, go to
[Link] You will find
2019, 2017, 2015, and older versions, choose the 2019 one and
click on download in the window which just opened. You will find 49
products to download; choose the Visual Studio Community 2019
one (Version 16.11 11/April/2023). You may at, at this time, be asked
to login to a Microsoft account; do so or create one and login.
Afterwards, do the download as you would download any other
product.

3.
Visual Studio 2019 vs. Visual Studio
2022
I cannot
<x>Visual Studio[AUX]Straightforward installation, two Arduino dev solutions</x><x>Visual Studio[AUX]Choose between two versions: 2019 and 2022</x>

report on fundamental differences between the two most recent

versions: 2019 vs. 2022. But I did note some differences, as follows:

Upon installing the two, you will find that the desktop icons are
identical. You can differentiate one icon from the other by
hovering over the icon; a popup opens with the Visual Studio
version the icon is linked to.
Both variants can coexist on the same computer. They can be
used simultaneously.
AtmelStudio being based on Visual Studio 2019, you will find
their user interfaces practically identical. The 2022 version's
user interface differs, but the differences are not substantial.
Each version offers some plugins by default:
The 2019 version offers a console apps plugin by default. An
Arduino plugin, aside from the Visual Micro one, does not
seem to be available.
 The 2022 version offers an ArduinoProjectTemplate by
default but not a console app. A console app is nevertheless
available.
You can install Visual Micro into both versions. These require
their own activation keys.

4.
Visual Studio for Arduino
developers
Since AtmelStudio is a customized version of Visual Studio 2019, my
coverage of Visual Studio is scant as practically everything I wrote
concerning AtmelStudio applies to Visual Studio. Differences
between the two are mostly cosmetic; if you want more details using
Visual Studio, skim through the AtmelStudio (page 37) chapter.
There is one important difference though, AtmelStudio is a
standalone product which is Atmel microcontroller compatible right
out-of-the-box. Visual Studio 2022 does incorporate an Arduino
development plugin by default; however, this plugin is somewhat of a
cheat, by this I mean that it bypasses uploading and running the
application on a board, you run it directly as an .exe file from a DOS
box which implies that you are not driving any electronics. It does
nevertheless possess a saving feature: it enables the developer to
work directly on the .ino file and interfaces with the Arduino IDE so
that an upload and execution directly from the microcontroller be
possible. See Visual Studio (Arduino project template) (page 82).
If you really want to run the application from an Arduino board
without using the Arduino IDE, as described above, you need to
install Visual Micro for Visual Studio, the same Visual Micro you may
have installed to enhance AtmelStudio.
Visual Studio 2022 is probably the best choice for futureproofing
your development needs, including embedded systems. I
recommend that you start with AtmelStudio (page 37), without Visual
Micro and get familiar with it. Once that is done, download Visual
Studio 2022 and install Visual Micro for Visual Studio (next chapter).
Your learning curve will be practically nil.
With this combination you get the best of possible worlds:
AtmelStudio is the best tool I found for doing Arduino Hardware-
based debugging (page 119). Visual Studio 2022 is, IMHO, the best
C++ development tool although one might argue in favor of
PlatformIO and possibly other tools I am not familiar with.

5.
Visual Studio (Arduino project
template)
I knew, from my
<x>Visual Studio[AUX]Out-of-the-box Arduino Project Template solution creates both .ino file and command line .exe program</x>

AtmelStudio experience, that Visual Micro would respond to my

needs; however, curiosity got the better of me upon seeing that
Visual Studio proposes an Arduino project template solution. I
decided to give it a try, limiting myself to doing a Hello World.
In the Type of Project creation window, I scrolled down to Arduino
project template. I clicked on it and proceeded to create a project I
named SimplePT. Little was I prepared for what I got. These
Microsoft guys are smart. The build generates both an .ino file which
you compile, upload, and run from the Arduino IDE, and an .exe file
which outputs its prints to a DOS box.
The application's structure is:

[Link] file does one thing only, it calls Sketch.h.

Sketch.h contains setup and loop, just as a .ino file normally
contains.
[Link] is the console version (.exe) of your program.
It does an include on Sketch.h.
Other files are added to the build if present.

Upon doing a build, Visual Studio generates a new .ino and a new
.exe console program; they do exactly the same. The .ino file can be
compiled/uploaded via the Arduino IDE and ran from the board; the
.exe file is ready to run from a DOS box. Of course, all you are going
to get from the .exe console program are prints in the DOS box since
it does not drive any electronics. But this fact alone may be useful for
debugging purposes. Neat!
But there could be issues as I am not sure that the DOS box
executable is created with the same compiler as the Arduino IDE
(GNU toolchain). I have not tried this solution on a large application.
Just remember that it exists and can be useful if you want to test
your application via a DOS box prior to testing it in a microcontroller.
In creating the ArduinoProjectTemplate, Microsoft combined the
console extension with a hook into the Arduino IDE so that you could
do a build/upload from it; simple solution, big rewards.

6.
Visual Studio (Visual Micro)
<x>Visual Studio[AUX]Create Arduino apps via Arduino project template or via Visual Micro</x><x>Visual Studio[AUX]Visual Micro for Visual Studio identical with Visual Micro

The Visual Micro route to create Arduino applications from

for AtmelStudio</x>

within Visual Studio seems, IMHO, to be the more straightforward

route: Open Visual Studio and click on the top toolbar Extensions -->
vMicro --> Open Existing Arduino Project; select an .ino file. Do a
build, click on the View Port Monitor in the bottom of the vMicro
selection box; it opens a serial monitor similar to the Arduino IDE
one. And that is it. Pure magic, it works!
You can debug with the Serial debugging (page 125) from the vMicro
menu selection box. For more info on using vMicro see the Visual
Micro chapter (next chapter).
Note that there are two Visual Studio versions: version 2019 and
version 2022. AtmelStudio is based upon Visual Studio 2019. Visual
Micro licenses being Visual Studio based, one license covers both
AtmelStudio and Visual Studio 2019. If you also want Visual Micro
for Visual Studio 2022, you will need to get a second license.
 7.
Visual Studio (debugging)
Visual Studio, with or without
<x>Visual Studio[AUX]Debugging - only Serial debugging via Visual Micro seems feasible</x>

Visual Micro, does not seem to support Arduino Hardware-based

debugging (page 119) (but I might be wrong). A priori, the only real
practical hardware-based debugging of Arduino apps running on an
Arduino board would be using the ATmeg328P-Xmini board with
AtmelStudio (without Visual Micro). See Hardware-based debugging
(page 119).
But, you could envision Serial debugging (page 125) from within
Visual Studio via Visual Micro.
 10.
Visual Micro

Visual Micro
<x>PlatformIO (general)[AUX]IDEs: Arduino, AtmelStudio, Visual Studio, VS Code, PlatformIO, Visual Micro</x><x>Visual Micro[AUX]Three identical Arduino plugins: for

AtmelStudio and for Visual Studio (2019 and 2022)</x><x>Visual Micro[AUX]Develop applications for Arduino, ESP32, RaspberryPi, and others</x><x>Which IDE to work with?

[AUX]Visual Micro - plugins forVisual Micro is a plugin for

AtmelStudio and Visual Studio Arduino work</x>

AtmelStudio and Visual Studio to undertake Arduino work. It handles

the build and upload of Arduino programs, provides a serial terminal,
and supports serial debugging. AtmelStudio and Visual Studio get
relegated to being code editors and project managers. Visual Micro
for Visual Studio is practically identical with Visual Micro for
AtmelStudio, the downloads are separate though. Note that there are
three versions of Visual Micro: one for AtmelStudio, one for Visual
Studio 2019, and one for Visual Studio 2022. However, since
AtmelStudio is Visual Studio 2019 based, although you need two
separate downloads for these two products, you need only one
license key because AtmelStudio is Visual Studio 2019 based.
Visual Micro brings to AtmelStudio and to Visual Studio some
features they are lacking for Arduino development, namely: a serial
terminal, an upload into the microcontroller (no need to create an
external tool), serial debugging (Serial debugging page 125), and
other.

Visual Micro provides a

<x>Visual Micro[AUX]Provides a serial monitor, compile/link/upload, serial debugging</x>

serial monitor which interfaces with the board and launches the
program.
Build and upload to microcontroller - When working with
AtmelStudio, there is no need to provide an external tool to
upload the program into the microcontroller (Upload code into
the microcontroller (create external tool) page 44).
<x>Debugging serial[AUX]Visual Micro enables serial debugging w/o dedicated hardware</x>Serial debugging provides

limited breakpoint capabilities, undertake conditional stops,

look at data values, and more. Despite its limitations, it offers
 features not available with print-based debugging but it is not
as powerful and versatile as Hardware-based debugging (page
119).

Visual Micro for Visual

<x>Visual Micro[AUX]Three identical Arduino plugins: for AtmelStudio and for Visual Studio (2019 and 2022)</x>

Studio, as far as I can tell, is identical to the AtmelStudio variant. See

Visual Micro for Visual Studio (page 85).
A note on pricing: a one machine hobbyist yearly fee costs $25/yr.
Upon downloading you do get a 45-day trial period. Note that you
can install Visual Micro into AtmelStudio and Visual Studio 2019 with
the same license. You would need a different license to run Visual
Micro from within Visual Studio 2022.

1.
Visual Micro for AtmelStudio
(MicrochipStudio)
Downloading Visual Micro for Atmel
<x>Visual Micro[AUX]vMicro for AtmelStudio in top-level toolbar</x>

Studio was a breeze. Having downloaded and successfully installed

it, the challenge was getting to run an Arduino program. I opened my
MemMgt AtmelStudio application, the build/upload process was
immediate; and it ran successfully right away as witnessed from
Visual Micro's console. Great!
I found two major differences when using AtmelStudio with Visual
Micro as compared to using it without:

Single-level directory structure - The classical way of importing

an Arduino project into AtmelStudio generates a two-level
directory structure and creates a [Link] file which
contains setup and loop; the .ino file is ignored. When using
Visual Micro, AtmelStudio works on a single-level directory
structure.
Build from Visual Micro - Building the Arduino application is now
done via Visual Micro's menu Build item. The build from
 AtmelStudio's Build top toolbar menu item automatically
invokes Visual Micro's Build.

Having successfully incorporated Visual Micro into AtmelStudio and

tested it, the next step was doing the same with Visual Studio (next
section).

2.
Visual Micro for Visual Studio
<x>Visual Micro[AUX]vMicro for Visual Studio tucked away in Extensions of top-level toolbar</x> Installing Visual Studio 2022
and adding the Visual Micro plugin was a breeze, nothing to it. But,
upon opening Visual Studio 2022, you will not find Visual Micro
(vMicro) right away; you will find it inside the Extensions top toolbar
menu item, not as a top-level toolbar menu item as in AtmelStudio.
But, if you look carefully, you will find a vMicro icon in the second
toolbar down. I opened both vMicros, Visual Studio's and
AtmelStudio's, they were identical except that the Atmel Studio
variant has one extra line: Configure Arduino IDE location.
<x>Visual Micro[AUX]vMicro for Visual Studio project location defined from Arduino IDE preferences</x>Upon clicking on vMicro, a

pane opened which contained, amongst other items, New Arduino

Project and Open Existing Arduino Project. This was most
reassuring; I created a new project and then %#@£*, where did
Visual Studio store my project? vMicro provided me with no clue as
to where the project got created. After searching around, I
discovered that the project was tucked away somewhere in
C:\users\... Drats, I don't want it there. So, I searched around and
found that I could change the project's location in vMicro-->General--
>Project and Solutions-->Locations-->Project Location. Good, I set
my project's location to where I wanted my Visual Micro Visual
Studio to reside. Upon recreating an Arduino project, I found that my
new project, %#@£*, was still in the old location. Why? After
googling the Internet, I found a clear answer: vMicro in Visual Studio
gets its file setting location from Arduino, not from Visual Studio. I
consequently opened the Arduino IDE, went to Preferences--
>Sketchbook location and changed it to my target directory. Sure
enough, it worked. My project was indeed created where I wanted it
to be.
From then on, vMicro for Visual Studio was the same as vMicro for
AtmelStudio.
<x>Debugging serial[AUX]Visual Micro enables serial debugging w/o dedicated hardware</x>I tried a serial-based debugging

session, it worked fine except that I was in for an extended learning

curve (see Serial debugging page 125).
 11.
VS Code

VS Code
Visual Studio Code, referred to as VS Code henceforth, is
Microsoft's programming foundation platform. Off <x>VS Code[AUX]IDEs: Arduino, AtmelStudio,
Visual Studio, VS Code, PlatformIO, Visual Micro</x><x>Which IDE to work with?[AUX]VS Code - a foundation upon which many tools are created, including PlatformIO</x>

<x>Which IDE to work with?[AUX]VS Code - a foundation upon which many tools are created, including PlatformIO</x><x>VS Code[AUX]Microsoft's programming foundation

the shelf, you cannot do much with it except

platform</x><x>Visual Studio Code[AUX]See VS Code</x>

edit source code such as you might do with Notepad++. Several

amongst many extensions transform it into a powerful full-fledged
programming tool. It can be morphed into just about anything: an
intelligent C++ editor which can compile, link, and upload programs,
an IDE such as PlatformIO (page 98), and a lot more.
The first order of business after downloading VS Code is to
understand that VS Code is meant to be enhanced via extensions.
You will find, by clicking on the extensions icon, the bottom one in
the left-hand toolbar, that there is a huge number of extensions,
some of them may be gadgets; yet others may be just what you are
looking for; it is up to you to explore. A search option enables you to
search for extensions which meet specific needs (e.g. C++, Arduino,
embedded systems, Python...).
Once installed, extensions need to be configured (VS Code
extensions page 91). This phase is important as extensions need to
be told how and with what to do things (e.g. compile/link/upload).
I worked with VS Code progressively. I started by experimenting VS
Code as a stand alone tool. At this stage it is similar to Notepad++. I
then progressively added extensions (C++, serial monitor, Arduino,
PlatformIO).

VS Code standalone - Being aware as to what one can do with

VS Code is important to later differentiate VS Code features
from extension features. See VS Code extensions (page 91).
Arduino awareness - There are many VS Code extensions,
 one of which is a Microsoft VS Code Arduino extension (page
93). Please note that installing the Arduino extension also
installs the VS Code C++ extension (page 94) and the VS
Code serial terminal extension (page 95).
PlatformIO full-fledged IDE - The PlatformIO team has done a
tremendous job extending VS Code into becoming a
professional level embedded systems IDE (PlatformIO page
98).

Throughout my VS Code testing, I found it important to differentiate

VS Code features from installed extensions features. Not doing so
can lead to confusion. See VS Code extensions (page 91).

1.
VS Code features
VS Code out-of-the-box does not do much - at best,
<x>VS Code[AUX]VS Code features</x>

it is a Notepad++ equivalent. However, there are hundreds if not

thousands of extensions which transform it into any of many
powerful tools. It is inherently infinitely flexible as it is up to the
extension's creator to use it as a foundation for his/her new ground-
breaking application.

Hundreds/thousands of
<x>VS Code[AUX]Extensions can be installed/uninstalled and enabled/disabled</x>

extensions extend VS Code. They can be installed/uninstalled

and enabled/disabled directly from VS Code thereby bringing
flexibility to what can be done with it.
<x>VS Code[AUX]Provides source control via Git repository</x>It provides source control via a Git

repository. Not having the need for source control, I have not
tried it. Source control addresses the needs of teamwork, not
individual work.
<x>VS Code[AUX]File, folder, workspace used for project management</x>File, folder, workspace are the

supports for project management.

<x>VS Code[AUX]User space vs. workspace</x>User space vs. workspace settings enable
 differentiating overall settings from specific project settings via
the user space /workspace concepts. User space refers to any
VS Code you might open in your computer. Workspace applies
to a specific project. A Workspace is a folder (directory aka
project) or group of folders. This distinction means that one
could set preferences at the User level and have them apply to
all work done via VS Code, irrespective of the folder being
worked on. Individual workspaces/projects may then be
customized.
<x>VS Code[AUX]Multilanguage: there are extensions for C++, Python, Perl...</x>VS Code is multi-language - It

can recognize C, C++, Python, and many other languages. It

supports color syntax, name completion, and much more, but it
does not go as far as checking for proper constructs. To do
this, you need to install an appropriate extension.
Find/replace in files - VS Code, being a Microsoft product,
supports the traditional ctrl-F and ctrl-H find/replace.
<x>VS Code[AUX]Find/replace supports regular expressions</x>Regular expressions (regex) are

supported in the VS Code find/replace. It seems to be a

standard regex, along with captures, not the wildcard Word
version.

2.
VS Code installation
Installing VS Code is simple. Google Microsoft Visual Studio Code;
you will quickly get to a download page for Windows, Mac, and
Linux. I choose Windows, clicked on download, and within a minute
or so, had VS Code installed in my computer. The file is small, less
than 500Kb.
Windows usually installs programs in C:\Program files. Do not look
for a VS Code installation in this folder, you will find the VS Code
installation in:
C:\Users\myself\AppData\Local\Programs\Microsoft VS Code
Having installed VS Code, you need to configure a few items before
using it (next section).
As I tried VS Code with the C++, Arduino, and PlatformIO
extensions, I ran across a lot of problems, as if certain items were
already defined and loaded in by default. For example, when
compiling .cpp code, it would consistently use the GNU compiler in
C:\Strawberry which is an old Perl installation. I consequently
removed all .vscode files from my C:\Users directory, there were
more than a hundred despite the fact that VS Code and its
extensions were removed. It seems that Windows does not do a
good job when uninstalling VS Code. Fearing that there might be
more VS Code files in my C: drive, I did another .vscode search; it
took a while. Sure enough, I did find two VS Code related files in my
C:\Strawbery Perl folder. I removed them.
I then reinstalled VS Code and searched for .vscode files figuring
that this time I would know where VS Code stored its settings. Upon
opening it, I was surprised to find that it remembered my VScode
folder. It should have lost the info because I removed all .vscode
files. Perhaps the info is in a registry; I consequently opened regedit
to check. Bad idea, lots of complicated stuff which could damage
Windows.
I did a C:\Users\MySelf search for .vscode again; explorer found 40
VS Code related files. They were mostly system files but it did
uncover two interesting .vscode files:

C:\Users\Myself\.vscode - There is an [Link] file which

contained an enable crash reporting set to true and an
extension folder which contained an [Link] file which
was empty.
C:\Users\agir\AppData\Local\Programs\Microsoft VS
Code\resources\app\extensions\debug-auto-launch\.vscode -
This .vscode folder contained a [Link] file which had the
${workspaceFolder} variable.

The question remained: Where is the ${workspaceFolder} variable

defined? This variable seems to be what I am looking for: How does VS
Code know which folder to work from? At this point, I asked myself:
what are .json files. As I quickly discovered, these are ubiquitous -
they are data files used by many application's .json files (page 96).
Still in the dark as to where VS Code is getting its folder info from, I
searched for all .json files in C:\Users\MySelf . It found a total of
1767 files. I sorted results by directory and searched for VS Code
related files. There were many Adobe (I use InDesign), Microsoft
Edge, Office, Autodesk (I use Fusion 360), and more. And, I did find
a [Link] file in
C:\Users\agir\AppData\Roaming\arduino-ide
which contained a definition for ${workspaceFolder} to be my target VS
Code working directory. It seems that this .json file is Arduino extension
related, not an Arduino IDE related.
Having removed the ${workspaceFolder} definition from the file, I
reinstalled VS Code. No luck, it still opened with my VScode folder. At
this point I decided to give up. I have no idea as to where VS Code is
getting its folder info.
The VS Code installation was otherwise simple, except for the
general appearance, which I was not able to customize via the VS
Code Settings (click on the gear icon (Manage) in the bottom of the
left-hand toolbar).

3.
VS Code navigation
Having installed VS Code and loaded a file, start experimenting.
Open the different menu items and try them; slowly but surely, you
will become familiar with VS code. Since access to many VS Code
features are keyboard shortcut based, open the shortcuts window
(ctrl-P - go to the bottom of the text box). You can also do a ctrl-K
ctrl-R for a printable list of commands, ctrl-K ctrl-S for an interactive
alphabetical list.
If you are to use VS Code, there are two commands you should
engrave in your mind: ctrl-P (open file in current folder) and ctrl+shft-
P, open a text box full of self-explanatory commands - VS Code and
extension related ones. The many commands available will get you
to realize that VS Code is much more then a programmer's editor.
The next order of business is improving VS Code's appearance as
off the shelf, it is not ideal (next section).

4.
VS Code configuration
Appearance is important. Window colors, font
<x>VS Code[AUX]User interface appearance</x>

sizes, and toolbar positioning and contents should be such that you
feel comfortable with them.
Having downloaded VS Code, I was confronted with a black screen,
large fonts in white, this seems to be the default VS Code look.
Furthermore, when editing a file, the font size was much too large. I
much prefer the AtmelStudio/Visual Studio default looks.
The first item of business is the VS Code general appearance; it is
defined by a theme; there are pre-set choices available via the
Manage-->Themes button. I chose Light (Visual Studio). I prefer dark
text on a light background as opposed to the default light text on a
dark background.
Then there is the font size issue as the default visual interface font
sizes are much too large. You can change font sizes by choosing
Settings and entering Font Size in the top horizontal search text bar.
VS Code will narrow down choices so that you can concentrate on
font sizes only. You can change them from here; but, you must do so
in three locations: User, Workspace, and Folder; this is what I did
(VS Code - folder vs. project (page 91) and VS Code features page
87). The problem was that doing so did change the font size of the
editor, but the VS code user interface font size remained the same.
After dedicating more than an hour trying to customize the VS Code
environment, I went to ChatGPT for help: I asked it how to; it
responded customize the [Link] file. But this is not possible
from within VS Code since it considers the file to be read-only (it is
read-write if you use notepad or other). I told ChatGPT that I could
not modify the [Link] file since it was read-only, it responded
with:
 If you are unable to edit the [Link] file because it is read-
only, you may need to take additional steps to modify the file.
Here's what you can do:
Open VS Code and go to the "File" menu.
Select "Open Folder" and navigate to the folder where your project
is located.
Right-click on the folder and select "Open in Terminal" (or "Open in
Command Prompt" on Windows).
In the terminal, enter the command code --user-data-dir=".". This
will launch VS Code with a new user data directory that is specific
to your project.
Go to the "File" menu and select "Preferences" > "Settings". This
will open the AND HERE IT STOPPED.
I tried regenerating the ChatGPT response since it did not finish, no
luck. It kept trying to get me to change the [Link] file, which is
read-only from inside VS Code. Sooo, I tried the ChatGPT
suggestion above, it worked. The problem was that the change is not
permanent. Upon closing and reopening VS Code via its desktop
icon, I was back in the old configuration: font size 14, not 10.
ChatGPT was not as clever as it could have been. In fact, the
command
code --user-data-dir=".\.vscode"
is really saying: launch VS Code (its .exe name is simply code) from
a DOS box ([Link]) with the user-data-dir option. So, instead of
opening VS Code and going through the convoluted process
suggested by ChatGPT, I created a batch file with the suggested
command above and placed it inside the VS Code project directory.
It says: launch VS Code by using data from the .vscode directory
starting from here (".\" means here). From hence on, do not start VS
code from the Windows task bar icon; start by clicking on the batch
file you created.
Doing this has one important draw back. The command code --user-
data-dir creates many directories and files inside the .vscode
subdirectory, a whopping 129MB spread across 85 folders, 441 files.
Disk space is not a problem, as I have plenty of that. The problem is
that I do regular backups onto a USB key, these 129MB get backed
up, absolutely unacceptable - takes too long and consumes too
much precious USB key memory. Surely there must be a way of
avoiding this.
Sooo, I searched for equivalent files in my C:\users directory. Sure
enough, there was C:\User\Myself\.vscode. I modified my batch file
to
code --user-data-dir="C:\Users\Myself\.vscode"
I removed the 129MB from my project directory and launched the
batch file, still located in my project directory; it worked. The 129MB
were created in C:\Users\agir\.vscode, no problem since I do not
backup anything from my C: drive.
I reinstalled VS Code to get a clean start (you should first uninstall
extensions) to see what gets defined where. My project directory's
.vscode subdirectory remained empty - no .json files; they are now
spread across several C:\Users\ subdirectories.
I did find a [Link] file in
C:\Users\Myself\AppData\Roaming\Code\User. It contained the
Visual Studio theme I had selected but no font size info. I was still in
a dilemma as VS Code opened from the desktop icon had large
fonts; VS Code opened from the batch file had small fonts. They
both used the same Visual Studio (light) theme. Upon examining the
properties of the VS Code launch icon, I found that all it did was
invoke [Link] without parameters. If you were to type code inside
a DOS box, you will find that VS Code gets opened just as if you had
invoked it from the icon. The ideal solution would be that the desktop
icon option should open with small fonts. I tried adding the --use-
data-dir parameter to the icon VS Code launch; I was not successful.
There must a configuration file somewhere which would modify this.
Doing so from the settings menu does not work.
Next step was to do a search on .json files. Looking for all .json files
in C:\users is impractical - there are too many of them (2000+). I
started again limiting my search to C:\Users\Myself\.vscode; I got
342 files, manageable. I looked at [Link] and other .json files,
no luck; none provided font size settings.
The --use-data-dir solution above is a hack, but that was the only
way I found to get a VS Code interface to my liking.
 5.
VS Code - folder vs. project
I found that VS Code Arduino does not
<x>VS Code[AUX]File, folder, workspace used for project management</x>

support the project concept. If you look for open a project or create a
new project, you will be disappointed because none of the menu
items mention Project. Instead, you have to deal with files and
folders. For example, to create the [Link] file I had to do a File--
>New File; this opened an editor text box. I created it, navigated to
the VScode project folder by doing an Open Folder from the Files
menu, and saved it there.
It is important to remember that VS Code does not know projects, it
only knows folders. If you decide to work with PlatformIO (page 98),
you will find that it does refer to projects. If you look for a project as
PlatformIO proposes, you will get frustrated. Doing an open project
in AtmelStudio or in Visual Studio is straightforward; In PlatformIO,
open project requires that you navigate from within the VS Code
folder box to the target directory and select it. This is most counter
intuitive, but it may be the price to pay for an as highly customizable
tool as VS Code.

6.
VS Code extensions
As indicated in the introductory
<x>VS Code[AUX]Extensions can be installed/uninstalled and enabled/disabled</x>

section above, VS Code requires extensions for it to be useful. It is

otherwise a Notepad++ like editor. Upon clicking on the Extensions
icon, bottom gear like icon in the VS Code left-hand toolbar, a pane
opens with the current installed extensions. A horizontal textbox with
the grayed-out Search extensions in the market place serves to
search for extensions by typing whatever it is you are looking for.
From then on, you will get many choices, up to you to install some
and test. Extensions get installed and enabled. You can, at any time,
disable an extension, thereby rendering it inoperant. You may even
uninstall it.
Before continuing, please note that extensions require that they be
configured. You will find that extensions are presented in a standard
way:

When clicking on an installed extension, a window opens

whose top part contains the extension's logo, who created it,
other info, enable/disable, install/uninstall, and a gear icon
(extension settings).
Below that are instructions, tutorials, access to extension
resources, etc. which you should read.

After installing an extension, you should:

RTM (read the manual), i.e., read the info as to what needs
doing to get the extension to work.
Open the extension's settings by clicking on the gear icon as
described above; do whatever it takes to get the extension to
do its work.
Close and reopen VS Code for changes to apply.

Within the context of Arduino C++ programming, I installed the

following extensions:

C++ editing - This is the most basic extension to undertake

C++ work. It enhances VS Code into being an intelligent C++
editor. Please note that if you install the Arduino extension, you
need not install the C++ one, as the Arduino one will do so.
See VS Code C++ extension (page 94).
The Arduino extension installs the C++ extension and the
Serial monitor extension. See VS Code Arduino extension
(page 93), VS Code C++ extension (page 94), and VS Code
serial terminal extension (page 95).
PlatformIO - The PlatformIO extension transforms VS Code
into a powerful embedded development IDE (Arduino,
ESP32...) which rivals with AtmelStudio and Visual Studio.
 Note that it also installs the VS Code C++ extension (page 94).
See VS Code PlatformIO extension (page 95).
Awk - There are many Awk extensions. I installed the Awk IDE
one to see what it could do for me; results were encouraging
but I did not explore further since I have long ago decided to
replace Awk with Perl. See VS Code Awk extensions (page
96).
<x>VS Code[AUX]Edit Awk, Perl programs with Notepad++ or VS Code</x><x>Perl (general)[AUX]Edit Awk, Perl programs with Notepad++ or VS

Perl - There are Oh so many Perl extensions. Which one to

Code</x>

choose got to be a dilemma since my immediate need was a

syntax sensitive Perl editor but I did finally decide on one. See
VS Code Perl extensions (page 96).
Regular expressions (regex) - VS Code supports regexes
through its find/replace feature (traditional Microsoft ctrl-F and
ctrl-H); a specific regex extension is consequently not needed
when editing code. Some extension authors named their
extension regex something, others named them regular
expressions something. Since the VS Code search is dumb, I
did searches on both regex (521 extensions) and regular
expressions (123 extensions); that is a total of 644 extensions
to look at, select, and test according to need. Which one(s) to
install is a daunting task. See VS Code regular expressions
(regex) extensions (page 96).

The marketplace Web site

([Link] is another source for
VS Code extensions. You will find the same extensions as those
accessed from within VS Code but presented in a more user-friendly
way; and navigation is faster .
As a sideline, please note that there is a VS Code extension for just
about all programming languages, Python in particular.

1.
Arduino CLI
Arduino-cli is what its name implies: Arduino development via a
command line interface. It is what any user interface may use to
manage Arduino source code, from compiling it to uploading it into a
microprocessor. It is what the Arduino IDE uses behind the scenes to
achieve what you and I ask it to do as we develop an Arduino
program. Since practically all Arduino work is done via one of many
IDEs, be it Arduino's, or AtmelStudio, there is no a priori need to
spend time on the Arduino CLI. But, the VS Code Arduino extension
requires it. I consequently downloaded and installed it.
If you google Arduino CLI, you will get mostly GitHub responses. The
one to download from is:
[Link]
Once done, extract the zip file to a new location (same location as
the VS Code install location)
C:\Users\agir\AppData\Local\Programs\Arduino-cli
You then need to define a new PATH environment variable so that it
may be accessed from anywhere. Do this by going to Windows
Settings; search for System variables, open the bottom Environment
variables button, select PATH in the user area (top box), do a new,
and add the full file path above.
Check that your arduino-cli command works form anywhere by
opening a DOS box and typing arduino-cli. You should get lots of
output informing you what the different arduino-cli options are.
Having downloaded and successfully installed Arduino-cli, I did a
Arduino-cli help > [Link]
This command redirected output to a notepad compatible text file. It
contains all of the command line options. You can then do a
Arduino-cli help compile > [Link]
to get help on a specific command. At this point I began to realize
just how powerful Arduino CLI can be. I qualify it as being a tool for
very advanced users. I personally have no need for it, but the
Arduino IDE extension does (next section).

2.
VS Code Arduino extension
Microsoft proposes an Arduino extension to enable VS Code C++
work on Arduino boards. This extension could be an alternative to
the PlatformIO extension (see next section).
Upon opening the VS Code Arduino extension presentation page,
the first thing it says is:
Either the legacy Arduino IDE or Arduino CLI are required. The
recommended approach is to use the version of Arduino CLI that
comes bundled with the extension, which works out-of-the-box.
Support for the legacy Arduino IDE will be removed in a future
version of the extension.
After installing the VS Code Arduino extension, I used the default
configuration which is based on the Arduino CLI (command line
interface). This got me nowhere as it seems that the Arduino CLI
interface did not get installed as verified by doing arduino-cli from a
DOS box (command not found message); despite the words that
comes bundled with the extension. I consequently tried using the VS
Code Arduino extension without Arduino CLI,; i.e., by using the
Arduino V2 installation instead. I shall spare you the details; note
that I spent a great deal of time and got nowhere. Note also that
installing the VS Code Arduino extension also installs the VS Code
C++ extension (page 94) and the VS Code serial terminal extension
(page 95)
I then installed the Arduino CLI (see preceding chapter). Before
doing anything complex such as compiling, I opened the [Link]
file to check out the C++ editor. Sure enough, the editor was doing
its work: syntax coloring, name completion, error detection,
refactoring, and lots more. I had a highly intelligent C++ editor. But
there was one major flaw which indicated that more configuration
work was needed. The editor could not find the Arduino.h file, the
corresponding #include was tagged as an error; it blocked all
subsequent error detection.
At this stage, the question is: Where does one define the Arduino
distribution path?
I decided to forgo where to define the include path and try using the
VS Code Arduino extension. Doing a ctrl-shft-P brings up a list of
commands, one of which is Arduino-CLI:Verify. This command
compiles the files in the active folder, in my case only one file
[Link] in the VScode folder. To my surprise, it compiled the .ino
file successfully and did find the Arduino.h file. It seems that the not
found Arduino.h file reported above when editing the file is a VS
Code C++ extension configuration issue. I tried to fix this but was
beset by a very long list of settings, two of which, cpp > Default:
Include path or cpp > Default: Compiler path which I set as
C:/Program files (x86)/Arduino. Doing this solved the problem, The
VS Code C++ extension did recognize the Arduino.h file include.
Now that I had a quasi IDE which contained an intelligent C++ editor
and compiled/linked an .ino file successfully, I needed to upload my
program into the board.
The bottom of the VS Code workspace displays a tool bar which
contains ISP, Board, and Comport settings. I chose ArduinoISP. I did
a ctrl-shft-P and chose ArduinoCLI:Upload. Success, it uploaded my
program into the board.
I went up to the top menu bar, chose Terminal-->New and from there
was able to launch my program - Success.
One quirk remained though: Upon doing a ArduinoCLI:Verify, I got
the message
[Warning] Output path is not specified. Unable to reuse previously
compiled files. Build will be slower. See README.
This message is in effect saying that make is not doing its work;
everything gets recompiled. As for the README, I could not find it.
At this point I decided to stop work on the VS Code Arduino
extension as extensive work to iron out misdoings remained. The VS
Code Arduino extension exercise demonstrated how powerful VS
Code can be; however, my experience with it is such that it does not
warrant using it in lieu of the Arduino IDE, Atmel Studio, Visual
Studio, or possibly PlatformIO.
I did learn one important lesson though: just about all work done by
the extension was done via ctrl-shft-P items. If you try doing Run
without debugging from the top toolbar, it will let you do so; but you
will run into a series of warnings/errors which will get you nowhere. I
did this; it was very time consuming; I shall spare you the details.

3.
 VS Code C++ extension
The Microsoft VS Code C++ extension transforms VS
<x>VS Code[AUX]C++ extension</x>

Code into being an intelligent C++ programmer's editor. It will do

name completion, signal undeclared objects, and detect bad
constructs such as a horribly wrong for statement. If you type
for i = 0 i < 20; i++ ) a++}
it will detect missing parentheses, missing semicolons, undefined
variables, missing curly braces. All these get signaled via red
squiggly underlines. Upon hovering on one of the errors, a small
popup opens suggesting how to fix the problem. VS Code for C++
can do a lot more, just click on the right mouse button, an extensive
popup opens. Oh, I should have mentioned that its find/replace
supports regular expressions (regex).
Upon opening my [Link] file, the VS Code C++ extension
triggered a file not found on the Arduino.h file include. This blocked
all error detection. I could, of course, comment it out, but then it
would trigger undefined variables on Serial and on other system
variables. The solution was to set the C++ extension setting cpp >
Default: Compiler path to C:/Program files (x86)/Arduino. The not
found file issue solved, the C++ extension turned out to be an
intelligent C++ editor.

4.
VS Code serial terminal extension
This is what the VS Code serial terminal extension presentation
window says:
The Serial Monitor extension provides a serial monitor to view
output from as well as send messages to serial ports. This is often
useful when testing or debugging programs on embedded devices.
It combines the bottom pane of the Arduino IDE with the IDEs serial
monitor. It has five tabs:

Output is where [Link] get directed to.

 Terminal - This is a DOS box. You can type any command, no
need to open a DOS box.
Serial terminal - This tab displays compiler messages and
other messages. It is the equivalent of the bottom pane of the
Arduino IDE.
Debug console - It is activated when debugging code. I have
not tried it.
Problems - I have not found any use for this pane.

All in all, the VS Code serial terminal extension brings to the VS

Code Arduino extension that which it needs to view what the tools
and the application are doing.

5.
VS Code PlatformIO extension
The VS Code PlatformIO extension transforms VS Code into a C++
IDE for embedded development (Arduino, ESP32...). See PlatformIO
(page 98).
After testing the VS Code Arduino extension, I removed it, removed
the C++ and serial monitor extensions, and installed the VS Code
PlatformIO extension into a bare VS Code (no extensions installed).
There is no need to go to the PlatformIO Web site to do this.
The VS Code PlatformIO extension installation was quick, a couple
of minutes at most. The installation process also installed the VS
Code C++ extension, but IT DID NOT install the serial terminal
extension, it has its own. It turns out that a setting somewhere was
left over after uninstalling the Arduino/C++/serial terminal extensions
because after installing the PlaformIO extensions, the newly
reinstalled C++ extension had its compiler path remained set to the
Arduino directory.
Risking being repetitive, if you are to use PlatformIO for VS Code, it
is vital that you should understand that you are working in a dual
environment: VS Code proper and the PlatformIO extension. One of
the problems working with VS Code and its extensions is
understanding who does what. This can lead to some confusion. I
recommend exploring VS Code with all extensions disabled (they
can be disabled/enabled after being downloaded). Having gotten a
good working knowledge of VS Code (page 86), Install the
PlatformIO extension and explore what it can do for you.
Unlike the Arduino extension, in which ctrl-shft-P grants access to
tasks, PlatformIO creates a shortcut in the left-hand toolbar. You will
find a new icon, the PlatformIO bug. Clicking on it opens a second
left-hand toolbar of tasks. These are all PlatformIO specific. You will
find a complete chapter dedicated to PlatformIO (page 98) which
covers using the VS Code PlatformIO extension.

6.
VS Code Awk extensions
<x>VS Most Awk extensions refer to an Awk language
Code[AUX]Awk extensions</x>

server which I know absolutely nothing about. There were Awk

syntax highlighting extensions; they could be somewhat useful. I
decided to forego installing any Awk extensions since I had long ago
decided to use Perl instead of Awk .

7.
VS Code Perl extensions
I counted 89 Perl extensions; which one(s) should I
<x>VS Code[AUX]Perl extensions</x>

install? To do what? I finally installed Perl Navigator by bscan. It

does syntax checking and more, has excellent reviews, and is recent
(May 2023). I loaded my Perl documentation from C++ source files
extraction program. It pointed out a few potential problems which
merit reviewing, something I shall do once my two C++ for Arduino
books get finished and published.

8.
 VS Code regular expressions (regex)
extensions
I installed two extensions to create a list of
<x>VS Code[AUX]Regular expressions extensions</x>

functions. I need this list within the context of the Class and Function
Names Referencing Framework (page 176). The objective is to
create a file which contains the functions defined in .h files, figuring
that there would not be any functions implemented in .cpp not
declared in .h files. The [Link] Perl program would use the
content of this functions file to search for improper use of function
names.

7.
.json files
I
<x>VS Code .json files[AUX]Data files organized as text key-value pairs</x><x>VS Code .json files[AUX]Value is string, number, Boolean, array, other json object</x>

tried VS Code as stand alone, with the C++ extension, and with the
Arduino extension. In the process of doing so, I discovered a .vscode
subfolder in my ArduinoDev main projects directory which contained
four .json files:
<x>Arduino (general)[AUX]IDEs: Arduino, AtmelStudio, Visual Studio, VS Code, PlatformIO, Visual Micro</x>But, before continuing, a

word on .json files.

.json files are text files in which data is organized as key-value pairs
enclosed as groups inside curly braces, each group can have a
name; e.g.
Contact = {
"firstName" = "Sasha",
"lastName" = "Orgalov",
...
Address" = {
"street" = "Main st.",
...
}
}
In the above, the group named Contact encloses its data inside curly
braces. It contains a group named Address. To access an item from a
.json aware program, you can do [Link] or [Link].
Notice that I use uppercase first letter for group names, lowercase
first letter for group items; this choice is based on my personal
naming conventions.
And now, here are the four .json files in the .vscode subfolder in my
ArduinoDev projects directory.

[Link] was probably created by the Arduino extension. It

contains board, comm port, and other configuration info.
[Link] was also probably created by the Arduino extension.
It contains a compiler path, not the same one as in the
[Link] file.
c_cpp_properties.json was created after doing a successful
ArduinoCLI: Upload from the Arduino extension. It contains the
compiler path, the include path, and other build specific info.
[Link] contains a link to a Web page which details how to
define VS Code launch configurations.
[Link]
-configurations

I decided to forgo examining the myriad possibilities (once more

steep learning curve).

8.
VS Code caveats
VS Code did manifest some annoying issues.

Unable to remove folder from

<x>VS Code caveats[AUX]Unable to remove folder from active folders list</x>

active folders list - I do all of my Arduino work in a folder

named ArduinoDev. At some point during my experimenting, I
opened this folder via the File-->Open folder menu item.
Having done this, I found no straightforward way of removing it
from my folders list. But there is a way. Select the folder then
do a menu File-->Close folder; this works. There is a shortcut
(ctrl-K F); I tried it, no luck.
<x>VS Code caveats[AUX]VS Code does not support the concept project content/disk content</x>Remove folder deletes it

from the disk - VS Code does not support the concept project
content/disk content such as AtmelStudio does. A right click on
the folder's name does offer delete; however, this is not good
as it deletes it from the disk. To remove the folder from the VS
Code workspace, you should invoke the menu item File--
>Close folder as indicated above.
<x>VS Code caveats[AUX]Arduino-cli failed to get installed</x> VS Code Arduino extension (page

93) - Arduino-cli failed to get installed despite the extension's

saying that Arduino-cli is embedded.
 12.
PlatformIO

PlatformIO
When I first learned about PlatformIO's existence, I went to the
[Link] Web site for info as to <x>PlatformIO (general)[AUX]IDEs: Arduino, AtmelStudio,
Visual Studio, VS Code, PlatformIO, Visual Micro</x><x>Which IDE to work with?[AUX]PlatformIO - free, professional grade, easy import of Arduino projects</x><x>PlatformIO

(general)[AUX]IDE supports cross-platform, cross-architecture, multiple frameworks</x><x>PlatformIO (general)[AUX]Plugin for Microsoft VS Code and CLion</x>

<x>PlatformIO (general)[AUX]Free, professional grade, easy import of Arduino projects</x><x>ESP32[AUX]Supported by Arduino IDE and PlatformIO, not by

what it was. Perhaps the best approach at defining it is to cite

AtmelStudio</x>

what the PlatformIO Web site says about it:

PlatformIO’s unique philosophy in the embedded market provides
developers with a modern integrated development environment
(Cloud & Desktop IDEs) that works cross-platform, supports many
different software development kits (SDKs) or Frameworks, and
includes sophisticated debugging (Debugging), unit testing (Unit
Testing), automated code analysis (Static Code Analysis), and
remote management (Remote Development). It is architected to
maximize flexibility and choice by developers, who can use either
graphical or command line editors (PlatformIO Core (CLI)), or both.
I infer from the above that it is a tool designed for businesses
engaged in professional embedded development.
If you have not read the chapter dedicated to VS Code (page 86),
you will be misled by what you see in the PlatformIO Web site. To
make a long story short, PlatformIO is presented as an IDE, an
entity, VS Code being detail (C++ code editing). If you follow along
this path by downloading/installing PlatformIO, frustration sets in
because you will not realize that VS Code is considerably more than
a C++ editor. Upon hitting a snag, you will naturally go to the
PlatformIO Web site for help. Since the snag was a VS Code related
issue, you will not get far. It will take you a while to realize that you
are dealing with two different tools. I insist: Get familiar with VS Code
before installing PlatformIO.
If you proceed with PlatformIO and do not check out VS Code, you
will find the going progressively more confusing. This is exactly what
I did because I did not find, from looking at the PlatformIO website,
how important understanding the interrelationship between
PlatformIO and VS Code was. After installing PlatformIO, part of
what you are looking at is VS Code, part is PlatformIO. It is important
to differentiate one from the other; you should first work on VS Code
without the PlatformIO extension to get to know what is specific to
VS Code; otherwise, you will waste a lot of time trying to figure out
what you're doing wrong. So, read on and discover the mistakes I
made so that you should avoid them.
PlatformIO is unique when compared to other IDEs because it is a
plugin for VS Code, CLion (Jet Brains), and other foundation tools.
Microsoft VS Code is free. CLion costs $90/year the first year
(individual user), less the following years. Judging from the short
presentation of CLion in the JetBrains Web site, it appears to be well
worth the money. It seems to be a formidable tool for C++
development. I have not tested it with PlatformIO and cannot report
on its merits compared with PlatformIO over Microsoft VS Code.
With this in mind, I am limiting myself to PlatformIO/VS Code. The
two, PlatformIO/VS Code, combined into one entity, become a
features rich IDE which rivals with AtmelStudio and Visual Studio.
PlatformIO brings to VS Code all that VS Code does not have as far
as Arduino development is concerned: build, link, upload, run and
hardware-based debug. Note that PlatformIO is not at all Atmel
specific the way the Arduino IDE, AtmelStudio, and Visual
Studio/Visual Micro are. It can target other microcontrollers. This
feature alone merits investing time and effort getting to know it if you
intend to develop for other microcontrollers. Note that if you do
intend to use PlatformIO for Arduino development, you nevertheless
need to download the Arduino distribution because it needs the
Arduino libraries.
After installing PlatformIO, it did take me a while to understand what
I was looking at, how to create and how to import an existing Arduino
project. I must confess that my first encounter with it was
disconcerting. I first expected another IDE. I expected a tool
centered around a workspace containing files to be worked on, and a
toolbar with buttons and menus above. PlatformIO is entirely
different; it requires getting used to. As I explored PlatformIO, I was
led to a first conclusion: it is powerful, however, it requires a
significant learning curve.
Upon opening PlatformIO the first time, it does so with its home
page. The question I asked myself was: What am I looking at? I was
faced with an intimidating set of toolbars and windows. There are
two distinct toolbars above, one for VS Code and one for PlatformIO.
You will find two toolbars on the left edge, sometimes three and even
four, and one status bar at the bottom. That is a total of 6 toolbars.
Add to this that some panes will contain toolbars. That is a lot of
content to get familiar with.
PlatformIO's home page features a big orange bug emoji staring you
down, as if to say: enter if you dare. It contains buttons on the right,
one of which is Import Arduino Project, and just underneath there is
a window full of news you can scroll through horizontally, several
Web news, and access to the PlatformIO Web site. Below that, out of
sight but which you can bring up, is a list of PlatformIO projects.
Enough said, you understand that it is a crowded busy screen,
implying a significant learning curve.
After a lot of searching around, I was able to run an application and
visualize output in the built-in serial monitor. It does some syntax
checking the extent of which I yet need to determine.
At this time, I was not aware that the PlatformIO installation
automatically installed the VS Code C++ extension (see VS Code
extensions page 91). When you edit C++ code inside PlatformIO, it
is the C++ extension which is doing the work, not PlatformIO.
PlatformIO uses the same toolchain as the Arduino IDE and
AtmelStudio (GNU compiler, linker, avrdude), and supports
hardware-based debugging, more on this later as it is burdened by
constraints.
I ran PlatformIO through its paces with two objectives in mind: (1) to
learn enough about it to describe the experience and (2) possibly
adopt it as my main Arduino development tool in lieu of AtmelStudio
or Visual Studio (Visual Micro) (page 82). The outcome is that I have
mixed feelings. PlatformIO does just about everything Visual Studio
and AtmelStudio can do which I value most: code completion, code
checking as you type, quick access features. However, its interface
is just too busy, I much prefer AtmelStudio's; and I find its learning
curve excessive given my needs.
You will find in the pages which follow more details on PlatformIO,
organized as follows:

PlatformIO Installation (page 100) - If you know nothing about

PlatformIO and discover that it could be a well worth
investigating alternative to Arduino development, you will go to
the PlatformIO Web site ([Link] and proceed to
download it. You should not do this because you will fail to fully
realize how important it is to recognize that PlatformIO is a
plugin over a foundation tool (VS Code or CLion).
PlatformIO features (page 101) - This section summarizes
what PlatformIO can do for you.
PlatformIO managing projects (page 102) - It is vital to
understand that a PlatformIO project is a VS Code folder.
PlatformIO creating projects (page 103) is simple. If you
choose an Arduino board, it creates an Arduino skeleton
project and uses the Arduino distribution upon doing the build.
The PlatformIO project configuration ([Link]) (page 106)
is defined in the [Link] file. This is a data file which lies
at the root of an PlatformIO project.
PlatformIO interoperability (page 107) with the Arduino IDE
and AtmelStudio is possible and simple although there are
restrictions.
PlatformIO hardware-based debugging (page 108) -
PlatformIO begins to support hardware-based debugging of
Atmel microcontrollers (Arduino ٨-bit boards). At time of writing
(May ٢٠٢٣), it requires specific hardware, namely an Arduino
Nano ٣٣ or Arduino MKR series boards and a debug adapter.
Neither Serial debugging (page ١٢٥) nor debugging with the
Microchip Xplained boards (see Hardware-based debugging
page ١١٩) are supported.
PlatformIO Documentation is good, easily accessible, plenty of
tutorials exists. See Bibliography - PlatformIO (page 336).
 PlatformIO gotchas and caveats (page 109) - This section
describes problems I encountered; refer to them as gotchas,
caveats, traps and pitfalls, whatever.

All in all, PlatformIO is a powerful tool which merits serious

consideration as an alternative to AtmelStudio and Visual Studio, the
more so since Platform IO is beginning to provide hardware-based
debugging for AVR Arduino boards.
I am not, by a long shot, an expert on using PlatformIO. These
pages condense what I learned concerning it. Despite its steep
learning curve, it could possibly be my choice for future
developments, particularly if I were to develop applications for
microcontrollers other than Arduino and ESP32.

1.
PlatformIO Installation
My advice is that you should forget
<x>PlatformIO (general)[AUX]Installation - do it progressively, VS Code first</x>

the PlatformIO download instructions, and proceed in three-stages:

Download VS Code and become familiar with it. A lot of what

you will be doing with PlatformIO is VS Code work. You will, at
this point, realize that VS Code may be enhanced with many
extensions, one of which is PlatformIO.
From within VS Code, install the VS Code C++ extension
(page 94) and work with it. PlatformIO would have installed it
anyway.
From within VS Code, install the PlatformIO extension. At this
stage, you have reached what the PlatformIO Web site
instructions would have led you into doing.

Installing PlatformIO as per the above will help you understand what
really PlatformIO is; i.e., distinguish between PlatformIO, VS Code,
and the VS Code C++ extension.
 2.
PlatformIO features
<x>PlatformIO (general)[AUX]Offers features beyond those offered by other IDEs</x><x>PlatformIO (general)[AUX]Plugin for Microsoft VS Code and

PlatformIO, being an embedded systems development tool for

CLion</x>

professionals and business, teems with features. Describing them all

would fill pages. From a nonprofessional Arduino developer
standpoint, its main features may be summarized as follows:

Powerful Arduino development tool - By powerful I mean that it

does a lot which the Arduino IDE does not do. Using
PlatformIO in lieu of the Arduino IDE will increase your
productivity and render the application more robust. But there
is a cost, its learning curve is fairly steep.
Multi-platform, multi-microcontrollers - If you want to develop
applications for Micro controllers other than the Arduino ones
and ESP32, by all means, consider using PlatformIO.
<x>PlatformIO (general)[AUX]Compatible with the Arduino project structure</x>PlatformIO supports the

Arduino project structure - The simplest definition of an

Arduino project I could come up with is that it has a main entry
C++ file, whose extension is .ino, not .cpp, which contains two
functions (setup and a loop). Remember that all C/C++
applications start with a main function. The Arduino environment
provides this function in the background upon doing a build;
main calls setup and loop, which are supplied by the programmer.

PlatformIO adheres to the Arduino project structure by creating a

[Link] file which contains empty setup and loop functions; it is up to
the programmer to transfer the contents of the Arduino .ino file into
[Link]. Note that PlatformIO ignores the .ino file. Also note that
PlatformIO source files are placed in a subdirectory defined by the
[platformio] src_dir variable. The default location being a subdirectory
named src, you can change this (see PlatformIO creating projects
page 103).
PlatformIO as plugin to VS Code - PlatformIO is not a
standalone program such as the Arduino IDE and other IDEs
are. It is a plugin on top of VS Code (the one I will cover), on
top of CLion, and others. Understanding who does what is
important to remain efficient using PlatformIO.
<x>PlatformIO (general)[AUX]Supports hardware-based debugging wide range microcontrollers</x>Hardware-based

debugging - PlatformIO proposes hardware-based debugging

for ESP32 and for Arduino boards (see PlatformIO hardware-
based debugging page 108).
<x>PlatformIO (general)[AUX]Provides facilities for lint like inspect utility</x>Lint like inspect utility - This

feature can be priceless. Static analysis detects suspicious

assignments, possibly bad pointers, uninitialized variables,
unused variables and functions, and other potential flaws. I
tested it on my programs. Sure enough, it revealed a number
of items which required a bit of house cleaning. But it did not
detect one of my favorite recurrent flaws, such as an = sign
instead of the comparison == sign in an if statement. That is
why I created a Perl program to detect these and other hard to
find flaws (see the Misdoings page 117). I should add that
PlatformIO's lint inspection can be improved upon by plugging
in external lint utilities.
<x>PlatformIO (general)[AUX]Supports multiple libraries packs</x>Project libraries - Locating all

libraries in a central Arduino library directory is one of the

Arduino team's simplifications which have made Arduino so
successful for small applications. As applications get bigger,
they may be called upon to run on several platforms, possibly
in different environments, thereby requiring variants of the
same library. PlatformIO fully supports a project specific library
configuration; the Arduino IDE does not. As for AtmelStudio
and Visual Studio, it might be possible to achieve this; I cannot
say.
<x>PlatformIO (general)[AUX]Supports version control, configuration management, continuous integration</x>Configuration

management - A multi-programmer professional context

requires that source files be checked out and checked in so
that no two programmers can make changes to the same file
 unaware of one another's doings. One refers to this as
configuration management. PlatformIO refers to this as
Continuous Integration. Any of a several tools can be
integrated into PlatformIO to achieve these tasks.

3.
PlatformIO managing projects
<x>PlatformIO (project management)[AUX]PlatformIO projects are VS Code folders</x><x>PlatformIO (project management)[AUX]Open/Close project is Open/Close VS Code

Most IDE project management (creating, closing, opening

folder</x>

projects) is straight forward. The Arduino IDE has menu items for
these: New sketch, Open recent, Close. AtmelStudio has New,
Recent projects and solutions, Close. Not so PlatformIO because of
its dual nature. PlatformIO is a layer over VS Code. PlatformIO
refers to projects; VS Code refers to folders. This is where confusion
may lie.
During your initial work with PlatformIO, you created a project;
suppose that it was the Simple project as described in PlatformIO
creating projects below. Having created the project, you want to pass
on to more substantial work and wish to remove the Simple project
from your PlatformIO workspace. You can visualize opened projects
by clicking on the top icon of the left-hand toolbar, this is the folders
Explorer. You found your Simple project. If you right-click on the
Simple line, you get a popup which contains a few items but fails to
offer the possibility of removing the project from your workspace.
This is where the aha! I have to resort to VS Code comes in. To
remove a project from your PlatformIO workspace, click on File (top
bar) and choose Close folder (this is VS Code). To open a project,
click on the Explorer (top icon in the left-hand toolbar) and click on
Open folder (this is PlatformIO).
So far things are simple and not subject to making bad mistakes. But
there is a gotcha. If you select a file from the second left-hand menu
bar, it will open it in the editor. If, after selecting it, you right click, a
text pane opens with a delete option towards the bottom, and here is
the gotcha: clicking on delete will not just simply remove it from the
workspace as AtmelStudio would, it physically removes it from your
storage area. It is a real Windows delete. See PlatformIO gotchas
and caveats (page 109).

4.
PlatformIO creating projects
<x>PlatformIO ([Link] file - project config.)[AUX]PlatformIO project is folder which contains [Link] file</x><x>PlatformIO ([Link] file - project config.)

[AUX]Operate in single- or two-level directory structure, code in subdirectory pointed to by src_dir</x><x>PlatformIO (project management)[AUX]Creates [Link] with empty

Creating projects is easy.

setup and loop</x><x>PlatformIO (general)[AUX]Creates [Link] with empty setup and loop</x>

Open the PlatformIO home page by clicking on the Platform IO bug

in the left-hand toolbar. Go to Quick access --> PIO Home --> Open;
this opens a window which offers the possibility of creating a new
project - do this; do not import an existing Arduino project as
PlatformIO will put it somewhere in C:\Users; it will not put it where
you want it.
Here are some facts to consider when creating a new project with
PlatformIO:

A Project folder's name can be anything. PlatformIO does not

care. There is no constraint such as is the case with the
Arduino IDE whereby the project's folder must have the same
name as the sketch's name without the .ino file extension.
Project properties are defined in the [Link] file. The
project's name is the name of the folder the [Link] file is
located in.
Source code may be located anywhere. The only constraint is
that it be located in a folder relative to the project's folder
defined by src_dir in the PlatformIO project configuration
([Link]) (page 106) file.

You can create a PlatformIO Arduino project in one of two ways,

each by clicking on the PlatformIO bug in the left-hand toolbar. This
opens PlatformIO's main tool bar next to the left-hand toolbar. In the
bottom half you will find PIO Home-->Open. Click on Open; it brings
up the PlatformIO homepage. You will find in the top right-hand side
of the homepage several textboxes under Quick Access proposing
two ways of creating a project: New project and Import Arduino
project.
I recommend the first option, New project, as it will let you choose
the directory into which you will create the project. The second
option, Import an Arduino project, does not grant you this option; it
puts the project in C:\Users\Somewhere; you have no choice. I may
be wrong but after an hour or so trying to specify where to import the
project to, I failed. Whichever you choose, the fundamental project
structure will be the same. A [Link] file which contains empty setup
and loop functions in an src subdirectory is created, the .ino file is
ignored when doing a build.
Having created an Arduino project, the good news is that you do not
have to create external tools such as you need to do with
AtmelStudio. The not so good news is that you need to set all your
configuration parameters inside the project specific [Link] file.
When you decide to create a PlatformIO Arduino project, the
directory structure will depend upon how you want to work. Do you
want interoperability? If you do want interoperability, it all depends on
the IDEs you want to interoperate with. See Interoperability (page
21).
Here is how to create an Arduino project by clicking on New project:

Click on the +New Project in the PlatformIO home page, the

one with the big ant-like emoji.
Define the project's name, use the Arduino project's name.
Choose the board, one of 2000+ possible choices at time of
writing. I chose the Arduino ATmega2560.
Unclick the Use default location check button; this extends the
textbox downwards with the following
Favorites - VS Code defined in
C:\Users\Agir\platformio\[Link].
Places - My username Agir (C:\Users\Agir) and Projects
directory (C:\User\Agir\Documents\PlatformIO\Projects)
Disk drives (letters) contains C: and D:
 Disk drives (names) contains the names of the disk drives, in
my case: OS and Data (Windows, my computer).

You can use any of the above to navigate to where you want to
locate your project. The Favorites folder is defined in
C:\Users\Agir\.platformio\[Link].
The default location pointed to a C:\Users location; I had to
manually modify the .json file to get the favorites menu item to point
to the VScode folder in my D: drive.

Click on Finish.

Suppose that you selected an Arduino IDE project located in a folder

named Simple which contains [Link]. You now need to
undertake a few minor changes to render your PlatformIO project
workable.

[Link] file - PlatformIO will have created a [Link]

file (project configuration) in the PlatformIO project root
directory, the one you selected. You will find details on
PlatformIO's project configuration in the next section.
Project source code location - PlatformIO creates a source
files subfolder inside the PlatformIO project root directory
named src. You need to add a new section and variable to the
PlatformIO project configuration ([Link]) (page 106) file
to tell it where the source files are to be placed. In my case, I
chose [platformio] src_dir = . to tell PlatformIO that the source
files should be in its project root directory (single-directory
structure).
PlatformIO Arduino project entry file - PlatformIO creates a
[Link] file which contains empty setup and loop functions. The
contents of the .ino file must be copied into [Link] or you
may opt to transfer the contents of the .ino file into a function
named Aardvark to enhance interoperability and render changes
less error prone. See Interoperability (page 21).
Compiler flags - Arduino libraries are built using the -fpermissive
 option to prevent compilation warnings from blocking the build
process. It seems that this flag is brought in by default. I added
it to the [env] section. Here is a typical error when the -
fpermissive flag is missing:
type char* className = "this is the Employee class"; C++
forbids...

The -fpermissive flag transforms the above from being an error, which
stops the build, to being a warning, which allows the build.
You are all set to go. Do build from the left bar by clicking on the
Build button; the application should compile error free, assuming it
compiled error free with the Arduino IDE. You can upload the
application into the microcontroller and then run it from the
PlatformIO terminal (it does have a serial console).
I encountered one first obstacle because of my not recognizing that
the PlatformIO project is in fact a VS Code folder. If you want to work
on a specific project, do not think in terms of opening a project as
you would do in AtmelStudio, instead think in terms of opening a VS
Code folder: do File-->Open folder from the top toolbar (VS Code
toolbar).
Another issue is that there is no such thing as a Project Properties
dialog box as in AtmelStudio. A project's configuration must be
undertaken via configuration parameters defined manually in the
project specific PlatformIO project configuration ([Link] - next
section) file located in the project's directory, much like the Arduino
[Link] file. For example, when creating a project, it asks for the
board. I responded with the ATmega2560; PlatformIO wrote the
following line of code in the [Link] configuration file:
<x>PlatformIO ([Link] file - project config.)[AUX]Define board with board = mega2560 or AtMega328 in [Link]</x>board =

ATmega2560
If you later decide to run the program on an Arduino UNO
(ATmega328P), you need to manually enter the new board into
[Link] by changing the board line to:
board = ATmega328
For more details on the [Link] file, see the PlatformIO project
configuration section ([Link] - next section).
To run the program, you must compile it, upload it, and run it in a
terminal (PlatformIO calls this monitor).

Compile and upload - Just as with the Arduino IDE, you can
compile and upload; both of these actions are undertaken from
the PlatformIO menu attained by clicking on the bug emoji in
the left-hand toolbar.
Run task - Opening the Terminal menu item in the top toolbar
displays several options, one of which is run task. This option
enables to trigger execution of the program.

Uploading was surprisingly successful. I say surprisingly because I

did not have to specify a com port such as is required with the
Arduino IDE and AtmelStudio. PlatformIO recognizes the com port
automatically. I cried victory too soon. I started experimenting
PlatformIO by making changes to source code and upon uploading, I
got the message:
avrdude: ser_open(): can't open device "\\.\COM11": Access is
denied.
This is not a bug, it is a lack of user friendliness explained in the
PlatformIO gotchas and caveats (page 109) section. Having
overcome the access denied problem, I was, from then on, able to
compile, link, upload, and run Arduino projects.
Since I was working with an ATmega2560, I decided to try the
ATmega328P. This required that I specify a new com port in the
[Link] file, thus:
upload_port = COM14
monitor_port = COM14
@#$µ&! PlatformIO did not recognize the new comport setting
defined in the [Link] file. To make a long story short, I ended
up creating a new project with the ATmega328P to solve the
problem. I did spend time searching directories to find where the old
COM11 specification remained, no success. See PlatformIO gotchas
and caveats (page 109).

5.
 PlatformIO project configuration
([Link])
<x>PlatformIO (general)[AUX]Project configuration in [Link] file</x><x>PlatformIO (project management)[AUX]Project configuration in [Link] file</x>

<x>PlatformIO ([Link] file - project config.)[AUX]Project configuration in [Link] file</x><x>PlatformIO ([Link] file - project config.)[AUX][Link] file

The [Link] file is the key to defining a project's

contains sections/variables</x>

properties. It contains what AtmelStudio puts behind the scenes in its

Project Properties window and what the Arduino IDE puts behind the
scenes in its [Link] file. After a while, you will be modifying it
often because it controls just about everything which relates to your
PlatformIO project. Here is an example of such a file:
; PlatformIO Project Configuration File
; Build options: build flags, source filter
; Upload options: custom upload port, speed, and extra flags
; Library options: dependencies, extra library storages
; Advanced options: extra scripting
; Please visit documentation for the other options and examples
; [Link]
[env:mega2560] ;microcontroller
platform = atmelavr ;microcontroller type
board = megaatmega2560
framework = Arduino ;development environment
<x>PlatformIO ([Link] file - project config.)[AUX]Com port defined in upload_port in [Link] file</x>upload_port = COM11

monitor_port = COM11
monitor_speed = 115200
[platformio] ;overrides default PlatformIO configuration
src_dir = MyProject ;source code location relative to project
directory
The [Link] file is a data text file whose syntax is rather simple;
it is very similar to .json files (page 96).

File structure - The file is a text file organized as distinct lines

of information, i.e., one line of text at a time.
Comments - everything after a semi colon is ignored. This is
equivalent to the C++ style // comment.
 Sections - Two sections (platformio and env) define the build
process. Specific data values are defined within one of these
two sections.
platformio section - There is only one platformio section, it
overrides the PlatformIO default configuration parameters.
env section, possibly followed by some specification, enables
specifying the type of build (release vs. debug, mega2560 or
mega328P...)

Please note that

<x>PlatformIO ([Link] file - project config.)[AUX]Source code location defined in src_dir in [Link] file</x>

variables are not user-defined; they are predefined by PlatformIO.

Each one belongs to a specific section (src_dir goes in platformio
section, not the env section). The [Link] file structure is
flexible as sections can inherit from other sections, thereby creating
a hierarchy.

Data is defined as variable/value pairs; an = sign assigns a

value to variable (e.g. upload_port = COM11). As indicated above,
variable/value pairs belong to specific sections. Name
completion will help you avoid doing something wrong. For
example, if you are trying to add src_dir inside the env section, a
popup opens as soon as you type an s; it will contain test_speed,
buil_src_filter,... but you will not find src_dir. This should alert you
that you are doing something wrong. If you do the same inside
the platformio section, as soon as you type an s you will get
shared_dir, and src_dir, you are good.
Environment - the [Link] file should define the type of
microcontroller and framework being targeted.
platform = atmelAVR
framework = Arduino

Theoretically, all of a project's configuration gets defined in the

project's [Link] file as illustrated above. But I did run into a
snag. I decided to change the board, which led me to define a new
board (board =) and new comport (upload_port =) in the
[Link] file. The problem is that upon uploading, PlatformIO
obstinately tried to upload into the old comport; I could not get it to
recognize the new comport. See PlatformIO gotchas and caveats
(page 109).
After working extensively with AtmelStudio, getting used to doing
things via parameters in a file took a little getting used to. It was a
question of knowing where to find answers. The link
htpp://[Link]/page/[Link] will tell you just
about all you need to know to configure the application. Acquiring a
good working knowledge of [Link] files is vital as these define
how a project gets built and uploaded.

6.
PlatformIO interoperability
<x>Interoperability[AUX]PlatformIO creates [Link] with empty setup and loop</x><x>Interoperability[AUX]Seamlessly switch between AtmelStudio, Visual Studio, PlatformIO,

Arduino IDE</x><x>PlatformIO (interoperability)[AUX]Creates [Link] with empty setup and loop</x><x>PlatformIO (interoperability)[AUX]Seamlessly switch between

Achieving interoperability between PlatformIO,

AtmelStudio, PlatformIO, Arduino IDE, Visual Studio</x>

the Arduino IDE, AtmelStudio, Visual Studio, VS Code and Visual

Micro could not be easier. The idea is to organize source files in
such a way that a simple #define in a global include file should enable
you to select which IDE you want to work with.
Having decided which IDEs you want to interoperate with, you must
decide whether you want a single-level or a two-level directory
structure. PlatformIO can be adapted to work with either. Here is a
summary of possible options:

Arduino IDE plus AtmelStudio plus PlatformIO requires a two-

level directory structure.
Arduino IDE plus AtmelStudio(Visual Micro) plus Visual Studio
(Visual Micro) plus PlatformIO requires a single-level directory
structure.

If you opt to work with AtmelStudio without Visual micro, you will be
in a two-level directory structure. You should start by creating an
AtmelStudio project as described in the PlatformIO creating projects
(page 103). Once you have done that, you have achieved
interoperability with AtmelStudio and the Arduino IDE. If you opt to
work with Visual Micro, you would be in a single-level directory
structure. You can use your existing Arduino project directory (two-
level directory structure) or the Arduino project directory (single-level
directory structure).
The next step is to create a new PlatformIO project (see PlatformIO
creating projects (page 103) above). Specify the PlatformIO's project
root directory as being the AtmelStudio project's root directory.
There is PlatformIO peculiarity you should be aware of: PlatformIO
considers that all .h/.cpp files in the source code directory and
corresponding subdirectories belong to the project. It does not
differentiate project files from physical non-project files such as
AtmelStudio does. It will load them and compile them. You may have
to move such subdirectories somewhere else.
Your source code will need a PlatformIO specific macro to
include/exclude PlatformIO code, such as:
#define PLATFORMIO_IDE
That is it. See the Interoperability (page 21) section for more details.

7.
PlatformIO hardware-based
debugging
<x>Debugging hardware-based[AUX]PlatformIO supports Arduino, ESP32 hardware-based debugging, other</x><x>ESP32[AUX]PlatformIO supports Arduino, ESP32 hardware-

based debugging, other</x><x>PlatformIO (hardware-based debugging)[AUX]Supports Arduino hardware-based debugging, ESP32, other</x><x>PlatformIO (hardware-based

Being
debugging)[AUX]Requires special Arduino Nano 33 or MKR boards</x><x>PlatformIO (hardware-based debugging)[AUX]Serial debugging not supported</x>

in the process of finalizing this book (May 2023), I checked the

PlatformIO Web site for Arduino hardware-based debugging
possibilities. The solution proposed is based on using specific
Arduino boards along with a debug adapter. The ATmega328P-Xmini
hardware debugger (see Hardware-based debugging page 119)
does not seem to be supported.
You will find full details on debugging possibilities in the PlatformIO's
Web site ([Link] I
looked at one section in particular: Arduino In-circuit Debugging with
PlatformIO; it contains:

Hardware - PlatformIO hardware-based debugging requires

installing the appropriate hardware (Nano 33 and the MKR
series) and undertaking appropriate connections.
Configuration - Some specific variables must be set in the
[Link] configuration file.

Having set up the environment, you can debug. I have not tried it
since I did not have the required equipment at time of writing.

8.
PlatformIO help and
documentation
[Link]) rich with info</x><x>PlatformIO (general)[AUX][Link] presents interesting tutorial on
<x>PlatformIO (general)[AUX]Web site (

PlatformIO</x>The extensive PlatformIO help and documentation shortens its

learning curve. There are two help and documentation locations you
can access from within PlatformIO:

Help menu item in the top bar - This is VS Code help, not
PlatformIO specific.
A Get Started and Docs selection item in the bottom of the PIO
Home-->Open page - This is PlatformIO specific help. These
contain extensive information as to what PlatformIO is, its
supported boards, platforms, frameworks, how to install it, how
to use it, and it details some features. It also contains help on
configuring a project via [Link] and getting started
tutorials. It will direct you to YouTube PlatformIO introductory
20-minute videos which I found enlightening. There is an
interesting tutorial in the [Link] Web site not listed in
the PlatformIO Web site. See Bibliography - PlatformIO (page
 336).

9.
PlatformIO gotchas and caveats
IMHO, PlatformIO is equivalent with
<x>PlatformIO (gotchas and caveats)[AUX]Has its share of gotchas and caveats</x>

AtmelStudio. But, as with all complex systems, I encountered a few

problems due in part to my failing to uncover how to do something,
or due to an outright PlatformIO problem.

Import Arduino project

<x>PlatformIO (gotchas and caveats)[AUX]Import Arduino project location hard-coded</x>

location hard-coded - Creating a new project and deciding where

you want it installed is simple; uncheck the default location checkbox
and specify where you want it to go. However, doing same on an
Arduino import is not possible; it will place your project in a
directory which it refers to as Places (left in the Import projects
window) - there are two icons:
Home icon labeled MySelf: C:\Users\MySelf
Folder icons labeled Projects:
C:\Users\MySelf\PlatformIO\Documents\Projects

If you do an import Arduino project, it will place it in the C: drive

Documents/projects above. I did not find a way to modify these
default locations. ChatGPT was of no use since it kept directing me
to a create project settings icon which seems to no longer exist. I
tried PlatformIO's help, no success; I gave up. But, since creating a
new project seems to be identical to importing an Arduino project,
they both create a [Link] file in an src subfolder, I went the New
project way.

[Link] file
<x>PlatformIO (gotchas and caveats)[AUX][Link] file variable in wrong section not reported</x>

variable in wrong section - If you do not put the src_dir = .

variable/value pair in the platformio section, you will get the
following misleading error message.
 Error: Nothing to build. Please put your source code files to
the PROJECT\src' folder.

The problem is that the error message is misleading. At some point

during my travails, I discovered that the mistake was not related to
formulating how to tell PlatformIO where to put it, but simply that
the variable/value pair was in the wrong place.

<x>PlatformIO (gotchas and caveats)[AUX]Uploading a program fails: access denied</x> Uploading a program fails:
access denied - As I progressed using PlatformIO, I became
ever more successful doing things. The output in the terminal
window looked the same as the one you get from a build with
the Arduino IDE. Then I tried uploading, I would on occasion
get an Access denied message on the comm port I was using
$#%*@ What is going on? The reason is that when you run a
program, because the loop function never stops, the application
does not exit. This means that the terminal is busy. So, when
you rebuild and try to reload, it will not reload because the
terminal is busy. Logical. But the messages is Access denied
as if it were some mysterious Windows permission problem.
You will find in the right part of the terminal window, bottom
part of the PlatformIO work area, lines of text which start with
PlatformIO: and end with a check mark. The last line of text
probably ends with a 220° arc going in circles - it is telling you
that the terminal is busy. Select it; click on the garbage can;
this stops the program thereby releasing the terminal. Problem
solved.
<x>PlatformIO (gotchas and caveats)[AUX]The build/upload process works fine but no output from program</x>The build/upload

process works fine except that you get no output - You finally
mastered compiling and loading an application and much to
your dismay, there is no output. You scream into the night but
no, that does not solve the problem. You simply forgot that
PlatformIO creates empty setup and loop functions. You probably
forgot to transfer your .ino code into the PlatformIO's [Link]
file.
<x>PlatformIO (gotchas and caveats)[AUX]Contents of directory not updated after directory content changes</x> Contents of
directory not updated after directory content changes - When
creating a project, I navigated up to a target directory. Having
found that it contained unwanted items, I opened Windows
explorer and deleted some items. Upon reopening PlatformIO's
import Arduino project window the items were still there. I had
to close and restart PlatformIO to get a fresh start.
<x>PlatformIO (gotchas and caveats)[AUX]Removing file from project deletes it</x>Be careful with the delete

option when you right click on a selected file in the Explorer

window. Deleting a file from a project deletes it from the disk.
PlatformIO's behavior within this context is dangerous as
precious work can be lost. The reason is that removing a file
via PlatformIO translates as deleting a file with VS Code.
Again, differentiate VS Code from PlatformIO. You cannot
remove a file from a project.
<x>PlatformIO (gotchas and caveats)[AUX]Build uses files in the project's source code directory and subdirectories</x>Builds use files in

the project's source code directory and files in subdirectories. I

had been doing Perl work in a subdirectory of the source file
directory. PlatformIO compiled files in the source directory and
in the subdirectories, thereby leading to lots of errors. I do not
think that it is a good idea for the tool to consider that all
.h/.cpp files in subdirectories of the project's source code
directory be included in the project.
<x>PlatformIO (gotchas and caveats)[AUX]File editing window can be unpinned, editor reverts to being simple notepad like</x>Individual

source files cannot be unpinned to benefit from a multi-screen

setup. If you try to do so, they will be added to the Windows
desktop as a link. You can then work on it from outside
PlatformIO with whatever program the file's extension is tied to.
Professional programmers will be working in multi-screen
environments. I argue that being able to unpin source files and
maintain access to C++ editing features are a requirement.
<x>PlatformIO (gotchas and caveats)[AUX]New com port not recognized</x>New com port not recognized -

After working with an ATmega2560; I thought I would try

working with the ATmeg328P-Xmini board (see Hardware-
based debugging page 119). Despite my having changed the
 comport to COM14 in the [Link] file, PlatformIO kept
trying to upload to COM11. I had to create a new project with
the new board as the comport specification got hardcoded
somewhere. I failed to find where the COM11 port specification
was hard coded despite considerable time searching.

Some of the above are related to not differentiating PlatformIO from

VS Code. I argue that before installing PlatformIO one should limit
the installation to VS Code (page 86) and work with VS Code with
the C++ extension to be familiar with what it can do; what it is good
for. Then install PlatformIO and work with it knowledgeably
differentiating VS Code from PlatformIO. Doing this will save you a
lot of frustration. Other problems may be due to my lack of
PlatformIO specific knowhow; but in my defense, I plead a long
learning curve.
 13.
Other IDEs (Code::Blocks and MPLAB)

Other IDEs (Code::Blocks and

MPLAB)
The preceding chapters describe the main IDEs for Arduino
programming in a Windows environment I could find. There certainly
could be others. This chapter covers two IDEs which I feel should be
mentioned:

Code::Blocks - I include Code::Blocks because it is a general

purpose C++ IDE which is referred to in the popular C/C++ for
Dummies books (see Bibliography - C/C++ programming page
332).
MPLAB (page 112) is a Microchip tool. I downloaded it, tried
using it, and stopped. Its learning curve seemed steep.

There may be other tools I should have included in this book;

however, the ones covered are professional grade, enhance
productivity, and provide ample means to develop Arduino
applications with.

1.
Code::Blocks
<x>Code::Blocks[AUX]Free, low learning curve C++ IDE; does not seem suited for Arduino dev</x><x>Which IDE to work with?[AUX]Code::Blocks does not seem suited for

You may have come across Code::Blocks in one of the for

Arduino development</x>

Dummies books (Bibliography - C/C++ programming page 332). It is

an IDE which resembles AtmelStudio. Its big drawback is failure to
import, compile, link existing Arduino applications seamlessly. IMHO,
I do not find it adapted for Arduino development.
<x>Code::Blocks[AUX]Free, low learning curve C++ IDE; does not seem suited for Arduino dev</x><x>Free IDEs[AUX]Arduino, AtmelStudio, PlatformIO, Visual Studio, VS

Code, Code::Blocks</x><x>Interoperability[AUX]Code::Blocks not suited for Arduino development, does not create project</x><x>Code::Blocks[AUX]ArduinoBuilder - third-

I first encountered
party plugin to Code::Blocks</x><x>Code::Blocks[AUX]FreematicsBuilder - third-party plugin to Code::Blocks</x>

Code::Blocks when reading the for dummies series of books (C,

C++, Arduino). I downloaded Code::Blocks and proceeded to test it.
My first encounter was rather positive as I was threading on familiar
ground, by this I mean that I was in an environment I was familiar
with, namely a C++ IDE whose interface resembles Arduino IDE's. I
was encouraged to go on since there was the possibility of creating
an Arduino sketch file, one which contained setup and loop. I proceeded
by compiling this file and was quickly set back as the compiler
reported that it could not find the Arduino.h header file. I searched
around and found that I could tell the compiler where to look for
header files. After telling Code::Blocks where to find the Arduino.h
file, it did find it but it could not find the .h files that were included in
it. This is when I decided to stop. Apparently, Code::Blocks does not
seem to support seamless Arduino project developments as the
other IDEs do.
To be fair, I must confess that I discovered a third-party product:
Freematics ([Link] proposes an application it has
named FreematicsBuilder (ArduinoBuilder). This tool enables
building an Arduino application; however, working with two
independent tools, one to edit code, the other to build the
application, is far from ideal. I decided not to devote any more time
to it.

2.
MPLAB
MPLAB
<x>Which IDE to work with?[AUX]MPLAB may be overkill for Arduino development</x><x>MPLAB[AUX]May be overkill for Arduino dev</x>

(Microchip) seems to be designed for professional embedded code

development. This is what Microchip says about its tool:
 MPLAB X Integrated Development Environment (IDE) is an
expandable, highly configurable software program that incorporates
powerful tools to help you discover, configure, develop, debug and
qualify embedded designs for most of our microcontrollers and
digital signal controllers. MPLAB X IDE works seamlessly with the
MPLAB development ecosystem of software and tools, many of
which are completely free.
After downloading it, I found that it required downloading other tools
and that, its being a NetBeans based tool, proficiency with the
NetBeans environment was a prerequisite, thereby presaging a
steep learning curve. The outcome was that I decided to forego
MPLAB.
 14.
Debugging

Debugging
Creating an application is a round-robin design, code, test, debug
process. Each phase of the <x>Debugging hardware-based[AUX]AtmelStudio with specialized boards enable hardware-based
debugging</x><x>Debugging hardware-based[AUX]Bugs categorized as glitches, code thrashing, misdoings</x>process is equally
important. The problem is that debugging can be excessive; it can
consume from 30% to 90% of the overall development time. Given
the extreme debugging time in the overall development process, it is
a given that the number one priority is doing everything possible to
reduce the likelihood of bugs. This is fundamentally what this book
and its companion, Pragmatic C++ Arduino Programming, are all
about.

Avoiding bugs (next section) - This chapter starts with the all-
important avoiding bugs (next section).
Types of problems (page 114) - I have categorized bugs into
three categories: Glitches (page 114), Code thrashing (page
115), and Misdoings (page 117). Although they overlap to
some extent, their cause and how they manifest themselves
differ. And they do have a common feature, their cause can be
extremely difficult to find.
Print-based debugging (page 118), although decried by some
as being obsolete (when compared to hardware-based
debugging), remains nonetheless a useful technique. Events
may be selectively displayed on the computer's serial monitor.
Were a problem to show up, pertinent data, parameter values,
and return values, could be displayed. It is based on placing
print commands which display information on the application's
behavior in a serial monitor. The programmer gets an overall
view of what is happening. I developed a formal framework
(Print-based Debugging Framework page 188) to organize the
 debugging process.
Hardware-based
<x>Debugging hardware-based[AUX]Atmel ICE - Not successful debugging Uno or Mega</x>

debugging (page 119) is based on the use of an external

hardware tool (Atmel ICE (page 122) or ATmega328P-Xmini
(page 120) for example) to enable debugging at code level.
Such a tool enables you to stop at a specific source code line,
monitor variables, watch for out of bounds values, change a
variable's value, inspect memory, resume, etc. It brings in an
enhanced dimension to debugging.
Serial debugging (page 125) is proposed via the Visual Micro
plugin into AtmelStudio and Visual Studio. It is similar to
hardware-based debugging. Its main advantage is that there is
no need for an external debugging board; its inconvenience
resides in its limitations.

The three main debugging methodologies (print-based, hardware-

based, serial) are complementary: visualize both the forest and the
trees. You will find these described in the following sections.

1.
Avoiding bugs
<x>Print-based Debugging Framework[AUX]Avoiding bugs - ounce of prevention is worth pound of cure</x><x>Bugs (avoiding them)[AUX]Ounce of prevention is worth pound of

The key to reducing debugging time is avoiding them. You should

cure</x>

adopt good tools, implement procedures, adopt good programming

practices, and use frameworks. Debugging time in general, whether
it be print-based debugging or hardware-based debugging (or serial
debugging), can be reduced by tailoring development for error
prevention or early error detection. Any discussion on debugging
should start with a discussion on avoiding bugs. It is a proven fact
that productivity depends, amongst other factors, on early error
detection. The sooner you catch an error, the less costly the bug.
This is one of the main reasons for using AtmelStudio or Visual
Studio as they check your syntax as you write code. You need not
wait for the compiler to tell you that you made the mistake of using a
comma instead of a semicolon in a for statement.
Furthermore, you should adopt good programming practices to avoid
errors or catch them early. See Good programming practices (page
129).

2.
Types of problems
As you develop your application, you will run into all sorts of
problems. They can be categorized as:

Glitches (next section) - These are the most difficult bugs to

find as they are not systematic. The application runs well but
every once in a while, it fails and you are hard put to determine
a cause and effect relationship.
Code thrashing (page 115) - A runaway print displays
gibberish. Or a variable has wrong values. If the problem is
systematic, you are lucky in that methodology and patience will
enable you to trace back to what is causing the problem.
Misdoings (page 117) - Small mistakes, but perfectly legal
code, such as an = instead of an ==, can cause a lot of
debugging time trying to figure out why the application is not
working properly.

1.
Glitches
Glitches are the
<x>Bugs (possible causes and cures)[AUX]Glitches worse programming problems one can encounter; how does one fix them?</x>

worse programming problems one can encounter. When trying to

find their cause, the application works well and, unexpectedly,
kaboom. Here is a scenario I was faced with as I thought that I was
just about to finish my beehive weighing system. Early tests were
satisfactory and then, as I tested a real live scenario, I started getting
occasional crashes. Notice the word occasional. Most of the time the
application worked properly but at times it hiccoughed - a glitch
raised its ugly face.

My beehive weighing system is a parent/child-based data

acquisition system. All modules have an XBee radio and a
DS3231 clock. The parent has these plus a GSM board to
send me SMSs.
Work is undertaken according to a schedule which sets an
alarm to trigger the next task, what to do, and when to do it. In
between tasks, the system is asleep. When a task is done, the
clock's alarm is set to wake up the system for the next task.
I occasionally send an SMS to the parent module to add,
modify, or remove a schedule.

I got the system to run perfectly with five modules in the network, the
maximum being 16. Things were going well but unfortunately, I cried
victory a little too soon, the parent crashed occasionally when
rescheduling the next job. This was a nasty glitch because I would
do five runs without a hitch, and on the 6th, doggone it!!!
Question: How do you go about finding the cause of a glitch?

Phase 1: Start by looking at your code. Review it. Try to guess

where the problem might lie but do not work too hard on it. Do
not dive too deeply into solving the problem right away. Go
away and do something else. Let your unconscious keep
working because your mind is such that when something is
bothering you, it will keep dwelling on it. Once refreshed and a
little more confident, come back, roll up your sleeves, and get
back to work.
<x>Bugs (possible causes and cures)[AUX]First step to find glitch's cause, eliminate randomness</x>Phase 2: If you are still

in the dark as to why your application is crashing sporadically,

try to transform a random occurrence into a regular one. This
is the key to ultimately fixing the problem. You must remove
randomness. Run tests, many tests, each time jotting down
test conditions, until a pattern emerges. Once you have
 successfully reproduced the problem, it is a just question of
time. You will ultimately find the offending code.
Phase 3: Removing randomness is easier said than done. If
you are able to transform the glitch into a repeating
occurrence, you have succeeded in setting up conditions
required to find the bug. I ran more tests and jotted down test
conditions. After a while, a pattern emerged. I discovered that
when doing a run without a new SMS, things worked well but,
when doing the same with a new SMS, I got a crash. Half the
problem was solved because I had successfully removed the
randomness of the crashes. The bug was now reproducible.
The problem seemed to be inside the function which handles
new SMSs. It is complex in that it does a lot of error checking
on the SMS's validity: its format, its content, etc. From then on
debugging was far simpler as I had narrowed down the bug's
location. As soon as I discovered this cause-and-effect
relationship (removed randomness), I was well on my way to
finding the bug. Correcting this glitch did cost me a full day's
work, but I got it fixed!!!

<x>Bugs (possible causes and cures)[AUX]Brute force approach to finding glitches; pare down application and rebuild progressively</x><x>Bugs (possible causes and cures)

[AUX]ATmega328P-Xmini - watch variables to find glitches</x><x>Debugging hardware-based[AUX]ATmega328P-Xmini - watch variables to find glitches</x><x>Debugging

What if, after doing phases 1 to 3

hardware-based[AUX]Watch corrupted variable to pause execution and find glitch's cause</x>

above, you still have not found why the application bugs. This
happened to me. The only solution I could think of was the brute
force approach. Start from setup and exit early after printing
something. It should run correctly. Progressively include more and
more functionalities, all the while checking for proper execution.
Sooner or later the program will bug. This will help you narrow down
where the problem lies. From then on, selectively include/exclude
blocks of code - you will ultimately find the cause. Using a hardware
debugger such as the ATmega328P-Xmini (page 120) hardware-
based debugger would be a better solution - place watches on
variables you deem get thrashed. From then on, persevere.

2.
 Code thrashing
The application is getting a little
<x>Bugs (possible causes and cures)[AUX]Application displays gibberish, restarts, etc.</x>

large and complex. It has been working well, however, suddenly, a

char array displays gibberish and the application restarts, which
means that it would loop forever. Something is trashing the address
the application should return to after exiting the function.
<x>Bugs (possible causes and cures)[AUX]Apply methodology for hard to find bug, understand the logic</x>This was a systematic

error, not a glitch. There was no randomness. This bug differed from
the one I described in the glitch problem above in that I was unable
to narrow down where the problem was. I concluded that some sort
of bug finding methodology over and beyond the one developed in
the glitch handling section above imposed itself.
Question is: A new methodology is required, now what? I decided to
go back to basics, a sort of "clean slate" approach.

<x>Bugs (avoiding them)[AUX]Inline comments clarify logic, fewer bugs</x><x>Comments[AUX]Program Documentation Framework - use Awk or Perl regexes to

extract comments</x><x>Comments[AUX]Inline comments clarify logic, fewer bugs</x><x>Bugs (avoiding them)[AUX]Inline comments clarify logic, fewer

First step: Understand the

bugs</x><x>Perl (general)[AUX]Perl program creates program documentation</x>

application's logic. I first looked at the code to figure out what I

was doing. The code was a little complex. It took me a while to
understand the logic. I had documented the code somewhat,
but not enough. So, I reviewed the code. I managed to
understand how the algorithm worked, what parameters were
used for, and more. This led me to enhance the function
documentation, which I could print via my Perl [Link]
program (Program Documentation Framework page 155). This
turned out to be a valuable step. Documenting the code,
function by function, deepened my understanding of the
program's inner workings.
Second step: Determine when things start getting bad. I
proceeded by systematically printing the offending pointer's
address and content to narrow down where the problem could
come from. I created a special macro to handle this:
PRINT_ADDRESS_CONTENT(desc, var, contents)
 Here is what gets printed:
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAA
SMS_PrepareSend lineNo 1214
Just after SMS_PrepareSendBG
variable: smsObtainedCompleteAddress: 8156
contents: smsObtainedComplete EMPTY
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAA
Printing the above in several places in the program revealed when
the problem occurred - at what point during the program's
execution the address got changed.
<x>Bugs (possible causes and cures)[AUX]Type checking leniency creates bugs due to undetected misplaced parameters</x>The bug was due to

the Arduino compiler's weak type checking. I had a function

declaration with 14 parameters, one of which was a default, hence
not necessary when using the function. As I programmed, I
somehow omitted one parameter, the compiler was quite happy
with this, but the program was not happy at all. An int became char*,
a char* became an int. You can imagine what this can do when doing
a strcpy on a char* which, in reality, is the value contained in an int - it
trashes the memory location pointed to by the char*. This will wreak
havoc, or not, depending on what the thrashed location is used for.
<x>Bugs (avoiding them)[AUX]Two-step approach: review code and apply systematic validity checks</x>This two-step approach,

review code followed by systematic checks of variable addresses

and contents, proved successful. I found the bug. And what is more,
it forced me into modifying code to make it cleaner and easier to
maintain.
<x>Bugs (possible causes and cures)[AUX]ATmega328P-Xmini - watch variables to find glitches</x><x>Debugging hardware-based[AUX]ATmega328P-Xmini - watch variables to

Hardware-based
find glitches</x><x>Debugging hardware-based[AUX]Watch corrupted variable to pause execution and find glitch's cause</x>

debugging could have helped me find the bug described above more
quickly by putting a watch on the variable which was printing
gibberish. But this was not an option as I could not get Atmel ICE
hardware-based debugging with AtmelStudio to work. Unfortunately,
at the time, I did not know that Atmel proposed the Xplained
hardware- based debugging boards, namely the ATmega328P-Xmini
(page 120), a miracle board which combines an ATmega328P
microprocessor and on-board debugging capabilities. Had I used this
board, I would have quickly found where the variable got thrashed.
 3.
Misdoings
[Link] program</x>Misdoings are
<x>Misdoings[AUX]Dumb mistakes not detected by compiler; found by Perl

seemingly banal errors due to wrong keystroke, being tired,

becoming momentarily distracted, stretching your multitasking
abilities, and the like. The problem is that it is nearly impossible to
write code without occasionally making a mistake. The compiler will
find these most of the time. They can be rectified immediately. But
there are others, considerably more subtle, which the compiler will
let pass since the code has no syntax errors. An example of such
errors is using an = sign instead of the logical operator == in an if
statement. This can wreak havoc. Your code does compile but
occasionally fails to run properly. This type of error, a glitch, can be
extremely difficult to find. I call these types of errors misdoings.
<x>Perl (general)[AUX]Perl program looks for dumb mistakes: [Link]</x>So far, I have identified

four misdoings which could penalize the programmer with substantial

debugging time. None of them are easy to detect. I have developed
a Perl program ([Link]) to detect these. Please not that it
is not a replacement for C lint utilities which do similar work.

<x>=[AUX]= instead of == in an if or while, == instead of = in an - assignment</x><x>Misdoings[AUX]= instead of == in an if or while or == instead of = in an

or while - A typical misdoings when doing an if or while

assignment</x> if
would be if(x=0). A comparison was obviously intended, not an
assignment. The seasoned C/C++ programmer does not
generally commit such an error; however, he/she does do this
when a little tired. Newbies, on the other hand, will often make
this mistake which will lead to inexorably long nights figuring
out why the application is not working properly.
Doing a==0 instead of an assignment such as a=0 should not
happen. There are circumstances whereby one is prone to
make this type of mistake. Imagine, you are coding away and
have written an if statement such as
if (thisIsAlongVariableName == 5)...
 and then, inside the if's body you want to do
thisIsAlongVariableName = 22;
Are you going to retype the long variable's name? Nope! You will
do a copy paste, modify the assignment, and modify 5 to 22.
Unfortunately, in the process, you forgot to change == to =. Happy
debugging!

statement default case missing or

<x>Misdoings[AUX]switch default case missing or empty</x>switch
empty - There are two mandatory rules relating to the default
case of switch statements: (١) there should always be a default
case, and (٢) the default case should always contain code to
either process a normal default or trigger an error which informs
that the switch should never exit via the default.
[Link] -
<x>Misdoings[AUX]Regex missing F() macro in [Link]</x><x>Macros[AUX]Misdoings: [Link] missing F() macro</x>

When an application starts, all strings are loaded into RAM,

thereby reducing heap space considerably. The use of the F()
macro enables storing strings in flash memory to be loaded
into RAM on an as needed basis, which avoids needlessly
keeping them in RAM. Consider the following example:
[Link](F("This string is loaded on an as needed basis."));

When coding prints, it is easy to forget the F() macro, thereby

unnecessarily decreasing available RAM. The detect misdoings
Perl program will list all prints which contain (" - an open parens
followed by a double quote. Prints which contain quoted strings
should be followed by (F(", not by (".
The program [Link] should be applied on a regular basis
as coding progresses. It should detect these errors thereby saving
on debugging time. I applied it on my 35 files beehive weighing
system, this is what I got:
Total errors found : 14
if = instead of == : 0
assign == instead of =: 1
SwitchNoDefault : 1
SwitchDefaultNoCode : 2
F macro missing : 10
The 10 missing F() macros are just a RAM usage optimization issue.
However, assign == instead of =, the two SwitchNoDefault and the
one SwitchDefaultNoCode errors could be a source of bugs. I quickly
corrected these. See Misdoings (appendix) (page 293).

3.
Print-based debugging
<x>Print-based Debugging The most immediate
Framework[AUX]Printing strategic data values, where the program is at</x>

approach at trying to locate why a program is not working and where

the problem stems from consists in printing strategic data values and
where the program is at as it runs. As one debugs, prints will pepper
the application and quickly get out of control. The solution is a
macro-based system which selectively allows certain prints to be
active while others lie dormant. The basic premise consists in
defining macro categories and a three-level macro hierarchy: global,
file, and function level.

Macro
<x>Print-based Debugging Framework[AUX]Macro categories: navigation, memory use, critical situation, data values</x>

categories - Several macro types may be categorized:

navigation macros (enter/leave a function, where is the
program at), memory use macros (fragmented and contiguous
memory, outstanding unreleased pointers), critical situation
macros (voltage getting low, values below/above threshold...),
data value macros, and others.
<x>Print-based Debugging Framework[AUX]Debugging macro level: global, file, function</x>Global level macros

enable/disable macros at global level. A program's production

version should not have any active debugging [Link].
Several category specific global level macro control prints.
File level - One's focus when debugging usually centers
around very few files, often one only. File level macros
enable/disable all debugging macros inside a specific file.
Function - Since a file may have many functions, most of which
have been validated, limiting macro activation to selected
 functions unclutters serial monitor output. The programmer can
concentrate on the functions which seem to be where the
problem lies.

These mechanisms have been formalized in the Print-based

Debugging Framework (page 188)

4.
Hardware-based debugging
<x>Print-based Debugging Framework[AUX]Complementary tool relative to hardware-based debugging</x><x>Debugging hardware-based[AUX]Complementary tool relative to

print-based debugging, requires special hardware and compatible IDE</x><x>Debugging hardware-based[AUX]Watch corrupted variable to pause execution and find glitch's

As I wrote before, hardware-based debugging and print-based

cause</x>

debugging are useful and complementary tools. Both should be

used, as they address different needs. There are times when one
wants to get a global picture of the program's execution (print-based
debugging). There are other times when one wants to look at code
with a magnifying glass to determine exactly what is going on
(hardware-based debugging).
Print-based debugging is easy - just do prints of the variables you
are interested in along with comments. Hardware-based debugging
is an entirely different matter - it requires special equipment. There is
a third debugging option: serial debugging. This third option is
identical with hardware-based debugging, but it suffers from severe
constraints - it is not as flexible. The discussion which follows covers
hardware-based debugging.
The first obstacle to overcome when attempting hardware-based
debugging is that the microcontroller must implement a mechanism
to enable it to control the program's execution (stop, look at a
variable, resume). The second difficulty is that a separate piece of
equipment (hardware debugger) needs to be connected to the
microcontroller to control execution. The microcontroller contains the
mechanics for hardware-based debugging. The hardware debugger
contains the interface to tell the microcontroller what to do (stop,
resume...).
Hardware-based debugging requires that two concepts be
understood:

<x>Microcontrollers[AUX]Microcontroller architectures: AVR, ARM, PIC</x><x>Debugging hardware-based[AUX]Debugging hardware must be compatible with

microcontroller architecture (AVR, ARM)</x><x>Arduino (general)[AUX]Arduino chips are AVR type</x><x>Microcontrollers[AUX]ESP32 chips are ARM

devices</x><x>ESP32[AUX]ESP32 chips are ARM devices</x><x>Microcontrollers[AUX]PIC (Programmable Intelligent Computer): both AVR and ARM

The microcontroller's
chips</x><x>Debugging hardware-based[AUX]Olimex - hardware-based debugger for ESP32</x>

architecture: AVR, ARM or PIC: Both the ATmega328P and the

ATmega2560 are AVR microcontrollers. The ESP32 is an ARM
microcontroller. AVR means Automatic Voltage Regulation.
ARM means Advanced RISC Machine. Both AVR and ARM
microcontrollers are RISC based microcontrollers. RISC
means Reduced Instruction Set Computing. PIC, which you
will occasionally find in the literature, means Programmable
Intelligent Computer. Hardware-based debuggers are
generally designed to support one or more such architectures.
Thus, if a particular debugger supports ARM only (Olimex), it is
useless to try to use it on AVR microcontrollers.
<x>Debugging hardware-based[AUX]ATmega328P and ATmega2560 compatible with hardware debugging with debugWire and JTAG respectively</x>Two

debugging standards are supported by microcontrollers:

debugWire and JTAG. The ATmega328P supports debugWire.
Both the ATmega2560 and the ESP32 support JTAG.
debugWire is Atmel specific for small microcontrollers. JTAG is
an industry standard.

Another important issue should be noted: Should debugging be done

on the board the program is to run on, or on a bigger development
board? I already indicated that due to the ATmega328P's small size,
I develop programs on the larger ATmega2560. Final testing should,
however, be done on the target board, not on the development
board.
<x>Debugging hardware-based[AUX]Arduino does not support hardware-based debugging; Arduino V2 does</x><x>Debugging hardware-based[AUX]AtmelStudio with

specialized boards enable hardware-based debugging</x><x>AtmelStudio (hardware-based debugging)[AUX]Hardware-based debugging with ATmega328P-Xmini successful, with

On the development toolchain side, depending on the

Atmel ICE not so</x>

microcontroller you will be developing on, you are faced with a

choice of environments to develop with (Arduino IDE, AtmelStudio,
Visual Studio + Visual Micro, PlatformIO, MPLAB, Code::Blocks) and
several hardware debuggers. Hardware debugging requires that the
IDE support it. The Arduino IDE 1.8.19 does not have a debug
feature; the new version 2 does. PlatformIO supports Arduino
hardware-based debugging, but it is limited to very specific
hardware. There is also MPLAB (page 112) from Microchip but, as
indicated in the section which describes it, it seems to be overkill for
Arduino development and has a major learning curve.
<x>Debugging hardware-based[AUX]Debugging hardware must be compatible with microcontroller architecture (AVR, ARM)</x><x>Debugging hardware-based[AUX]Atmel

ICE - Not successful debugging Uno or Mega</x><x>Debugging hardware-based[AUX]ATmega328P-Xmini Uno pin compatible supported by AtmelStudio, success using it</x>

<x>Debugging hardware-based[AUX]ATmega2560RFR2 Xplained Pro - specialized hardware for RF communications, not Arduino boards pin compatible, success using

On the hardware debugger choice, Atmel has been for years

it</x>

proposing Atmel ICE. It supports debugWire and JTAG. I tried using

it - I must report utter misery trying to get it to work. More on this in
the Atmel ICE (page 122). And to my great surprise, I discovered the
Xplained Xmini line of debugging boards. These are two-in-one
microcontroller/hardware debugger boards - I tested the
ATmega328P-Xmini (Xplained Mini) with my MemMgt program,
success, bliss, pure plug & play!!! Too bad I was unaware of this
board's existence when I was developing my beehive weighing
system and attempting to debug it with Atmel ICE. It would have
saved me tons of time. You will find in the following section a full
description on how to hardware debug a program with the
ATmega328P-Xmini (next section). There is an ATmega2560 variant
(ATmega2560RFR2 Xplained Pro) which I have tested successfully
on my MemMgt application. However, it has two major drawbacks:
getting output into the serial console may take some doing and it is
not at all Arduino ATmega2560 pin compatible. The search is still on
for an Arduino ATmega2560 pin compatible hardware-based
debugging solution.
There are other hardware-based debuggers, their prices range from
$15 to $400. It is a jungle. I have not tried any of them. You might
want to look at the new Arduino IDE V2's debugging possibilities - it
supports SAMD boards ([Link]
v2/tutorials/ide-v2-debugger). PlatformIO also proposes limited
hardware-based debugging capabilities for some Arduino boards
(see PlatformIO hardware-based debugging page 108).
After installing Visual Micro for AtmelStudio (MicrochipStudio) (page
84), I discovered that it proposes Serial debugging (page 125). I did
not know that such a feature existed. I had read about simulators,
but this is not a simulator; it behaves like a hardware-based
debugger.

1.
ATmega328P-Xmini
As mentioned in this
<x>Debugging hardware-based[AUX]ATmega328P-Xmini Uno pin compatible supported by AtmelStudio, success using it</x>

chapter's introduction above, the Xplained Xmini boards combine a

microcontroller and on-board debugging capabilities, the idea being
that you should load your application into the board's microcontroller
and be able to debug it by adding checkpoints, watches on variables,
display memory content, view variable values, and benefit from a
host of other possibilities. Physically, the ATmega328P-Xmini looks
like an Arduino Uno breakout board. You can solder header pins so
that you might connect whatever as if it were a regular Arduino Uno
board.
Having said that the ATmega328P-Xmini is an Arduino Uno board
with on board hardware based debugging capabilities, its looks,
when first pulled out-of-the-box, are surprising. It is slightly larger
than an Arduino Uno board with a small protrusion into which you
can plug a Mini USB cable. It is not shipped with header pins; you
are immediately struck by the seven rows of holes along one edge,
and six rows along the other edge. These are referenced in the
product documentation (Bibliography - Hardware-based debugging
page 336). Two of the rows are meant to be identical with the
Arduino pin configuration (RX, TX, analog pins, etc.). When you get
the board, start by soldering the Arduino like header pins.
Before continuing, note that the board's name may be confusing:
ATmega328P-XMINI (the box it came in), ATMega328P Xplained (on
the board), ATmega328P-Xmini (User guide). Throughout this book,
I'll refer to it as the ATmega328P-Xmini.
Hooking up the ATmega328P-Xmini and getting it to run in
AtmelStudio V7 was incredibly easy, pure plug & play. Here is what
needs doing:

Connect the board to a comm port via the USB cable.

Open AtmelStudio and import an Arduino project (I did this on
my MemMgt application). You will immediately be confounded
by AtmelStudio's colorful display of the board in the left-hand
portion of the work area.
Configure the project's target to DEBUG.
Turn optimizations off in the C and C++ options section and
add -fpermissive to the compiler options.
Set the device to ATmega328P.
Open an Arduino IDE, any project, open the tools menu and
set its comm port to whatever the Xmini is hooked to and the
board to a Uno. Open the Arduino IDE's console and set its
baud rate to 9600. This will enable you to visualize [Link] in
the Arduino console thereby enabling you to undertake both
hardware-based debugging and print-based debugging
simultaneously. Note that the 9600 baud rate seems
hardwired. I tried other baud rates; output was unreadable.

That is it. After doing this, assuming you have imported an Arduino
project properly, do a build. It will automatically upload the .hex
executable into the microcontroller. There is no need to do an upload
via an external tool such as is required when using a conventional
Arduino. You may launch the program in debug mode or non-debug
mode and enjoy.
<x>Interoperability[AUX]Develop with Visual Studio 2022 + Visual Micro, hardware debug with AtmelStudio + Xplained boards</x><x>Debugging hardware-based[AUX]Develop

with Visual Studio 2022 + Visual Micro, hardware debug with AtmelStudio + Xplained boards</x><x>Visual Studio[AUX]Develop with Visual Studio 2022 + Visual Micro,

hardware debug with AtmelStudio + Xplained boards</x><x>Visual Micro[AUX]Develop with Visual Studio 2022 + Visual Micro, hardware debug with AtmelStudio + Xplained

boards</x><x>AtmelStudio (hardware-based debugging)[AUX]Develop with Visual Studio 2022 + Visual Micro, hardware debug with AtmelStudio + Xplained

You might want to establish interoperability between Develop

boards</x>

with Visual Studio 2022 + Visual Micro, hardware debug with

AtmelStudio + Xplained boards board. This may be possible but I
have yet to try it.
 2.
ATmega2560RFR2 Xplained Pro
<x>Debugging hardware-based[AUX]ATmega2560RFR2 Xplained Pro - specialized hardware for RF communications, not Arduino boards pin compatible, success using

The ATmega2560RFR2 Xplained Pro is not a standard Arduino

it</x>

ATmega2560 plus debugger hardware such as the smaller

ATmega328P-Xmini is. The 2560 Xplained Pro is specialized
hardware for testing RF communications. It is not pin compatible with
the standard Arduino ATmega2560 and is not meant to enable
hardware-based debugging of Arduino ATmega2560 applications. I
decided to nevertheless purchase it since I could not find an
alternative to hardware debug in a large memory space. And the
price was low - cost me $35.
I received the board just as I am finalizing this book. As with the
ATmega328P-Xmini, I hooked up the board to the computer (PC -
Windows 10), opened AtmelStudio V7 and imported the MemMgt
program, set the compiler options, the device, and the debugging
board. All went well except for one drawback: I could not get any
output into the serial terminal. This was my initial experience:

AtmelStudio compiled MemMgt error free and uploaded the

executable into the board. So far so good.
I could place breakpoints wherever I wanted to stop the
program and look at variables. Fantastic!
I could not get any output into the console via [Link]. The
board does provide a virtual COM port which might be used for
visualizing prints in a serial terminal. This remains to be tried.
The Arduino IDE failed to recognize the board whereas it did
recognize the ATmega328P-Xmini as being a UNO.
The ATmega2560RFR2 Xplained Pro lacks just about all the
pins one finds on the standard Arduino ATmega2560.
There could be problems with interfacing the application via
serial communications or other protocols.

The conclusion is that hardware debugging an Arduino program on

an ATmega2560RFR2 Xplained Pro is possible however work
remains to get it to handle pin based I/O and serial communications.
I am still searching for an ATmega2560 Arduino pin compatible
hardware debugger.

3.
Atmel ICE
A couple of years ago, since my
<x>Debugging hardware-based[AUX]Atmel ICE - Not successful debugging Uno or Mega</x>

environment was and still is the ATmega328P/ATmega2560

combination, I had opted for Atmel ICE. The reasoning was simple:
Use an Atmel tool to develop on an Atmel microcontroller whose
code was compiled inside an Atmel IDE to be debugged by an Atmel
hardware debugger. Little did I know how much time and frustration
this choice entailed. If you are interested in the experience I lived
through, read on otherwise, jump to the next section (Serial
debugging page 125). Here is the story.
I naively thought that by following the instructions, I would
experience a successful hardware debugging session. I diligently did
my homework: YouTube videos, the Web, and Atmel manuals. I then
proceeded to connect Atmel ICE to the ATmega2560 board via the
ISP connector, then clicked on Start Debugging and Break in
AtmelStudio.
Hardware-based debugging is possible only if the microcontroller
manufacturer has built support mechanisms into the microcontroller.
The standard mechanism today seems to be JTAG. It does a lot
more than allow debugging. It also enables one to debug PCBs by
managing individual microcontroller pins. These may be
programmed to send and receive data to test the PCB's reaction. It
also provides hardware-based debugging features: breakpoints can
be set in source code, variables looked at and changed, etc., thereby
giving the programmer a close look at his/her application's behavior.
debugWire is a protocol similar to JTAG. It is provided by Microchip
for some of its small Atmel microcontrollers. To make a long story
short, the ATmega328P (Arduino UNO) supports debugWire but not
JTAG. The ATmega2560 is the reverse - it supports JTAG but not
debugWire. Whatever standard is used, whether it be debugWire or
JTAG, a hardware-based debugger is needed.
Atmel ICE is an inexpensive tool considering what it does. At around
$120 it is well worth the investment since it should save lots of
debugging time. So, I bought one and worked my way through a few
how-to YouTube videos. It seemed fairly simple. Basically:

Install AtmelStudio V7 first.

<x>AtmelStudio (hardware-based debugging)[AUX]YouTube Atmel ICE tutorials</x>

Be sure NOT to connect Atmel ICE before having installed

AtmelStudio. If you do so, Windows will install the wrong Atmel
ICE drivers which will lead to long nights of frustration as you
try to unravel driver issues.
Connect Atmel ICE to the computer via the supplied micro-
USB cable. The correct driver will be automatically
downloaded.
Connect the board you will be working on. Open AtmelStudio
to define the device in the Project Properties window.
Turn off C and C++ optimization and set the profile to
debugging.
Go to Project Properties Tools. Choose the Atmel Ice debugger
and select debugWire.
You are set to go. Set breakpoints and start debugging,
advance in the program one line at a time, look at the values of
variables. There are a ton of things you can do. This is what an
Atmel rep said in a YouTube video. Simple enough? In a word,
NO!!!

What follows is my experience trying to get Atmel ICE to work with

the ATmega2560 in debugWire mode and later, with the
ATmega328P microcontrollers. I did mention above that the
ATmega2560 did not support debugWire. It supports JTAG only. But
I did not know this when I began to debug with Atmel ICE. I should
start off by saying that I am a complete novice doing hardware-
based debugging. My approach is to follow instructions and hope for
the best. After reading through the Atmel ICE manual, viewing
YouTube videos, and searching the Internet, I believed that all I
needed to do was to hookup Atmel ICE to the board and activate
debugWire. At the time, I was not aware that there was a second
standard (JTAG) and that choosing one (debugWire or JTAG)
depended on the microcontroller being used. In my defense, none of
the videos, manuals, etc. explicitly mentioned this dichotomy.
I started by attempting to use Atmel ICE on my Arduino
ATmega2560 board in debugWire mode because the Atmel YouTube
video indicated that all that was needed was to choose debugWire in
the application's project properties tool option. My recollection is that
the video was not explicit as to debugWire having been chosen
because the target was an ATmega328P. Atmel ICE, instead of
proposing ISP and debugWire, proposed ISP and JTAG. To my
surprise, I could not configure the ATmega2560 in debugWire mode.
No matter what I did, I got ISP and JTAG as the proposed options,
not ISP and debugWire; I then set the ATmega2560's OCDEN fuse
which I mistakenly took to be the debugWire enable fuse. This did
not work. After a while, it dawned on me that Atmel ICE or the
ATmega2560 might not support debugWire. Another thought crossed
my mind, perhaps I got a bad Atmel ICE. Or, there are several Atmel
Ice versions and I got the wrong one. Hours later, after learning all
about fuses and how to set them with avrdude or with AtmelStudio, I
abandoned the idea of using the ATmega2560 board and Atmel ICE
combination. At this point, I had not yet realized that the
ATmega2560 supported JTAG only, not debugWire. I clung on to the
ATmega2560/debugWire combination as a child clings to their stuffy.
In the process of trying, I damaged two ATmega2560 boards.
Luckily, I had spare ones on hand.
Failing to get AtmelStudio to propose a debugWire choice on my
ATmega2560, I went on the Internet to find a solution. A number of
YouTube videos encouraged me to endeavor along this path. Others
expressed their frustration which provided zero encouragement. The
Atmel ICE Manual's debugWire section (10/2016) does explain that
the DWEN fuse must be turned on but it does not specify that
debugWire does not work on the ATmega2560.
<x>Debugging hardware-based[AUX]Dragon Programmer, STK500/600 should enable fixing bad microcontroller using HVPP</x>I failed to find in

forums or anywhere else the answer to why AtmelStudio with Atmel

ICE on a ATmega2560 was proposing ISP and JTAG and not ISP
and debugWire. The forums teem with talk about fuses, bricked
microcontrollers, high voltage reset (12V HVPP - high voltage
parallel programming), turning things on and off in the correct order,
and unsoldering bridges on the board to name a few. In a nutshell:
everything but the kitchen sink. As I plodded along, the experience
proved to be extremely confusing and time consuming. Luckily for
me, I do this for fun. I do not make a living off of it and no one is
watching the clock. And I learned a new term: bricked
microcontrollers. It means that your microcontroller is now a little
brick, useless for anything except as a little brick in some small
construction project. A bricked microcontroller does not mean it is
damaged, a fuse may be reset to render it usable again. The Atmel
ICE documentation p. ٤٦ does say ...not doing this will render the
device stuck in debugWire mode, and high voltage will be required to
revert the DWEN setting... In other words, resetting it with a 12V high
voltage source (HVPP or HVSP) is supposed to fix the problem.
I searched the Microchip site for info on HVPP to no avail. The data
sheets do not refer to it. I could not find specific references on what it
was and how to implement it. I did however find an Atmel YouTube
video on using the Atmel development board (STK500/600) to fix a
bricked microcontroller by applying HVPP on it. This video gave me
a glimmer of hope, but I did not pursue the idea.
I then had an eureka moment. What if I tried Atmel ICE on an
Arduino Uno (ATmega328P)? I knew this would be a pointless
exercise since I would not be using a ATmega328P for debugging. It
is too small, but I wanted to experience hardware-based debugging.
I hooked it up and to my surprise, I finally got the choice of
configuring Atmel ICE in debugWire mode!!! Good, I just got a
chance to start fresh. I might still get hardware-based debugging to
work. I chose debugWire and looked forward to doing hardware
debugging. I really wanted to try it out and discover the promises
which underlie the concept. I would finally be doing real professional
work. But that is not counting on the little devilish urchins which lurk
in the dark corners of Atmel ICE. My first try at debugging produced
an error condition, the often seen Got 0xC0 instead of 0x00. Silly of
me! I had not set the DWEN fuse (debugWire enable). So, I opened
the AtmelStudio Programming device window and set the fuse. I
naively believed that things would work but to no avail. I still got the
infamous message, so, I decided to undo what I had done - revert
back to the DWEN fuse being disabled. To my surprise, I could no
longer access the fuses. They were all grayed out. Furthermore, I
discovered that I could no longer upload a sketch into my board. It
seems I had one more damaged board (actually it is the
microcontroller that is damaged, not the board). I verified this by
replacing the ATmega328P with a fresh one. This fixed the problem.
I could upload sketches again. The damaged microcontroller refused
to play along. Off it went into the bad boards, bad microcontrollers,
and bad components bin.
Trying to hardware-based debug with Atmel ICE was an exercise in
frustration, lots of time spent with little or no success. I did learn a lot
about microcontrollers, but I ultimately failed to configure the
ATmega2560 or the ATmega328P for hardware-based debugging
with Atmel ICE.
Another point is worth mentioning. The ATmega328P datasheet has
a couple of pages on debugWire which contains interesting material,
but it does not mention JTAG. On the other hand, the ATmega2560
datasheet (445 pages) makes no reference to debugWire nor to the
DWEN fuse, but it does have several pages on JTAG. This led me to
conclude that the ATmega2560 does not support debugWire. This
idea is corroborated by the choices offered when configuring Atmel
ICE in AtmelStudio for the ATmega2560, only ISP and JTAG are
proposed. It then dawned on me that the videos I had watched were
based on the use of ATmega328P, not ATmega2560. But the videos
did not explicitly mention that debugWire was ATmega328P specific
and that it would not work on a ATmega2560.
There is not much more I can say about hardware-based debugging
with Atmel ICE. I had purchased Atmel ICE and tried to debug with it.
The videos I had seen presented the process as if it were a piece of
cake. No! It was a truly frustrating experience. It led me to abandon
hardware-based debugging with Atmel ICE on an ATmega2560.
Having damaged two Arduino ATmega2560 boards, I decided that
enough is enough. In the future I will eschew further investments in
another programmer, and avoid risk breaking more ATmega2560
boards, or cope with the learning curve, nor will I again spend an
inordinate amount of time on such a fruitless endeavor. At any rate,
that is my sincere hope.
<x>Debugging hardware-based[AUX]Dragon Programmer, STK500/600 should enable fixing bad microcontroller using HVPP</x>One more thing,

the Atmel Ice documentation does warn the user that a false move
might render the microcontroller useless. They indicate that if that
were to happen, the microcontroller could be recovered via their
HVPP protocol (high voltage parallel programming - apply 12V on
the reset line). As I said earlier on, Microchip's Web site and
documentation was of no use. There is info in YouTube: The Dragon
Programmer (an old, discontinued Atmel piece of equipment no
longer available, even on eBay) or the Atmel AVR STK500/600
evaluation board ($150+ on eBay). Complex, they require
plugging/unplugging, stripping a wire from a ribbon connector,
soldering pins, and more hack-like stuff. This route being totally
unprofessional, I decided that it was not the way I wanted to go.
Thankfully, by way of a post scriptum, Microchip proposes the
Xplained Xmini microcontroller/debugger boards (ATmega328P-
Xmini page 120). These enabled me to do that which I had earlier
failed miserably to do with Atmel ICE. And there is of course the
reliable time proven print-based debugging solution (see Print-based
Debugging Framework page 188).

5.
Serial debugging
<x>Debugging serial[AUX]Visual Micro enables serial debugging w/o dedicated hardware</x><x>AtmelStudio (general)[AUX]Visual Micro enables serial debugging w/o dedicated

Serial debugging seemed

hardware</x><x>Visual Micro[AUX]Visual Micro enables serial debugging w/o dedicated hardware</x>

promising, but getting results turned out to be a saga. Read on!

After installing Visual Micro for AtmelStudio (MicrochipStudio) (page
84), I was greeted with a new top-bar menu item vMicro. It contained
a Debugger choice which opened a window-full of items, one of them
being the type of debugging to undertake: serial, hardware, or off.
Given that I had already succeeded debugging with the
ATmega328P-Xmini board from AtmelStudio, I thought I would give
serial debugging a try. This is where the fun began.
Please note that the discussion which follows applies to vMicro for
both AtmelStudio and Visual Studio.
After downloading a new application, I usually give it a try without
reading the manual. Most software is user-friendly enough which
makes the learning curve short. I find that I can usually get started
and need to resort to occasional Internet searches. This was the
case with AtmelStudio and with Visual Micro for AtmelStudio.
However, when it came to serial debugging, I managed to
compile/link/upload my MemMgt application in debugging mode, but
was unable to actually debug the application. It just ran and exited;
breakpoints were inoperant. This is when I thought that a little
outside help would be helpful. Sooo, I opened the debugging help
provided by Visual Micro.
To debug, breakpoints have to be enabled. The online help states
how to do this; the problem is that the menu items in the online help
do not jibe with the vMicro menu items: namely concerning the
Breakpoint Manager in the Serial monitor. In a nutshell, the
Breakpoint Manager renders breakpoints active or passive. It
contains: Jump to breakpoints, Trace to breakpoints, Auto continue,
and Breakpoints off. I chose Jump to breakpoints.
I loaded my MemMgt program; upon doing a Build & Upload from
vMicro, the application got built in DEBUG mode and displayed a
slew of help messages in the Output window. The application ran
and displayed print messages in the serial monitor, which proved
that the build was successful. But, it did not stop at breakpoints. The
Micro Build output window displayed the following:
WARNING: Unable to find the start of the setup() method. This is
required to initialize the software debugger.
WARNING: Unable to find the start of the loop() method. This is
required for automatic pin reports.
Aha, where might the problem lie? Back to googling; luck was not on
my side. I could not find anything concerning the start of setup in the
Visual Micro forum. Sooo, I decided that I had a sufficient grasp of
the context to try the Visual Micro inline debugging tutorial.
I loaded the BlinkWithoutDelay example and followed the tutorial
step by step and it worked. I set a breakpoint inside the loop function;
it did stop the LED from blinking. Why did breakpoints in the Blink
program work and why did they not work in my MemMgt program?

Directory structure - It took me a while, but I finally unraveled

why one worked and not the other. It relates to how the
application was created. The BlinkWithoutDelay program was
created directly from the Visual Micro (vMicro) New Arduino
Project menu choice. This creates a single level directory
structure with two subdirectories: _vm and DEBUG. The
MemMgt program was create from the standard AtmelStudio
New Arduino Project menu choice. This creates a two-level
directory structure which incorporates the ATmelStudo solution
concept (Managing directories page 39). It appears that this
directory structure does not support debugging. No error
messages are issued; however, serial debugging simply does
not work.
Where should setup and loop be located? - BlinkWithoutDelay is
short, it fits in a single .ino file; there is no [Link] file. This
was a surprise; I thought that AtmelStudio required that setup
and loop had to be in [Link]. I tried emulating the
BlinkWithoutDelay program structure by removing [Link]
from the project and adding the [Link] file with the same
content. Since I had established interoperability, this was
simple enough. Upon rebuilding via vMicro, much to my
surprise, I got an undefined reference from the linker.
Apparently, the build failed to recognize the [Link] file,
but it did recognize [Link] file. Why? It seems
that when creating an AtmelStudio Arduino project, setup and loop
get placed in the [Link] file. When doing the same via
vMicro, these files get placed in the .ino file, not in the
[Link] file. Not abiding by this rule generates a cannot
find setup and loop error.
How to enable debugging - The AtmelStudio-based Create
Arduino project does not seem compatible with vMicro serial
debugging. The solution is to create a new project in which setup
and loop get positioned in the .ino file. You can then transfer all
 project .h/.cpp files to the vMicro application project directory
and import them into vMicro via the File Explorer. One benefit
from doing this is Arduino/AtmelStudio interoperability; no need
for a #define macro to differentiate them since they work on
exactly the same code.
Debugging session recompiles application - You just cannot
restart a debugging session; doing a Continue recompiles the
program. If the program were long, this would be a hassle, you
would then be better off using a hardware-debugging board
such as the ATmega328P-Xmini Uno. But, you might run into
not having enough RAM space. Serial debugging on a
ATmega2560 offers considerably more RAM space.
Caveat - Although I got serial debugging to work on the larger
MemMgt application, I found that I was limited with stopping at
predefined breakpoints. Step over, Step into, Run to cursor,
and Set new breakpoint do not work. Activating one of these
resumes program execution; it stops at the next preset
breakpoint. Not being able to define new breakpoints during a
debugging session is a serious limitation.
Project properties - AtmelStudio and vMicro project properties
were the same.
Data watch - The documentation indicates that there is a locals
window; it exists but is inoperant. It remains possible to inspect
local variables by configuring a breakpoint but this takes a little
planning. You can do this by setting a breakpoint and
configuring it. For example, if you want to look at myVar
somewhere along the program's execution, place a breakpoint
and add to its settings:
myVar = {myVar}

This is equivalent to doing a pair of Serial prints such as

[Link]("myVar = "); [Link](myVar);
That which is inside curly braces are variables, their value gets
printed; that which is outside is literal text.
Breakpoints can also be made to conditionally get activated.
Suppose you want to detect a null condition on myVar and preview
values of some variables, you can add a break point somewhere in
your code with the condition that myVar == null and add to the
breakpoint variable/value displays such as done above.
 15.
Good programming practices

Good programming practices

Countless books, articles, blogs, seminars, authors, and others detail
how to improve program maintenance, <x>Golden rules[AUX]KISS principle - simple solutions often better than
complex ones</x>detect errors early, provide robustness, enhance productivity,

etc. Programming procedures and standards, be they in-house,

generally accepted, or formally proposed during conferences, are
important to success. All large-scale complex endeavors, whether
they be airplanes, skyscrapers, or operating systems, require that
procedures and standards be defined and adopted. These are
overkill for Arduino developers, but we do need to adopt rules. I call
this good programming practices. These were touched upon in my
first book Pragmatic C++ Arduino Programming. They are revisited
and expanded upon in this chapter:

Be consistent (page
<x>Good programming practices (GPP)[AUX]Be consistent, particularly when naming files, functions, etc.</x>

132) - Whatever you do, do it consistently. This will save you

time and will avoid bugs. For example, if you decide that
functions start in upper-case and variables in lower-case, stick
to it. You will thereby be less prone to using function names as
variables and vice versa.
<x>Bugs (avoiding them)[AUX]Apply good programming practices and adhere to Golden rules - check data, never assume anything; do error handling</x>

Naming conventions (page 132) - Names should reflect what a

variable or function does. Lower-case/upper-case should
differentiate variables from functions and other entities. You
may want to use prefixes to differentiate functions from
classes, from enum declarations. You should define your own
standards and use them consistently. It will improve readability,
maintenance, and will save you time.
<x>Good programming practices (GPP)[AUX]Indentation, code collapsing not cosmetic</x><x>AtmelStudio (editor)[AUX]Indentation, code collapsing not

cosmetic</x><x>Arduino (editor)[AUX]Indentation, code collapsing not cosmetic</x> Code formatting (page

 133) means make code readable, verify indentation, and
collapse it to avoid being overwhelmed by reams of code
flooding your screen.

Readability - let your code reflect what it is supposed to do.

Indentation may seem to be a cosmetic attribute for source code.
No! It is a debugging tool. Use the editor to indent often and look at
the result.
Collapsing code unclutters your viewing area, display what you are
concentrating on, hide the rest.

<x>Good programming practices (GPP)[AUX]Plan work offline, think before typing; thinking hardest thing to do</x><x>Think[AUX]Thinking hardest thing to do -

Think (page 135) - One cannot repeat it often enough.

requires effort</x>

Thinking is the hardest thing to do. It is so easy to embark on a

long journey without paying due attention to potential
alternatives. This is where smarts come in. The better your
thinking, the better the code, but be careful not to outsmart
yourself.
<x>Good programming practices (GPP)[AUX]Countless reasons for commenting code</x><x>Comments[AUX]Countless reasons for commenting code</x>

Document code (page 136) - There are countless reasons why

you should comment your code. The first one is that when you
come back to a section of code months after having written it,
your comments will be a life raft because you will have
forgotten just about everything you have done. Furthermore, if
somebody else maintains your code, he/she will be immensely
grateful to you upon finding detailed explanations on what you
have done. With a little extra work, comments may be
formatted so that they could be extracted via a Perl or Awk
program to produce documentation.
<x>Think[AUX]Planning offline before coding can save loads of time</x> Plan your work offline (page

137) - I have said it before and shall repeat it: Given today's
incredibly easy and fast access to computer resources, it is
easy to yield to the temptation of sitting down at the computer
and coding right away without giving the problem due
consideration. The result will often be to back track and start
over from a new angle - wasted time.
<x>Golden rules[AUX]Validate function parameters and return values; never assume anything</x><x>Good programming practices (GPP)[AUX]Adhere to the

'Golden rules' (see above)</x><x>Good programming practices (GPP)[AUX]Don't neglect validity checks and error handling - Check, Check, Check!</x>

Never assume anything (page 137) - Check, Check, Check! -

Validity checks increase an application's robustness. From the
outset, perform validity checks on function parameters and
return values. Do error handling if there is a problem and print
values. Detecting faulty parameters early will save you ages of
debugging time. When declaring a variable, always, I mean
ALWAYS, initialize it, to zero or something else. Do not
assume that the compiler will initialize things for you. Variable
initializations will improve your algorithm, debugging,
readability, and understanding. You should also validate
function parameters before using them. Error handling and
developing algorithms are distinct tasks. Always start coding
functions with checks on parameter values - Are they within
acceptable ranges?
<x>Validate data[AUX]Apply good programming practices and adhere to Golden rules - check data, never assume anything; do error handling</x> Error

handling (page 138) - As the application unravels, data

validation will uncover errors, which brings up several issues:
What should the application do when an error gets detected?
Should the application continue? Abort? Plug in a default
value? What about reporting the error? The developer or the
user needs to be informed of the error's occurrence, location,
and cause. I cannot emphasize it enough: error detection and
error handling are crucial to an application's robustness. Since
it is logic developed alongside the algorithm, its status tends to
be relegated to a TO DO list. No! Do it as you code.
<x>Good programming practices (GPP)[AUX]Organize your code as separate files, .ino file should be small</x><x>Project Files Framework[AUX]Common sense

Project organization (page 139) - The

dictates how to organize an application's source code</x>

Arduino IDE saves your work in an .ino sketch file (a C++ file).
For small apps this is good enough; it will contain your classes,
functions, variables, and whatever is needed by the
application. However, as the application expands, you will
need to place your code inside separate files. Create a file for
macros (Macros.h), a file pair for globals (Globals.h/.cpp), the
.ino file which contains setup and loop, and separate file pairs for
classes (.h/.cpp). Doing it all inside the .ino file will overwhelm
you and lead to situations difficult to manage.
<x>Good programming practices (GPP)[AUX]Monitor memory - prevent stack overflow and lack of heap space</x><x>Memory (use)[AUX]Memory allocation

failure avoidable with memory monitoring functions</x> Monitor memory use (page 139) to
proactively prevent stack overflow and insufficient heap space.
Understand where your program is consuming precious RAM.
Beware of memory leaks (failure to release memory).
Careful with lenient compiler type checking (page 140) - When
passing parameters as integers, char pointers, or floats, the
compiler is permissive. It will let integers pass instead of floats
and much more; it is extremely lenient.
<x>Golden rules[AUX]KISS principle - simple solutions often better than complex ones</x> Abide by the KISS

principle (page 140) (keep it simple stupid) - It is often so

comfortable to embark on a long, complicated journey when a
simple solution could have done the job just as well. This is the
EGO trap - think twice or try to imagine several ways of
handling a specific problem. You will often find that your first
approach may be intellectual overkill.
<x>Good programming practices (GPP)[AUX]Wrap-up means stay focused 'til the task is completed</x><x>Wrap-up phase[AUX]Adhere to good programming

Task wrap-up phase (page 141) - When

practices; stay focused 'til the task is completed</x>

doing a task, one’s concentration often wanes just about when

one is about to finish the task. Many mistakes are made at this
point in time. As you are just about to finish a task, make it a
habit to tell yourself that this is when most mistakes are made.
<x>Golden rules[AUX]Good mental condition crucial to good work</x> Mental condition (page 143) - As

you are writing code, your brain is operating on two-levels: the

autopilot level and the conscious level. You can improve your
chances of error free coding by giving your unconscious level
ample room to do its job correctly: take your time, pull back,
and stop regularly to think things out. This will improve your
productivity. Do not rush. Listen to what your body is telling
you.
Use C++ macros (page 143) - C++ provides a preprocessor
which enables defining macros. These simplify coding and
improve program robustness.
Define constants only
<x>Constants[AUX]#define or const variables to define constants; centralize their location</x>

once (page 144) - Do not hard code constants inside code.

Instead, use a #define macro or const variable, located in
Globals.h/.cpp files or other.
<x>Good programming practices (GPP)[AUX]Use #define macros, const variables, enums instead of hard coding values</x><x>enums[AUX]enums better than

Use Use enum lists (page 145) (symbolic meaningful lists)

#define macros</x>

instead of numeric values to index into arrays. They clarify code.

Parameter default initialization (page 146) means that you can call
a function with fewer parameters than the ones defined. This,
combined with lenient type checking, can be the source of extremely
insidious bugs.
<x>Good programming practices (GPP)[AUX]Initialize variables, whether local or global, preferably with curly braces</x><x>Curly braces[AUX]Initialize using

Initialize using curly braces (page 146) to futureproof

curly braces</x>

your application. Given that there are three ways to initialize

variables when declaring them: assignment via an equal sign,
parentheses, and curly braces, there are good reasons for
using curly braces such as int x{0}. This is explained in the
companion book Pragmatic C++ Arduino Programming.
<x>Good programming practices (GPP)[AUX]Comment closing curly braces and #endif</x><x>Curly braces[AUX]Comment closing curly braces and #endif</x>

<x>Macros[AUX]Comment closing curly braces and #endif</x> Comment closing curly braces and
#endif (page 147) to facilitate untangling misplaced or missing
opening or closing curly brace or #endif.
Use the auto-indent feature of the editor (page 147) as you
code. You will thereby be reassured that curly braces are not
missing and that they are properly placed.
<x>Good programming practices (GPP)[AUX]Start application with code skeletons</x> Always start with code

skeletons (page 147) - An application consists of files which

contain related items and functions which do specific work,
whether they be global or inside classes. The programmer will
save time by copy-pasting code skeletons when creating a
new project or writing a new class or function.
<x>Good programming practices (GPP)[AUX]Exploit C++ features sparingly</x><x>Golden rules[AUX]Do not fall into the EGO trap</x><x>EGO

Exploit C++ features sparingly (page 147) -

trap[AUX]Exploit C++ features sparingly</x>

Adapt C++ coding to your skills - do you really need lambda

 functions, templates, bit-level coding, multiple inheritance,
virtual functions, pragmas? Do not fall into the EGO trap. Just
because you think that inheritance is really cool, do not use it
unless you are sure that it is the correct tool for the job at
hand. The mind is subtle. The subconscious will push you into
doing things you should not do.
<x>Golden rules[AUX]Do your homework, know your tools, update your C++ skills, do not reinvent the wheel</x> Do your

homework (page 148); this is a question of discipline. It is

much too easy to succumb to haste, the desire to get the job
done, to the exhilaration of coding and getting results, without
doing absolutely necessary preliminary work. Before you start
a major task you should ensure that (1) someone has not
already brought a solution to your endeavor; (2) that you
master the tools you are working with; and (3) that your C++
skills are up to par considering what you envision doing.

This list of good programming practices is not exhaustive. Adhering

to these practices is an excellent start. They will save you lots of time
and reduce debugging by helping write well organized properly
validated code. The sections which follow describe these in greater
detail.

1.
Be consistent
<x>Good programming practices (GPP)[AUX]Be Adopting a set of
consistent, particularly when naming files, functions, etc.</x>

procedures and submitting oneself to these consistently contributes

to your being more efficient as you write code. A good example
would be how one writes variables and functions. If you adhere to
variables being written with a leading lowercase letter and functions
with a leading uppercase letter, and do so consistently, you will
always know, when you look at a name, that it is either a variable or
a function. You will avoid making the common mistake of assigning a
function to variable which is perfectly legal in C. When doing so, you
assign the function’s address to the variable, which is not at all what
you intended doing; the compiler will not complain. This is a typical
gotcha which you can avoid by being consistent in your naming
conventions. As you perused through the C++ gotchas in the
companion book Pragmatic C++ Arduino Programming, you would
have come across several gotchas which could have been avoided
had you been consistent in your way of writing code.

2.
Naming conventions
<x>Good programming practices (GPP)[AUX]Be consistent, When I started
particularly when naming files, functions, etc.</x>

programming with C, my first reference books were Kernighan and

Richie's The C Programming Language, first edition, and of course
later on, when C++ came into being, I poured over Bjarne
Stroustrup's The C++ Programming Language, 2nd edition. As I read
the books, I was struck by their programming style: lower-case
letters for all variables and functions, cryptic names (p for pointer, s
for char string, v or val for value). It was difficult going. I decided to adopt
rules which would make life easier:

Names - Name sizes have no impact on RAM; so, do not

hesitate to write long compound meaningful names, for
example: value instead of val or v, firstOccurence instead of f, etc.
When defining a string value do not do String s1{""}, instead write
String myString1{""}. The name should explicitly reflect what the
item does, for example: ValidateEepromID(). Also, capitalize the first
letter of the parts which make up a compound name such as
GetNumberOfChars(). Underscores may be used instead of
capitalizing word fragments as in Get_number_of_chars(); this is a
matter of personal preference. The issue is that you should
adopt a convention and be consistent.
Variables should start with lower-case; functions, enums, and
classes, with uppercase, for example: Joblist* joblist = new Joblist().
Notice how useful this naming convention is. The lowercase
 variable jobList is an instance of the uppercase class JobList. Neat!
Remember! C/C++ is case sensitive.
Parameters start with an underscore. For example, in the
function int MyFunction(char _delimiter), the underscore clearly
differentiates the parameter from local/global variables as you
use it inside the function.
Functions and other constructs such as struct, class, enum, union
start with an upper-case. Code such as Record nextRecord =
NextRecord() clearly differentiates variables from functions and
clarifies logic.
Macros are systematically written in upper-case with
underscores to facilitate reading. Here are a couple of
examples:
#define XBEE_SERIAL_RX 9 // Serial receive port number
#define XBEE_SERIAL_TX 10 // Serial transmit port number

Maintaining a consistent naming style improves program readability,

facilitates program maintenance, reduces debugging, and improves
robustness.

3.
Code formatting
Adhering to a few cosmetic
<x>Good programming practices (GPP)[AUX]Indentation, code collapsing not cosmetic</x>

details can make a significant difference in your productivity. Being

careful with code formatting such as readability, indentation, and
code collapsing make programs more readable and less error prone.
These are described below:

1.
Readability
Small details such as where and when to include spaces are
important. Get used to doing things in a consistent way. For
example, write char* myString instead of char *myString. In the first case you
are clearly saying that myString is a char*. The second case is not clear;
you are implying that the dereferenced *myString is a char - wrong!
As part of Good programming practices (page 129), make names
meaningful, not cryptic. Write GetNext() instead of Gnxt().
Also, use spaces intelligently. When writing lists (parameters,
enums...), put a space after the comma, do not compact them.

2.
Indentation
<x>AtmelStudio (editor)[AUX]Collapses classes, functions, enums, ifs, whiles, etc. and #ifdef...#endif pairs</x><x>Curly braces[AUX]Extra curly braces facilitate code

collapsing</x><x>Curly braces[AUX]AtmelStudio does not collapse curly braces; Arduino IDE does</x><x>Arduino (editor)[AUX]Collapses curly braces, not AtmelStudio</x>

Besides making source code

<x>Macros[AUX]AtmelStudio collapses #ifdef; extra curly braces facilitate cross platform code collapsing</x>

look good and organized, indentation has a real programming purpose; it

reflects scope hierarchy. One could think in terms of level 0 scope (global
scope), level 1 scope (a function), level 2 scope, and so on. This
means that a variable defined in a block at scope level 1 is visible
from enclosed blocks at scope 2 and lower.
Indentation serves another purpose; it helps you discover errors.
Misplaced braces and parentheses will be revealed by suspicious
left margin offsets. At best, the error will be caught by the compiler
which reveals where the problem lies. At worst, the braces/parens
are balanced but misplaced. The program compiles error free but
does not function correctly. You wonder why.
The example below illustrates how correct indentation can improve
readability and how misplaced indentation can reveal errors. Here is
an example of nested statement blocks:
|Foo() { // scope level 0 // global
| int x{0}; // foo wide scope
| ...
| if ( x == 0) {
| int y{0}: // if wide scope
| ...
| x = ...;
 | } else {// if
| ... Do something else
| } // if ( x == 0)
| x++;
| ... continue
|} // Foo()
x is visible in the if and else blocks but not y, which is visible only in the
if block, not in the else block.
The Foo() function defined above contains three scope levels, two
indentation spaces per level, thus:

Level 0 against the left margin. This is global level. Functions

have global scope unless they are defined inside a class, in
which case they have class level scope.
Level 1 is 2 spaces in, this is the function's body level.
Level 2 is 4 spaces in, these are the if and else content levels.

If curly braces were to be improperly positioned, the application

could well compile error free but would not run correctly. Examining
for correct indentation thus becomes an important diagnostic tool.
Throw in a couple of mistakes and see what happens:
|Foo() {
| int x{0};
| ...
| if ( x == 0) {
| int y{0}:
| ...
| x = ...;
| } else {// if
| ... Do something else
| x++; // mistake 1 - x++ should be after the else's closing curly brace
| } // else
|} // mistake 2 - one too many closing curly braces
|... continue
|} // Foo()
Notice how automatic indentation has revealed that there are
problems:

x++ is
indented too far in but it looks fine because it has been
misplaced. It should be positioned after the else closing curly
brace, not before. The x++ indentation level reveals the
 mistake.
There are two closing curly braces against the left margin at
the end of the function definition. This is a dead giveaway that
there is an extra closing curly brace. The question is: Which of
the closing curly braces is the extraneous one? You would
quickly know if you got into the habit of commenting closing
curly braces and #endifs.

Automatic indentation is a precious tool to uncover programming

mistakes.
The Arduino IDE is good at indenting and collapsing code. But it
does have one failing - it does not collapse #ifdef...#endif pairs.
AtmelStudio, on the other hand, does collapse #ifdef...#endif pairs. It
does a decent indentation job but is poor at code collapsing. When
looking for curly braces positioning bugs, it is better to use the
Arduino IDE to collapse code, one of the reasons for establishing
IDE interoperability.

3.
Collapsing
When a file or function is long, you may want to isolate an area you
are working on to facilitate your work. Most editors support
collapsing code, this means make it so that you see the first line of a
section of code. For example, collapse a function, or a while
statement.
The Arduino IDE is good at code collapsing; it uses curly braces. It
will collapse classes, functions, while, if, etc. but it will not collapse
#ifdef...#endif pairs. The workaround is to add open/close curly braces to
enclose #ifdef...#endif lines of code thereby enabling the Arduino IDE to
simulate #ifdef...#endif code collapsing. AtmelStudio is good at
collapsing #ifdef...#endif but will not collapse curly braces. It will however
collapse classes, functions, enumerations, and other high-level
constructs, but will not collapse if, while, for. There will be occasions
whereby you will want to use the Arduino IDE to collapse code. Note
that code collapsing does not carry over when pasting; pasted code
gets uncollapsed.
Code collapsing, tied to indentation, is another useful feature. Code
collapsing means hiding details located within curly braces. Another
way of saying this is collapse functions, ifs, whiles, etc. The Foo()
function above, when completely collapsed, should yield:
Foo() {
Collapsing the second variant above, the one with mistakes, yields
the following:
Foo() {
... continue
} // Foo()
Code collapsing can be of major help to unearth a missing or extra
curly brace buried somewhere in the code. Note that the compiler
does a fairly decent job at locating missing ones.

4.
Think
<x>Think[AUX]Good programming practices</x><x>Good programming practices (GPP)[AUX]Plan work offline, think before typing; thinking hardest thing to do</x>

Good programming practices dictate that

<x>Think[AUX]Thinking hardest thing to do - requires effort</x>

one should focus on the problem at hand. There are times when one
can code as if one were on autopilot such as laying out a switch
statement along with the cases it should handle, or when one is
laying out a function's skeleton. But when it comes to handling
individual cases or filling a function's body with code, one should
concentrate on content, i.e., think.
I have said it in the companion book Pragmatic C++ Arduino
Programming under psychological factors: "Thinking is the hardest
thing to do". It is worth repeating that thinking things out before
letting the unconscious intuitive mind take over is one of the good
programming practices directives. It is so easy to sit down and code
right away. You have given your algorithm some attention, quickly
reached a comfort zone which translates as "Yeah: This is what I
should do" and immediately started typing away. This is incremental
programming (straight from mind to fingers).
The combined reticence to sit back and think (laziness) and the
extremely easy access to a computer generate an urge to indulge in
incremental programming. This leads inexorably to poorly thought-
out algorithms, the result being increased debugging time,
undocumented code (lack of comments), poor logic, etc., aka shoddy
work. One should recognize when one is following a path of least
resistance, i.e., when one shunts out that most important task: think
before you do.

5.
Document code
The
<x>Good programming practices (GPP)[AUX]Countless reasons for commenting code</x><x>Comments[AUX]Countless reasons for commenting code</x>

human brain is remarkable. Its capabilities far surpass our appraisal

of its potential. As we program, our mind concentrates on the job at
hand however, in the background, it keeps churning away on other
things; it is multi-tasking. As we develop an algorithm, our mind
builds a mental image of the logic. It uses this image to guide you,
whether you are drawing a flow diagram on paper or typing code. As
you are doing one or the other, you should continuously comment
what you are doing; it will help you think things out. The mental
picture gets progressively clearer. The logic gets transposed into the
flow diagram, or into code. You then test it, and when found to run
correctly, you finalize it. Memory is fresh with the newly developed
logic. Comments inserted in the code will be of precious help later on
when you come back to enhance it, adapt it to new situations. Had
you not documented what you did, much later on, you would need to
rethink and partially redevelop the algorithm.
<x>Comments[AUX]Program Documentation Framework - use Awk or Perl regexes to extract comments</x><x>Program Documentation Framework[AUX]Create program

documentation via Awk/Perl regex based comment extraction</x><x>Golden rules[AUX]Comment! Comment! Comment!</x><x>Comments[AUX]Comment! Comment!

I soon realized that I needed

Comment!</x><x>Program Documentation Framework[AUX]Comment! Comment! Comment!</x>

to add one more rule to my list of golden rules; and this was in no
uncertain way: Comment! Comment! Comment! (See the section on
golden rules in the companion book Pragmatic C++ Arduino
Programming).
 The importance of program documentation
<x>Comments[AUX]Countless reasons for commenting code</x>

can hardly be overstated. By doing two tasks at the same time (task
1: develop the algorithm and task 2: document it) programmers
enhance their understanding of the algorithm they are developing by
the shear act of explaining what the code is doing. One's insight into
the algorithm gets enhanced which leads to better code, shorter
development time, less debugging. Furthermore, since one's mind is
volatile, documentation will serve later to explain the algorithm's
inner workings. It will also be of significant help when doing
maintenance and when enhancing the application.
I was struck by the importance of commenting code properly as I
developed my Arduino-based beehive weighing system. When I
started developing it, I planned things out on paper first and then
coded. I felt that the work on paper was good enough to serve as the
application's documentation. This approach turned out to be
satisfactory at the beginning however, as things got more complex, it
quickly became apparent that I was a little lax. At first, I added
comments to describe what a function did. I did not adopt specific
rules as to what to include in comments nor as to their format. As I
progressed, I decided to formalize the way that comments were
written: what a function did, how it did it, parameters and what they
served for, validity checks, return values, functions called, etc. This
led me to define comment templates which gradually morphed into a
framework so that comments could be extracted via a Perl or Awk
program to produce part of the application's documentation. See the
Program Documentation Framework (page 155) and the Function
Creation Framework (page 157).

6.
Plan your work offline
<x>Think[AUX]Good programming practices</x><x>Good programming practices (GPP)[AUX]Plan work offline, think before typing; thinking hardest thing to do</x>

It is so easy to sit at the computer and start

<x>Think[AUX]Easy to sit at the computer and code</x>

coding away, and so gratifying. Some incredibly smart hackers, one

sees this in TV series, can undertake the most complicated tasks
directly from mind to fingers which race along typing code, not a
single mistake. I can do this for simple repetitive tasks, perhaps not
as quickly, but as soon as a task gets a little complex, I find that I can
save lots of time by sitting down at a table to lay out the algorithm as
a flow diagram on paper before doing any coding.
Upon starting a new task, begin with the big picture, leave the details
for later. These get filled in after you have tested the overall logic.
Draw flow diagrams and document your thinking. These should
clearly explain the program's architecture, underlying algorithms, and
its functionalities. After working on your algorithm for a while, you will
have a clear picture as to what you are trying to do; you may then
start coding.
My doing such planning had the benefit of forcing me to think and
thereby, once I got onto the computer, increased the likelihood that
the algorithm would work correctly. It also has the benefit of enticing
me to comment the algorithm extensively.
You might argue that this falls under the human factors category and
not in the good programming practices category. But I shall argue
that understanding how one functions and acting accordingly is part
of good programming practices.

7.
Never assume anything
A golden rule many programmers
<x>Good programming practices (GPP)[AUX]Adhere to the 'Golden rules' (see above)</x>

fail to apply, including I, is to systematically check data validity. If you

do not abide by this rule, the application will sooner or later bite you.
You should keep in mind that the cost of a programming error
(debugging and testing time) increases as development progresses.
It is a fact proven time and again; if a programming error inside a
function is discovered while the function is being developed, its cost
will be minimal. However, if it is discovered once the application has
been deployed, the cost can become exorbitant. An example: the
Ariane 5 rocket exploded during its first flight because of one line of
bad code. This implies that applications should contain validity
checks and be tested extensively - find the bug as early as possible. All
values passed to a function should be checked for coherence, as well as all
return values. Do not assume anything; this is a golden rule. If a
variable is passed to a function which in turns passes it to another
function and so on, check its coherence in every function, because
an intermediate function may modify it in an unintended way.
<x>Golden rules[AUX]Validate function parameters and return values; never assume anything</x><x>Validate data[AUX]Apply good programming practices and adhere to Golden

Always check that a parameter's value is

rules - check data, never assume anything; do error handling</x>

within acceptable bounds. If you are passing a pointer, make sure

that it is valid. If you are passing an index, make sure that it is within
bounds. There are numerous circumstances which require
verifications so that you can proceed safely. Data needs to be
validated before being used; never assume anything.
By keeping within the maxim never assume anything, validity checks
should be the norm, a vital process which favors program
robustness. You never know when you shall be bitten because you
assumed something. I should venture to say that more time is spent
doing validity checks and reporting than implementing the algorithm.
A function will either perform correctly or encounter an error which
may require interrupting its normal execution. Once an error has
been detected, decide on what to do, stop the application, or notify
and continue. Here is a typical example: Did the malloc do its job?
char *lastName = (char*) malloc (100);
if (lastName == 0) {
... handle error
}
After doing the malloc, test the value of lastName and handle the error, if
any. The above is typical. The questions are: How will you implement
data validity checking? How does the calling function handle the fact
that the called function could not do its job properly? This subject is
covered in the next section.

8.
Error handling
<x>Validate data[AUX]Apply good programming practices and adhere to Golden rules - check data, never assume anything; do error handling</x><x>Good programming practices
 Data validation has detected that a
(GPP)[AUX]Don't neglect validity checks and error handling - Check, Check, Check!</x>

parameter's value is out of bounds. But what do you do then?

Should a function return a specific value to signal that an error has
occured? Do you continue with the program's execution? Under
what conditions? Most of the time, the application should stop and
report an error. Here is a list of potential mishaps:

Is a parameter within acceptable bounds?

Is the algorithm generating correct values?
Is a function being used triggering an error?
Is a switch statement's exit via default acceptable?
Was opening the database successful?
Was a search into the database successful?
And so on. More things could go wrong.

I propose several error handling solutions:

Return a specific error value which may be tested for by the

calling function to ensure that all is well. The return value could
be a unique value which specifies that an error occurred.
Function returns true/false - Functions could be designed so that
they always return true/false to indicate whether they did their job
correctly. If an error gets encountered while the function was
doing its work, it would return false. For example, return false if it
could not open the database when looking for a name but
return true if it succeeded in opening the database but the name
searched for was not found. The first result is an error: Was the
database opened? The second result is normal: Was the name
searched for found? Not opening the database is a major error
which could interrupt the program's flow. Not finding the name
may be a normal condition, such as inserting a new contact
instead of modifying an existing contact.
Abort the application via an error handling function is a clean
way of handling errors. The function may print things on the
console, send a message to the programmer, or convey info
 via other means such as an SMS.
Using C++'s exception handling rolls the application back to a
predefined landing point (when an error occurs, it does a throw).
It is a powerful error handling mechanism. Unfortunately, it is
unavailable in the Arduino environment. The alternative
consists in doing a long jump back to a predefined landing
point by using the longjmp/setjump mechanism. There is an
important advantage to using this solution over using an error
handling function which aborts the application: The destination
of the long jump can be tailored to continue execution
according to the type of error. See Pseudo Exception Handling
Framework (page 183).

In a nutshell, if a validity check reports NOT OK, the problem must

be handled in some way. It is up to the programmer to decide on
what to do next.

9.
Project organization
<x>Good programming practices (GPP)[AUX]Organize your code as separate files, .ino file should be small</x><x>Project Files Framework[AUX]Common sense dictates how to

An application resembles a textbook; it is organized

organize an application's source code</x>

as chapters, sections, subsections, etc. There may be an

introduction, possibly an appendix. Chapters cover a specific
subject, which is why there is a table of contents. An application's
organization follows along the same pattern except that a
programmer does not usually create a group of files before
beginning to write code. The usual process is to start small, in one
file, the .ino file, and then expand into other files as needed. This
type of development can become chaotic. As development
progresses, files get reorganized constantly. It would have been
better to organize files right from the beginning based on the kind of
contents they will hold. At first, they will be empty but, as
development progresses, they will slowly but surely receive content.
There will be files for classes, for helper functions, for macro
definitions, for global values, a main application entry file, and
others.
Imagine a chest of drawers; each drawer, properly labeled, should
contain items in accord with what the drawer's label refers to. Your
project should be organized the same way. Create all the pertinent
files before writing a single line of code. It goes without saying that
code should be documented, preferably in such a way that an Awk
or Perl program be able to pull out comments to serve as application
documentation. A skeleton project with all the files would help you
get started. See Project Files Framework (page 152) and Always
start with code skeletons (page 147).

10.
Monitor memory use
<x>Memory (use)[AUX]Memory allocation failure avoidable with memory monitoring functions</x><x>Good programming practices (GPP)[AUX]Monitor memory - prevent stack

overflow and lack of heap space</x><x>Memory (use)[AUX]Total heap is fragmented plus contiguous</x><x>Memory (use)[AUX]Understand memory structure to optimize

memory use</x><x>Memory (stack)[AUX]Stack overflow thrashes memory allocations, may go undetected a long time</x><x>Memory (leaks)[AUX]Could be a cause of lack of

You have tested your

memory, depletes available memory slowly but surely, do not manifest themselves during testing phase</x>

application under multiple test conditions and found it to run well.

You have reached a confidence level which reassures you as to its
robustness. But under real conditions, an urchin may pop up and
cause it to fail: You ran out of memory. This can happen in one of
two ways:

Memory allocation failure - Such an event is easy to detect, just

check whether the malloc was successful; the pointer to which you
assigned the malloc's return should be non-zero. But this is after
the fact; your application cannot continue doing its job; it is
suddenly forced to stop. It would have been a lot better to
monitor available memory so that the application could
undertake some preventive measures prior to a memory
allocation failure. You may have to redesign the application so
that it consumes less RAM or run it in a microcontroller with
more RAM. Whatever you do, checking that allocations are
 successful prevents the application from continuing on a bad
footing.
Stack overflow is a more insidious bug because the application
may thrash the top of allocated RAM, which will not necessarily
manifest itself right away. As functions get called, the system
allocates memory for stack frames down from the top of
unallocated memory. As stack frames get added downwards,
they may well extend into allocated memory. The application
may chug along for a while, it could be weeks or months,
before the damaged zone gets used. It is only at this point in
time that you are alerted as to their being a bug. But you are
clueless as to the cause. There is no magical mechanism such
as checking that a pointer is non-zero to prevent stack
overflow. The only solution is to monitor memory regularly and,
just as with allocations, undertake preventive measures to
prevent damage. The ultimate cause could be too much
allocated memory (could there be memory hogs, aka memory
leaks?) or too many nested function calls.

One of the objectives of good programming practices is to help the

programmer be less prone to making mistakes. Another objective is
to incite the programmer to design his application in ways that it
should become more robust. The companion book Pragmatic C++
Arduino Programming covers memory management extensively. It
explains memory structure, how to measure both contiguous and
fragmented memory, how to detect memory leaks, and what to do to
optimize memory use.

11.
Careful with lenient compiler type
checking
Do not rely on the
<x>Bugs (avoiding them)[AUX]Careful with compiler function parameter type leniency and default initializations</x>

compiler's function parameters type checking - the compiler will not

differentiate and int from a char* function parameter from a type
checking perspective. Be careful. There is only one way of avoiding
these errors. Double check your work. Examine declarations,
implementations, and use. Here is an example: from a strictly C legal
syntax perspective, you can write a function prototype as follows:
bool MyFunction(uint8_t, uint8_t, uint16_t);
This will compile but, when looking at the prototype, it does not tell
you much. You will be much better off by adding meaningful labels to
the parameters such as;
bool MyFunction(uint8_t _index, uint8_t _count, uint16_t _value);
Unfortunately, you cannot add a label to the return value such as:
bool _success MyFunction(uint8_t _index, uint8_t _count, uint16_t _value); // ERROR
It will not compile. But you can insert a C-style comment as follows:
bool /*_success*/ MyFunction(uint8_t _index, uint8_t _count, uint16_t _value);
Doing this systematically clarifies what the function does.

12.
Abide by the KISS principle
<x>Golden rules[AUX]Da Vinci: "Perfection lies in details, but perfection is not a detail"</x><x>Golden rules[AUX]KISS principle - simple solutions often better than complex

Da Vinci said "Perfection lies in details, but perfection is not a

ones</x>

detail". As one programs, the desire for excellence can be a strong

motivating factor, but it can be a false friend. In trying to do very
good work, you may complicate things, you may get caught in some
EGO trip and thereby indulge in some unnecessarily complicated
work (see Exploit C++ features sparingly page 147). A simpler
solution lies at your fingertips, but no, you plunge ahead and spend
hours laying down your convoluted thinking into code.
It is a given that the probability of failure increases with the numbers
of components the system has. The more complex a machine, the
more there are parts in the machine, the higher the likelihood a part
will fail. It is the same with programming. As code size increases, the
more if statements, while loops, switches, functions, classes, etc., the
higher the likelihood of mistakes. Reducing code size, aka
simplifying the system as much as feasible, should be a priority.
Abiding by the KISS principle requires that one should analyze the
problem along several paths. If you plunge ahead with your first idea
and fail to consider other approaches, you could have created an
unnecessarily complex solution containing many potential failure
points. Had you persisted, you might have found a simpler, more
robust solution.
The underlying purpose of the KISS directive is to make the
application robust - unnecessary complexity generates bugs. Abiding
by this rule may cost you time during the design phase, but you will
save lots of debugging time because your application will be less
error prone and more robust.

13.
Task wrap-up phase
<x>Good programming practices (GPP)[AUX]Wrap-up means stay focused 'til the task is completed</x><x>Wrap-up phase[AUX]Adhere to good programming practices; stay

Programmers generally focus on one task at a time:

focused 'til the task is completed</x>

handle an error condition, cycle through a series of items, create a

function, handle multiple situations via a switch statement, etc.
Undertaking each one of these requires focus, concentration. As one
gets close to finishing a task, one's concentration wanes, thereby
being less diligent with the task's wrap-up phase. By wrap-up phase,
I mean details which complete the task worked on, such as including
the function's closing curly braces, inserting the last return, or
managing a switch's default.
I shall illustrate the above with a function such as int Foo(int param1){ ... }.
It will undertake calculations, do a database lookup, call other
functions, whatever. It is designed to return an int value. It sounds
simple, but there are times when the function's innards are such that
mistakes are made during the wrap-up phase. Consider the following
example which contains three wrap-up omissions:
int Foo( int param1) {
...
if (a > b) {
[Link](F("Error: a > b"));
return; // error 1 - should return something
} // if
 switch(x) {
case 1:
return 5;
break;
case 3:
return 7;
break;
default: // error 2 - default should always contain code
break;
} // switch
} // Foo() // error 3 - functions should always have a last return, even void ones
This example contains two return errors and one switch default error.
<x>Think[AUX]Linear thinking means focus and follow a path; peripheral thinking means let your mind loose, let it dwell around a subject</x>The first error

concerns the return after the validity check if(a > b). As written, if the
validity check detects a failure, the function returns without
specifying what it is returning. It thus returns anything. The
programmer focused on the validity check task: handle the error. As
soon as he/she finished the handle the error task, concentration
waned, and he/she thereby omitted to specify what to return. This is
a human factor type of error. The programmer concentrates on
dealing with handling an error condition. Mental momentum drives
the mind along a path. I call it linear thinking (focus, think one step at
a time, and follow a path). When the programmer finishes handling
the task, he/she takes on another task, forgetting to specify a return
value because the return is not part of the task. Call this the wrap-up
phase of coding which requires peripheral thinking (let your mind
loose, let it dwell around a subject). The mind should have multi-
tasked along parallel paths (linear and peripheral thinking). This kind
of human factor error can be dealt with given a little discipline. As
soon as the programmer decided that a > b should be checked,
he/she should implement a skeleton error handling code , such as
if (a > b) {
...
... handle error
return 0;
} // if
When a programmer codes a complex construct such as an if, switch,
function, or other, had he/she started with a skeleton code, the error
would have been avoided.
The second error concerns the switch statement. The default case
should contain code, either to process a normal condition, or to
signal an error because the switch should never exit via the default case.
Again, the programmer was intent on coding the several case
statements, the default being an afterthought. The programmer did use
a snippet which included the default, but neglected to write code for it.
His/Her focus was on code the cases. Having finished with them, the
programmer neglected the switch's default, which is part of the wrap-up
phase - human factor again.
The third error concerns the missing return, located at the end of the
function, just before the Foo function's closing curly brace. The
programmer was sure that the function would always exit via the
validity check or via one of the switch cases, thereby neglecting the
bottom return. There should always be a return just prior to a function's
closing curly brace despite the fact that the function should never
return via the bottom. If the function should never exit via the bottom,
error code should nevertheless be inserted just in case it does.
Omitting such a safety feature can lead to glitches difficult to detect.
The programmer should have started by defining a skeleton Foo()
function such as:
int Foo(int param1) {
...
return 0; // good - the function exits with something
} // Foo()
The
<x>Wrap-up phase[AUX]Code skeletons help ensure items not forgotten</x><x>Code skeletons[AUX]When creating functions, templates ensure items not forgotten</x>

examples above illustrate how relaxing one's attention when the end
is in sight leads to omissions during the code's wrap-up phase. A
programmer may avoid committing such mistakes by starting with
code skeletons prior to working on details. By doing this, the wrap-up
phase of task creation is taken care of right from the start. Most good
IDEs provide code snippets to handle this. However, coding tasks,
such as writing code for the switch's default case above, require practice
and discipline so that the wrap-up phase should not be neglected.
When creating functions, you may want to use a particularly
complete function code skeleton such as I propose in the Function
Creation Framework (page 157). It contains comment templates to
ease function documentation.
 14.
Mental condition
Make sure that your mental
<x>Think[AUX]Good mental condition crucial to good work; do not rush, take breaks</x>

condition is up to par given the job at hand, thereby reducing the

likelihood that your subconscious should play tricks on you. Thinking
is a complex process; it combines conscious mental gymnastics
such as finding an answer to the question "What happens if I do
this?" with subconscious mental gymnastics such as writing correct
C++ syntax as you type code. If you are tired, stressed, close to
burnout, hungry, cold, whatever, it is a sure bet that your mind will
perform poorly and that your work will suffer from it.
Creating code is similar to speaking: your brain picks out the right
words and assembles them correctly without your thinking about
them. The idea, the concept you are trying to convey, requires that
you give it hard thought. When coding, your mind is intuitively
making sure that syntax is correct (such as == instead of = in an if)
however, you are also conscientiously focusing on the logic.
Engaging in a sports competition such as tennis is exactly the same.
Your hitting the ball is a purely unconscious process developed
through years of training. As you play, the conscious part of your
brain is analyzing your opponent's strategy, strengths, and
weaknesses, which leads you into deciding where to drive the ball to
and how to hit it (forward slash for speed or back slash for short
returns).
You should recognize that your mental condition is crucial to doing
good work. As you code, there comes a time when you realize that
you should take a rest, take a walk, do whatever it takes to restore
your condition to an acceptable level. Being in a hurry to finish, lack
of concentration because your focus is somewhere else, working late
hours, having taken just a tad too much alcohol, being depressed...
are factors which will impair the quality of your work. Be on the alert.
Do whatever it takes so that you should be in top condition, just like
an athlete who is doing a competition, or an airplane pilot who must
ensure that if an emergency arises, he/she would be in top shape to
ensure that he/she would respond quickly and correctly. Coding
successfully depends on human factors (Psychology 101). The
better your mental condition, the better your performance. Because
of this, I have devoted an entire chapter to this subject in the
companion book Pragmatic C++ Arduino Programming.
Being aware of one's mental condition is definitely part of good
programming practices.

15.
Use C++ macros
C++ is one of the rare
<x>Bjarne Stroustrup[AUX]Suggests not using macros</x><x>Macros[AUX]Bjarne Stroustrup suggests no macros</x>

languages which provides a preprocessor. It is a simple but

particularly useful text replacement and conditional inclusion tool. If
you find that you repeatedly rewrite certain lines of code, replace the
few lines of repeat code with an appropriate macro, or create a
function.
Bjarne Stroustrup says, in his excellent The C++ Programming
Language, 4th edition: the first rule about macros is: do not use
macros unless you have to. He further says that almost every macro
demonstrates a flaw in the programming language, in the program,
or with the programmer. You get the gist of it. If you use macros
extensively, according to Bjarne, you are not programming correctly.
I must be a bad programmer because I use macros extensively; but,
I would argue that using them intelligently, for good purpose, is good
programming.
<x>Macros[AUX]Preprocessor does text/replace via #define and conditional inclusions via #ifdef</x>When you click on build in

your IDE to compile/link your application, macros get processed by

the preprocessor. It is the first tool in the toolchain to be invoked. It is
a plain text/replace and conditional inclusion tool which is
characterized by two fundamental features:

Text replacement macros - For every occurrence of a macro, it

will replace the macro with the corresponding text which it
defines. For example, suppose you have defined the macro
 #define BUFFER_SIZE 256,
every time the preprocessor encounters
the macro BUFFER_SIZE, it will replace it with 256. It can also be
used to avoid repeating lines of code when doing print-based
debugging; you are, in effect, inlining.
Conditional inclusion macros such as #ifdef...#endif pairs enable
code to be selectively included based on a macro's having
been defined or not. It is used to include header files only once
and to selectively include parts of code according to some
criterium.
The #include macro pastes the pointed to file's source code.

Using macros diligently can be a boon to productivity, hence part of

good programming practices. It improves readability, reduces
programming time, enhances robustness. The companion book
Pragmatic C++ Arduino Programming covers creating and using
macros extensively.
You will find an example on using macros for debugging purposes in
the Appendix (Print-based Debugging Framework (appendix) page
285).

16.
Define constants only once
<x>Constants[AUX]#define or const variables to define constants; centralize their location</x><x>Good programming practices (GPP)[AUX]Use #define macros, const variables,

Do not hard code anything

enums instead of hard coding values</x><x>Macros[AUX]Is const instead of #define better choice?</x>

directly into a statement; use #defines or const variables. For example,

#define BUFFER_SIZE 256 or const uint16_t bufferSize{256}. There will be times
when one is tempted to hard code some values directly. The
following would be a typical example:
WriteDataToEEPROM(0, name, 50);
In the above, name shall be recorded at offset 0 in the micro
controller's EEPROM. Its offset and length are hard coded at 0 and 50
respectively. Doing such hard coding should be avoided, but it does
happen. As coding progresses, name has always been the first item in
EEPROM, of length 50, but new requirements impose that something
else be placed as the first item in EEPROM. Now, suppose that you
want to write category's value in EEPROM. Somewhere in your code,
you will write
WriteDataToEEPROM(0, category, 10);
If this second line of code is physically near the first line of code
above, you will catch that there is a conflict and modify the first line
accordingly. However, if the two lines are physically distant from
each other, the category line will conflict with the name line. Obviously,
two data values cannot occupy the same space at the same time.
This type of bug can be avoided by using appropriate const variables
or #defines, such as:
#define NAME_LENGTH 50 or const uint8_t nameLength{50};
#define NAME_OFFSET 0 or const uint8_t nameOffset{0};
A simple rule is to get into the habit of defining quantities once only.
Use const variables or macros whenever you need to use numbers or
other to be used repeatedly throughout the application.

17.
Use enum lists
<x>enums[AUX]enums better than #define macros</x><x>enums[AUX]enums synchronize multiple arrays</x><x>Good programming practices (GPP)[AUX]enums more flexible

Few programming
than #define macros</x><x>Constants[AUX]#define or const variables to define constants; centralize their location</x>

languages provide features such as C+'s enum. These prove to be

extremely useful but, since they may be considered an advanced
feature, I suspect that most newbie C++ programmers do not use
them. They create #define macros instead, not realizing that enum lists
can be considerably more practical. I propose that using them
knowingly is part of good programming practices. The companion
book Pragmatic C++ Arduino Programming covers these extensively.
Here is a brief enum use example.
Suppose that the application needs to manage a set of different
fruits, such as apples, oranges, and grapes. Define a fruits enum list
such as:
enum class EFruit: uint8_t {
start = 0, apple = 0, orange, grape, ..., end
}; // EFruit
Array items can be accessed via enum indices (array
synchronization). Consider the following fruitLabels array:
char* fruitLabels[] = {
"Apple", "Orange", "Grape", ...
}; // fruitLabels
More such arrays could be defined, each array item specifying some
feature of a specific fruit. To display a fruit name, using the
appropriate enum makes the code explicit (uint8_t cast required):
[Link]( fruitLabels[(uint8_t)EFruit::orange]);
All arrays managed by EFruit would be synchronized by the enum list
defined above.
enum definitions contain several features which enhances their
practicality:

class
<x>enums[AUX]class in enum declaration is scope specifier</x> - This keyword enables enum items
to be tied to a specific scope, the EFruit enum list. Thus, an enum
item must be scoped to the corresponding class, such as is done
in EFruit::orange. Please note that the class keyword in the enum
definition list above is not the same class keyword that you
would use when defining a class. They are distinct constructs.
<x>enums[AUX]Specifier defines size of individual enum items</x>: uint8_t - This size specifier construct

specifies that the enum items be sized as uint8_t, the default type
being an int, two bytes long. There is no need to waste bytes
when defining short enum lists. Remember that behind the
scenes, enums are numeric values.
<x>enums[AUX]enum lists begin with start, finish with end</x>start = 0 and end - Since enums transpose

as a list of numeric values, start = 0 sets the value of the

beginning of the list, end sets the total number of items in the
list. The values of the individual enums sequence are
incremented by one with apple set to zero to render the list zero
based. This mechanism enables cycling through array items
synchronized with the enum list such as:
for ( int i = (int)EFruit::start; i < (int)EFruit::end; i++ ) {
[Link]( fruitLabels[i] );
}
If more fruits were added to the list, or if the list were to be shuffled
around, there would be no need to modify the for statement above.
The above is a short introduction on enum lists; I cannot imagine any
significant application not using them. They are an important tool
which constitutes part of good programming practices.

18.
Parameter default initialization
Parameter default initialization - Because default initializations
enable you to call a function with fewer parameters than specified in
the function declaration, proceed with caution. This feature is useful,
but it can play tricks on you, particularly when combined with poor
type checking as illustrated in Careful with lenient compiler type
checking (page ١٤٠). Here is an example of a dangerous default
parameter initialization.
bool MyFunction(uint8_t _index, uint8_t _count, uint16_t _value = 0);
The last parameter (_value) is initialized to 0 if only two parameters are
handed to the function. Suppose you called the function as follows:
success = MyFunction( index, value);
You forgot the _count parameter. The compiler will not complain since
it will accept two parameters instead of 3. And, since its type
checking will consider a uint8_t and a uint16_t to be equivalent, the
compiler will let it pass. There is no warning. The application will bug
because the function's count variable is receiving whatever happens to
be located in value and the function's value variable will be set to 0.
Nasty! Finding this bug is difficult.

19.
Initialize using curly braces
There is a golden
<x>Good programming practices (GPP)[AUX]Initialize variables, whether local or global, preferably with curly braces</x>

rule: when declaring variables, ALWAYS initialize them. There are

good reasons for doing this:
 Global variables are guaranteed to be initialized to 0 when
declared. Local variables, i.e., variables declared in functions,
do not benefit from a guaranteed value. Initialize them!
Explicit initialization makes for cleaner code. When debugging,
a variable's value can be checked against its initialization
value.
The algorithm may require that the variable be initialized.

Initializing variables can be done in one of three ways : curly braces,

parentheses, or equal sign. Some examples: char* initialValue{"Sasha"},
initialValue("Sasha"), char* initialValue = "Sasha" are equivalent. Curly braces
differentiate initializations from assignments and from function calls.
Furthermore, note that there are situations where curly braces
initialization is the only option; example - C++11 forbids using
parentheses to initialize member variables.
So, initialize with curly braces, one more good programming
practices directive.

20.
Comment closing curly braces and
#endif
<x>Good programming practices (GPP)[AUX]Comment closing curly braces and #endif</x><x>Curly braces[AUX]Comment closing curly braces and #endif</x>

pairs #ifdef...#endif
<x>Macros[AUX]Comment closing curly braces and #endif</x><x>Comments[AUX]Comment closing curly braces and #endif</x>

may at times lead to confusion or worse, to errors. When lines of

code separate the enclosing curly braces or #ifdef...#endif pair,
commenting the closing curly brace or #endif to determine whom they
belong to will prove invaluable.
Finding misplaced or missing closing curly braces or #endif can be
extremely time consuming. Misplaced ones are particularly vicious
since the compiler will not report an error; hence, the programmer
will proceed with confidence not suspecting that he/she has just
introduced a flaw in his/her logic. Systematically tagging closing curly
braces and #endif reduces such mishaps and helps uncover where the
problem lies.
All ifs, fors, and whiles closing curly braces should be tagged, as well
as the end of function and class definitions. All #endifs located tens of
lines down should also be tagged with their corresponding #ifdef.
Doing so helps the programmer locate problems, particularly when
autoformatting code reveals an indentation problem.

21.
Use the auto-indent feature of the
editor
Auto-indentation reveals improper use
<x>Bugs (avoiding them)[AUX]Editor's auto indent detects bad curly braces</x>

of parentheses and braces. The Arduino IDE is good at auto-

indenting - use it even if you are developing with AtmelStudio, since
it does not indent as well (see examples in Code formatting page
133).

22.
Always start with code skeletons
Modern
<x>Good programming practices (GPP)[AUX]Start application with code skeletons</x><x>Code skeletons[AUX]Start application with code skeletons</x>

IDEs such as AtmelStudio provide the user with code fragments from
which one can jump start; these are referred to as snippets. But the
process may be taken a few steps farther. When creating a new
application, or a new class, or a new function, it is advisable to start
from a skeleton project and skeleton code which contains much of
the infrastructure of what is being created.
I found, as I developed my Arduino-based beehive weighing system,
that function skeletons which included fill in the blanks comment
sections saved me a lot of time. Furthermore, when creating a new
application, instead of starting from scratch from an Arduino .ino file,
I started from a skeleton application which include the main functions
(setup and loop), and all the other files one needs to differentiate the
individual application components (Macros.h, Aardvark.h/.cpp,
Globals.h/.cpp, etc.). As I developed these skeletons, they quickly
morphed into frameworks: See Project Files Framework (page 152),
Program Documentation Framework (page 155), and Function
Creation Framework (page 157).

23.
Exploit C++ features sparingly
<x>Good programming practices (GPP)[AUX]Exploit C++ features sparingly</x><x>Golden rules[AUX]Do not fall into the EGO trap</x><x>EGO trap[AUX]Exploit C++ features

sparingly</x><x>EGO trap[AUX]Avoid proving to yourself that you are an absolute genius</x><x>EGO trap[AUX]Psychological factors extensively covered in companion book

Psychologists have coined an expression which I

Pragmatic C++ Arduino Programming</x>

find particularly pertinent when programming: the EGO trap. By EGO

trap they mean self-fulfillment, self-congratulatory behavior, patting
oneself on one's back, etc. Programming a complex, all
encompassing, highly generic algorithm, can be extremely satisfying;
the more so if one uses the many advanced C++ mechanisms
(multiple inheritance, lambda functions, operator overloading, etc.).
The mind is reinforced by "See how clever I am" and by "I am a very
good C++ programmer". Both are delusions because the algorithm
will probably be difficult to finalize, may turn out to be bug ridden,
and will undoubtedly consume a great deal of time. The programmer
has fallen into an EGO trap.
This has happened to me time and again. I fell into the EGO trap
when designing an inheritance based linked list framework. I spent
hours trying to get it to work properly, a nice piece of work. It was
way overkill given the problem I was trying to solve. It turned out I
just wanted to exercise my newly acquired C++ knowhow, pure EGO
programming! I finally scrapped the work and came up with a much
simpler solution.
One should listen to one's unconscious mind. As you embark on a
long journey designing an unknowingly and unnecessarily complex
algorithm, a little background voice will progressively make itself
heard: "Hey, are you sure this is the right path?". Some can hear this
little voice relatively soon as they plod along (lucky for me, I fall in
this category); others will continue despite strong tides, heavy seas,
locking themselves inside an inexorable long journey. The outcome
may be functionally correct but the development and debugging
costs may be excessive, and the solution may lack robustness. Such
behavior can be attributed to the EGO lure. You are not
programming to solve a problem; you are programming to prove to
yourself how clever and good you are.

24.
Do your homework
<x>Golden rules[AUX]Do your homework, know your tools, update your C++ skills, do not reinvent the wheel</x><x>Good programming practices (GPP)[AUX]Do your

How often have you embarked head

homework: know your tools, update your C++ skills, do not reinvent the wheel</x>

on into doing work not having done preliminary research which could
have saved you a huge amount of time. There might be a better tool
to do what you intend to do; furthermore, part of the solution may
already exist in libraries. Exhaustively searching the Internet and
GitHub should be the first step when starting a new project. Do not
reinvent the wheel.
Having searched the Internet, the next step is that you should make
sure that you know your tools. Use C++ reasonably - define classes
but do not overdo it. Multiple inheritance, operator overloading,
lambda functions, templates, etc. may not be necessary. When
passing variables to functions, be sure that you understand the
differences amongst pass by value, pass by address, and pass by
reference.
And last but not least, you should update your C++ skills - A few
features, easy to learn, could prove highly beneficial to your
productivity and to the application. It is therefore important that you
review C++ and be aware of features you may consider advanced,
just in case one of these may prove to be a solution to the problem
you are working on.
The companion book Pragmatic C++ Arduino Programming contains
two chapters which covers C++ features you will most certainly need
(What one needs to master) and which are advanced, but which you
are unlikely to use (What one needs to be aware of).
 16.
Frameworks

Frameworks
My Arduino-based beehive weighing system is relatively complex.
From a back-of-the-envelope perspective, it <x>Frameworks[AUX]Application independent tools designed to
accomplish specific tasks</x><x>Frameworks[AUX]Three types of frameworks: organizational, data handling, specialized</x><x>Beehive weighing system[AUX]Frameworks

seemed simple. Modules radio sensor data to

required to improve beehive weighing system program</x>

a coordinator; the coordinator collates the data and sends

corresponding SMSs. Behind the scenes, when all the little details
have to be dealt with, things get rather complex, such as
synchronizing the modules, determining how to make sure that the
clocks' times are within one second of each other, and other runtime
requirements. There is also the need to parse commands sent to the
coordinator so that it should know when to weigh the beehives -
once a day or more, and when? Every 10 minutes starting at
3:00PM? Handling such details is the reason why I am up to 35 files
and 15000+ lines of code, and two years into programming the
application.
In the process of developing my Arduino-based beehive weighing
system, I had to organize code, encapsulate low level mechanisms,
manage data efficiently to save memory, handle errors, do event
reporting, etc. These led me to create a set of tools which I refer to
as frameworks. These reduce debugging; enhance the application's
robustness; improve productivity; and enable generating program
documentation. Since these frameworks are not application
dependent, I could use them again. You will find below seventeen
such tools. They are grouped into three distinct categories:
Organizational Frameworks (page 151), Data Handling Frameworks
(page 158), and Specialized Frameworks (page 173). Note that
source code for these frameworks is available from (see A note on
the book's source code page 338).
<x>Organizational Frameworks[AUX]Three frameworks help structure your files and document source code</x>Organizational

Frameworks (page 151) manage what goes into files, classes,

macros. There are three of them, as follows:

Project Files Framework (page 152) - As soon as an

application becomes nontrivial, i.e., as soon as the contents of
the one .ino sketch file gets so large that it becomes unwieldy,
code needs to be broken up into separate files to organize the
application's contents. This has led me to define nine or more
.h/.cpp file pairs. A skeleton project helps you jumpstart a new
application.
Program Documentation Framework (page 155) - I found that
each time I documented my work prior to writing code, my
coding was more efficient. Describing a function clarified my
thinking, the result being that I got the algorithm up and
running sooner than had I not documented the code.
Formatting these comments enables them to be extracted by a
Perl program to produce documentation. I have written such a
program (see Misdoings (appendix) page 293 and A note on
the book's source code page 338).
Function Creation Framework (page 157) - Functions contain
the algorithms. They do the work. When starting a new
function, I use a function template to organize its content,
including inline documentation presented as a fill-in-the-blanks
comment template to describe what the function does. This
template turned out to be a substantial time saver.

You will find more info on these frameworks in Organizational

Frameworks (appendix) (page 243).
<x>Data Handling Frameworks[AUX]Frameworks to handle data storage, data conversion, data compacting</x> Data Handling
Frameworks (page 158) store, manage, and transmit data. There are
seven of these, as follows:

Class Data Framework (page 159) - When a class contains

identically typed data items, generalized enum-based Get and Set
functions lead to easier and less error prone programming.
Data Packets Framework (page 161) - Radio transmission
requires that data be subdivided into chunks of manageable
 size. Chunks are then partitioned as packets to be individually
transmitted.
Format Driven float to byte Conversion Framework (page
162) - Sensor data storage space may be reduced by
converting float data into byte values. These are stored as
offsets from a minimum value and adjusted to account for the
required precision. Doing this can often reduce 4-byte floats to
one byte, thereby providing significant storage savings.
DataGroup Framework (page 164) - This mini database
system enables data to be stored and accessed sequentially or
via a key, for both fixed length and variable length records.
Bitfield Storage Framework (page 166) - Classes may contain
variables stored in bitfields rather than in full 8-bit sized bytes,
thereby saving space. Individual bitfield sized items can be
managed as one contiguous memory space. Data may then be
stored in EEPROM, SD card, or wherever a block of memory
to store data can be set aside.
Event Storage Framework (page 171) - A large application
may have predefined events it needs to keep track of. These
true/false conditions are stored as one contiguous multi-byte
memory space and accessed via enum based IDs.
Linked List Framework (page 172) is an inheritance-based
system for linked list like storage. Classes derived from the
base LinkedList class can be stored as in a link list.

You will find more info on these frameworks in Data Handling

Frameworks (appendix) (page 253).
<x>Specialized Frameworks[AUX]Seven framework toolkits to accomplish sundry tasks</x> Specialized Frameworks (page

173) respond to specific needs such as memory management and

error handling. There are seven of these, as follows:

Algorithm Test Framework (page 174) - Efficiently creating an

algorithm requires planning prior to coding. Logic needs to be
separated from the work being done. This leads to defining
execution paths, each of which needs to be identified and
tested against the appropriate control variables.
Class and Function Names Referencing Framework (page
176) - Meaningful reporting requires that class and function
names be displayed as opposed to displaying cryptic IDs.
When memory availability is not critical, classes and functions
may incorporate string variables to spell out their name when
used by the reporting system. But, if memory is an issue, it is
preferable to use numeric codes (enum) to identify them. These
are used by the reporting system to transpose IDs into class and
function labels.
Memory Management Framework (page 179) monitors
memory use. A log of all memory allocations and releases
reveals the application's overall memory use as it runs,
memory leaks, and excessive stack growth.
Pseudo Exception Handling Framework (page 183) enables a
long jump back to a predefined landing point. It mimics C++'s
exception handling (try/catch/throw mechanism) not available in
the Arduino environment.
Error Reporting Framework (page 185) - As mentioned
repeatedly, never assume anything; always verify. Bad values
generate warnings and errors; these are logged into a linked
list to generate reports.
new and new[] Operator overloading (page 186) and their
corresponding deletes bestows them with the possibility of
seamlessly generating memory allocation events.
Print-based Debugging Framework (page 188) - When
debugging by doing prints to view variables in the serial
monitor, the number of [Link] tend to get out of hand.
Organizing what gets printed by hierarchical #define macros
(global, file, and function level) enables to selectively choose
what gets printed, thereby minimizing memory requirements
and clarifying source code.
Misdoings (page 117) are basic programming mistakes one
can make, such as an = instead of an == in an if statement. The
 Perl [Link] program detects some potential bugs.

You will find more info on the above in Specialized Frameworks

(appendix) (page 272).
These frameworks require a little preparation. The rewards are
improved productivity, reduced debugging, and increased application
robustness.

1.
Organizational Frameworks
<x>Organizational Frameworks[AUX]Three By Organizational
frameworks help structure your files and document source code</x>

Frameworks I mean formal procedures to organize code over and

beyond good programming practices - three frameworks are
presented within this context:

Project Files Framework (next section) - Organizing files (what

goes into which file) is the first phase in getting a project
organized. Create .h/.cpp file groups, one for each class with
some exceptions when classes depend on each other, one for
macros, one for global variables, and others. This organization
leads to interoperability between the Arduino IDE, AtmelStudio,
Visual Micro, Visual Studio, VS Code, and PlatformIO. See
Interoperability (page 21).
Program Documentation Framework (page 155) -
Documenting what you have created is time consuming, but
you will be rewarded by an enhanced insight of your program,
reducing your overall development time. Furthermore, months
later, when you come back to documented code, you will
congratulate yourself for having such foresight as you quickly
get up to speed. Having documented code via the Program
Documentation Framework (page ١٥٥), you may use the Perl
program [Link] to extract comments
from your code to produce program documentation. See
 Misdoings (page 117) and A note on the book's source code
(page 338).
Function Creation Framework (page 157) - A function should
be properly structured right from the beginning. After defining a
function's name, its parameters, and the return type, more
needs to be defined before writing a single line of code. When
creating a new function, copy/paste a function template and fill
in the documentation template's content. It will force you into
giving the function more attention thereby enhancing your
insight and creativity.

These three frameworks require a little discipline. They do initially

consume setup time, but the results will reward the time invested.

1.
Project Files Framework
<x>Project Files Framework[AUX]Common sense dictates how to organize an application's source code</x><x>File organization[AUX]See Project Files

When beginning an Arduino project with the Arduino IDE, upon

Framework</x>

clicking on the Arduino icon in the Window's toolbar or somewhere in

your screen, the Arduino IDE opens a skeleton .ino file which
contains a call to setup and to loop; you then add code into each one.
Click on the second icon from the left in the Arduino IDE's toolbar
(the first one being compile only). The IDE takes over; compiles the
code; and, if error free, uploads it into the board. This assumes that
the correct comm port, board and microcontroller have already been
set from the Arduino IDE's Tools menu. This setup is good enough
for a small program such as Blink, the Arduino equivalent of C's "Hello
world".
When things start getting complicated, the simple .ino setup/loop model
will not do. My beehive weighing system project, at 35 files, 15000
lines of code, required my being organized. As the project grew, I
progressively developed a Project Files Framework. This has
contributed to robustness and productivity. Furthermore, I devised a
scheme to ensure interoperability between the various IDEs by using
IDE specific #defines. Instead of coding the application's startup code
in setup, I coded it in a function named Aardvark. All setup does is call
Aardvark thereby obtaining interoperability amongst IDEs. See
Interoperability (page 21) for details.
<x>File organization[AUX]Start development in Aardvark, called by setup, then expand into other files</x><x>File organization[AUX]Aardvark() contains original setup code, key

to interoperability</x><x>File organization[AUX]Class definitions in ClassSpecific.h/.cpp files</x><x>File organization[AUX]HelperFunctions.h/.cpp: service functions</x>

<x>File organization[AUX]Globals.h/.cpp contain #includes and global variables; macros in Macros.h file</x><x>File organization[AUX]Functions.h/.cpp and FunctionsSKL.h/.cpp

The Project Files Framework is structured

contain general and application specific functions</x>

around nine file sets, each of which responds to a specific need

independent of the project's nature:

<x>Project Files Framework[AUX]Entry files: Arduino: .ino file; AtmelStudio and Visual Studio/Visual Micro: [Link]; PlatformIO: [Link]</x><x>Project

Files Framework[AUX]setup and loop located in application entry file</x><x>File organization[AUX]Entry files - Arduino IDE: .ino file; AtmelStudio: [Link];

PlatformIO: [Link] </x><x>Application entry files[AUX]Arduino: .ino file, AtmelStudio and Visual Studio/Visual Micro, PlatformIO: [Link]</x><x>Arduino

(build)[AUX].ino file is Arduino's entry file (C++ source code)</x><x>AtmelStudio (interoperability)[AUX]AtmelStudio Arduino project entry file [Link]

Application entry files -

contains setup and loop</x><x>Interoperability[AUX]setup and loop located in application entry file</x>

Each IDE requires a main entry file into the application. The
.ino file is the Arduino IDE's application entry file; it contains
setup and loop. [Link] generated by AtmelStudio and
[Link], generated by PlatformIO, are the respective .ino file
equivalents. They should contain whatever the Arduino .ino file
contains. Visual Micro does not create its own Arduino entry
file such as AtmelStudio and PlatformIO do; it uses the
AtmelStudio [Link] file. Note that when AtmelStudio
imports an existing Arduino IDE project, it will generate a
[Link] file which will contain everything that the Arduino
IDE .ino file contains. They will be identical except for the
names. One is named [Link], the other is named
[Link]. PlatformIO also creates an Arduino .ino
equivalent, named [Link], but, unlike AtmelStudio'
[Link], [Link] will contain empty setup and loop functions;
it does not fill them with the contents of the Arduino IDE .ino
file.
<x>Project Files Framework[AUX]Aardvark() contains original setup code, key to interoperability</x><x>Aardvark[AUX]Aardvark() contains original setup code,

key to interoperability; in Aardvark.h/.cpp</x><x>Interoperability[AUX]Aardvark() contains original setup code, key to

interoperability</x>Aardvark.h/.cpp - Each IDE has its strong points; it is

 not an issue of replacing one with another; all should be
explored; use the one you feel most comfortable with.
Interoperability is based on one set of files to be used by any
of the environments seamlessly. The problem is that to achieve
interoperability, whatever changes you make in the .ino file,
you must also make in the AtmelStudio's [Link] file and in
the PlatformIO's [Link] file. To avoid having to make
changes in three files, I transferred all the code which would
normally be located in setup into the Aardvark function. By doing
this, the three IDE specific application entry files never change.
Each one's setup function contains one line of code: a call to
Aardvark. As to the Arduino imposed function loop, I leave it
empty; but it must exist, otherwise the linker will generate an
undefined reference.

is declared and defined in its own Aardvark .h/.cpp file pair.

Aardvark
As the project evolves, only the Aardvark function in [Link]
needs to be worked on. Aardvark does the work, not setup. See
Importing an Arduino project (AtmelStudio) (page 40),
Interoperability (page 21), and PlatformIO interoperability (page
107).

<x>Project Files Framework[AUX]Atest.h/.cpp contain ATest() to do testing</x><x>File organization[AUX]Do your testing in the ATest()

function</x>Atest.h/.cpp - When testing code, I do so in the Atest

function. It is called from Aardvark. It enables me to undertake
tests without polluting the contents of the Aardvark function.
<x>Project Files Framework[AUX]Globals.h/.cpp contain #includes and global variables; macros in Macros.h file</x>Globals.h/.cpp -

enum lists, global variables, #includes (other header files are

limited to doing #include "Globals.h"), and more, which span across
the entire application, are put in this file pair. For example, const
uint16_t charBufferSize{256} should be placed in [Link] and extern
const uint16_t charBufferSize in Globals.h.
<x>Project Files Framework[AUX]Macros.h contains macro definitions</x><x>File organization[AUX]Macros.h contains macro

Macros.h - Preprocessing is one of C/C++'s valuable

definitions</x>

features. Over time, an increasing number of application wide

macros get defined, such as #define BUFFER_SIZE ٢٥٦. These
should be placed in the Macros.h file. Macros specific to a
class could be placed in the class's header file .h. There is no
need for a [Link] file since macros are declarations, they
do not reserve memory space.
<x>Project Files Framework[AUX]ClassSpecific.h/.cpp files contain classes</x>ClassSpecific.h/.cpp - Top-down

development via data and functions encapsulation requires an

extensive intelligent use of classes. These should be stored in
dedicated .h/.cpp file pairs, generally one file pair per class. It
may occasionally be preferable to store several related classes
in one file pair, ex. linked list classes. The linked lists
implement two classes: LinkedItem and LinkedList. Both classes are
declared and defined in the LinkedList.h/.cpp pair. The general
rule is to have one .h/cpp file pair per class or related classes.
An application could have 10, 20, 50... classes. By adopting
the rule one class - one file, productivity, robustness, and
maintainability are enhanced.
<x>Project Files Framework[AUX]HelperFunctions.h/.cpp: service functions</x>HelperFunctions.h/.cpp - It is

sometimes necessary to create low-level general purpose non-

application specific functions to be used throughout the
application, such as string-to-number conversions and hexa-to-
base ١٠ ASCII conversions. These should be stored in a
dedicated HelperFunctions.h/.cpp file pair.
<x>Project Files Framework[AUX]Functions.h/.cpp and FunctionsSKL.h/.cpp contain general and application specific

Functions.h/.cpp - There are high-level functions,

functions</x>

differentiated from helper functions, which are not application

dependent, such as DoStartup to turn things on and DoShutdown to
shut the system down. In a parent/child application, parent and
child have functions/classes common to both and
functions/classes of their own (next paragraph).
FunctionsSKL.h/.cpp - Some high-level functions may be
specific to the application, ex., the parent and the child. These
would be stored in a separate file pair identified by the keyword
Functions and initials to signal what the file pair relates to. It
could be FunctionsPAR for parent and FunctionsCHD for child.
 SKL refers to skeleton, a file pair I use to kick start creating
new non class specific files.

The file groups defined above could be located in a directory named

Skeleton so that when undertaking a new project, you could copy the
files into the newly created project directory and adapt as needed.
The above may seem a bit burdensome when starting a project.
Over time, as the application grows, it becomes a solid foundation to
build the project on. See Project Files Framework (appendix) (page
243) for details on header files and the Globals.h file.
Before closing this presentation of the Project Files Framework, a
couple of details concerning file inclusions merit an explanation. The
contents of header files (.h) are encapsulated within a #ifndef
MY_CLASS_H #endif pair to avoid double inclusions, thus:
#ifndef MY_CLASS_H
#define MY_CLASS_H
...
...
#endif // MY_CLASS_H
<x>Header files (.h)[AUX]#includes in Globals.h, pivot around which all header files (.h) get included</x><x>File organization[AUX]#includes in Globals.h, pivot around which all

Globals.h is the pivot around which all header files (.h)

header files (.h) get included</x>

get included. Its structure is:

#ifndef GLOBALS_H
#define GLOBALS _H
#include <Arduino.h>
#include "Macros.h"
...
global declarations
...
#include "HelperFunctions.h"
#include "Functions.h"
#include "FunctionsSKL.h"
#include "BasicClass.h"
#include "Atest.h"
#include "Aardvark.h"
...
#endif // GLOBALS _H
The #ifndef GLOBALS_H ... #endif pair ensures that the file gets included
only once. As for the other project header files, inclusion is
centralized in the one Globals.h file.
<x>Header files (.h)[AUX]Arduino.h, Macros.h included in Globals.h file, #include <Arduino.h> in Macros.h</x>Notice in the above that
Arduino.h and Macros.h get included at the top of the Globals.h file
and that the application's header files are included at the bottom.
This is because the body of the Globals.h file may require items
defined in Arduino.h and Macros.h. The other header files are
located in the bottom of the Globals.h file because the application's
header files require items defined in the body of the Globals.h file.
<x>Header files (.h)[AUX]Respect header file dependencies (top-down) and beware of interdependencies</x>You should be on the

lookout for dependencies. If a specific header file uses items defined

in another header file, the other header file comes first.
Interdependent header files must be avoided. If two header files
depend on each other, we have a problem. If ClassABC.h contains
items defined in ClassXYZ.h and vice versa, the application must be
reprogrammed so that there be a dependence and not an
interdependence.

2.
Program Documentation Framework
It is a given that
<x>Program Documentation Framework[AUX]Create program documentation via Awk/Perl regex based comment extraction</x>

code should be amply documented. Doing such work should be a

self-imposed discipline. Its benefits cannot be overemphasized.
Many reasons underly documenting code:

Additional thinking as you document code improves your

algorithm .
Recall past work - You may have long forgotten what you did
months ago.
Provide help for those who may be in charge of doing
maintenance.
Knowledge base for upgrades - When improving an algorithm,
documentation on past work will prove priceless.

One forgets things over time. If code contains well thought out
comments, these become precious assets during later development
and debugging. They explain what a function being worked on does.
Furthermore, if several programmers are to work on the project,
proper documentation becomes an absolute requirement so that the
team members can work efficiently.
C/C++ programs contain lots of distinct components. They are an
assembly of files which contain classes, functions, macros, enums,
global data. Small one .ino file programs are mostly self-
documenting, no special attention is required, however, when an
application gets large, it becomes increasingly vital to document
work, for oneself as the programmer, and for others who will have an
active role in the project. Good in-code documentation provides help
to better use and maintain the program.
You will at first write occasional comments here and there but, after a
while, the need for a formal template-based documentation system
becomes obvious. Slowly but surely, as the application's size
increases, documenting files, classes, and functions becomes
necessary. This is where the Program Documentation Framework
comes into play.
We tend to follow the path of least resistance and have an innate
desire to get the job done as quickly as possible. The result is that
comments usually get added in, if any, as an afterthought. To
circumvent this tendency, program documentation templates help
you document code right away. It becomes a fill-in-the-blanks
process. Telling oneself I must document what I am doing is not
good enough. Such an attitude is like a new year's resolution - good
intentions, no more. Documentation templates greatly help reduce
the tendency to postpone inserting comments because they are
there, crying out to be filled-in with appropriate info.
Questions: What needs to be documented? How should
documentation be structured? Should documentation be a
separately written document, or should its content rely upon
comments inside code?
Having clearly understood the need for program documentation, the
question remains. What needs to be documented? When I first
started work on my Arduino-based beehive weighing system, my
priority was getting the program to work. I had a nagging feeling that
I should document code as I advanced. So, I did write comments in
places where logic got to be a little tricky but, in the beginning, I did
not formalize this or make it a habit. After a while, documentation
started to become an increasingly important need. This led me,
slowly but surely, to organize comment-based documentation as I
developed code; I described what I was doing. I initially did this free-
style and later, via templates.
Documenting code is a two-pronged process, top down and bottom
up. Top down (the way you think, a global mental image of the
application) starts with the big picture at the application's entry point
in the application's entry file ([Link], [Link], [Link]
files), followed by file level documentation. Bottom up starts with
statement level comments, upwards towards variables level, function
level, and finally, class level.

Files - C++ files generally come in pairs: header files .(h) which
declare classes, functions, macros, enums, and implementation
files (.cpp), which contain code and global and static variables.
The first objective when documenting code is to compile info
on what an .h/.cpp file group contains, what it does, how it
functions, governing algorithms and ideas, key variables, and
any relevant top-level information which will help the
programmer. Describe the forest first, then the patches, and
finally the trees. These file level comments should be located
in the header file (.h). It should contain a PROGRAM FILE template
inside of which you will write the important features of the file's
content. See Program Documentation Framework (appendix)
(page 244) for details on the templates.
Classes -The header file should contain as many PROGRAM
CLASS templates as there are classes. The classes' functions
are individually documented in the .cpp file via the
corresponding PROGRAM FUNCTION templates.
Functions and methods - Functions are global or may belong
to a class. Their bodies contain algorithms. They take
parameters, do validity checks, and may return a value. They
may be documented via the PROGRAM FUNCTION template which
encompasses description, parameters, error checking, and
 return a value.
Program flow - As the application gets large, it will contain
hundreds of methods and functions. Because of the sheer
number of inter-relationships, it is well worth knowing who calls
who and who gets called by whom. Whenever a function gets
called, the programmer can insert a PROGRAM FLOW macro which
contains the name of the function being called. Since the Perl
program knows at this point which function is being processed
because of the PROGRAM FLOW START macro, it can generate a
two-fields record, the first field being the calling function, the
second field being the called function. These can be exported
into a CSV file whose content can then be plugged into Excel
so that it may be sorted by calling function and by called
function.
Macros (#define) can be especially useful. It is well worth
documenting them, at least to know that they exist and where
they are defined, whether they be commented or not. A list of
unused macros may also prove useful. These are macros
defined but not used. A separate list of commented out macros
would also be useful.
<x>enums[AUX]Program Documentation Framework - enums are often an alternative to macros</x>enums are often an

alternative to macros. It is well worth while documenting them,

as well as enums which have been created yet not used.

The final framework I came up with, referred to as the Program

Documentation Framework (appendix) (page 244), encompasses
documentation templates inserted into code files, and Perl based
programs to extract documentation from files, classes, functions,
program flow, macros and enums.
The templates used to extract documentation are based on
keywords. Their general format is:
//PROGRAM KEYWORD START
// The following is a description of the .h/.cpp file contents
// ...
//PROGRAM KEYWORD END
In the above, KEYWORD is a place holder which stands for FILE, CLASS,
or FUNCTION.
The process
<x>Program Documentation Framework[AUX]PROGRAM KEYWORD START and END basis of generalized info extraction mechanism</x>

is simple. When the Perl application encounters a sequence

PROGRAM KEYWORD START, it starts exporting lines. When it
encounters the sequence PROGRAM KEYWORD END, it stops
exporting lines. After having extracted lines of text, the Perl program
does just a tad of output formatting and inserts them into program
documentation files.
You will find detailed descriptions of the documentation extraction
process in Program Documentation Framework (appendix) (page
244).

3.
Function Creation Framework
Starting a function's definition with a
<x>Function Creation Framework[AUX]Function template to create function</x>

skeleton template reduces the risk of errors by omission. Functions

are complex, they require that they be documented, not just any
which way according to the programmer's whims. Documentation
should be formalized so that it be complete. It should include the
function's objective, what it is meant to accomplish, its return value,
its parameters and their purpose, and validity checks.
There is much more to creating a function than creating a declaration
in an .h file and its implementation in a .cpp file. Inline documentation
should be inserted to formally document the function along with the
corresponding print-based debugging macros. All this taken together
constitutes the Function Creation Framework.
When creating a new function, it is advisable to use a template. It will
help you adhere to the following:

Name - Assign the function a meaningful name. It should

reflect what the function does. Write the function's declaration
in the .h file, and an implementation skeleton in the
corresponding .cpp file.
<x>Function Creation Framework[AUX]Document functions exhaustively</x>Function documentation - Insert
 the Function documentation template as per the Program
Documentation Framework and start filling it with appropriate
info and describe what the function does.
Debugging macros - Define the function's print-based
debugging macro, place it at the head of its .cpp file along with
the other debugging macros.
Print macros - Place the enter/exit function macros enclosed
by the function's #ifdef debugging macro.
Parameter validations - Insert macros to print parameter
values and comments to signal that parameter validation code
is required.
Return value - Define and describe what the function should
return, if anything (void function).

Test before adding any code to ensure that the function's preliminary
definition compiles correctly.
Since you will be creating many functions, it is advisable to create a
sample Foo function which you can copy/paste and modify to create
new functions. Starting with a complete function skeleton reduces
errors, gets you going with comments, and gets you started with a
clean complete structure on which to build.
Doing the preliminary work detailed above before beginning to code
does consume a little time. However, as the old adage says, a stich
in time saves nine. See Function Creation Framework (appendix)
(page 249) for full details.

2.
Data Handling Frameworks
Microcontroller memory is
<x>Data Handling Frameworks[AUX]Frameworks to handle data storage, data conversion, data compacting</x>

usually a scarce resource, which requires that data should be

packed as much as feasible. Within the context of my beehive
weighing system, I developed seven frameworks to handle data,
collectively referred to as the Data Handling Frameworks. These
accomplish more efficient memory use, programming ease, and
improved maintainability.

Class Data Framework (page 159) - When a class contains

identically typed variables, accessing them via enum driven
offsets into a structure containing them adds to the program's
robustness and maintainability.
Data Packets Framework (page 161) prepares data for radio
transmission. Data is broken up into chunks which are in turn
broken up into packets and transmitted, one packet at a time.
Checksums to verify that data has not been corrupted, packet
sequencing, total data sent, etc. are mechanisms which
improve data integrity.
The table-driven Format Driven float to byte Conversion
Framework (page 162) converts floats to packed bytes and
vice versa. Since the quantity of data sent via radio
transmission is limited, every effort must be undertaken to
squeeze out a byte here and a byte there. For example, when
the humidity sensor provides a reading of 64.7%, my reaction
is: Meh! So what? All I need to know is that it is closer to 60%
than 70%. Therefore, I can limit data transfer to one byte which
holds humidity values in 10s of percent. Instead of transmitting
a value of 64.7, which requires four bytes, a value of 6 gets
transmitted, which requires only one byte. This reasoning can
lead to substantial memory savings.
The DataGroup Framework (page 164) is an index
based/sequential data storage system to store fixed or variable
length records. It may be used to store data in EEPROM, SD
cards, RAM, or in any device in which a block of memory may
be set aside and its contents managed via an offset.
Bitfield Storage Framework (page 166) - Upon examining the
values which variables can take, the conclusion is that
individual storage requirements of some variables may be
reduced down to 1 bit. For those variables which require less
than 8-bits, the C++ bitfield storage mechanism may be used
 to limit a variable's memory use to the strictest minimum. A bool
value, for example, requires only one bit. If a class contains
numerous bitfield size variables, they may be grouped across
several bytes. The Bitfield Storage Framework manages a
table driven contiguous byte array with one Get(myVarID, value)
function and one Set(myVarID, value) function to retrieve and set
values. Substantial storage savings may be achieved.
Event Storage Framework (page 171) - It is at times necessary
to define a list of true/false conditions such as errors, work
accomplished, and milestones. Call these events. Storing
these implies storing bool values. These should be stored inside
1-bit sized bitfields and not in individual 8-bit bytes. The space
saving is 8-fold. Event storage is handled as a contiguous
address space so that accessing individual events can be
handled by a pair of functions: Get(myEventID, value) and Set(myEventID,
value).
Linked List Framework (page 172) - When individual variable
sizes vary and the number of items to store is unknown, array
storage can get cumbersome and may prove inefficient
memory-wise. A linked list is a better alternative. The Linked
List Framework provides base classes which can be used to
provide linked list storage to any class via inheritance.

These frameworks are described in individual framework sections

below; and, for some of them, in more detail in the appendix (Data
Handling Frameworks (appendix) page 253).

1.
Class Data Framework
Some classes may need to store
<x>Class Data Framework[AUX]Access data via Get/Set enum driven offsets</x>

several identically typed variables; these could have the same

length, in which case they may simplify data access via enum driven
offsets in arrays, or have different lengths, in which case they may
be stored in structures. Whatever the data's nature, the use of enums
enriches data manipulation possibilities by enabling cycling through
the items in for and while loops to check for data validity and the like.
Imagine a class in which variable types differ. They must be accessed
individually via dedicated functions. In the class Employee below,
accessing items is usually done via dedicated Get/Set functions such
as GetName/SetName to get/set the name variable. This class would have one
dedicated Get and one dedicated Set function for each private variable.
When creating such a class, one would begin by creating the
variables and corresponding Get/Set functions individually, as follows:
class Employee {
public:
Employee( ... ) { ... } // constructor
bool SetName() { strcpy(name, _name ); } // there should be validity checks
char* GetName() { return name; }
...
private:
char civility[6]; // group items of the same type
char name[60];
char address[100];
...
}; // Employee
To access the name of an employee, use its dedicated function GetName:
Employee empSashaDoe( ... );
char tempName[50];
strcpy(tempName, [Link]()); // accesses an employee's name
The above is what most newbie programmers would do. There is not
much one can do to reduce the programming burden inherent in
designing such classes: as many dedicated Set/Get functions as there
are variables to be implemented. However, if variables are of the
same type, such as in the Employee class above, a little planning and
rearranging can reduce the programming burden and improve
maintainability.
A convenient improvement would be to handle the char arrays as a
contiguous memory zone inside a struct. Instead of storing Employee data
in separate char arrays, it is more flexible to store all the Employee data
in a struct and access individual variables via appropriate offsets from
the struct's base address via generalized Get and Set functions. Using
#defines to specify char array sizes constitutes the first step:
 #define EMP_SIZE_CIVILITY 6 // includes null ending
#define EMP_SIZE_NAME 60
#define EMP_SIZE_ADDRESS 100
...
The arrays are synchronized by an
<x>enums[AUX]enums synchronize multiple arrays</x> enum list as
follows:
enum class EEmployee: uint8_t {
start = 0,
civility = 0, name, address, ...,
end
}; // EEmployee
Individual item sizes is obtained via a table such as:
uint8_t varSizes[] {
EMP_SIZE_CIVILITY, // size 6
EMP_SIZE_NAME, // size 60
EMP_SIZE_ADDRESS, // size 100
...
}; // varSizes
Thus the class Employee becomes:
class Employee {
public:
Employee( ... ) { ... } // constructor
void Set(EEmployee _eItem, char* _item) { See appendix for code details };
char* Get(EEmployee _eItem) { See appendix for code details };
...
private:
struct {
char civility[varSizes[EEmployee::civility]]; // group items of the same type
char name[varSizes[EEmployee::name]];
char address[varSizes[EEmployee::address]];
...
} charData; // struct
...
}; // Employee
To get the offset of a specific variable, add up the sizes of the
preceding variables, thus the offset of address from the beginning of the
structure's address will be at 6 + 60 = 66.
Note that the dedicated Get/Set functions in the original class above
have been replaced with two generalized Get/Set functions, which take
an enum to identify the variable being handled.
Setting up the Employee class to use generalized Get/Set functions
requires design work and testing but once done, using the Employee
class becomes a lot easier, less error prone, and renders it
considerably more maintainable. What is more, its data handling
mechanics could be implemented as a base class. You will find full
details in the Class Data Framework (appendix) (page 254).

2.
Data Packets Framework
When transmitting data across a serial
<x>Data Packets Framework[AUX]Send/receive packets via serial ports</x>

port or via radio transmission or whatever, a chunk of data must be

broken up into packets which the system can manage, each of which
is given an ID, checksum, and sequencing number so that data
received will be assembled into a complete data set. This is what the
Data Packets Framework does.
<x>Data Packets Framework[AUX]Break up data into chunks/packets</x><x>Data Packets Framework[AUX]Packet has header (start code, packet size, ID, sequencing info), body

The data to be sent is partitioned into chunks, each of

(data), tail (end code, checksum)</x>

which is in turn partitioned into packets, packet size being a fixed

value based on the transmission protocol. In my application's case,
serial communications having been chosen, packet size is based on
the serial port's protocol buffer size, which is defaulted at 64 bytes on
Arduino - to be safe, I chose a 48 bytes package size. I found that
using the full buffer size (64 bytes) posed instability problems. WiFi,
radio, or Bluetooth could be managed in much the same way.
A packet will contain user data and overhead to manage packet
transmission. I chose a maximum of four packets per chunk of data,
which corresponds to 169 bytes of data total. If more data than the
maximum chunk size is to be sent, the application would have to be
structured to send several 169 bytes max sized chunks.
The Data Packets Framework algorithm kicks off by defining the
number of packets to be sent. For a maximum of 169 bytes per
chunk of data, one first and three consecutive packets may be
required. Since the first packet contains chunk info, it has less data
capacity than the consecutive packets. When a packet is being
created/received, data is injected into/extracted from the packet, the
number of data bytes being:

First packet 40 - 48 minus packet header bytes (6), minus tail

 bytes (2).
Following packets 43 - 48 minus consecutive packets header
bytes (3), minus tail bytes (2).

This means that due to the header/tail overheads, a maximum of 169

bytes can be sent in one chunk, first packet (40) plus three
consecutive packets (3 * 43 = 129).
Assembling a packet to be sent begins with start/end delimiter bytes:
DATA_PACKETS_START_CODE 0x7E (126 decimal)
DATA_PACKETS_END_CODE 0x7F (125 decimal)
<x>Data Packets Framework[AUX]Validity checks: data received and checksum on data</x> The user data being sent may
contain these values. As soon as a start code is received, 47 bytes
are expected. They could have any value. The 48th byte should be
the end code 0x7F. If byte 0 contains the start code 0x7F and the end
byte contains the end code 0x7E, the packet is a priori complete. It
must then be validated via its Checksum, the 47th byte. The
algorithm, having acquired 48 valid bytes, may proceed with
additional reliability checks.
<x>Data Packets Framework[AUX]Packet is broken up into header (start code, packet size, ID, sequencing info); body (data); tail (end code, checksum)</x>Each

Packet has three sections: head, body, tail. The first packet's head
contains more info (packet info) than the following packets' head.
packets start with a header to identify the packet (3 bytes):

start code - first byte, signals begin packet.

packet sequence - nth packet of consecutive packets (٤
packets max, ١-based).
data ID identifies chunk - user defined.

The packet body contains user data spread across one or more
packets, depending on data size.
All packets end with a tail to signal end of packet and provide a
checksum (2 bytes):

Checksum - An algorithm calculates a checksum which is

checked at the receiving end to ensure that data has not been
garbled. This byte is the next to the last one in the packet.
 End code - Last byte, signals end of packet.

The first packet contains additional info to define its size (3 bytes):

packet count - number of packets being sent. This info is

redundant since the packet count can be derived from data
size, it is used to check data size coherence.
packet size - Since packet size is hard coded fixed size, this
parameter is not required. It is there for possible future use.
data size - The receiving end needs to know the data size so
that it can determine the last data byte location in the last
packet. It also checks coherence with packet count.

The receiving end manages two error conditions:

Data sent - Data has been

<x>Data Packets Framework[AUX]Validity checks: data received and checksum on data</x>

received or not received. It does not specify whether the data

is OK.
Data OK - Signals whether received data is OK or not. Does
the checksum compute correctly?

The Arduino-based beehive weighing system became a test bench

for the Data Packets Framework. It has proven to be quite reliable.
The mechanics of the Data Packets Framework are complex. Its
use, however, is simple. See Data Packets Framework (appendix)
(page 257) for more details.

3.
Format Driven float to byte Conversion
Framework
<x>Format Driven float to byte Conversion Framework (FDFBC)[AUX]Converts float values to packed bytes and vice versa, packing controlled by parameters, FDFBC class does all

Sending data over a network via radio communications requires

the work</x>

that data be compacted as much as possible. Whatever gets

transmitted must be packed as bytes at the sending end and rebuilt
from bytes at the receiving end. Sensors generate float values which
cannot as such be transmitted because radio communications
operate on a one byte at a time basis. Under such circumstances, a
4-byte float union could be created thereby sending the float's value one
byte at a time. However, this may prove wasteful if, after analyzing
the data being sent, it turned out that some float values could be
packed into one byte, or into a few bits, thereby reducing the quantity
of data to transmit.
Such packing is accomplished by the class FDFBC (Format driven
float byte conversion). It encodes an array of float sensor values into a
byte array for transmission; and decodes the bytes at the receiving
end back into floats. Parameters define the packed byte size of
converted floats, decimals, data reduction via a divider, and the range
of values.
<x>Format Driven float to byte Conversion Framework (FDFBC)[AUX]Parameters include packed byte size, decimals, divider, range, offset</x>Defining the

conversion process starts with examining the possible range of

values the sensors may generate. Since sensor values and range
differ from one another, each sensor's float-byte conversion needs to
be customized to achieve compaction. This poses issues in terms of
maintenance. The FDFBC class replaces hard coded item-by-item
conversions with a generic format-driven conversion process. It is
based on the use of conversion parameters which drive the
conversion process, as follows:

Number of bytes - Each float variable can be packed into four

bytes or less. This is manually determined after examining
characteristics underlying each sensor's possible range of
values.
Number of decimals defines data resolution. Should a
temperature of 28.7°C (1 decimal) be transmitted? When
managing beehives, 29°C is good enough, no need for the
extra precision provided by one decimal. Since beehive
temperatures will vary from 15°C to 45°C, and the bees perish
when below 15° or above 45°, the 15°/45° range is the life-
sustaining temperature range needed. This leads me to offset
 the temperature by -15° and transmit values which range from
0 to 30. One byte (5 bits) will suffice.
Divider - The raw data value may contain unnecessary
precision. Humidity displayed in 10s of percent (40%, 50%...)
is good enough. From a practical perspective, 60% humidity is
as good as 57%. Since humidity values range from 30% to
100%, apply an offset of -30, round off humidity to nearest 10%
and divide by 10. Hence, data transmitted will be 0 to 7. One 1
byte (3 bits) suffices.
Min-Max values - Sensor values lie between thresholds.
Humidity never drops below 30%. Temperatures inside the
beehive vary from 15°C to 45°C. Regulated voltage readings
will range from 4.7 to 5.0 volts, etc. The min value defines an
offset.
Offsets - There is no need to transmit an absolute value.
Transmitting the increment above an offset value is adequate
and may reduce the number of bits required. For a 4.9
regulated voltage reading, the transmitted increment will be
(4.9 - 4.7) = 0.2/divider hence 2 (divider set at 0.1). The inverse will be
done at the receiving end.

Before delving into the mechanics of converting floats to bytes to

transfer data, the following radio constraints need to be accounted
for:

Data gets transmitted as byte sequences. Sensor data,

presented as floats, must be converted to byte sequences of
appropriate size, and decoded back at the receiving end into
the corresponding float values.
Sensor data may be positive only (voltages, humidity, weights)
or may be both, positive or negative (outside temperature) in
which case an offset is needed to account for negative values.
Furthermore, the range may be small (4.7 to 5.0 for volts) or
large (0.00 to 50.00, 2 decimals for weight voltages). These
considerations require that data sizes be a few bits or one, two
 or more bytes.
The (min/max) may be used to reduce the byte size by
restricting transmission to increments above a minimum
(offset). For example, regulated voltage ranges from 4.7 to 5.0
volts - given a one decimal requirement, data will range from 0
to 3, thereby requiring 1 byte only (2 bits).

The FDFBC class applies these constraints. It starts by converting

sensor data expressed as floats, into a set of data bytes to be
transferred via the radio network. At the receiving end, the data
stream is converted back into individual float items.
The algorithm begins with an enum list which defines a group of
sensors whose data is to be transmitted, thus:
enum class EHiveSensors: uint8_t {
start = 0,
volts5 = 0, volts8, tempHive, tempAmbient, humidHive, humidAmbient, body, super,
end
}; // EHiveSensors
The algorithm converts float values to packed bytes and vice versa,
packing being controlled via parameters which specify how sensor
float values are to be converted into byte sequences. These
parameters (destination bytes, decimals...) are stored in the
SensorDataParams structure:
struct SensorDataParams {
uint8_t bytes: 2; // NumberBytes (values: 1 - 4)
uint8_t decimals: 2; // NumberDecimals (values: 0 - 3)
uint8_t divider: 2;// divide by multiples of 10 (values: 0-3)
float min{0.0}; // Minimum possible value
float max{0.0}; // maximum possible value
}; // SensorDataParams
The outcome of this
<x>Format Driven float to byte Conversion Framework (FDFBC)[AUX]float values stored in SensorFloatData[]</x>

conversion process reduces the size of the DataPacket to be

transmitted. Sensor float data and sensor byte packed data are stored
in their own respective arrays: float SensorFloatData[] holds sensor values
as floats and uint8_t SensorByteData[] holds converted sensor values as
packed bytes.
Using the class FDFBC to transmit my beehive weighing system's sensor
data reduced byte requirements from 32 to 9 bytes - that is a huge
saving. Furthermore, being table driven, conversion changes are
limited to modifying the control parameters, thereby rendering
debugging easier and improving maintainability. Further compaction
could be achieved by using bitfield storage, however, I felt that the
benefits did not warrant the extra work required. I calculated that
using bitfields on the 6 sensors would save an additional two bytes.
Details of class FDFBC can be found in Format Driven float to byte
Conversion Framework (appendix) (page 258).

4.
DataGroup Framework
It is a
<x>DataGroup Framework[AUX]Index/sequential fixed/variable length data</x><x>Memory (pools)[AUX]EEPROM writes limited to 100,000</x>

certainty that when the microcontroller is turned off, all data

contained in RAM is lost. Since practically all applications need to
use data from previous runs, the questions raised are: Where could
permanent data be stored? In an SD card? Or somewhere wireless
(Bluetooth, the Internet...)? Or in a USB card? Or in an external
EEPROM? In the microcontroller's EEPROM? If the quantity of data
to store can fit in the microcontroller's EEPROM, this would be the
preferable choice because it is the least expensive, simplest, most
immediate solution. There would be no need for extra wiring and
interfaces. There is a drawback though: EEPROM writes are limited
to 100,000. Once in production, if you do 10 writes per day, the
microcontroller is good for 27 years, however, if you do 100 writes
per day, the limit goes down to 2.7 years and, because time flies,
you find yourself changing the microcontroller too often. This being
said, EEPROM reads are unlimited.
Whatever gets stored, think in terms of mini databases. They can be
a list of contacts, an array of beehive IDs, a list of jobs, etc. Each
one of these can be thought of as being a DataGroup of homogeneous
data characterized by several features:

ID - A DataGroup
<x>DataGroup Framework[AUX]DataGroup class does the work: header defines characteristics, body contains data</x>

should have a name so that it could be identified.

DataGroup size and location - The DataGroup needs to be stored in a
 block of memory somewhere. It can be a temporary block in
RAM provided by a malloc or new or a permanent block in
EEPROM or somewhere else.
Storage method - The DataGroup may be thought of as being a
consecutive list of records, each of which holds data. These
records are either fixed length or variable length and may be
accessed sequentially, one after the other, or via a key to
access a specific one. The above may be summarized by
saying that a DataGroup is a block of memory designed to store
consecutive fixed or variable length records accessed directly
or sequentially, aka an indexed sequential variable length data
access method.

The DataGroup concept described above is implemented in the

DataGroup Framework. It contains classes and functions to manage
data in the microcontroller's EEPROM but could be easily adapted to
store data in other supports.
A DataGroup memory zone contains two components. The first one is a
header; it contains the DataGroup's name, a description, the total size of
the DataGroup storage including the header, and a flag to indicate that
records are fixed or variable length. The second one is the body; it
holds data organized as fixed or variable sized records.
The header contains the DataGroup's characteristics:

Name and description - 10 and 50 bytes. These sizes are hard

coded by #defines.
Size is the total memory space the DataGroup requires, including
the header.
A record length if fixed length record, otherwise 0 for variable
length records.
A key flag which defines whether keys are used.
A corresponding key length, equal to 0 if keys are not used.

The DataGroup's body contains records whose length is either fixed or

variable:
 Fixed length records -
<x>DataGroup Framework[AUX]Index/sequential fixed/variable length data</x>

Accessing these is simple! Each record being of a predefined

length, accessing a record via its offset inside the DataGroup's
body is the record's index number (relative position - zero
based) times the record's size.
Variable length records - Accessing these is a little more
complex. Records are written into the DataGroup's body with the
record's size at its head. This enables traversing the records
by jumping forward from one record to the next until the
desired record has been reached.

<x>DataGroup Framework[AUX]Functions enable finding, inserting, removing records and defragment DataGroup
storage</x> functions
accomplish the following:

Finding records - Accessing a particular record is the

application's responsibility. Some kind of key may optionally be
built into the record for direct access. Naturally, keys should be
unique to avoid duplicates. Keys may be of any length, up to
the record's max size. The algorithm searches for a desired
record by examining records one at a time until the desired
one is found. While progressing through the list, if dealing with
variable length records, offsets are calculated one at a time.
The Find function then passes the offset of the desired record to
the Read function.
Inserting a record depends on its type, fixed or variable. For
fixed length records, the algorithm searches for an empty slot,
and inserts it there. If there are no empty slots, it appends the
record at the end, assuming there is space. For variable length
records, the algorithm appends the record at the end or in a
big enough hole.
Record removal - Records may be removed one at a time. To
remove a record, the algorithm renders its slot inactive by
replacing its data with the key word EMPTY. This creates holes.
Defragmentation - When dealing with variable length records,
after numerous random removals, the DataGroup will increasingly
 appear to be a chunk of Swiss cheese. If the empty slot's total
size exceeds a certain tolerance, then the DataGroup's body
would necessarily have to be reorganized to remove the holes.
This is referred to as defragmentation.

Using the DataGroup Framework as opposed to doing direct hard

coding into the EEPROM has proven to be immensely practical and
less error prone. See the DataGroup Framework (appendix) (page
265) for details.

5.
Bitfield Storage Framework
The Bitfield Storage Framework
<x>Bitfield Storage Framework[AUX]Optimize data storage via bitfield packed variables</x>

optimizes a class's data storage via the use of bitfield packed

variables. When examining values which variables may attain, it is
clear that attributing the full 8-bits to store certain variables is
overkill. Some values will range from 0 to 31 (5 bits), others from 0 to
63 (6 bits), bool values (true/false) 1 bit, etc. When working with multiple
instances of classes which contain variables, using bitfields may
save a lot of space. The bitfield declaration syntax is simple - add a
colon and the number of bits at the end of the variable's declaration,
as follows:
uint8_t myVar: 3;
declares a variable uint8_t myVar
<x>Bitfield Storage Framework[AUX]Optimize data storage via bitfield packed variables</x>

which occupies three bits of memory only, not the full 8-bits of its
data type, thus allowing for other variables to be packed into the
same byte. Note that the bitfield mechanism may be used in
structures and classes only, not directly at global level. Data
intensive classes which consume a sizable chunk of memory, and/or
for which there are many instances, can benefit from this
mechanism. When creating a class, the standard step-by-step
procedure would be:

Create a bare-bones class with its constructor and private and

public sections.
 Add variables; assign bitfields to the variables; and reorganize
them into groups of 8-bits.
Add Get/Set functions for each variable.

Here is a first draft for

<x>Bitfield Storage Framework[AUX]Bitfield storage example, 3 arrays: JobData, VarsMinMax, JobLabels data arrays</x>

the Job class of my beehive weighing system:

class Job {
public:
Job(){ };
bool GetStatus( );
void SetStatus( bool _status);
uint8_t GetJobType();
…
private:
bool status: 1;
uint8_t jobType: 3;
…
bool repeatType: 1;
uint8_t repeatNumber: 4;
dummy1: 3;
}; // class Job
Notice that bitfields have been used. Variables occupy the space
they need, no more. Reducing storage space and not straddling byte
boundaries is best done by initially listing variables in a spreadsheet
and rearranging them. If necessary, dummy variables are added to
complete 8-bit blocks. Here is the result of the Job class's variables
presented as a table. Dummy variables were not necessary except
for the optional last one (dummy1) to complete the last 8-bit block.
Min Max Siz Offse Byte
. . e t #
status 0 1 1 0 0
jobType 0 3 3 1 0
repeatInterval 0 10 4 4 0
day 1 31 5 8 1
weekday 0 6 3 13 1
scheduleType 0 3 3 16 2
hour 0 23 5 19 2
 1/4 hour units 0 3 2 24 3
repeatType 0 1 1 26 3
repeatNumb
0 10 4 27 3
er
dummy1 0 1 1 31 3
Table 16.1 - Job class - <x>Tables[AUX]Bitfield Storage Framework - Table of bitfield based variables of Job class</x><x>Bitfield Storage
Framework[AUX]Job class inherits from the Bitfield class</x>Table of bitfield based variables.

Note that four bytes suffice to pack the 10 variables, a considerable

savings if many instances of the class were needed. If the class Job
were to contain more variables, such as year, month, etc., the storage
savings would be significantly greater.
The above accomplishes one first objective: memory requirements to
store data have been reduced from 10 to 4 bytes thereby saving 6
bytes per Job instance. However, programming remains tedious since
data access has to be performed on a variable per variable basis.
Because there are 10 variables, 10 pairs of Get/Set functions must be
created such as the GetStatus/SetStatus pair. Whenever the class needs to
be modified by adding, removing, or modifying variables, the
positioning in the variable list may need reshuffling and
corresponding Get/Set functions need to be created or removed.
As usual, never assume anything. When dealing with data, a
validation scheme is needed to ascertain that values lie within
acceptable limits. Such a scheme would require a two-dimensional
array of min/max values: (status 0/1, jobType 0/3...) to assert that values lie
within the min/max range, which should, of course, be coherent with
the number of bits assigned to store the data.
uint8_t varsMinMax[][2] = {
{ 0, 1 }, // status
{ 0, 3 }, // jobType
...
{ 0, 1 }, // repeatType
{ 0, 10 } // repeatNumber
}; // varsMinMax[][2]
Accessing a particular item, for example, jobType's min/max values of 0/3,
requires that an offset be used for positioning inside the array. An
enum list of variables would be a convenient way to access the
different min/max values:
enum class EJobVars: uint8_t {
start = 0,
status = 0, jobType, ..., repeatType, repeatNumber,
end
}; // EJobVars
Having gone this far, define an array of jobVarLabels for reporting
purposes:
char* jobVarLabels[] {
"status",
"jobType",
...
"repeatType",
"repeatNumber"
} // jobVarLabels
Note that the JobData
<x>Bitfield Storage Framework[AUX]Bitfield storage example, 3 arrays: JobData, VarsMinMax, JobLabels data arrays</x>

data array, the VarsMinMax array, and the JobLabels array could have been
assembled into a struct array, thus:
struct VarsCharacteristics {
enum EJobVars; // type of variable
uint8_t size; // Number of bits to store data
uint8_t min, max; // Min and max values to validate data
char* label; // the variable's name
}; // VarsCharacteristics
The
<x>Bitfield Storage Framework[AUX]BitfieldStorage class requires info on variables stored in VarCharacteristics; uses bit-masks to operate on bytes</x>

complete data set which characterizes the bitfield packed data would
be stored in a VarsCharacteristics array as follows:
static VarsCharacteristics varsCharacteristics[] = {
{EJobVars::status, 1, 0, 1, "status"}, // status
{EJobVars::jobType, 3, 0, 3, "jobType"}, // jobType
{EJobVars::repeatinterval, 4, 0, 10, "repeatInterval}, // repeatInterval
...
} // varsCharacteristics
Putting all the data into a VarsCharacteristics struct array, rather than in
separate arrays synchronized via an enum list, makes for a more
maintainable and less error prone program.
So far, no framework has been defined, just a little bit of savvy
programming to render code more compact, more robust, and
somewhat easier to manage. Three properties have been added to
the Job class to store data efficiently:

Variables have been dimensioned via bitfields and aligned

 properly to avoid straddling 8-bit boundaries. Dummy variables
are added if necessary.
A class specific enum list identifies which variable to work with.
A VarsCharacteristics struct array contains an enum based ID, bit size,
min/max values, and label for each variable.

The tables above are

<x>Bitfield Storage Framework[AUX]Variables accesses via enum driven generalized Get/Set functions</x>

synchronized by the enum list. However, accessing variables, whether

via a Get or a Set, requires defining two dedicated functions for each
variable, thus jobType requires GetJobType and SetJobType, etc. Since the Job
class contains 10 variables, 20 such functions must be defined.
Surely, there must be a better way, particularly since a variables enum
list has already been defined. This is where the Bitfield Storage
Framework comes in and does away with the individual Get/Set
functions. It replaces them with one generalized Get/Set function pair
which takes an EJobVars enum to identify which item to Get or to Set,
irrespective of the total number of bitfield variables. It simplifies
programming and improves maintainability. It uses a simple
technique to accomplish this: variables get accessed via their offset
from a base address.
Because the compiler will pack together the four bytes which the Job
class requires to store its variables, accessing a variable can be
done via its offset from the first bitfield sized variable. For example,
ReportType is at offset 26 and has a bit size of 1.
The Bitfield Storage Framework's algorithm is based on the
following:

Identify the byte the variable is located in via its total offset
from the beginning of the byte array. RepeatType is in byte 3, the
4th byte (26/8 = 3).
Determine the variable's offset inside the byte by doing a
modulo 2. Call it localOffset (26%8 = 2).
Create a mask to Get/Set the ReportType's value in byte 3 via an AND
or an OR on the byte. For RepeatType, the mask would be b00000100.
It is a 1-bit long variable.
 If doing a Get, copy byte 3 into a uint8_t result byte, do an AND onto
this byte with the mask and shift right two times.
When doing a Set, put the value you want to set on RepeatType in a
uint8_t result byte, shift left two times as indicated above and do an
OR onto byte 3 with the RepeatType's mask.

For this approach to work, a bitfields byte array, whose base address
is that of the first byte of the Job's storage, needs to be defined. The
algorithm described uses this base address to Get or Set a bitfield
variable packed inside a byte array. It needs the following
components to do its job:

mapped bytes - A byte array maps the variables in memory:

uint8_t jobVarsArray[].
enum ID - The particular item to be worked on is determined by
the appropriate enum. In the example above, it would be
EJobsVar::repeatType which evaluates to 8 (the ninth variable since
enums are zero based).
bit sizes - The VarsCharacteristics structure contains the sizes of
each variable. It is used to calculate the total offset a variable
is located at. RepeatType is located at total offset 26, its bit size is 1.

The mapped bytes

<x>Bitfield Storage Framework[AUX]Put bitfields in a structure to access them easily via an offset from the structure's address</x>

and VarsCharacteristics arrays are parameters which functions of the class

BitfieldStorage will use when doing work on a variable identified by the
corresponding enum, such as EJobVars::jobType. The bitfield variables base
address space is the address of the class instance. For example, in
the class Job above, taking the address of the first item status returns the
same address as that of the class Job instance. Another solution, more
elegant, less error prone, and more maintainable, consists in
enclosing the bitfield packed variables inside a structure so that the
structure's address becomes the beginning of the byte array which
holds the data, thus:
 struct JobVars {
bool status: 1;
uint8_t jobType: 3;
…
bool repeatType: 4;
uint8_t repeatNumber: 4;
} jobVars; // struct JobVars
Instead of taking the address of the class instance &job1, or that of
the first variable status, to access data in the mapped byte array, take
the address of the struct &[Link]. Using the structure's address to
get the base address of the bitfield sized variables renders the code
more maintainable. Non bitfield packed variables added before
and/or after the struct will not impact access to the individual bitfield
packed variables.
The 20 pairs of Get/Set functions are replaced by the two BitfieldStorage
class's generalized Get/Set functions. They access the desired variable
via an enum. From a maintenance standpoint, modifying variables in
the class Job variables list gets reduced to modifying the JobVars structure
and the contents of the varsCharacteristics array.
The class is now:
class Job {
public:
Job( ... ){ ... };
...
BitfieldStorage bitfieldStorage( ... ) { ... }
private:
struct JobVars {
uint8_t status: 1;
uint8_t jobType: 3;
…
uint8_t repeatNumber: 4;
} jobVars; // struct JobVars
static JobVarsCharacteristics jobVarsCharacteristics[EJobVars::end];
};
The entire set of Get/Set functions is replaced by two functions defined
in class BitfieldStorage:
bool SetBitfieldStorageItemValue((uint8_t) EJobVars _eVar, uint8_t _valueToSet);
bool GetBitfieldStorageItemValue((uint8_t) EJobVars _eVar, uint8_t& _valueToGet);
These take two parameters: an enum to identify which variable to deal
with cast as a uint8_t and a uint8_t variable to Set/Get its content.
Implementing the Bitfield Storage Framework does have a RAM
usage cost due to memory being required for tables. Given the need
for 16 instances of the Job class, the memory saving is 19x6=96
bytes. Considering the 10 bytes cost of the bit-size table, the net
saving is 86 bytes. I have not included the cost of jobVarsCharacteristics
since these are needed anyway for control and reporting purposes.
<x>Bitfield Storage Framework[AUX]Job class inherits from the Bitfield class</x>The class Job now relies on the bitfield

class's Get/Set functions. These are accessed via a Bitfield instance

added to the Job class. Doing this works; however, it would be more
elegant to have the Job class inherit from the Bitfield class thereby
enabling doing a [Link] instead of doing a job->[Link].
You will find more details on the class BitfieldStorage in the Bitfield Storage
Framework (appendix) (page 261) implemented as a base class
using the Job class above as an example.

6.
Event Storage Framework
The fact that an event
<x>Event Storage Framework[AUX]Predefined true/false conditions stored in 1 bit per event byte array, saves RAM</x>

occurred or not is a yes/no condition such as "Is Sasha home?". He

is either home or not. An application could test for event categories:
radio transmission, program error, etc. If the number of events in a
single category does not exceed 7 or 8, they can be stored in a
single byte as individual bits via bitfields. However, if the number of
events is greater, several bytes are required.
<x>Event Storage Framework[AUX]enum driven Get/Set functions get/set events packed in bytes</x>The governing concept of the

Event Storage Framework is that bytes containing predefined events

be handled as one contiguous memory zone. Thus, if 45 possible
programming errors have been identified, getting or setting event 31
may be undertaken by doing a Get or Set on event 31. Behind the
scenes, event 31 will be located in the 5th byte in a 7 bits per byte
setup, or in the 4th byte in an 8-bits per byte setup. But the
programmer who is using the framework does not need bother with
these details, all he/she should be concerned with is Set or Get event
31.
<x>Event Storage Framework[AUX]256 events (8-bits), 244 event (7-bits)</x>The Event Storage Framework limit is

256 events when storing on an 8-bit basis, 244 events on a 7-bit

basis. This framework resembles the Bitfield Storage Framework in
that the underlying principles are similar: the difference lies in the
size of variables - they are one bit long in the Events Storage
framework. They could be of any size up to 7 bits for the Bitfield
Storage Framework.
The class EventStorage is characterized by two generalized functions
Get/Set, which use enums to get or set the desired event.
class EventStorage {
EventStorage (uint8_t* _eventsArray, uint8_t _registerSize,
uint8_t _varsCount, char** eventLabels);
public:
bool Set(uint8_t _eventID, uint8_t _eventToSet);
bool Get(uint8_t _eventID, uint8_t& _eventToGet);
...
}; // EventStorage
An enum list identifies individual
<x>Event Storage Framework[AUX]enum driven Get/Set functions get/set events packed in bytes</x>

events and a corresponding string labels array. The enum list EProgErrors
below defines program errors along with the corresponding labels
array labelsProgErrors.
enum class EProgErrors: uint8_t {
start = 0,
outOfBounds = 0, badAlloc, illegalDefault, ...
end
}; // EProgErrors
char* labelsProgErrors {
"out of bounds",
"bad allocation",
"illegal default",
...
}; // labelsProgErrors
Implementing the Event Storage Framework requires that the
following be done:

Create enums to identify events.

Create an array of string labels to identify events for reporting
purposes.
Create an array of bytes to store the events. This array is
managed by the EventStorage class.
Define whether the events should be stored on a 7- or 8-bit
basis.
 The example is generic.
<x>Event Storage Framework[AUX]Specific classes could be created to handle distinct event categories</x>

Specific classes could be created to handle distinct event categories,

such as the following, taken from my beehive weighing system:

SMScommands - 8-bits - Identifies events tied to managing SMSs.

Chores - 8-bits - Events related to work between parent and
child.
JoblistErrorCheck - 8-bits - Error reporting when checking job lists.
ProgramError - 7 bits - Error reporting on events which should not
happen such as bad memory allocation, illegal default in a switch
statement, no default case, out of bounds variables, etc. It is 7
bits because errors will be transmitted to the parent by radio.
Programming error events in the parent can be sized to 8 bits
since they do not get transmitted.
radioTransfer - 7 bits - Events from radio communications between
parent and child.

Upon reporting, the application cycles through all the events in all
event categories to determine whether anything out of the ordinary
occurred. See Event Storage Framework (appendix) (page 269) for
full details.

7.
Linked List Framework
A collection of items
<x>Linked List Framework[AUX]Linked list not constrained by item size and number of items; can grow and shrink</x>

may be stored in memory in one of two basic ways: as an array or as

a linked list. Arrays benefit from being simple to use, just index into
the array via a subscript such as myArray[37]. This will access the 38th
item directly (38th because C++ arrays are zero based, 37 being the
38th item). This simplicity incurs a cost. It brings in two limitations:
the number of items in an array is predefined, and each array
element's length is also predefined to be that of the array's type.
A linked list is considerably more flexible. It can grow and shrink.
Heap space is the limit as to the number of items linked one after the
other. Storing variable length items is an added benefit. Linking items
together is based upon tacking to each item a pointer which holds
the address of the next item, thereby enabling forward list traversal.
Start at the beginning and move forward one linked list item at a
time. A pointer to the previous item could also be included, thereby
enabling backward list traversal.
Any class can be transformed to support linked list storage. Add a
private variable next, a private static first, and public functions GetNext() and
SetNext(). Notwithstanding the ease whereby classes can be made to
support linked list storage, grouping list management into a base
class and having derived classes concentrate on the job they are
supposed to do is a more flexible and maintainable solution. The
linked list part of a class's definition will thereby be physically
separated from the derived class.
Implementing linked lists requires two classes:

<x>Linked List Framework[AUX]Based on LinkedItem and LinkedList</x><x>Linked List Framework[AUX]LinkedItem::GetNext traverses linked list forward 'till

The LinkedItem class handles items to be linked

desired item is attained</x>

together for storage purposes. It adds a private variable next to

an item to be stored in a linked list. It has one function: GetNext()
and one variable LinkedItem* next.
<x>Linked List Framework[AUX]LinkedList class: GetFirst, GetLast...</x>The LinkedList class manages the

list. It has three functions Append, GetFirst and GetLast and one
private variable LinkedItem* first. These enable the list to be
traversed forward. It also has print list functions.

The LinkedList class can be minimal - cycle from the head of the list
and traverse it forward. Lists can be enhanced to enable backward
traversal with previous added to the LinkedItem class. They can be further
enhanced by adding InsertAfter, Remove, Sort...
I used the Linked List Framework in my application for:

Memory management - list of

<x>Linked List Framework[AUX]Memory management and error reporting</x>

objects which trace memory allocations and releases.

Error reporting - list of objects which contain events such as
 warnings and errors.

Linked lists may be used to store lists of just about anything.

Implementing them as base classes to tack to items renders them
extremely flexible and easy to maintain and debug. See the Linked
List Framework (appendix) (page 270) for implementation details.

3.
Specialized Frameworks
Specialized Frameworks groups
<x>Specialized Frameworks[AUX]Seven framework toolkits to accomplish sundry tasks</x>

seven frameworks such as testing, reporting, memory usage, error

handling and more. They group frameworks which may neither be
categorized as Organizational Frameworks (page 151) nor as Data
Handling Frameworks (page 158).

Algorithm Test Framework (page 174) - An algorithm may be

visualized as being a decision tree: a hierarchy of if statements
which define work to be done. When creating an algorithm,
undisciplined programmers tend to do incremental
programming. They jump straight from thinking to code (I place
myself in this category). It is not necessarily the most efficient
path. Logic should be worked out on paper with flow diagrams
and then coded and tested. The Algorithm Test Framework
proposes such a scheme, it is a formal procedure which saves
on development time.
The Class and Function Names Referencing Framework (page
١٧٦) contains the mechanics which enable using enums to
access class and function name arrays for reporting purposes.
This requires that each class and function be identifiable via an
enum based ID.
Memory Management Framework (page 179) - Memory is a
precious resource - knowing how much remains is vital. This
may be achieved via the function TotalHeap, but it is of little use
when the heap size becomes critically low or highly
fragmented, except for triggering an abort to avoid a crash. It is
important to know what consumes memory and whether it is
properly released. This framework contains mechanisms to
report on memory use as the application runs, thereby
enabling the programmer to pinpoint memory hogs and act
accordingly.
Error Reporting Framework (page 185) - The adage is: Do not
assume anything! Always verify! If, after verifying a variable's
value, a problem is detected, it must be reported. This is where
the Error Reporting Framework comes in. It logs errors and
warnings as they occur for immediate or later reporting. After
reporting the error, The application must handle it; this is where
pseudo exception handling comes in (next paragraph).
Pseudo Exception Handling Framework (page 183) - Error
handling starts with detecting and reporting errors - deciding
what to do once an error has been detected is the second
issue. C++ proposes exception handling to handle errors. It is
a powerful and versatile construct but unfortunately not
supported by the Arduino environment. But the Arduino
environment does support the setjmp/longjmp mechanism which
the Pseudo Exception framework uses to mimic C++'s
exception handling.
Operator overloading (page 186) - Logging memory allocations
events records how the application uses memory. Such
recording could be done by functions, but C++ provides a more
elegant alternative. The new and new[] operators and their
corresponding deletes can be overloaded to incorporate event
logging and other tasks.
Misdoings (page 117) - Newbies and tired seasoned
programmers will invert = and == in if statements, and commit
other such seemingly banal misdoings that lead to pernicious
bugs, awfully difficult to detect. The Perl program
[Link] helps detect some of these.
These frameworks are described in individual framework sections
below; and, for some of them, in more detail in the appendix
(Specialized Frameworks (appendix) page 272).

1.
Algorithm Test Framework
<x>Algorithm Test Framework[AUX]Formal procedure to test program logic; logic and work done are distinct concepts</x><x>Algorithm Test Framework[AUX]Start with flow

At some
diagrams to peg down algorithm's logic</x><x>Algorithm Test Framework[AUX]Define execution paths and controlling parameters, then fill in real code</x>

point during the development of my beehive weighing system, an

algorithm's complexity led me to develop a methodology which I
have since developed into a framework: the Algorithm Test
Framework. This framework's underlying mechanisms rely on
identifying distinct execution paths and testing that the
corresponding control variables do indeed follow the desired paths.
Designing an application should begin on paper. Describe what the
application should do. Define its characteristics and behavior. Draw
flow diagrams. When the result is considered mature enough, start
coding. Professional teams which create large applications have set
up complete procedures ranging from initial analysis to final delivery.
We amateur programmers will often sidestep this process and dive
straight into coding an application without first planning things on
paper, or we perhaps limit ourselves to a back of the envelope
design. This was my case. I dove straight into writing code as the
program logic seemed clear enough to me, or so I thought. At first,
things went smoothly but soon enough, getting a piece of code to
work correctly proved frustrating. I spent considerably more time
debugging code than creating it. I had to stop coding and lay things
down on paper to get the algorithm to function properly.
<x>Algorithm Test Framework[AUX]Process: decision tree several levels deep</x>Think of an application as being a

series of processes, a process being a unit of work such as get

sensor values, transmit data, prepare and send an SMS. Seen from
afar, an application may be characterized as being linear. Do one
process, then do the next process, and so on. A process, however,
may be a little complex in that it has an entry point into a decision
tree which may run several layers deep. The nodes of the tree are if
statements, either do this or do that, this is the process's logic. The
branches may lead to work-to-do end points or to other if statements.
Such a decision tree may be characterized by its level - the depth of
if statements. A 1-level tree has two end points. A 2-level tree has
four end points, a 3-level tree has 8 endpoints, and so on. An N
levels tree has 2N end points. These end points may otherwise be
referred to as execution paths since it takes a unique combination of
control variable values to get there. The logic is represented by the
tree and the corresponding control variables. In such cases, creating
a flow diagram, i.e., laying the program's logic down on paper, can
be a crucial and essential step getting code to work correctly.
Logic and work done are two distinct concepts. For example, in the
conditional statement
if X equals 0; do A otherwise do B
the algorithm defines a single-level deep decision tree. do A and do B
are work done. if X equals 0 is logic. The key point is that logic and work
done are distinct. This means that when developing logic, use
dummy place holders for work done. When logic is deemed correct,
plug in the real work. Logic testing means checking that control
variables, X in the example above, do indeed produce the desired
results. The example is trivial but remember that decision tree sizes
are exponential. If there were three levels, there would be eight
possible execution paths.
In my experience, logic testing is a round-robin process: draw a flow
diagram --> define execution paths --> code --> check execution and
start again. Do this until the algorithm yields satisfactory results. It is
relatively easy to test that the logic does indeed do what it is
supposed to do. All it takes is a little organization. Nail down the big
picture before filling in the details. When the logic has been
validated, replace dummy work with real work.
I propose the following methodology:

Start with flow diagrams; work on paper; and think out the logic
with appropriate if/switch statements. Remember that the switch
statement is practically the same as a cascading set of if...else
statements.
Identify execution paths; logic can be tested via a skeleton
 decision tree which simulates the various execution paths the
application may follow - test them with appropriate #ifdefs, one
per execution path. For each execution path, set the
appropriate variables to control program flow. The number of
execution paths increases exponentially with the number of
levels.
Define the controlling parameters - Remember that the number
of possible execution paths increases exponentially with the
number of control parameters: two execution paths require 1
control variable; 4 paths require 2; 8 paths require 3; and so
on. The number of execution paths is exponential, it is a
maximum of 2N where N is the number of levels. The number
of control variables is the number of levels. Create place
holder work items to verify conformance with the flow diagram.
Test - After defining all possible execution paths, create/test
individual path scenarios. This should be done in a skeleton
program, not in the application being developed. Next step,
transfer the code to the application and test the different
scenarios with real live variables.
Fill in the real code - Once the logic has been validated, fill in
the real code which is to do the work and test it.

You will find a complete example taken from my beehive weighing

system in the Algorithm Test Framework (appendix) (page 272).

2.
Class and Function Names Referencing
Framework
<x>Class and Function Names Referencing Framework[AUX]Identify classes/functions via IDs</x><x>Class and Function Names Referencing Framework[AUX]Framework used

When doing any kind of event reporting (memory used and

for event reporting</x>

errors detected), it is vital to identify the event's origin - class,

function, and line number. A first approach to identifying events relies
on defining a variable char* ClassName in the class's definition, and a char*
FunctionName in the function's definition. These would be used for
reporting. When I first did this, reporting worked but memory
requirements were excessive. Strings to identify classes and
functions consume a lot of memory.
My application is a parent/child system. The parent runs on a
ATmega2560 which has ample memory capacity to handle strings to
identify classes and functions, however, the children run on smaller
ATmega328Ps. Since they have limited memory, a string-based
solution to identify where events come from could not be used.
The solution to identifying events without consuming much memory
is based on using enum based IDs - class IDs and function IDs to
identify an event's origin. These enums would be used by the reporting
system to transpose IDs into corresponding class and function
names. The children report with IDs thereby enabling class/function
identification without the memory overhead of strings. They transmit
the event which contains the class and function IDs to the parent
which then uses these to report with full names - this supposes that
the parent has lots of memory because it resides in a larger board
(ATmega2560).
The mechanism underlying converting IDs into names relies on
tagging classes and functions with a unique ID (enum) defined in an
enum class list and in class specific enum function lists.
The mechanics of the Class and Function Names Referencing
Framework requires the following:

An enum list identifies classes.

For each class, an enum list of functions and an array of names
identify the class's functions.
A struct StructCandFnames array groups a class's name info. It
contains a char* className and a function names arrays char*
functionNames[]. This array contains as many rows as there are
classes in the application.
struct StructCandFnames {
char* className; // pointer to a class name
char* functionNames[]; // An array of function name pointers
}; structCandFnames[(uint16_t)EClasses::end];
Names can be accessed directly by indexing into the structure's
contents. For example, to get the class name and its corresponding
function name from enum based IDs, do:
char* className = structCandFnames[classID].name;
char* functionName = structCandFnames[classID].functionNames[functionID];
The problem with using these arrays directly is that the validity of IDs
has not been verified. The CandFnames class has been specifically
designed to retrieve names from IDs and to verify that class and
function IDs are valid and coherent. Using the class's CandFnames Get
functions with the respective IDs follows:
CandFnames* cAndFnames = new CandFnames();
...
char* className = cAndFnames->GetClassName(classID);
char* functionName = cAndFnames->GetFunctionName(classID, functionID);
The system requires that IDs be inserted into source code class and
function definitions so that events could use them and properly
identify their origin. The class Employee, for example, would be
modified to contain a private variable for the ID and its corresponding
Get function:
class Employee: ...
{ // Collapse braces
// BEGIN insert Class Names framework
#ifdef USE_CLASS_NAMES
private:
static const uint16_t classID = (uint16_t) EClassID::EmployeeID;
public:
uint16_t GetClassID() { return classID; }
#endif // USE_CLASS_NAMES
// END insert Class Names framework
} // Collapse braces
...
};
Functions would, in a similar way, be modified to contain a EFunctionID
enum with the corresponding class level GetFunctionID function. You will
find full code for the class CandFnames and more details in the Class and
Function Names Referencing Framework (appendix) (page 276)
section.
To summarize:

Memory allocations/releases and errors are recorded as

events. Reporting requires that full class and function names
 be used but this consumes significant memory which memory-
tight applications cannot afford.
enums identify classes and functions thereby enabling proper
event identification when logging them in memory-tight
applications.
The application which handles reporting uses enums as indices
into the class CandFnames to extract names from corresponding
arrays.

From a practical perspective, manually adding the required enum lists,

structure items and adding class and function IDs to source code is
feasible for small applications (10 classes, 50 to 100 functions). If the
application is large, doing the required work manually becomes
overwhelming and error prone. It needs to be automated - this
process is referred to as instrumentation. I started work on a Perl
program to generate the support files and insert into source code the
necessary class and function IDs. See the next section which deals
with inserting the code (instrumenting ) to accomplish the work and
Class and Function Names Referencing Framework (appendix)
(page 276).

3.
Class and Function Names Referencing
Framework instrumentation
Reporting errors and
<x>Class and Function Names Referencing Framework[AUX]Automate class and function IDs insertion</x>

events requires displaying class and function names which triggered

these, thus, each class and function should be able to identify itself.
This can be done via a string in the class's or function's definition, or
by a unique ID which may be used to pull out the name from a string
array. The process of inserting such IDs into classes and functions is
referred to as instrumenting code. It can be done manually, or via a
Perl program.
The Class and Function Names Referencing Framework needs:
 Class and
<x>Class and Function Names Referencing Framework[AUX]Instrumentation synchronizes class and function enum lists with names</x>

function enum lists and corresponding names arrays.

IDs inside classes and functions index into the string arrays for
reporting purposes.

Since any sizable program will have many classes and functions,
manually creating the contents of the enums and class/function names
is time consuming and terribly error prone. Perl is the ideal tool to
automate the process since it can extract data from source files,
format the output, and generate support files. But, before delving into
the details of the Perl program to automate the process, here is a
brief review of what is needed so that reporting can use class and
function IDs to generate full names.

A CandFnames.h file which contains (1) the class and

structure StructCandFnames declarations, (2) one class enum list and
one function enum list per class plus extern declarations for the
corresponding char* function name arrays, and (3) the
StructCandFnames array.
A [Link] file which contains (1) the struct items (class
names and corresponding function name arrays) and (2) the
individual one per class function name arrays.
All class and function definitions in .cpp files should be
modified to incorporate IDs which the reporting system uses to
identify an event's origin.

The content of the CandFnames.h file is structurally identical to that

of the [Link] file, the enums defined in the CandFnames.h
file are indices into individual array items in the [Link]
file. Note that the code is encapsulated inside a pair of #ifdef
_USE_CLASS_AND_FUNCTION_NAMES ... #endif to deactivate its contents if no
reporting is deemed necessary. The framework's algorithm requires
that a #include "CandFnames.h" be incorporated in the framework's
declaration files.
Instrumenting the project files is undertaken by the [Link]
Perl program. As it runs through the source code, it recognizes
classes and functions and thereby builds an array containing these.
It becomes the raw material to create enum lists, string arrays, and
other framework needs. The difficulty centers around cycling through
source code to find classes and functions. See Class and Function
Names Referencing Framework instrumentation (appendix) (page
251).

4.
Memory Management Framework
<x>Memory Management Framework[AUX]Memory allocations verify heap availability and reveal memory leaks</x><x>Bugs (possible causes and cures)[AUX]Strange behavior

You are running your code and suddenly, strange

or crash - probably ran out of memory</x>

things start happening: the application restarts, again and again; a

function does not seem to be called; values are out of whack;
gibberish gets printed. Something stops working whereas it worked
before!!! What is going on? Enter panic thinking… You may be
running out of memory! Look at the build size displayed in the
compilation output window. If it is a little large (in my case greater
than 77% of RAM use on a ATmega2560), the program may be
running out of heap space and crash. Unfortunately, the crash may
occur long after a bug began its mischief. That is because the crash
occurs when the damaged code gets used, not when the code gets
damaged.
<x>Bugs (possible causes and cures)[AUX]Strange behavior or crash - probably ran out of memory</x>All sorts of things could

cause a crash: array overruns, bad pointers, and other such niceties.
But one of them, running out of memory, could be the culprit. If this is
the case, two issues need to be dealt with when trying to diagnose
the cause of the crash:

Confirm that there is a memory problem - Did a malloc fail? Or

did the heap and the stack collide? This must be ascertained.
The Memory Management Framework contains tools to assist
you in this endeavor.
<x>Memory (leaks)[AUX]Memory Management Framework discloses memory gluttons and memory leaks</x><x>Memory (stack)[AUX]Stack overflow thrashes

memory allocations, may go undetected a long time</x><x>Memory (stack)[AUX]Memory allocation failure avoidable with memory monitoring functions</x>
 Having confirmed that
<x>Memory (use)[AUX]Memory allocation failure avoidable with memory monitoring functions</x>

there is a memory problem, the second question is "What is

the cause? Are there too many function calls hence excessive
stack requirements? Or are there memory gluttons? The latest
new/malloc allocations may not necessarily be the cause. The
problem could have been a memory leak (aka glutton - failure
to release memory) much earlier on, or excessive memory
demand by a class or function. Stack growth due to deep
function nesting could also be the cause, particularly if you
implement recursion. Or local variables may have been too
greedy; they may require excessive quantities of memory.
Remember that the heap's size shrinks upward from the
bottom due to memory allocations and that it extends
downward from the top due to function call nesting. Stack
frames get piled down one under the other as functions get
called. Since the stack and heap are competing for the same
memory space, they could very well meet. This is when
problems occur.

The response to the two issues

<x>Memory (heap fragmented)[AUX]Has holes from random memory releases</x>

above (memory allocations from the heap and stack growth from
function calls) requires that (1) available memory be monitored for
allocation and function call purposes and that (2) all memory
allocations and memory releases be logged somewhere to examine
how memory has been used. That is what the Memory Management
Framework does. It monitors memory use as objects get allocated
and released from and to the heap. These events are logged into a
linked list then used for reporting on the application's memory use.
<x>Memory (use)[AUX]Bottom of RAM (system RAM) used for globals, static, system variables</x><x>Memory Management Framework[AUX]Heap has two parts: fragmented

and contiguous heap</x><x>Memory (heap contiguous)[AUX]Contiguous heap between top of allocated memory and bottom of stack</x><x>Memory (heap contiguous)

[AUX]Functions: TotalHeap, FragmentedHeap, ContiguousHeap, MaximumPossibleAlloc</x><x>Memory (heap fragmented)[AUX]Functions: TotalHeap, FragmentedHeap,

Before
ContiguousHeap, MaximumPossibleAlloc</x><x>Memory (stack)[AUX]Function nesting and recursion consume too much contiguous heap space</x>

delving into the details of memory management, here is a quick

refresher on an ATmega's microcontroller memory structure.
The microcontroller's memory (RAM) is divided into two parts, from
the bottom-up: unavailable memory (system defined at start up), and
available memory which contains the heap and the stack. As the
application runs, function calls consume available memory (stack)
top-down, thereby reducing the heap's size. Allocations (malloc and
new) consume available memory from the bottom-up (heap). If the two
were to meet, the heap growing upwards, the stack growing
downwards, memory would no longer be available. The application
would not function correctly or crash. Please note that the stack and
the heap do not shrink the same way. When functions exit, they
release memory from the bottom of the stack upwards. This means
that the stack consumes and releases contiguous space. This
process, unlike memory allocations from the heap, does not
generate holes. When memory is released via a free or delete, memory
gets released from the first allocation, from the last allocation, or
from any one in between; there is no particular sequence. After a
while, the heap gets divided into two parts: the bottom part which
contains holes (fragmented heap) and the top part under the stack
(contiguous heap.) These concepts are illustrated in the table below,
taken from the companion book Pragmatic C++ Arduino
Programming.
Stack Stack Stack
main, setup,
main, main, setup,
Aardvark
setup other functions
other functions
available
RAM contiguous contiguous
heap heap fragmented
fragmented heap heap
global data global data global data global data
Crash
Powerup Startup Execution
stack and heap collide
Table 16.2 - <x>Memory (use)[AUX]Memory used as the application runs</x>Memory used as the
application runs.
As the application runs, the two memory management issues
become:

Is there enough
<x>Memory (heap contiguous)[AUX]Function calls require contiguous memory space for stack frames</x>
 contiguous heap space to accommodate the intended function
call? If not, the function will thrash the top allocations. The
problem will manifest itself when the thrashed data gets used,
not when the thrashing occurs.
<x>Memory Management Framework[AUX]Enough contiguous heap? Hole large enough?</x>Is contiguous heap space

large enough, or is there a hole large enough to accommodate

the intended allocation? If not, the returned pointer gets set to
0. This should not pose a problem if the programmer checked
allocation calls to ensure that they were successful.

The programmer must manage memory to render the application

robust. If the application runs out of memory, it cannot continue. The
programmer should prepare for this so as to not get caught
unawares. The following three sets of tools provide the means to
handle memory management issues:

<x>Memory Management Framework[AUX]Functions: TotalHeap, FragmentedHeap, ContiguousHeap, MaximumPossibleAlloc</x><x>Memory (heap total)

[AUX]Total heap = fragmented plus contiguous heap; reveals total available memory</x><x>Memory (heap fragmented)[AUX]Largest possible allocation: either

Memory monitoring functions - The functions

largest hole or contiguous memory</x>

TotalHeap, ContiguousHeap, and FragmentedHeap can be inserted

anywhere to monitor memory and the heap's fragmentation
severity (total = fragmented + contiguous). When doing
allocations while fragmentation is high, total heap may seem
adequate when, in fact, contiguous heap could be nearly used
up and fragmented heap may not contain a hole big enough to
accommodate the intended allocation. The function
FragmentedHeap sets the MaximumPossibleAlloc parameter which could
be used to check that there is enough space for the intended
allocation or for memory hungry function calls. Note that this
check prior to allocating space is not necessary. It is sufficient
to do the allocation and to check the return pointer's value. If
the allocation failed, the pointer will be set to 0. Act accordingly.
One should do these checks systematically after every malloc or
new. There is no such safety net when calling a function. If a
called function requires more space for its stack frame than is
 available in contiguous memory, it will silently thrash the top of
allocated memory. You will know that there is a problem when
you try to use the damaged data, which could be long after the
damage has actually been done. The solution lies in checking
contiguous heap space prior to calling memory hungry
functions (easier said than done).
Function calls monitoring - If fragmented heap space were
large, and contiguous heap space low, the programmer may
falsely assume from the total heap reported that all is well. But,
there could very well not be enough contiguous heap space, in
which case, the stack would grow down into the top of the
heap allocations and thrash them. One solution to avoid this
mishap would be to define a minimum contiguous memory
threshold which will guarantee that there is sufficient memory
for the next function call. The ContiguousHeap function may be used
to determine whether this threshold has been reached. See
Determining memory usage (page 297).
<x>Memory Management Framework[AUX]MemAllocEvt and MemAllocList - base classes for managing allocations</x>Memory event

logging and reporting - Events created by the class

MemAllocEvtMgt contain the info required for reporting memory
use. For each allocation, a unique tag is added to the event to
identify possible memory leaks, i.e. allocations which never get
released. Memory events may be logged by systematically
using an overloaded new operator which records such events.
See Operator overloading (page 186).

Implementing memory management consumes memory. It should be

used during the development phase only. Once the application has
been fine-tuned and is pretty well guaranteed not to suffer from
memory leaks or excessive memory requirements, the Memory
Management Framework could be turned off, thereby saving the
memory it consumes.
Summarizing, the Memory management concept is simple:

Allocate memory as needed. Check allocations; if one fails,

 handle the error.
Create and store MemAllocEvts in a list to examine detailed
allocation requirements later. The overloaded operators new and
delete will undertake this task automatically. See Operator
overloading (page 186).
Identify memory allocations with a unique ID to verify memory
releases thereby revealing possible memory leaks. See
Memory Management Framework (appendix) (page 278) for
details.

1.
Memory allocation monitoring functions
<x>Memory Management Framework[AUX]Functions: TotalHeap, FragmentedHeap, ContiguousHeap, MaximumPossibleAlloc</x><x>Memory (heap total)[AUX]Total heap =

fragmented plus contiguous heap; reveals total available memory</x><x>Memory (heap fragmented)[AUX]FragmentedHeap function monitors extent of memory

Memory allocation monitoring relies on three functions: TotalHeap,

holes</x>

FragmentedHeap, and ContiguousHeap (see Determining memory usage page

297). They yield the available heap. If the contiguous heap or
fragmentated heap are below thresholds, the functions trigger an
error. Memory allocations and error handling can be implemented by
the overloaded operator new (see Operator overloading page 186).
The FragmentedHeap function may reveal critical situations due to
cumulative allocations and releases, the result being lots of small
holes. Ultimately, contiguous heap or released allocation holes could
end up being too small to accommodate a moderately sized
allocation request. After a while, all available heap is spread out over
lots of holes, somewhat like Swiss cheese, thereby increasing the
risk of failed allocations due to the biggest hole not being big
enough. A minimum allocation size can be set to stop execution if
the largest hole or contiguous heap got to be too small. The
FragmentedHeap function returns the total fragmented heap. It also
informs on the largest possible allocation.
Memory allocations and releases are implemented by overloading
the two operators new and new[] and their corresponding delete to:
 Allocate/release memory from/to the heap.
Check that memory allocation has succeeded, if not, handle
the error and abort or do something else. This will require
doing a long jump (see Pseudo Exception Handling
Framework page 183).
If the allocation or release is successful, create a MemAllocEvt
described below. See the Memory Management Framework
(appendix) (page 278) for more details.
Prior to calling a memory hungry function, check that
contiguous heap space is large enough by using the
ContiguousHeap function.

See Operator overloading (page 186) for more details on managing

memory, mainly via the overloaded new operator.

2.
Memory event logging and reporting
A second set of tools,
<x>Linked List Framework[AUX]Memory Management Framework logs allocations and releases in linked list</x>

Memory event logging and reporting (the first set being Memory
allocation monitoring functions described above) logs allocations and
releases as events in a linked list containing the necessary data to
produce a report. The class MemAllocEvt contains the means to create
allocation events. The programmer can then cycle through the
events to determine if there is a problem or where to edge out a few
bytes.

<x>Operator overloading[AUX]Overloaded new: memory management, verify allocation success, log events</x><x>Memory Management

Log events - When doing

Framework[AUX]Overloaded new: memory management, verify allocation success, log events</x>

allocations, check the return value of an allocation (the

overloaded new and new[] operators do this). If the allocation is
successful, create and store MemAllocEvt in a linked list. If the
allocation fails, report the error by providing info on what
caused the problem and where (class, function, and line
number). Then, handle the error: abort or do something else.
 See Error Reporting Framework (page 185).
<x>Memory Management Framework[AUX]Memory allocations verify heap availability and reveal memory leaks</x><x>Memory Management

Display results as a list

Framework[AUX]Memory Management Framework discloses memory gluttons and memory leaks</x>

of allocation/release events as the program unravels and/or

just prior to the program's exiting. Examine the complete
memory allocation report to determine what, where, and when
there might have been a problem (memory leaks) or room for
memory optimization.

Each MemAllocEvt event contains the following data:

Class and
<x>Class and Function Names Referencing Framework[AUX]MemAllocEvt uses function IDs to report with function names</x>

function indices - These are used by the Class and Function

Names Referencing Framework (page 176) to produce reports
with meaningful class and function names.
The source code line number defines where the allocation or
release occurred.
The number of bytes allocated or freed is tallied to determine
how much net memory remains from allocations and releases
when the program ends. Such memory accounting reveals
whether there are outstanding allocations which have not been
released; these are referred to as memory leaks.

<x>Memory (use)[AUX]Memory allocation failure avoidable with memory monitoring functions</x><x>Memory (heap contiguous)[AUX]Function nesting and recursion consume

Available heap
contiguous heap space</x><x>Memory (stack)[AUX]Function nesting and recursion consume too much contiguous heap space</x>

space reveals the available memory situation. It reflects both

memory allocations and releases, and stack growth and shrinkage.
The stack could be consuming excessive heap space from too much
function nesting, from significant local variable space requirements,
and/or from excessive recursion.

3.
Function stack frame memory requirements
<x>Memory (stack)[AUX]Hard code function memory requirements to monitor stack needs</x><x>Memory (stack)[AUX]Function nesting and recursion consume too much
contiguous heap space</x><x>Stack frame[AUX]Functions require contiguous heap space for stack frame</x><x>Memory (heap contiguous)[AUX]Function calls require

So far, the discussion has focused on managing

contiguous memory space for stack frames</x>

allocations and releases based on functions which calculate

fragmented and contiguous heap. These can be used to monitor
memory thereby issuing alerts when a critical condition occurs,
which imposes preventive measures.
When you allocate memory, check the memory address returned by
malloc or new. If it is 0, the allocation failed, and you must act
accordingly. No harm done. When you call a function, a stack frame
gets created; it is placed in the contiguous heap's top. If something is
there already, too bad, the stack frame will overwrite it and the
damage is done. The solution is to check contiguous heap space to
ensure that there is enough memory prior to calling the function.
Unfortunately, I have not found a way to automate function memory
needs prior to their being called. Sure, you could build a table of
function stack frame sizes and use it to verify if contiguous heap is
large enough, but this is unwieldy, time consuming, and a
maintenance nightmare. The alternative is to set a threshold. If
minimum contiguous heap happens to be less than the threshold,
you can abort or do something else.
Pragmatic C++ Arduino Programming describes an Arduino
program's memory structure; function stack frames are part of it. In
nutshell, function stack frames contain the function's return address,
its parameters, its variables, and its return value. You could
incorporate into each function which requires substantial memory a
SizeOf function which would report on its stack frame size.
Unfortunately, their values cannot be generated automatically; they
must be defined individually function by function; since the function's
structure is known, it's stack frame size may be calculated. I placed
this mechanism in my TODO list. See Table 16.2 Memory
Management Framework (page 179).

5.
Pseudo Exception Handling
Framework
<x>Pseudo Exception Handling Framework[AUX]Alternative to C++'s exception handling based on setjmp and longjmp</x><x>Exception handling (C++)[AUX]Pseudo Exception

As
Handling Framework - alternative to C++'s exception handling</x><x>Golden rules[AUX]Validate function parameters and return values; never assume anything</x>

mentioned in the section Good programming practices (page 129),

the golden rule is: Never assume anything! The application should
always test a value to ensure that it lies within acceptable limits. If a
test returns false, the application must handle the problem via an error
handling mechanism which must:

Do reporting - The
<x>Validate data[AUX]Upon detecting error, undertake reporting and decide what to do next</x>

application's user or programmer should be informed that an

error occurred along with pertinent info: What error? And
where?
Handle the situation - An error having been detected,
determine its severity. Either issue a warning and continue
after making adjustments or issue a critical error and abandon
processing; i.e., abort or bypass a chunk of code.

<x>Pseudo Exception Handling Framework[AUX]setjmp (try) sets landing point</x><x>Exception handling (C++)[AUX]try establishes landing point, throw returns back to landing

There are several solutions to handling critical errors,

point, catch handles the problem </x>

the preferred one being C++'s exception handling mechanism

(try/throw/catch). Its underlying concept is based upon transferring
program control from the error's location backwards to a landing
point, a return back to a safe location. The problem is that the
Arduino environment does not support C++'s exception handling but
it does have the setjmp/longjmp mechanism. These are equivalent to a
long application-wide goto. setjmp defines a landing point; longjmp is a
return back to the landing point. This mechanism provides the
means to mimic C++'s exception handling. I named it pseudo
exception handling.
C++'s exception handling mechanism is a three-stage process: try
some piece of code (try - this is a landing point). If deep down in the
function calls beyond the try, something fails, do a throw to abort
processing and return back to the try's location (landing point) and
handle the problem (catch via one of the several catches attached to the
try). Each throw is tied to a corresponding catch via their respective
parameters. A throw is a long return to one of the catches attached to
the try.
The key to C++'s exception handling is a long jump. The application
transfers control backwards to the try landing point. Since the Arduino
environment provides the long jump feature, it becomes a simple
matter to create a pseudo exception handling mechanism by using
C++'s built-in functions setjmp and longjmp.

int setjmp(jmp_buf _env) is

<x>Pseudo Exception Handling Framework[AUX]longjmp (throw) - return landing point</x>

equivalent to a try in C++'s exception handling. It defines a

landing point for a corresponding longjmp. It returns 0 if no longjmp
gets triggered. If a longjmp is triggered, setjmp returns whatever
value the longjmp has set, thereby revealing which longjmp triggered
an exception.
<x>Pseudo Exception Handling Framework[AUX]EErrorID identifies longjmp</x> void longjmp(jmp_buf _env, int _eErrorID) -
longjmps could be triggered, all of which should land in their
corresponding setjmps. The longjmp's jmp_buf env identifies which
setjmp (landing point) to return to. The longjmp's int _eErrorID
identifies the error. Note that several longjmps could be tied to a
given setjmp (landing point); they are differentiated from each
other by their respective EErrorID. Each distinct EErrorID could be
defined as an enum which identifies the error.
jmp_buf defines the environment when setjmp gets called. It
distinguishes one setjmp (landing point) from another so that
longjmps will land in their corresponding setjmps.

To summarize: C++'s longjmp mechanism relies on one structure

(jmp_buf) and two functions (setjmp and longjmp). Implementing the
mechanism is a four-step process, two for setting it up and two for
obtaining results.

Define landing point (setjmps are equivalent to trys). The setjmp

analogy with C++'s exception handling is one setjmp/jmp_buf pair
for each try/throw pair you would have created.
Create one global jmp_buf env variable for each landing point
(setjmp). C++'s exception handling does this automatically.
 Trigger a jump back to the landing point - Deep down inside
the call stack, if an error is found, trigger a longjmp(env, errorID) with
the appropriate jmp_buf env to identify the landing point, i.e., which
setjmp to return to with the enumError so that the error could be
identified and acted upon. A longjmp and C++'s exception
handling throw are equivalent.
Get the error identifier errorID = setjmp(env). A value of 0 indicates
that no longjmps have been triggered (i.e., no errors detected). If
an error did get triggered, proceed accordingly. Obviously, the
value of longjmp's parameter errorID should never be 0. The
programmer should be careful when defining the enum EErrors list.

The above explains the long jump error handling mechanism - it

mimics C++'s exception handling. Issues remain on what to do when
errors (critical conditions) and warnings (something is not quite right)
have been detected. The difference between errors and warnings is
that an error requires that the application be aborted, or an entire
section of code be by-passed, both requiring a longjmp. A warning
allows the application to continue after doing adjustments. Both
cases require that information be logged somewhere for proper
reporting. This is where the Error Reporting Framework comes in
(see following section Error Reporting Framework). Implementing
this pseudo exception handling 4-part mechanism is a little tricky and
not at all intuitive. See Pseudo Exception Handling Framework
(appendix) (page 281) for more details.

6.
Error Reporting Framework
The previous section
<x>Error Reporting Framework[AUX]ErrorEvt class stores info on event and links them as linked list (ErrorList)</x>

covered the use of the setjmp/longjmp mechanism to mimic C++'s

exception handling by redirecting program flow when an error
occurs. But this is half the problem. The other half concerns
informing the outside world, whether it be the programmer or the end
user, that an error has occurred. It should report the error's nature
and where it occured.
A well-designed application will undertake tests and may trigger
warnings and errors: C++ related errors, logic errors, and runtime
resource problems. The C++ related errors are explained and
explored extensively in the section C++ gotchas in the companion
book Pragmatic C++ Arduino Programming. The errors due to flawed
logic can be as elusive as the C++ gotchas. The third type of error is
insufficient resources. This can include lack of memory or low battery
voltage. As errors go, it is particularly vicious since the program may
crash yet not provide a clue as to why things went wrong. This is
where error reporting can save lots of debugging time.
<x>Error Reporting Framework[AUX]switch default case missing or empty</x>I shall illustrate error reporting with a

common programming error. Imagine that you have written a switch

with several cases. There should always be a default case - it is used to
handle situations not being dealt with by one of the predefined cases.
It is part of the logic, a catch-all to handle unforeseen circumstances.
Clearly, if the switch unexpectedly falls through default, there is a
programming error if the algorithm is such that this should never
happen; it should, under such circumstances, trigger an error. The
application reports an illegal default exit along with function name, line
number, and possibly other info.
<x>Error Reporting Framework[AUX]Handles error logging and reporting</x>During the development phase, every

time something bad happens, error logging should be undertaken. It

involves listing the class, the function, the line number, and the error
type. The Error Reporting Framework does this via two classes, one
to handle error registration and the other to provide error reporting:

The class ErrorEvt contains the event's ID, i.e., class/function and
line number and the nature of the event expressed as an enum
(EError). ErrorEvt inherits from LinkedItem so that it could be stored in
a linked list.
The class ErrorList manages the list of ErrorEvts - Individual errors
(ErrEvt) are appended to the errors linked list. ErrorList inherits
from LinkedList.
 An enum EErrors list along with a
<x>Error Reporting Framework[AUX]enums synchronize events with array strings</x>

corresponding string array enables reports to display errors as

names, instead of displaying them as numbers. The errors report
output would look like this:
class/function: Sample::Constructor lineNo: 345: outOfBounds
class/function: Contact::SetName lineNo: 498: failedAlloc
When faced with a warning or an error, logging the info is vital to
debugging and providing feedback when the application has been
released. Being alerted that there is a problem is an important step
towards achieving application robustness. See Error Reporting
Framework (appendix) (page 284) for more details.

7.
Operator overloading
<x>Operator overloading[AUX]Overloaded new: memory management, verify allocation success, log events</x><x>Memory Management Framework[AUX]Overloaded new:

One of C++'s distinguishing features is its

memory management, verify allocation success, log events</x>

extensive set of operators. It contributes to its being a powerful

programming language, powerful enough to use it to write operating
systems such as Unix and Linux, and mission critical applications.
Not content with the extensive set of operators provided, the C/C++
creators judged that enabling the programmer to customize
operators would enhance the language. Such customization is
referred to as operator overloading.
<x>Operator overloading[AUX]AddHive += illustrates operator overloading</x><x>+=[AUX]AddHive illustrates += operator overloading</x>Behind the

wording operator overloading lies a simple concept. Imagine an

operator as being a function which can belong to a class or which
can be global. Imagine two classes, Beeyard and Beehive. When adding a
beehive to the bee yard, you might do something like:
[Link](beehiveNalini);
The AddHive function takes a beehive and adds it to the bee yard. It
would be more elegant, cleaner, to do:
beeyard += beehiveNalini;
Instead of using the AddHive function, the customized += operator is
created so that it may be used to add a Beehive to a Beeyard.
Customizing such an operator is referred to as operator overloading.
Inside the Beeyard class definition, declare the overloaded += operator
just as if it were a function thus:
void operator += (Beehive& _beehive) { code }
The code is identical to the AddHive function's code. Note that the -=
operator to remove a beehive could be overloaded in the same way.
To create an overloaded operator just write a function which will do
the desired job and replace the function's name with operator xyz where
xyz is the target operator. Taking the example above, the Beeyard
function void AddHive(Beehive &) becomes void operator+=(Beehive &). It could
not be simpler. Instead of doing
[Link](beehiveNalini);
do
beeyard += beehiveNalini; .
When overloading the global new operator, do the same. Start out by
defining a function which does what you want the operator to do:
memory allocation, check allocation validity, register an event, and
more. Thus do:
void* AllocateMemory(size_t _memToAllocate,...) { do work, return pointer or null }
Having created such a function, test it. Once you are satisfied,
replace the function's name with operator new. You now have
void* operator new(size_t _memToAllocate,...) { do work, return pointer or null }
To use it do:
uint8_t* myArray = (uint8_t*) new(50,...); // there, you are good.
There is no fundamental reason for
<x>Operator overloading[AUX]First create function to do the work</x>

overloading the new operator, a function could have done just as well.
I find that the new operator makes coding easier and that when
reading code, it is cleaner and more user-friendly than using the
function AllocateMemory. Here is the full code of the global operator new I
created for my application:
void* operator new (size_t _allocSize, uint16_t _classIndex,
uint8_t _functionIndex, char** _functionLabels) {
#ifdef USE_CLASS_NAMES
[Link](F("Entered global operator new with allocSize = "));
[Link](_allocSize);
[Link](F(" class "));
[Link](structCandFNames[(uint16_t)_classIndex].classLabels );
[Link](F(" function "));
[Link](_functionLabels[_functionIndex]);
#endif // USE_CLASS_NAMES
 void* allocPtr = malloc (_allocSize); // standard allocation
if (allocPtr == 0) { // check that malloc worked
#ifdef _DEBUG_OPERATOR_NEW
[Link](F("Error: ITEM alloc failed"));
#endif // _DEBUG_OPERATOR_NEW
delay(100);
// Abort from here or return
return 0; // return null
} // if (allocPtr == 0)
#ifdef _DEBUG_OPERATOR_NEW
[Link](F("Success: ITEM alloc successful - entering HandleMemoryConstructor"));
#endif // _DEBUG_OPERATOR_NEW
// adjust alloc size to account for MemAllocEvt requirements for pointers
uint16_t allocSize = _allocSize + 2 + sizeof(MemAllocEvt) + 2;
memMgt->HandleMemoryConstructor (_classIndex,
_functionIndex, _functionLabels, allocSize);
return allocPtr;
} // operator new
Please note the use of #ifdefs to control prints (expanded upon in the
next section).

8.
Print-based Debugging Framework
<x>Debugging hardware-based[AUX]Complementary tool relative to print-based debugging, requires special hardware and compatible IDE</x><x>Print-based Debugging

Framework[AUX]Complementary tool relative to hardware-based debugging</x><x>Print-based Debugging Framework[AUX]Display function specific data and errors</x>

<x>Print-based Debugging Framework[AUX]Macro categories: navigation, memory use, critical situation, data values</x><x>Macros[AUX]Milestone macros highlight program's

location</x><x>Macros[AUX]Critical reporting macros triggered when resource runs low (voltage, memory...)</x><x>Macros[AUX]Memory use macros inform on current

All of us have
available memory and heap fragmentation</x><x>Macros[AUX]Error reporting macros print highly visible text to describe errors</x>

used print commands to get info on what an application is doing.

This is print-based debugging. Such debugging is usually
undertaken haphazardly according to needs. After a while, the
application ends up being peppered with prints, commented out or
not. Source code gets cluttered. Furthermore, the programmer will
want to know the context - where the application is. These are
milestones. If critical situations occur, such as insufficient resources;
or, if an error occurs, he/she should be notified. Finally, since the
application may quickly overwhelm the console with screenfuls of
output, print-based debugging should be controllable. In other words,
it should be selectively limited to areas being worked on.
 When undertaking
<x>Print-based Debugging Framework[AUX]Debug control macros turn debugging on/off global, file, and function level</x>

print-based debugging, there should be the right number of prints,

not too many, otherwise the screen will be swamped by pages of
output, and not too few, as it will fail to display pertinent info.
Inevitably, there will be substantial output in the serial monitor
despite efforts at limiting it. Debugging control macros, eye-catchers,
lines of highly visible repeat characters, will help search for the right
info in the long screen output.
Time being the most precious commodity, why waste it with searches
through reams of serial monitor's output. The Print-based Debugging
Framework limits and organizes output to quickly locate the info
which may reveal the bug's cause. The solution is simple, just insert
prints to display whatever info is deemed necessary to understand
what the program is doing. But it has one important drawback:
memory consumption. Since print-based debugging requires a lot of
text to display info in the serial monitor, lots of RAM and/or flash
memory is consumed. During the early development stages, the
application being small with memory to spare, print-based debugging
memory needs may seem inconsequential. But after a while, as the
application grows, debugging memory requirements may take a
significant toll. The application runs out of RAM and/or flash memory.
This being a certainty, the Print-based Debugging Framework was
created to selectively choose, via macros, what gets printed. This
requires that you focus on problem areas to prevent saturating
available RAM and/or flash memory with unnecessary prints.
A large application could have hundreds of functions, all of which
could be a source of bugs. Print-based debugging requires inserting
suitable print statements into code. Insert macros which signal entry
into and exit from functions, macros to display parameter values and
return values, and macros to display pertinent debugging info. If
functions are to be instrumented this way, such memory needs may
exceed available RAM and/or flash memory. To manage these
memory needs, I developed a set of hierarchical global, file, and
function debugging macros. They control what gets printed and
structure the debugging process. These are grouped together inside
the Print-based Debugging Framework. It provides three types of
debugging macros:
 <x>Macros[AUX]Print-based Debugging Framework defines macros hierarchy</x> Debug control macros - The
application's structure may be visualized as being a three-level
hierarchy (global, file, and function) which leads to a
corresponding hierarchy of control macros. Selectively
commenting and uncommenting these controls what gets
printed via #ifdef...#endif pairs, thereby limiting printouts to problem
areas. These macros are grouped according to the
application's structure: global, file, and function levels. The
global-level level macros turn global macros on and off. The
file-level macros do the same at file level. Ditto for the function
level macros at function level.
<x>Macros[AUX]Function entry and exit macros print details upon entering and exiting functions</x>The function reporting

macros display function debugging info: enter/exit a function,

display parameter values on entry, display return values on
exit. The entry macro provides various details when entering a
function: function name, line number, calling function and
corresponding line number, etc. The long version of the exit
macro provides similar info. The parameter values macro
prints parameter names along with their value.
<x>Macros[AUX]Error reporting macros print highly visible text to describe errors</x><x>Macros[AUX]Print-based debugging macros display total, contiguous,

Four
fragmented heap</x><x>Memory (use)[AUX]Print-based debugging macros display total, contiguous, fragmented heap</x>

application wide macro types are provided: milestone, critical,

error, and memory use. The milestone macro reveals a
program's location. The critical macro is triggered when a
resource runs low. The error (warning) macro gets triggered
when something bad happens (value out of bounds, failed
allocation, etc.). The memory-use macro displays memory use.

These three macro groups (debug control, function reporting,

application wide) make up the Print-based Debugging Framework.
You will find extensive details in the Print-based Debugging
Framework (appendix) (page 285).
 17.
Should know tools

Should know tools

Professional programmers should not limit themselves with acquiring
a good programming knowledge of C++; they should learn other
skills. The following are text processing tools inherited from the Unix
environment; they may prove particularly useful.

Regular expressions (regex) (page 191) - How often have you

used a wildcard when searching for files in Windows Explorer,
such as doing a search on *.txt (find all files which end with
.txt). The star '*' is a wildcard, it says any text pattern. This
wildcard concept has been expanded into a powerful
mechanism referred to as regular expression (regex) to enable
searching for just about anything in a line of text. Microsoft
Word does present this possibility in a pared down version
when you activate the Use wildcards. I you have been using
this Word feature, you are somewhat familiar with regular
expressions. But the Windows wildcard mechanism is just the
tip of the iceberg as to what can be done with regular
expressions. VS Code, Visual Studio, and now Arduino V2
support the full features regular expressions engine; why didn't
Microsoft use the full version in Word, I have no idea. In trying
to make its use simpler for non-developers, they made it just
as complex and, what's more, not well documented. Awk and
Perl, introduced below, both support regular expressions.
Awk (page 204) - This text processing tool's main objective is
to grant its user the possibility of doing something to a line of
text if the line contains some specific text found via a logical
statement and/or regular expression (regex) search. Briefly, if
the line contains something you are looking for, do this or do
that to it; otherwise, process the next line. Awk has one
 drawback relative to regular expressions - it does not support
captures (see Regex groupings page 198).
Perl (page 217) - Perl can do anything you can do with Awk
since it also contains a regular expression (regex) search
engine. Perl is also a general-purpose programming language
and, as such, you can draw a parallel between a Perl program
and a C/C++ console application. If you are to do some Awk
type work on text files, consider doing it with Perl instead, it is
a lot more versatile, and you can do a lot more with it. Note
that Perl, unlike Awk, does support captures (see Regex
groupings page 198).

I strongly recommend that you should learn Perl (page 217), Regular
expressions (regex) (page 191) being a pre-requisite to using it. If
you know Perl, no need to learn Awk; you will find more on this
farther down.

1.
Regular expressions (regex)
You might not know this, but chances are you have been using
regular expressions (regex) ever since you started <x>Regex (general)[AUX]Regex engine is a
generalized intelligent wildcard protocol to find/replace text</x>using a computer. When you do a search in

your computer by typing *.txt, you are using a variant of a regex (a

regex would be .*?txt). You were probably thinking in terms of
wildcard searches. You are telling the computer to search for all files
whose name ends with txt. The asterisk-period-txt *.txt is saying:
match anything before the .txt extension. The outcome will be
[Link], [Link], etc. Your using the *.txt in your search
means that you have just created a wildcard search for something.
The Arduino IDE legacy version 1.8.19 does not support such search
features. Arduino V2, AtmelStudio Visual Studio+Visual Micro, VS
Code, and PlatformIO do.
I have so far used the wildcard terminology; it is a simplified subset
of regexes. The question is "What is a regex?". The simplest answer
I could come up with is that it is a text pattern matching mechanism.
It is a set of rules (grammar) which enables searching for specific
patterns in a line of text. For example, search for a ZIP code in an
address, or the date of an email. Note that it is limited to matching
plain text (text which does not contain special characters such as
text in a Word document).
Regular expressions are simple to use and extremely handy. Here is
a simple regular expression example search. Suppose you want to
find all Serial calls except for [Link]. You are looking for [Link],
[Link], etc. Finding these via a search tool requires that you do one
search per [Link]-you-are-looking-for. The problem is that there could
be many Serial calls such as write, read, open, close... You want to find all of
them, yet you want to avoid cluttering output with the [Link] which
pepper your code. With a regular expression, you need do a search
once only. This Serial\d*\.(?!print) regex searches for all serial non-prints.
It is saying: Search for the character sequence Serial, followed by zero
or more digits \d*, followed by a period \. (notice that d and the period .
are escaped with a backslash \), followed by not !, followed by the
word print. Notice the question mark before the not !. This means look
ahead. Make sure that the character sequence print is not there.
Powerful.
<x>Regex (general)[AUX]Many tools contain a regex engine - AtmelStudio and PlatformIO find/replace, Perl, Awk, Word (wildcards)</x><x>Perl (general)[AUX]Powerful C-like

programming language with regex support</x><x>Awk (general)[AUX]Extract info from text files via logical statement & regex</x><x>AtmelStudio (find/replace)[AUX]Supports

Applications which
whole word, regular expressions, past searches</x><x>PlatformIO (general)[AUX]Find/replace supports regular expressions</x>

support searches by way of regexes contain a regex engine. Just as

C++ IDEs need a compiler, Perl and Awk incorporate a regex
engine. It is what gives them their power. Arduino V2 (not the legacy
1.8.19), AtmelStudio Visual Studio+Visual Micro, VS Code, and
PlatformIO incorporate a regex engine in their search/replace tool.
Some editors (Atom) allow search/replace using regexes. As said
above, Word does allow the use of wildcards; they resemble regexes
though a lot less powerful; but note that Word does support regexes
more fully via VBA programming (Visual Basic for Applications - The
Microsoft Office programming language).
Regexes may respond to a multitude of needs, for example:

Intelligent searches
<x>Regex (general)[AUX]Regex engine is a generalized intelligent wildcard protocol to find/replace text</x>
 such as find all Serial statements which do not contain print such
as: [Link], [Link], [Link]... but not [Link] nor [Link].
<x>Regex (examples)[AUX]Perl and Awk program use regexes to extract program documentation from source code</x>Generate

program documentation via Perl and Awk scripts by extracting

comments from source files.
<x>Regex (examples)[AUX]Regex to find = instead of == in an 'if', or == instead of = in an assignment</x>An interesting use

consists in finding matches such as using the assignment

operator = instead of the logical operator == in an if statement. A
similar search finds using == instead of = in assignment
statements.
<x>Regex (examples)[AUX]Uncovers repeat words like the red red fox</x>Find repeat words in a document:

in the red red fox, red is repeated.

<x>Regex (examples)[AUX]Regex used to list enums and #defines, find unused ones</x>List enums, #defines in a

program. Find unused ones.

<x>Regex (examples)[AUX]Generate subset of a file</x>Create a subset of a file based on each

line's content.

Once you get to know regexes, you will find increasing uses for
them. More examples are presented in Misdoings (appendix) (page
293).
This chapter expands regex concepts; these should help you to
quickly get up to speed creating regexes.

Regex terminology (page 193) - As with all disciplines, regexes

possess their own lexicon (a set of terms) and grammar (a way
of doing things). These rules are based on entities which have
names. C++ has classes, variables, functions, operators, etc.
Regexes have characters, metacharacters, alternates, etc.
<x>Regex (general)[AUX]Regex segments combine individual patterns to form elaborate ones</x> Regex primer (page

194) - The various regex building blocks are assembled to

create pattern matching expressions. Simple regex fragments
may be used as building blocks for more complex regexes.
<x>Regex (metacharacters)[AUX]Metacharacters are letters or symbols which become pattern matching operators</x> Metacharacters

and literal characters (page 195) - Algebra uses symbols such

as arithmetic operators, percentage, modulo, and more to
undertake work on numbers. One can speak in terms of an
arithmetic engine which would drive a handheld calculator.
Regular expressions are similar in that they use characters
which are given a special role to undertake pattern matching,
such as the familiar wildcards when you use Word. Learning
how to use regexes is in part learning what these
metacharacters are, do, and how to use them. They are
ordinary characters which get transformed into special
characters based on context. One can speak in terms of a
regex engine which would undertake text pattern matching.
Regex tidbits (page 196) - A few regex segments can serve as
building blocks for more elaborate search patterns.
Regex example - search for enums (page 197) - This section
presents a search for enums via a regex.
<x>Regex (look ahead/behind)[AUX]Ties regex segment to what precedes/follows</x> Regex look ahead/behind

(page 198) - A special mechanism, referred to as look ahead

or look forward and look behind or look backwards, enables a
regex segment to be tied to what immediately follows or
precedes it. This mechanism enables one to search for all
[Link] in which something is anything but a print.
<x>Regex (captures/groupings)[AUX]Grouping (capture) extracts of parens pair content</x> Regex groupings (page

198) - Regexes do more than match text. They also allow parts
and pieces of the found text to be captured and used later on
in the regex (back tracking or look ahead) or by the application
(capture).
<x>Regex (greediness)[AUX]Greediness means grab all it can before relinquishing control (speed); laziness means grab one character only and let the regex continue

Regex greediness (page 199) - Regex segments can

(functionality)</x>

be greedy (grab all it can before relinquishing control) or lazy

(grab one character only and let the regex continue).
Understanding greediness is important as the regex segment .*
(dot-asterisk - search for anything) does not work the same
way as .*? (dot-asterisk-question mark). .* (dot-asterisk) means
anything from here on. .*? (dot-asterisk question-mark) means
anything from here on BUT look at the next character just in
case the next regex segment finds a match.
 1.
Regex terminology
As with most programming tools, regexes have their own
terminology. The following are a few terms specific to regexes:

Regexes
<x>Regex (general)[AUX]Regexes use two types of characters: literal characters (what to search for) and metacharacters (regex operators)</x>

use two types of characters: literal characters (what to search

for) and metacharacters (regex operators). See
Metacharacters and literal characters (page 195).

Literal characters - These would be the letters, digits, and special

characters one uses when writing or coding. Some can have a
special use depending on context, they are then referred to as
metacharacters.
A metacharacter is a literal character which takes on a special
meaning to accomplish a specific task. For example, * means zero
or more times when in the body of a regex, hence it is a meta-
character. It means an asterisk when it is inside a character class
as in [-+/*]. This means look for a - or a + or a / or a *. (See below for
explanations as to what character classes are). Notice also how the
'-' symbol is placed first. When inside of a character class, the -
sign means range, such as in [a-z], it is then a meta-character, but
when placed at the beginning of the character class, it gets
demoted to being a literal character and thereby loses its meta
nature. A character's meta nature is contextual, i.e., it may be a
metacharacter in one context and be a literal character in another.
Metacharacters are the key to using regexes, they are itemized and
explained in Metacharacters and literal characters (page 195).

Match refers to the act of

<x>Regex (general)[AUX]Match - return true if regex is successful, false otherwise</x>

applying a regex to text which returns true if a match is found,

false otherwise.
<x>Regex (metacharacters)[AUX]Escape (backslash \) transforms metacharacter into ordinary character</x>Escape - A backslash

\ before a metacharacter transforms it into a simple character

or may give it a special meaning, such as \s which means
space or tab. When a backslash needs to be demoted to being
a literal character do a double backslash \\, the first \ means
treat the next character literally (escape).
<x>Regex (general)[AUX]Character class [...] defines what characters to search for</x>Character class means that

the current character can be any of the characters enclosed in

square brackets: digits are identified by 0-9, letters by a-zA-Z. For
example, valid characters in a C++ variable's name include
one or more underscores, digits, and letters - the regex which
defines a C++ variable becomes [_a-zA-Z][_0-9a-zA-Z]*. Notice that
there are two square bracket groups (two character class
groups). This tandem is saying that a valid C++ variable name
starts with an underscore and/or letters followed by zero or
more underscores and/or letters and/or numbers. The * sign
means zero or more times (+ instead of * would mean one or
more times). This is an example of putting together complex
regexes from basic regex segments. The process is expanded
upon in the Regex primer (page 194) section below.
<x>Regex (general)[AUX]Character class [...] defines what characters to search for</x>Character class shorthand -

There are a number of built-in short hands to represent specific

character classes.
\d instead of doing [0-9] to specify a digit.
\w instead of doing [_0-9a-zA-Z] to specify an underscore, digit, or
letter.
\s instead of doing [ \t] to specify a space or a tab.
\D means not a digit.
\W means not a letter.
\S means neither a space nor a tab.

Quantifier -? means optional. * means zero or more times. +

means one or more times. A character may be repeated
several times; {٣} means three times. [a-zA-Z]{٣} (three
successive letters, lower or upper-case). {٠,٣} means repeat ٠
to ٣ times.
Alternation means a choice usually represented as items
separated by the OR symbol | inside parentheses ( red | blue | white ).
<x>Regex (general)[AUX]Anchor means look for something at the beginning ^ or at the end $ of the line of text</x>Position or anchor

^ means at the beginning of the line, $ means at the end of the

line. Consider the following regex: ^The.*?fox\.?$ - you can
decompose it as follows: ^The means search a line which
begins with The, is followed by anything .*?, and ends with fox
(the $ sign to mark the end of the line) with an optional end of
line period \.?. Notice the .*? construct; it means any character
(the period), zero or more times (asterisk), lazy search
(question mark). See Regex greediness (page 199).
Group, capture, and back reference - When a regex contains
parentheses, each opening parens represents a group which
can be referred to as $٢$ ,١, etc. to be used by the application
or as \٢\ ,١... to back reference inside the regex. See Regex
groupings (page ١٩٨) below.
<x>Regex (general)[AUX]Regex segments combine individual patterns to form elaborate ones</x>Regex segments - A

regex can be broken up into its components, these would be

referred to as segments. For example, the regex labo[u]?r has ٦
segments: the four letters in labo are four segments, followed by
an optional [u]?, followed by r.
<x>Regex (greediness)[AUX]Greediness means grab all it can before relinquishing control (speed); laziness means grab one character only and let the regex continue

Greediness - Yes, there is such a thing as greediness.

(functionality)</x>

This means that a regex segment will attempt to grab all it can
before handing control over to the next regex segment. A
greedy segment can be made lazy. It modifies how the regex
engine operates. This is particularly important when using the
construct .* which means grab anything zero or more times
(greedy grab). There are situations which require greediness,
others which require laziness. See the Regex greediness
(page 199) section later on for explanations and examples
relating to the mechanics of greediness. I cannot
overemphasize its importance. If you are to use regexes, do
yourself a favor by dedicating time so that you understand the
concept.
 2.
Regex primer
A regular expression is a
<x>Regex (general)[AUX]Regex segments combine individual patterns to form elaborate ones</x>

pattern used by a regex engine to determine whether a match inside

the character string being processed is found. It returns true if the
entire regex matches, false otherwise. It can also return values which
can be used by the application (captured) or used farther down by
the regex as it parses the text (forward or back reference). See the
Regex groupings (page 198) section below.
The simplest regex is a pure text matching statement. Thus, the
regex brown, will search all occurrences of the five successive letters
b, r, o, w, n in a line of text. If the regex brown were applied to the text
below
The little brown fox
it will return true. Had the regex been ^brown, it would have returned false
because it is being asked to find lines which begin with brown. The line
of text above does contain brown, but it is not located at the beginning.
Please note that for convenience's sake, I refer to the sequence brown
as being a word. To be accurate regex-wise, note that brown is a
sequence of five regex segments, each one being a search for a
specific character. I could have rewritten the regex as [b][r][o][w][n]
which means b followed by an r followed by an o... This is what the
regex engine does. It does not look for a word, it undertakes a
search for a match one regex segment at a time, this transposes as
one character at a time in the regex brown.
We could also look for brown foxes instead of brown dogs, so the regex
would be brown\s+fox(es)? which means look for brown, then look for one or
more spaces \s+, then look for fox, then look for an optional (es)?.
Simple so far. A regex is easy to decrypt once you have understood
the vocabulary. Notice in the above the use of ^ to refer to the
beginning of the string. ^ and $ are position indicators also referred to
as anchors. ^ means at the beginning. $ means at the end. These
are, in regex parlance, metacharacters, which brings me to introduce
the type of character concept.
 3.
Metacharacters and literal characters
Regexes use
<x>Regex (general)[AUX]Regexes use two types of characters: literal characters (what to search for) and metacharacters (regex operators)</x>

two types of characters:

<x>Regex (metacharacters)[AUX]Metacharacters are letters or symbols which become pattern matching operators</x> Metacharacters
(aka special characters): \ ^ $. [] ( ) | * +? { }
<x>Regex (metacharacters)[AUX]Literal characters: characters searched for</x>Literal characters are any

character not defined as being a metacharacter or any

metacharacter in a context outside of that which makes it a
metacharacter. If a metacharacter is to be used as a literal
character, it needs to be escaped with a backslash. For
example, the asterisk *, which means zero or more times, can
be transformed into an ordinary asterisk by escaping it \*.

If your search includes looking for a metacharacter, prefix a

backslash \ to the character you are looking for (ex: \$, \., \\, \*) to
transform it into a literal character.
<x>Regex (metacharacters)[AUX]Context defines whether pattern matching character is metacharacter or literal</x>The literal vs. meta

nature of a character depends on its context. For example,

parentheses are literal characters inside a character class,
metacharacter otherwise. Thus ([(]hello[)]), although a little contrived,
means look for a (hello) sequence inside a group. The first parens is a
metacharacter which starts a group, the second parens (inside
square brackets) means look for an opening parens, same with the
closing parens. An alternative would be to escape the parentheses
instead of placing them in a character class; the pattern becomes (\
(hello\)).
Here is a detailed list of metacharacters:

<x>Regex (metacharacters)[AUX]? means preceding segment is optional, also means greedy or lazy when doing a search</x><x>Regex (metacharacters)

The four quantity quantifiers are::

[AUX]Quantifier {1-x} means repeat 1 to x times</x>

* - any or more times.

+ - one or more times.
 ? - optional, not necessarily there.
{1-x} repeat the preceding regex segment, in this case 1 to x
times. For example, a five-digit ZIP code would be [0-9]{5}. An
extended ZIP code would be [0-9]{5}(-[0-9]{4})? - the second
group of four digits is optional.
<x>Regex (general)[AUX]Character class [...] defines what characters to search for</x>[] defines a character class - it

encloses characters to be searched for. For example, [a-zA-Z]

means look for one lower- or one upper-case letter. In the zip-
code example above [0-9]{5} means look for five successive
digits. Another example: Americans differ from the British when
using nouns which end in 'or'. Americans will write labor,
harbor... The British will write labour, harbour... Both are
correct - a regex to handle these would be labo[u]?r, harbo[u]?r. The
? means optional, not necessarily there which is different from
the question mark after a period asterisk (.*?) which means use
lazy traversal (see Regex greediness (page 199) below).
<x>Regex (metacharacters)[AUX]Escape (backslash \) transforms metacharacter into ordinary character</x>The \ escape

transforms metacharacters into ordinary characters or defines

shortcuts such as \d to express a digit or \1 to back refer to
group 1. Example: \* look for an asterisk. It converts the
metacharacter * into the literal character *.
<x>Regex (metacharacters)[AUX]Caret ^ used both as an anchor (beginning of text) and negation in character class</x>The caret ^ has

two uses depending on context - use it as an anchor to match

the beginning of the searched text. For example, ^The means
search for the The at the beginning of the text. The ^ is also
used as negation inside a character class, as [^ ] to mean
neither X, nor Y, nor Z.
<x>Regex (metacharacters)[AUX]$ sign anchors search to end of text</x>The $ sign is an anchor. Look for

match at the end of the string. (fox\.?)$ returns true in The little brown
fox since the string ends with fox or fox. (fox followed or not
followed by a period). Notice that this particular regex allows
for an optional period to mark the end of the line and that the
period is escaped to inhibit its metacharacter nature.
 A period
<x>Regex (greediness)[AUX]Period/asterisk '.*?', period/plus '.+?' followed by question mark means lazy search anything</x>

outside a character class means match any character.

Example: the period at the beginning of the following
sequences .[A-Z] means any character followed by an upper-
case letter. The often used .*? means look for anything zero or
more times, lazy version (see Regex greediness (page 199)
below).
<x>Regex (metacharacters)[AUX]OR symbol '|' means one of several alternatives (choices) inside a parens group</x>The vertical bar |

means OR, i.e., an alternate such as (white|black) looking for a white

or black something.
<x>Regex (captures/groupings)[AUX]Grouping with '|' (OR) lists alternatives</x><x>Regex (captures/groupings)[AUX]Grouping (capture) extracts of parens pair

The parens pair ( ) groups things together when providing

content</x>

alternatives such as (white|black). The opening parens marks the

beginning of a group. It also enables the regex engine to store
a selected value into a numbered buffer - example: In
(white|black), $3 will contain white if white was found, 3 being the third
opening parentheses in the regex pattern. See Regex
groupings (page 198) below.
Spaces - When writing C++ code, spaces are used to enhance
readability. Do not do this when writing regexes. A space is a
literal character just as the letter a is. For example ( == | != )
means look for equal or not equal with a space before and a
space after; this regex will not find a==b because the equality is
not bounded by spaces. A C++ compiler does not care.

4.
Regex tidbits
Here are a few regex segments which respond to particular
purposes:

<x>Regex (greediness)[AUX]ZIP code example gotcha or how laziness solved the problem</x><x>Regex (examples)[AUX]ZIP code example gotcha or how laziness

solved the problem</x> ([0-9]{5}(-[0-9]{4})?) ZIP code - parens pairs are used to
 capture data - see Regex groupings (page 198).
\s* zero or more spaces or tabs, or \s+ one or more.
<x>Regex (examples)[AUX]Extract a variable's name; define floating-point numbers</x>\d+\.\d{4}+ is a four decimals

floating-point number, i.e., one or more digits followed by a

period followed by four digits. In this regex, the period outside
a character class being a metacharacter needs to be escaped
to make it a literal character.
[_a-zA-Z]\w* represents a valid C++ variable name. The first
character should be an underscore or letter but no digits.
Consecutive characters, if any (reason for the * after \w
character), could be underscore, digit, letter. Notice the
shorthand \w instead of the long [_0-9a-zA-Z]
(abc|12345|really neat) - search for an alternate: the string abc, or the
string 12345, or the string really neat.
<x>Regex (examples)[AUX]Greedy vs. lazy search anything</x>.* means anything, greedy version; .*?

means the same, lazy version. Be sure you understand the

difference between .* and .*?. There are searches you could not
do without; not recognizing the difference is a regex gotcha.
See the section Regex greediness (page 199) below.

5.
Regex example - search for enums
Imagine a regex as being a sequence of
<x>Regex (examples)[AUX]Step-by-step example regex to find enums</x>

little building blocks, call them regex segments. Isolate and examine
them one at a time. I shall use some of the above to illustrate a
regex I created to search for enums and their name. The regex
assumes good programming practices and standardized code
formatting conventions. The enum declaration should begin a line and
the declaration should follow up to and include the size qualifier:
uint8_t but not necessarily include the opening brace, which could be
on the next line. Here is a typical enum:
enum class EMyEnumList: uint8_t { list of comma separated items };
The syntax of an enum having been defined, a regex can be created
for each component above and assembled to form a complete regex.
Creating the regex to find such enums starts with the analysis of an
enum's syntax:

The enum keyword is the first word in a line of text. This

assumes that enum declarations start on a newline. The regex
starts with zero or more spaces \s* followed by the keyword
enum, followed by one or more spaces \s+ thus: ^\s*enum\s+
class qualifier (optional) - the enum keyword may be followed by
the optional class keyword followed by one or more spaces thus:
(class\s+)?.
enum names are an assembly of one or more underscores,
upper/lower-case letters and digits (first character cannot be a
digit) [_a-zA-Z]\w*. Since the name is to be captured, the entire
find name regex segment is enclosed in parentheses thus: ([_a-
zA-Z]w*).

There is no need to go further since all we want is the name. The

complete regex to have Perl find enum names becomes an assembly
of the above, as follows:
/^\s*enum\s+(class\s+)?([_a-zA-Z]\w*)/
The begin and end forward slashes / delimit the regex. This is
required by Perl and Awk because the regex syntax uses all
characters, spaces included. The above does look a little cryptic,
however, decomposition, taking pieces one at a time, simplifies its
analysis. Plug it into AtmelStudio, Visual Studio, PlatformIO, or the
Arduino IDE V2 to check the syntax (without the forward slash
delimiters).

6.
Regex look ahead/behind
Look ahead and look behind are
<x>Regex (look ahead/behind)[AUX]Condition search based on existing item</x>

assertions. Look ahead says: Use the following regex segment to

look for text from this position. If found, return true, otherwise return
false. For example, suppose the following line of text:
The little brown fox scrambled through the field. The red fox stayed
behind the fence.
The look ahead regex segment fox\s+(?=scrambled) matches the first fox,
but not the second one. The cursor gets positioned just before
scrambled.
<x>Regex (look ahead/behind)[AUX]Look ahead to find all Serial.??? which are not prints</x><x>Regex (examples)[AUX]Look ahead to find all Serial.??? which are not

Another example: I used the look ahead regex feature (?= and ?!)
prints</x>

to detect all Serial statements which are not prints: Serial\d*\.(?!print). This
is saying find all Serial followed by an optional digit (Serial, Serial3...),
followed by a period not followed by print (?!print).
There is a lot more to lookahead and look behind. They can get
complex. You will find extensive details in the following Web site:
[Link]/[Link].

7.
Regex groupings
Grouping enables specific values
<x>Regex (captures/groupings)[AUX]Grouping (capture) extracts of parens pair content</x>

inside text to be extracted and used later on inside the regex (back
reference) or by the application which contains the regex engine
(capture). It is based upon parens pairs, such as the following regex
skeleton:
(... (..)...)...( ... ).
The skeleton above has three open/close parens, hence three
groups whose values are defined by the contents inside the opening
and corresponding closing parens. Notice that parens may be
nested.
Suppose the following text:
A dark blue background hinders visibility.
and the following regex which contains three groups, one of which is
nested.
.*?((blue|green)\s+background).*?(visibility|appearance)
In the above, \s+ refers to one or more spaces. .*? is the lazy look for
anything.
The three groups yield respectively:
\1 or $1 = blue background
\2 or $2 = blue
\3 or $3 = visibility
The regex above has three groups numbered from 1 to 3 since there
are three open parens, from left to right. The open parens is the
begin marker for a group. In this example, we would have $1, $2 and
$3. These $x are variables which contain the value of the group's
match. The application can use these to carry on with whatever it is
supposed to do if a match is found. If there is no match do not use
these since they are empty. As the regex parses the line of text,
these groups can be used inside the regex to accomplish back
referencing. The semantic changes to \1, \2 and \3. A typical use for
such back referencing would be to detect double words such as this
this. The regex would be (\w+\s)\1. This regex is saying: find any
number of letters or digits followed by spaces followed by whatever
is in group 1, in other words, a repeat. Try it in your favorite IDE. In
the replace part, plug $1 for example, and observe how your original
line gets modified. One additional comment concerning $x variables:
Be careful with $0, it stores the name of the Perl program being
executed. It has no relationship with the line of text being processed.
In Awk, $0 contains the entire line of text being processed. If you
want the line of text in a Perl program, use $ _.
When applying a regex to a line of text, think in terms of a regex
cursor advancing inside the regex one regex segment at a time, and
a string cursor advancing in the line one character at a time. In the
line of text The little brown fox., the regex brown will start at the
beginning of the line and search for b. Once it finds it, it searches for
an r and so on. Once it has found the n, it stops and returns true. It is
up to the application to handle what is does after getting the true/false
result of the regex match.
When using regexes in Perl, there is a subtle uninitialized variable
situation which can crop up during regex captures. Suppose your
regex contains (\w+)? as the third open parens. You are looking for an
optional word which you want to capture into $3. When you do
if($3 ne ""), if text is not found, Perl will issue a verbose warning telling
you that $3 is uninitialized. The reason is that since text was not
found, $3 did not get a value, it is therefore used but not initialized.
The solution lies in providing the regex with an alternate search
which will always be found, not matter what. Thus, change the regex
segment to (\w+|\s*)?, which is telling it to look for a word, or zero or
more spaces. If it does not find a word, it will definitely find zero or
more spaces, $3 will be initialized to either that which it found or to an
empty string "", thereby avoiding the verbose message.

8.
Regex greediness
<x>Regex (greediness)[AUX]Greediness means grab all it can before relinquishing control (speed); laziness means grab one character only and let the regex continue (functionality)

In all of my programming, I have never encountered a concept

</x>

such as greediness. But yes, regular expressions do differentiate

how the regex engine operates as it applies a particular regex
segment to text. It can do so in one of two modes: it is either greedy,
which means that it grabs anything it can; it prioritizes things for
itself; it is being completely selfish. Or, it is lazy; it hands over control
to the consecutive regex segment; it is being polite; it is being
altruistic; it is saying to the consecutive regex segment, would you
care to look at the following character just in case you might be
interested? Please note that greediness is official regex parlance.
<x>Regex (greediness)[AUX]ZIP code example gotcha or how laziness solved the problem</x><x>Regex (examples)[AUX]ZIP code example gotcha or how laziness solved the

Here is an example of a greedy regex whose objective is to

problem</x>

extract a ZIP code from a US address. The regex could be .*\d+. It is

saying look for anything .* followed by one or more digits \d+. This is
what our mind does - we scan the line, look for anything, stop when
a number shows up, then grab as many consecutive numbers as
possible.
To fully understand what the regex engine is doing, add a parens
pair to create a group to enable the application to use what it found
as $1. The same regex with one grouping becomes .*(\d+).
Now, apply the regex to the string Freeport ME 04032, USA. $1 will contain 2,
not 04032, which is totally unexpected - 04032 was expected, not 2. This
is what the regex does:
 Start with the .* regex segment. Have it grab all it can. It grabs
the entire line because .* is telling it to grab anything, including
the ZIP code, the comma, and USA to the end of the line.
Since the second regex segment \d+ has not had a chance to
do its job, the regex engine backtracks the line one character
at a time from the end, until the second regex segment (\d+)
finds something it is looking for. When this second regex
segment encounters the last digit in the ZIP code, 2, it says:
OK, I am good, I found a digit; since I want one or more, one is
good enough. I am done.
Since there are no more regex segments to satisfy, the regex
returns with a success: $1 contains 2.

The result above is surprising as the entire ZIP code was expected.
The explanation is that the engine is optimized for speed. The first
segment grabs everything then backtracks. The grab anything .* is
qualified in this case as being greedy.
<x>?[AUX]Regex lazy search</x>Adding a ? to the greedy regex segment .* transforms it

into a lazy regex segment .*?. The complete regex then becomes .*?
(\d+). The following is the lazy processing sequence:

Apply the regex segment by segment from left to right. It

begins with the first regex segment .*?.
It grabs one character and passes control to the next segment
\d+.
The \d+ segment tests the character, if it fails, it passes control
back to the preceding segment which advances one character.
If the second segment \d+ succeeded (it encounters a digit) it
grabs it and checks the next digit. In the meantime, the first
segment .*? is not invoked since it finished doing its work. The
second segment \d+ continues until it encounters a non-digit.
Then stops.
When the complete regex is satisfied (i.e., both .*? and \d+ are
satisfied), the regex stops and returns true. The first regex .*?
 found a sequence of all possible characters until the second
regex \d+ took over.

$1 will return the entire ZIP code 04032. This back-and-forth control
segment transfer slows down the search process considerably, but it
gets the job done.
The difference between the greedy and lazy behavior of the grab all
.* is that the greedy one processes the entire string then backtracks
giving the second segment a chance to find something. In the lazy
way of doing things, the first segment plods along one character at a
time. It finds something, asks the second segment whether it is
something it wants. If not, the first segment continues plodding along
one character more, control being passed back and forth until the
second segment finds something that it wants: a digit. Once the
second segment gets going, the first segment says, OK, I am done,
you continue. Since the second segment is looking for repeats on
digits, it grabs every consecutive digit it can find, one by one, until it
finds that there are no more consecutive digits. At this point, since
both segments have completed their respective jobs, the regex
returns true with $1 containing the full ZIP code. The greedy version is
fast but does not yield the correct result; the lazy version is slow but
does yield the correct result. A compromise may be found with .*
(d{5}). It is saying forge head; look for anything; when you reach the
end of the line backtrack until you find five consecutive digits. This
solution is fast and gets the job done.
<x>Regex (greediness)[AUX]Find text enclosed in square brackets gotcha, greediness issue</x><x>Regex (examples)[AUX]Find text enclosed in square brackets gotcha, greediness

Here is another example. The regex is meant to select text

issue</x>

enclosed in square brackets.

A [garden] offers great [outdoors] potential.
This is what you might do.
\[(.*)\]
The regex searches for an opening bracket [, then searches for
anything .*, then searches for a closing bracket ]. You would expect $1
to return garden. The parentheses are there to do a grab, they are not
part of the search. The outcome is that it selects everything from the
first bracket to the last bracket, including the text,
$1 returns garden] offers great [outdoors
because once an opening bracket was found, the regex segment .*
grabbed everything it could up to the end of the line of text then
handed control over to the closing bracket segment ] which
backtracked until it found a closing bracket. As it backtracked one
character at a time, it stopped as soon as it found a closing bracket
which happens to be the second closing bracket, not the first one.
That is because the sequence .* is greedy.
Placing a ? after .*, (.*?), converted the regex into being lazy, the
result being that the regex plodded along one character at a time.
The first segment passed control to the second segment which tried
to find a closing bracket; not finding one, it returned control to the
first segment which advanced one character and so on, back and
forth, until a closing bracket was found, which is the first closing
bracket. The outcome being that $1 contains what was expected:
garden instead of garden] offers great [outdoors.
Greediness is an important concept in the grab all .* (any character
zero or more times) construct. Be sure that you understand how to
differentiate the greedy (.*) from the lazy (.*?) usage, as you may use
it often.

2.
Awk vs. Perl
<x>Awk (general)[AUX]Why I started with Awk, then migrated to Perl</x><x>Perl (general)[AUX]Why I started with Awk, then migrated to Perl</x><x>Perl (Awk vs. Perl)

As I progressed on
[AUX]Perl is better than Awk</x><x>Perl (general)[AUX]Powerful C-like programming language with regex support</x>

my Arduino-based beehive weighing system, I needed to use

functions I had created earlier on. I found that I had forgotten what
some functions did, which I had unfortunately not documented. I
wasted a lot of time understanding what I had done. As I unraveled
the code, I inserted comments. While doing so, I realized that if
comments were sufficiently organized, I could pull them out to
generate program documentation with a simple Awk program, a text
processing tool I had used way back in my old Unix days. It was just
a matter of using keywords which Awk's pattern matching
capabilities could use to pull out relevant info. This led me to slowly
create keyword-based templates which would encapsulate inline
program documentation. I started with an Awk program to pull out
template-based documentation from my source code; but, I then
migrated to Perl. See Program Documentation Framework (page
155).
<x>Awk (general)[AUX]Extract info from text files via logical statement & regex</x><x>Awk (rules)[AUX]Process text files with sequence of condition/action-blocks (rules)</x>

In a nutshell, Awk pattern

<x>Awk (rules)[AUX]Program is sequence of condition/action-blocks (rules) and functions</x>

matching makes it an ideal tool to extract info from text files. The
fundamental concept rests on a two-step mechanism: (1) analyze a
line of text for a match against a pattern. If the match is successful,
(2) do something. I named this two-step mechanism a
condition/action-block. In Awk parlance, this is a rule. Processing a
file with Awk thus becomes a simple process of getting lines of text,
one line at a time, and applying all of the rules to the current line.
<x>Awk (rules)[AUX]Condition based on pattern search: simple logical statements or regexes</x>A condition determines
whether a line of text matches a pattern. Here are some examples:
Does the line contain the word PROGRAM? Is there both an opening
parenthesis and a closing parenthesis with optional text in between?
Does the line start with //? Is there a \ at the end of the line? There is
an infinite variety of such questions which are collectively referred to
as pattern matching.
Awk pattern matching conditions can be one of two types: logical
statements or regular expressions (regexes).

The simplest approach to defining a condition is to use a

logical statement such as $1 == "James". It means: do a pattern
match on the first token $1 to check whether the fist word is
James. Awk breaks up the line of text into words (tokens) and
places them into the variables $1, $2, etc. This is automatic
each time a newline is read in. Such a rule could be used to
make a subset of a source file by excluding lines which are
commented out.
A condition may also contain a regular expression (regex) such
as ^\s*//, which means start at the beginning of the line ^, look
for zero or more spaces \s*, then look for two consecutive
forward slashes //. If there is a match, return true. A regex may
 look cryptic at first, but you quickly get used to the syntax and
find that unraveling it is simple. See Regular expressions
(regex) (page 191).

Awk conditions can mix both logical statements and regexes by tying
them together with AND (&&) and OR (||) operators.
If a pattern matching condition on a block of text is successful, work
needs to get done. This is referred to as an action-block - it is a
group of statements contained within a set of curly braces such as
one does when defining a function. Doing work means use
programming statements to obtain the desired result: print a line, set
a state variable, undertake some logic, or store info somewhere.
<x>Awk (general)[AUX]Uses Awk may be good for</x>After invoking Awk with your command line

arguments (program and files), you do not need to open and load the
files or to specifically read lines one at a time. Awk will automatically
cycle through all the lines of all the files and apply the rules. When all
the lines have been dealt with, the program ends. It could not be
simpler. Here are some uses Awk may be good for:

<x>Awk (examples)[AUX]Generate documentation from code comment templates, list macros and enums</x> Program
documentation - Commenting code is a golden rule. You
should write what each function does, what parameters and
variables do, and whatever it takes to clarify how a function
works. Given a well thought out comment structure, it becomes
child's play to pull out comments from the .h/.cpp files. This led
to creating the Program Documentation Framework (page 155)
presented in the Frameworks section, initially developed with
Awk, later on with Perl.
<x>Awk (examples)[AUX]Generate function call reference - who calls who and who gets called by whom</x>Function call

reference - As the number of functions gets large, code

maintenance can be improved by listing cross references of
function calls: who calls who and who gets called by whom.
List macros - My beehive weighing system application is large.
Over time, I have created macros (#define) throughout the
application. After a while, I may have stopped using some but
inadvertently left them in. Awk can find them - it can also list
 macros which have been commented out. With Perl it is also
possible to list unused active macros, but I digress.
Enumeration list - enums, much like macros, may have been
defined a bit everywhere. Listing them is desirable, as well as
those which have been commented out. The same goes for
Perl. It can be used to list unused enums.
<x>Awk (examples)[AUX]List potential misdoings e.g. = instead of ==</x>Error detection - Some hard-to-

find bugs, such as = instead of == in an if statement, or missing

or empty defaults in switch statements, can be easily found. See
Misdoings (page ١١٧).
<x>Awk (examples)[AUX]Print subset of a file</x>Print part of a file - An Awk rule can

duplicate a file leaving out commented lines.

<x>Awk (regex)[AUX]Parens-based captures not supported - Perl does support it</x><x>Perl (Awk vs. Perl)[AUX]Does not support parens-based captures - Perl does</x><x>Awk

(general)[AUX]Learning curve is shallow - nothing to it</x><x>Awk (regex)[AUX]Parens-based captures not supported - Perl does support it</x><x>Perl (general)[AUX]Perl is

better than Awk</x><x>Perl (Awk vs. Perl)[AUX]Perl is better than Awk</x><x>Perl (general)[AUX]Perl supports regex parens-based captures - Awk does

Developing Awk programs to cover some of the above proved to

not</x>

be easy and not particularly time consuming, though I did run into
one major snag. When working on the Documentation framework, I
discovered, much to my dismay, that Awk did not support regular
expression parens-based captures. This refers to using the regex to
customize what gets put into the built-in variables $1, $2, etc. Not
supporting this feature prevented me from undertaking certain tasks.
There were things that I wanted to do which I just could not. I had to
change tack and find another text processing language with a built-in
regex engine which did support parens-based captures. Luckily, I
had looked at Perl long ago and pulled out my old book Learning
Perl. I also watched Derek Bana's YouTube tutorial on Perl (see the
Bibliography - Perl page 334). Since Perl supported the regex
parens-based capture, I migrated my Awk applications to Perl. The
learning curve proved to be short. Perl can do everything which Awk
can and much more, which is why you may want to dedicate time
learning it.
The following pros of Perl relative to Awk stand out:

It is a powerful programming language - It may not stand up to

 the power of a full-fledged C++ program, but it can go a long
way without the overhead of C++ programming. It can handle
programming needs other than text processing. Its capabilities
go far beyond its being a better Awk.
<x>Perl (arrays)[AUX]Perl arrays are associative, can be mix of numeric (float) and string values</x>Perl arrays are

associative - From a C++ programmer's perspective, you must

unlearn everything you know about arrays and relearn how to
define and use Perl arrays. They are powerful; a few lines of
code will accomplish tasks which would otherwise require
many lines of C++ code.
<x>Perl (build)[AUX]Lists enable doing tasks on multiple items in one line of code</x>List constructs, such as ($x, $y,

$z) = (10, 20, 25), are supported. You guessed it: $x gets assigned
10, $y gets assigned 20, etc.
Regular expression parens-based captures - One of Awk's
shortcomings lies in its lack of support of regular expressions
parens-based capture. Perl does support this mechanism.
<x>Perl (Awk vs. Perl)[AUX]Process same set of files several times - Awk limited to processing set once only</x>Flexible file

processing - Awk loads files, processes them and that is it. It

has finished. True, you can use Awk to process a file a second
time via a trick which consists in duplicating the file in the
command line, thus:
Awk -f [Link] fileA fileA

Awk will load fileA twice and concatenate them together. With Perl,
you can process files individually umpteen times. This feature
means that a single Perl program can run through a set of files time
and time again to respond to different needs.
The above are good reasons for adopting Perl. But, be warned, if
you are a C++ programmer, you will run into oddities. You will have
to dedicate time and effort to understand new ways of doing things.
Since we build know-how by laying new concepts on top of ones we
already know, there are situations which require backtracking and
learn anew. For example, the Perl way of handling arrays requires a
clean mental slate. Forget all you know about C arrays. Significant
time dedicated to using arrays was the not so welcome news. On the
upside, Perl is extremely powerful. You can do work which would
otherwise be laborious, if not downright impossible, with other
programming languages.
My conclusion is that you may use Awk to analyze a text file one line
at a time when you want to get up to speed quickly. Awk is simple, it
is based on choosing lines to work on via a pattern matching
mechanism. For each match, specific work needs to be done.
However, if you were in the need of a powerful tool, not necessarily a
full-fledged C++ one, you might deem it worthwhile to invest time in
learning Perl. Finally, if regular expressions parens-based capture is
important to you, then Perl becomes unequivocally necessary (that is
what triggered me to switch from Awk to Perl).
Notwithstanding my having advised to forget Awk and learn Perl, I
endeavored to present Awk and describe its workings so that you
might experiment with it.

3.
Awk
You may skip this entire section if you intend to work with Perl, but I
suggest that you read this introductory part which describes Awk
briefly.
<x>Awk (general)[AUX]Why I started with Awk, then migrated to Perl</x><x>Awk (general)[AUX]Extract info from text files via logical statement & regex</x>Awk is

a command line text processing utility which contains mechanisms to

undertake a search for text via a pattern match (logical statement or
regex), one line at a time, and do work via a C-like set of
instructions, an action-block. Call this the Awk programming
language. You give it a set of plain text files (not Word files, nor any
files which contain formatting code). It concatenates them into one
large sequence of text lines and processes the entire lot one line at a
time (call it a record). A set of rules, activated one after the other, will
test each line for a match against a pattern to determine whether the
line should be worked on.
As Awk processes the lines of text one at a time, it tries to match the
line against a pattern (condition). If a match is found, it does
something with it (action-block). The condition/action-block pair
defines a rule. Awk reads a line and applies the first rule to the line. If
the condition yields true, the corresponding action-block is applied to
the line. It then tries the next rule and so on until all the rules have
had a chance to process the line. When finished with the rules, Awk
reads the next line and starts again, applying all the rules one at a
time, until all the lines have been processed.
<x>Awk (rules)[AUX]After loading new line, Awk tokenizes it</x>When Awk reads a line of text, it
decomposes its content into fields (tokens), a token being a
sequence of characters surrounded with spaces or tabs. The first
token is placed into the built-in variable $1, the second into $2 and so
on, $0 being the entire line. If the rule's condition evaluates to true, a
block of code (action-block) tied to the condition, processes the line.
Each rule (condition/action-block) can be thought of as being a
function, the condition being the criterium to activate the function's
body, the action-block being C-like statements to process the line.
Please note that if the condition is a regular expression (regex) and
evaluates to true, it will not capture anything via regex parens
grouping. This is unfortunate. Awk capture is limited to automatic
tokenization into $1, $2, etc. upon reading a line, before applying any
rules.

<x>Awk (rules)[AUX]Program is sequence of condition/action-blocks (rules) and functions</x><x>Awk (general)[AUX]Program is sequence of condition/action-

Conditions are one of two pattern matching

blocks (rules) and functions</x>

mechanisms which evaluate to true or false: logical statements

and regexes. Please note that these can be mixed to create a
complex condition via the AND (&&) and OR (||) logical operators.
A logical statement is any Awk statement which evaluates to
true or false. Example: the logical statement $1 == "Sasha" looks at
the first token, if it contains Sasha it returns true. A regex pattern
search is a technique based on a character-by-character
search to find a match. Looking for Sasha with a regex would be
/^\s*Sasha/, ^ means start at the beginning of the line, \s* look for
zero or more spaces or tabs, look for the five consecutive
characters S, a, s, h, a. Note that you are not asking the regex to
look for the string Sasha - you are asking it to look for a
 sequence of five specific characters, one after the other. It is
important to understand that regexes implement a one
character at a time search mechanism. It is not a word-based
search engine. See Regular expressions (regex) (page 191).
Awk actions-blocks are C-like functions blocks. They contain
C-like statements: variables, function calls, control statements
such as if, for, while, etc. A C++ programmer will find Awk's
syntax familiar.

If you have not opted to using Perl, I recommend you read on to

understand what Awk can do, how to use it. The rest of the Awk
coverage is detailed in the following sections:

Awk terminology (page 206) - Awk has its specialized

terminology, as all programming languages have.
Awk program structure (page 206) - Awk is an extremely basic
programming language specifically oriented towards
selectively doing work on individual lines of plain text files by
using a pattern matching mechanism (logical statements
and/or regexes to select the lines work to be done on). Its
syntax resembles C's syntax. An Awk program is an assembly
of user defined functions and variables, built-in functions and
variables, one BEGIN block, one END block, and a sequence of
rules (condition/action-blocks).
<x>Awk (Awk vs. C)[AUX]Origins trace back to Unix, hence based on the C language</x> Awk vs. C (page 214) - Awk

being remarkably similar to C, the unaware C-savvy Awk

beginner may assume incorrectly how Awk behaves; this
section is a must read before you start programming in Awk.
<x>Awk (command line)[AUX]-Wlint flag displays errors</x>Caveats - As with all tools and

programming languages, Awk has its share of traps and

pitfalls. Do yourself a favor, always run Awk with the -Wlint
command line option. See Invoking Awk from a DOS box
(page 301) and Awk gotchas (page 216).
The Awk (appendix) (page 301) section covers downloading
and getting started with Awk and using a DOS command box
 ([Link]) to launch an Awk program. It also contains example
programs and other info to help you get started with Awk.

Given Awk's usefulness I encourage its use, however, before diving

into learning Awk and writing Awk programs, I strongly suggest that
you should examine Perl, since it can do whatever Awk can do, and
much more. Its learning curve, on the other hand, is somewhat
longer. Go directly to the Perl (page 217) chapter to get an idea as to
what Perl is and come back here if you are still convinced that you
should use Awk.
I have tried to summarize Awk in the few sections below. Hopefully, it
will help you get started. My advice is that you should experiment
and try things out in order to understand how to use it. As for regular
expressions, which seem so cryptic when first encountered, open
your favorite IDE (except the old Arduino IDE V1), create any file,
and start experimenting with regexes and its find/replace dialog box
after clicking the regex icon to on.
I have tried, in this chapter and in the Awk (appendix) (page 301), to
summarize enough so that you can write most Awk programs you
might need. There is more to it though, as may be inferred from the
gawk official manual which is 500+ pages long, but there is no need
to read the manual before starting, as long as you understand a few
basic concepts. Refer to the manual to clarify concepts. If you
already program in C++, the learning curve is a couple of hours at
most, just enough time to understand its particularities. See Awk vs.
C (page 214) below.

1.
Awk terminology
Awk does not have a specialized
<x>Awk (terminology)[AUX]Rules, conditions, action-blocks, statements, fields...</x>

vocabulary but there are a few terms often associated with it.

A rule is a condition/action-block such as $1 == "Sasha" { statements }.

Awk conditions (page 209) are either logical statements,
 regexes, or a combination of both.
Awk action-blocks (page 209) are a sequence of Awk
statements enclosed in curly braces. They resemble C
program functions.
Awk statements are C-like statements inside an action-block.
The use of semicolons to mark the end of a statement is not
mandatory but may be required to avoid confusion. I
recommend that you always end statements with semicolons.
Records and fields - A record is a line of text. A field is an item
in the line of text, also referred to as being a token. It is a
character sequence separated by spaces. It can be thought of
as being a word. Note that a = b (note the spaces) defines three
fields: a, =, and b; a=b (no spaces) defines one field: a=b. Such
details can lead to confusion.

2.
Awk program structure
Awk is a command line text
<x>Awk (rules)[AUX]Process text files with sequence of condition/action-blocks (rules)</x>

processing utility. You feed it an Awk program you created and text
files to process. It will concatenate the files into one continuous set
of text lines and then process the lines one at a time according to a
set of rules (condition/action-blocks) you created.
Awk code ressembles C code. Your Awk program may contain four
parts, none of which are mandatory. It may contain functions,
variables, arrays. Functions may contain the basic control operators
if, for, while, etc. When looking at an Awk program, a C++ programmer
will immediately be on familiar ground. Furthermore, its grammar
resembles C++'s grammar. I should say deceptively similar since
C++ expertise will play tricks on the newbie Awk programmer.
When creating programs with most languages, the first lines of code
contain global declarations, housekeeping code such as #include
"MyIncludeFile.h", global variables, various declarations, initializations,
and other. These are single lines of code. Not so with Awk, all lines
of code must be inside one of the four constructs: user functions,
BEGIN block, END block, and rules. Writing a standalone line of code
outside of these is a mistake which Awk will silently let pass as it will
consider that it is a condition-only rule. This extraneous line will be
executed, the result being that the text line being processed will be
printed or not depending on the extraneous line's content.
<x>Awk (rules)[AUX]BEGIN and END blocks do work at start and end of program</x><x>Awk (rules)[AUX]Concatenates files into set of continuous text processed one line at a

An Awk program starts by running

time</x><x>Awk (rules)[AUX]Rule could be condition only or action-block only</x>

the BEGIN block. It then concatenates the files into a set of continuous
lines of text. It loads the first line, partitions the lines as tokens into
the built-in variables $1, $2, etc. ($0 contains the entire line), and
applies the entire set of rules (condition/action-block) to it. Rules
processing then starts over with the next line. When all lines have
been processed, the END block is invoked. The program ends after
this.
<x>Awk (rules)[AUX]After loading new line, Awk tokenizes it</x>When creating an Awk program,
concentrate on the four parts which constitute it: user functions, a
BEGIN block, an END block, and a set of rules (condition/action-block).
None of these are mandatory.
Here is a skeleton Awk program:
# The pound sign is the comment symbol
function MyFunction1(_param1, param2, ... ) {
#Awk statements
} # end MyFunction1
... more user functions
BEGIN {
# code executed when the application starts
} # end BEGIN block
END {
# code executed when the application has processed all the lines of all the files
} # end END block
#rules
condition1 { action-block 1 }
condition2 { action-block 2 }
condition3 { action-block 3 }
...
That's it - not that much to it.

A BEGIN block is executed before processing the lines of text. It

can be used to initialize global variables, print text before
processing the files, change the value of some built-in
 variables, i.e., anything you may want to do before doing any
work on the lines of text.
BEGIN { statements...}
An END block is identical to a BEGIN block except that it is
executed after processing the files. It may be used to report on
total values and indicate that Awk has finished doing its job.
This means at the very end, not at the end of each input file
when working on several files.
END { statements...}
<x>Awk (rules)[AUX]Condition based on pattern search: simple logical statements or regexes</x><x>Awk (rules)[AUX]Rule could be condition only or action-block

rules (condition/actions blocks) are the heart of an Awk

only</x>

program. They are applied one at a time on each line. They

differ from functions in that they do not have a name, do not
take parameters, and do not return a value. A rule has two
parts: a condition and an action-block, neither of which is
mandatory. A condition is a pattern (logical statement or regex
or mix of the two) which defines what to search for in the line
being processed. If the pattern finds a match, the action-block
is executed. There can be as many rules as deemed
necessary within the limits of the system's resources.

Awk reads one line; tokenizes it into $1, $2, etc. and tests it for a
match with the rule's condition, one rule at a time, top-down. If
there is a match, the rule's action-block is executed. When all rules
have been processed, it starts anew by loading the next line;
tokenizes it and processes it with the first rule. When all the lines
have been processed, it invokes the END block and finishes.
Remember, if you are processing several files, Awk concatenates
the files into one continuum. The entire set of rules is applied to
each line. When all the lines have been processed, Awk
terminates. This being said, Awk provides a means of knowing
which file the line belongs to as they are being processed (see Awk
built-in variables page 313).
An Awk rule could be a condition only or an action-block only,
neither is mandatory. If the condition is absent, the action-block is
applied on every line being read. If the action-block is absent, lines
which meet the condition will be printed. This last feature enables a
subset of the original file to be generated, based upon one or
several criteria. Example: if you wanted to print all lines which are
not commented, you could do so with the rule /^\s*[^/][^/]/. This
means: if the line does not start with a comment, print it. The regex
is saying: start at the beginning of the line ^, look for zero or more
spaces \s*, search for two consecutive characters that are not
forward slashes [^/][^/]. If you wanted to print only commented lines,
you would use the following regex: ^\s*//.

User defined
<x>Awk (variables and functions)[AUX]Functions similar to C functions, may return value of any type</x>

functions - Just as in C, Awk supports user defined functions.

These are similar to C functions. They have a name, take
parameters, contain statements to do work, and may return a
value. They encapsulate code to accomplish something.
Functions start with the keyword function, have a name,
receive parameters enclosed in parentheses, define a block of
statements enclosed in curly braces, and optionally return
values from anywhere in the function's body. Functions must
be defined before being used. Awk does not support function
declarations, mainly because an Awk program is one file; it is
not multiple files such as C++ programs are. Here is a typical
Awk function.
function MyFunction( parameters ) { statements... }

Notice that no return type has been defined, which means that
when looking at the function's first line, you cannot tell whether it
returns anything. You must look at the code to know this. To make
life easier, systematically write comments to document the function,
thereby saying whether the function returns something.

Built-in functions and variables

<x>Awk (rules)[AUX]Provides useful built-in functions and built-in variables</x>

are useful to control Awk's behavior and to process strings.

Variables - Notice in the program skeleton above that no global
variables have been defined; that is because all variables used
in functions, except parameters, are global.
 Comments are similar to those in
<x>Awk (rules)[AUX]Comment is # sign; down to end of line</x>

C++ except that the pound sign # is used instead of the double
forward slashes //. Everything after the pound sign # up to the
end of the line is considered to be a comment.
<x>DOS box[AUX]Awk invoked via DOS box - output screen or file</x><x>Awk (command line)[AUX]-Wlint flag displays errors</x>When

Invoking Awk from a DOS box (page 301) be sure to do so

with the -Wlint option. It will save you grief and pain as
otherwise Awk will silently let pass all sorts of mistakes you
might make, such as inadvertently creating new variables due
to misspelling or forgetting to enclose a string in double
quotes.

<x>Awk (general)[AUX]Edit Awk, Perl programs with Notepad++</x><x>Notepad++[AUX]Edit Awk and Perl programs - configure it for Perl: also useful for C++

Then there is the issue of using an Awk-friendly editor to create

editing</x>

your Awk applications. You may use notepad but there is a better
option. I found Notepad++ configured for Perl to be a good tool to
edit Awk applications with. You could also use VS Code with an Awk
extension. See VS Code Awk extensions (page 96).

3.
Awk conditions
Conditions may be simple statements such as $5 == "Sasha" where $5 is
a built-in variable which refers to the fifth field in the line being
processed, or may be any C-like statement which evaluates to true
(not zero) or false (zero), or may be a regular expression such as
finding an if statement \s*if\s*\(. This means search for zero or more
spaces \s*, followed by the two characters i and f, followed by zero or
more spaces \s*, followed by an opening parens \(. A condition is
either a logical statement such as $1 == "Sasha" or a regex enclosed in
slashes such as /^\s*Sasha/. In this case the regex is saying: start at the
beginning of the line ^, search for zero or more spaces \s*, then
search for the five letters S, a, s, h, a. Notice that the logical statement
and the regex function differently. The logical statement compares
the variable $1 with the string "Sasha". The regex works on the entire
line - it is looking for a sequence of five specific letters. The opening
and closing slashes signal that the condition is a regex instead of a
logical statement. Logical statements and regexes may be mixed via
the AND (&&) and OR (||) logical operators.

4.
Awk action-blocks
Action-blocks and function
<x>Awk (rules)[AUX]Action-block similar to function - contains C-like statements enclosed in curly braces</x>

blocks are identical. They are a set of statements enclosed in curly

braces {} just like a C function. Action-blocks can contain any
number of valid Awk statements, thus:
condition {
statement;
statement;
...
}
where statementis valid syntax such as c=a+b or print($3). There are two
differences relative to C though:

Action-blocks are nameless. Function blocks have a name.

Action-blocks do not return values and do not take parameters.
They may however incorporate the $1, $2, etc. built-in
parameters; these contain the individual words which
constitute the line of text.

As rules get processed, Awk makes it possible to stop processing

the current line and jump to the next line. Two built-in functions do
this:

The next function stops immediate

<x>Awk (rules)[AUX]getline, next stops processing line</x>

processing of the current line, gets the next line and starts
again with the topmost rule. It operates just as the continue
statement in a for loop does.
The Getline function, when used without parameters, is similar to
next, except that it continues processing with the next line. It does
 not return to the top rule. getline accepts parameters to do all
sorts of work. It is complex, best reserved for advanced Awk
programmers.

5.
Awk rules
The key to using Awk is a good understanding of condition/action-
blocks (rules):
condition1 { action-block1 } # rule 1
condition2 { action-block2 } # rule 2
condition3 { action-block3 } # rule 3
…
A condition is a logical statement or regex which evaluates to true or
false, or a combination thereof, linked by AND (&&) or OR (||) operators.
A condition may be anything, a variable, a constant, lines of code, a
logical statement, a regex, etc. Whatever it is, it is evaluated, the
result being 0 (false) or non 0 (true). When a condition's pattern matching
search on a line yields true, the corresponding action-block is
executed. Whether the action-block is executed or not, Awk
continues with the next rule. In the code segment above, Awk will
apply rule1, then rule2, and so on to the current line. It will execute the
action-block if the corresponding condition returns true. Note that it is
possible to interrupt processing the current line by using the keyword
next. Doing this stops processing the current line, reads in the next
line, and starts again with the first rule (see the previous section:
action-blocks).
I should add, before continuing, that rules are condition/action-block
pairs in which both the condition and action-blocks are optional. This
is explained further down in the condition-less rules and action-block
less rules.

1.
Awk rules - A question of semantics
 The semantics which govern
<x>Awk (general)[AUX]Program is sequence of condition/action-blocks (rules) and functions</x>

how statements in action-blocks are defined are what I refer to as

the Awk programming language. Think of it as being a simplified
version of C. However, some of the features which make C a
powerful language, such as type checking, variable declarations,
multiple types, etc., are absent in Awk. But, when you look at an Awk
program, it looks remarkably similar to a C program.
<x>Awk (rules)[AUX]Files broken up into records, records into fields (lines of text/words)</x><x>Awk (rules)[AUX]Default record separator ORS is newline (line of text); default

Let
field separator OFS is spaces or tabs (word in line of text)</x><x>Awk (built-in variables and functions)[AUX]Output record separator - default is newline (ORS = "\n")</x>

me add a note on records and fields. From an informal perspective, I

have been referring to lines of text and tokens or words. Awk is a
little more specific. It processes records and fields. Records are
strings delimited by the record separator RS which, by default, is set
at \n (newline). Therefore, by default, a record is a line of text. In the
default case, record numbers and line numbers are identical.
Changing this can be useful to handle multi-line records, ex. change
ORS to two carriage returns (ORS = "\n\n"). In this case, records and lines
of text get out of sync as soon as a multiline record is found.
<x>Awk (built-in variables and functions)[AUX]FS - fields separator (default " "); RS input record separator (default "\n")</x>In much the same

way, fields are words inside a line of text delimited by the field
separator FS which by default is a space or tab. Therefore, fields,
tokens, and words are the same; words and lines of text are the
default. Processing CSV (Comma Separated Values) files however,
would require redefining the field separator as a comma FS = ",".
Since the default record separator (RS) is a newline (\r) and the
default field separators (FS) are spaces and tabs (/t), C/C++ source
files may be processed directly with no changes in RS and FS. But, if
you wanted to be really clever, you could define the semicolon as
being the RS record separator; by doing this, you could read one C++
statement at a time, whether it seats in one line or in several lines.
See Awk built-in variables (page 313).

2.
Awk rules - Tokenization
As mentioned earlier, Awk reads lines of text, one at a time. It uses
the rule's condition to determine whether work is to be undertaken
on the current line, i.e., do the work defined in the corresponding
action-block.
<x>Awk (rules)[AUX]Rules use fields $ 1, $ 2, etc. from line tokenization</x>Awk breaks up the record's content

into fields labeled as $1, $2, etc. The variable $0 contains the entire
line. Quite simple, if you want to find a line which defines a macro,
just do $1 == "#define". When it finds a line whose first field is #define, it
invokes the corresponding action-block. The problem is that
commented out #defines will not be found because the condition would
fail since $1 could contain // or //#define.
<x>Awk (regex)[AUX]Regex finds macros: #define, #ifdef...</x><x>Awk (examples)[AUX]Regex finds macros: #define, #ifdef...</x>Pattern matching

with a regex is much more powerful. Its use is preferable since it

brings flexibility which basic logical statements do not have. The
regex ^\s*(//)?\s*#define will find both commented and uncommented
#defines. It says: starting from the beginning of the line ^, find zero or
more spaces \s*, followed by an optional C++ comment (//)?, followed
by zero or more spaces \s*, followed by the 7 characters #define. Notice
the parentheses around // and the question mark. It is saying: look
for two consecutive optional forward slashes //. This means that the
regex will find all #defines, whether there are forward slashes or not,
whether attached to #define or not.
If you want to determine whether the macro is a commented macro,
you need to test for the presence of a comment // attached to #defines
or not. Doing this is complex because $1 could contain #define or // or
//#define. Since Awk does not support regex captures, you cannot
benefit from regex groupings to test for $1's content directly via the
regex (//)?#define. Awk requires code to determine the presence of //. In
Perl, you may use a regex capture such as (//)?. $1 would be empty or
would contain //, thereby defining whether the macro is active or
commented out.
The condition based on logical statements which finds #defines is:
$1 == "#define" || $1 == "//#define " || ($1 == "//" && $2 == "#define")
These logical statements handle commented and uncommented
#defines. However, if you want to also handle #endif, #if, #else..., each of
which could be commented or not, you would need a total of 15
basic logical statements grouped by the OR || operator. Using a regex
is a lot simpler and less error prone. This is the regex you would
write (begin/end forward slashes are not part of the regex, Awk
requires their use to differentiate them from logical statements):
/^\s*(//)?s*#(define|if|endif|else|error)/
This regex accomplishes the same task as the 15 logical statements.
If the first capture reveals that it is indeed a //, the conclusion is that
the macro is commented out. This regex may seem a little cryptic; it
can nevertheless be analyzed by taking items from left to right, one
at a time. After a little practice, you will get used to it.
The Awk alternative for finding macros would be to look at the first
two tokens $1 and $2:
Does $1 start with a pound sign (#)? If so, got a macro.
Does $1 start with two slashes, followed by a pound sign (//#),
followed by one of the macro keywords? If so, got a macro.
Does $1 start with two slashes (//) and $2 with a pound sign (#)? If so,
got a macro.
Using a regex would be a lot simpler but Awk does not support regex
captures. This is one more reason for migrating to Perl.

3.
Awk rules - Condition less rule
<x>Awk (rules)[AUX]No condition A rule without a condition is
rule is always true - action-block invoked</x>

equivalent to a rule whose condition is always true. The condition-less

rule {do something} is equivalent to the always true 1 {do something} rule.
Since the condition 1 evaluates to 1, the result is that the rule always
evaluates to true; hence, the action-block always gets invoked.
Suppose you want to create a copy of your source files with line
numbers at the beginning of each line. You could create the rule
{ print "/* line", NR, "*/ ", $0 }
Since the rule has no condition, it will print the contents of every line
$0 preceded by the line number. This action-block produces:
/* line 37 */ the line's content
Here is another simple rule: it prints every line with filename and line
number at the beginning of each line:
{ print ARGV[ARGIND+1],":", NRF, " - ", $0; }
The output will be
MyFile:348 - this is the line of text
Notice that this rule does not have a condition - this means that
every line will be printed with file name and line number at the
beginning of the line. The built-in variable FILENAME could have been
used instead of ARGV[ARGIND+١].

4.
Awk rules - Action-block less rule
A rule without an action-block, i.e., a
<x>Awk (rules)[AUX]Condition only rule prints line if condition successful</x>

condition only rule, will print the line if the condition evaluates to true.
Such a rule is equivalent to condition { print $0 }. Example: suppose
you want to make a copy of your C++ file without comment lines.
Use the regex ^\s*[^/][^/]. It is saying: start at the beginning of the line
^, look for zero or more spaces \s*, then make sure that the first two
non-space non-tab characters are anything but forward slashes [^/]
[^/]. The result is that all lines which do not start with two forward
slashes will be printed. You could have done this with a logical
statement such as $1 == "//" but then what if $1 contained "//#define". The
condition would evaluate to false. Using a regex is a lot simpler.
One final word on conditions: they can evaluate to anything. Any
valid Awk statement will evaluate to something: either zero or
nonzero. Suppose that you create a standalone statement, a
statement which seats against the left margin, neither in a function,
nor in a rule, such as c=a+b. You just created an action-block less rule.
Lines will be printed whenever c evaluates to nonzero. Gotcha!

5.
Awk rules - Use logical statement or regex
Most rules have both
<x>Awk (rules)[AUX]Logical statements for simple conditions, regexes for complex ones, or mix of the two</x>

conditions and action-blocks. Here is a simple example:

$1 == "class" { foundClass = 1 }
The $1 == "class" is a logical statement. This rule says: if the first token
$1 contains the string class (contained in quotes), the action-block is
invoked - it sets the variable foundClass to 1. There are a few things to
note:
 The condition is a C-like logical statement, aside from the $1
variable which is Awk specific.
The action-block looks familiar. It is a C-like function body.
Notice the use of curly braces to enclose statements, and the
use of the assignment operator =.
The assignment above does not contain an end semicolon as
you would have in C/C++ and Perl. This is optional in Awk. If
there is an ambiguity as to what follows next, the semicolon is
required. I systematically use a semicolon to mark the end of
statements. It makes code more readable.
Awk does not know true/false. It considers 0 to mean false. Any
nonzero value, positive or negative, means true.

What follows is a more

<x>Awk (rules)[AUX]Logical statements for simple conditions, regexes for complex ones, or mix of the two</x>

complex logical statement example. The programmer's intent is to

search for classes, structures and enums. Easy - just look at $1. Since
these always begin with the corresponding keyword, assume that
such declarations start on a newline.
Define the following rule to get the name of a class, struct or enum:
$1 == "class" || $1 == "struct" || $1 == "enum" { ... }
The name should be in $2 or possibly in $3 in the case of enums (2nd or
3rd token). Having gotten the name, some processing is required to
clean it since C/C++ does not always require spaces to delimit items.
A class name could be followed immediately by an open brace such as
class myClass{ ... }
thereby polluting the name. In this case $2 will contain myClass{.
enum declarations are a little more complex. They may be defined in
one of several ways, such as:
enum EEmployee {... // standard short definition, spaces used
enum class EEmployee{... // class added and space before opening brace omitted
enum EEmployee: uint8_t{... // size qualifier added, spaces used
enum class EEmployee:uint8_t{... // class qualifier and size qualifier added, spaces omitted
This presents two issues:

The enum's name could be in $2 or $3.

The name could be polluted with a brace { or a colon : or more
 such as EEmployee:uint8_t{ in the 4th line above.

The programmer would have to test $2 as it might contain either the

enum's name or the word class. Having found the name, he/she would
have to clean it by removing potential unwanted trailing characters
such as opening brace {, or a colon :, or other.
As may be judged from the above, using conditions based on logical
statements can be cumbersome. Using a regex capture would be
considerably more efficient, as follows:
(class|struct|enum[\s]+(class)?)[\s]+(\w)+[{:]?
That's it! Took me five minutes. The name or the enum, class, struct is in
$3, the 3rd grouping (3rd opening parens). The regex effectively
differentiates the class keyword between a class definition and the class
attribute of an enum. But, you cannot do this with Awk as it does not
support regex captures; you would need to do it with Perl.
The conclusion is that you should use logical statements for simple
conditions; use regexes for complex conditions, and possibly migrate
to Perl if you indeed need to capture text via regexes.

6.
Awk BEGIN and END
Awk BEGIN and END blocks are built-
<x>Awk (rules)[AUX]BEGIN and END blocks do work at start and end of program</x>

in empty function skeletons, much like the empty setup and loop when
you start a new Arduino sketch - it is up to the programmer to decide
what to use them for. The statements they contain can be any legal
Awk statement. BEGIN is invoked before any line of text gets
processed. END is invoked after all the lines of text from all the files
have been processed. This enables the Awk program to undertake
housekeeping when the program starts and when it ends.

BEGIN may contain user variables and built-in variables

initializations. It could do program header prints and any other
initializations.
END could print results from overall processing such as totals,
 do wrap-up work, and finish with a message such as "End of
Program".

7.
Awk functions and variables
Awk functions are practically
<x>Awk (caveats)[AUX]Functions similar to C functions, may return value of any type</x>

identical to C++ functions. They may take parameters and return a

value just like in C++; however, return values and parameters are not
explicitly typed. They are typed by their use. The parameters are
enclosed in parentheses, and the body in curly braces. Unlike C++
and Perl, there is no such thing as a function declaration. Functions
are defined first and used afterwards.

The keyword function

<x>Awk (variables and functions)[AUX]Function declaration starts with 'function'</x>

precedes the function's name.

A function can return
<x>Awk (variables and functions)[AUX]Functions similar to C functions, may return value of any type</x>

a value via the keyword return, but it does not have a return type.
A return statement is optional however, if omitted and used as
if the function returned something, there is no error message,
and the calling function will receive garbage.
There cannot be a space between the function's name and the
opening parens.

The structure of an Awk function is

function MyFunction( parameter list ) {
statement
statement
...
}
Variables defined inside a function are global, however, the
parameters of a function are local to the function even though the
spelling may be the same as that of an existing global variable. See
Awk some features illustrated (page 304).
 8.
Awk vs. C
<x>Awk (Awk vs. C)[AUX]Origins trace back to Unix, hence based on the C language</x><x>Awk (Awk vs. C)[AUX]Your C knowhow can be a false friend</x><x>Awk (general)

Awk's origins
[AUX]Your C knowhow, a false friend when learning Awk</x><x>False friends[AUX]Your C knowhow, a false friend when learning Awk</x>

trace back to Unix. The C language was invented to develop Unix.

Many utilities were created to accomplish various tasks (grep, sed,
Awk, Perl…). All of these profited from the C-language experience.
That is why they are C-like, but there are differences since they were
meant to be used to meet different needs. If you have never
programmed in C, you will learn Awk and not make many of the
mistakes a seasoned C programmer will make because he/she
would program Awk as if it were a C program, thereby making
incorrect assumptions as to how Awk works. So, seasoned C
programmers be warned - there are differences, some of which are
subtle. See Awk some features illustrated (page 304).
Here are some notable differences.

<x>Awk (Awk vs. C)[AUX]The # sign signals a comment to end of line</x> The # sign signals that the rest of
the line is a comment, like the C++ //. The C comment /*...*/ pair
is not supported.
<x>Awk (Awk vs. C)[AUX]Two variable types: floats and strings - they are not declared, they are used</x>Awk has two variable

types: floats and strings - no chars, no integers, no signed/unsigned, no

pointers, etc. Variables are not declared. They are used.
<x>Awk (Awk vs. C)[AUX]Variables used in functions are global</x>Awk variables are global, which can

lead to surprises, particularly in for loops which use i as the

indexing variable. Test it by defining a first function which
contains a for loop indexed with i. Create a second function
which prints the value of i. Call the first function, then the
second function. You will find that i's value in the second
function will correspond to the value i had when the first
function finished with it, this despite the fact that it has not
been declared to be global, nor used as such.
<x>Awk (Awk vs. C)[AUX]Strings are 1-based</x><x>Awk (strings)[AUX]Strings are 1-based</x>Awk strings are 1-based.
 The index of the first character in a string is 1, not zero as in
C++. This means that functions which return a character
position in a string, will return a 1-based position, 0 if not
found.
<x>Awk (Awk vs. C)[AUX]Array indices can be integer, float, string</x><x>Awk (arrays)[AUX]Array indices can be integer, float, strings, similar to key into

Awk Arrays look like C arrays, however, their

database</x>

implementation is entirely different. Array indices are keys

which could be a floating-point number or a string. Think of
Awk arrays in terms of a database whose records are
accessed by a key. A special for statement enables traversing
an array. See arrays in the Awk reference (page 309).
<x>Awk (Awk vs. C)[AUX]Supports regular expressions</x>Awk supports regular expressions, but

it does not support the regexes' parens-based capture

mechanism.
<x>Awk (Awk vs. C)[AUX]End of statement semicolon not mandatory</x>The end of statement semicolon is

not mandatory. It should however be used to remove

ambiguity. Doing two consecutive print statements print a print b is
legal but print a x = 10 is not legal. For the sake of readability and
good programming practices, I recommend that you
systematically use semicolons to mark the end of statements
and that statements be on separate lines.
<x>Awk (Awk vs. C)[AUX]Print statement not quite literal, requires understanding its mechanics</x>The print statement does

not require that the print list be enclosed in parentheses,

furthermore, the corresponding items need not be separated
by commas. However, using commas will influence whether
spaces between printed items are added or not. See Awk print
(page 315). I recommend that you systematically use commas
and set OFS (output field separator) or "" (no space) in the BEGIN
block. See Awk print (page 315).

This is just the tip of the iceberg. Consult any Awk reference, you will
find it to be a good resource. It is a basic tool useful to accomplish a
variety of tasks. The Awk Appendix contains additional details. See
Awk some features illustrated (page 304).
This being said, risking being repetitious, I strongly recommend that
you use Perl instead of Awk. With Perl you will be able to accomplish
everything you can do with Awk and a lot more. See Perl (page 217).

9.
Awk gotchas
In the process of designing a simple user-
<x>Awk (caveats)[AUX]Excessive simplicity cause of gotchas</x>

friendly C-like language without the complexities of C, its designers

inadvertently built into Awk an important potential for gotchas. Here
are some of the ones I ran across.

<x>-Wlint flag[AUX]Awk - use -Wlint command line flag to display errors</x> Programming errors - I got used
to having the C compiler being my safety net against errors I
might make. This generally worked well. With Awk however,
you cannot rely on the interpreter to protect you as it will
silently let pass errors unless you use the -Wlint command line
option to issue warnings on some of the mistakes you might
have made. You must be particularly attentive as you code. For
example, if you misspell return x by writing returns x, it does not
complain, no error message. You will just get garbage.
Unknowingly, by misspelling return to returns, you created the
variable named returns leaving you with two variables sitting side
by side (returns and x), which is legal. That is why there is no
error message. Your mistake is a legal Awk statement. Using
the -Wlint option will trigger an unused/uninitialized warning on
returns. If you do something silly such as a b c d, you are defining
four uninitialized variables, which do nothing. Such a statement
is legal and will remain unreported unless you use the -Wlint
option.
<x>Awk (caveats)[AUX]Careful with spaces in if, while, functions, etc. parens must be against if, etc.</x>Spaces when defining

a function - Careful, Awk uses spaces to parse things, thus if(

means start an if statement because an opening parens follows
if. However, 'if (' means concatenate if and ( to form a string.
This will report an error since what follows is gibberish to the
 Awk interpreter. Likewise, function definitions and function calls
do not allow a space before the opening parens.
function MyFunction ( ... ) { ... }

is illegal because of the space between the MyFunction and the (. The
problem is that Awk will not always report an error, you will get a
blank output and wonder why. Definitely frustrating. Again, use the -
Wlint command line option to get warned of mistakes.

Awk does not know characters; it only

<x>Awk (caveats)[AUX]Individual characters are strings</x>

knows strings - C++ programmers think both in terms of

characters and strings: 'A' is a character; "A" is a string. They
have been trained to understand the difference. Awk works
only in terms of strings. If you want to define a single character,
you must define a string whose length is one.
<x>Awk (caveats)[AUX]Variables typed when used - no declarations</x>There is no need to declare

variables since they are created when first used. Such

simplicity can be treacherous. For example, suppose you
wrote
if( name == Sasha) instead of if( name == "Sasha")

You meant to test that name does indeed contain Sasha but forgot the
double quotes. In C++ you will get an undefined variable error; Awk
will oblige by creating a new variable named Sasha - it will be
considered to be an uninitialized variable and not a string. The
program will run, with erratic results.

Side effect from stray lines of

<x>Awk (caveats)[AUX]Stray line of code silently interpreted as rule</x>

code - As mentioned earlier, an Awk program has four

sections: BEGIN block, END block, functions, and rules. If,
through inattention, you happen to write a line of code outside
of these such as "I made a mistake", Awk will consider it to be a rule.
Remember! A rule which contains a condition without an
action-block is legal; it prints the entire line if the condition
evaluates to true. So, if you wrote "I made a mistake", you created a
rule which contains a condition but no action-block. Since a
 string evaluates to nonzero, it becomes a rule whose condition
always evaluates to true. Since there is no action-block, this rule
will print each line, i.e. all the file. If you had written b = 0, the
condition would evaluate to 0, thus the line would not get
printed.

As a sideline, a rule which contains an action-block without a

condition is legal. It executes the action-block on every line, it is as
though you wrote 1 { statements }. You might get reams of output or
nothing at all, depending on the contents of the action-block.

<x>Awk (caveats)[AUX]Strings are 1-based</x>Awk strings are 1-based, not 0-based as in

C++ and Perl. The index into a string starts at 1. A 0 return
from a string search function usually means that the search
was not successful.
<x>Awk (caveats)[AUX]Function variables global, parameters local</x>Function variables are global,

function parameters are local - Variables are defined as global

as soon as you use one for the first time, whether inside a
function, inside BEGIN/END or in a rule. You must be careful with
indices such as in a for loop which uses i. If other functions use
i, they are the same i because i is global.

4.
Perl
Perl is a 3rd generation procedural
<x>Perl (general)[AUX]Powerful C-like programming language with regex support</x>

programming language much like C, Fortran, Java, Basic, etc. After

working with Awk, Perl turns out to be a real delight. My best
description of Perl is that it is a powerful C-like programming
language which contains a regular expression engine (regex). Its
origin lies with Unix, along with grep, ed, Awk, etc. I should
emphasize that it is more than a useful command line utility. It is a
full-fledged programming tool capable of delivering sizable
applications. However, it is limited to one program file, but the
programmer may encapsulate code in modules which may be
brought in much as library components are brought in. Note that
there are many Perl third-party modules which could enhance your
program.
Perl is loaded with far reaching features. It can load files, extract
lines of text one at a time, undertake pattern matching, do work on
text via its C-like programming language, and export results into text
files. Perl can pretty well do anything which can be done with Awk,
and much more. You will find more details on Perl in the following
sections :

Perl's Terminology (next section) - It is important to understand

Perl's vocabulary as it brings in concepts unfamiliar to the C++
programmer.
<x>Perl (general)[AUX]Edit Awk, Perl programs with Notepad++ or VS Code</x><x>Notepad++[AUX]Edit Awk and Perl programs - configure it for Perl: also

useful for C++ editing</x> Perl primer (page 219) contains just enough info to
get you started doing real work such as loading files, cycling
through them, doing pattern matching, and generating results.
<x>Perl (build)[AUX]Perl's program structure resembles C's</x>The Perl program structure (page

220) is free style, much like C's but it has its quirks such as
passing parameters to functions and arrays. There is not much
which needs to be done to get a Perl program up and running.
Try print "Hello World from Perl". That is it; you have created your first
Perl program.
<x>Perl (build)[AUX]Quick overview of Perl programming</x> Perl quick overview (page 221)

presents a bird's eye view of a Perl program.

<x>Perl (Awk vs. Perl)[AUX]Simulate Awk with Perl or how to transform Awk program to Perl</x> Simulate Awk using Perl

(page 223) runs through a group of files and does line by line
pattern matching just as Awk does with condition/action-
blocks. One of Perl's important advantages over Awk lies in
Perl's being able to process a group of files times over as
opposed to Awk's processing them only once.
<x>Perl (general)[AUX]Your C knowhow, a false friend when learning Perl</x><x>False friends[AUX]Your C knowhow, a false friend when learning

As usual, knowing how to program C leads one to

Perl</x>

preconceived ideas as to how things work, aka false friends.

 This leads to gotchas. See Perl vs. C (page 224).

Note that you can use VS Code with Perl extensions as an

environment to create Perl programs with. See VS Code Perl
extensions (page 96).
The Appendix presents details on using Perl constructs. See Perl
(appendix) (page 318).

1.
Perl terminology
<x>Perl (built-in variables and functions)[AUX]$PROGRAM_NAME, or $0, contains the Perl program launched</x><x>$ (dollar sign)[AUX]Perl

$INPUT_RECORD_SEPARATOR, or $RS, or $/ - default \n</x><x>$ (dollar sign)[AUX]$0 - Awk: record just read in, Perl: program name (also $PROGRAM_NAME)</x><x>Perl

Perl uses some familiar names

(file handles)[AUX]$filehandle created from file name string via the open statement</x>

which can be a trap to the unwary C programmer since they hide

Perl specific concepts. It has its own terminology such as:

<x>Perl (file handles)[AUX]Files in read, write, or update mode</x> File handles are created from
filenames via the open function. They are used to open/close
files for input, output, or both.
<x>Perl (build)[AUX]Perl handles records and fields</x>Records and fields - In both Awk and

Perl, a record is a string of text in one or more lines.

<x>Perl (build)[AUX]Importing packages (use directive): strict, diagnostic, features 'switch'</x>Pragmas are Perl

directives placed at the beginning of a program.

modules, aka packages, are external components. They get loaded
via the keywords use or package such as package DataGroup. This
enables Perl to encapsulate the application's logic inside
individual files (packages are the equivalent of libraries in
C++).
Statements are C-like lines of code. They may be used at
global level or inside a function (Perl calls them subroutines).
Perl statements basically follow the same syntax as C
statements. Note that Perl requires that statements end with a
semicolon. A C++ programmer will discover unfamiliar
constructs which will require hard thinking to unravel.
<x>Perl (build)[AUX]Perl provides three variable types: $scalars, %hashes, @arrays</x>Perl proposes four data

types: scalar numeric, scalar string, array, hashes - Whereas

Awk does not have particular data types save float and string,
which are implicit upon first use.
<x>Perl (build)[AUX]Pass by value and by reference</x>Pass by value, pass by reference - Perl's

function parameter passing is identical to C's pass by value.

Pass by reference may also be implemented. Passing a
pointer is not supported. Pass by reference is particularly
useful when passing an array to a function which may require
that some of the array's content be modified. See Perl pass by
reference (page 329).
<x>Perl (build)[AUX]Scope resolution operator :: implements namespace concept</x>Scope: namespace and my -

Given that Perl programs can import modules, the creators of

Perl added a namespace concept implemented via the scope
resolution operator ::, just like in C++, so that variables could
be qualified according to the module they belong to.
Subroutines - Fortran has subroutines, Perl also. C, Java, Awk
and others have functions. They are blocks of code identified
by a name which can take parameters when invoked. Calling
them functions, procedures, methods, or subroutines is a
matter of terminology since, aside from how they are referred
to, they are identical. I prefer using the term function.
Parameters - All ٣rd generation languages I have used are
similar as to defining function parameters and how to write
function calls. Perl is peculiar - it takes getting used to. See
Perl subroutines (functions) (page 322).
<x>Perl (build)[AUX]The text match operator =~ enables using a regex in a statement</x>Text match operator as in

$myString =~ /regex/ is saying: return true if the regex finds a match in

$myString.
<x>Perl (general)[AUX]Perl supports regex parens-based captures - Awk does not</x>Capture means getting the

regex to extract part of the matched text's content. Items get

captured via parens pairs in the regex.
Utilities are programs which are part of the Perl distribution.
 There are several utilities, one of which is an Awk to Perl
translator: a2p. I have not found it in my Strawberry download
and thus cannot comment on it, but it does exist somewhere.

2.
Perl primer
Perl programs resemble C programs. They may include functions,
variables, and arrays, but there are differences. The C++
programmer should beware of false friends. The issue here is
avoiding doing something in a way which is not indigenous to Perl.
A Perl program has no particular structure. It is an assembly of
interpreter directives, imported packages, variables, statements, and
functions which form a unit to achieve a task. It can be as short as
print "Hello World from Perl"
or as long as the system will allow.
<x>Perl (general)[AUX]Edit Awk, Perl programs with Notepad++ or VS Code</x><x>Notepad++[AUX]Edit Awk and Perl programs - configure it for Perl: also useful for C++

A Perl program is a single text file whose extension is generally

editing</x>

.pl or .PL. It is invoked from a [Link] console (DOS box). For

details on [Link] see the DOS box (appendix) (page 237). Unlike
C which is multimodule, a Perl program is single module, one text
file. It can, however, bring in external content as packages (modules)
via the pragma or use directives (see below).
Here is a typical Perl program invocation (DOS box command line):
Perl [Link] arg1 arg2 arg3...
It contains a call to Perl, the program's name, and arguments:

Perl is the
<x>Perl (command line - DOS box)[AUX]Perl command line file/directory in arg1, arg2, etc.; in @ARGV array</x>

[Link] you need to invoke.

[Link] is the name of the Perl program you have
created. Notice that there is no -f flag, such as in Awk, to
identify the program. It is the first argument after invoking Perl.
arg1 arg2 arg3 ... are the Perl program arguments which will
be passed into the Perl built-in array @ARGV. They are just
names which the programmer may use for whatever purposes
 he chooses. They can be anything, it is up to the programmer
what to use them for. In my case arg1 and arg2 are the input
and output file paths respectively, arg3 and what follows are
the input files. These arguments are located in the @ARGV array
as of index = 1. The first argument ($ARGV[0]) contains Perl, the
calling program's name.

Here is a typical Perl program's structure:

When creating a Perl program, you are

<x>Perl (general)[AUX]Handles text files and binary files</x>

limited to placing all your code inside one file (.pl or .PL
extension) just like with Awk. But unlike Awk, you may import
modules, be they native Perl modules, third-party modules, or
your own. Start your program with use directives. They are Perl
directives to import packages. Perl programmers use these
frequently. You should of course comment extensively (use #
instead of // or /*..*/). Please note that Perl, unlike Awk, can be
used to process both text files and binary files such as Word
files. I shall limit my coverage of Perl to text files. I have not
investigated using Perl with Word files.
<x>Perl (variables and functions)[AUX]Functions have a name and a body but do not have a parameter list</x><x>Perl (variables and functions)[AUX]Functions may

be called with parameters - unique way of handling (via array @_)</x><x>Perl (variables and functions)[AUX]Functions can pass by value or by

reference</x> Define functions as needed. Be careful with parameters -

Perl has a rather unique way of handling them; however, both
pass by value and pass by reference are supported.
<x>Perl (variables and functions)[AUX]BEGIN and END blocks executed at program start and finish</x>Define the optional

BEGIN and END blocks. The BEGIN block is executed once when
the program starts. The END block is executed once just before
the program exits.
<x>=~[AUX]Perl pattern matching operator</x>Perl supports regexes. Pattern matching is

handled via the pattern matching operator =~. Regex items may
be parens-based captured into $1, $2, etc. to be used as
variables to do work. The following statement illustrates pattern
matching captures:
 my $line = "The little brown fox ran away";
if ( $line =~ /.*?(brown|red|\s*)/) { print " the fox's color is $1"; }

This regex has only one parens group. Its content is assigned to $1.
Note the match operator =~. If there is a match, the print statement
will print the fox's color. Perl's pattern matching is quite specific, it is
regex based. It does not support pattern matching via logical
statements nor automatic tokenization into $1, $2, etc., such as is
done in Awk.

Perl captures parts and

<x>Perl (general)[AUX]Perl supports regex parens-based captures - Awk does not</x>

pieces of text into $1, $2, etc. via the regex pattern matching
parens-based capture mechanism. Note that Perl's $0 does not
return the entire line. To get the entire line use $ _. Perl's $0
contains Perl, the calling program being run. It is the same as
$ARGV[0], i.e., the first item in the @ARGV array.

The above is expanded upon in the Perl program structure section

which follows.

3.
Perl program structure
The Perl program structure is rather basic. It
<x>Perl (build)[AUX]Perl's program structure resembles C's</x>

does not differ much from other procedural languages program

structures. A program is a single text file with a .pl or .PL extension.
It is not multi-file such as C++ is, but you can import packages which
are equivalent to libraries in C++.

Define imports - The

<x>Perl (build)[AUX]Importing packages (use directive): strict, diagnostic, features 'switch'</x>

program should start with use directives to import packages,

three of which are highly recommended:
use strict; # enforces explicit declarations
use diagnostics; # provides useful info on errors detected
use warnings; # does not work on my Strawberry distribution
 After defining basic packages to use, you may import library
components. It is done via use something. Two important such
components should be imported:
use 'say'; # it is a print clone which adds a newline
use "switch"; # imports the switch/case/default mechanisms named given/when/default.
There are other ways of importing packages, each with its
advantages. The use pragma is not the only way. Notice the use of
single or double quotes. You can use one or the other, it does not
matter since string interpolation is not implemented (see Perl string
interpolation - single or double quotes (page 323) below).

Write your code. After having defined pragmas and imported

packages, you are free to do whatever you want. It could be as
short as:

print "Hello World from Perl!"

or extensive. See Perl (appendix) (page 318) for Perl program
examples and extensive details on programming Perl (function
parameter passing, arrays, strings, etc.).

Perl supports the optional

<x>Perl (built-in variables and functions)[AUX]Perl supports the Awk like BEGIN and END</x>

Awk like BEGIN and END blocks. They are parameter-less

functions invoked when the program starts and when the
program ends. BEGIN may be used for initializations, defining
global variables, setting built-in variables, and other program
start housekeeping tasks. END may be used for totalizations,
closing files, notifications, and other program wrap-up tasks.

Just like other programming languages, Perl has its syntax,

grammar, functions and is shipped with utilities to assist you in your
work. The [Link] web site is a good resource.

4.
Perl quick overview
The previous section Perl program structure introduced what a Perl
program looked like. This section explains some of the fundamental
mechanisms underlying Perl.
Since the context is to have a Perl program process C++ source
code, one line at a time, a back of the envelope definition as to what
a Perl program should contain follows:

Preliminaries are initializations and housekeeping chores

before doing any work. You may want to define a BEGIN block. It
would be invoked upon startup. Pragmas are defined and
packages are loaded (library components). Both are based on
the use keyword. You may also want to define an END block to
handle wrap-up phases when the program ends.
You may want to import packages (library components),
depending on your needs. They are the equivalent of library
modules in a C++ environment. See the [Link] web site or
Google Perl modules. I recommend the following :
use strict - Ensures that variables are formally declared.
use warnings - Issues warnings on code which compiles but may
be obsolete.
use diagnostics - My Perl Strawberry download does not include
this package. The warnings package is included. It does a good
job.
Use feature 'say' - say is identical to print except that it adds a
newline, whereas Perl print does not automatically add a
newline. It is no big deal but it saves systematically adding \n
to prints.
- Provides the switch/case/default mechanisms,
use feature 'switch'
referred to as given/when/default. The Perl switch statement is
similar to C++'s but there are differences. These are
presented in Perl program flow control (page ٣٢٤).

These modules are part of the Perl distribution. You might want to
consider using other modules available from the [Link] Web site,
or third-party modules, or try creating your own Perl modules.
Google Perl modules.

<x>Perl (variables and functions)[AUX]Perl types differentiated by special characters: $scalars, @arrays, %hashes</x><x>$ (dollar sign)[AUX]Perl provides three

variable types: $scalars, %hashes, @arrays</x><x>Perl (arrays)[AUX]Perl provides three variable types: $scalars, %hashes, @arrays</x><x>%hash[AUX]Perl

Variable declarations - Remember that

provides three variable types: $scalars, %hashes, @arrays</x>

there are three types of variables: $scalars, @arrays, %hashes.

Each variable type must start with the corresponding special
character. If the strict pragma is on, variables, when first used,
must be preceded with the scope resolution my to render them
local within the scope they are declared in. If the use strict pragma
is in use, both global and local variables must be declared.
Note that Perl supports variable declarations and initializations
within the same statement. Global variables should be
declared at the top of the program file, after the module
imports (use package) .
<x>Perl (file handles)[AUX]$filehandle created from file name string via the open statement</x>Importing files - File

names can be passed as command line arguments. It is the

name that is imported, not the file. The Perl program must
explicitly create a file handle to open the file. Files are not
automatically loaded such as is done in Awk. This requires
appropriate command line arguments and file handle creation
within the application.
open $fileHandle, '<', $fileName;

The < means open for input (read). The file handle is then used to
physically access a file.

<x>Perl (build)[AUX]Process files: foreach, while on $FileHandle</x><x>Perl (file handles)[AUX]Process files: foreach, while on $FileHandle</x><x>Perl (build)

[AUX]Convert $fileHandle to array via <...>, cycle with foreach</x><x>Perl (arrays)[AUX]Convert $fileHandle to array via <...>, cycle with foreach</x><x>Perl

Cycling through files - Since a

(file handles)[AUX]Convert $fileHandle to array via <...>, cycle with foreach</x>

file is considered by Perl to be an array, once opened its lines

can be read in one at a time via a foreach loop and acted upon
just as would be done in Awk.
foreach $line ( <$fileHandle> ) { ... } # Cycle through a file one line at a time.

Processing the file may also be done via a while loop in one of two
ways. The first way relies on using the implicit file handle to array
conversion operator <...> directly, in which case the while functions as
if it were a foreach;
while ($line = <$fileHandle> ) { ... }
An internal cursor is positioned at the head of the file when it is
opened and incremented one line at a time after each while iteration.
<x>Perl (arrays)[AUX]Process file via a shift function on array</x>The second way of cycling through files

relies on using the file handle via the array conversion operator <...>
to create an array, thereby enabling getting/removing the array's
first item via :
@tempArray = <$fileHandle>;
while ( $line = shift(@tempArray) ) { ... }
The while action-block can work on the variable $line which will
contain the current line of text. Each time shift is called, the variable
$line gets the first text line from the array. It is then removed from the
array, thereby preparing it for the next shift. Cycling stops when the
temporary array has been emptied. This get the item on the top of
the pile and remove mechanism is unique to Perl; C++ does not
support such a mechanism.

Implementing Awk-like rules (condition/action-blocks) means

cycling through the lines in the file so that they are processed
one by one and applying condition/action-blocks, with regex
pattern matching and regex parens-based captures, on each
line. Awk rules can be simulated via an if statement on a regex
pattern match attempt. In Awk, you could write a rule in one of
two ways, using a regex or using a logical statement. For
example:
/\s*Sasha/ { ... } # regex - does the line contain the five characters S, a, s, h, a
$5 == "Sasha" { ... } # logical statement - does the fifth token contain the string "Sasha"

In Perl, you would look for a regex match as illustrated by

$line ~= /\s*Sasha/

Perl supports regex based rules only, i.e., match text against a
regex:
if ( $line =~/\s*Sasha /) { ... }
 Simply put, if text in the variable $line matches the
<x>=~[AUX]Perl pattern matching operator</x>

regex (=~ means matches), do work. In other words, if $line contains

the five consecutive letters Sasha anywhere, the match is successful.
If the regex contains parens pairs, a match will capture the parens
pair contents, one open parens at a time, from left to right, into $1,
$2, etc. Note that the two forward slashes surrounding Sasha are not
part of the regex - they signal start and the end of the regex.

5.
Simulate Awk using Perl
<x>Perl (Awk vs. Perl)[AUX]Perl is better than Awk</x> Many choose Awk because it is a highly
specialized text processing tool which contains a regular expressions
(regex) pattern matching engine uniquely adapted to do its job; i.e.,
process lines of text contained in text files by applying a set of rules
on each line, thereby accomplishing work based on a line's content.
Awk reads in files, concatenates them together into one continuous
set of text lines, and reads lines one at a time. The programmer will
have defined a set of rules (condition/action-blocks) which will be
applied to each line. Having said this, since Perl can do the same
and much more, why not use Perl. Sure, the learning curve is
steeper, but you can do so much more, such as processing a set of
files a number of times, not being limited to one pass such as with
Awk.
<x>Perl (Awk vs. Perl)[AUX]Simulate Awk with Perl or how to transform Awk program to Perl</x>Before delving on how to

simulate an Awk program or convert an Awk program into a Perl

program, one must first look at what an Awk program does. The
following is an overall schematic presentation of what an Awk
program does:
Do the BEGIN block
# Get the files as list in the command line arguments
while ( get file from file list ) { # this is automatic in Awk
 while ( get line from current file ) { # this is automatic in Awk
# apply the rules one at a time on the current line
if ( match1 on line is successful ) { action-block 1 }
if ( match2 on line is successful ) { action-block 2 }
if ( match3 on line is successful ) { action-block 3 }
...
} # handle line
} # handle file
Do the END block
The code schematic above demonstrates Awk's handling of a set of
files. You will find full details on simulating Awk with Perl in the Perl
simulate Awk (page 320) section.
<x>Perl (Awk vs. Perl)[AUX]Process same set of files several times - Awk limited to processing set once only</x>Using Perl instead of

Awk to process rules on file text lines provides multi-pass file

processing capabilities via multiple foreach cycles on the file set. A first
foreach cycle could fill an array with data. A second cycle could do
work on the lines of text based on the contents of the array obtained
from the first pass. Example: Do a first run to get enum names, then do
a second one to identify enums declared but not used and enums used
and where. The same could be done on #defines.

6.
Perl vs. C
<x>Perl vs. C[AUX]Perl code resembles C but there are notable differences</x><x>Perl (general)[AUX]Your C knowhow, a false friend when learning Perl</x><x>False

Perl code resembles C but there are

friends[AUX]Your C knowhow, a false friend when learning Perl</x>

notable differences. A prior knowledge of C/C++ could disserve you

so, approach Perl with caution. Be aware of the following:

Perl is a command line

<x>Perl vs. C[AUX]Perl is a command line utility/interpreter</x>

utility/interpreter whereas C/C++ relies on three tools:

preprocessor, compiler, and linker (GNU C++, Microsoft Visual
Studio...) to create a stand-alone executable file. C/C++ are
multifile programs. Both Awk and Perl are based on a single
file which contains the program.
<x>Perl vs. C[AUX]# sign same as C++ comment - ignore up to end of line</x>The # sign signals that the rest

of the line is a comment. The C++ // comment is not supported;

 neither is the C-style /*...*/ comment.
Perl
<x>Perl vs. C[AUX]Special character identifies variable's type: $ scalar, @ arrays, % hashes</x><x>Perl vs. C[AUX]Pointers not supported</x>

has three variable types: scalar, arrays and hashes - no integers,

no signed/unsigned, and no pointers. All variables take a special
character to identify their type: $ for scalar, @ for arrays, and %
for hashes.
<x>Perl vs. C[AUX]Arrays are associative; index is integer based</x>Perl Arrays are associative, integer

index based. There are no array declarations, they are defined

when first used. Not all data items within an array's index
range occupy a memory location. This means that the array
could contain undefined items.
<x>Perl vs. C[AUX]print statement items not necessarily inside parentheses</x>The print statement does not

require that the print list be enclosed in parentheses.

<x>Perl (strings)[AUX]Supports literal strings (single quotes) and interpolated strings (double quotes)</x>Perl introduces two

types of strings: literal strings (single quotes) and interpolated

strings (double quotes). The later enables variables to be
located inside the quotes and have their value printed: print
"\$myVar = $myVar" will print $myVar = 5. The first $myVar is escaped,
therefore printed as such, the second $myVar is taken as being a
variable despite the quotes, its content gets printed. This is
referred to as string interpolation.
<x>Perl vs. C[AUX]Regular expressions supported</x>From the beginning, both Awk and Perl

have incorporated a regular expression (regex) engine. This is

one of the features which makes them similar, and which
differentiates them from C/C++, at least until C++11 which
does support regexes.
<x>Perl (variables and functions)[AUX]Functions have a name and a body but do not have a parameter list</x><x>Perl (variables and functions)[AUX]Functions may

Functions parameter passing is

be called with parameters - unique way of handling (via array @_)</x>

very unusual - it requires a learning curve: See Perl

subroutines (functions) (page 322).

Semicolons to end statements are not required; I strongly urge using

them systematically.
The above is by no means exhaustive. Perl is extremely rich, making
it a viable full-fledged programming language.
 7.
Perl caveats
I did find a minor problem using regexes using Perl. If you try a
capture and nothing gets captured, you might get a cryptic
uninitialized variable error message. This can be easily overcome;
see Regex groupings (page 198), end of section.
 18.
From back of the envelope to final product

From back of the envelope to final

product
You are reading this book because you are creating some gizmo
which requires getting a C++ program into a microcontroller so that it
could accomplish some useful work. The microcontroller is
connected to some electronics which seat on a board (be it a
breadboard, a prototype soldered board, or a PCB). The PCB sits in
some enclosure and there most probably are devices which need to
be supported (servos, motors, LEDs, switches...). This means that
you will need tools to create schematics, create PCBs, and a CAD
tool to design parts for 3D printing.
When I started work on my Arduino-based beehive weighing system,
I soon found that hand drawn schematics were not good enough. I
discovered Tinkercad (page 226), a free Autodesk tool which
enables one to design circuits visually. You can lay down
breadboards, connect wires from components on the breadboard to
an Arduino, and generate a schematic. You can even incorporate
C++ code and thus simulate running the schematic, as long as you
keep it simple. It is a useful tool to get started on a circuit. You may
have seen breadboards in YouTube presentations.
Once you are past the breadboard and prototype board stage, if you
want to migrate up to the PCB stage, you will need a schematic
which can be converted to Gerber files to manufacture your PCB (I
used Eagle (page 227) to design the schematic and generate the
Gerber files). Your PCB having newly arrived, you will need to create
home made parts to enclose it in, holders to house various
components, and other paraphernalia. I had used 123 Design from
Autodesk to create 3D components but Autodesk discontinued in
favor of Tinkercad (page 226) or Fusion 360 (page 227), a
professional grade CAD design tool, a sort of mini version of
AutoCAD. As luck would have it, Autodesk offered a free full features
Fusion 360 license for hobbyists, constrained by a limited number of
active designs. It has become my tool for creating 3D parts which I
export to my Zortrax M200 3D printer (see 3D printing page 228).
Eagle (page 227) is one of the best tools to create schematics and
the corresponding Gerber files to fabricate PCBs. Autodesk
purchased Eagle Software and integrated Eagle into Fusion 360.
They made it so that you can export a Tinkercad schematic to Fusion
360, refine your schematic and from there, create your PCB.
Fantastic! Furthermore, I was able to import my old Eagle
schematics into Fusion 360. After dedicating time to this new
enhanced Eagle software, I concluded that it was the way to go.
There is of course a learning curve, but the skills acquired are
immensely useful: create schematics, use the schematics to create
the boards, then use the Fusion 360's 3D design features to create
the hardware and export the files as .stl files for your 3D printer.
Do yourself a favor; read the following introductory sections to
discover a complete gizmo fabrication toolchain. See Tinkercad
(page 226), Eagle (page 227), Fusion 360 (page 227), and 3D
printing (page 228), and Fused deposition modeling FDM (page
229).

1.
Tinkercad
<x>Fusion 360[AUX]Can import Tinkercad schematics</x><x>Tinkercad[AUX]Create visual breadboard wiring/schematics</x><x>Tinkercad[AUX]Schematics exported to Eagle

(Fusion Unlike most IDEs, Tinkercad is a Web application

360)</x>

([Link] You don't download it; you create a link and

open an account to log in. Your designs are stored by Autodesk
somewhere out there. This means that you need a fast connection.
ADSL would probably not cut it; so, if you work remotely in some
lonely place far from highspeed Internet, you may encounter some
difficulties using it.
Starting out with Tinkercad is just like starting out with a new Arduino
project. Get a breadboard, an Arduino Uno, and start putting
components on the breadboard. Connect these to each other and to
your Uno via colored wires. Once your design has advanced
enough, you can add C++ code to simulate its operation (turn a LED
on and off, reset the board via a button, etc.). You can also generate
a schematic (PDF) and a parts list. Nifty. The interface is quite user-
friendly and can be a great time saver getting breadboard prototypes
to work.
Icing on the cake, Tinkercad allows the schematic to be exported to
Eagle, now part of Fusion 360. Having finished with the breadboard
part of your design, you can pass on to a soldered prototype and to a
PCB using Eagle.
That is not all that Tinkercad can do. It is also a CAD design tool for
you to create 3D objects. It does not have the power of Fusion 360
but you can do fairly complex stuff with it.
Tinkercad also proposes Codeblocks (do not confuse this with the
C++ IDE Code::Blocks (page 111) introduced earlier in this book).
When creating 3D objects with Tinkercad, code is what lies behind
the user interface; code is generated each time the user does
something. This is what most applications do. They are driven by
proprietary languages designed specifically for them. Word and
Excel, for example, can be made to do things via VBA programs.
You can use JavaScript to automate InDesign work you would
otherwise do manually. When you do something inside Tinkercad,
code is generated to do the work. This is why you can undo work;
code gets rolled back. Codeblocks is behind the scenes code
rendered visible so that you could modify something in the code and
thus undertake changes in the 3D object being worked on. It renders
changes much easier to undertake and less error prone. For
example, if you want to put a cylindrical hole through a part, you will
create a circle and extrude it out through your part in cut mode. If
you wanted to make the hole smaller, you would normally have to
refill the hole and recut with a new cylinder. Using codeblocks, all
you need to do is change the diameter of the cylinder in the
cylinder's codeblock, the hole gets automatically resized.
The bottom line is that Tinkercad enables you to design the
electronics, insert C++ code of an Arduino-based gizmo, and test it
before doing any physical work. It also enables you to create the
physical parts of your project. I shall stop here and leave it up to you
to look at Tinkercad ([Link]

2.
Eagle
<x>Fusion 360[AUX]Includes Eagle software (electronic schematic and Gerber files)</x><x>Eagle software[AUX]Electronic schematic and Gerber files; now in Fusion

Eagle is one of the major professional grade electronic schematic

360</x>

and PCB design tools. I used it on numerous occasions to design

through-hole based PCBs for my Arduino-based beehive weighing
system. Since Eagle is now integrated inside Fusion 360, you can
design your PCB and the surrounding hardware as a coherent entity.
The PCB becomes a component of your gizmo as opposed to being
an independent piece of hardware.
When developing an electronic circuit from within Eagle, you start
out by designing the schematic. An extensive library of electronic
components enables you to pick the ones you wish to integrate
inside your circuit. And if a specific component does not exist, you
can create your own; you create its schematic representation and its
physical 3D layout.
When your schematic is sufficiently advanced, you can pass on to
the physical layout. You are presented with a board (single layer,
double layer, or multi-layer - your choice) and all your components
sitting outside the board, interconnected as per your schematic. You
then start moving and placing components inside the board and
position electrical traces on the top layer, the bottom layer, or
intermediate layers, with through holes to establish electrical
conductivity from layer to layer; doing so is somewhat of an art.
Traces are the little printed circuit lines which conduct electricity from
one place to another - on a breadboard or prototype board, these
would be wires.
It takes time to fit the parts and pieces together in such a way that
there are no short circuits. You obviously cannot have a ground trace
cross a 5V trace. One first approach is to place all 5V traces on top
the top layer and all ground traces on the bottom layer. Having done
this, you then need to place the other traces. At this point you may
have to rearrange your schematic so that traces would run parallel to
each other. For example, when connecting an LCD display to an
Arduino, you need to connect 4 or 8 LCD data terminals to 4 or 8
Arduino digital ports. You should choose your Arduino ports in such
a way that data traces should run parallel from the LCD to the
Arduino.
Once the schematic is done, and your circuit proven correct after
checking it out on a breadboard and on a prototype board, it is time
to create the PCB (probably though hole first and then SMT (surface
mounted) if you are really ambitious).
The Eagle learning curve is short. Time spent learning how to create
your PCBs with it is well worth the effort. You will feel immensely
gratified when the delivery man rings at your door to deliver a
package containing professional looking PCBs created from your
Gerber files.

3.
Fusion 360
Tinkercad
<x>Fusion 360[AUX]High end CAD design tool</x><x>Fusion 360[AUX]Includes Eagle software (electronic schematic and Gerber files)</x>

may certainly be good enough for some 3D design work. It all

depends on what you are trying to accomplish. It is an entry level
CAD tool which may meet the needs of many of you. But if you want
a more powerful tool, Fusion 360 offers features which enables you
to create the most sophisticated components and assemblies. It can
generate files for 3D printing, laser cutting, CNC machining, and
more. It has rendering features to show what the assembly looks
like; it has animation features to show how the parts interact with
each other (a differential for example); and it has features to plan
tool paths for manufacturing the part. It also can detect interferences
between components.
Fusion 360 may not be as powerful as Autocad, Solidworks, or
Catia, but it certainly offers possibilities which extend far beyond the
needs of practically all Arduino gizmo developers. What's more, it is
free for hobbyists, students, and educators, the limit being the
number of simultaneous active components being worked on, yet
remains full featured.
Fusion 360's underlying user interface concept relies on 2D
drawings which are expanded out into 3D components. For example,
draw a circle; it can be expanded out into a sphere, a cylinder
(extrusion), a doughnut (revolve around an axis offset from the
circle's position), a pipe (sweep along a path), etc. This initial
drawing is referred to as a sketch (do not confuse Fusion 360
sketches with Arduino sketches). When you create a second solid
which intersects with the first solid, the second one can fuse with the
first one or cut it. For example, a cylinder through a cube can create
a hole. As you create 3D parts for your gizmo, you can do things to
edges such as beveling them, or to surfaces, such as tweaking them
in or out. These are referred to as features.
And if you created your PCB inside Fusion 360, you could create an
enclosure to house it in, complete with openings for switches, LEDs,
whatever. Having used a common coordinate system, you can place
them together and have Fusion 360 check for interferences.
There are many CAD tools, some of which you might need to pay
for, others being free; cost is not the only criterium. There is also
reliability, tech support, longevity, features... Autodesk being a major
player in the CAD field, I felt that I was on safe ground and since I
had come to Fusion 360 by accident since I had started with 123
Design (legacy entry level Autodesk free 3D CAD tool) and that it
fully met my needs, I looked no further. My toolchain today is
AtmelStudio for C++ hardware-based debugging, Visual
Studio/Visual Micro to create C++ code, Tinkercad to create
breadboard prototypes, Eagle in Fusion 360 to enhance schematics
exported from Tinkercad and create PCBs, and finally Fusion 360 to
create 3D components. As for 3D printing, I use a Zortrax M200
([Link] All this software is free and full featured
(except for Visual Micro - 25$/yr and Fusion 360 limited as to the
number of active components being worked on). If my work were to
become more substantial and/or become a commercial venture, I
would pay subscriptions according to need.
 4.
3D printing
<x>3D Printing[AUX]Several 3D printing techniques available</x><x>3D Printing[AUX]3D printing starts with CAD tool</x><x>3D Printing[AUX]Use Fusion 360 to design

The final
part</x><x>3D Printing[AUX]Standardized file format .stl for 3D printing</x><x>3D Printing[AUX]I chose a Zortrax 200M 3D printer</x>

component of the gizmo fabrication toolchain is 3D printing. Having

fabricated your PCB and defined interconnections with the outside
world, you may need physical enclosures, adapters, supports, gears,
axles, whatever. Some of these you can buy; others, you will have to
fabricate. My solution is to create an object with Fusion 360 and then
3D print it. Having completed the 3D design, the next step is
exporting it as an .stl file for 3D printing; it can also be exported it as
an .svg file for laser cutting. The question remains: Which 3D printing
technology should be used? The criteria are:

Size - The part may be large; you may consequently need to

create it as a subassemblies or it may be so large that you
may not find any suitable 3D printing solution; you would
consequently have to fabricate it by other means.
Material - Given the stress levels and environment the gizmo
will be subjected to, you may need to choose a material not
adapted to 3D printing. Your design criteria may be met by low
melting point plastics to (LED arrays support) or they may be
such that sintered tungsten imposes itself (jet engine turbine
blades).
Complexity - If all you need is a box, buy one and adapt it to
your needs, or you might need a box with inserts, holes,
flanges, in which case you might be better off fabricating one,
possibly via 3D printing.
Precision - How precise should the fabricated part's
dimensions be? 2%, 0.1%? Your decision will impact your
technical options.
Physical characteristics - What stresses will the part be
subjected to? Some 3D printing technologies will not do as the
 parts may end up being too fragile.
Environmental factors -Will the gizmo be outside, in the cold,
subject to salt spray, or inside, in a dry location at room
temperature?
Cost - Some 3D printing techniques are industrial grade, not
within the reach of Arduino makers. You will find others quite
acceptable.

The above is a short list of criteria. Here is a list of various

technologies which may be characterized as 3D printing:

Fused deposition
<x>3D Printing[AUX]Fused deposition modeling, Stereolithography, Selective layer sintering</x>

modeling FDM (next section) - The plastic is extruded through

a heated nozzle as thin layers which pile up to create the
object. The physical properties differ according to direction,
parallel to 3D print plane or perpendicular to it.
Stereolithography SLA - Liquid resin gets hardened by a laser
one layer at a time. This technology is particularly suited for
small objects and precision fabrication. Physical properties are
homogeneous.
Selective layer sintering SLS - This process is similar to the
SLA process above, except that the medium is a powder
(plastic, metal...).

There are other techniques; but, they are mainly adapted to

industrial grade fabrication of complex components.
The Arduino gizmo maker will be limited by cost, space constraints,
and ease of fabrication. The first process described above, FDM,
IMHO, is the most practical and cheapest solution for fabricating 3D
parts (see next section).

5.
Fused deposition modeling FDM
<ALSO>3D Printing[AUX]Fused deposition modeling FDM</ALSO><x>Fused deposition modeling FDM[AUX]Cheapest most practical for 3D printing</x><x>Fused deposition

As the Fused
modeling FDM[AUX]Deposit molten plastic via nozzle</x><x>Fused deposition modeling FDM[AUX]Each printer mfg. has own slicer tool</x>

deposition modeling FDM name implies, this 3D printing technique is

based upon depositing a molten something unto a base, one thin
layer at a time. The idea is simple. Take a part, any part, slice it up
into thin slices. Each slice will have its unique shape; stacked on top
of each other, correctly placed, recombined, they create the part.
This is basically what 3D printing is all about. The process may be
summarized as follows:

Create an object with a suitable tool (I chose Fusion 360).

Have the CAD tool export the object according to a
standardized file format (.stl).
Define how the part will be printed - When deciding how to
print the part, two important factors must be brought in: 1 - the
part is considerably more solid along the print plane, i.e. print a
tube longitudinally or vertically? And 2 - overhangs need
support.
Define supports to handle overhangs - If you print the tube
longitudinally, it will be more resistant to flexion along its axis,
but a support must be printed to prevent the filament from
collapsing as the top part of the tube gets printed. Beyond a
certain angle, say 30°, the filament will tend to collapse. If you
printed the tube horizontally, you would have to physically
remove the support from inside the tube. If the tube were S-
shaped, you might find it difficult or impossible to remove some
of the tube's internal support material.
The Slicer tool generates support for overhangs as described
above and will cut up the object into thin slices; The result is
converted into machine tool commands. Most 3D printer
manufacturers provide their own slicer tool. This process
generates a print file which is specific to a given 3D printer.
You are all set - Import the 3D print file into the 3D printer and
press the print button.

My personal choice for 3D printing has been Fusion 360 along with a
Zortrax M200 printer. The reason for Fusion 360 is that it is the only
high-end professional grade tool available free of charge for
hobbyists I could find. As for the Zortrax M200, after going to 3D
print expos, talking with various people, looking at YouTube videos, it
seemed to be the best compromise. Neither cheap, nor outrageously
expensive, it proved to be reliable; part size was acceptable
(20x20x20 cms.); overhangs were adequately handled; its slicer
seemed efficient; the range of materials was fairly extensive; the
print area is enclosed thereby enabling the print surface to remain
hot; in a nutshell, a favorable experience. Furthermore, support and
warranty were excellent. I purchased the machine six years ago. My
next machine would probably be the Zortrax M300, one reason being
its auto detection of out of material (the M200 just keeps on printing
with an empty nozzle), another reason being support for rubbery like
materials.
 19.
Appendix

Appendix
This appendix contains material which expands upon the main body
of this book. It covers the following:

RAM requirements (next chapter)

<x>Code size[AUX]Are the IDEs as efficient code size wise</x>

depends on the toolchain being used and on the build options.

Ascertaining which IDE yields better code requires that the
objective be defined. Is it to be size or speed? Applications
may be under memory constraints or execution speed
constraints. It all depends. In my case, code size takes
precedence over speed. At this point, it is a given that
AtmelStudio and Visual Studio/Visual Micro are far more
programmer friendly tools than the Arduino IDE to develop
C++ code with. From a code size standpoint, which to use
remains debatable.
DOS box (appendix) (page 237) and DOS batch files (page
241) primer - Old programmers, raised on IBM PCs or on Unix,
are familiar with command line interfaces. This will usually not
be the case with younger programmers who have always
worked from within graphical environments. The DOS box
([Link]) opens a window which simulates a command line
console. It is supports old DOS commands and can be used to
launch command line utilities such as Awk, Perl, the GNU
compilers, and avrdude.
<x>Frameworks[AUX]Application independent tools designed to accomplish specific tasks</x> Frameworks (appendix)

(page 243) - These are toolboxes which meet specific needs.

The appendix covers additional details on most of them.
<x>Misdoings[AUX]Dumb mistakes not detected by compiler; found by Perl [Link] program</x> Misdoings

(appendix) (page 293) - The compiler will catch most typos one
makes. But a few elusive ones, exceedingly difficult to find,
may cause insidious glitches. A Perl program helps uncover
silly mistakes just as = instead of == in if or while statements.
Memory structure (appendix) (page 296) - The various types of
macros are expanded upon (memory use, fragmented and
contiguous memory, handling fragmentation, freeing and
deleting memory).
The Awk (appendix) (page 301) contains additional details on
using Awk.
The Perl (appendix) (page 318) contains details on
downloading Perl, using Perl, file handling, subroutines,
strings, lists, and many other constructs, are expanded upon .
 20.
RAM requirements

RAM requirements
Compilers are not created equal; some may be better than others for
compactness, others for speed. Add to this the fact that compile/link
options can have an enormous influence on an application's code
size. Since the IDEs compared use the same GNU compiler, which
compiler to use is not a code size criterium, compiler options define
code size.
You will find below two sections, one which compares compiler code
sizes, the other examines compiler/link options.

<x>Code size[AUX]Identical compile/link options yield different RAM/flash memory requirements</x> Code size (IDE
comparisons - next section) - Testing for code size under
seemingly identical build options yield different RAM and flash
memory requirements.
<x>Code size[AUX]Code size depends on compiler options and on -flto flag; reduces size by ٤٠٪</x> Code size (build

options) (page ٢٣٣) - Given that the same GNU compiler is

used by the IDEs, code size depends on the compiler and
linker options. By default, AtmelStudio generates code which
requires considerably more RAM than the code generated with
the Arduino IDE or with PlatformIO. It turns out that a missing
linker optimization option (-flto) was the cause. Having fixed
this, AtmelStudio RAM requirements turned out to be about the
same as compiling with the Arduino IDE or with PlatformIO;
however, flash memory requirements turned out to be quite
different.

1.
Code size (IDE comparisons)
<x>-flto flag[AUX]Code size depends on -flto flag; reduce by 40%</x><x>Code size[AUX]Code size depends on compiler options and on -flto flag; reduces size by 40%</x>

From
<x>Memory (pools)[AUX]PROGMEM and F() macro reduce code size</x><x>Code size[AUX]RAM used depends on IDE and use of PROGMEM and F() macro</x>

a RAM requirement perspective, as measured by available heap

space upon application startup, the Arduino IDE and PlatformIO
produced equivalent code sizes, but AtmelStudio required
approximately 40% more RAM than the other two IDEs. To make a
long story short, it turns out that both the Arduino IDE and
PlatformIO use the -flto (link time optimization) option by default;
AtmelStudio does not. After plugging in this option into the
AtmelStudio build, I got equivalent RAM requirements. Flash
memory requirements were not quite the same, but this is another
issue. See next section Code size (build options) (page 233).
<x>AtmelStudio (caveats)[AUX]Code size by default 40% too large, needs -flto flag</x><x>PlatformIO (build)[AUX]-flto and -fpermissive flags included by default in PlatformIO

Once the -flto

project properties</x><x>Arduino (build)[AUX]-flto and -fpermissive flags included by default in Arduino IDE project properties</x>

option has been added to AtmelStudio, the three IDEs revealed

equivalent RAM requirements although the Arduino IDE is slightly
better by approximately 0.7% which is not significant. However, the
Arduino IDE is much better at using flash memory, PlatformIO next,
and AtmelStudio last. These are tests I undertook on my MemMgt
application.
IDE with -flto flag Flash use RAM use Heap space at startup
Arduino IDE 1.9.18 8202 724 7426
AtmelStudio 10434 760 7383
PlatformIO 9352 754 7396
Table 20.1 - RAM memory <x>Tables[AUX]RAM memory and flash memory requirements</x><x>Memory (use)[AUX]RAM memory
and flash memory requirements</x>and flash memory required by the three IDEs to run the

MemMgt application with the -flto option.

<x>Code size[AUX]Compile time code size corroborated by heap space at startup</x>The table above shows results on

building my MemMgt application with the three IDEs (-flto link option
applied). They show that heap space required by the MemMgt
application at program start corroborates the memory requirement
figures reported by the IDEs. The Arduino IDE's build reports 5%
less RAM needed however this slight improvement might not be
significant.
<x>Code size[AUX]Develop with AtmelStudio, final release may be preferable with Arduino</x>My final analysis is that
AtmelStudio is a great tool for both developing code and building the
final release. It increases programmer productivity and facilitates
more robust code. However, there might be an edge in building the
final release with the Arduino IDE due to slightly less RAM and
considerably less flash memory requirements.
Note that these tests were undertaken before my exploring the
Visual Studio/Visual Micro combination and Arduino V2. I shall
venture to say that these would yield similar results.

2.
Code size (build options)
<x>Code size[AUX]Code size depends on compiler options and on -flto flag; reduces size by 40%</x> When comparing the output
of one development tool with that of another, one should make sure
that build options are similar. What compiler options were used? Was
it Debug or Release? Was the output optimized or not? There may
also be other pertinent code size or execution speed options.
<x>Code size[AUX]AtmelStudio code size by default 40% too large, needs -flto flag</x><x>AtmelStudio (caveats)[AUX]Code size by default 40% too large, needs -flto flag</x>I

started by verifying the impact of AtmelStudio's build options on code

size. Since AtmelStudio's executable RAM requirements were, at
first, considerably greater than the Arduino IDE's and PlatformIO's, I
tweaked all the compiler and linker options in AtmelStudio to
determine whether options could explain such a large difference.
Regretfully, none of the compiler options proposed in the Project
Properties windows seemed to have a significant impact on code
size.
<x>-flto flag[AUX]Code size depends on -flto flag; reduce by 40%</x>From these results, I concluded that

AtmelStudio was missing an option which would reduce RAM

requirements. It turned out that the -flto linker optimization option
was missing. I discovered this discrepancy by comparing build
options from the three IDEs. I list them here since they enhance
one's vision as to what is going on when one presses the
compile/build button or menu item.

The Arduino IDE's compile/link command is:

 <x>Code size[AUX]Code size depends on compiler options and on -flto flag; reduces size by 40%</x><x>Arduino (build)[AUX]Build command

"C:\\Program Files
details</x>

(x86)\\Arduino\\hardware\\tools\\avr/bin/avr-g++"
-c -g -Os -w -std=gnu++11 -fpermissive -fno-exceptions -
ffunction-sections -fdata-sections
-fno-threadsafe-statics -Wno-error=narrowing -flto -w -x c++ -
E -CC -mmcu=atmega2560
-DF_CPU=16000000L
-DARDUINO=10812
-DARDUINO_AVR_MEGA2560
-DARDUINO_ARCH_AVR
"-IC:\\Program Files
(x86)\\Arduino\\hardware\\arduino\\avr\\cores\\arduino"
"-IC:\\Program Files
(x86)\\Arduino\\hardware\\arduino\\avr\\variants\\mega"
The AtmelStudio executable was built in NDEBUG (Release)
mode, but this makes no difference since the DEBUG mode
yields the same code size results.
<x>AtmelStudio (build)[AUX]Build command details</x>"C:\Program Files

(x86)\Atmel\Studio\7.0\toolchain\avr8\avr8-gnu-
toolchain\bin\avr-g++.exe"
-funsigned-char -funsigned-bitfields -c -std=gnu++11 -
fpermissive -v -MD -MP -Os
-fno-threadsafe-statics -ffunction-sections -fdata-sections -
fpack-struct -fshort-enums -mrelax -w
-mmcu=atmega2560
-DNDEBUG
-DF_CPU=16000000L
-DARDUINO=108012
-DARDUINO_AVR_MEGA2560
-DARDUINO_ARCH_AVR
-DUSB_VID=0x2341
-DUSB_PID=0x0010
-DUSB_MANUFACTURER="\"Arduino LLC\""
-I"C:\Program Files
 (x86)\Atmel\Studio\7.0\Packs\atmel\ATmega_DFP\1.3.300\incl
ude"
-I"..\\..\ArduinoCore\include"
-I"..\\..\ArduinoCore\include\core"
-I"..\\..\ArduinoCore\include\variants\mega"
-B "C:\Program Files
(x86)\Atmel\Studio\7.0\Packs\atmel\ATmega_DFP\1.3.300\gcc\
dev\atmega2560"
The PlatformIO command line is:
<x>PlatformIO (build)[AUX]Build command details</x>-c -fno-exceptions -fno-threadsafe-

statics -std=gnu++11 -fpermissive -w -v -Os -Wall

-ffunction-sections -fdata-sections -flto -mmcu=avr6
-specs=device-specs/specs-atmega2560
-D PLATFORMIO=40301
-D ARDUINO_AVR_MEGA2560
-D F_CPU=16000000L
-D ARDUINO_ARCH_AVR
-D ARDUINO=10808
-I include -I MemMgt
-I C:\Users\agir\.platformio\packages\framework-arduino-
avr\cores\arduino
-I C:\Users\agir\.platformio\packages\framework-arduino-
avr\variants\Mega2560

Build commands, as illustrated above, contain a lot of text. I copied

the three groups of options and removed those which were common
to all three and those which would certainly not have an impact on
code size, such as -w for warnings, -v for verbose. Other options
were common to all three environments - these would not explain the
difference. And finally, the I (include directories), D (macros) and B
(AtmelStudio specific) options were also removed, since they would
have no impact on code size. Here are the results after removing
common and irrelevant options:
Arduino IDE options not in the AtmelStudio options:
-flto -E -CC
AtmelStudio options not in the Arduino IDE options:
 -funsigned-char -funsigned-bitfields -c -MD -MP
-fpack-struct -fshort-enums -mrelax
PlatformIO options not in the AtmelStudio options:
-flto
It immediately became obvious that AtmelStudio did not use the -flto
option (link time optimization). Rebuilding the AtmelStudio
application with this option manually added to the compiler options
reduced code size to the equivalent Atmel IDE and PlatformIO
levels.
Why does AtmelStudio not propose this option in one of their Project
Properties choices? I do not know, but it definitely should. I consider
it to be an important AtmelStudio flaw. More on this can be found in
the AtmelStudio caveats (page 62) section.
The conclusions I have drawn are that:

AtmelStudio does not propose the -flto option (link time

optimization) which has a significant impact on code size
confirmed by an equivalent reduction in heap space measured
at program start. Manually adding this flag to project properties
brought down AtmelStudio's generated code size to levels
equivalent to that of the Arduino IDE and PlatformIO, and
increased available heap space accordingly.
<x>Code size[AUX]Compile time code size corroborated by heap space at startup</x>Determining available heap

space at program entry confirmed that the increase of an

application's RAM requirements translates as a decrease in
available heap space, thereby confirming the significance of
the build memory requirement numbers.
<x>Code size[AUX]RAM used depends on IDE and use of PROGMEM and F() macro</x>Flash memory requirements

are significantly lower with the Arduino IDE then with the other
two IDEs. I have no explanation for this since the executable
physical code sizes should be equivalent.
AtmelStudio and PlatformIO use the GNU C++ version 5.4.0
compiler. I could not determine the Arduino IDE's compiler
version. All three yielded __cplusplus equals 201103, which means
that they are C++11 compliant. Determining the compiler
version remains an issue with the Arduino IDE. However, since
the build sizes are equivalent, I am making an educated guess
that the Arduino IDE uses the same compiler version as the
two other IDEs.
There remains an unanswered issue: AtmelStudio has a -
DDEBUG option which may imply extra memory requirements.
Recompiling in release mode (NDEBUG) yielded the same
results. Why? Unfortunately, I am unable to provide an
explanation.
 21.
DOS box (appendix)

DOS box (appendix)

<x>DOS box[AUX]Windows DOS PC users today interface with the
box for text interface programs</x>

computer via graphical user interfaces (GUI). Practically all

applications, be they Word, Excel, accounting applications, etc. are
GUI based. The screen is filled with icons, graphical toolbars,
images, and symbols. This has not always been the case. In the
days of the first PC's (IBM PC, Apple II, Atari...) the interface was
text-based. It was rather drab, no pretty graphics. You typed the
application's name and options after the console's command line
prompt. The application got loaded and the PC churned for a while.
Finally, the screen displayed output or wrote the output into a text
file. Familiar tools such as Awk, Perl, avrdude, the GNU C/C++
toolchain are command line based. If you plan to do work with any of
these, you will need to do it from a DOS box (aka command box -
[Link] - old PC text screen interface). So, read on. The following is
a DOS box crash course.

What is a box retraces the history of desktop computers

explaining how the DOS box came into being (next section).
Launching a DOS box explains how to get the DOS box up
and running. See DOS box launch (page 238).
DOS commands - DOS has its own shell, i.e., a command line
language such as Unix has (Bourne shell, Korn shell). They
provide the user with commands to perform various tasks. This
section lists the more common DOS commands (page ٢٤٠).
Batch files are ordinary text files with a .bat extension. They
encapsulate DOS commands so that they could be launched
as one unit, as opposed to typing them all sequentially. Put
your commands in a batch file and run them just as you would
run any executable. See DOS batch files (page 241).
 Other info you should know reveals undocumented DOS
command tidbits. See DOS box - other info (page 242).

1.
DOS box - What is it?
The
<x>DOS box[AUX]Microsoft created DOS operating system for the IBM personal computer</x><x>DOS box[AUX]Text user interface (old Unix type console)</x>

IBM Personal Computer (1981) is the ancestor of our current laptop

and desktop Windows based computers. It became an instant
success because, being 16-bit based, it was the first desktop
computer with sufficient capacity to be a bona fide business
computer. It could run accounting programs. It was eight times more
powerful than its rival, the Apple II, an 8-bit based computer. The
interface was a text screen which displayed 80 columns by 40 lines
of fixed size letters. The operating system was named MS-DOS,
short for Microsoft Disk Operating System, commonly referred to as
DOS. Programs were invoked as command line text such as was
done on Unix terminals. Output was plain text to the console or to
files.
Bill Gates signed a contract with IBM whereby DOS would be
installed on IBM PCs on a royalty basis. At the time, IBM was not
interested in making desktop PCs. It had designed the PC as an
intelligent terminal for its mainframe computers. IBM's big mistake
was to not foresee how small desktop computers would become so
ubiquitous. Its second big mistake was not recognizing that the
software market would ultimately be much bigger than the hardware
market (mainframe computers). These two mistakes partly explain
Microsoft's huge success. Saying an I told you so with the benefit of
hindsight is easy. At the time, things were not at all clear. On a
similar scale, who could have predicted the far-reaching
consequences of the Internet, originally developed as a file
exchange tool for the scientific community. And what about
smartphones, and Facebook?
Apple responded to the newly acquired IBM/Microsoft lead by
introducing the Graphical User Interface (GUI) based Apple Lisa in
1983, a commercial flop, followed by the MacIntosh (the Mac -1984),
a huge success. By this time, the IBM PC was a business-oriented
computer, i.e., applications which did a lot of number crunching. The
Mac had more of an audience with research, universities, and other
sectors which required a graphical user interface. Microsoft
responded in 1985 with Interface Manager 1.0 renamed Windows
1.0 later on. It is only with Windows 3.0 (1990) that IBM PCs with
GUI interfaces received a wider acceptance.
<x>DOS box[AUX][Link] (DOS box) launched from screen or toolbar icon or run menu item</x><x>DOS box[AUX]DOS box in

c:\Windows\System32\[Link]</x>To make a long story short, Windows-

based 1990 PCs had to be compatible with 8+ years of text-based
applications. Microsoft therefore included a Windows utility which
could run these: the DOS box. When the DOS box is launched, it
opens a window which looks like the old IBM-PC's screen. It
supports old DOS commands and more. A category of useful Unix
tools, command line based such as Awk and Perl ported to DOS,
avrdude, the GNU C/C++ toolchain, require that they be run in a
DOS box. This is why the DOS box remains an important Windows
tool for Arduino developers.

2.
DOS box launch
D:\Dev\Atmel>
<x>DOS box[AUX][Link] (DOS box) launched from screen or toolbar icon or run menu item</x><x>DOS box[AUX]Prompt looks like this:

</x><x>DOS box[AUX]DOS box in c:\Windows\System32\[Link]</x>Using a DOS box is not

particularly difficult, but there are things one should understand and
know. The DOS box is an application [Link] located in
C:\Windows\System32. You can invoke it directly by double clicking
on [Link] or by typing cmd in the run option of the Windows start
menu at the bottom left corner of the Window's screen. Upon
invoking [Link], a black window opens. It contains a prompt such
as C:\Windows\System32>. This is the DOS command prompt. From
here on, type anything, the system will react by responding with text
displayed after the text you typed. If the DOS box does not recognize
your text, i.e., it is neither a DOS command nor an executable nor a
batch file. It responds with
<x>DOS box[AUX]Cryptic not found message</x>DNS server not authoritative for zone.

Why this absolutely nonsensical text? I cannot say. It means that you
typed something it does not recognize.
To get a DOS box into your directory, copy paste [Link] from the
C:\Windows\System32 into your directory. Once in your directory,
double click on it. It will open with a prompt pointing to your directory.
Assuming that your directory is D:\AtmelDev\Perl, the command line
prompt becomes D:\AtmelDev\Perl>.
<x>DOS commands[AUX]Do HELP for DOS commands</x>Unix users of old interact with the computer

by typing commands in a terminal's window. DOS is the same in that

it provides the user with a rich set of commands but, before delving
into DOS commands, here are details on using the DOS box.
If you plan to use a DOS box, the first task is knowing where to get
info on using one. YouTube would be a good place. Unfortunately, I
failed to find a decent DOS video. The next source of information is
in-line help: Type HELP after the DOS box command prompt. DOS
will respond with a list of DOS commands. To get help on a specific
command, type HELP command.
The next task is to learn how to change file paths and how to
navigate from one directory to another. When the DOS box opens, it
displays a prompt which identifies the directory the DOS box was
launched from. In the prompt D:\AtmelDev\Awk>, D: is the disk such
as Windows Explorer displays. \ATmelDev is a root directory in disk
D:. \Awk is a subdirectory in \ATmelDev. These three items identify
the current directory's name and location. Notice that the items
which make up the DOS prompt are separated by backslashes, not
forward once as you would find with Unix, this is a Microsoft attempt
at enforcing its sense of self.
<x>DOS commands[AUX]CD command to navigate to other directories</x>Having understood what the command

prompt refers to, the next task is being able to navigate to other
directories by typing CD something (short for change directory) but
first, type HELP CD (aka CHDIR). It will respond with help on using
the change directory command. If you wanted the current directory to
be D:\AtmelDev\framework, you could type:
CD D:\AtmelDev\framework
but fortunately, there is a shorter way which involves relative file
paths. Instead of typing the above, assuming you are in
D:\AtmelDev\Awk>, do:
CD ..\framework
<x>DOS commands[AUX]..\ to backtrack one directory, .\ to proceed from current directory</x>You are saying: go back one

directory and go to framework from there. The ..\ means backtrack

one directory level. There is another shortcut: .\ means from the
current directory. You can also use the backtrack one directory
without a destination to navigate with the CD command. Doing CD ..
takes you one directory up the hierarchy to D:\AtmelDev. Doing CD ..
again takes you to D:. Doing CD .. a third time does nothing since
you are at the disk's root.
One final word on the CD command: If your destination is another
disk, use the /D option when doing a CD something. If you do not,
the command will be ignored, no message. You can also change
disks by typing 'C:'; your new prompt will be C:>.
Now that you understand file paths, how to navigate up and down
directories, and how to go from one disk to another, you need to
know how to invoke an application. When you type something after
the command prompt, if it does not recognize it as being a DOS
command, the system will search the system for executables with
that name, executables being files which have an .exe, .com, or .bat
extension. From a pragmatic standpoint, you do not need to bother
with understanding the difference between .exe and .com files, just
assume that they are identical. You will, however, create your own
text based executable files (.bat - batch files). These are described
further down in DOS batch files (page 241).
<x>DOS commands[AUX]PATH displays or sets environment variables</x>If you type foo, since it is not a DOS

command, DOS will look for a foo executable in the current directory.
If it does not find one, it will search in directories defined by the
PATH environment variables to determine whether one of them
contains it. A typical PATH variable for foo could be
C:\FooInstallDir\bin. If there is indeed such a PATH variable, and
there is a foo executable in the corresponding directory, it will launch
it, otherwise, it will respond with the meaningless message (at least
to me):
<x>DOS box[AUX]Cryptic not found message</x>DNS server not authoritative for zone.

Try this: type PATH in the DOS box, it will respond with a long list of
PATHS which define where Windows will look for executables. A
word of advice: PATH is a system command - do not experiment with
it since mistakes could irreversibly mess up your system. After typing
foo, if Windows finds it as per the PATH environment variables
mechanics described above, foo will be launched and will do
whatever it was designed for.

3.
DOS commands
Now that you understand
<x>DOS commands[AUX]basic set: CD, DIR, DEL, EXIT, MKDIR, COPY, XCOPY, PAUSE, ECHO, REM</x>

how to use the DOS box, you should enrich your know-how with a
basic set of DOS commands. The following list is a good starting
point:

CD, elaborated upon above, is used to navigate up and down

directory trees and disks.
DIR lists files in the current directory and subdirectories. The
output depends on options used.
DEL deletes selected files. Be careful, the command accepts
wildcards. If you type del*, it will erase everything in the current
directory, including subdirectories.
EXIT closes the DOS box.
MD and MKDIR create a directory.
COPY copies something from one file path/file name to
another file path/file name.
XCOPY is an extended version of COPY. I use it to backup
files from selected directories whose timestamp has changed.
One of the XCOPY options is the /D option - copy files newer
than the destination file. I use this feature in a little batch file to
backup critical files to an external hard disk. It works like a
charm.
PAUSE interrupts processing. Type any letter to continue. It is
useful to see the screen's output before it scrolls out of sight. It
 is used in batch files, described below.
ECHO and REM (remark) are used to display a message in
the DOS box as the batch file executes. It may be useful when
combined with pause to monitor program flow.
<x>DOS batch files[AUX]:: undocumented comment for batch files</x>Comments :: - Most programs have

a means to include comments, /*...*/ in C, // in C++, # in Awk and

Perl. DOS batch files use the undocumented '::' at the
beginning of a line to signal that it is a comment. DOS does not
do anything with it. It ignores it. It enables documenting the
batch file's contents.
<x>DOS batch files[AUX]Manage batch file execution: START, TIMEOUT</x>START launches a program in a

new DOS box.

TIMEOUT /t 10 - stop 10 seconds before continuing. This is
necessary when running several programs successively from
inside a batch file. It gives a program time to finish before
another program gets launched.

4.
DOS batch files
A batch file is an ordinary text file you can create with notepad; its
extension must be .bat. Windows considers that it is a DOS box
executable. If you double click on a batch file from Windows
Explorer, Windows will open a DOS box and launch it; the DOS
commands it contains will be executed. Batch files are particularly
useful when command lines are long or when launching several
DOS commands sequentially. This would be the case when
launching Awk and other command line programs.
If you are new to DOS boxes and batch files, try this:

Open notepad and create a file you will name [Link].

Insert the following lines into the [Link] file you just
created:
 :: Example using a DOS batch file
:: REM (remark) displays a message
REM Good, you just started your DOS batch file
:: PAUSE interrupts processing the batch file's commands
PAUSE
REM Continuing processing batch file
PAUSE
REM Finished processing batch file; after this DOS box will
be closed.
PAUSE
Experiment.

Instead of
<x>DOS batch files[AUX]Batch files enable multiple DOS commands</x><x>DOS batch files[AUX]Launch Awk and Perl from batch files</x>

typing Awk or Perl lengthy command lines, it is a lot easier to write

complete commands in a batch file and launch the batch file as
illustrated above. In the example below, the batch file named
[Link] is launched by typing its name along with any
parameters it might require.
GenerateTypes ..\Personnel.h
<x>DOS batch files[AUX]Batch file example</x>The command above launches the
[Link] batch file with one parameter ..\Personnel.h: The
[Link] batch file contains:
::Uses the Awk program [Link]
ECHO [Link] BEGIN
Awk -f [Link] %1 > [Link]
ECHO [Link] END
PAUSE
EXIT
In the above notice several items:

:: means this is a comment; it is not documented in DOS

HELP. Standard comments such as //, ;, #, /*...*/ do not work.
<x>DOS batch files[AUX]ECHO prints a message in the DOS box</x>ECHO prints a message in the

DOS box.
Awk is the name of the program being invoked along with its
parameters.
 <x>DOS batch files[AUX]May take parameters such as %1, %2, etc.</x> %1 is a batch file parameter, there
is only one in this case; otherwise, you would do %2, %3, etc.
In the example above, it would contain the full or relative file
path of the file(s) you wish to process.
<x>DOS batch files[AUX]Displays output or redirects to file (> and >>)</x>> means redirect output to

[Link] or >> to append to [Link]. It could have been

passed as a second argument %2 instead of hard coding it in
the batch file, such as %1, %2, etc.
<x>DOS batch files[AUX]PAUSE interrupts processing, EXIT and ctrl-C closes DOS box</x>PAUSE means just what it

says, interrupt processing the batch file. It gives you time to

look at results. Pressing any key will resume the batch file's
processing.
EXIT means close the DOS box. This command its useful if
you want to stop processing before reaching the end of the list
of commands.

The above is a simple batch file. Since a batch file can invoke other
batch files, you can build a top-down work session hierarchy. Please
note that DOS commands have been capitalized; this is my personal
convention as DOS is not case sensitive.

5.
DOS box - other info
You will find below other details on using
<x>DOS box[AUX]Other details you ought to be familiar with</x>

the DOS box you ought to be familiar with:

Closing the box - Use ctrl-

<x>DOS batch files[AUX]PAUSE interrupts processing, EXIT and ctrl-C closes DOS box</x>

C, or click on the X in the top right corner, or select close from

the top left icon menu, or send the EXIT command.
<x>DOS box[AUX]Copy/paste to/from other apps</x>Copy/paste - The Windows workspace

and the DOS box use the same buffer. It is possible to copy
any content in the DOS box to anywhere outside the DOS box
and vice versa. You can also copy buffer contents to the
command line as you are typing it, but be careful, the
destination is the DOS cursor, a little blinking horizontal
rectangle. It is not wherever the mouse's cursor is. You have to
use the horizontal arrow keys to move this cursor forwards and
backwards along the command line you are typing.
<x>DOS box[AUX]Recalcitrant app? TaskManager (ctrl+alt+del)</x>Stopping a recalcitrant application -

When the screen fades and you get [Link] not responding,
open the TaskManager (CTRL+ALT+DEL), look for [Link] in
the running processes and stop the process. This should close
the DOS box.
Redirecting output - Programs launched from the DOS box
usually put out results inside the box's screen. You can redirect
it into a desired file. To do this, tack on > [Link] (or >>) at
the end of your command. > will replace whatever is in the
destination file with your program's output; >> will append
output to the end of the destination file.
 22.
Frameworks (appendix)

Frameworks (appendix)
<x>Frameworks[AUX]Three types of This appendix provides
frameworks: organizational, data handling, specialized</x>

additional information on frameworks presented in the main body of

this book. Frameworks are grouped inside three categories:

Organizational Frameworks (next section) or how to structure

an application. It includes Project Files Framework, Program
Documentation Framework, and Function Creation
Framework.
Data Handling Frameworks (appendix) (page 253) or how to
use data efficiently. It includes Data Packets Framework,
Format driven float byte conversion, DataGroup, Bitfield
Storage Framework, and Event Storage Frameworks.
Specialized Frameworks (appendix) (page 272) or how to
handle specific tasks. Includes Algorithm Test Framework,
Class and Function Names Referencing Framework, Linked
List Framework, Memory management, Pseudo exception
handling, Error reporting, and General Debugging Macros.

1.
Organizational Frameworks
(appendix)
These frameworks help
<x>Organizational Frameworks[AUX]Three frameworks help structure your files and document source code</x>

structure an application (organize source code). It contributes to

programmer productivity. You will find below additional information
on the following:
 Project Files Framework (next section) - How to avoid double
including header files and how to structure the project's files.
Program Documentation Framework (appendix) (page 244) -
This framework proposes a Perl program to extract
documentation from source code.
Function Creation Framework (appendix) (page 249) - When
creating functions, it is advisable to start by copying/pasting a
complete function template skeleton.

1.
Project Files Framework (appendix)
<x>Organizational Frameworks[AUX]Project Files Framework - Organize your project's files</x><x>Project Files Framework[AUX]Common sense dictates how to organize an

As described in the body of the Project Files Framework

application's source code</x>

(page 152) presentation, common sense and practical rules to

organize an application's source code render an application more
robust, source code more readable, and improves the programmer's
productivity. This chapter covers several details relating to a project
file's organization.

Avoiding header file multiple

<x>Header files (.h)[AUX]#ifndef prevents multiple .h file inclusions</x>

inclusions is accomplished by encapsulating header files (.h)

within a #ifndef...#define...#endif macros, thus:
#ifndef MY_CLASS_H
#define MY_CLASS_H
...
...
#endif // MY_CLASS_H
The two main files
<x>Project Files Framework[AUX]Globals.h/.cpp contain #includes and global variables; macros in Macros.h file</x>

to include in all .h/.cpp files are Arduino.h and Globals.h. The

Arduino.h should be located in the top of the Globals.h file so
that all project files should have access to its content. Organize
header file inclusions into one master Globals.h file. It is the
pivot around which all .h files are included. Its structure is:
 #ifndef GLOBALS_H
#define GLOBALS _H
#include <Arduino.h>
#include "Macros.h"
...
global definitions
...
#include "HelperFunctions.h"
#include "Functions.h"
#include "FunctionsSKL.h"
#include "BasicClass.h"
...
#include "Atest.h"
#include "Aardvark.h"
#endif // GLOBALS _H

<x>Project Files Framework[AUX]Globals.h/.cpp contain #includes and global variables; macros in Macros.h file</x><x>Header files (.h)[AUX]Arduino.h, Macros.h included in

Notice in the above that Arduino.h and

Globals.h file, #include <Arduino.h> in Macros.h</x>

Macros.h get included at the top of the Globals.h file and that the
application's header files are included at the bottom. This is
because the body of the Globals.h file may require items defined in
Arduino.h and Macros.h. The application's header files require
items defined in the body of the Globals.h file.

<x>Project Files Framework[AUX]Avoid interdependencies between .h files</x><x>Header files (.h)[AUX]Respect header file dependencies (top-down) and beware

Be careful
of interdependencies</x><x>Header files (.h)[AUX]Respect header file dependencies (top-down) and beware of interdependencies</x>

with dependencies, i.e., content which depends on other

content. The inclusion sequence of the application's header
files must account for these. If a specific header file uses items
defined in another header file, the other header file comes first.
Interdependent .h files must be avoided. If two header files
depend on each other (interdependence), you have a problem.
If ClassABC.h contains items defined in ClassXYZ.h and vice
versa, the application must be reprogrammed so that there will
be a top-down dependence and not a sideways dependence.

2.
Program Documentation Framework
 (appendix)
<x>Organizational Frameworks[AUX]Program Documentation Framework - Document your code</x><x>Program Documentation Framework[AUX]Starts with inserting comments,

The first step underlying program documentation

organize them as templates (fill in the blanks)</x>

(Program Documentation Framework page 155), short of formal

documentation written during the design phase, is to insert
comments as you code. It is best to do this from the start when
creating functions. This may be accomplished via comment
templates, one each for files, classes, functions, and program flow.
When creating a file group for a new class (.h/.cpp files), copy a file
template from a skeleton project and start describing the file's
content. Do this in the header file (.h file). After you have
documented the file, document classes in the same .h file, the
methodology is identical with file documentation except that the class
template name differs. As to functions and methods defined in .cpp
files, copy/paste a function template, modify names appropriately,
and fill in the function specific comments: description, parameters,
error checks, and return values. Do this in the .cpp files. Using these
templates and documenting them as much as possible before writing
code has the important side-effect of clarifying ideas, thus reducing
overall development time. The maxim should be: Document your
files and functions right from the beginning.
<x>Program Documentation Framework[AUX]Create program documentation via Awk/Perl regex based comment extraction</x><x>Program Documentation

Having documented your file by inserting

Framework[AUX]Example of comments extraction Aardvark</x>

appropriate comments, it becomes possible to extract these to

create program documentation. The two tools, Awk and Perl, enable
creating a program to selectively extract text via regex-based rules
(see Regular expressions (regex) page 191). It relies on keyword
phrases to extract documentation, which it redirects into an output
file.
The following illustrates file level documentation. It is taken from my
Aardvark.h header file which contains comments describing the file's
content.
 // PROGRAM FILES START
// Traditional Arduino IDE programs have setup() and loop() functions.
// loop() not being used, the program starts and ends in setup()
// Interoperability amongst IDEs achieved by transferring code located in setup of the
// original .ino file into Aardvark, located in Aardvark.h/.cpp.
// setup() remains unchanged, not requiring that the entry files of the IDEs be modified.
// Changes are made in Aardvark, which is IDE independent.
// PROGRAM FILES END
In the above, file
<x>Program Documentation Framework[AUX]Class and file documentation template store class and file info</x>

documentation is encapsulated in templates which contain START/END

keywords used by the regex to activate/deactivate the extraction
process. These keywords set the Perl variable OKtoPrint to true or false.
The algorithm is rather simple, three rules define what needs to be
done:
Rule #1 sets okToPrint to false when it encounters the key phrase
PROGRAM FILES END. Process this rule before the others to stop
printing if printing has been turned on.
Rule #2 - If okToPrint is true, the Perl program extracts the line,
removes the leading comment //, and appends the line to the
output file. Output formatting could be envisioned.
Rule #3 sets okToPrint to true when it encounters the keywords
PROGRAM FILES START. Also prints a header to identify the file whose
description is being printed.
<x>Comments[AUX]Program Documentation Framework - use Awk or Perl regexes to extract comments</x>Before delving into the

details of the three rules above, I shall illustrate the process with the
regex corresponding to rule #1: find the end of the comments block.
If the regex below encounters the keyword PROGRAM FILES END, the
corresponding condition/action-block's code gets invoked:
/^\s*//\s*PROGRAM\s+FILES\s+END/
As usual, regexes get to be a little cryptic but if you patiently
examine its components one at a time, they become clear:
^ means at beginning of line
\s* means zero or more spaces
// two forward slashes (C++ style comment)
\s* zero or more spaces
PROGRAM\s+FILES\s+END three consecutive keywords separated by one
or more spaces.
The outcome is that the expression will match lines which start with a
comment // possibly indented with spaces or tabs into the line,
followed by the three words looked for, no matter how many spaces
separate them.
Project Files Framework contains a Perl-based program which
implements the three rules indicated above. In a nutshell, the
regexes above find the START/END of documentation comments,
thereby setting the state variable $okToPrintLine to on or off which decides
whether to append the line into the output file. The three rules are
illustrated below as constructs:
if ( $line =~ regex ) {
do something
} // if
In the above, =~ is the Perl match operator. It means: if there is
match, the expression evaluates to true, false otherwise.

Program segment #1 - Search for END, set the $okToPrintLine

variable to ٠ (OFF).
# match // PROGRAM FILES END
if ( $line =~ /^\s*(//)\s*PROGRAM\s+FUNCTION\s+END/) {
$okToPrintLine = 0;
print ("$filename:$lineNumber File description END");
} # if
Program segment #2 - Print the current line if $okToPrintLine is ١;
the leading // is removed and the line is appended into the
output file.
if ( $okToPrintLine == 1 ) {
$line = RemoveLeadingComment();
print $outputFileHandle $line; # append $line to the output file
lineCount++;
# if ( $okToPrintLine == 1 )
Program segment #3 - Search for START, set the $okToPrintLine
variable to ١ (ON).
# match // PROGRAM FILES START
if ( $line =~ /^\s*(//)\s*PROGRAM\s+FUNCTION\s+START/) {
$okToPrintLine = 1;
print ("$filename:$lineNumber File description START");
} # if

For the above to work, source code should be formatted accordingly

so that the rules can operate. The output of the above is bare bones
documentation. Here is an Example output:
 Aardvark.h:130 File description START
Traditional Arduino IDE programming has a setup() and a loop()
function.
...
Changes are made in Aardvark, which is IDE independent.
Aardvark.h:145 File description END
The above is quite simple, straightforward, immensely useful.
Properly documenting work does take a little time and attention, but
it is not excessive. It is a question of discipline.

1.
File documentation template
The file documentation template is the same as the program
documentation template. The keywords FILE START and FILE END
encapsulate everything you would like to say concerning the .h/.cpp
file pair. As follows:
// PROGRAM FILES START
// The following is a description of the files .h/.cpp contents
// ...
// PROGRAM FILES END
In the above, anything you may insert between the START and END
comments will be extracted. Since the Perl program knows the name
of the header file being processed, the programmer need not
explicitly insert the file's name when documenting it.

2.
Class documentation template
<x>Program Documentation Framework[AUX]Class andInserting class
file documentation template store class and file info</x>

descriptions into the header source file is handled exactly the same
way as is done for file documentation. There is a minor difference in
that the class's name has been added to the CLASS START keywords,
thus: CLASS START class name .
There should be as many class documentation templates as there
are classes in the file.
 // PROGRAM CLASS START MyClass
// The following is a description of MyClass's content
// ...
// PROGRAM CLASS END

3.
Function documentation template
The function
<x>Program Documentation Framework[AUX]Function template contains description, parameters, error checking, return values</x>

documentation template is a little more elaborate than the file and

class documentation templates. It is better to place function
documentation in the .cpp files, class documentation being more
logically placed in the .h files.
Distinct function features need to be documented: description,
parameters, error checking, and returns, as illustrated by the
following code sample which documents the function CountLinesInFile:
bool CountLinesInFile(int& _totalCount, const char* _fileName) {
// PROGRAM FUNCTION START CountLinesInFile
// DESCRIPTION
// Counts the number of lines in file _fileName.
// Updates the _totalCount variable passed by reference.
// PARAMETERS
// Parameters total: 2
// int& _totalCount - count in _ fileName added to _totalCount
// const char* _fileName - file whose lines are to be counted
// ERROR CHECKING
// Error checks total: 1
// Checks that file exists and that it is not empty
// RETURNS
// Returns bool - success/failure
// PROGRAM FUNCTION END
...
return true;
} // CountLinesInFile()
The template above can be inserted into the function being
documented. It is then up to the programmer to fill in the info as
illustrated above. The key phrases are used by the Perl program to
extract info which is then added to the application's documentation.

4.
 Program flow template
Program flow documentation (who calls who and who gets called by
whom) is based on the use of three commented key phrases
arranged as follows:
// PROGRAM FLOW START callingFunction MyFunction
...
// PROGRAM FLOW CALLS AnotherFunction
x = AnotherFunction ();
...
// PROGRAM FLOW END
The START phrase activates the
<x>Program Documentation Framework[AUX]Who calls who? Who gets called by whom?</x>

program flow capture process; it contains the calling function's

name. This is necessary since the function, inside of which you are
capturing program flow info, might not contain a description
template. Whenever a PROGRAM FLOW CALLS phrase is encountered,
the Perl program extracts the called function's name and creates a
record which contains the calling function, which it knows from the
FLOW START phrase described above, and the called function, which it
knows from the FLOW CALLS phrase. The FLOW END phrase ends the
extraction process. The extracted info is appended to a CSV file so
that it can be imported into a spreadsheet to sort by calling function
and by called function. This responds to the need who calls who and
who gets called by whom.

5.
List macros - Perl program
Macro names get
<x>Program Documentation Framework[AUX]Create program documentation via Awk/Perl regex based comment extraction</x>

extracted via a Perl regular expression. It is easy to extract #defines,

just look at the first token preceded by an optional comment //.
/^\s*(//|\s*)?\s*#define\s+(\w+)/
This regex is saying: start at the beginning of the line ^, followed by
zero or more spaces \s*, followed by an optional comment symbol or
zero or more spaces //|\s*, followed by zero or more spaces \s*,
followed by #define, followed by one or more spaces \s+, followed by a
word (\w+). The comment symbol gets captured into $1 by (//|\s*)?, which
enables active macros to be differentiated from commented out
macros. The macros name gets captured into $2 by (\w+).
The regex will pull out active and commented out macros, Perl will
create a report which lists all macros, active and commented out
ones, with file name and line numbers.

6.
List enums - Perl program
Extracting enums is simple, just look at the first token (this assumes
that enum declarations start on a newline):
/^\s*(//|\s*)?\s*(class\s|\s*)?\s*enum\s+(\w+)/
The enum regex is identical to the macro extraction regex except that
an additional regex segment has been added to account for the
optional class keyword between enum and the enum's name (class\s|\s*)?. The
regex will capture the optional comment into $1 to differentiate active
enums from commented out ones, the class keyword into $2 which is not
necessarily used, and the enum's name into $3.
The Perl program lists all enums, active and commented out ones,
with file name and line numbers.

3.
Function Creation Framework
(appendix)
<x>Organizational Frameworks[AUX]Function Creation Framework - Use function creation template to create new functions</x><x>Function Creation Framework[AUX]Function

template to When professional programmers create functions

create function</x>

(Function Creation Framework page 157), they describe the

function's operation and innards before writing code. Along the same
line of thought, when an Arduino developer creates a new function,
he/she may save overall development time by starting with a function
skeleton which includes comment-based description templates.
Suppose that you want to create the function bool Foo. You can start
coding directly in the .h and in the .cpp files. You get the impression
that you are forging ahead and making progress. Later on, you will
start adding debugging features and will insert function
documentation as comments. This is well and good, but if you were
to invest time by doing preliminaries, you would be rewarded by time
saved later on. When you create a new function, I recommend that
you do so according to a well-established procedure which involves
five steps. These cover structuring code, inserting comments, and
inserting print-based debugging macros.

Step one - Start by creating the function's name and the

function's parameters in the .h and .cpp files, with an ending
semicolon for the .h file and opening and closing braces for the
.cpp file. Make sure that the function's name is meaningful and
that it reflects what the function does. Do not be cryptic. In this
example, Foo is a perfect name since it means this is a throw
away function.
// Foo.h file
bool Foo (bool _handleNegatives, float& _value, char* _callingFunction, int _lineNo);
// [Link] file
bool Foo(bool _handleNegatives, float& _value, char* _callingFunction, int _lineNo) {
...
} // Foo

Notice the commented closing brace. Always do this. It is a great

time saver when you try to untangle missing or extra braces in your
code. #endifs should also be commented if they are far from the
corresponding #ifdef. These will help later during debugging. Also,
notice the leading underscore on the parameters, it distinguishes
them from global variables or local variables. I have also added two
parameters for debugging purposes: the calling function and line
number. These two parameters are positioned systematically as
the last two. No more remains to be done in the .h file, the rest of
the work is in the .cpp file.

Step two - Define the debugging macro for the function and
insert it at the top of the .cpp file. Each function has its own
debugging #define so that debugging prints could be turned on
and off via the corresponding #ifdef. The debugging macros
always start with _DEBUG_ followed by the files's name (FOO) and
the function's name (FOO):
#define _DEBUG_FOO_FOO
Step three - Insert into the function implementation debugging
entry and exit macros and appropriate print parameter macros.
Do not forget the default return value at the end of the function.
In this case it returns a bool value.
bool Foo(bool _handleNegatives, float& _value, char* _callingFunction, int _lineNo) {
#ifdef _DEBUG_FOO_FOO
PRINT_FUNCTION_ENTER
PRINT_VALUE(_handleNegatives)
#endif
// code starts here
... your code
// code ends here
#ifdef _DEBUG_FOO_FOO
PRINT_FUNCTION_EXIT
#endif
return true; // always insert a return at the function's end
} // Foo ()
Step four - Insert the function documentation template to
properly document the function. Write a comment as to what
the function does and the parameters' roles, which can be
expanded upon later.
// PROGRAM FUNCTION START Foo
// DOCUMENTATION
// Describe here what the function does, the algorithm...
// PARAMETERS
// Parameters total: 2
// bool _handleNegatives - needs to be described
// float& _value - needs to be described
// ERROR CHECKING
// Error checks total: 1
// bool _handleNegatives - should evaluate to 0 or 1.
// RETURNS
// Returns bool - success/failure
// PROGRAM FUNCTION END
// PROGRAM FLOW START calling function: Foo
...
// PROGRAM FLOW END
Step five - Comment the function as much as possible and
compile to make sure you are starting out with clean code.
<x>Function Creation Framework[AUX]Complete function skeleton Here is the completed function
example</x>

skeleton:
bool Foo (bool _handleNegatives, float& _value, char* _callingFunction, int _lineNo) {
// PROGRAM FUNCTION START Foo
// DOCUMENTATION
// Describe here what the function does, the algorithm...
// PARAMETERS
// Parameters total: 2
// bool _handleNegatives - needs to be described
// float& _value - needs to be described
// ERROR CHECKING
// Error checks total: 1
// bool _handleNegatives - should evaluate to 0 or 1.
// RETURNS
// Returns bool - success/failure
// PROGRAM FUNCTION END
// PROGRAM FLOW START calling function: Foo
#ifdef _DEBUG_FOO
PRINT_FUNCTION_ENTER
PRINT_VALUE(_handleNegatives)
#endif
// Check parameter validities
// Check bool _handleNegatives validity
if ( _handleNegatives != 0 && _handleNegatives != 1 ) { handle bad value }
// code starts here
... your code
// code ends here
#ifdef _DEBUG_FOO
PRINT_FUNCTION_EXIT
#endif
return true;
// PROGRAM FLOW END
} // Foo ()
That is it, 30+ lines of code and you have not started on the logic.
Before you start filling in the logic, I recommend you start filling in the
documentation section with what you know: what the function is
supposed to do and how, what the parameters are for, insert
parameter validity checks, etc. It will help you clarify your thinking
and initiate documentation. After doing this, compile to make sure
you did not inadvertently introduce an error; then start coding the
algorithm.

4.
 Class and Function Names Referencing
Framework instrumentation (appendix)
<x>Class and Function Names ReferencingThe key to identifying
Framework[AUX]Identify classes/functions via IDs</x>

functions so that documentation should inform on who is doing the

reporting is to insert class and function IDs inside classes and
functions to identify them. Such insertions are referred to as
instrumenting code. It could be done manually on small applications
and automated on large applications by using regexes.
Classes are easy to identify, just look at the first word of a line. This
assumes that class declarations start in a newline. The regex for
finding a class is
if ($line =~ /^\s*class\s+(\w+)/ { do something with $1 }
When the regex evaluates to true, $1 contains the class's name - the
content of the first parens pair. Classes may be included from a first
pass through the files to identify them and thus build a classes table.
<x>Program Documentation Framework[AUX]Finding functions using regexes complicated</x>Finding functions is a little more

complex since function and variable declarations are almost

identical; furthermore, function declarations will contain type
qualifiers, thereby complicating matters.
Instrumenting functions requires two steps:

Identify functions by looking for function declarations in .h files

- create a table of file/function names.
Instrument function definitions in .cpp files identified via the file
function/table created from .h files.

The following rather lengthy function declaration is a model to define

the regex.
virtual char** MyClass::MyFunction ( params ) { code }

Start from the beginning with optional spaces.

Continue with a search for optional keywords (static, virtual,
friend...) followed by spaces.
Search for a type char, int...
 Search for optional type qualifiers (pointer or reference, ** in
the above).
Search for an optional scope qualifier such as myClass:: - this is
part of what we are looking for.
Search for a name followed by optional spaces. This is the
function's name we are looking for.
Search for an optional open parens, optional in that it could be
located in a consecutive line.

So far, this regex does find function definitions. The problem is that
the open parens could be in the next line. If it were in the same line,
the search ends, a function has been found. If it were not in the
same line, a flag should have been set to enable searching for an
open parens as the first item in the following non commented line.
With the flag set and an open parens found, the previous line gets
qualified as being a function definition.
The function's structure analysis above leads to the following regex:
/^\s*(static\s+|virtual\s+|friend\s+){0-3}(\w+)\s*(\**)\s*((\w+)\s*::\s*)?\s*(\w+)\s*(\(|\s*)/
This regex may be separated into its constituent parts:

Start at the beginning of the line with optional spaces: ^\s*

Handle the keywords static, etc.; try as many as 3 times as there
might be as many as three keywords: (static\s+|friend\s+|virtual\s+){0-3}
Handle the type. There is no need to differentiate one type
from another. From a Perl perspective, a type is a name,
consequently, just plug (\w+) to identify a name. It is in parens
so that it could be captured and checked against legal types
(char, int...). If it is not a legal type, discard the line since it is not
a function definition. This can get a little complex since
enumerations, structures and classes are also types.
Handle the optional zero or more pointer qualifiers (\**)?.
Handle the optional scope specifier ((\w+|\s*)\s*::\s*)?. The class's
name is provided by (\w+|\s*).
Search for the function's name followed by zero or more
spaces. The function's name is provided by (\w+).
 Finish off by searching for an open parens (\(|\s*).

Granted, it is a little long, but it does find function definitions

For this regex to work, variables should not be initialized with
parentheses. For example, suppose that a class contains int x(99), the
regex would consider x to be a function; do a curly braces
initialization such as int x{99}. Note that this regex works only if the
opening parentheses is in the first line of the function declaration,
easy to achieve.
The two regexes above demonstrate how to obtain class and
function names to instrument the files. After processing the .h files,
the Perl program contains the info required to generate the
CandFnames.h/.cpp files and instrument the .cpp code files. The
Perl data extraction and code instrumentation program which
accomplishes this is [Link]. At the time of writing,
this program is being worked on.
The Perl automated data extraction and code instrumentation
system is based on several assumptions:

The C++ source code compiles error free.

Code is reasonably formatted. By reasonably I mean that class
declarations and function definitions should start on a newline.
The function declaration's opening parenthesis is in the first
line.
There are no blocks of code commented out via /*...*/ pairs. The
Perl program will be fooled by whatever is included in these.

All application functions which may trigger events and need to have
IDs inserted in their definition, are defined in .cpp files. Functions
defined in the .h files and inlined ones are presumed to not have
much code, thereby not triggering events. These are ignored during
the instrumentation process.
At time of writing (May 2023), some features of the Perl Class and
Function Names Referencing Framework program have not yet been
completed. The following TO DO work remains.
 Refine the search for functions algorithm.
Instrument classes in .h files and functions in .cpp files.

2.
Data Handling Frameworks
(appendix)
The Data Handling
<x>Data Handling Frameworks[AUX]Frameworks to handle data storage, data conversion, data compacting</x>

Frameworks (page 158) category groups data compacting to save

space and to prepare data for transmission.

Class Data Framework (appendix) (page 254) - When a class

contains identically typed variables, accessing them via enum
driven offsets into a structure containing them clarifies
programming and renders the application more robust and
maintainable.
Data Packets Framework (appendix) (page 257) - Byte level
data transmission is based upon breaking up data into chunks
which are in turn broken up into packets.
Format Driven float to byte Conversion Framework (appendix)
(page 258) - float data items occupy four bytes each. The value
range of individual items leads to the conclusion that some of
these could be stored in as little as one byte. The conversion
process is table driven on a data item by data item basis.
The Bitfield Storage Framework (appendix) (page 261) stores
byte data into bitfields accessed via enum based Get and Set
functions.
DataGroup Framework (appendix) (page 265) - Storing
multiple fixed or variable length records sequentially or key
based is handled via generalized Get and Set functions. This
system is in effect a mini database which can store data in any
device which supports byte level access.
Event Storage Framework (appendix) (page 269) - As an
 application runs, it can generate all sorts of events: errors,
SMS handling, radio communications. Event types are stored
in contiguous byte arrays accessed seamlessly via appropriate
enums.
Linked List Framework (appendix) (page 270) - Storing records
as linked lists is easy and efficient. Two linked list classes
serve as base classes to enable any class to store values
inside linked lists.

1.
Class Data Framework (appendix)
<x>Data Handling Frameworks[AUX]Class Data Framework accesses class data via enums</x><x>Class Data Framework[AUX]Access data via Get/Set enum driven

Organizing and accessing identically typed variable length

offsets</x>

related class variables makes for low error prone programming and
maintenance ease. The mechanics to do this is referred to as the
Class Data Framework (page 159). The specific variables access
functions Get/Set are replaced by two generalized Get/Set functions
which take an enum list-based index to identify which variable to work
on.
Organizing identically typed related variables requires that their
individual sizes be known. If sizes are identical, items may be stored
in an array and accessed via an enum-based index, however, if data
sizes differ, an array of variable sizes (employeeItemSizes below) driven by
an items enum-based list, may be used to calculate the offset into a
struct which contains the items. To access a specific item, add up the
sizes of the individual items but do not include the size of the desired
item.
The table of variable's sizes uses data size #defines for each item, as
follows:
static const uint8_t employeeItemSizes[] = {
EMP_SIZE_CIVILITY,
EMP_SIZE_NAME,
EMP_SIZE_ADDRESS,
...
}; // employeeItemSizes
This array is synchronized via the EEmployee enum list:
enum class EEmployee: uint8_t {
start = 0,
civility = 0, name, address, ... ,
end
};
An item's offset is the cumulative sum of sizes up to but not including
the size of the item whose offset is being calculated. To get a specific
item's offset, use the function ItemOffset below. It verifies that the item's
index being searched for is valid. It then returns the offset of the
desired item by summing offsets of preceding items or -1 if an error
was found, as follows:
int16_t Employee::ItemOffset (EEmployee _item, EEmployee _end )) {
// enum based index to get offset
// returns offset of _item in array; if error, returns -1
if (_item < 0 || _item >= EEmployee::end) {
// Handle error - index out of range
return -1;
}
// index OK - search for some offset
int16_t offset{0};
for ( int i = 0; i < _item; i++ ) {
offset += employeeItemSizes [i];
}
return offset;
} // ItemOffset ()
The class Employee now becomes:
class Employee {
public:
Employe( ... ) { ... } // constructor
bool Set(EEmployee _eItem, char* _item); // Code implemented below
char* Get(EEmployee _eItem); // Code implemented below
int16_t ItemOffset ( EEmployee _eItem, EEmployee::end );
private:
struct {
char civility[EMP_SIZE_CIVILITY]; // group items of the same type
char name[EMP_SIZE_NAME];
char address[EMP_SIZE_ADDRESS];
...
} charData; // struct
public;
static const uint8_t* employeeItemSizes;
...
}; // Employee
The generalized Get function becomes:
 char* Employee::Get(EEmployee _eItem) {
int16_t itemOffset {0};
itemOffset = ItemOffset (_eItem) )
if ( itemOffset == -1) {
// handle error
return 0;
} // if
return (char*) ((uint16_t) &charData + itemOffset)
}; // Get
The generalized Set function becomes:
bool Employee::Set(EEmployee _eItem, char* _data) {
int16_t itemOffset {0};
itemOffset = ItemOffset (_eItem );
if ( itemOffset == -1) {
// handle error - item ID problem
return false;
} // if
if ( strlen(_data) > employeeItemSizes[_eItem] -1 ) { // -1 to account for null byte
// handle error - data too large
return false;
} // if
strcpy((char*) ((uint16_t) &charData + itemOffset), _data);
return true;
} // Set
If you decide to add more items, or shift things around, all that is
needed is to modify the enum, the #define lengths sequence, and the
item lengths array.
The above may be taken one step further. Suppose that you want
labels for each of the variables. You can do this by replacing the item
lengths array employeeItemSizes with a struct array which includes both
labels and item lengths, such as:
struct EmployeeItemsSupportData { char* label, length }; // struct Employee
EmployeeItemsSupportData employeeItemsSupportData[] = {
{ "Civility", EMP_SIZE_CIVILITY },
{ "Name", EMP_SIZE_NAME },
{ "Address", EMP_SIZE_ADDRESS },
...
}; // employeeItemsSupportData
so that, with appropriate formatting, you could use the generalized
Get function and the employeeItemsSupportData array info to display data.
Given that item lengths are now inside a structure, the ItemOffset
function needs to be modified, as follows:
int16_t Employee::ItemOffset (EEmployee _item ) ) { // enum based index to get offset
// returns offset of _itemIndex in array; if error, returns -1
 if (_item < 0 || _item >= EEmployee::end) {
// Handle error - index out of range
return -1;
}
// index OK - search for some offset
int16_t offset{0};
for ( int i = 0; i < _item; i++ ) {
offset += employeeItemsSupportData[i].length;
}
return offset;
} // ItemOffset ()
Having coordinated variables with labels via the enum list, obtaining
data from the class becomes trivial, as follows:
[Link](Employee::empItemsSupportData[EEmployee::name]. label); [Link](F(": "));
[Link]( empSashaShrivek->Get(EEmployee::name); // supposes empSashaShriveck has
been created
The outcome is: name: Sasha Shrivek
The above shows how to generalize access to private data in
classes which contain variable length identically typed variables such
as multiple char* arrays. The mechanics in the Employee class are hard
coded inside the class. If the application were to contain several
similar classes, the mechanics could be placed inside a base class.
Derived classes would handle the specifics of the class. The base
class could be named CharDataAccess, derived classes could be
imagined, such as:
class Employee: CharDataAccess { ... }
class Staff: CharDataAccess { ... }
class Manager: CharDataAccess { ... }
class Consultant: CharDataAccess { ... }
I leave it up to you to create such a base class.

2.
Data Packets Framework (appendix)
<x>Data Handling Frameworks[AUX]Data Packets Framework breaks data into chunks for serial transmission</x><x>Data Packets Framework[AUX]Break up data into

The mechanics of the Data

chunks/packets</x><x>Data Packets Framework[AUX]Send/receive packets via serial ports</x>

Packets Framework (page 161) is encapsulated in the class DataPackets.

It handles radio transmission by breaking up a chunk of user data
(169 bytes max size) into manageably sized packets, sent one at a
time.
class DataPackets {
// handles basic send/receive data
public:
DataPackets(uint8_t _dataSize, char* _dataBuffer);
bool PacketsSend();
bool PacketsGet();
uint8_t GetDataBufferArraySize() {return dataSize;}
uint8_t GetPacketSize() {return packetSize;}
private:
bool GetByte(uint8_t _byteToGet, long _timeLimit);
uint8_t Calculatehecksum();
bool VerifyChecksum();
bool VerifyParametersCoherence();
uint8_t packetSize{0}; // size of hard coded at 48
char* dataBuffer{(char*)0}; // number of packets to send/receive
uint8_t dataSize{0}; // total bytes in data to be sent/received, from constructor
}; // class DataPackets
The class's basic characteristics follow:

The DataPackets(uint8_t _dataSize, char* _dataBuffer) constructor receives

the data size to send and a pointer to the data to be sent. It
breaks the data into packets as needed: up to a maximum of
169 bytes.
The PacketSend() and PacketGet() functions send/receive packets, one
at a time.
A VerifyCoherence() function checks that the chunk parameters in
the first packet are coherent with the chunk's characteristics.
This function serves mainly to detect programming errors.
CalculateChecksum() and VerifyChecksum() handle checksums. These
ensure that the data has not been corrupted.

3.
Format Driven float to byte Conversion
Framework (appendix)
<x>Data Handling Frameworks[AUX]Format Driven float to byte Conversion Framework reduces data size</x><x>Format Driven float to byte Conversion Framework (FDFBC)

The Format Driven

[AUX]Converts float values to packed bytes and vice versa, packing controlled by parameters, FDFBC class does all the work</x>
float to byte Conversion Framework (page 162) converts a group of
sensor data expressed as an array of float values into a group of
packed bytes and vice versa, its objective being to seamlessly
compact sensor data for radio transmission via a serial port. This job
is accomplished by the class FDFBC (Format driven float byte
conversion). It converts a floats array into an array of bytes. It also
does the reverse at the receiving end by converting a byte array into
a corresponding floats array.
Each sensor's conversion is controlled by parameters to optimize
packing. Sensor values, expressed as floats, occupy four bytes of
memory. Upon analyzing each sensor's possible range of values, the
conversion's outcome may be as low as one byte. My Arduino-based
beehive weighing system monitors eight sensors which provide float
values, i.e., 32 bytes. As is shown further down, the data for the
eight Arduino-based beehive weighing system sensors can be
reduced to 9 bytes, a significant 23 bytes reduction in data to
transmit.
<x>Format Driven float to byte Conversion Framework (FDFBC)[AUX]Converts float values to packed bytes and vice versa, packing controlled by parameters, FDFBC class does all

The class FDFBC, which does all the work, is characterized by the
the work</x>

following:

The FDFBC constructor receives pointers to three externally

defined arrays: a control parameters array sensorDataParams, a
sensor floats array sensorFloatData, and a sensor byte array
sensorByteData.
Public top-level functions undertake the conversion between
the raw float sensor values and the sensor byte values.
Private auxiliary functions are used by the top-level functions
to do the work.
Private pointers to arrays contain the control parameters and
the data to work with.
class FDFBC{
public:
FDFBC( ... ); // Constructor
// Main top level functions
bool TransferFloatToInt8array(... );
bool TransferInt8arrayToFloat(... );
 private: // private functions
// Auxiliary functions called by main functions above
uint8_t CalculateBytesRequiredFloat( ... );
uint8_t GetNumberOfDecimals( ... );
bool GetNegativeOK( ... );
uint8_t GetControlVariablesArraySize( ... );
SensorDataParams GetAllControlParameters( ... );
uint8_t GetOffsetSensorInDataPacket( ... );
uint8_t GetGlobalArraySize( ... );
private: // private data
SensorDataParams* sensorDataParams; // control parameters
float* sensorFloatData; // Holds sensor values
uint8_t* sensorByteData; // Holds converted packed sensor data
}; // class FDFBC
The data conversion algorithm starts off with a global enum
which define the sensors:
enum class EHiveSensors: uint8_t { // beehive data holds sensor values (floats)
start = 0,
volts5 = 0, volts8, hiveTemp, hiveHumid, ambientTemp, body, super,
end
}; // EHiveSensors
The mechanics of float to byte DataPacket conversion is based on
parameters which characterize each sensor's data, namely:
bytes needed, the number of decimals, data reduction by
division, its range of possible values. The structure
SensorDataParams contains all sensor specific parameters used by
the FDFBC class to do the conversion.
struct {
uint8_t bytes: 2; // NumberBytes (values: 0 - 3), apply offset of 1
uint8_t decimals: 2; // NumberDecimals (values: 0 - 3)
uint8_t divider: 2;// DivideInstead (values: 0-3) - in tens
float min{0.0}; // Minimum possible value
float max{0.0}; // maximum possible value
} sensorDataParams [ number of sensors];
Number of bytes required to send the data, from 0-3 bytes, an
offset of 1 is applied to set the number of bytes.
Number of decimals could be 0 or more. Battery voltages
require 1 decimal; temperature - 0 decimals, humidity - 0
decimals, weight voltages require two decimals.
Divider - Some sensor values, as obtained, contain more
precision than required. Humidity, for example, can be reported
 in 10s of percent thus, 56% should be converted to 60% and
divided by 10. The value to be transmitted then becomes 6 and
not 56, thereby reducing the number of bytes required.
Furthermore, since the range is 30% to 100%, a humidity value
can be further reduced by taking 30 as a base, thereby limiting
values to transmit increments from 0 to 7 which requires only
one byte (4 bits if you want to push it).
Min-Max values define a range. For example, regulated
voltage measurements should lie between 4.7 and 5.0. Given
a 4.7 minimum, increment transmission from 0 to 3 (1 byte) is
sufficient. The same applies to the battery voltage, values will
range from 6.1 to 8.1; given a minimum of 6.1, transmitted
increments will range from 0 to 20 (1 byte). In both cases,
subtract the minimum value from the data value and multiply
by 10 to get the increment. Increments get transmitted instead
of full values.
Sensor parameters are stored in a structure array, one set per
sensor.
extern const SensorDataParams sensorDataParams[]; // holds control parameters
A floats array contains sensor data. The converted data is in a
uint8_t array:
extern float sensorFloatData[]; // Holds sensor values
extern uint8_t sensorByteData[]; // Holds converted packed sensor data

You
<x>Format Driven float to byte Conversion Framework (FDFBC)[AUX]Beehive weighing system sensor values, FDFBC reduces data transfer from 24 to 9 bytes</x>

will find below sensors of the Arduino-based beehive weighing

system examined one by one to specify control parameter values.

Voltage - Voltages can vary from ٤.٧ to ٥.٠ for regulated

voltage, and ٦.١ to ٨.١ for battery voltage. In both cases, a ٠.١-
volt resolution is satisfactory, one byte is adequate. Given a ٤.٧
volts minimum, data transfer may be limited to transferring the
increment above the minimum multiplied by ١٠, as opposed to
transmitting the entire voltage value, thus requiring one byte
only to send increments from ٠ to ٣. As for the battery voltage,
increments range from ٠ to ٢٠ given a ٦.١ minimum voltage,
 thus needing one byte each.
The beehive temperature sensor provides a three significant
figure precision temperature. Two significant figures is good
enough for managing the beehive. 28.7°C would be sent as 29
given a 15°C to 45°C beehive temperature range. The
transmitted values will range from 0 to 30 after applying an
offset of -15 - one byte suffices.
The ambient temperature sensor provides temperatures
from -20°C to 50°C. This represents a range of 0 to 70 - one
byte suffices.
Humidity - If the raw sensor data yields ٦٠ ,٪٥٦٪ is good
enough. The algorithm would need to round off the raw value
to the nearest ١٠٪ then divide by ١٠ - humidity values to be
transmitted would range from ٠ to ١٠. Furthermore, if humidity
ranges from ٣٠٪ to ١٠٠٪, sending a ٦٠٪ humidity can be
further reduced to sending a value of ٣ = ١٠/(٣٠-٦٠) ٣). At the
receiving end, multiply by ١٠ and add ٣٠ to obtain ٦٠٪. One
byte suffices for both the ambient humidity and hives humidity.
Weights can vary from 6.0 to 50.0 kilos. The question is: What
should the weight resolution be? 100 grams, 200 grams,
1kilogram? Since the beekeeper may want to receive weight
data throughout the day, he/she would need a 100-gram
resolution. Two bytes would be required for weights ranging
from 0.0 to 5.00 volts (transfer 0 to 500). If 5.0 volts
corresponds to 50.0 kgs, 0.01 volts is equivalent to 100 grams.
Thus a 3.62 volts sensor value gets converted and sent as
3.62*100 = 362, thereby requiring two bytes.

The item-by-item sensor value characteristics itemized above reveal

that 9 bytes suffice to send the eight sensor data values, as opposed
to 32 bytes which would correspond to a packet size based on float
values. The data size could be further reduced by two bytes if some
variables are packed into bitfields.
This sensor data analysis defines the following control variables
array:
 const SensorDataParams sensorDataParams [] {
// bytes, decimals, divide, minimum, maximum
{ 1, 1, 0, 4.7, 5.0}, // Regulated voltage 5V
{ 1, 1, 0, 6.1, 8.2}, // Battery voltage 8V
{ 1, 1, 0, 15.0, 45.0}, // Beehive temperature
{ 1, 1, 0, -20.0, 50.0}, // Ambient temperature
{ 1, 1, 1, 30.0, 100.0}, // Hive humidity - divide by 10, 10% increments sufficient
{ 1, 1, 1, 30.0, 100.0}, // Ambient humidity - divide by 10, 10% increments sufficient
{ 2, 1, 0, 0.00, 5.00}, // Weight body
{ 2, 1, 0, 0.00, 5.00} // Weight super
}; // sensorDataParams
The functions of the class FDFBC transform sensor float values into
the appropriate byte values for data transfer, summarized as:

Hard code a SensorDataParams parameter table based on sensor

characteristics.
Obtain raw sensor data for the specified sensors.
Convert sensor data into byte data via the Format Driven float
to byte Conversion Framework.
Send the data.
On the receiving end, do the reverse.

4.
Bitfield Storage Framework (appendix)
<x>Data Handling Frameworks[AUX]Bitfield Storage Framework stores data inside bitfields</x><x>Bitfield Storage Framework[AUX]Optimize data storage via bitfield packed

The underlying concept of the Bitfield Storage Framework

variables</x>

(page 166) relies on storing small-valued integer variables inside a

few bits, instead of storing them inside entire bytes. The most
notorious example of wasted space would be to store a true/false
condition inside one byte instead of using one bit. Fortunately, C++
proposes the bitfield operator to save space. When defining a
sequence of variables inside a structure or a class, you could do:
bool status: 1 // true or false
uint8_t jobType: 3 // any value from 0 to 7
...
The variables status and jobType get attributed 1 bit and 3 bits of storage
respectively. Note that there are four bits left to store other data in
the same byte. This mechanism enables classes with many
variables to save a lot of space.
Once bitfield storage has been decided upon, it is possible (via
adequate memory mapping and bitwise manipulations) to
seamlessly access data via the use of Get/Set enum-based functions
instead of using dedicated variable specific Get/Set function pairs such
as:
bool success = GetStatus(status); // status passed by reference
SetJobType(jobType); // jobType passed by value
The above may be replaced with generic Get/Set function pair which
rely on variable IDs provided by an enum list
bool success = Get( EJob::status, status); // status passed by reference
Set(EJob::type, jobType ); // jobType passed by value
In the above, EJob::status and EJob::type are enums which identify the
variable to be dealt with.
<x>Bitfield Storage Framework[AUX]Variables accesses via enum driven generalized The above replaces
Get/Set functions</x>

variable specific Get/Set functions with one generic Get function and one
generic Set function. Using enums also enables the entire set of
variables to be traversed via a loop. These possibilities enable
creating algorithms otherwise not feasible.
<x>Bitfield Storage Framework[AUX]Variables accesses via enum driven generalized Get/Set functions</x>The enum list identifies

variables:
enum EJob { start = 0, status = 0, jobType, ..., end } // Job class specific enum list
<x>Bitfield Storage The Bitfield Storage
Framework[AUX]BitfieldStorage class provides bitfield storage via inheritance</x>

Framework contains the base class BitfieldStorage to store and retrieve the
Job class's variables mapped inside a sequence of bitfield packed
variables. Each variable is defined as a bitfield inside a sequence of
bytes.
class BitfieldStorage {
public:
BitfieldStorage (
uint8_t_varsCount, // number of variables
int8_t* _varsArray, // pointer to the beginning of bitfield data
varCharacteristics* _varCharacteristics // struct contains data
);
bool Get(uint8_t _itemNumber, uint8_t& _valueToGet);
bool Set (uint8_t _itemNumber, uint8_t _valueToSet);
void PrintVariables();
void PrintMasks();
bool VerifyMasks();
 private: // Suppport functions not called from the outside
bool CreateMask( uint8_t _varID, uint8_t & _mask);
bool VarsArrayCoherence(uint8_t& _faultyIndex);
bool GetOffsetInMap(uint8_t _varID, uint8_t &_offsetInMap);
bool GeByteID(uint8_t _varID, uint8_t &_byteID);
bool GetOffsetInByte(uint8_t _varID, uint8_t &_offsetInByte);
bool GetItemBitSize( uint8_t _varID, uint8_t &_itemBitSize);
private: // data
uint8_t* varsArray{0}; // maps values of bitfields
uint8_t varsCount{0}; // number of items including dummy filler values
VarCharacteristics* varCharacteristics{0}; // pointer to array of var characteristics
} // BitfieldStorage
The BitfieldStorage class requires three support items: a pointer
to the Job
class's data, the number of variables, and a means to get individual
variables' bit sizes.

uint8_t* _varsArray is the address of the first variable in the list of

bitfield variables. This is the address of a structure which
encloses the variables to map the derived class's bitfield data
into an indexed byte array.
uint8_t _varsCount - The number of variables required to do a
coherence check against the enum based variable sizes used to
access the var array items.
VarCharacteristics varCharacteristics[varsCount] - This array contains the
variables' bit sizes, min and max values a variable may attain,
and labels for each variable.

The variables of the derived class should be enclosed in a struct

whose address maps to a uint8_t array.
uint8_t* varsMapp = &varsStruct; // Job class structure which contains bitfields
Please note that the structure's address yields the address of the
beginning of the bitfield variables, irrespective of what lies before or
after the structure's position in the derived class.
The BitfieldStorage constructor undertakes the following tasks:

It initializes the three derived class BitfieldStorage support items

above.
It runs VarsArrayCoherence to verify that bitfields fill 8-bit blocks and
 do not straddle byte boundaries. If there is a problem, it
triggers an error.
The BitfieldStorage constructor runs VerifyMasks to ascertain that
masks are correct. Masks are used to selectively extract data
from a byte by appropriate bitwise operations and bit shifts. It
does not require much code, but it is a little complex to set up.
Once masks have been verified to work well, the VerifyMasks
function could be #ifdefed out.

The derived Job class below and the base BitfieldStorage class above
(which handles access to the Job class's data) are used to illustrate
the workings of the Bitfield Storage Framework.
When creating a class, start by defining bit lengths of individual
variables as #defines organized so that byte boundaries do not get
straddled. If necessary, add dummy variables. You may want to do
this in a spread sheet. See Bitfield Storage Framework (page 166).
#define SIZE_JOB_status 1
#define SIZE_JOB_jobType 3
...
Having defined
<x>Bitfield Storage Framework[AUX]Bitfield storage example, 3 arrays: JobData, VarsMinMax, JobLabels data arrays</x>

individual variables and their corresponding bit sizes, the derived class
Job may now be defined:
class Job: BitfieldStorage {
public:
Job(
uint8_t _jobType, uint8_t _repeatInterval, uint8_t _day, uint8_t _weekday,
uint8_t _scheduleType, uint8_t _hour, uint8_t _fifteenMin, bool _repeatType,
uint8_t _repeatNumber, uint8_t _varsCount )
: BitfieldStorage(_varsCount, (uint8_t*) &jobData, varCharacteristics) {
// I could not get inline initialization of structure variables to work.
[Link] = _jobType; [Link] = _repeatInterval;
[Link] = _day; [Link] = _weekday; [Link] = _scheduleType;
[Link] = _hour; [Link] = _fifteenMin; [Link] = _repeatType;
[Link] = _repeatNumber;
varsCount = _varsCount;
} // Job constructor
void PrintValues();
struct JobData {
bool status: SIZE_JOB_jobStatus; // 1 bit
uint8_t jobType: SIZE_JOB_jobType; // 3 bits
uint8_t repeatInterval: SIZE_JOB_repeatInterval; // 4 bits
 // byte boundary
uint8_t day: SIZE_JOB_day; // 5 bits
uint8_t weekday: SIZE_JOB_weekday; // 3 bits
// byte boundary
uint8_t scheduleType: SIZE_JOB_scheduleType; // 3 bits
uint8_t hour: SIZE_JOB_hour; // 5 bits
// byte boundary
uint8_t fifteenMin: SIZE_JOB_fifteenMin; // 2 bits
bool repeatType: SIZE_JOB_repeatType; // 1 bit
uint8_t repeatNumber: SIZE_JOB_repeatNumber; // 4 bits
} jobData; // JobData
uint8_t varsCount{0};
static const VarCharacteristics varCharacteristics[];
}; // class Job
The derived class Job, having encapsulated the variables inside a struct
with appropriately dimensioned bitfields, transparently uses the
BitfieldStorage class's Get and Set functions. An enum identifies which
variable is to be worked on.
Get( EJob::status, status);
Set( EJob::type, jobType);
These two functions are similar in that they take two parameters, one
to identify the variable taken from the enum list of variables, the other
being the variable's reference to Get its value, or the variable's value
to Set its value. They both return true or false to specify whether or not all
went well.
<x>Bitfield Storage Framework[AUX]BitfieldStorage class requires info on variables stored in VarCharacteristics; uses bit-masks to operate on bytes</x>Getting

and setting data from/into variables require that the variables' bitfield
sizes be defined. This info is contained in the structure VarCharacteristics:
struct VarCharacteristics {
uint8_t bitSize; // bitfield size
uint8_t min, max; // min and max values variable may attain for validity checks
char* label; // name of variable
}; // VarCharacteristics
An array of VarCharacteristics contains info on all the variables, as follows:
static const VarCharacteristics Job::varCharacteristics[] = {
// uint8_t size; uint8_t min, max; char* label;
{SIZE_JOB_jobStatus, 0, 1,"status" },
{SIZE_JOB_jobType, 0, 4, "jobType" },
...
}; // varCharacteristics
The algorithmic details governing setting and getting data from the
derived class variables follow. Three items are needed:
 Total offset of a variable within the byte array. This is obtained
by adding the bit lengths of all preceding variables.
Specific byte in which the variable is located. This is obtained
by dividing the total offset by 8-bits per byte.
Offset of the variable within the byte. This is obtained by
summing the sizes of the target variable's preceding variables.

Suppose you are looking for the variable hour (the Job variables table is
described in the section Bitfield Storage Framework page 166). It is
located at total offset 19, i.e., in the 3rd byte, at offset 3 within the
byte.
Behind the scenes, this is what happens:

Get the variables offset in the overall byte array -

GetOffsetInMap(EJob::hour, totalOffset).
Identify which byte the variable is located in - GetByteID(EJob::hour,
byteID).
Get the variable's offset in the byte - GetOffsetInByte(EJob::hour,
offsetInByte).
When doing a Get, extract the variables data into a work byte
via a bitwise AND with the appropriate mask (CreateMask), then do
a shift on the work byte towards the right (towards 0).
When doing a Set, the function sets the desired value in a work
byte followed by an appropriate left shift (away from 0). This is
followed by setting the variable's bits to 0 with the appropriate
xored mask (CreateMask) applied on the mapped array's target
variable. The work byte is then added to the mapped byte.

Bit manipulation maps are created on the fly via the CreateMask function
which uses an EJob enum to identify the variable being worked on. It
returns false if an error occurs. The masks for the first three
variables of the Job class are:
statusMask = 1: // first variable 1 bit
jobTypeMask = 1110; // second variable 3 bits; offset 1
repeatIntervalMask = 11110000; // third variable 4 bits, offset 4 ( 1 + 3 )
The first three variables' masks contain 1s according to their bitfield
size and position in the sequence. The result is verified by adding
the 3 masks together.
totalMaskSequence = 11111111
Notice how the 1s in the individual masks do not overlap. By adding
them together, the outcome is a byte filled with ones. A total mask
sequence not equal to 255 indicates a programming error.
Here is a sample walkthrough to Set the hour variable to 14. Remember,
the hour variable is inside the 3rd byte at offset 3. Start by locating the
hour bitfield within the byte array, identify the byte, and finally find its
offset within the byte:
CreateMask( EJob::hour, mask);
GetTotalOffset( EJob::hour, totalOffset);
GetbyteID( EJob::hour, byteID);
GetOffsetInByte( EJob::hour, offsetInByte);
GetSize( EJob::hour, size);
Having gotten the basic data required to accomplish the bitwise
manipulation, define a work byte and set it to the desired value:
uint8_t tempByte = 14;
Shift it left offsetInByte to place it where it belongs:
tempByte <<= offsetInByte;
Set target hour bits to zero and then add tempByte to target byte:
mappedBytes[regIndex] &= ~maskHour;
mappedBytes[regIndex] += tempByte;
Voilà. Q.E.D. The above is certainly a workload and somewhat
complex but, once implemented, using it is simple.

5.
DataGroup Framework (appendix)
<x>Data Handling Frameworks[AUX]DataGroup Framework supports index/sequential fixed/variable length data</x><x>DataGroup Framework[AUX]Index/sequential

fixed/variable The DataGroup Framework (page 164) is a mini-

length data</x>

database management system. Being support independent, it can

manage data in the microcontroller's or external EEPROM, in an SD
card, somewhere in the Internet, in some cloud, in RAM, or
somewhere else. Minor changes will be required in the low level
Read/Write functions to adapt access to records according to the
specifics of the memory system being used (RAM, EEPROM,
cloud...). Which memory support to use depends on the quantity of
data to be managed. Its only requirement is that it be possible to
reserve a contiguous data space with direct byte level access from a
base offset.
<x>DataGroup Framework[AUX]DataGroup class does the work: header defines characteristics, body contains data</x>The mechanics of this

framework are encapsulated inside the DataGroup class and support

functions. Reads and writes from/into memory are undertaken N
number of bytes at a time. An array of uint8_t is populated with data
and then written into memory. The reverse holds when getting data
from memory. Since storing and retrieving data is based on a
contiguous memory zone, data needs to be stored inside a structure
which maps the bytes into individual components. For example, the
DataGroupHeader contains 8 different items as detailed in the struct
DataGroupHeader:
struct DataGroupHeader {
char name[DATA_GROUP_NAME_LENGTH]; // default 10 bytes
char desc[DATA_GROUP_DESCRIPTION_LENGTH]; // default 50 bytes
uint16_t offset{0}; Offset of DataGroup in memory
uint16_t datagroupSize{0};
uint16_t recordLength{0};
bool fixedType{true}: 1; // fixed (defaults) or variable
bool hasKeys{false}: 1; // If true, record has keys, default is false
uint8_t keyLength{0}: 3; // key limited to max size of 7 bytes
: 3; // bitfield buffer to bring total bitfields to size = 8
} storageGroupHeader ; // DataGroupHeader
For maintainability reasons, use sizeof(DataGroupHeader) to get the struct's
size. Then write or read the corresponding bytes in EEPROM or
other using the address of the structure storageGroupHeader defined
above.
The class DataGroup contains functions which the application can
use to manage data and to defragment the data when needed.
class DataGroup {
public:
// This constructor creates a new DataGroup .
DataGroup(char* _name, char* _desc, uint16_t _packageSize, ERecordType _type,
uint8_t _keyLength = 0, uint16_t recordLength = 0);
// Writes in first available slot fixed size, writes at end of records variable size
// If key based fixed length record, overwrites it if it exists
void WriteRecord(uint8_t* _data); // Write for fixed size records
void WriteRecord(uint16_t _recordLength, uint8_t * _data); // Write variable size
 // Key based get, returns offset
bool GetRecordKey(uint8_t keySize, uint8_t& _key, uint16_t* _data);
bool GetRecordOffset(uint16_t& _offset, uint16_t* _data); // Offset based get
bool GetRecordIndex(uint16_t& _index, uint16_t* _data); // Index based get
bool RemoveRecord(uint16_t _offset);
uint16_t GetRecordCount(); // returns number of records in DataGroup
uint16_t GetLastRecord(); // returns offset of last record
private:
void Defragment(); // Removes holes from variable length records.
DataGroupHeader storageGroupHeader;
}; // class DataGroup
The DataGroup constructor creates a new DataGroup. Functions
manipulate the data. These operate on one of two kinds of records:
fixed length records or variable length records. Both types may
contain a unique key to identify the record when searching data.
Fixed length records get inserted into the first available slot or after
the last existing record. Variable length records get inserted in a hole
if big enough or appended after the last record. After numerous
writes and deletes, the memory could be littered with empty space.
When this empty space reaches a critical size, it is time to
defragment the DataGroup - traverse the entire record list and
consolidate it to eliminate the empty spaces. Defragmentation is
undertaken if empty space is less than ١٠٪ of the group's allocated
space. Whatever storage method is used, there is one last record
whose content is #END# to mark the end of stored data.
The class DataGroup provides ٨ public functions to manage data.

There are two WriteRecord

<x>DataGroup Framework[AUX]GetRecord/WriteRecord/RemoveRecord get/set/remove data</x>

functions. One is for fixed size records; it inserts a record in the

first available slot or at the end, after existing records. The
other one is for variable size records; it appends a record after
the last record. If the record is key based fixed size and exists,
it gets modified instead of being inserted as a new record. If
the record is key based variable size and exists, it gets
removed from wherever it is located and inserted in a hole if
big enough or appended as the last record. If a problem is
encountered, the functions return false.
bool WriteRecord(uint8_t* _data); // Fixed size
bool WriteRecord(uint16_t _recordLength, uint8_t * _data); // Variable size
 There are three GetRecord functions. They differ from each other
according to the access method: key based, offset based, and
index based. Each one returns a bool value so that the
application could determine that the record has been found.
The key based GetRecordKey iterates through the records until it
finds a key-based record. The offset based GetRecordOffset
accesses a record directly because it knows its offset in the
DataGroup's body. This function can be used for both fixed and
variable length data. The index based GetRecordIndex gets the Nth
record, whether it be fixed or variable length records. For fixed
length records, it calculates the offset as N-1 times the size of
the record. For variable length records, it calculates the offset
by iterating progressively through the records N-1 times using
the record's size to step forward one record at a time.
// Key based get, returns offset
bool GetRecordKey(uint8_t& keySize, uint8_t* _key, uint16_t* _data);
bool GetRecordOffset(uint16_t& _offset, uint16_t* _data); // Offset based get
bool GetRecordIndex(uint16_t& _index, uint16_t* _data); // Index based get

The GetRecord functions above return false if an error occurred. If a

record is not found, _data is set to zero, no data found.

There is one remove

<x>DataGroup Framework[AUX]GetRecord/WriteRecord/RemoveRecord get/set/remove data</x>

record function RemoveRecord. It has a parameter to return the fact

that the record did indeed exist and that it has been removed.
The corresponding slot is tagged with "#EMPTY#".
bool RemoveRecord(uint16_t _offset);

This function returns false if a problem occurs, like attempting to

remove a nonexistent record.

There is a GetRecordCount function which traverses the records to

return the total number of records, excluding the records
tagged as "#EMPTY#".
uint16_t GetRecordCount(); // returns number of records in DataGroup
The GetLastRecord function gets the offset of the last record. This
 defines the total space used by data. It could signal a possible
resource problem if the space allocated to the DataGroup were to
be insufficient, given the needs of the application.
uint16_t GetLastRecord(); // returns offset of last record
There is one Defragment function; it cleans up memory space if
free space after the last record is less than 10% of the
allocated space.
uint16_t Defragment(); // Defragment if free space < 10% of allocated space

This function returns the size of free space including holes.

Currently, defragmentation has not been completed. This is part of
work to be done.
The above summarizes the contents of the class DataGroup.
<x>Beehive weighing system[AUX]Example of DataGroup Framework use</x><x>DataGroup Framework[AUX]Beehive weighing system good example</x>What

follows illustrates this framework within the context of my Arduino-

based beehive weighing system. It is a star shaped coordinator-
beehives radio network. The coordinator and beehives send and
receive data to/from each other. The coordinator stores a list of jobs
(Jobs) and a list of beehives (Beehives) it is connected to. Jobs is a
variable length DataGroup. Beehives is a fixed length one. Both DataGroups
are managed by their corresponding DataGroup instantiations.
The Jobs DataGroup stores variable length jobs sequentially one at a
time. Since there are few jobs, a maximum of 10, access is
sequential, jobs are keyless.
struct StructJobsData {
// Sequential access only, no keys, variable length
uint16_t lengthData{0};
char* data; // Application needs to allocate space
}; // StructJobsData
The Beehives DataGroup lists the beehives (fixed length) which are
attached to the coordinator. The number of beehives is currently a
maximum of 16. It stores the radio module's serial number which
serves as a key, and the beehive's user defined name.
struct StructHivesData {
// Key based access
uint8_t hiveID[STRUCT_BEEHIVES_DATA_KEY_LENGTH];
char moduleName [STRUCT_BEEHIVES_DATA_MODULE_NAME_LENGTH];
}; // StructHivesData
Two DataGroup constructors create and manipulate the DataGroup, one
constructor per DataGroup.
DataGroup storageJobs("Jobs", "Stores jobs data", 100, ERecordType::variable);
DataGroup storageHives("Beehives", "Stores beehive data", 300, ERecordType::fixed,
DATA_GROUP_BEEHIVES_RECORD_LENGTH);
Having created DataGroups with their corresponding parameters
(structures), managing these via the functions of the DataGroup class
becomes trivial.

6.
Event Storage Framework (appendix)
<x>Data Handling Frameworks[AUX]Event Storage Framework stores evens in contiguous byte arrays</x><x>Event Storage Framework[AUX]Predefined true/false conditions

Events are predefined true/false conditions, one bit

stored in 1 bit per event byte array, saves RAM</x>

per event (Event Storage Framework page 171). They are mapped
as a sequence of 7- or 8-bit bytes via enum lists. This framework
reduces memory requirements 7- or 8-fold as compared with a
standard one byte per event storage scheme.
<x>Event Storage Framework[AUX]256 events (8-bits), 244 event (7-bits)</x><x>Event Storage Framework[AUX]enum driven Get/Set functions get/set events packed in

The class EventStorage contains Get and Set functions that access
bytes</x>

events via their IDs as defined in an enum list.

enum class EEventID: uint8_t { start = 0, outOfBounds = 0, badAlloc..., end };
class EventStorage {
public:
EventStorage (uint8_t* _eventsArray, uint8_t _registerSize,
uint8_t _varsCount, char** eventLabels);
bool Set(uint8_t _eventID, uint8_t _eventToSet);
bool Get(uint8_t _eventID, uint8_t& _eventToGet);
void PrintVariables();
void PrintRawData();
void PrintMasks();
bool VerifyMasks();
private: // private functions
uint8_t GetOffsetInMap(uint8_t _eventID);
uint8_t GetSequenceIndex(uint8_t _eventID);
uint8_t GetOffsetInSequence(uint8_t _eventID);
uint8_t CreateMask( uint8_t _eventID, uint8_t& _mask);
 private: // private data
const uint8_t registerSize{8};
const uint8_t* eventsArray{(uint8_t*) 0 }; // maps values of bitfields
const uint8_t eventCount{0}; // contains number of items
const char** eventLabels;
}; // EventStorage
The class needs the following:

uint8_t RegisterSizespecifies whether to use 7- or 8-bit blocks. 8-bits

is the default, 7 bits if event data is to be radio transmitted.
int8_t* EventsArray is a sequence of bytes which contains events
mapped as one bit each per event into 7- or 8-bit blocks.
These are initialized to 0 by default. The array of bytes required
to store the events will be instantiated as follows:

eventsArray = new uint8_t ( eventCount/registerSize + eventCount%registerSize? 1: 0 );

For a set of 48 events, six 8-bit bytes are required or seven 7-bit
bytes.

uint8_t EventCount contains the number of events (EEvents::end).

char** EventLabels is a string array of event labels that are used for
reporting purposes.

The constructor initializes the events to 0. This means that all events
are set to false, the event did not get triggered. It is the default
behavior. An event set to true is an exception; the event got triggered.
<x>Event Storage Framework[AUX]Masks used to manage individual events</x>The Get/Set functions use masks to

selectively extract data from a byte by appropriate AND/OR bitwise

manipulations and bit shifts. Masks are created on an as needed
basis by CreateMask. VerifyMasks checks that CreateMask works correctly by
creating masks on a byte-by-byte basis up to the number of events
in the even list (eventCount). VerifyMasks can be #ifdefed out since it is a
debugging tool. As demonstrated below, the first three events' masks
contain a bit set to 1 which defines its respective position in the
sequence.
 outOfBounds mask = b1; // rightmost position
badAlloc mask = b10; // 2nd rightmost position
illegalDefault mask = b100; // 3rd rightmost position
...
VerifyMasksadds the masks of each byte together. The sum should
yield B11111111 for 8-bit/byte (255), B1111111 for 7-bit/byte (127). Note that
the leading bits set to zero are not printed. Also note how the 1s in
the individual masks do not overlap. By adding them together, the
result is a byte-full of ones. If the total mask sequence were to not be
255 (8-bits) or 127 (7 bits), there would be a programming error
somewhere.
The last three public functions, PrintVariables, PrintMasks, VerifyMasks are
development aids. Once a target class has been set up, these may
be ifdefed out. One additional print function, PrintRawData, prints events
as groups of 8-bits.
The first three events illustrate using the eventsStorage class's Get
function. When eventsArray[0] yields B00000010, it means that both
OutOfBounds and IllegalDefault (above) are fine, but BadAlloc has been
triggered. A for loop could print all triggered events as follows:
for ( int i = 0, i < EProgErrors::end; i++ ) {
uint8_t result{0};
[Link](i, &result) {
if ( result == true ) {
[Link](labelsProgErrors [i]); [Link](F(" got triggered."));
} // if
} // for
Notice how the loop cycles through all the events, irrespective of the
number of registers used to store them, regardless of the fact that
they may be stored as 7 events/byte or 8 events/byte. When an
event gets triggered, such as the badAlloc in the example above, it is
up to the application to decide what to do.

7.
Linked List Framework (appendix)
<x>Data Handling Frameworks[AUX]Linked List Framework enables any class to store values inside linked lists</x><x>Linked List Framework[AUX]Based on LinkedItem and

The Linked List Framework (page 172) defines base classes

LinkedList</x>

which grant linked list functionalities to derived classes:

 LinkedItem - It contains LinkedItem* next, which enables forward
traversal via GetNext.
LinkedList - It manages the list via the functions Append, GetFirst and
GetLast. It contains one private variable LinkedItem* first. It also has
print functions to visualize the list's content.

The mechanics underlying the two

<x>Linked List Framework[AUX]LinkedList class: GetFirst, GetLast...</x>

linked list classes relies on the LinkedItem class which enables the
derived class to call next. The LinkedList class manages traversing the
list forward and enables items to be appended to the end of the list.
It does not remove items, reorder items, nor does it do an InsertAfter.
These and other functionalities could be easily added to the basic
implementation.
class LinkedItem {
public:
LinkedItem (LinkedItem* _next);
LinkedItem* GetNext();
virtual void PrintItem(bool _insertNewline) = 0;
private:
LinkedItem* next{0};
}; // class LinkedItem
As may be judged from the above,
<x>Linked List Framework[AUX]Individual items or entire list may be printed</x>

the LinkedItem class is extremely simple. It has a private variable next

and a function GetNext to get the next item. It has one pure virtual
function PrintItem, defined in the derived class, used by the LinkedList
class below to print the entire list.
class LinkedList {
public:
LinkedList();
void AppendItem (LinkedItem* _next);
LinkedItem* GetFirst();
LinkedItem* GetLast();
void SetFirst (LinkedItem* _first);
virtual void PrintList() = 0; // Customized list includes a header and a tail
void PrintItems(); // uses LinkedItem::PrintItem
private:
LinkedItem* first{0};
}; // class LinkedList
The LinkedList class manages the linked list. It has one private variable
first, functions to set and get it, a function to get the last item, and a
function to append an item. Notice that the LinkedItem class has a pure
virtual function PrintItem whose code is implemented in the derived
class. The LinkedList class has two print functions. PrintItems prints the
items sequentially, and one pure virtual function PrintList, which
enables the derived class to customize the print job, for example:
MyPersonalLinkedList::PrintList() {
... Do header work prior to printing the items
PrintItems();
.. do finishing work
} // PrintList
The above is a minimalist linked list class in that it is limited to
appending items and traversing the list forward. It can be enhanced
to traverse the list backwards, to insert an item somewhere inside
the list, to remove items, to sort items, to do validity checks when
items are inserted, etc. It could also include other data common to
derived classes. In my case, I added line numbers, and class and
function indices to the LinkedItem base class since the two derived
classes, error, and memory management, use these. Every item
which is common in derived classes may be incorporated into the
base class. Items which are specific to derived classes are handled
individually by the respective derived classes.
There is really not much to it, but it has proven to be practical, easy
to implement, and a time saver.

3.
Specialized Frameworks (appendix)
Six frameworks, which do not fall
<x>Specialized Frameworks[AUX]Seven framework toolkits to accomplish sundry tasks</x>

under Organizational Frameworks (page 151) or Data Handling

Frameworks (page 158)s, are categorized as Specialized
Frameworks (page 173). These are:

Algorithm Test Framework (appendix) (page 272) is a

methodology for developing and verifying an algorithm's
multiple execution paths.
Class and Function Names Referencing Framework
(appendix) (page 276) - Memory-tight applications cannot
 afford to identify classes and functions via dedicated strings.
Class and function enums are used instead and are transmitted
as IDs to the application which handles reporting. The IDs are
then converted into class and function names.
Memory Management Framework (appendix) (page 278) - A
clear insight into how the application uses memory is crucial to
its robustness. Details on heap space measurement and
memory allocation are presented.
Pseudo Exception Handling Framework (appendix) (page 281)
- C++'s exception handling is unfortunately not part of the
Arduino distribution. C's setjmp/longjmp is used instead to mimic
C++'s exception handling.
Error Reporting Framework (appendix) (page 284) - As the
application runs, it may undergo problems. These are logged
as errors or warnings inside a linked list for reporting purposes.
Print-based Debugging Framework (appendix) (page 285) -
Print-based debugging requires standardized prints such as
entry/exit into functions, memory use reporting, and critical
conditions display. These are encapsulated inside macros.

1.
Algorithm Test Framework (appendix)
<x>Specialized Frameworks[AUX]Algorithm Test Framework plan and test multiple execution paths</x><x>Algorithm Test Framework[AUX]Define execution paths and

Whenever you attempt to code something that

controlling parameters, then fill in real code</x>

gets a little complex, it helps to lay things down on paper first -

develop a visual representation as to what you are trying to do
(Algorithm Test Framework page 174). Begin by drawing a flow
diagram which reveals execution paths, each of which will depend
on control variables. Create one #define for each execution path in
which you set the control variables' corresponding values. Having
done this, test each execution path. If the logic fails, modify the flow
diagram or values, and repeat the process.
I developed the logic testing methodology described above due to
difficulties getting part of my Arduino-based beehive weighing
system to work properly.
<x>Algorithm Test Framework[AUX]Start with flow diagrams to peg down algorithm's logic</x><x>Algorithm Test Framework[AUX]Unravel add beehive logic with flow

diagram</x><x>Algorithm Test Framework[AUX]Unravel add beehive logic with flow diagram</x><x>Beehive weighing system[AUX]Test beehive weighing system execution

paths with Algorithm Test Framework;I knew what I wanted to

unravel add beehive logic with flow diagram</x>

accomplish and thought I had clearly defined the logic. I therefore

started by coding directly, not having drawn a flow diagram. My initial
code led to bad results, which was a surprise because I was sure of
myself. This experience led me to draw a flow diagram. I quickly
discovered that the process was more complex than I had originally
anticipated. After testing code, I had to go back to the flow diagram
and draw, test, and modify code until I finally got it right. In the final
version, three if statements and a switch statement control the process.

Figure 22.1 - Test Arduino-based beehive weighing system execution paths with Algorithm Test Framework.

The diagram above contains horizontal rectangles to indicate actions

and 45° squares to indicate true/false decisions. It reveals the logic
underlying registering or not registering a beehive when the
coordinator gets data from the beehive. There are 6 execution paths,
each path is labeled as P1, P2, P3... Three variables control execution
flow.
Having finalized the flow diagram, I wrote code to simulate the
various paths. The flow diagram I ended up with may be
characterized by:

An entry point during which preliminary tasks are undertaken

and control parameters are set.
It contains a total of 6 execution paths which need to be tested
individually via appropriate #define #ifdef pairs labeled P1, P2...
It has four exits (P1, P2, P5, P6) grouped into one, and two error
conditions (P3, P4) which need to be handled. Error handling is
outside the scope of the flow diagram. It could be an abort, a
longjmp, a return false.
It contains three if statements and one switch statement.

In the code below, each execution path is tested by uncommenting

the appropriate #define to reveal whether coded logic matches that of
the flow diagram.
The following 6 macros define the execution paths to be simulated.
Only one at a time should be active:
#define PATH_1 // Data mode, registered, real name
//#define PATH_2 // Data mode, registered, temp name, set real name
//#define PATH_3 // Not registered, beehive number limit reached - ERROR1
//#define PATH_4 // Not registered, transmit mode bad - ERROR 2
//#define PATH_5 // Data mode, not registered, register temp name "Registered xx"
//#define PATH_6 // Maintenance mode, not registered, register with real name
Each path is managed by an #ifdef which defines a set of control
variables. These will channel execution to the corresponding end
point, PATH_1 for example:
#ifdef PATH_1 // Data mode, registered, real name
isRegistered = true;
dataMode = ETransmitMode::data;
[Link](F("Setting PATH 1 - isRegistered TRUE, data mode"));
#endif // PATH_1
When PATH_1 gets defined, the variables will be such that flow should
proceed along the defined path. The code will print whatever path
ends up being selected. Hopefully, the correct one gets attained as
verified by the path's definition and the path's outcome being
identical. Once the logic is nailed down, code to do the real work can
be plugged into the appropriate locations.
Here is the outcome for PATH_1:
Start simulation register module
Setting PATH 1 - isRegistered TRUE, data mode.
Module is registered
PATH 1 - Module is registered, data mode do nothing
End simulation
The output clearly verifies the logic's conformance with the flow
diagram. It could be improved by appropriate formatting; however, it
serves its purpose adequately enough. The next major step consists
in verifying the logic with real input parameters.
Here is the complete code to test the beehive registration according
to the flow diagram above.
#define REGISTRATION_TEST
#ifdef REGISTRATION_TEST
[Link](F("Start simulation register module"));
// Controlling variables
enum DataMode { eData, eMaintenance, eDefault};
DataMode dataMode{eData};
const int numberModulesLimit{16};
int numberModules{16};
int registrationID{0};
bool isRegistered{true};
char moduleName[20];
//Uncomment one of 6
//#define PATH_1 // Data mode, registered, real name
//#define PATH_2 // Data mode, registered, temp name, set real name
//#define PATH_3 // Not registered, ERROR - beehive number limit reached
//#define PATH_4 // Not registered, transmit mode bad - ERROR
//#define PATH_5 // Data mode, not registered, register temp name "Registered xx"
#define PATH_6 // Maintenance mode, not registered, register with real name
#ifdef PATH_1 // Data mode, registered, real name
isRegistered = true;
dataMode = ETransmitMode::data;
[Link](F("Setting PATH 1 - isRegistered TRUE, data mode"));
#endif // PATH_1
 //Paths 2 to 6 are similar:
if ( isRegistered == false) { // outer if
[Link](F("Module is not registered"));
if ( numberModules >= 16 ) { // inner if
[Link](F("PATH 3 - module not registered - ERROR Beehive limit reached"));
} else {
switch (dataMode) {
case ETransmitMode::data: // PATH 5
[Link](F("PATH 5 - Data mode, not registered, register with temp name"));
break;
case ETransmitMode::maintenance: // PATH 6
[Link](F("PATH 6 - maintenance mode, not registered,"
" register with real name"));
break;
default: // PATH 4
[Link](F("PATH 4 - ERROR default transmit mode"));
break;
} // switch
} // inner if
} else { // Module is registered
[Link](F("Module is registered"));
if (dataMode == eMaintenance && strncmp(moduleName, "Registered", 10) == 0) {
[Link](F("PATH 2 - maintenance mode, module registered temp name, update real name"));
} else {
[Link](F("PATH 1 - data mode, module is registered with real name, do nothing"));
} // inner if
} // outer if
[Link](F("End simulation."));
#endif // REGISTRATION_TEST

2.
Class and Function Names Referencing
Framework (appendix)
<x>Specialized Frameworks[AUX]Class and Function Names Referencing Framework - IDs identify classes/functions</x><x>Class and Function Names Referencing

The Class and Function Names Referencing

Framework[AUX]Identify classes/functions via IDs</x>

Framework (page 176) provides the mechanics to identify where

events come from: Which function in which class triggered the event.
Class and function names could be used to identify the event but
these require a lot of memory because of the need to store names
as strings inside class and function definitions. If the application were
to run on a microcontroller with plenty of memory such as an
ATmega2560, there should not be any memory issues, unless the
application were to be exceptionally large.
The enum-based event identification solution, as opposed to the
string-based solution, reduces memory requirements considerably.
Three bytes suffice to identify the class and function (2 for the class
ID - uint16_t, 1 for the function ID - uint8_t). The memory overhead
becomes minimal. IDs can be transmitted to the server which will
convert these into names for reporting purposes.
The system is based on enum class lists that index into class structs
array which tie function name arrays to their respective classes.
struct StructCandFnames {
char* classLabel; // class name
char** functionLabelsArray; // array of function name pointers
};
Having defined the class/function structure (StructCandFnames), an array
of class and corresponding class specific function name arrays get
populated.
...
{ "Employee", employeeFunctionNames },
{ "Manager", managerFunctionNames },
...
A enum list defines indices to access individual rows in the
class/function structure.
enum class EClasses: uint16_t {
start = 0, Employee = 0, Manager, ..., end
}; // enum EClasses
There should be as many StructCandFnames instances as there are
classes. A coherence check may be undertaken by checking the
value of the last enum EclassNames::end with the struct's number of items,
thus:
uint16_t numberOfClassNames = sizeof(structCandFnames)/ sizeof(structCandFnames[0]);
uint16_t numberOfClassEnums = (uint16_t) EclassNames::end;
if ( numberOfClassNames != numberOfClassEnums) {
We have a problem!!!
}
On a class-by-class basis, set up as many function enums as there are
functions in the class such as:
 enum class EEmployeeFunctions: uint8_t { // 256 functions per class should be enough
start = 0,
EmployeeConstructor = 0, EmployeeDestructor, GetMgrID, GetEmployeeID, ...,
end
}; // enum EEmployeeFunctions
Function names are contained in class specific string arrays
accessed by the corresponding class specific function enums
described above, thus:
char* employeeFunctionNames[] {
"Employee constructor",
"Employee destructor",
"GetMgrID",
"GetEmployeeID",
...
};
The enum list's name and the string array's name both incorporate the
class's name Employee to identify the class these belong to. In EEmployee,
the E prefix is a mnemonic for enum, Employee being the class.
Furthermore, notice that the enum class list, presented earlier, is sized
16 bits. The enum functions list is sized 8-bits. I believe that large
applications may have more than 256 classes, but an individual
class would generally have less than 256 functions. If you want
more, just change the enum declaration accordingly.
By using sizeof, check that the function's enum list size and the
corresponding function names array size match (do the same as was
done above on class names).
<x>Class and Function Names Referencing Framework[AUX]class CandFnames does all the work</x>The class and function

names arrays are managed by the class CandFnames (class

name/function name array struct, class enum list, class specific function
enum list, and function names arrays). This class's Get functions uses
IDs (enums) to generate class and function name strings for reporting
purposes:
class CandFnames{
public:
CandFnames();
~CandFnames();
char* GetClassName (uint16_t _classID);
char* GetFunctionName ( uint1""6_t _classID, uint8_t _functionID);
uint16_t GetNumberOfClasses();
uint8_t GetNumberOfFunctions (uint16_t _classID);
 bool CheckClassID (uint16_t _classID);
bool CheckFunctionID (uint16_t _classID, uint8_t _functionID);
void PrintClassList();
void PrintFunctionList (uint16_t _classID);
private:
ClassesAndFunctionsNames classesAndFunctionsNames[EClassNames::end];
uint16_t classID{0};
uint8_t functionID{0};
}; // class CandFnames
Most of this class's functions are there for housekeeping purposes.
GetClassName and GetFunctionName are the two relevant functions from a
user perspective. This class incorporates behind-the-scenes checks
to prevent invalid class and function indices from being used. If such
errors are found, it is up to the programmer to decide what to do:
abort and fix the problem or use default class or function names for
reporting.
<x>Class and Function Names Referencing Framework[AUX]Automate class and function IDs insertion</x>For the system to be

operational, enum lists and name arrays need to be created, and enum
identifiers need to be inserted in each class and function. Doing such
insertions manually for a small application is feasible however, a
large application may have 100+ classes and 10+ functions in each
amounting to 1000+ names to manage. Manually creating the enums,
struct contents and char* arrays would be time consuming and error
prone. There is the further issue of inserting IDs into all application
class and function declarations. Given the extent of work required to
accomplish this, automation is highly recommended. This would
require a special purpose program which would extract class and
function names in a first pass and modify source code accordingly in
a second pass by inserting the IDs as variables and GetFunctionIDs as
class functions. This process is referred to as instrumentation.
A mini compiler could be written to recognize classes and functions
however, it is a task reserved for compiler writers, which requires
specialized skills. lex (lexical analysis) and yacc (yet another
compiler compiler) could also be used; but, its implementation is
complex and it risks not being flexible enough. Then there is Perl,
with its C-like programming language and regex engine. It could be
the appropriate tool to create a program to extract the required data
from the C++ source files. It can be made to produce one .h file and
one .cpp file that contains the enums and string arrays which can be
added to the project. It can also instrument the project source files by
inserting the enum identifiers and GetID functions in classes and
functions, the modified files being placed in a separate directory to
preserve the original files. Once instrumentation is done, the
programmer needs to modify one application source file - do an
#include "CandFnames.h" file, and add a #define USE_CLASS_NAMES to turn the
use of the framework on and off. The application should, of course,
be retested. See Class and Function Names Referencing
Framework instrumentation (page 178).
One last word: class and function names could have been stored in
EEPROM to save precious RAM space. Doing this could be handled
by the DataGroup class of the DataGroup Framework. This has been put
aside for later work.

3.
Memory Management Framework
(appendix)
<x>Specialized Frameworks[AUX]Memory Management Framework informs on heap space, memory use, memory gluttons</x><x>Memory Management

Framework[AUX]Memory allocations verify heap availability and reveal memory leaks</x><x>Memory (leaks)[AUX]Memory Management Framework discloses memory gluttons

Getting a hold on memory use as the application runs is

and memory leaks</x>

vital. During the development and testing phases, the Memory

Management Framework (page 179) registers info on allocations
and releases and verifies that allocations are successful. When the
application ends, a report reveals how memory was used, by whom,
and whether there are memory leaks. The Memory Management
Framework components are:

<x>Memory Management Framework[AUX]MemMgt - main class for handling memory management events</x> MemMgt - Main class
for handling memory management events.
<x>Memory Management Framework[AUX]MemAllocEvt and MemAllocList - base classes for managing allocations</x> MemAllocEvt -
Memory allocation events are created for each memory
allocation and release. They contain the following: unique
event identifier, bytes allocated, heap space available, class
 and function ID, and line number. It inherits from the LinkedItem
class.
MemAllocList - This class manages a linked list of MemAllocEvts. It
displays the memory allocation history and displays active
items (items not released). It inherits from the LinkedList class for
storage as a linked list. See Linked List Framework (page
172).
<x>Memory (use)[AUX]Memory allocation failure avoidable with memory monitoring functions</x><x>Memory (heap fragmented)[AUX]Essential component of

Memory Management Framework</x><x>Memory (heap contiguous)[AUX]Essential component of Memory Management Framework</x> TotalHeap
calculates heap space, fragmented and contiguous. The
FragmentedHeap function calculates the holes total and the biggest
possible allocation (biggest hole or contiguous space). The
ContiguousHeap function calculates the space between the top of
allocated space and the bottom of the stack.
<x>Operator overloading[AUX]Overloaded new: memory management, verify allocation success, log events</x><x>Memory Management

Overloaded new/new[] and

Framework[AUX]Overloaded new: memory management, verify allocation success, log events</x>

corresponding overloaded delete/delete[] - If an allocation is

successful, the overloaded operator new will append a
MemAllocEvt to the memory allocation event list, otherwise, it will
register it as an error event for later reporting. See Operator
overloading (page 186) and Error Reporting Framework (page
185).

A linked list of memory allocation events is the underlying building

block of the Memory Management Framework reporting system.
These events contain the following:

Unique allocation number identifier - When an object gets

created via a new, it is assigned a unique allocation number
which can be used to verify which objects have not been
deleted, thereby revealing possible memory leaks. This
allocation number ties an object's creation to its corresponding
release.
classIndex and functionIndex - The class is registered to inform as to
 the class (could be global if a global function did the
allocation). Upon reporting, classIndex (hard coded inside each
class) is used to obtain the class's name via the mechanisms
of the Class and Function Names Referencing Framework.
The functionIndex, also hard coded as a local enum item inside each
function, is used to obtain the function's name. See Class and
Function Names Referencing Framework (page 176).
Memory allocation events are create events and release
events. Each event is identified by a corresponding enum.
Number of bytes - When items get created or released, the
number of bytes is recorded by the overloaded new operators. It
includes the overhead memory requirements of the MemAllocEvt
instance.
Heap space used - As objects get created/released, heap
space shrinks/expands. But another mechanism is
consuming/releasing heap space: function calls. The stack
grows and shrinks as functions get called and exit. Displaying
heap space regularly, as allocations/releases occur and as
functions get called and exit, helps detect when the stack and
heap risk colliding.

Logging an event's memory allocation or release requires summing:

The total of a class's memory requirement via the sizeof operator

thus: sizeof(MyClass) plus 2 bytes for the pointer.
MemAllocEvt overhead memory requirements, included in the
allocation byte count.
Whatever additional memory requirements the class might
require are logged separately by the overloaded new/new[]
operators.

The class MemMgt, instantiated once at the beginning of the application,

provides functions to monitor memory use.
class MemMgt {
<x>Memory Management Framework[AUX]MemMgt - main class for handling memory management events</x>
 public:
MemMgt();
// Memory requirements
int16_t CalculateMemAllocEvtSize (uint16_t _classIndex, uint16_t _classSize,
EMemCreateRemove labelCreateRemove);
void HandleMemoryConstructor (uint16_t _classIndex,
uint8_t _functionIndex, char** functionLabels, uint16_t _classSize);
void HandleMemoryDestructor (uint16_t _classIndex,
uint8_t _functionIndex, char** functionLabels, uint16_t _classSize);
// Objects created numbering
uint16_t GetNetObjectsCreated();
uint16_t GetGlobalMemObjectCreated();
void NetObjectsCreatedIncrement();
void NetObjectsCreatedDecrement();
// Heap space measure
void SetInitialHeapSpace (uint16_t _initialHeapSpace);
int16_t GetInitialHeapSpace();
void AdjustTheoreticalHeapSpace (int16_t _delta);
// Create/destroy F/U items
void CreateMemAllocItem (
uint16_t _classIndex, int16_t _memAlloc, char* _callingFunction, int _lineNo);
void DestroyMemAllocItem (
uint16_t _classIndex, uint8_t _functionIndex, int16_t _memoryAllocation,
char* _callingFunction, int _lineNo);
MemAllocEvtList* GetMemAllocEvtList();
void IncrementTheoreticalHeapSpace (int16_t _increment);
uint16_t GetTheoreticalHeapSpace();
void CompareRealMinusTheoreticalHeapSpace (uint16_t _heapSpace);
void Print();
void PrintReportingMemAllocStats();
private:
MemAllocEvtList* MemAllocEvtList;
// These are the net number of objects active at any given time new and malloc
// Each object creation gets a unique new number, these are never decremented
static uint16_t netObjectsCreated;
static uint16_t globalMemObjectCreated;
static uint16_t initialHeapSpace;
static uint16_t theoreticalHeapSpace; // F/U as objects get created/destroyed
}; // class MemMgt
The class MemAllocEvt inherits from LinkedItem so that events can be stored
inside a linked list. It contains four variables, with their attendant
Set/Get functions. It passes class and function indices and line number
to the base class LinkedItem:
 class MemAllocEvt: public LinkedItem {
public:
MemAllocEvt ( uint16_t _classIndex, uint8_t _functionIndex, int _lineNo,
EMemCreateRemove _status,
uint16_t _allocNumber, int16_t _memAlloc );
void PrintItem (bool _insertNewline);
int16_t GetMemoryAllocated();
uint16_t GetAllocNumber();
uint8_t GetStatus(); // 0 or 1 (0 means deleted 1 for created)
private:
uint16_t allocNumber{0}; // alloc identifier
int16_t memAlloc{0}; // allocated memory
int16_t heapSize{0}; // heap
uint8_t status; // 0 or 1 (0 means deleted 1 for create)
}; // class MemAllocEvt
The code below indicates that the function DoSomething, located in line
285, allocation number 23, has allocated 26 bytes of memory, and heap
space is 554 after the allocation. Later on, the same function
DoSomething, line number 326, allocation number 23, releases 26 bytes,
heap space is increased to 580 bytes.
MemEvent 23 class Sample::DoSomething:285 create 26 heap
space = 554
MemEvent 23 class Sample::DoSomething:326 release 26 heap
space = 580
Since allocation 23 has been both created and released, it is no
longer an active allocation.
The individual MemAllocEvt items are managed by the class
MemAllocEvtList which inherits from LinkedList.
class MemAllocEvtList: public LinkedList {
public:
MemAllocEvtList(): LinkedList() {};
void PrintList();
private:
//int16_t totalMemoryAllocated{0};
}; // class ObjectCreationFollowUpList
It prints items sequentially via the customized PrintList function.
When the application ends, all memory allocations should have been
released. Allocations that have not been released are memory
leaks - the count of active components should be zero. The report
reveals which items have not been released.
 4.
Pseudo Exception Handling
Framework (appendix)
<x>Specialized Frameworks[AUX]Pseudo Exception Handling Framework - alternative to C++'s exception handling</x><x>Pseudo Exception Handling

It took me a while to understand

Framework[AUX]Alternative to C++'s exception handling based on setjmp and longjmp</x>

the underlying concepts of C++'s exception handling, unfortunately

not supported by Arduino, which got me to create a substitute:
Pseudo Exception Handling Framework (page 183).
Imagine the following: you are a mountain trek group leader. At some
point you stop and tell the group members: "If a problem arises,
stop, come back here. We would then decide what to do next". The
key to understanding the C++ exception handling mechanism lies in
the stop and come back to decision. When you do a try something,
you are defining a landing point to come back to if somewhere inside
the code you are trying the application fails, in which case it would
throw an exception which brings execution back to try. In much the
same way, setjmp defines a landing point to come back to if a longjmp
gets triggered downstream. This mechanism enables doing pretty
much the same work as exception handling: define a landing point,
do work from there on, and if something fails, come back to the
landing point. See Pseudo Exception Handling Framework (page
183).
Imagine a situation whereby a function would check that a variable's
value should not be zero. If it happens to be zero, the application
experiences a critical error. Program execution might need to be
aborted. I shall illustrate this behavior in the scenario below with the
function DoWork.
<x>Pseudo Exception Handling Framework[AUX]Deep down the function call chain trigger longjmp back to setjmp landing point</x><x>Pseudo Exception Handling

Deep down
Framework[AUX]setjmp (try) sets landing point</x><x>Pseudo Exception Handling Framework[AUX]longjmp (throw) - return landing point</x>

the function call chain, the function DoWork detects an error - it must
therefore handle it. It does so by triggering a longjmp with appropriate
parameters which transfers execution from inside DoWork all the way
back to the setjmp(envGlobal) landing point located in setup (see the code
below). The longjmp sends back the error number EErrorType::illegalZero to
identify the problem.
...
// could be deep down inside a function call chain
void DoWork() {
...
if (newItem == 0 ) {
// do error reporting HERE
longjmp(envGlobal, EErrorType::illegalZero); // got an error
} // if
...
} // DoWork()
The if(setjumpReturn != 0) statement in setup (below) tests the error number;
if it is nonzero, code inside the if statement gets executed (goto
ErrorExit).
The mechanics of this error handling system is somewhat subtle in
that setjmp is entered twice, a first time upon entering setup thereby
returning 0 since no longjmp has yet been triggered because DoWork has
not been called, a second time if a longjmp gets triggered from inside
DoWork as it does its work.
If the value returned by the landing point setjmp is not 0, this means
that a longjmp containing the error's cause got triggered. Execution is
rolled back to the setjmp's location. Since setjmp returned a non 0, the
if(setjumpReturn != 0) code prints a message and redirects execution to the
location defined by the label ErrorExit.
The fact that program flow may get rolled back to setjmp by a longjmp
requires that the if(setjmp return value) check must be placed before
continuing with code which contains the longjmps. This is counter-
intuitive - one would think that you should start out by doing work
inside of which you should test for an error. If you do this, you get
into an infinite loop. Let me explain. Suppose you code as follows
(pseudocode):
error = setjmp()
DoWork()
if (error != 0 ) goto ErrorExit
This is the flow of events:

Do error = setjmp() - error is set to 0 since no longjmp has yet been

triggered.
Enter DoWork - An error is encountered deep down the call
 chain. It triggers a longjmp which transfers control back up to the
error = setjmp(). error is set to a non-zero value.
DoWork is called once more; an error gets encountered;
execution is transferred again back up to error = setjmp().

The above gets repeated indefinitely - you have just created for
yourself an infinite loop.
Another reason which requires that the setjmp be called before code
containing a longjmp is that the longjmp should know where to go to if it
gets triggered. setjmp must be called a first time to position it in case it
gets invoked by a longjmp. Here is how to set up the setjmp/longjmp
mechanism:

Begin by defining a setjmp, it will return 0 the first time around

since no longjmp will have been triggered.
After the setjmp, insert an if to test setjmp's return value, thereby
determining whether a longjmp has been triggered. The first time
around, setjmp returns 0, the if's content is skipped.
Continue by doing work as usual, if there is an error, the longjmp,
deep inside the call chain which begins with DoWork, transfers
control back up to setjmp before the if.
If an error triggers a longjmp, setjmp gets called a second time. It
now returns a non-zero value. Program flow enters the
if(setjumpReturn != 0). The error is handled, and control is transferred
out to ErrorExit, DoWork gets bypassed.

Important!! The key to understanding the above relies on the

if(setjumpReturn != 0) being traversed twice (see code below):

A first time upon entering setup or whatever - As program flows

into setjmp, it returns 0 since no error has yet been detected. The
if(setjumpReturn != 0) is not activated because setjmp's return value is
0, thus program flows into doing regular work after the if (DoWork
above).
 A second time - If deep down the function call chain an error is
detected, a longjmp gets triggered. Program flow unwinds back
to the setjmp, but this time it returns a non-zero value and
program flow gets transferred inside the if(setjumpReturn != ٠). It
handles the error by transferring out of normal execution down
to ErrorExit.

Once a longjmp has been triggered, it is up to the programmer to

decide what to do. A goto end may be executed or another setjmp/longjmp
downward pair could be implemented to step down closer to the
application's beginning. One could imagine error handling to have a
tree-like structure. Each node would contain a setjmp to serve as a
landing place for longjmps up the tree.
Here is the setjmp/longjmp pseudo exception handling code (the catch
equivalent of C++'s exception handling).
You first need to set global jmp_buf variable which ties the longjmps to
specific setjmps:
// global space
...
jmp_buf envGlobal; // global variable for the example setjmp
...
You then need to define a landing point. In this case, landing point
(setjmp) is placed in setup, before DoWork.
setup() {
...
int setjumpReturn = setjmp (envGlobal); // This is the landing point
if ( setjumpReturn != 0 ) { // setjmp error processing
[Link](F("setjumpReturn = ")); [Link](setjumpReturn);
// goto handle error - record info somewhere based on longjmp parameters
goto ErrorExit;
} // if
// Handle normal processing
DoWork();
[Link](F("Finished normally - normal EXIT"));
delay(100);
return;
 // Handle error processing
ErrorExit:
[Link](F("setup() Error EXIT"));
delay(100);
return;
} // setup()
You finally need to undertake error checking in your code. If an error
does occur, do a longjmp call with an error code (this is illustrated in
DoWork below).
// The longjmp could be deep down inside a function call chain starting with DoWork
void DoWork() {
...
if (newItem == 0 ) {
// do error reporting HERE
longjmp(envGlobal, EErrorType::zeroValue); // got an error
} // if
...
} // DoWork()
Try it and get familiar with the three components (env_buf, setjmp,
longjmp). The overall mechanism is subtle but rather simple to
implement.

5.
Error Reporting Framework
(appendix)
<x>Specialized Frameworks[AUX]Error Reporting Framework - log errors and warnings inside a linked list</x><x>Error Reporting Framework[AUX]ErrorEvt class stores info on

Error reporting is similar to memory management

event and links them as linked list (ErrorList)</x>

reporting (Error Reporting Framework page 185) in that they both

inherit from a linked list for storage and reporting purposes. An enum
list and a corresponding array of names provide the mechanisms for
reporting with names.
enum class EErrorType: uint8_t {
start = 0,
outOfBounds = 0, badAlloc, illegalDefaultExit, ...,
end
}; // enum EErrorType
 char* errorTypeLabels[] {
"outOfBounds = 0,
"failedAlloc",
"illegalDefaultExit",
...
}; // errorTypeLabels
Individual errors are
<x>Error Reporting Framework[AUX]ErrorEvt and ErrorList inherit from LinkedItem and LinkedList respectively</x>

instantiated as events of the ErrorEvt class. These contain the error's

type (EErrorType), the class and function IDs, and the source code line
number. This ErrorEvt class inherits from LinkedItem for linked list storage.
It contains one private variable errorType, all other info is located in
LinkedItem. It also contains a PrintItem function.
class ErrorEvt: public LinkedItem {
public:
ErrorEvt ( int _lineNo, EErrorType _errorType );
void PrintItem (bool _insertNewline);
static const uint16_t classIndex = (uint16_t) EClass::ErrorEvt;
private:
EErrorType errorType{0};
}; // class ErrorEvt
Individual error events are appended to the error list (ErrorList) for
reporting purposes. The ErrorList class inherits from LinkedList which
contains a PrintList function to print the list's entire content.
class ErrorList: public LinkedList {
public:
ErrorList();
void PrintList();
private:
}; // class ObjectCreationFollowUpList

6.
Print-based Debugging Framework
(appendix)
<x>Specialized Frameworks[AUX]Print-based Debugging Framework - selectively choose debugging print #defines</x><x>Print-based Debugging Framework[AUX]Debug control

The Print-based Debugging Framework

macros turn debugging on/off global, file, and function level</x>

(page 188) contains three macro categories: global level, function

level, and debug control. Each category contains an extensive set of
macros to selectively choose what gets printed, to inform on
application-wide events, and to provide standardized function level
debugging.

<x>Print-based Debugging Framework[AUX]Macro categories: navigation, memory use, critical situation, data values</x><x>Macros[AUX]Function entry and exit

macros print details upon entering and exiting functions</x><x>Print-based Debugging Framework[AUX]Entry/exit macros provide parameter and return values, and

Global-level macros (next section) - These macros

inform on selected items</x>

inform on application events: milestones, critical conditions,

errors, and memory-use.
Function level macros (page ٢٨٩) - Each function may contain
standardized macros to signal entry/exit, provide parameter
and return values, and inform on selected items.
Debug control macros (page ٢٩١) - These macros are grouped
according to the application's architecture namely: application,
files, functions.

These three macro categories are detailed in the sections which

follow.

1.
Global level macros
Global level macros may be used anywhere in the application: deep
inside a function or up close in the program's entry function. Four
macros contribute to this group (these are expanded upon farther
down):

<x>Macros[AUX]Milestone macros highlight program's location</x> Milestones macros (next section)

help identify key events by being highly visible so that they
may be easily located in the serial monitor's output. These
messages, used sparingly, clearly delimit blocks of output. An
application such as my beehive weighing system, which
sends/receives SMSs, reports, and gets data from the
beehives may be organized as distinct execution blocks.
Milestones clearly delimit these in the serial monitor's output.
<x>Macros[AUX]Critical reporting macros triggered when resource runs low (voltage, memory...)</x> Critical reporting
 macros (page 287) sit silently in the background in strategic
locations. They get activated when a resource becomes critical
such as low voltage, memory getting low, or the clock not
working. Here is an example from my beehive weighing
system. Since my prototype runs on batteries, I use a critical
macro to signal that voltage is getting low and it is time to
replace the batteries. Another example: if heap space were to
be running low or heap fragmentation high, a critical macro
displays the info.
<x>Macros[AUX]Error reporting macros print highly visible text to describe errors</x> Error reporting macros (page

288) - As suggested earlier in Good programming practices

(page 129), never assume anything. Always undertake validity
checks on parameters. When a validity signals that there is a
problem, the macro will report the function's name and line
number, the calling function and line number, and text which
describes the problem. There is no error handling, the macro
merely reports that something happened.
<x>Macros[AUX]Memory use macros inform on current available memory and heap fragmentation</x><x>Memory (use)[AUX]Print-based debugging macros

Memory use macros (page 288) - A heap

display total, contiguous, fragmented heap</x>

space macro can be placed anywhere to inform on current

available memory and heap fragmentation.

1.
Milestone macros

Milestones highlight the program's location by printing text with

leading and trailing lines of highly visible character sequences. A
typical milestone macro would look like this:
#define MILESTONE_DISPLAY(desc) \
[Link](F("<MMMMMMMMMMMMMMMMMMMMMMMMMMMM>")); \
[Link](F(desc)); \
[Link](__FUNCTION__); [Link](F(":")); [Link](__LINE__); \
PRINT_VALUE_EQUAL(millis()) \
[Link](F("<MMMMMMMMMMMMMMMMMMMMMMMMMMMM>"));
This macro meets several needs:

Visibility - It prints two highly visible lines of repeated

 characters. This is an eye-catcher which helps one find it in
reams of printouts.
Identification - It receives a string parameter to reveal the
milestone's location, and the function's name and line number.
Timing - It prints milliseconds since startup via the macro
PRINT_VALUE_EQUAL(millis()).

The milestone macro

MILESTONE_DISPLAY("Just before getting an SMS.")
would print:
MMMMMMMMMMMMMMMMMMMMMMMMMMMM
Just before getting an SMS.
GetSMS:155
millis() = 1245
MMMMMMMMMMMMMMMMMMMMMMMMMMMM
This output is highly visible and reveals where the application is. Two
similar milestone macros meet enhanced needs:
#define MILESTONE_DISPLAY_VALUE(desc, val)
#define MILESTONE_DISPLAY_VAL1_VAL2(desc, val1, val2)
They display a description and one or two parameters with their
corresponding values and millis(). The macro described above is
minimalist. It could be enhanced to display other info.

2.
Critical reporting macros

Critical reporting macros(ex: CRITICAL_DISPLAY) resemble the

milestone macro - they display highly visible text.
#define CRITICAL_DISPLAY(text) \
[Link](F("<CCCCCCCCCCCCCCCCCCCCCCCCCCCCCC>")); \
[Link](F(text)); \
[Link](__FUNCTION__); [Link](F(":")); [Link](__LINE__); \
PRINT_VALUE_EQUAL(millis()) \
[Link](F("<CCCCCCCCCCCCCCCCCCCCCCCCCCCCCC>"));
A typical use of the macro:
if (heapSpace < HEAP_SPACE_MINIMUM ) {
#ifdef CRITICAL_REPORTING_ON
CRITICAL_DISPLAY("Memory prob, heap space getting low")
PRINT_VALUE_EQUAL (heapSpace);
 [Link](F(" minimum = "));
[Link](HEAP_SPACE_MINIMUM);
#endif // CRITICAL_REPORTING_ON
// Handle low heap space error
...
} // if (heapSpace < HEAP_SPACE_MINIMUM )
The output would be:
CCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
Memory prob, heap space getting low
GetSMS:160
millis = 1247
CCCCCCCCCCCCCCCCCCCCCCCCCCCCCC
heapSpace = 255 minimum = 350
...
Again, the output is highly visible. If there were no such critical
problem issues, commenting out the REPORTING_ON macro would
deactivate them.

3.
Error reporting macros

The error reporting macro resembles the milestone and critical

macros. A chain of characters is printed before and after the
contents to highlight them (PROGRAM_ERROR_TEXT macro). The macro
prints a description, the function and line number, and the function
which called it with an attendant line number.
#define PROGRAM_ERROR_TEXT(PARAM_text) \
[Link](F("ERROR - EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE>"));\
[Link](F("\t")); [Link](F(PARAM_text)); \
[Link](F("\tFrom: ")); [Link](__FUNCTION__);\
[Link](F(":")); [Link](__LINE__);\
[Link](F("\tcalled by ")); [Link](_callingFunction);\
[Link](F(":")); [Link](_line number);\
[Link](F("\tmillis ")); [Link](millis());\
[Link](F("ERROR - EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE>"));
A typical output would be
ERROR - EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE>
Parameter should not be zero
From: DoWork:426 called by ATest:235
millis() = 1260
ERROR - EEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEEE>
The data reported above is minimal, other macros derived from the
one above could contain considerably more info.

4.
Memory use macros

The Print-based Debugging Framework uses two macros to obtain

info on memory use:

PRINT_HEAP_SPACE- provides total, fragmented, and contiguous

heap data and the largest possible alloc.
#define PRINT_HEAP_SPACE(paramDesc) \
[Link](F(paramDesc));\
[Link](F("Largest possible alloc = "));
[Link](LargestPossibleAlloc(__FUNCTION__, __LINE__));
[Link](F("Fragmented heap = ")); [Link](FragmentedHeap(__FUNCTION__,
__LINE__));
[Link](F("Contiguous heap = ")); [Link](ContiguousHeap(__FUNCTION__,
__LINE__));µ
[Link](F("Free RAM = ")); [Link](TotalHeap(__FUNCTION__, __LINE__));
PRINT_MEMORY_CONSTANTS provides values on all system memory
constants:
#define PRINT_MEMORY_CONSTANTS(_description) \
{\
[Link](F("&__data_start ")); [Link]((uint16_t)&__data_start); \
[Link](F("&__data_end ")); [Link]((uint16_t)&__data_end); \
[Link](F("&__bss_start ")); [Link]((uint16_t)&__bss_start); \
[Link](F("&__bss_end ")); [Link]((uint16_t)&__bss_end); \
[Link](F("&__heap_start ")); [Link]((uint16_t)&__heap_start); \
[Link](F(" points to ")); [Link]((uint16_t)__heap_start); \
[Link](F("&__heap_end ")); [Link]((uint16_t)&__heap_end); \
[Link](F(" points to ")); [Link]((uint16_t)__heap_end); \
[Link](F("__brkval ")); [Link]((long)__brkval); \
[Link](F("__ctors_start ")); [Link]((long)&__ctors_start); \
[Link](F("__ctors_end ")); [Link]((long)&__ctors_end); \
}

For more on Memory use macros see Memory structure (appendix)

(page 296).

2.
 Function level macros
There are two function level debugging macro categories:

Function entry and exit (next section) - These macros print

details upon entering and exiting functions, such as the
function's name and line number, the calling function name and
line number, parameter lists via the parameter name/value
macros below and return values.
<x>Macros[AUX]Parameterized name/value macro simplifies printing a parameter's value</x> Print a variable's value

macro (page 290) - This macro prints both a parameter's name

and its value. There is no need to repeatedly do [Link] to
print the name and value.

1.
Function entry/exit macros

Function entry
<x>Print-based Debugging Framework[AUX]Entry/exit macros provide parameter and return values, and inform on selected items</x>

debugging macros have been standardized. They list:

Function name and line number obtained via the compiler's

built-in variables __FUNCTION__ and __LINE__. __FILE__ could have
also been used, however, it takes up too much memory space
as the full path gets printed, easily 100 bytes per call.
When printing entry info, it is often useful to know which
function called the one being reported from. This is
accomplished by adding two parameters to the function: char*
_callingFunction and int _line number. The function's debug macro prints
the function/line number being debugged as well as the calling
function and line number.
If the function is called with parameters, the pertinent ones get
printed via the macro PRINT_VALUE_EQUAL (see Print a variable's
value macro page 290). It prints the parameter's label and its
value such as: count = 27.
The exit debugging macros lists the function's name and line
number, which may be useful when a function contains several
returns. The entry macro has already supplied pertinent information.
On exit, the return value, if any, could also be printed.
void SetMgrID (uint16_t _mgrID, char* _callingFunction, int _line number) {
#ifdef _DEBUG_PERSONNEL_SET_MGR_ID
PRINT_FILE_FUNCTION_LINE_CALLED_BY_NL
PRINT_VALUE_EQUAL_TAB(_mgrID)
#endif // _DEBUG_PERSONNEL_SET_MGR_ID
...
#ifdef _DEBUG_PERSONNEL_SET_MGR_ID
PRINT_FUNCTION_EXIT
#endif // _DEBUG_PERSONNEL_SET_MGR_ID
return;
} // SetMgrID()
The following entry macro prints START, the function's name, line
number, and the corresponding calling function's name and line
number.
#define PRINT_FILE_FUNCTION_LINE_CALLED_BY_NL \
[Link](F("START: ")); \
[Link](__FUNCTION__); [Link](F(":")); [Link](__LINE__); \
[Link](F(" from ")); [Link](_callingFunction); \
[Link](F(":")); [Link](_line number);
Variants of this macro could contain parameters to print the pertinent
function parameters, their label and value. The matching exit macro
will print END, the function's name and exit line number.
#define PRINT_FUNCTION_EXIT \
[Link](F("END: ")); \
[Link](__FUNCTION__); [Link](F(":")); [Link](__LINE__);
Another variant of this macro could print the return value.
Here is a typical debugging output for the SetMgrID function of the
Personnel class. It says that SetMgrID was called at line 143 by the function
ATest at line 253. The parameter _mgrID has a value of 31. The exit macro
prints the return line number 166.
START: SetMgrID:143 from ATest:253
_mgrID = 31
...
END: SetMgrID:166

2.
Print a variable's value macro
 Instead of repeating a [Link] to
<x>Macros[AUX]Parameterized name/value macro simplifies printing a parameter's value</x>

print the variable's name first and its corresponding value next, the
PRINT_VALUE_EQUAL macro uses the C++'s preprocessor stringizing
operator # to print a parameter's label.
#define PRINT_VALUE_EQUAL(var) \
[Link](F(#var)); [Link](F(" = ")); [Link](var);
For example, the following code:
int count = 27;
PRINT_VALUE_EQUAL(count)
yields count = 27. This macro saves a lot of typing. Variants can be
created to customize output formatting, such as start with a tab, do
not do a newline after printing the value, include the variable's
address in the output, and so forth.

3.
Debug control macros
Debug control
<x>Print-based Debugging Framework[AUX]Debug control macros turn debugging on/off global, file, and function level</x>

macros turn debugging on/off at three levels: global, file, and

function.

The global level application-wide debugging control macros

are located in Globals.h. If any of these macros are
commented out, the corresponding global level macros will be
deactivated.

#define GENERAL_REPORTING_ON // Controls file and function level control macros

#define MILESTONE_REPORTING_ON // Milestones are turned on and off
#define CRITICAL_REPORTING_ON // Controls critical reporting macros
#define ERROR_REPORTING_ON // Controls error reporting macros
#define MEMORY_MANAGEMENT_ON // Controls memory-use reporting macros

The file level control macros, located in Globals.h, control

debugging prints on a file-by-file basis. For example,
DEBUG_PERSONNEL_ON will turn on/off prints in functions located in
the file [Link] via the #ifdef on the global control macro.
 #ifdef GENERAL_REPORTING_ON
....
#define _DEBUG_PERSONNEL_ON
...
#endif // GENERAL_REPORTING_ON

The function level control macros, located in individual .cpp

files, control which functions' prints get activated. For example,
the _DEBUG_PERSONNEL_SET_MGR_ID macro will turn on/off prints in
the function SetMgrID().

#ifdef _DEBUG_PERSONNEL_ON
....
#define _DEBUG_PERSONNEL_SET_MGR_ID
...
#endif // _DEBUG_PERSONNEL_ON
In the example below, the programmer wants to debug the function
SetMgrID. To do this, he/she must enable the three debug control
macros (global level, file level and function level):
#define GENERAL_REPORTING_ON // global level - allows debugging info, in Globals.h
#define _DEBUG_PERSONNEL_ON // file level - located in Globals.h allows debugging in
[Link]
#define _DEBUG_PERSONNEL_SET_MGR_ID // debug function SetMgrID in [Link]
The function specific control macros turn debugging inside a
particular function on and off, via appropriate #ifdef...#endif pairs.
void SetMgrID(uint16_t _mgrID, char* _callingFunction, int _line number) {
...
#ifdef _DEBUG_PERSONNEL_SET_MGR_ID
...
#endif // _DEBUG_PERSONNEL_SET_MGR_ID
...
} // SetMgrID()
In the above, the 3-level hierarchy of control macros enables
debugging code inside the SetMgrID function.
Summarizing: Globals.h contains global level and file level
debugging control macros. Each .cpp file will contain function level
debugging control macros to turn on and off debugging specific
functions. This macros-based hierarchical control system enables
fine tuning of what gets printed.
 23.
Misdoings (appendix)

Misdoings (appendix)
[Link] program</x><x>Misdoings[AUX]= instead of == in an if
<x>Misdoings[AUX]Dumb mistakes not detected by compiler; found by Perl

or while or == instead of = in an assignment</x><x>Misdoings[AUX]switch default case missing or empty</x>Misdoings are subtle

programming mistakes which the compiler does not catch because

the ensuing erroneous statement is a valid C construct. The Perl
program [Link] can detect the following five misdoings via
appropriate regexes (see A note on the book's source code page
338):

ifor while - Typically if(x = 0), etc. The intention was if(x == 0).
== instead of = in assignment - Typically x == ٠. The intention
was x = ٠.
Missing default or empty default case in a switch statement - When
writing switch statements, always include a default case with code
in it. This raises two potential misdoings: no default case and
empty default case (no code).
Missing F() macro in [Link] - Most programmers will write plain
prints such as [Link]("Some text"). By not using the F() macro,
they missed out on saving RAM by storing strings in flash
memory.

These misdoings are described in the sections which follow.

1.
= instead of == in if/while
Typing an equal sign = instead == (comparison ) in an if or while
statement or in a logical statement involving && or || is a common
programming mistake. These are usually caught right away however
the errors slip by occasionally which causes the program to bug .
Finding these errors can be extremely time consuming because it
could well not be systematic. It is a glitch. Note that when you do a = b
in an if statement, if b is not zero, the result is true, if b equals zero, the
result is false. Same for a while, it may throw the application into an
infinite loop unless it contained a break statement.
The Perl find misdoings program uses the following regex to detect
if/while statements which contain an equal sign = instead of a
comparison == sign:
/(if\s*\(\s*(\(\s*)?|while\s*\(\s*(\(\s*)?
<x>Regex (examples)[AUX]Regex to find = instead of == in an 'if', or == instead of = in an assignment</x>

|&&\s*(\(\s*)?|\s*\|\|\s*(\(\s*))?\s*\w+\s*=[^=] /
The regex above will find simple bad statements including logical
statements with and without parentheses such as:
if((a = b) || (x = y) ).
Granted, the regex is a little long and difficult to decipher but, taking
things one at a time, and understanding that it is repetitious, the job
gets to be a little easier.
The regex may be broken up into its individual constituents as
follows. It starts by looking at one of two patterns: an if or while with a
parens, or an && or an || without a parens:

The if with a parens is if\s*\(, same with the while: while\s*\(. The
logical statements do not have a mandatory parens, &&\s*,
same with the ||: ||\s*.
It then looks for an optional parens after the if, while, &&, and ||: (\
(\s*))?. So far it found something like if((.
Having found an if or while or a logical operator (&& or ||), it looks
for a name \s*\w+. The regex has processed the line of text up
to if((a.
It ends the process by looking for an equal sign not followed by
another equal sign \s*=[^=].

This regex can be plugged into my Perl programs as an Awk like

rule:
 if ($line =~ / regex above / {
print "************* Warning: = instead of == in if/while/&&/|| ********************";
print "$filename:$lineNo - $line";
} # end rule
After developing this regex, I tried it on my Arduino-based beehive
weighing system application. It found three such errors; it saved me
hours of debugging time.

2.
== instead of = in assignment
Typing the logical operator == instead = (equal sign) in an
assignment statement is another common programming mistake.
You might be comparing a to b as in a==b instead of assigning b to a as
in a=b. This usually happens when doing copy/paste from an if
statement when the variable's name is long. The == can be
inadvertently not changed to =. Again, this is a mistake difficult to find
if not caught immediately. The following regex may be used to detect
such == assignment statements.
/((\w+|\s*)\s*\()?\s*\w+\s*(\[\s*\w+\s*\])?\s*==/
This regex is saying:

((\w+)\s*\()? look for an optional first word surrounded by optional

spaces followed by a parens,
\s*\w+\s* followed by a word surrounded by optional spaces,
(\[\s*\w+\s*\])? followed by an optional index of the sort a [55],
spaces optional,
followed by an equality comparison == .

The rule will test $2, the second open parens. If it contains if/while, the
line is discarded since it is not a mistake because == is part of the
if/while test. This regex is used by my [Link] program inside
a rule:
 if ($line =~ / regex above / {
if ( $2 ne if' && $2 ne 'while' ) {
print "************* Warning: assignment has a bool operator == instead of an assignment =");
print "$filename:$lineNo - $line";
}
} # end rule
After developing this regex, I tried it on my Arduino-based beehive
weighing system application. I was reassured to find that I had no
flaws of this type.

3.
switch statement empty or missing
default case
<x>Misdoings[AUX]switch default case missing or empty</x><x>Bugs (avoiding them)[AUX]switch default case missing or empty</x><x>Error Reporting

A program's robustness depends in part upon

Framework[AUX]switch default case missing or empty</x>

foreseeing and handling potential problems. One of these concerns

switch statements. C does not require that a default case be defined, but
then, what if the switch were trying to handle a situation for which no
case had been defined? How does the application handle this? Which
brings me to define two rules as a matter of good programming
practices when coding a switch statement:

default cases are mandatory.

default cases should do something. Either handle the default
normally as one of the cases or trigger an error stating that this
particular default should never be called because the
programmer may feel that all the possible case values have
been covered.

The lint like Perl [Link] Perl program did indeed find
missing default cases and empty default cases in my beehive weighing
system application.
 4.
F() macro in [Link] missing
<x>Misdoings[AUX]Regex missing F() macro in [Link]</x><x>Macros[AUX]Misdoings: [Link] missing F() macro</x><x>Code size[AUX]RAM used depends on IDE

The F() macro used in [Link] saves RAM space. It

and use of PROGMEM and F() macro</x>

stores non-active print strings in flash memory. [Link] should be

written with the F() macro to save on RAM requirements, such as
// F() macro stores text in flash memory, loads it into RAM temporarily
[Link](F("This print uses the F macro"));
The following rule in the [Link] Perl program detects
prints which do not contain the F() macro.
/\s*Serial\d*\.print(ln)?\s*\(\s*["]/{
<x>Regex (examples)[AUX]Regex missing F() macro in [Link]</x>

print # **************** Missing F macro in [Link] ********************

print FILENAME ":", NR, "Missing 'F' macro in [Link]/ln", $0;
} # end rule
Notice that the regex above contains \d* to cover Serial, Serial1, serial2...,
and (ln)? to cover both print and println. The key to the regex's logic is to
detect a double quote after the first opening parens \(\s*["] behind a
[Link].

5.
Work to do
Regular expressions within the context of a Perl program may be
used to uncover other misdoings. This section is an early start on a
TODO list.

Missing () when calling a function - Using a function's return

value is one of C/C++'s shortcuts, as in x = MyFunction(). But what
if you forget the parentheses and do x = MyFunction? Your
program will fail because you are assigning to x the address of
MyFunction, not its return value. Unfortunately, the compiler will let
it pass - no errors, no warnings. Once the Perl program is able
to build a table of functions, it can cycle through all assignment
statements to uncover such mistakes. I did make this mistake
twice in my beehive weighing system program.
Implement a mechanism so that functions would return their
stack frame memory requirements. See Function stack frame
memory requirements (page 183).
 24.
Memory structure (appendix)

Memory structure (appendix)

<x>Memory (use)[AUX]Understand memory structure to optimize memory use</x><x>Memory Management Framework[AUX]Heap has two parts: fragmented and contiguous

The Memory Management Framework (page 179) relies on

heap</x>

several macros and functions to verify that there is enough memory

to run the application during its entire execution and to verify that
memory has been properly released when the application ends.
These macros and functions are divided into four groups:

Memory use
<x>Memory Management Framework[AUX]Memory use macros provide a snapshot of available memory</x>

macros - Several macros provide a snapshot of memory use

as the application runs. Since these consume little memory,
they may be used anywhere deemed strategic by the
programmer.
Determining memory usage - Available memory, the heap, is
partitioned into two blocks: the bottom block which contains
allocated memory throughout which there may be holes of
available memory (fragmented memory), and the block which
runs from the top of allocated memory to the bottom of the
stack (contiguous memory).
<x>Memory (heap fragmented)[AUX]Has holes from random memory releases</x>Highly fragmented heap - After

the application has been running for a while, there may be little
contiguous memory. Most of the available memory may be
spread out as holes in fragmented memory; the stack will
probably drop down into allocated space thereby creating a
stack overflow (possible crash or, as a minimum, corrupted
memory).
How do free and delete know how much memory to
deallocate - Doing several consecutive mallocs or news reveal
that memory allocation consumes two extra bytes. This info is
 used by delete and free to determine how much memory to
release.

1.
Memory use macros
The memory-use macro
<x>Memory Management Framework[AUX]Memory use macros provide a snapshot of available memory</x>

may be put anywhere within the application to quantify heap space.

#define PRINT_MEM_STATUS(_description, _callingFunction, _lineNo, \
_parentCallingFunction, _parentLineNo) \
{\
[Link](F("*************************************************")); \
[Link](F("Memory status: "));[Link](_description); \
[Link](F("Called from ")); [Link](_callingFunction); \
[Link](F(":")); [Link](_lineNo); \
[Link](F("Parent calling function ")); [Link](_parentCallingFunction); \
[Link](F(":")); [Link](_parentLineNo); \
uint16_t _heapSpace = HeapSpace (); \
PRINT_VALUE_EQUAL(_heapSpace) \
[Link](F("*************************************************")); \
} // PRINT_MEM_STATUS
The following shows a typical PRINT_MEM_STATUS output.
*************************************************
Memory status: Test - at beginning of Aardvark()
Called from AardvarLG:43
Parent calling function setup:99
_heapSpace 7539
*************************************************
Notice the lines of asterisks to catch the eye and get your attention.
Variants of this macro could be envisioned. Long repeat prints of a
single character consume a lot of flash memory. If this were to be a
problem, such prints could be replaced by a for loop inside a function
to print individual characters:
void PrintLongSingleChar(char _charToPrint, uint8_t _numberToPrint) {
for ( int i = 0; i < _numberToPrint; i++ ) {
[Link]((char) _charToPrint);
}
[Link]();
} // PrintLongSingleChar
Instead of printing a long string, call the function and print many
characters sequentially.
PrintLongSingleChar('*', 80)
You could even define a macro to make life easier:
#define PRINT-_80_ASTERISKS ...

2.
Determining memory usage
<x>Memory (use)[AUX]Memory allocation failure avoidable with memory monitoring functions</x><x>Memory (use)[AUX]Total heap is fragmented plus contiguous</x>

<x>Memory (heap contiguous)[AUX]Contiguous heap between top of allocated memory and bottom of stack</x><x>Memory (stack)[AUX]Function nesting and recursion consume

Available RAM starts somewhere in the bottom of the

too much contiguous heap space</x>

microcontroller's address space and extends up to the top. The stack

grows downwards as dangling stack frames into this space.
Allocated memory (malloc and new) grows upward into this space. The
heap is the part of available RAM not used by the stack. During
program execution, allocated space gets created and released
randomly, thereby leaving holes. The consequence is that the heap
contains two parts: fragmented heap space and contiguous heap
space above it. The problem underlying fragmented heap space
relates to the hole sizes. If an allocation request requires heap space
small enough such that a hole can be taken advantage of, the
system will allocate memory from the hole, thereby making the hole
smaller. On the other hand, if the memory request is larger than the
biggest hole, allocation will be obtained from the contiguous space
area. After a while, there will not be any significant contiguous heap
space left and holes will get progressively smaller, unless a large
block of memory gets released. Given such a situation, if the holes
are small, available heap space, although possibly significant in
absolute terms, may not be of much use.
<x>Memory (use)[AUX]__flp (holes) and __brkval (no holes) address of start of heap</x><x>Memory (heap fragmented)[AUX]__flp (holes) and __brkval (no holes) address of start

of heap</x><x>__flp[AUX]Address of begin heap when there are holes</x><x>__brkval[AUX]Address of begin heap instead of __flp when there are no

Thankfully, the system tracks these holes. As a result, it is

holes</x>

possible to track both fragmented and contiguous heap space The

beginning of heap space is defined by one of two system variables.

__brkval is the address of the beginning of heap space if there

 are no holes, the beginning of contiguous heap space if there
are holes.
__flp is the address of the beginning of heap space if there were
holes. If __flp equals 0, there are no holes, all of available heap
space is contiguous.
The system keeps a linked list of available holes via the FreeList
structure (see below).

The bottom of stack is found via a

<x>Memory (stack)[AUX]Bottom of stack found via programming trick</x>

programming trick which consists in defining and calling the very

small function BottomOfStack which returns the address of its one and
only local variable, as follows:
uint16_t BottomOfStack () {
uint8_t localVariable{0};
return (uint16_t) &localVariable + 3 + FUNCTION_CALL_OVERHEAD;
} // BottomOfStack
The address of the
<x>Memory (stack)[AUX]Stack frame structure and size</x><x>Stack frame[AUX]Determining stack frame size</x>

local variable is incremented by the size of the stack frame, which

yields the address of the bottom of the stack. See Function stack
frame memory requirements (page 183).
<x>Memory (heap fragmented)[AUX]Largest possible allocation: either largest hole or contiguous memory</x><x>Memory (heap total)[AUX]Total heap = fragmented plus

Two functions, FragmentedHeap and ContiguousHeap,

contiguous heap; reveals total available memory</x>

provide TotalHeap, one of the values the application should monitor.

Since the function FragmentedHeap travels the holes to get the total
fragmented heap, it also determines the size of the largest hole. This
is the second value which should be monitored.
Contiguous heap is the space between the bottom of the stack and
the top of allocated space:
uint16_t ContiguousHeap() {
return = BottomOfStack () - __brkval;
} // ContiguousHeap
Fragmented heap space uses the address of the first hole __flp to
travel up the holes linked list by using the structure:
// free list structure as maintained by the avr-libc memory allocation routines
struct FreeList {
size_t size;
struct FreeList* next;
}; // FreeList
It becomes a simple matter to start with the first hole and travel up
the list until there are no more holes. The ContiguousHeap function does
this. As it travels up the holes list, this function finds the largest hole.
If contiguous heap space is larger than the largest hole found,
LargestPossibleAlloc is set to it.
uint16_t FragmentedHeap(uint16_t &_largestPossibleAlloc) {
// Starts at __flp and moves up the free list
// returns total fragmented heap
// Finds largest hole or contiguous heap space if larger than the largest hole
uint16_t* allocatedMemory{0};
uint16_t totalFragmentedHeap{0};
FreeList* currentHole{__flp};
// Start by setting largest hole equal to contiguous heap space
_largestPossibleAlloc = ContiguousHeap();
if (__flp == 0) {
return 0;
}
while(true) {
// Break if at end of list
if (currentHole == 0) {
break;
} // if
totalFragmentedHeap += currentHole->size;
_largestPossibleAlloc = currentHole->size > _largestPossibleAlloc
? currentHole->size : largestPossibleAlloc;
currentHole = currentHole->next;
} // while
return totalFragmentedHeap;
} // FragmentedHeap
The total heap function TotalHeap merrily adds fragmented heap to
contiguous heap:
uint16_t TotalHeap () {
return FragmentedHeap() + ContiguousHeap();
} // TotalHeap

3.
Highly fragmented heap
The problem with the function
<x>Memory (heap fragmented)[AUX]Small holes may collectively yield a deceptively large heap</x>

TotalHeap in the previous section is that it includes fragmented heap

space. It could well yield a size which appears comfortable for the
application's requirements; however, the heap could be very
fragmented with no significant contiguous heap space, i.e., full of
small holes which collectively yield a deceptively large size.
<x>Memory (heap fragmented)[AUX]FragmentedHeap function monitors extent of memory holes</x>The function FragmentedHeap is

designed to yield the sum total of holes and the size of the largest
possible allocation. Most of the time the largest possible allocation
will be the size of contiguous heap space, but after a while, it may
start getting small since it is reporting the size of the largest hole. At
this point, the programmer might decide to review his/her code.
<x>Memory (heap fragmented)[AUX]Fragmentation ratio: fragmented/total heap</x>The fragmentation ratio fragRatio,

defined as the ratio between FragmentedHeap and TotalHeap, should be

monitored.
uint8_t fragRatio = (uint8_t) ((float)FragmentedHeap()/(float)TotalHeap()*100.0);
Determining whether the heap is seriously fragmented can be useful
in applications which do lots of allocations followed by releases. The
ratio between fragmented heap and total heap (free RAM) is a
measure of the fragmentation's severity expressed in % - a ratio of
100% indicates that all available memory is fragmented. As the
application unravels, fragmentation will increase; the ratio will start
increasing. If the ratio were to rise to unacceptable levels, the
programmer would have to rethink his/her application.
One of my TODO list items is to devise a heap management
framework to undertake allocations via an overloaded new. It would
allocate memory from a reserved pool and defragment automatically.
This framework could be based on the DataGroup Framework (page
164).

4.
How do free and delete know how much
memory to deallocate
and new need a value for the malloc
<x>Memory (use)[AUX]malloc and new need how much memory to allocate</x>

quantity of memory to allocate. However, when doing a free or delete,

this information is not provided. The question is: How do free and new
know how much memory to release? The programmer provides the
address of the memory allocation from which to release memory;
but, he/she does not provide how much memory to release.
<x>Memory (use)[AUX]Allocations add 2 bytes for allocation size</x>My Internet search has revealed that an

additional word is appended to the allocated space whenever a new

or malloc is called. This extra word contains the number of bytes
allocated and possibly a flag which indicates whether the block of
memory is currently allocated or available. Under such
circumstances, since the address of the beginning of the allocated
block is passed to malloc or new, the deallocation algorithm would get
the size of memory to deallocate from the additional data tacked
onto the allocated space referred to above. I have not pursued the
matter further. If you are interested in this subject, look at:
[Link]
It contains a detailed analysis on identifying heap memory blocks
and interesting links to related material.
 25.
Awk (appendix)

Awk (appendix)
The following sections contain various useful Awk (page 204)
specific details.

Download Awk (next section) covers downloading it from the

[Link]/software/gawk.
Invoking Awk from a DOS box - [Link] (this page just below)
- Awk is a command line application. You will find information
on invoking Awk with its command line arguments, namely
specifying an Awk program and passing file names.
Awk short example (page 303) - A small application which
uses grocery sales data to generate a report illustrates some
of Awk's features.
Awk some features illustrated (page 304) - Awk resembles C
but there are differences. Awk peculiarities are presented and
illustrated in a program.
Awk reference (page 309) - This section details important
features of the Awk programming language.

1.
Download Awk
To download Awk, go to Source Forge
<x>Awk (general)[AUX]Download Awk, go to SourceForge</x>

([Link] It proposes a free download of the

Software Foundation's Awk. The instructions detail how to set it up.
In my case (Windows 10), I downloaded the 64-bit Windows
implementation: [Link]. Run it with the defaults.
Awk will be installed in a C:\Program Files x86\GnuWin32 directory.
Having installed the program, copy/paste
C:\Windows\System32\[Link] (DOS box) into the directory where
you will be running Awk. Test it by typing Awk in the command box. It
should respond with text informing you how to get going with Awk.

2.
Invoking Awk from a DOS box
Invoking an Awk program requires that you
<x>Awk (command line)[AUX]-Wlint flag displays errors</x>

first open a command box (C:\Windows\System32\[Link] aka

DOS box). Once inside the DOS box, you can invoke Awk directly by
typing Awk or by calling a batch file (.bat file), which does the
invoking. The command could be a one liner Awk program (enclosed
in single quotes) directly typed such as:
Awk -Wlint '$1 == "#define"' file1 file2 ... > [Link]
or a complete structured program contained in an .awk file, in which
case the command would be:
<x>Awk (command line)[AUX]Command line: Awk -Wlint -f [Link] file1 file2 ... > [Link]</x>Awk -Wlint -f

[Link] file1 file2 ... > [Link]

or you can invoke a batch file [Link] which contains the
above (see DOS batch files page 241). The file [Link] may
contain hundreds of lines. It is the Awk program.
In the above:

Awk invokes [Link] which is the program you downloaded.

Just type it from inside the DOS box.
-Wlint - When you program in Awk, you will make mistakes.
Many of these would have been caught by the compiler had it
been a C/C++ program; not so in Awk. A new variable having
been created due to a spelling mistake will pass silently
because it is a legal Awk expression. The -Wlint command line
option ensures that many such mistakes be detected and
reported as warnings. Note that this option must be placed
before the -f [Link] option. Having said this, beware;
do not rely too much on -Wlint since the Awk interpreter is not
as good as a C++ compiler at detecting programming errors.
' $1 == "#define" ' is a complete Awk program, a one liner. It
may be typed in directly in the DOS box as long as it is
enclosed in single quotes. This one liner says: print the line if
the first word in the text is "#define". Please note that the Awk
program is enclosed in single quotes; the string is enclosed in
double quotes.
-f [Link] is a multi-line Awk program you created. It
is a text file containing Awk statements which you have stored
somewhere and given it an .awk extension.
file1 file2 file3... are the files which Awk will concatenate and
process as if it were one continuous file. Filenames should
include a complete path. I generally create a directory named
Awk inside the directory which contains my project files (.ino,
.h, .cpp). I prefix file1, file2 file3... with the relative path (..\). For
example, ..\BasicClass.h (here ..\ means directory single-level
up, i.e., backtrack one level towards the root directory).
Awk's print outputs text directly to the console, in the same
way that a C++ [Link] does. You can capture this output by
redirecting it to a file thus: > [Link]. The > symbol means
redirect the output to somewhere else, erase whatever is in
there, and insert the Awk output. The alternate symbol >>
means append to whatever is already there. [Link] is the
destination of Awk's redirection.
<x>Awk (rules)[AUX]Multiple file processing passes possible</x>Multiple passes file processing - You

may want to process a file twice, a first pass to gather info

followed by a second pass whose logic is based on info
gathered from the first pass. The problem with Awk is that
once it finishes processing the lines, it has finished and you
cannot direct it to start over. For example: you may want to
create a table of values with a first pass and then process the
file using the table to accomplish something. Awk cannot do
this, Perl can. If you really want to do a second pass with Awk,
you can trick Awk by twice feeding it the file you want to
process:
 Awk -Wlint -f [Link] fileA fileA > [Link]

The Awk program will be working on two identical sets of lines

tacked together. Detecting when Awk has finished the first pass
relies on watching Awk's ARGIND variable; ARGIND changes from 1 to
2 when it starts processing the second set of lines; it does not care
that the files are identical. All you need do is watch ARGIND's value
to determine when to switch from pass 1 or pass 2.

3.
Awk short example
This small example illustrates a complete
<x>Awk (examples)[AUX]Short database example Awk program</x>

Awk program. It produces a weekly sales report based on data

contained in a text file.
<x>Awk (examples)[AUX]Short database example Awk program</x><x>Tables[AUX]Awk - short database example Awk program</x>Suppose you

have a text file which contains all sales for the week. The data in
[Link] is structured as follows:
Product Supplie Sales Purchase Quantity
r price Price Sold
$1 $2 $3 $4 $5
Peanut MyVegan $5.90 $4.75 10
butter
Peanut BodyFit $10.50 $8.50 5
butter
Yogurt plain Yoplait $0.80 $0.65 30
Yogurt plain Dannon $0.85 $0.65 25
Yogurt Chobi $1.10 $0.90 0
Greek
Table 25.1 - Weekly grocery sales
You want to list the total sales and profit of each product, one at a
time: supplier first, product, sales for the week and margin generated
for the week. You also want totals.
In this example the condition is sales greater than 0 ($5 > 0). The
corresponding action-block is a series of prints and the multiplication
of two fields whose result is added to the global variables totalSales and
totalMargin. You need a BEGIN block to initialize things, a set of rules (in
this case only one rule) and an END block to wrap things up. There
are no functions.
BEGIN {
print "Total sales for the week Sales Margins";
totalSales = 0;
totalMargin = 0;
OFS = ""; # avoid inserting spaces when prints contain commas
}
END{
print "Sales and margin for the week\t$", totalSales, " \t$", totalMargin ;
}
This is our one and only rule: condition { action-block}:
$5 > 0 && $1 != "#" {
sales = $3*$5;
totalSales += sales;
margin = ($3 - $4 ) * $5; # (sales price - purchase price) * quantity sold
totalMargin += margin;
print " ", $2, " - ", $1, "\t", "$", sales, "\t", "$", margin;
}
Note that $1 != "#" in the rule above is there to prevent comments from
being processed.
The Awk program can be invoked directly in a DOS box:
Awk -Wlint -f [Link] [Link] >
[Link]
The output, contained in [Link], will be
Total sales for the week Sales Margins
MyVegan Peanut butter $47.50 $11.50
BodyFit Peanut butter $42.50 $10.00
Yoplait Yogurt plain $19.50 $4.50
Dannon Yogurt plain $16.25 $5.00
Sales and margin for the week $125.75 $31.00
That is it. There is not much to it and the work is done easily, quickly.
Awk can nevertheless do much more. But, if you are contemplating
learning Awk, look at Perl (page 217) first; it can do everything Awk
can do, and much more.

4.
Awk some features illustrated
The following sections contain code segments to illustrate a number
of Awk specificities.

Awk scope (next section) - In Awk, all variables are global,

except for function parameters.
Awk strings (page 315) - Awk proposes somewhat more
powerful string handling than C does. Concatenation in
particular is similar to C's but there are a few subtle differences
and enhancements.
Awk one-dimensional arrays - Code samples show some of
the specificities of array processing. It is quite different from
C's way of using arrays. See Awk arrays (page ٣٠٥) and Awk
one-dimensional arrays (page ٣٠٧) .
Awk two-dimensional arrays (page ٣٠٨) - Awk
multidimensional arrays must be thought of as being flattened
to one dimensional access via the concatenation of the array's
indices.
Awk condition only and action-block only rules (page 311) - A
rule without a condition prints the line. This feature can be
useful to print all the lines of a file after doing something on
each line.

A rule without an action block prints the line if the condition

evaluates to true. This can be useful to print all uncommented lines
in a file

1.
Awk scope
Awk variables are global except
<x>Awk (variables and functions)[AUX]Function variables global, parameters local</x>

when used as parameters. For example, the two variables of FunctionA

below (i and a) are global even though they are first used inside a
function. Remember, Awk variables are not declared and are no
typed.
 function FunctionA() {
print "Start FunctionA";
i = 5; a = 10;
print "i =", i; print "a =", a, "\n";
} # FunctionA
In FunctionB below, i becomes a local variable of the function because it
is a parameter. The variable a remains global.
function FunctionB( i ) {
print "Start FunctionB";
print "i =", i; print "a =", a, "\n";
} # FunctionB
In FunctionC below, both i and a are global.
function FunctionC( ) {
print "Start FunctionC ";
print "i = ", i; print "a = ", a, "\n";
} # FunctionC
This piece of code tests the three functions above. It is inserted in
the BEGIN block:
...
FunctionA();
x = 99;
FunctionB(x);
FunctionC();
...
These are the results. FunctionA initializes i and a.
Start FunctionA
i=5
a = 10
A function's parameter makes the corresponding variable local to the
function. FunctionB takes a parameter which is assigned to the local
variable i.
Start FunctionB
i = 99
a = 10
Had you used the -Wlint command line option, Awk would have
reported: i shadows global i.
FunctionC prints i and a, despite i's being locally used as a parameter in
FunctionB. i and a remain global in FunctionC. They retain the values set
initially in FunctionA.
Start FunctionC
i=5
a = 10
The above illustrates how variables are global despite their being
first used inside a function (FunctionA), and how function parameters
are local to a function (FunctionB) despite their names being the same
as global variables.

2.
Awk arrays
If you are a C/C++ programmer, you
<x>Awk (arrays)[AUX]Arrays are associative, mix strings and numeric values</x>

must do a clean mental swipe of what you know about arrays. Awk
arrays are different in that they are associative. Indices are not
necessarily numeric. They can also be strings. An Awk array index is
a key into a pseudo-database which contains records, as opposed to
array items in C which are accessed sequentially via a numeric
index.

<x>Awk (arrays)[AUX]Multi-dimensional arrays supported</x> An Awk one-dimensional array item is

written as in C: MyArray[a]. An Awk two-dimensional array is
written differently: MyArray[a,b].
<x>Awk (arrays)[AUX]Array indices can be integer, float, strings, similar to key into database</x>Awk arrays are not

contiguous data spaces as in C. They are pseudo-databases

accessed via a key, the array key being the concatenation of
the indices separated by a character defined in the built-in
variable SUBSEP. This means that Awk arrays, behind the
scenes, are one dimensional, but they can be operated on as if
they were multi-dimensional. Whereas C array indices are
integers, Awk array indices can be anything: integers, floats,
strings. Indices inside an array may be of mixed type. Part of the
index could be numeric; another part could be a string.
At the most basic level an Awk array looks exactly like a C
array:
n = 10;
names[n] = "Sasha Gomez";

Even though a number is used to point to an array item such as in

names[n], this index does not point to the 10th item in the array. It
points to the item whose key is "10". Nevertheless, it remains
possible to simulate access to sequential items in a for loop such as
for( i=1; i<=10; i++) { print(myArray[i]); }
The for loop will operate as
<x>Awk (caveats)[AUX]Cycling through array in C-style for loop may create unwanted array items</x>

expected. Array items which contain string indices will be ignored.

You must be careful when doing this, though. Assuming all you
have done is define one array item such as name[10] above (the array
contains one item only) and do a standard for loop over 10 items,
Awk will create nine undefined array items.

Traversing an array may be

<x>Awk (arrays)[AUX]Loop construct for (idx in array) facilitates array traversal</x>

better accomplished by the special array specific Awk loop

construct for (idx in array) { ... }. It resembles a foreach Perl construct
except that the array items must be accessed via the index
variable. The loop will be iterated N times where N is the
number of items in the array. On each iteration, idx will contain
the value of the individual array index. Traversing the entire
array may be accomplished by doing:
for ( idx in myArray) { print "idx = ", idx, " - myArray[idx] = ", myArray[idx]; }

This feature is presented in Awk some features illustrated (page

304).

Traversing multidimensional arrays

<x>Awk (arrays)[AUX]Using multidimensional arrays is complex</x>

gets a little complex. Awk concatenates multidimensional

indices into one index whose components are separated from
each other via the built-in SUBSEP variable which by default is
\034 (decimal 28, the FS file separator character). Since this
default separator is not printable, you might want to change it
to @ (SUBSEP="@") or something else. When printing the
concatenated index, you will get something like Name@City. See
the section Awk some features illustrated (page 304) above.
<x>Awk (arrays)[AUX]delete removes array items</x>Removing an array item is possible with

the delete operator. When doing:

delete myArray[10];
 Awk will remove the item in myArray accessed by the key 10. myArray[9]
and myArray[11] may well still exist. You are not setting myArray[10] to
zero; you are physically removing it.

The length function may be

<x>Awk (arrays)[AUX]length function yields number of items in one-dimensional arrays</x>

used to get the number of items in an array:

myArrayLength = length(myArray);

If the array were to be multidimensional, the length function yields

the total number of items because, behind the curtain, a
multidimensional array is a one-dimensional array whose indices
have been concatenated into one index with an index separator
defined in the built-in variable SUBSEP. Getting the number of items
in each subarray gets a little complex (Awk two-dimensional arrays
page 308). You would have to count the number of different indices
at each index level.

1.
Awk one-dimensional arrays
Awk supports the one- and multi-dimensional arrays concept. Since
two-dimensional arrays differ substantially from one-dimensional
arrays, these are covered separately.
<x>Awk (arrays)[AUX]Arrays defined when array items used</x>The following program segment illustrates

one dimensional arrays. Start by defining four array elements. Notice

that there is no declaration to specify that arrayIDX is an array. The
array has four items: two are indexed via numbers, the other two via
strings.
print "Begin array to test for loops";
arrayIDX[2] = 99;
arrayIDX["firstN"] = "Sasha";
arrayIDX["lastN"] = "Gomez";
arrayIDX[1] = 98;
Getting the number of items in an
<x>Awk (arrays)[AUX]length function yields number of items in one-dimensional arrays</x>

array is accomplished via the length function, such as in the following:

arrayIDXLength = length(arrayStr);
print " arrayIDXLength = ", arrayIDXLength;
The result is 4 as expected since four items were defined. Note that it
yields the number of items, not the array memory size such as the
C's sizeof operator does.
arrayIDXLength = 4
Now take a look at what happens when doing a traditional for loop on
five items (this is a gotcha - read on). Remember that the array has
four items, two being indexed by a number, the other two by strings:
for ( i = 1; i <= 5; i++ ) {
print "\ti = ", i, " arrayIDX[i] = ", arrayIDX[i];
}
It is interesting to note that only the items with numeric indices got
printed, the other two, with string indices, failed to get printed. This
corroborates the fact that array indices are keys. They are not
positions relative to the head of the array. Here is the print result.
i = 1 arrayIDX[i] = 98
i = 2 arrayIDX[i] = 99
i = 3 arrayIDX[i] =
i = 4 arrayIDX[i] =
i = 5 arrayIDX[i] =
After doing the for loop, get the array's length. The result is:
arrayIDXLength length = 7
@#$µ% Awk created additional array elements when trying to print
inexistent array items: indices 3, 4 and 5. This is one of Awk's traps
and pitfalls. Now traverse the entire array via Awk's special for array
traversal construct:
for ( idx in arrayIDX) {
print "\tindex = ", idx, " - arrayIDX[idx] = ", arrayIDX[idx];
}
The result is a jumble:
index = 4 arrayIDX[idx] =
index = firstN arrayIDX[idx] = Sasha
index = 5 arrayIDX[idx] =
index = lastN arrayIDX[idx] = Gomez
index = 1 arrayIDX[idx] = 98
index = 2 arrayIDX[idx] = 99
index = 3 arrayIDX[idx] =
Notice how the unexpectedly created empty items show up as
indices 3, 4 and 5 when doing the print arrayIDX[idx] in the for loop and
that their values are undefined.
Also notice how there is no particular order. Why is index = 4 printed
first? Such unexpected behavior will push C++ programmers over
the edge.
 2.
Awk two-dimensional arrays
The
<x>Awk (arrays)[AUX]Arrays are associative, mix strings and numeric values</x><x>Awk (arrays)[AUX]Arrays defined when array items used</x>

following two-dimensional array is a table of rows and columns which

contain data on contacts. The left most index (row) contains the
contacts' IDs. The right-most index (columns) contains several float
and string data items specific to one contact. Since Awk arrays are
associative, you can mix numeric data items with string data items.
Furthermore, indices can be anything: numeric (integers and floats)
and strings. In C/C++, you would have to create arrays of structs to
accomplish this; indices would have to be integers.
Getting started with Awk arrays is disconcerting as you do not go
through the process of declaring an array before using it as you
would need to do in C/C++. You are free to start defining individual
array items such as the following contact data for an individual:
#Data for Sasha Gomez
<x>Awk (arrays)[AUX]Two-dimensional array example</x><x>Awk (examples)[AUX]Two-dimensional array example</x>

myContacts[1067, "lastName"] = "Gomez";

myContacts[1067, "firstName"] = "Sasha";
myContacts[1067, "street"] = "41 Sumner street";
myContacts[1067, "city"] = "Freeport";
myContacts[1067, "zip"] = "04032";
Printing the array's content with the specialized Awk Array traversal
for loop:
for ( idx in myContacts ) {
print "Index =", idx, "\t", myContacts[idx];
}
yields:
Index 1067@city Portland
Index 1067@firstName Sasha
Index 1067@ZIP 04032
Index 1067@street 41 Sumner street
Index 1067@lastName Gomez
The print output is surprising in that there does not seem to be any
particular order.
If you wanted to pull out all of Sasha Gomez's data, you would do a
search with the for traversal loop and select items whose first index is
1067 using the index(str, find) function:
 for (idx in myContacts) {
if (index (1067, idx) != 0) {
print "Index =", idx, "\t", myContacts[idx];
} # if
} # for
Transforming multiple indices into one index and handling the array
as if it were a one-dimensional array raises interesting possibilities.
Indices are assembled from their component parts and joined with
the Index separator SUBSEP inside a variable. Example:
myContacts[1067, job] = "Manager";
can be replaced with
myContacts[1067@job] = "Manager";
This example illustrates another interesting feature of Awk
multidimensional arrays. A two-dimensional array may be visualized
as being a spread sheet, rows and columns. The total cell size is the
number of rows multiplied by the number of columns. A C struct-
based array would have the rows times columns number of items.
Not so with Awk, add to the contact array above a new value such as
myContacts["1067@note200606"] = "Sasha turned his/her report in 3 days early";
Were you to do this in C, you would have to add a new column and
add as many new pointers as there are rows. In Awk you are adding
one item only. An added benefit is that you could pull out all notes
dated note200606 (yymmdd).

3.
Awk reference
You should be able to create fairly complex Awk programs based on
Awk's main features described below.

The Awk command line (page 310) should include, as a

minimum, the Awk program's name, the -w warnings switch,
and a list of files (full or relative paths).
Awk comments (page 310) - The Awk comment symbol is the
pound sign #. It is a C++ like // comment in that everything after
the # to the end of the line is ignored.
Awk rules (appendix) (page 310) - Awk processes a group of
text files as one continuous set of text lines. A user-defined set
of rules (condition/action-block pairs) are applied on the lines,
one line at a time. If a rule's condition evaluates to true, the
rule's action-block, a C-like function body, is executed.
Awk BEGIN and END (appendix) (page 312) - Awk provides
two function skeletons which are executed once: the BEGIN
function before processing the lines of text of all of the files,
and the END function, after processing all the lines of text. It is
up to the programmer to define what BEGIN and END do. See
Awk short example (page 303).
Awk functions and variables (appendix) (page 312) have a
name, can take parameters, contain statements in a function
body, and may return a value. Awk does not have function
declarations.
Awk functions and variables (appendix) (page 312) - Variables
are neither declared nor typed. They are created on the fly and
may contain numeric or string values.
Awk built-in variables (page 313) and built-in functions - Awk
provides a rich set of built-in variables and functions
specifically tailored to do line by line text processing. It also
provides string processing functions, math functions, and
more.
Awk scope (page 304) - Variables are always global, even
when first used inside a function. The exception is function
parameters, they are local to the function and may shadow
global variables if their spelling is the same as a global
variable.
Awk program flow control (page 314) - The standard C flow
control constructs if, while, for, do...while, switch are supported.
Awk operators (page 314) - Most of the C operators are
supported.
Awk arrays (page 305) - Awk arrays are associative. The best
analogy I came up with is to consider an Awk array to be a
table of values accessed via a key. An Awk specific for loop
 enables traversing the array. The standard C-style index-based
for loop is also supported, but it can generate unwanted side
effects.
Awk strings (page 315) - Awk does not differentiate characters
from strings the way C does. An Awk character is a 1-byte long
string such as "A". Awk strings may be delimited by single or
double quotes.
Awk print (page 315) - The Awk print statement resembles the
C++ [Link] statement. It is, however, provided with features
which grant it greater flexibility than its C++ counterpart. The
C-like printf is supported.
Basic Awk mathematical functions (page 316) are provided.

4.
Awk command line
Invoke Awk from a DOS box
<x>Awk (rules)[AUX]Invoke Awk from DOS box via command: program name, -w option, file names</x>

via a command which includes:

Program name such as -[Link].

-w option to trigger warnings.
file names (full or relative paths).

5.
Awk comments
The # sign
<x>Awk (rules)[AUX]Comment is # sign; down to end of line</x><x>Comments[AUX]Awk/Perl comment symbol is pound sign #</x>

signals that the rest of the line is a comment. Comments in Awk

operate just as C++ // comments do. Everything from the comment
symbol (#) to the end of the line is ignored. The C style comment /*...*/
is not supported.
 6.
Awk rules (appendix)
Though mentioned earlier, it is
<x>Awk (rules)[AUX]Condition based on pattern search: simple logical statements or regexes</x>

worth repeating. Awk is a text processing utility which operates on

lines of plain text files. It reads one line at a time and does
something to it if the line contains text which matches a pattern. The
combination of pattern matching (condition) with work done (action-
block) is referred to as a rule. A condition is a logical statement or a
regular expression (regex) or a combination thereof. A logical
statement is a sequence of one or more logical statements joined
together by the AND (&&) and OR (||) operators. A regex is a sequence
of characters enclosed in two forward slashes which define a text
match pattern (see Regular expressions (regex) page 191). If the
condition finds a match in the line of text, the corresponding action-
block is invoked. An action-block resembles a function body - it is a
sequence of C-like statements enclosed in curly braces.
<x>Awk (rules)[AUX]Rules use fields $ ٢ $ ,١, etc. from line tokenization</x>Whatever the nature of the condition

(logical statement or regex), when Awk loads a new line of text, it

breaks it up into fields which it assigns to the built-in variables $1, $2,
etc. The built-in variable $0 contains the entire line.
Searching for a #define macro is a good example; just do $1 == "#define".
When it finds a line whose first field is #define, it invokes the
corresponding action-block. Commented out #defines will not be found
because the condition would fail since $1 would contain // or //#define. If
you want to determine whether the macro is a commented macro,
the corresponding action-block would need to test whether a
comment // precedes a #define or not. To find commented and non-
commented #defines do:
$1 == "#define" || $1 == "//#define" || ( $1 == "//" && $2 == "#define" )
These logical statements handle commented and uncommented
#defines, but if you want to also handle #endif, #if, #else, #error, each of
which could be commented or not, you would need a total of 15
logical statements grouped by the OR || operator.
Using a regex is a lot simpler and less error prone:
 /^\s*(//)?s*#
<x>Awk (regex)[AUX]Find macros regex: #define, #ifdef, etc.</x><x>Awk (examples)[AUX]Find macros regex: #define, #ifdef, etc.</x>

(define|if|endif|else|error)/
As evidenced by this example, pattern matching with a regex is
much more concise than using logical statements. Its use may be
preferable since it brings flexibility which logical statements do not
have. The regex above accomplishes the same task as the 15
logical statements!
I admit that when first encountered, regexes are incredibly cryptic. It
is a question of learning the vocabulary. Once you are familiar with it,
they are easy to read since they are a sequence of character
searches. For example, \s translates as space, * means zero or more
times. The regex above says: start at the beginning of the line ^,
followed by zero or more spaces \s*, followed by an optional C++
comment (//)?, followed by zero or more spaces \s*, followed by a #
sign, followed by the six letters define, or the two letters if... Notice the
parentheses around // and the question mark. It is saying: look for
two consecutive forward slashes //, however, these are optional. This
means that the regex does not care whether there are forward
slashes before the pound sign #.

7.
Awk condition only and action-block
only rules
I have said it before, but it is worth repeating: Rules are
condition/action-block constructs. This being said, you may create
condition only rules and action-block only rules. Condition only rules
will print the line if the condition evaluates to true. Action only rules,
i.e., rules without conditions, will be applied to all lines.
<x>Awk (rules)[AUX]Condition only rule prints line if condition successful</x>Here is a simple useful action-block

only rule. It prints every line with the filename and line number as a
comment in front of the line:
{ print "/* ", ARGV[ARGIND], ":", NRF, " */ ", $0; }
The output will be
/* MyFile:348 */ this is the line of text
<x>Awk (built-in variables and functions)[AUX]FILENAME is file being processed, same as (ARGV[ARGIND])</x><x>Awk (rules)[AUX]No condition rule is always true -
 The inverse, a condition only rule, just prints the line if the
action-block invoked</x>

condition evaluates to true. It can be used to generate a subset of a

file, such as commented lines only or a file of non-commented lines.

8.
Awk interrupt processing lines
As rules
<x>Awk (rules)[AUX]getline, next stops processing line</x><x>Awk (built-in variables and functions)[AUX]next, getline interrupt rules processing</x>

get processed, it is possible to interrupt processing and load the next

line. Two built-in functions may be used to do this:

next - This keyword stops immediate processing of the current

line, gets the next line, and starts again with the topmost rule.
It operates just as the continue statement in a for loop does.
getline - This keyword, when used without parameters, is similar
to next, except that it loads the next line and continues
processing. It does not start again with the top rule.

9.
Awk BEGIN and END (appendix)
Awk supports two predefined
<x>Awk (rules)[AUX]BEGIN and END blocks do work at start and end of program</x>

function skeletons: BEGIN and END. BEGIN is invoked before any line of
text gets processed. END is invoked after all the lines of text from all
the files have been processed. This enables the Awk programmer to
undertake housekeeping when the program starts and wrap-up when
the program ends.

BEGIN initializesvariables to make code more maintainable,

sets built-in variables, does necessary program header prints,
and undertakes any other work deemed necessary before
processing lines.
END prints results from overall processing, such as totals, and
 finishes with clear messages such as "End of Program". It may also
be used to generate output based on tables of data acquired
during the rules processing.

Both BEGIN and END are optional and codeless. It is up to the

programmer to decide whether to use them and how.

10.
Awk functions and variables (appendix)
<x>Awk (Awk vs. C)[AUX]Origins trace back to Unix, hence based on the C language</x><x>Awk (variables and functions)[AUX]Functions similar to C functions, may return

Awk functions are practically identical to C functions. They

value of any type</x>

have a name, may take parameters, and may return a value just like
in C. The parameters are enclosed in parentheses, the body in curly
braces. There is no such thing as a function declaration. Functions
are defined first and used afterwards.

The function keyword precedes the function's name.

Functions may take parameters itemized in an optional
parameter list enclosed in mandatory parentheses.
There cannot be a space between the function's name and the
opening parens.
A function can return a value via the keyword return, but it does
not have a specific return type.

The structure of an Awk function is:

function MyFunction( parameter list ) {
statement
statement
...
return something; # optional, could be anywhere in the function block
}
A return
<x>Awk (variables and functions)[AUX]-Wlint flag warns nonexistent return</x><x>-Wlint flag[AUX]Awk - use -Wlint command line flag to display errors</x>

statement is optional, however, if omitted and the calling function

assumes that it did return something, there is no error message, and
the calling function will receive garbage. If the -Wlint command line
option had been used, there might be a warning.
<x>Awk (variables and functions)[AUX]Variables created on the fly when first used</x>Awk variables are created on the

fly when first used. They are not typed. They may contain floats and
strings. Assigning a value to a variable does not bind it to a specific
type. A variable may start out containing a float and later be assigned
a string or vice versa. Whether first used inside a function or globally,
variables are always global except for parameters which are local to
the function. See the section Awk scope (page 304) below.
<x>-Wlint flag[AUX]Awk - use -Wlint command line flag to display errors</x><x>Awk (caveats)[AUX]Program particularly sensitive to spelling errors, no

Note that because variables are not declared, Awk is

warnings</x>

particularly sensitive to spelling errors. For example, suppose you

define the variable errorCount and further down you write errorsCount. You
have unknowingly created a second variable. The application will fail
to run properly. You may run your Awk program with the -Wlint
option. It may report errorCount or errorsCount as being an
uninitialized/unused variable but, if you persist and use the
misspelled variable, there will be no warning and you will be in for an
unwelcome debugging stint. Do not rely on Awk to warn you of such
mistakes.

11.
Awk built-in variables
Awk provides built-in variables. Here are a few common ones (see
the Awk official reference in the Bibliography - Awk (page 334) for a
complete list).

ARGC, ARGV,
<x>Awk (built-in variables and functions)[AUX]Command line variables particularly useful: ARGC, ARGV, ARGIND</x>

ARGIND - A complete C/C++ program is a function whose name

is main:
int main(argc, argv) {}

argcis the number of command line arguments. argv is an array of

command line arguments including the program's name as the first
item.
Since Awk is a C program, it has a main function with its
corresponding argc and argv arguments. When invoking it, you do:
Awk -Wlint -f [Link] file1 file2...
<x>Awk (command line)[AUX]Contains flags, program, files to process</x>Upon invoking Awk from the DOS

box, ARGC and ARGV get generated.

ARGC is argc above. Its value is the number of files being processed
plus 1.
ARGV is the argv array above. Note that ARGV[0] returns "Awk".
ARGIND is an index into the ARGV array which contains the name of
the file being processed.
You can know what file is being processed by doing ARGV[ARGIND].

FILENAME
<x>Awk (built-in variables and functions)[AUX]FILENAME is file being processed, same as (ARGV[ARGIND])</x> is the file
being processed as written in the command line. FILENAME and
ARGV[ARGIND] are equivalent. You will use one or the other
according to circumstances. Note that FILENAME is the full name
as written in the command line, along with the file path.
<x>Awk (built-in variables and functions)[AUX]FNR is record number of file being processed; NR is overall input record number</x>NR is the

input record number (line number) from the top of the first file
being processed. The NR count starts with line 1 of the first file
and continues until all the files have been processed. An
empty line counts as one line. The final value of NR will be the
total number of lines contained in the files. Note that the built-in
variable RS, which defines the end of record, has a \n (newline)
default value - by default, record numbers and line numbers
are identical. If you modify RS to \n\n to handle multiline records,
then line number and record number will diverge.
FNR is the same as NR except that the number is the line
number in the file currently being processed. Each time a new
file gets loaded, FNR is reset to 1.
<x>Awk (built-in variables and functions)[AUX]FS - fields separator (default " "); RS input record separator (default "\n")</x>FS - input field

separator (default is space " ").

RS - input record separator (default is newline "\n").
NF - number of fields in current record (words in current line).
 OFS - output field separator (default space " ").
ORS - output record separator (default newline "\n").
SUBSEP - Multidimensional arrays can be flattened into one-
dimensional arrays. The corresponding one-dimensional index
is the concatenation of the indices separated via a special
character stored in SUBSEP (\٠٣٤ - decimal ٢٨ by default, special
ASCII character FS - file separator). Thus, if you do SUBSEP="@",
the ٢-dimensional array element myArray[i,j] becomes a one-
dimensional element myArray[i@j]. This mechanism works for
larger multi-dimensional arrays.
<x>Awk (built-in variables and functions)[AUX]$ ٢ $ ,١, etc. - contain tokens of record just read in, $٠ contains entire record just read in</x><x>$ (dollar sign)

[AUX]$٠ - Awk: record just read in, Perl: program name (also $PROGRAM_NAME)</x><x>$ (dollar sign)[AUX]Awk $ ١, etc. contains tokens of record just read in;

$٢$ ,١,
etc. are built-in variables which contain the
Perl contains captures from regex</x>

tokens of a line just read in. $0 contains the entire line. These
may be assembled on the fly, example: create a variable such
as $num where num is a variable that contains an integer:
for ( num = 1; num < NF; num++) { print "$", num " = ", $num; }

will print a list of tokens as follows (NF is the number of tokens in the
line):
$1 = whatever
$2 = something
$3 = else
...
This construct is useful if you want to cycle through the tokens to
test them for something. It is an alternate to using a condition.

12.
Awk program flow control
The
<x>Awk (rules)[AUX]Supports standard C program flow control operators if, while, for, etc.</x><x>Awk (rules)[AUX]getline, next stops processing line</x>

standard C program flow control operators if, while, for, do...while, continue,
break, switch can be used just as in C. Awk provides three additional
controls: exit, next and getline. See Awk interrupt processing lines (page
312).
 <x>Awk (built-in variables and functions)[AUX]exit stops processing files; END function called</x>exit is
used to stop
processing the files. Awk will nevertheless execute the END
statement and then exit.
<x>Awk (built-in variables and functions)[AUX]next, getline interrupt rules processing</x>next means stop processing

this line, get the next line, and resume processing starting with
the first rule in the rules sequence.
getline is similar to next, it stops processing the current line, gets
the next line, but resumes processing with the current rule, not
with the first rule. There is more to getline than this. Consult the
Awk documentation for more info. See Bibliography - Awk
(page 334).

13.
Awk operators
The following standard C operators may be used:

Arithmetic:
<x>Awk (rules)[AUX]Provides arithmetic, increment, assignment, unary, logical, text match operators and conditional expression</x>

+, -, *, /, %, ^
Increment: ++, --
Assignment: =, +=, -=, *=, /=, %=, ^=
Unary: +, -, ! (not)
Logical comparison: <, >, <=, >=, ==, !=
Logical operators AND &&, OR || can be used in the condition part
of a rule or inside a statement to compare two variables.
Logical statement match ~. instead of using == and != in a
condition, you can use match ~, not matched !~, for example:
$4 !~ "New York"

means select all lines in which $4 does not contain "New York"

The conditional expression condition ? result if true : result if false; is

supported.
 14.
Awk strings
Awk strings are similar to C++'s String class
<x>Awk (strings)[AUX]Similar to C++'s String class, char array</x>

and character chains. They can be concatenated but, unlike C, there

is no need to do a strcat to concatenate string variables:
firstName = "Sasha"; lastName = "Gomez";
fullName = firstName " " lastName;
print(fullName); produces Sasha Gomez.
Notice how the space has been concatenated inside fullName without
resorting to a strcat like function.
<x>Awk (strings)[AUX]String functions: index, length, substr, match</x><x>Awk (built-in variables and functions)[AUX]String functions: index, length, substr,

The Awk built-in string functions differ from those in C. Below is

match</x>

a sampling of a few common ones. There are more. Consult the Awk
reference for the full list.

index(str, substr) - returns the position of substr in str. Remember, Awk

strings are one based, hence it returns 0 if substr is not found, the
first letter of str is at position value 1.
length(str) - returns the length of the string str.
substr(str, i, n) - returns a substring n chars long starting at position i
(1 based) in string str.
match(str, regex) - returns the position (1 based) of the regular
expression regex if there is a match, or 0 if no match.

15.
Awk print
The Awk equivalent to C++'s [Link] is the
<x>Awk (rules)[AUX]Print equivalent to C++'s [Link]</x>

simple print statement:

print($3, " ", $4, " - salary = $", $8);
will produce on two successive prints the
<x>Awk (rules)[AUX]Variables OFS, ORS control print's behavior</x>

following (OFS set to "", no space when comma encountered)

 Sasha Gomez - salary = $20
Ted Smith - salary = $22
The basic print command is print a b c d where a, b, etc. can be anything
as long as it is printable. If not printable, it skips and continues.
Notice that the print command does not necessarily require that its
parameters be inside parentheses. Commas separating items to be
printed are not required, nor is a semicolon to mark the end print
statement required. All of the following are legal.
1 - print a b c d # bare bones
2 - print a b c d; # bare bones plus semicolon
3 - print a, b, c, d; # commas used systematically
4 - print a b, c d; # commas used occasionally
5 - print (a b, c d); # parentheses used
Using commas has repercussions on the output. Using semicolons
and/or parentheses does not.

The print command

<x>Awk (built-in variables and functions)[AUX]Output record separator - default is newline (ORS = "\n")</x>

adds a newline after printing. It is equivalent to a [Link].

This default behavior can be altered by modifying the value of
the built-in ORS variable.
The semicolon is not mandatory as long as there is no
ambiguity as to what follows the print statement. But good
programming practices dictate that they be used
systematically.
Commas mean print a space:
Lines 1 and 2 will concatenate the items together and print
abcd .
Line 3 will print one space after each item: a b c d.
Lines 4 and 5 will print ab then a space then cd.
Parentheses are optional. They have no effect on the output.
<x>Awk (built-in variables and functions)[AUX]OFS output filed separator controls whether space printed when comma encountered</x>You can

modify the prints behavior via two built-in variables:

ORS (output record separator) - controls whether a newline is
printed automatically. The default is ORS = "\n".
If you want several print outputs on one line do:
 ORS = "";
print...
print...
ORS = "\n";

OFS (output field separator) - controls whether a space is printed

when a comma is encountered. The default is a space OFS = " ". If
you want commas for greater code readability and not have them
transformed into spaces, do:
OFS = "";
print a, b, c, d; # abcd will be concatenated together, no spaces added
It may be preferable to use commas for readability's sake and not
have Awk automatically add spaces. To do so, insert OFS = "" and ORS
= "" in the BEGIN block and control formatting directly. By doing this,
commas do not add a space. If you want a newline, add \n explicitly
to your print statement.

The more complex printf statement can be used to refine

formatting. Please consult the official Awk reference if you
need to use this function.

16.
Awk mathematical functions
Awk provides a set of mathematical functions, these are:

Trigonometric functions:
<x>Awk (general)[AUX]Provides trig, log, type conversion, random numbers math functions</x>

sin(x), cos(x), atan2(y/x) - x and y in radians.

Type conversion: int(x) - truncates a number to int (all numbers in
Awk are floats).
Logarithmic: log(x), sqrt(x), exp(x) is e power x.
Random numbers: rand() returns random 0-1.0, srand(x) set seed
based on time of day.
 26.
Perl (appendix)

Perl (appendix)
The following sections contain additional Perl (page 217) specific
details:

Download Perl
<x>Perl (general)[AUX]Perl details: download, use it, simulate Awk, constructs strange from C/C++ perspective</x>

(page 319) covers downloading it from the [Link] Web

site.
<x>Perl (command line - DOS box)[AUX]Invoke Perl directly or use batch file</x> Invoking Perl from DOS box

(page 319) - Perl is a command line application, like Awk. This

section covers details as to how to invoke Perl with command
line arguments, namely passing file names to Perl.
<x>Perl (Awk vs. Perl)[AUX]Simulate Awk with Perl or how to transform Awk program to Perl</x> Perl simulate Awk (page

320) is a skeleton Perl program which simulates Awk. It reads

in files defined on the command line and processes the lines of
text, one line at a time. Multiple file passes may be undertaken,
a vast improvement over Awk.
<x>Perl (file handles)[AUX]$filehandle created from file name string via the open statement</x> Perl file handling (page

321) - Awk's file handling is automatic because Awk command

line arguments are files which are automatically opened.
Output must be done via redirections, either from inside the
program when doing a print, or at command line level to grab
non redirected outputs. Here Perl differs from Awk in that
command line arguments are names which could be used for
anything. Perl file handling requires that file handles be created
via the open statement. Screen output may be redirected just as
with Awk; however, output may be directed to files from within
the Perl program via file handles, which cannot be done with
Awk.
Perl subroutines (functions) (page 322) - Perl is unique in how
subroutines (functions) are declared and used, particularly with
respect to parameter passing.
Perl strings (page 322) resemble Awk strings. A rich set of
functions enables manipulating them. Two stand out: the split
function and the qw// function to break up a long string into an
array, and join, to join together the contents of an array of words
into a single string with the inclusion of an optional separator.
<x>Perl (strings)[AUX]String interpolation differentiates a variable's name from its value in print statements</x> Perl string

interpolation - single or double quotes (page 323) - This

feature, unique to Perl, is particularly useful when doing prints.
Most programming languages require that variables be
separated from strings when doing a print. Not so in Perl, you
can place a variable inside a string, Perl's print statement will
print its value. This process is called string interpolation.
<x>Perl (build)[AUX]Lists enable doing tasks on multiple items in one line of code</x> Perl lists (page 323) - The

list concept, a sequence of values such as (4, 20, 7...), not an

array, is something neither Awk nor C have. Lists can be used
to do in one line of code what would otherwise require several
lines of C++ or Awk code.
<x>Perl (build)[AUX]Program flow control mechanisms are supported, namely for, if, switch (given), etc.</x> Perl program flow

control (page 324) - The standard C-like program flow control

mechanisms are supported, namely if, switch (given), for, while,
do...while.
<x>Perl (arrays)[AUX]Perl arrays are associative, can be mix of numeric (float) and string values</x> Perl arrays (page 325) -

Perl arrays are associative. This is the only similarity relative to

Awk. Using them requires learning new mechanisms.
Perl multidimensional arrays (page 327) are unique. These
must be closely examined because they require experimenting
and getting used to.
<x>Perl (build)[AUX]Pass by value and by reference</x> Perl pass by reference (page 329) is

rather unique.
Perl built-in variables (page 329) - Perl has a rich set of built-in
variables.
Perl strings (page 322) - Perl proposes several string operators
 and functions to facilitate string handling.
<x>Perl (build)[AUX]Importing packages (use directive): strict, diagnostic, features 'switch'</x>Perl pragmas - Perl

programs should always start by importing packages such as

use strict and use diagnostic to ensure program robustness. See Perl
importing packages (page 330).
<x>Perl (variables and functions)[AUX]Perl types differentiated by special characters: $scalars, @arrays, %hashes</x> Perl variable

types (page 331) - Perl variables are differentiated as to type

via special characters.
<x>Perl (build)[AUX]Scope resolution operator :: implements namespace concept</x> Perl scope resolution

operator :: and my (page 331) - Perl adds to program

robustness by enforcing scope resolution.
<x>Perl (strings)[AUX]Supports regex text match operator =~</x>The Perl text match operator (page

331) =~ enables assigning true/false to a variable according to a

regex's success or failure.
Bibliography - Perl (page 334) - A few selected Perl Web links
point to specific sources of information.

1.
Download Perl
Perl is included in Unix,
<x>Perl (general)[AUX]Included in Unix, Linux, and MacOS but not Windows (download Strawberry Perl)</x>

Linux, and MacOS but it may not necessarily be the latest version. If
your computer is Windows based, you can download Perl from the
[Link] Web site. It recommends a free download of
Strawberry Perl for Windows 10 ([Link]
The instructions detail how to set it up. In my case (Windows 10), I
downloaded the 64-bit Windows version and ran it with the defaults.
Perl gets installed in a directory named C:\Strawberry.
Having installed Perl, copy C:\Windows\System32\[Link] into the
directory where you will be running Perl. Test it by typing Perl -h in
the command box. It should respond with a series of possible Perl
command line options.
 2.
Invoking Perl from DOS box
Using Perl requires that you open a
<x>Perl (command line - DOS box)[AUX]Invoke Perl directly or use batch file</x>

command box (see DOS box (appendix) page 237). Once inside the
DOS box, you invoke Perl along with its arguments as follows:
<x>Perl (command line - DOS box)[AUX]Perl command line file/directory in arg1, arg2, etc.; in @ARGV array</x><x>Perl (command line - DOS box)

[Link] arg1 arg2... > [Link]</x>Perl [Link] arg1 arg2...

[AUX]

> [Link]
<x>DOS batch files[AUX]Launch Awk and Perl from batch files</x>or you can invoke a batch file
[Link] which contains the above (see DOS batch files
page 241). The file [Link] may contain hundreds of lines.
In the above:

Perl invokes [Link] which is the program in the Perl

distribution you downloaded.
[Link] is the Perl program you created. It is a text
file containing Perl statements which you have stored
somewhere and given it a .pl extension. Please note that you
do not use a -f flag to signal that what follows is the program's
name. In Perl, the first argument is the program's name.
<x>@ARGV (Perl array)[AUX]Perl arg1 arg2... command line arguments stored in @ARGV array</x><x>Perl (command line - DOS box)[AUX]Perl command line

file/directory in arg1, arg2, etc.; in @ARGV array</x><x>Perl (command line - DOS box)[AUX]Perl command line file/directory in arg1, arg2, etc.; in @ARGV

arg1 arg2 are arguments which Perl will store in ARGV[1],

array</x>

ARGV[2], etc. They could contain anything. They are not

necessarily file names such as is the case with Awk. I used
arg1 and arg2 for input and output file paths, and filenames
from arg3 on
[Link] - Perl's non file specific output will be redirected into
a file named [Link] or whatever you choose. Or you could
direct output to a file from within your program via file handles.

3.
 Perl simulate Awk
Awk processes lines of text
<x>Perl (Awk vs. Perl)[AUX]Simulate Awk with Perl or how to transform Awk program to Perl</x>

contained in a set of plain text files by applying rules to each line,

thereby accomplishing work based on a line's content. This work can
be easily simulated in a Perl program. The main steps are:

Create BEGIN and END blocks just like in Awk.

<x>Perl (Awk vs. Perl)[AUX]Cycle through file names array with foreach, process files with file handles</x>In BEGIN, cycle

through the filenames array to create file handles complete

with file paths and extensions (.h or .cpp).
Create a function to process each file (file handle) and cycle
through the lines to implement the rules:
condition/action-blocks --> if ( $line =~ /regex/ ) { ... }

After cycling through the lines of a file, close the file handle and
continue with the next file, if any.

Having processed all files, you may reprocess them with a

different set of rules. When you are all done processing files,
the END statement gets invoked when program execution ends.

Here are the details: first create an array of file names based on the
command line arguments. This can be done inside the BEGIN block.
$NumberOfFiles = scalar(@ARGV) - 3; # ARGV[1/2] are input/output paths, ARGV[0] contains
"Perl"
@filenames = @ARGV[3..$NumberOfFiles-1]; # array of file names as subarray of @ARGV.
Cycle though all the files with a foreach, open one file at a time,
initialize variables. This stage replaces Awk automatic file loading.
my $totalLineCount = 0; # sums number of lines in all files
foreach my $filename (@filenames) {
open $fileHandle, '<', $fileName;
$lineCount = 1; # holds line count of file being processed
Traverse each file with a while loop:
while (my $line = <$fileHandle>) {
Apply the rules using the $line content
 if ( $line =~ /regex1/) { ... }
if ( $line =~ /regex2/) { ... }
if ( $line =~ /regex3/) { ... }
...
Having finished processing the rules on the current line, increment
$lineCount and continue on to the next line. When there are no more
lines to process, $line becomes zero thereby exiting the while loop.
$lineCount++; # increment to get total line count of file
} # while
Having finished work with a file, update $totalLineCount with $lineCount,
close the file, and continue with the next file via the files foreach loop.
$totalLineCount += $lineCount;
$lineCount = 0;
close $fileHandle;
} # foreach filename
print "Finished processing files";
The above does exactly what an Awk program would do. Each Awk
condition/action-block would be handled by a Perl if regex-based
match statement and its corresponding action-block so:
Awk's condition/action-block transposes to if($line =~ /regex/) { action-block }
The above represents a code block to process a group of files as
Awk would. The Perl code block above may be repeated with
different rules to do multi-pass processing which is not possible with
Awk. Example: A first pass could register all enums in a table. A
second pass could detect where enums are used and which are
declared but not used. It could also do a couple of additional passes
to do the same on #defines.

4.
Perl file handling
Perl provides a construct which
<x>Perl (file handles)[AUX]$filehandle created from file name string via the open statement</x>

it names file handle. These provide access to files for reading,

writing, or updating. A file handle is a variable created with the
function open for input or output. They return OK or undef if the file is
not found.
$openFileResult = open $myFilehandle, $direction, $myFileName; or die "message if cannot
open";
Note the die keyword above; it is Perl's way of handling an error when
trying to open a file.

<x>Perl (file handles)[AUX]Files in read, write, or update mode</x>Creating a file handle with open
requires five items:
$openFileResult - if successful, generates a value, otherwise,
generates undef.
$direction is < to read, > to write, +< or >> to update depending on
the Perl version.
$myFilehandle is the file handle being defined.
$myFileName is the file's name.
die is part of open, it issues a message in case of failure.

Once created and opened, think of a file handle as being an array

or list of records (lines). You can step through the file handle one
record (line) at a time, just as you would through a list, one list item
at a time. Once finished with the file you should close it via its file
handle close($myFilehandle).

Reading from a file with Perl requires that you first declare a
file handle such as
open $myFileHandle, '<', "[Link]";

You may then cycle through a file via

foreach $line ( $myFilehandle) { do something on $line }
This is unlike Awk, which automatically cycles through the files
provided as command line arguments.

Writing to a file requires that you declare a file handle and use
it to direct the print thus:
$myFilehandle;
open $myFilehandle, '>', "[Link]";
print $myFilehandle "This is text to append to the file [Link]";

The print's content will be appended to the file.

There is a short cut to

<x>Perl (file handles)[AUX]Convert $fileHandle to array via <...>, cycle with foreach</x>
 access all lines in opened files: the empty angle operator <>
performs operations on all lines in opened files. Similarly,
while(<>) { ... } will process all the lines in opened files. You can
limit processing to a specific file by plugging in the file's name
inside the <> such as while(<$myFilehandle>) { ... }.

5.
Perl subroutines (functions)
<x>Perl (variables and functions)[AUX]Functions have a name and a body but do not have a parameter list</x><x>Perl (variables and functions)[AUX]Functions may be called with

Subroutines are functions. It is a question of

parameters - unique way of handling (via array @_)</x>

terminology. They have a name and a body but they do not have a
parameter list due to Perl's peculiar way of passing parameters to
subroutines. A typical definition would be:
sub MySubroutine { subroutine's body }
Notice the absence of a parameter list. They may however be called
with parameters as usual, such as:
MySubroutine( param1, param2...);
The body of the subroutine accesses the function parameters via the
built-in array @_. Perl could hardly be more cryptic than that!
Individual parameters are accessed as successive items in the array:
param1 is $ _[0]
param2 is $ _[1]
...
Other than this parameter passing peculiarity, subroutines may be
used as functions as in other 3rd generation programming languages
(C, Basic, Fortran, Pascal, etc.).

6.
Perl strings
<x>Perl (strings)[AUX]Perl supports string concatenations</x>String concatenation - In Awk and C++ you
would do:
$myString = "first" " " "time" " " "around"; yields "first time around".
In Perl, you would have to use a period:
$myString = "first" . " " . "time" . " " . "around"; yields "first time around"
or you could do a join on a list with spaces in between the items:
join(" ", ("first", "time", "around"));
The join function takes two parameters, sep and items; sep is the
separator, often a space, to insert between the items. items is a list
of items to be joined together. This join function simplifies creating
customized lines of text such as for CSV files, where the separator is
a comma, not a space.

7.
Perl string interpolation - single or
double quotes
<x>Perl (strings)[AUX]Supports literal strings Perl strings may be
(single quotes) and interpolated strings (double quotes)</x>

enclosed in single quotes or double quotes. They may be of any

length, a single character, or no character at all such as '' (two single
quotes), or "" (two double quotes), or "A", or longer, such as 'this is a
longer single-quoted string'. Using one instead of the other is a question of
habit most of the time. C programmers should be careful in that there
is no such thing as a null value to terminate the string. They should
also beware of single quotes. In C, single vs. double quotes
differentiate single characters from strings. Whereas in Perl single-
quoted strings are literal strings. Double quoted strings are special -
they are referred to as interpolation strings; they serve to process a
variable inside a string, a feature unknown in most programming
languages. The following print statement:
<x>Perl (strings)[AUX]Supports literal strings (single quotes) and interpolated strings (double quotes)</x> $var = "This is a string.";
print '$var = $var';
will produce:
$var = $var
because the content of single quoted strings is taken literally. There
is no special treatment. However, if you use double-quotes such as:
<x>Perl (strings)[AUX]String interpolation differentiates a variable's name from its value in print statements</x> print "$var = $var";
The print statement will produce:
This is a string. = This is a string.
Perl calls this string interpolation. It prints the value of $var.
To get what you want, you should escape the first $var as in:
print "\$var = $var";
The above will produce:
$var = This is a string.
Notice how you do not have to put the variable outside the double-
quotes as you would need to do in Awk, in C, and in most other
languages.
String interpolation is further illustrated by the following example. In
practically all programming languages, you can do a print which
mixes quoted strings and variables such as:
print ("Full name: ", firstName, " ", lastName");
The variables and string components of the print statement are
separated by commas, and the lot is enclosed in parentheses. It
makes for difficult reading. In Perl, you would do:
print "Full name: $firstName $lastName";
This print statement is characterized by the fact that it all fits inside
one interpolated double-quoted string, and that it is the variable's
content which should be printed, not its name. You can print a
variable's name by escaping the $ sign such as in print "\$myVar = $myVar"
above.

8.
Perl lists
Perl allows list-oriented operations
<x>Perl (build)[AUX]Lists enable doing tasks on multiple items in one line of code</x>

such as multiple assignments:

($a, $b, $c) = (1, 2, 3)
1 is assigned to $a and so on. Or swap values:
($lunchDesert, $supperDesert) = ($supperDesert, $lunchDesert);
or assign values to arrays:
@daysOfTheWeek = ("Monday", "Tuesday"...);

9.
Perl program flow control
 Perl supports the C-style
<x>Perl (build)[AUX]Program flow control mechanisms are supported, namely for, if, switch (given), etc.</x>

program flow control constructs: if, while, for, foreach, given (switch), do...while.
Most of these work just like C++'s. Others differ, they are described
below.

Perl supports for/while, however, instead of interrupting the loop

with continue and break, the semantic is next for continue, last for break.
<x>Perl (build)[AUX]Process files: foreach, while on $FileHandle</x>The foreach construct can be

interpreted as being an automatic for on a list. For each item

inside a list (an array, for example), do something. The syntax
is:
foreach $item (array, or filehandle, or list ) { ... }

<x>Perl (built-in variables and functions)[AUX]$INPUT_RECORD_SEPARATOR, or $RS, or $/ - default \n</x><x>$ (dollar sign)[AUX]Perl

Unlike a for statement, there is not need to

$INPUT_RECORD_SEPARATOR, or $RS, or $/ - default \n</x>

define where to start from and how to end processing. It starts at

the beginning of the list, handles each item one by one
successively, assigns its value to $item, and stops at the end of the
list. If the list is a file handle, $item will contain a line of text since a
file is considered by Perl to be a list of lines of text, assuming the
input record separator $/ has not been modified from its default
(newline \n).

- The switch given (aka switch)

<x>Perl (build)[AUX]C's switch/case/default are named given/when/default in Perl</x>

statement, referred to as given in Perl, is supported but does not

work quite the same way as in C; switch/case/default are named
given/when/default respectively. There are no break statements to
prevent one case's execution to flow into the next case. If a
when is activated, none of the other when statements are tested
which is the equivalent of an automatic break. The consequence
is that you cannot group cases together as you would in C/C++
and cannot process one case then flow into the next case for more
processing. The syntax is:
use feature "switch";
...
 given ( $controlValue ) {
when( value1) { ... };
when( value2) { ... };
...
default { ... };
} // #controlValue

But it could also be used as follows:

my $var = 2;
given( $var ) {
say "first" when (1);
say "second" when (2);
default {print "default";};
}

10.
Perl arrays
Perl arrays and C arrays
<x>Perl (arrays)[AUX]Perl arrays are associative, can be mix of numeric (float) and string values</x>

differ. Perl arrays are associative - items may be of different types. In

this respect, they somewhat resemble Awk arrays, but this is where
the similarity ends. Here are a few of the features which characterize
Perl arrays.

Creating arrays - A variable

<x>Perl (arrays)[AUX]Perl provides three variable types: $scalars, %hashes, @arrays</x>

which is to be an array must be declared with the @ array

identifier such as @myArray. It could be initialized in one of
several ways:
<x>Perl (arrays)[AUX]Create array as subset of existing array, from a Perl list, or by tokenizing string</x><x>Perl (arrays)[AUX]Create individual array items on

the fly</x><x>Perl (arrays)[AUX]Convert $fileHandle to array via <...>, cycle with foreach</x> Via a list: @myArray = (4, 1,
'Sasha, 'Gomez' );
Via a subset of another array: @myArray = @biggerArray[4..7]; # indices 4
to 7
Create individual array items on the fly: $myArray[3] = 5;
Tokenize a string: @myArray = qw/Mr. Sasha Gomez Cambridge 02139/, '109
Mass. Ave..'/;
Transform a file into an array: @myArray = <$filehandle>;
 Arrays need not be
<x>Perl (arrays)[AUX]Create multidimensional arrays via nested square brackets initializations</x>

predefined as to size and as to the data type they will contain. They
are dimensioned automatically as a result of initialization and may
be re-dimensioned automatically upon assignments with indices
outside its range or by array item removals. As for data types,
arrays may contain a mix of numeric and string values.

Array indices, unlike

<x>Perl (arrays)[AUX]Array indices must be integers - will convert non integer indices to integer</x>

Awk, are integers only. If you use an integer, Perl will accept it
as such. If a float, it will use its integral part to transform it into
an integer. If a string, it will convert it to index 0:
index 5 becomes 5,
index 3.7 becomes 3,
index "first" becomes 0.
Array contents - Just like in Awk, array items may be numbers
(float or integer) or strings. You could thus do something like:
@myArray = ('Sasha', 'Gomez', ..., 55, 41...);
Using indices to get array items - Accessing items in the array
is done by changing the array item's name from being an array
to being a scalar item onto which an index is appended.
Example: the ٤th item in @myArray is $myArray[٣]. Since Perl defines
single valued items as being scalar, an array item takes the $
prefix. $myArray[3] is an array item which should not be confused
with @AoA[3], the 4th subarray of the multidimensional @AoA
array. Note that Perl array indexing is 0 based. Furthermore,
since Perl arrays are associative, $myArray[3] means get the item
whose key is 3. Items 2 or 4 could well not exist. In Perl,
undefined variables have the value of 0 in a numeric context, or
the value null string "" in a string context.
<x>Perl (arrays)[AUX]scalar gets number of items in one-dimensional array</x>Array size - To get the number

of items in a one-dimensional array, use scalar as in $numberItems =

scalar @myArray. But note that getting the array size of the first
subarray in a two-dimensional array by doing scalar @myArray[0]
does not work. See Perl multidimensional arrays (page 327)
 (next section). When using scalar @myArray to get a one-
dimensional array's size, the number of items reported is the
index value of the last item + 1 (remember, arrays are 0-
based). If the array is missing items, the number of items
reported will include them. In this respect, Perl and C function
the same way. Awk, on the other hand, will report the net
number of items. It does not include the missing items. In the
following:
myArray = (5, 20, 3); # indices are 0,1,2 - there are three items
$myArray[7] = 33; # index 7

Awk would report four items, Perl will report 8 items (last index + 1).

<x>Perl (arrays)[AUX]Cycle through the entire array, via a for loop, foreach loop, while loop</x>Managing Perl arrays -
Since arrays are lists, it is possible to cycle through the entire
list, via a for loop, via a foreach loop, or via a while loop.
<x>Perl (arrays)[AUX]Cycle through the entire array, via a for loop, foreach loop, while loop</x>for loop - Traversing an

array with a for loop is identical to doing so in C. Iterate through

the array one index at a time, starting with ٠ (Perl arrays are 0
based), stop after the last index. Both Perl and C return an
array's size equal to the last index plus 1. Thus, the iteration
continues as long as the index is less than the array size. To
cycle through the array, get the array's size, then implement
the for loop, just like in C:
my $myArrayLength = scalar @myArray;
for ( my $i = 0; $i < $myArrayLength; $i++ ) { do something with $myArray[$i] }

Since Perl arrays are associative, the array could have empty
spaces - these will be undefined. You can check for undefined
values with the defined function which returns true for defined
variables:
if ( !defined $myArray[$i] ) { handle the undefined array item }
As an aside, notice the my in front of $i in the for statement. It is the
scope operator; it makes $i local to the for loop.

The foreach loop construct

<x>Perl (arrays)[AUX]Cycle through the entire array, via a for loop, foreach loop, while loop</x>
 enables traversing an array item by item such as
foreach(my @myInfo) {print $ _;} or foreach $line (my @myInfo) {print $line;}

<x>Perl (built-in variables and functions)[AUX]$ARG or $ _ is the line of text (record)</x><x>$ (dollar sign)[AUX]Perl - $ _ contains the entire line that has just been

Note that $ _ contains the entire line that has just been read.
read</x>

Notice also that there is no need to explicitly get items from the
array one at a time via an index. Furthermore, the foreach construct
traverses the array one $line item at a time as if it contained a
contiguous set of variables. Undefined items, which the for loop
must handle, are ignored by the foreach loop.

A while loop may be used

<x>Perl (arrays)[AUX]Cycle through the entire array, via a for loop, foreach loop, while loop</x>

by defining a temporary array and using the shift function to get

and remove the first item at the start of each iteration:
my @tempArray = @myArray;
while( my $item = shift( @tempArray ) { do something with $item }

The while exits when there are no more $items to shift. Although this
solution is a little cumbersome, the foreach array traversal is cleaner,
no need to create a temporary array. There could be performance
issues which would favor this approach.

Adding and removing

<x>Perl (arrays)[AUX]push/pop add/remove last item in Perl array; shift/unshift does same on first item</x>

array items - Array items can be added to or removed from the

head or tail of the array with the functions shift/unshift
(prepend/remove first) and push/pop (append/remove last). This
feature may be used to process an array with a while loop as
demonstrated above. Individual array items may be physically
removed via the delete(item) function. Array items can also be
added by assigning a value to an array item whose index is
unused, whether it be within the array's current index range or
outside it. Note that if you add an array item outside the array's
current range, the array gets re-dimensioned automatically.
<x>Perl (strings)[AUX]Perl arrays to strings with list function</x><x>Perl (strings)[AUX]Perl string tokenization and concatenation functions: split, qw//,

Strings to arrays and arrays to string - A line of text can be

join</x>

transformed into an array of words with the split function or with

 the qw// function, as shown above in the Perl arrays (page 325)
section. Doing the reverse, transforming an array of words into
one contiguous string, can be done with join(sep, list) where sep is a
separator such as a space, comma, whatever; list is the array
whose items are to be joined together.
<x>Perl (arrays)[AUX]Create array as subset of existing array, from a Perl list, or by tokenizing string</x>Array copy - A subset

of an array can be copied into another array:

@newArray = @oldArray[0..4]

copies the five items of the array @oldArray indexed 0 to 4 into

@newArray.

Creating an array from a list is simple:

@myVars = (5,4,3,2,1);

The list of five values creates the @myVars array.

Arrays can be sorted - The sort

<x>Perl (arrays)[AUX]Perl arrays can be sorted via the sort function</x>

function will sort array items according to the items' ASCII

values. This is a basic string-oriented sort, not a numeric sort.
Numbers will be sorted as strings, hence the sort will be
numbers first, followed by lower-case letters and then upper-
case letters. The command to sort @myArray is:
@sortedArray = sort( @myArray );

If the array contains decimal values greater than 10, the ASCII-
based sort outcome would be similar to 1 11 13 2 21 5 55... which is not
a numeric sort.

Array merges - Two or more arrays

<x>Perl (arrays)[AUX]+ sign overloaded to enable merging arrays</x>

can be merged together into a larger array by adding them

together.
@array = @array1 + @array2.

11.
 Perl multidimensional arrays
Multidimensional arrays in Perl
<x>Perl (arrays)[AUX]Create multidimensional arrays via nested square brackets initializations</x>

are arrays which contain arrays. Thus @AoA[2] would be the third
array of the two-dimensional array @AoA. Note that the AoA notation is
a mnemonic for an array of arrays. Had you written $AoA[2], you
would be referencing the third item of the one-dimensional AoA array.
The following short example shows how Perl arrays can indeed be
confusing and error prone.

Arrays are dynamically dimensioned - This means that when

you define an array, such as @AoA (this is a shorthand to mean
array of arrays), it could be a one- or a multi-dimensional array.
Suppose the following:
@AoA = (
["Sasha", "Mary", "Paul", "Henry", 5],
["Abigail", "Joseph", "Eve", "Arthur", 8]
);

The statement above defines a two-dimensional array; it contains

two subarrays. Deeper arrays could be created by nesting square
brackets. Accessing an array item from one of the subarrays is
done just the same as in C: $AoA[1][3] of the array above yields Arthur.
Remember, Perl arrays are zero based.
<x>Perl (arrays)[AUX]Multidimensional arrays is cumbersome</x>Managing subarrays of multidimensional

arrays is cumbersome. A two-dimensional array variable is

obtained by using two indices. Just like in C, the first index (left to
right) defines the subarray; the second index defines the item in the
subarray: $AoA[1][3] yields Arthur. From a conceptual standpoint, [1] is
the subarray, [3] is the index in the subarray. Logic dictates that
printing all variables specific to the @AoA[0] subarray could be
achieved by cycling through it thus:
foreach $item ( @AoA[0] ) {say "\$item = $item";}
The above looks right, AoA[0] should be the first subarray, but it does
not work. It generates an error with a suggestion to use $AoA[0]
instead of @AoA[0]. This means converting AoA[0] from being an array
to being a scalar, which I do not really understand. So I did
 foreach $item ( $AoA[0] ) {say "\$item = $item";}
It compiled error free, but I got the following print result:
$item = ARRAY(0x652288)
Of course, I could use a for loop to cycle through the items, but to
do this I would have to know the subarray's size. This is a whole
other can of worms (illustrated below). So, I battled on, and finally
managed to create a subarray and print its content with a foreach loop
by doing:
my $subArray = $AoA[0][0];
foreach $item ( @$subArray ) {say "\$item = $item";}
In effect, the above is saying: (1) get the first array item and put it in
a scalar variable $subarray and, (2) transform the scalar variable thus
retrieved into an array by prefixing the scalar variable with the array
qualifier @. I have absolutely no idea as to why this works but the
fact remains, it does yield the desired result. Perhaps doing this is
really saying: get the first item of the subarray you want (a scalar);
after doing this transform this scalar to make it become the
beginning of an array by prefixing it with the @ array notation.

Multi-dimensional array sizes - Just as for one-dimensional

arrays, getting a two-dimensional array's size is based on
using the scalar function. @AoA is the two-dimensional array
introduced above.
$arraySize = scalar @AoA;

The above yields 2, which is surprising. AoA does indeed have two
subarrays. I would have expected the total number of variables -
10; this is what Awk would do. My objective being to get the
number of variables in the subarrays, logic suggests that I could
do:
$arraySize = scalar @AoA[0];
But no, this yields a warning and a bad result ARRAY(0x652108). Perl
suggests that I convert the subarray to a scalar. I then tried as
suggested:
$arraySize = scalar $AoA[0];
and did not get a warning but I still got the same bad result. Getting
the dimension of a subarray got to be a little more complex. The
solution lies in proceeding indirectly as illustrated in the handling
 subarrays section above. The key lies in transforming the sub array
of a two-dimensional array into a standalone one-dimensional
array. Thus:
$subArray = $AoA[0][0];
<x>Perl (arrays)[AUX]Getting dimension of subarray requires programming trick</x>

$arraySize = scalar @$subArray;

This works! It looks awkward. I have not figured out why, perhaps
an expert Perl programmer could find an elegant explanation.
There might be a reason though, something tied to implicit
addressing or dereferencing.

12.
Perl pass by reference
There are circumstances when passing by
<x>Perl (build)[AUX]Pass by value and by reference</x>

reference is the only way to get the job done. For example, if you
want to modify items of an array via a function, you must pass the
array as a reference so changes on array items inside the function
changes items in the original function as well.
<x>\[AUX]pass by reference Perl operator </x>The procedure for doing this relies on the pass by

reference operator \@. When invoking a subroutine with an array to

be passed by reference, do:
my $result = MyFunction(x, y, \@myArray, ...);
In the above, my array is the 3rd parameter. The function will contain
the following dereferencing code:
sub MySubroutine {
...
my @targetArray = @{$ _[2]}; # 3rd parameter, zero based
Wow, talk about being cryptic (@{$ _[2]}), Perl can be a geek's delight.
After doing the above, @targetArray is used as you would use any array.
As you work on @targetArray, it is as though you were directly working
on the original @myArray.

13.
Perl built-in variables
 You may access practically
<x>Perl (built-in variables and functions)[AUX]English pragma: AND instead of &&, OR instead of ||</x>

all of Perl's built-in variables in standard English or as characters

and symbols. By default, Perl built-in variables start with a $ or an @
sign. The use English pragma must be invoked to enable using the
standard English name version.

<x>Perl (built-in variables and functions)[AUX]$PROGRAM_NAME, or $0, contains the Perl program launched</x><x>$ (dollar sign)[AUX]$0 - Awk: record just

$PROGRAM_NAME,
or $0, contains the Perl
read in, Perl: program name (also $PROGRAM_NAME)</x>

program launched. It does not contain a line just read in from a

file as in Awk. To get a line just read in, use $ _.
<x>Perl (built-in variables and functions)[AUX]@ARGV array: command line arguments</x><x>@ARGV (Perl array)[AUX]Perl arg1 arg2... command line

arguments stored in @ARGV array</x> @ARGV is an array which contains the command
line arguments.
<x>Perl (built-in variables and functions)[AUX]$ARGV array contains current file name when reading from <> array</x><x>$ (dollar sign)[AUX]Perl $ARGV array

$ARGV
contains current file name when reading from <> array</x> is the current file name when
reading file as array from <>.
<x>Perl (built-in variables and functions)[AUX]$INPUT_LINE_NUMBER, $NR, $. line number of file being processed</x><x>$ (dollar sign)[AUX]Perl $NR

alternate $INPUT_LINE_NUMBER or $., line # of file being processed</x><x>$ (dollar sign)[AUX]Perl $. alternate $INPUT_LINE_NUMBER or $NR, line # of file

being processed</x>$INPUT_LINE_NUMBER, or $NR, or $., is the line number of

the file being processed.
<x>Perl (built-in variables and functions)[AUX]$INPUT_RECORD_SEPARATOR, or $RS, or $/ - default \n</x><x>$ (dollar sign)[AUX]Perl $RS alternate

$INPUT_RECORD_SEPARATOR or $/ - input record separator, default \n</x><x>$ (dollar sign)[AUX]Perl $INPUT_RECORD_SEPARATOR, or $RS, or $/ -

$INPUT_RECORD_SEPARATOR,
default \n</x> or $RS, or $/ - input record
separator. It defines how text is separated into records. The
default is a newline \n which means that source code is read
one line at a time. A file is thus a list of lines (records).
<x>Perl (built-in variables and functions)[AUX]$OUTPUT_RECORD_SEPARATOR, or $\ - default newline</x><x>$ (dollar sign)[AUX]Perl $\ alternate

$OUTPUT_RECORD_SEPARATOR (default newline)</x> $OUTPUT_RECORD_SEPARATOR, or $\ - By

default, print does not terminate with a newline. Set to $\ = "\n" to
get print to add a newline.
<x>Perl (built-in variables and functions)[AUX]Input field separator - Perl does not seem to have one</x>Input field separator -

Perl does not seem to have an explicit input field separator

such as Awk does, but it does have $OUTPUT_FIELD_SEPARATOR.
<x>Perl (built-in variables and functions)[AUX]$OUTPUT_FIELD_SEPARATOR, or $OFS, or $, separates fields from each other</x><x>$ (dollar sign)[AUX]Perl
 $OFS alternate $OUTPUT_FIELD_SEPARATOR or $, separates fields from each other</x><x>$ (dollar sign)[AUX]Perl $, alternate

$OUTPUT_FIELD_SEPARATOR, or $OFS separates fields from each other</x> $OUTPUT_FIELD_SEPARATOR, or $OFS,

or $, separates fields from each other.
<x>Perl (built-in variables and functions)[AUX]$ARG or $ _ is the line of text (record)</x><x>$ (dollar sign)[AUX]Perl $ARG - same as $ _, is the complete record

or $ _ is the $ARG
(line of text)</x><x>$ (dollar sign)[AUX]Perl - $ _ contains the entire line that has just been read</x>

complete record (line of text). You would typically access each

line by doing:
foreach $line ( $myFilehandle ) {use $line };

The alternative is to use $ _, no need to create the $line variable. The

loop then becomes:
foreach ( $myFilehandle ) { use $ _ }.

<x>Perl (built-in variables and functions)[AUX]@ARG or @_: arguments to function</x><x>@ (at symbol)[AUX]Perl @ARG or @_ - array passes arguments to a

function</x><x>@ (at symbol)[AUX]Perl @_ alternate @ARG array passes arguments to a function</x><x>Perl (arrays)[AUX]Array @_ is mechanism to pass

arguments to a function</x> @ARG or @_ - An array of arguments passed to a

function.
<x>Perl (built-in variables and functions)[AUX]$SUPSEP or $; - index separator</x><x>$ (dollar sign)[AUX]Perl $; alternate $SUPSEP - index separator, comma by

default</x>$SUPSEP or $; - index separator, comma by default.

There are more built-in variables, go to

[Link] to get the complete list.

14.
Perl string functions
From a C++ programmer's perspective, Perl provides a couple of
interesting string operators not found in C++:

<x>Perl (built-in variables and functions)[AUX]split and qw// functions automatically tokenizes a line of text</x><x>$ (dollar sign)[AUX]Awk $ 1, etc. contains

Splitting lines of text into words - Perl

tokens of record just read in; Perl contains captures from regex</x>

does not automatically tokenize a line into components as Awk

does; but it can automatically tokenize a line of text via the split
function. It can also extract data from a line via the regex
 parens-based capture mechanism and puts them into $1, $2,
etc. See the Perl text match operator (next page). In Awk, a
line of text is stored in $0. In Perl, a line of text is stored in $ _ ($0
contains the Perl program's name).
<x>Perl (built-in variables and functions)[AUX]split and qw// functions automatically tokenizes a line of text</x><x>Perl (strings)[AUX]Perl string tokenization and

Tokenization - The qw// function breaks up a

concatenation functions: split, qw//, join</x>

string into its individual tokens. As follows:

@myArray = qw/This is a string/

@myArray will
contain the four words of the original string as individual
array elements. Notice that the string is not enclosed in quotes; had
it been quote delimited, the first and last item would include the
quote thus: first item: "This and last item: string".

<x>Perl (built-in variables and functions)[AUX]qq{}function enables embedding double quotes inside of string without escapes</x><x>Perl (strings)[AUX]qq{}

Quoted strings inside quoted

function enables embedding double quotes inside of string without using escapes</x>

strings - Practically all programming languages require that a

double quoted string inside a double quoted string must be
escaped. For example: "He said \"yesss!\"". In Perl, you would have
to do the same, except that Perl has the qq{} operator; using it
frees you from escaping quotes; the example string could be
written as qq{He said "yesss!".}. It is cleaner and less error prone.
The q{} operator is the single quote equivalent.

15.
Perl importing packages
Perl, like Awk, is a single
<x>Perl (build)[AUX]Importing packages (use directive): strict, diagnostic, features 'switch'</x>

module program; one text file defines the program. But, unlike Awk,
it can import packages of predefined code, much as C++'s importing
libraries. Perl accomplishes this via the packages concept: pragma or
use.
pragmas and use are the first items in a Perl program, except for
comments of course, but these are ignored. Two pragmas should
always be loaded: the use strict and use diagnostics pragmas. strict forces the
programmer to adhere to strict syntax, which should improve
program robustness. diagnostics provides useful debugging info to the
programmer when Perl encounters a programming error.

1.
Perl variable types
<x>Perl (variables and functions)[AUX]Perl types differentiated by special characters: $scalars, @arrays, %hashes</x><x>$ (dollar sign)[AUX]Perl provides three variable types:

$scalars, %hashes, @arrays</x><x>%[AUX]Perl provides three variable types: $scalars, %hashes, @arrays</x><x>@ (at symbol)[AUX]Perl provides three variable types: $scalars,

Differentiating variable
%hashes, @arrays</x><x>Perl (arrays)[AUX]Perl provides three variable types: $scalars, %hashes, @arrays</x>

types requires that a variable's name be preceded with a special

character according to its type: $, @, % (scalar, array, hash). This can
play tricks on you as $myVar and %myVar may be unrelated variables. As
for arrays, they are declared with an @ (@anArray). However, its
individual items are accessed by qualifying the array as scalar by a $
sign such as $anArray[0]. Notice that the name stays the same, an
index inside square braces has been added, and the first character
has been changed from the array symbol @ to the scalar symbol $.
Note that doing @anArray[0] refers to the first array in what would be
deemed to be a multidimensional array, and does not refer to the first
item in the one-dimensional array @anArray - Scary!

2.
Perl scope resolution operator :: and
my
Suppose that an imported module
<x>Perl (build)[AUX]Scope resolution operator :: implements namespace concept</x>

called DataGroup contains a variable named $buffer and that you want to
create your own $buffer variable. You will do DataGroup::$buffer to access
the DataGroup module's version. As for your $buffer variable, you will
declare it as my $buffer, which you can use in your program without a
scope qualifier. The my keyword is stating that the variable being
declared belongs to the main program, not a module. Furthermore, it
limits its scope to the level in which it was declared, such as
happens in C++. One of the consequences of using the use strict
pragma is that variables must be explicitly declared with the my scope
resolution operator; omitting the my keyword will trigger an error.

16.
Perl text match operator
Regexes are extremely useful. The Perl
<x>Perl (strings)[AUX]Supports regex text match operator =~</x>

programmer can fortunately embed a regex in his/her program, apply

it to some text, and capture data in his/her algorithm via the $1, $2,
etc. capture mechanism. Here is an example:
my $text = "the little fox had glossy brown fur";
if ( $text =~ /(\w+)\s+had\s+(\w+)/ ) {
print("\$1 = $1 \$2 = $2");
}
This regex gets the two words on each side of the string "had". The
result is:
$1 = fox $2 = glossy
the parentheses around the two \w+, causes the contents of the two
\w+ to be captured into the built-in variables $1 and $2 respectively. If
the regex were longer and had more parens pairs, each open parens
would add an additional capture: into $3, $4, etc. Parentheses may be
nested which means that captures may overlap.
 27.
Bibliography

Bibliography
The bibliography covers resources I came across which I consider
relevant within the context of my two books: Pragmatic C++ Arduino
Programming and Defensive C++ Arduino Programming. This
bibliography is by no means exhaustive nor thorough, and some
crucial works may have been left out. It is organized according to
self-explanatory themes.

1.
Bibliography - C/C++ programming
The C programming Language, 2nd edition - Brian W. Kernighan
and Dennis M. Ritchie. Prentice Hall, 1988. This is the original book
on the C language.
The C++ Programming Language, 4th edition - Bjarne Stroustrup,
Addison Wesley, 2013. This is bible number 2. Comprehensive, a
must to get details on new features in C++. It remains outdated since
we are moving to C++20; however, it is the most current and
complete C++ book I have found.
Pragmatic C++ Arduino Programming, Amazon self-published
2023, Michèle Delsol. A practical guide for Arduino
C++ programmers or why you may be dedicating far too much time
to debugging, yet you think you know C++.
Defensive C++ Arduino Programming, Amazon self-published
2023, Michèle Delsol. The right tools for C++ Arduino programmers
(Arduino IDE V1 and V2, AtmelStudio, Visual Studio, Visual Micro,
VS Code, PlatformIO, C++ frameworks, Awk, Perl, regular
expressions).
Practical C++ Programming, 2nd edition - Steve Oualline, O'Reilly,
2003. A good practical textbook for those new to C++ programming.
Effective C++, 3rd edition - Scott Meyers, Addison Wesley, 2005.
This book gives you a taste as to how complex C++ can get.
Effective Modern C++, - Scott Meyers, O'Reilly, 2017. This book, a
sequel to Effective C++, introduced above, is for advanced C++
programmers. It goes into the arcane of C++, way beyond the
average Arduino programmer's needs. But, if you intend to become
an expert C++ programmer, this book is for you.
C - A Reference Manual, 3rd edition, Samuel P. Harbison, Guy L.
Steele Jr., Prentice Hall Software Series, 1991. A useful
complementary book.
The Standard C Library, by P.J. Plauger, Prentice Hall, 1992. This
is a good early compilation of The C Standard Library.
Expert C Programming - Deep C Secrets, Peter van der Linden,
Prentice Hall, 1994. Contains useful tidbits and illustrates how much
C language I was unfamiliar with.
C Programming for Dummies, 2nd edition, by Dan Gookin, Wiley
Publishing, Inc., 2020. This introduction to the C language may be
used as a reference. I consider the Dummies in the title to be a
marketing gimmick. If you are to develop small applications, opt for
the Perl language or Python, or other; but these simpler languages
come with a cost: larger applications and slower execution speeds.
C++ for dummies, 7th edition, by Stephen R. Davis, Wiley
Publishing, 2014. My comments are the same as for C for
dummies. It tends to mix C and C++. I much prefer an approach
which differentiates C from C++. Walk before you run. Both of the for
dummies books (C and C++) consider that C and C++ are different
languages. No! C++ is C enhanced with object-oriented
programming. Bjarne Stroustrup, the inventor of C++, says so. That's
why there is a ++ in the name - it means increment.
C++ Core Guidelines (do a Google) is a collaborative effort of 240
programmers who have put together a set of recommendations to
improve C++ programmer performance. Moderated by Bjarne
Stroustrup and Herb Sutter, it is remarkable in its completeness. At
around 520 pages long, it covers a lot of material.
ASCII Table - [Link] - Looking up values for ASCII
characters is a constant chore. This site presents the first 127
characters in a concise practical way.
The C language is documented in a GNU official manual
[Link] - 91 pages.
An excellent reference work, I recommend you download it so it will
be available to you at a moment's notice.
GNU C/C++ compiler - AtmelStudio, the Arduino IDE, and
PlatformIO implement the same C/C++ GNU compiler. Because of
this, the compiler documentation is the same :
[Link]
GNU preprocessor - GNU's manual on the preprocessor is a useful
complement
[Link]
Pragmatic C++ Arduino Programming, presented above, covers
the preprocessor extensively.
STL - All C++ programmers should be familiar with the Standard
Template Library. It contains advanced features which may be of
use. See [Link]
Unfortunately, the STL is not part of the Arduino distribution, but the
Arduino IDE does support templates.
Jumping into C++ - Alex Allain, 2013. Alex also created an
excellent Web site on C/C++ : [Link]
The Cherno is a knowledgeable C++ programmer. He regularly puts
out YouTube video tutorials - they are concise and well presented.
Andreas Spiess, the guy with the Swiss accent on YouTube,
presents knowledgeable, short, precise, thorough coverage of
C++ programming tools and of electronics in general.
Instant C/C++ info - [Link] is a good place
to get info on specific C/C++ functions.
[Link] is another excellent Web site which teams
with savvy info on C++.

2.
Bibliography - PROGMEM
framework
Here are some sources which cover the PROGMEM framework:
<x>PROGMEM (read-only data in flash memory)[AUX]Web sites which describes PROGMEM</x>[Link]

libc/user-manual/group__avr__pgmspace.html
[Link]
demystified
[Link]
m
Pragmatic C++ Arduino Programming, introduced above in the
Bibliography C++ section, covers the PROGMEM framework extensively.

3.
Bibliography - Software
Engineering
Software Engineering, Ian Sommerville, 3rd Edition, Addison
Wesley, 1989. Software engineering is part of the overall
development process. Although written 30 years ago, it is neither
outdated nor obsolete.
Good programming practices - There are lots of articles on good
non-language specific programming practices. Here is a good one by
Kim Moser, a programming and web consultant
[Link]

4.
Bibliography - Regular Expressions
(regex)
Regular Expressions (regex) site - [Link]
[Link] - This is a good place to get info and to access
tutorials. You can also search YouTube - it teems with tutorials on
using regexes.
Mastering Regular Expressions - Jeffrey E. F. Friedl - O'Reilly &
Associates - ١٩٩٧. It is an old reference manual, somewhat outdated
and incomplete. I had it in my bookshelf from way back; it got me
started on regular expressions.
Defensive C++ Arduino Programming, introduced above in the
Bibliography C++ section, covers regular expressions extensively.

5.
Bibliography - Awk
The AWK Programming Language - Alfred V. Aho, Peter J.
Weinberger, and Brian W. Kernighan - Addison Wesley - 1988. This
is the original Awk presentation by the authors of the language.
Official Awk manual -
[Link] - This is the official
GNU org's manual. Extensive, it is a good complement to the book
listed above.
GNU Awk - I use GNU's version of Awk downloaded from
[Link]
Defensive C++ Arduino Programming, introduced above in the
Bibliography C++ section, covers the Awk programming language
extensively.

6.
Bibliography - Perl
The main Perl documentation source is the [Link]
Web site. It is quite complete; however, its search function could
stand improvement. If you need info on Perl features, look in the Perl
Web site and then google for more info elsewhere. Here are some
useful links.

[Link]
<x>Perl Web site[AUX]Documentation on variables, functions, operators</x>
 lists Perl's built-in variables. You may find the descriptions
somewhat hard to understand. It seems to have been
designed by experts for experts.
[Link] - Perl proposes a rich set of
operators; most of these are described in this Web page.
[Link] details how to use
functions.
[Link] - Perl has a rich set of built-in
functions well described in this dedicated Web page.

Here are more sources on using Perl.

Strawberry Perl distribution - [Link] This is the
Perl distribution I use on Windows 10.
Perl online help -
[Link]
Perl book - [Link] -
You can download it free or buy it. It is a good instrument and I
recommend its acquisition if you are to do significant Perl work.
Perl Built-in variables - [Link]
Learning Perl - Randall L. Schwartz - O'Reilly & Associates, 1993.
This is an old book on Perl. It got me started when first using the
language.
Derek Banas published an excellent introductory Perl YouTube video
tutorial.
Defensive C++ Arduino Programming, introduced above in the
Bibliography C++ section, covers the Perl programming language
extensively.

7.
Bibliography - Arduino
Arduino home page - [Link] It starts here.
There are other languages you might try to program Arduino with.
This web site presents a few solutions: Which Programming
Languages Can You Use With Arduino? ([Link]
Adafruit guidelines - As usual, Adafruit does an excellent job
documenting the components they sell and the Arduino environment
in general. [Link]
Arduino's memory map - Mircea Diaconescu presents a clear view
of Arduino's memory map. He discusses heap/stack use and
PROGMEM. [Link]
Arduino memory usage - A good article by Gerd Wagner on
optimizing the Arduino memory usage -
[Link]
Usage.
Defensive C++ Arduino Programming, introduced above in the
Bibliography C++ section, covers the Arduino IDE extensively.

8.
Bibliography - AtmelStudio (now
MicrochipStudio)
AtmelStudio home page - [Link]
support/atmel-studio-7.
AtmelStudio V7 User guide -
[Link]
[Link].
Do an Internet search on Getting Started with AtmelStudio,
AtmelStudio Documentation, and MicrochipStudio (new name for
AtmelStudio). Search YouTube; it has tutorials on AtmelStudio.
Defensive C++ Arduino Programming, introduced above in the
Bibliography C++ section, covers AtmelStudio extensively.

9.
Bibliography - Visual Micro
Visual Micro (company name and product name) offers two
embedded development solutions, Arduino compatible: Visual Micro
for MicrochipStudio (AtmelStudio) and Visual Micro for Visual
Studio. Google these to get info on what they are.
Defensive C++ Arduino Programming, introduced above in the
Bibliography C++ section, covers both Visual Micro variants.

10.
Bibliography - PlatformIO
PlatformIO home page - [Link] - this is the official
site.
PlatformIO tutorial - [Link]
platformio-in-visual-studio-code-to-program-arduino. There are many
PlatformIO tutorials; do a Google and search YouTube.
Defensive C++ Arduino Programming, introduced above in the
Bibliography C++ section, covers PlatformIO extensively.

11.
Bibliography - Espruino and
JavaScript
Espruino is a new microcontroller board based on the 32-bit ARM
STM32F103xC microcontroller. See [Link] for details.

12.
Bibliography - Hardware-based
debugging
Hardware-based debugging relies on connecting debugging boards
to the microcontroller. I have identified several such boards :
ATmega328P Xplained Mini - Pin compatible with Arduino - I
successfully debugged an application with it.
[Link]
[Link]. See Hardware-based
debugging (page 119).
ATmega2560RFR2 Xplained Pro - Based on the larger
ATmega2560 (not Arduino pin compatible).
[Link]
tool/ATMEGA2560RFR2 XPLAINED PRO. User guide :
[Link]
cuments/UserGuides/Atmel-42203-RCB256RFR2-XPRO-User-
[Link].
Atmel ICE - I was not successful using this hardware debugger with
AtmelStudio; I have not tried it with the Arduino IDE V2.
[Link] Atmel ICE
user guide -
[Link]
ICE_UserGuide.pdf.
Atmel Embedded User Guide - You may want to download this
guide available from the Microchip ATmega2560RFR2 Xplained Pro
documents section.
[Link]
microcontrollers-embedded-debugger_user-[Link].
Defensive C++ Arduino Programming, introduced above in the
Bibliography C++ section, covers hardware-based debugging extensively.

13.
Bibliography - Programming
psychology
The Psychology of Computer Programming, Gerald M. Weinberg,
Computer Science Series, 1971. Not at all outdated - our brain
functions the same way it did 50 years ago.
The 10 Commandments of Egoless Programming is a concise
review of psychologically induced misguided endeavors. See
[Link]
programming and a video (search for Dave Xiang in YouTube).
Maslow's pyramid - Google and do a YouTube search on Maslow's
pyramid or how a hierarchy of needs conditions motivation.
Programmers at work, Susan Lammers, Tempus books of Microsoft
Press, 1986, 1989. A very interesting book which describes the early
programming days of industry leaders (Bill Gates, Charles Simonyi, John
Warnock...).
 A note on the book's source code
<x>Frameworks[AUX]Frameworks source code You may download the
available in Dropbox</x><x>Book's Web site (.

complete as is source code from [Link]

programming, modify code to your heart's content, and use it free of
charge at your own risk on a non-commercial or commercial basis. It
is subject to an MIT type open-source type license agreement, as
follows:.
Permission is hereby granted, free of charge, to any person
obtaining a copy of this software and associated documentation
files (the "Software"), to incorporate it inside of an executable or
other machine language component without restriction for personal
or commercial use. The Software may not be redistributed as
source code or in any recognizable human readable form in any
form whatsoever for any use whatsoever.
The Software is the property of Michèle Delsol (France), copyright
2023, USA and international. For any questions concerning use of
the software contact Michèle Delsol at [Link]
[Link].
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY
OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT
LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF
CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF
OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
OTHER DEALINGS IN THE SOFTWARE.
Note that some frameworks have benefited from extensive
development work, and they are quite solid; others are in their
infancy and may be bug ridden. I do not guarantee that the code will
work. If you use these frameworks, you must place copyrights as
indicated in the headers of all the Defensive Programming
Frameworks files you use.
You will find, throughout this book, code snippets, classes,
functions... to illustrate how C++ works. They cover a lot of material.
Some of them are short and to the point; others are longer and can
be more or less complex. I have tested all of them, at least I think I
have, which means that you might find omissions, mistakes,
inaccuracies... If you happened to come across such failings, please
send me an email at cppArduino@[Link] explaining what it is you
think is wrong. I shall look into it, try to respond, and bring in
corrections for the next edition of this book, currently 1st edition.
My Web site contains the frameworks, and Awk and Perl programs
presented in these two books. To download these, go to [Link]
[Link]/c-arduino-programming (a little over 1.5 megabytes).
You will also find notes and acknowledgements as to events
concerning these two books in my Web site [Link]
 About the author
The author, Michèle Delsol, born in France, educated in the USA
(MIT - [Link]., Sc.D.), now retired and living in France, has worked in
industrial firms in South America, the USA, and France. For the last
25 years of her career, she was CEO and CTO of the company she
created. As CTO she gained experience working with Fortran, Forth,
Lisp, Java, JavaScript, Visual Basic, PHP, HTML, and C++.
Not shying from rolling up her sleeves, she is practical and dives into
hands-on work. Her passion for flying led her to build and fly her own
airplane (RV8). She also claims some artistic capabilities (the
warthog and marmoset on the covers of this two-book tandem are
hers), did some acting (theater), and is now an enthusiastic
beekeeper.
Her most recent electronics endeavor is an Arduino-based beehive
weighing system to help monitor her bees' wellbeing (full details in the
author's book Defensive C++ Arduino Programming - see
Bibliography - C/C++ programming (page ٣٣٢). It began as a back of
the envelope idea which grew into a full-fledged system. As the project
progressed, from proof-of-concept of the individual components to a
comprehensive integrated system, the application grew to more than 35 files
(15,000 lines of code). Her early development was fairly undisciplined and
incremental, which led her to too much time debugging - what was initially
an enjoyable pastime became a gruesome burden. She consequently stepped
back and researched why programmers make mistakes. She found that she
needed to adopt good programming practices and use better tools. It
brought her to switch to AtmelStudio and later to Visual Studio/Visual
Micro in lieu of the Arduino IDE, to extensively review C++, to create
frameworks to handle specific tasks, to relearn adequate Awk and Perl to
extract documentation from the source files, to learn regular expressions,
and to undertake other useful programming chores.
This experience led her to write extensive notes on the material she covered
to keep track of things as the literature and the Internet were lacking in
resources for amateur programmers who already programmed in C++. Or
rather, there was too much material, most of which was unprofessional,
verbose, and did not address the question posed directly. Ask a simple
question, get long complicated responses. These notes gradually morphed
into two books: Pragmatic C++ Arduino Programming, a reference
work to help already C++ savvy Arduino programmers avoid the
many gotchas C++ can throw at them, and Defensive C++ Arduino
Programming which presents C++ tools and frameworks to improve
programmer productivity to write efficient, robust, maintainable, and
compact Arduino applications.
Her Arduino-based beehive weighing system is not quite finished.
Future work would focus on transferring her project to an Open-
source Software team, on redesigning the PCB for SMT technology,
on implementing alternate communications to handle remote areas
where GSM is not available, on developing Web and smartphone
based user-friendly interfaces, and more. Whether these will be
undertaken remains to be seen since she has other projects she
intends to work on. Amongst the many ideas that float in her mind, a
device to detect Asian wasps hovering in front of the beehives. She
also plans to interpret a colony's activities via the sound they make.
These projects are a tall order - they imply learning and
implementing digital signal processing, digital image processing, and
AI. These should keep her busy for years to come. Needless to say,
she is never bored. Aside from the shared benefits that beekeepers
might gain from her hard-won endeavors, her ongoing satisfaction
lies in the challenge and the doing.
 Beehive weighing system
<x>Beehive weighing system[AUX]How the beehive weighing system got me into writing these two books</x><x>Beehive weighing system[AUX]Monitor weights, temperature,

Life being a collection of chance encounters,

humidity; based on RF communications and GSM board</x>

a friend of mine (Charlie H.) introduced me to beekeeping, an

immensely rewarding activity. One of my first tasks as a new
beekeeper was to monitor beehive weights via a hand-held scale.
This information is crucial to assess how bees are doing.
Another chance encounter (Daniel T.) introduced me to Arduino. It
seems that life is conditioned by chance encounters. Anyway, as
soon as I understood what I could do with Arduino, I asked myself:
Why not hack an electronic Arduino-based beehive weighing
system? On the surface the idea was simple enough: get strain
gages cannibalized from ordinary bathroom scales, throw in
opAmps, voltage regulators and a few other little gizmos. Add to that
a GSM board, a radio, and a clock and I would get the beehives'
weights, temperatures, humidity, and battery voltages on my cell
phone. Simple! Nothing to it! Well, not quite.
To make a long story short: beekeeping + Arduino + C++ = electronic
Arduino-based beehive weighing system = challenge. Little did I
know how much I did not know and just how much work would be
required to bring this project to fruition. Having defined how to get
the data from the beehives, and how to get them into my cell phone,
I trudged along, slowly, and painfully. I grossly underestimated the
work and amount of learning required to get the system to function
as I would like it to.
My electronic know-how being limited to Ohm's law, the challenge
proved to be daunting. Being retired, with nothing in particular to do
aside from keeping fit, maintaining social ties, and being a law-
abiding citizen, this project quickly became a full-time job. I was
never bored. I had to crash course transistors, opAmps and voltage
regulators. Discovering this world was fun. Getting an NPN/PNP
transistor-based latch to work, and understanding why it worked,
brought me intense satisfaction.
As for Arduino, what a discovery! The microcontroller and the
Arduino IDE are fantastic and cost next to nothing. My enthusiasm
was immediate. I first got breadboard proof-of-concept prototypes
working. I got strain gages which I pulled out of an electronic
bathroom scale to deliver a signal via an opAmp. Radio
communications were based on XBee (Zigbee protocol). I chose the
DS3231 RTC clock to synchronize the units to minimize wakeup
time. An A6 GSM board established two-way SMS communications
with my cell phone.

Beehive weighing system architecture

As I became savvier with the electronic components I could use,
system characteristics slowly percolated out of a host of possibilities:

Parent-child architecture - I opted for a parent/child star

network system as opposed to independent beehive modules,
each reporting directly. Children would be on beehives. They
generate data and send it to the parent (coordinator) which
would be near-by. It would collate data from the beehives and
send the info to my smartphone.
<x>Beehive weighing system[AUX]Monitor weights, temperature, humidity; based on RF communications and GSM board</x>RF

communications - The coordinator and beehives communicate

with each other via RF communications, no wires between the
beehives. I chose a ZigBee based system (XBee).
Batteries only - Since beehives are in remote locations, they
require electrical autonomy, making batteries necessary
(١٨٦٥٠ LiIon). I rejected the use of solar panels since they
would be conspicuous, obviously valuable items. There was
every likelihood that they get stolen.
Cost - If the system were to become a commercially viable
product, cost would be an issue.
Ease of use - The system needed to be flexible: When to
weigh and how often, get data to trigger notifications on
specific events, etc.
<x>Beehive weighing system[AUX]Monitor weights, temperature, humidity; based on RF communications and GSM board</x>Interface with

the outside world - I had to decide how to interface with the

system (Firefox, Lora, WiFi, Bluetooth, cell phone). I chose an
 A٦ GSM board to communicate with my cell phone as it
seemed to be the easiest, most practical, and immediate
solution.

Given three years of hindsight, I must say that the electronic aspect
of the project was indeed easy. However, the C++ program to
manage the system proved to be a lot more complex than
anticipated. I was no longer inside an .ino sketch mindset - direct
programming from mind to computer. The application grew
inexorably to 35 files of code at last count. I had to improve my C++
know-how, develop new techniques, and change my mindset -
professionalize my work.
As for my know-how, I had C++ experience. I poured over Kernighan
and Ritchie's The C programming Language. I then went on to
Bjarne Stroustrup's The C++ Programming Language. But my
experience was rusty. I still have early editions of these books; I
consider them collectors' items.
I gradually improved and organized my programming by various
means: C/C++ know-how, good programming practices, frameworks,
and better IDEs (AtmelStudio, Visual Studio + Visual Micro,
PlatformIO). These helped me attain the desired objectives:
productivity, application maintainability and robustness, speed and
compactness.
I have finished the first version of my Arduino-based beehive
weighing system. The next phase is to run it through its paces on
several beehives and establish its resistance to weather. After that, if
I were to make it into a commercially viable product, I would need to
undertake fairly extensive work, incorporate less expensive
components, use a cheaper radio, create alternative interfaces to the
outside world, and use SMT based PCBs to lower costs.

Where to from now

The current version of the beehive is functional however, several
slight problems remain:

The XBee radio modules are far too expensive ($30.00

apiece). Current work centers around using
 BRF24L01modules. They come in at around less than $3
apiece.
Using a cell phone to get data is cumbersome because of the
limitations of an SMS message and costly because of the
requirement to pay for SIM cards.
A friendly Internet based user interface to visualize data and
schedule jobs needs to be put together.
Envision migrating from through-hole to surface mount
technology (SMT).

As soon as these books get published, I shall resume work on the

beehive weighing systems.
 Index table

<ALSO>@array[AUX]Perl (arrays)</ALSO><ALSO>$File handle[AUX]Perl (file handles)</ALSO><ALSO>$ProjectDir, $TargetDir, $TargetName[AUX]AtmelStudio (build)

</ALSO><ALSO># sign[AUX]Perl and Awk comment symbol, same as C++ style //</ALSO><ALSO>#define[AUX]Macros</ALSO><ALSO>#if, #ifdef, #ifndef,
#endif[AUX]Macros and Project Files Framework</ALSO><ALSO>#ifndef and #include[AUX]Macros and Header files</ALSO><ALSO>.h files[AUX]Header files (.h)</ALSO>
<ALSO>.hex files[AUX]bootloader</ALSO><ALSO>.ino[AUX]Arduino (build) - application entry file (sketch)</ALSO><ALSO>:: [AUX]DOS batch files</ALSO><ALSO>{ }
[AUX]curly braces</ALSO><ALSO>-Wlint flag[AUX]Awk (command line)</ALSO><ALSO>-Wundef[AUX]AtmelStudio (build)</ALSO><ALSO>Aardvark[AUX]Arduino,
AtmelStudio, Visual Studio/Visual Micro, PlatformIO(interoperability)</ALSO><ALSO>Action-blocks[AUX]Awk (rules)</ALSO><ALSO>Application code size[AUX]Code
size</ALSO><ALSO>ArduinoBuilder[AUX]Code::Blocks</ALSO><ALSO>ArduinoDev[AUX]Code::Blocks</ALSO><ALSO>arg1 arg2, etc.[AUX]Perl (command line)
</ALSO><ALSO>ARGC, ARGV, ARGIND[AUX]Awk (command line)</ALSO><ALSO>ARM (Advanced Risk Machine)[AUX]Microcontrollers</ALSO>
<ALSO>Arrays[AUX]Awk arrays and Perl arrays</ALSO><ALSO>Atest.h/.cpp[AUX]File organization</ALSO><ALSO>ATmega2560[AUX]Hardware setup</ALSO>
<ALSO>ATmega328P-Xmini, ATmega2560RFR2 Xplained Pro, Atmel ICE[AUX]Debugging hardware-based</ALSO><ALSO>AtmelStudio project and
solution[AUX]AtmelStudio (file management)</ALSO><ALSO>Avoiding bugs[AUX]Bugs (avoiding them)</ALSO><ALSO>AVR[AUX]Microcontrollers</ALSO>
<ALSO>avrdude[AUX]Build toolchain</ALSO><ALSO>Awk (Awk vs. Perl)[AUX]Perl (Awk vs. Perl)</ALSO><ALSO>Batch files[AUX]DOS batch files</ALSO>
<ALSO>BEGIN [AUX]Awk and Perl (build)</ALSO><ALSO>Bookmarks[AUX]AtmelStudio (editor)</ALSO><ALSO>Bootloader[AUX]Build toolchain</ALSO>
<ALSO>Bottom of stack[AUX]Memory (use)</ALSO><ALSO>Bottom-up design[AUX]Object-oriented programming</ALSO><ALSO>Build options[AUX]AtmelStudio and
Arduino IDE (build)</ALSO><ALSO>Breadboards[AUX]Hardware setup</ALSO><ALSO>C++ exception handling[AUX]Exception handling (C++) and Pseudo exception
handling</ALSO><ALSO>C++ Editor[AUX]Build toolchain</ALSO><ALSO>catch, try, throw[AUX]Exception handling (C++) and Pseudo Exception Handling</ALSO>
<ALSO>Capture[AUX]Awk and Perl (regex), Regex (captures/groupings)</ALSO><ALSO>Character class[AUX]Regex (general)</ALSO><ALSO>Chunks of data[AUX]Data
Packets Framework</ALSO><ALSO>CLion[AUX]PlatformIO</ALSO><ALSO>[Link][AUX]DOS box</ALSO><ALSO>Code collapsing[AUX]Arduino IDE and AtmelStudio
(editor)</ALSO><ALSO>Code thrashing[AUX]Bugs</ALSO><ALSO>Collapsing code[AUX]Arduino IDE/AtmelStudio (editor)</ALSO><ALSO>Color coding[AUX]Arduino
IDE/AtmelStudio (editor)</ALSO><ALSO>Command box[AUX]DOS box</ALSO><ALSO>Comments[AUX]Program Documentation Framework</ALSO><ALSO>Common
sense[AUX]Good programming practices</ALSO><ALSO>Compiler[AUX]Build toolchain, Arduino, AtmelStudio, Visual Studio/Visual Micro, PlatformIO(build)</ALSO>
<ALSO>Conditional inclusions[AUX]Macros</ALSO><ALSO>Conditions[AUX]Awk (rules)</ALSO><ALSO>Contiguous heap[AUX]Memory (heap contiguous)</ALSO>
<ALSO>Crash[AUX]Bugs</ALSO><ALSO>Critical available memory and critical reporting macros[AUX]Memory (monitoring functions), Macros</ALSO><ALSO>Curly
braces[AUX]Arduino IDE and AtmelStudio (editor), Good programming practices</ALSO><ALSO>Debugging print-based[AUX]Print-based Debugging Framework</ALSO>
<ALSO>debugWire[AUX]Debugging hardware-based Atmega328P</ALSO><ALSO>default case in switch statement[AUX]Bugs, Good programming practices, Error Reporting
Framework, Misdoings</ALSO><ALSO>delete[AUX]Perl (arrays)</ALSO><ALSO>Dependencies[AUX]Header files (.h)</ALSO><ALSO>Development tools[AUX]Arduino,
AtmelStudio, Visual Studio, Visual Micro, PlatformIO, VS Code, Code::Blocks, MPLAB</ALSO><ALSO>Documentation templates[AUX]Program Documentation
Framework</ALSO><ALSO>Editor C++[AUX]Arduino, AtmelStudio, Visual Studio, VS Code, PlatformIO (editor), Notepad++, VS Code</ALSO><ALSO>EEPROM (Electrically
Erasable Programmable Read Only Memory)[AUX]Memory (pools)</ALSO><ALSO>END[AUX]Awk and Perl (build)</ALSO><ALSO>Error handling[AUX]Validate
data</ALSO><ALSO>Error reporting macros[AUX]Macros and Print-based Debugging Framework</ALSO><ALSO>Event-based programming[AUX]Arduino (build - loop
function)</ALSO><ALSO>Events[AUX]Memory Management Framework and Event Storage Framework</ALSO><ALSO>Flash memory[AUX]Memory (pools)</ALSO>
<ALSO>exit[AUX]Awk (built-in variables and functions)</ALSO><ALSO>External tools[AUX]AtmelStudio (upload)</ALSO><ALSO>F() macro[AUX]Macros, Code size,
Misdoings</ALSO><ALSO>FDFBC class[AUX]Format Driven float to byte Conversion Framework</ALSO><ALSO>Fields[AUX]Awk (general)</ALSO><ALSO>File
handles[AUX]Perl (file handles)</ALSO><ALSO>FILENAME[AUX]Awk (built-in variables and functions)</ALSO><ALSO>Flash memory[AUX]Memory (pools)</ALSO>
<ALSO>Flow diagrams[AUX]Algorithm Test Framework</ALSO><ALSO>foreach[AUX]Perl (build)</ALSO><ALSO>Format document[AUX]AtmelStudio (editor)</ALSO>
<ALSO>fpermissive[AUX]-fpermissive (compiler options) (first page of index table)</ALSO><ALSO>Frameworks[AUX]Organizational, Data handling, Specialized
Frameworks</ALSO><ALSO>Fragmented heap[AUX]Memory (heap fragmented)</ALSO><ALSO>Freematics[AUX]Code::Blocks</ALSO><ALSO>Function call
nesting[AUX]Memory (use)</ALSO><ALSO>Function level macros[AUX]Print-based debugging</ALSO><ALSO>Function template[AUX]Function Creation
Framework</ALSO><ALSO>Functions[AUX]Awk and Perl (variables and functions)</ALSO><ALSO>Gerber files[AUX]Hardware setup; Eagle (electronics design)</ALSO>
<ALSO>getline[AUX]Awk (built-in variables and functions)</ALSO><ALSO>Gibberish[AUX]Bugs (avoiding them, causes, cures)</ALSO><ALSO>Glitches[AUX]Bugs
(avoiding them, causes, cures)</ALSO><ALSO>Greediness[AUX]Regex (greediness)</ALSO><ALSO>Groupings[AUX]Regex (captures/groupings)</ALSO><ALSO>Hardware-
based debugging[AUX]Debugging hardware-based</ALSO><ALSO>Hardware[AUX]Build toolchain</ALSO><ALSO>Hashes[AUX]Perl (variables) identified by % sign</ALSO>
<ALSO>Heap[AUX]Memory use</ALSO><ALSO>High voltage reset (HVPP )[AUX]Debugging hardware-based</ALSO><ALSO>Holes[AUX]Memory (fragmented heap)
</ALSO><ALSO>IDEs[AUX]Arduino, AtmelStudio, Visual Studio, Visual Micro, VS Code, PlatformIO, Code::Blocks, MPLAB</ALSO><ALSO>IDEs[AUX]Which IDE to work
with?</ALSO><ALSO>Image[AUX]avrdude (.hex file)</ALSO><ALSO>Import Arduino project[AUX]Interoperability</ALSO><ALSO>Incremental
programming[AUX]Think</ALSO><ALSO>index[AUX]Awk (strings)</ALSO><ALSO>Input record separator[AUX]Awk and Perl (built-in variables)</ALSO><ALSO>In-system
programmer[AUX]avrdude (ISP)</ALSO><ALSO>Intelligent C++ Editor[AUX]Build toolchain</ALSO><ALSO>Intellisense[AUX]AtmelStudio (editor)</ALSO>
<ALSO>Interdependencies[AUX]Header files (.h)</ALSO><ALSO>Interoperability[AUX]AtmelStudio (interoperability) and PlatformIO (interoperability)</ALSO><ALSO>ISP
(in-system programmer)[AUX]avrdude</ALSO><ALSO>jmp_buf[AUX]Pseudo Exception Handling Framework</ALSO><ALSO>join[AUX]Perl (string)</ALSO><ALSO>json
files[AUX]VS Code .json files</ALSO><ALSO>JTAG[AUX]Debugging (hardware-based ATmega2560)</ALSO><ALSO>KISS principle[AUX]Good programming
practices</ALSO><ALSO>Know your tools[AUX]Good programming practices</ALSO><ALSO>Largest possible allocation[AUX]Memory (fragmented heap)</ALSO>
<ALSO>Laziness[AUX]Regex (greediness)</ALSO><ALSO>length[AUX]Awk (strings), Awk (arrays)</ALSO><ALSO>Libraries[AUX]Arduino IDE (build)</ALSO>
<ALSO>Line of text[AUX]Built-in variables (Perl is $ _ ; Awk is $0)</ALSO><ALSO>Linear thinking[AUX]Think</ALSO><ALSO>Link time optimization[AUX]Code
size</ALSO><ALSO>Linker[AUX]Build toolchain, Arduino, AtmelStudio, Visual Studio, PlatformIO(build)</ALSO><ALSO>Lint like inspect utility[AUX]PlatformIO (general),
Misdoings (Perl program finds program flaws)</ALSO><ALSO>Literal characters[AUX]Regex (metacharacters)</ALSO><ALSO>Log events[AUX]Memory Management
Framework and Event Storage Framework</ALSO><ALSO>Logic testing[AUX]Algorithm Test Framework</ALSO><ALSO>Logical statements[AUX]Awk (rules)</ALSO>
<ALSO>longjmp[AUX]Pseudo Exception Handling Framework</ALSO><ALSO>Look ahead and look behind[AUX]Regex (look ahead/look behind)</ALSO>
<ALSO>loop[AUX]Arduino IDE (build)</ALSO><ALSO>main function[AUX]Arduino, AtmelStudio, Visual Studio, PlatformIO(build)</ALSO>
<ALSO>[Link][AUX]PlatformIO application entry file</ALSO><ALSO>Make[AUX]Build toolchain</ALSO><ALSO>Manage data[AUX]Data Handling Frameworks</ALSO>
<ALSO>Manage files[AUX]Project Files Framework</ALSO><ALSO>Map mode[AUX]AtmelStudio (editor)</ALSO><ALSO>Masks[AUX]Bitfield Storage Framework and
Event Storage Framework</ALSO><ALSO>match[AUX]Awk (strings)</ALSO><ALSO>MaximumPossibleAlloc[AUX]Memory (heap fragmented)</ALSO><ALSO>Memory use
macros[AUX]Macros</ALSO><ALSO>Mental condition[AUX]Think</ALSO><ALSO>Metacharacters[AUX]Regex (metacharacters)</ALSO>
<ALSO>MicrochipStudio[AUX]AtmelStudio (general)</ALSO><ALSO>Microsoft Disk Operating System[AUX]DOS box</ALSO><ALSO>Milestone
macros[AUX]Macros</ALSO><ALSO>Mind to keyboard[AUX]Think - incremental programming</ALSO><ALSO>Mindset[AUX]Good programming practices</ALSO>
<ALSO>Naming conventions[AUX]Good programming practices</ALSO><ALSO>new (overloaded)[AUX]Operator overloading</ALSO><ALSO>next[AUX]Awk (built-in
variables and functions) and Linked List Framework</ALSO><ALSO>Olimex hardware debugger[AUX]Debugging hardware-based</ALSO><ALSO>Other
IDEs[AUX]Code::Blocks and MPLAB</ALSO><ALSO>Outlining[AUX]AtmelStudio (editor)</ALSO><ALSO>Packages[AUX]Perl (build)</ALSO><ALSO>Parameter default
initializations[AUX]Avoiding bugs</ALSO><ALSO>Parameter validation[AUX]Golden rules</ALSO><ALSO>Pattern matching[AUX]Awk, Perl, and Regular
expressions</ALSO><ALSO>PCB (Printed Circuit Board)[AUX]Hardware setup; Eagle</ALSO><ALSO>Peripheral thinking[AUX]Think</ALSO><ALSO>permissive[AUX]-
fpermissive (first page of index table)</ALSO><ALSO>PIC[AUX]Microcontrollers</ALSO><ALSO>Plain text matches[AUX]Regex (general)</ALSO><ALSO>Plan your work
offline[AUX]Think</ALSO><ALSO>pop[AUX]Perl (arrays)</ALSO><ALSO>Preprocessor[AUX]Build toolchain, Macros</ALSO><ALSO>Printed Circuit Board (PCB)
[AUX]Hardware setup</ALSO><ALSO>Program logic[AUX]Algorithm Test Framework</ALSO><ALSO>ProjectDir[AUX]AtmelStudio (build)</ALSO><ALSO>Project
directory[AUX]Interoperability</ALSO><ALSO>Prototype board[AUX]Hardware setup</ALSO><ALSO>push[AUX]Perl (arrays)</ALSO><ALSO>qw// and qq{}[AUX]Perl
(string)</ALSO><ALSO>Radio transmission constraints[AUX]Format Driven float to byte Conversion Framework</ALSO><ALSO>Random Access Memory (RAM)
[AUX]Memory (use)</ALSO><ALSO>Redirection[AUX]DOS box</ALSO><ALSO>Refactoring[AUX]AtmelStudio (editor)</ALSO><ALSO>Regex groupings[AUX]Regex
(captures/groupings)</ALSO><ALSO>Regular expressions (regex)[AUX]Regex</ALSO><ALSO>Return values validation[AUX]Golden rules</ALSO><ALSO>Rules[AUX]Awk
(rules)</ALSO><ALSO>say feature[AUX]Perl (build)</ALSO><ALSO>scalar[AUX]Perl (variables) identified by $ sign</ALSO><ALSO>Scope[AUX]Awk (variables and
functions)</ALSO><ALSO>Scrollbars[AUX]AtmelStudio (editor)</ALSO><ALSO>Search and replace[AUX]AtmelStudio (find/replace)</ALSO><ALSO>Segmentation
faults[AUX]Arduino IDE (caveats) and AtmelStudio (caveats)</ALSO><ALSO>Sensor data conversion[AUX]Format Driven float to byte Conversion Framework</ALSO>
<ALSO>Serial communications[AUX]Format Driven float to byte Conversion Framework</ALSO><ALSO>Serial terminal[AUX]Build toolchain, Arduino IDE (editor) and
AtmelStudio (editor)</ALSO><ALSO>setjmp[AUX]Pseudo Exception Handling Framework</ALSO><ALSO>setup[AUX]Aardvark, Interoperability, and Arduino IDE,
AtmelStudio and PlatformIO (build)</ALSO><ALSO>shift[AUX]Perl (arrays)</ALSO><ALSO>[Link][AUX]AtmelStudio (interoperability)</ALSO><ALSO>Simulate
Awk[AUX]Perl (simulate Awk)</ALSO><ALSO>Size specifier[AUX]enum</ALSO><ALSO>SMT (surface mount technique)[AUX]Hardware setup</ALSO>
<ALSO>Solution[AUX]AtmelStudio (file management)</ALSO><ALSO>Specialized macros[AUX]Macros</ALSO><ALSO>Spell-checker[AUX]AtmelStudio (editor)</ALSO>
<ALSO>split[AUX]Perl (string)</ALSO><ALSO>SRAM (RAM)[AUX]Memory (use)</ALSO><ALSO>Stack overflow[AUX]Memory use</ALSO>
<ALSO>Startup[AUX]avrdude (bootloader)</ALSO><ALSO>strict[AUX]Perl (build)</ALSO><ALSO>String interpolation[AUX]Perl strings</ALSO><ALSO>substr[AUX]Awk
(strings)</ALSO><ALSO>Surface Mount Technology SMT[AUX]Hardware setup</ALSO><ALSO>switch feature[AUX]Perl (build)</ALSO><ALSO>switch statement default
case[AUX]Bugs, Good programming practices, Error Reporting Framework, Misdoings</ALSO><ALSO>Syntax checking[AUX]AtmelStudio (editor)</ALSO><ALSO>Tab
mode[AUX]AtmelStudio (editor)</ALSO><ALSO>TargetDir and TargetName[AUX]AtmelStudio (build)</ALSO><ALSO>Task creation[AUX]Code skeletons</ALSO>
<ALSO>Task wrap-up phase[AUX]Wrap-up phase</ALSO><ALSO>Text find/replace[AUX]Macros</ALSO><ALSO>Text match operator[AUX]Perl (text match operator =~)
</ALSO><ALSO>Text matches[AUX]Regex (general)</ALSO><ALSO>Through-hole technology[AUX]Hardware setup</ALSO><ALSO>throw, try, catch[AUX]Exception
handling (C++) and Pseudo Exception Handling</ALSO><ALSO>Tokens[AUX]Awk (terminology)</ALSO><ALSO>Top-down design[AUX]Object-oriented
programming</ALSO><ALSO>try, throw, catch[AUX]Exception handling (C++) and Pseudo Exception Handling</ALSO><ALSO>Type checking leniency[AUX]Bugs</ALSO>
<ALSO>Undefined references[AUX]AtmelStudio (caveats)</ALSO><ALSO>unshift[AUX]Perl (arrays)</ALSO><ALSO>Update your C++ skills[AUX]Good programming
practices</ALSO><ALSO>Uploader[AUX]Build toolchain, avrdude, AtmelStudio build</ALSO><ALSO>use[AUX]Perl (build)</ALSO><ALSO>Variables[AUX]Awk and Perl
(variables and functions)</ALSO><ALSO>Variables[AUX]Initializations</ALSO><ALSO>VAssist[AUX]AtmelStudio editor and documentation</ALSO>
<ALSO>Verbose[AUX]avrdude and AtmelStudio (import Arduino project)</ALSO><ALSO>Version control[AUX]PlatformIO (general)</ALSO><ALSO>Visibility[AUX]Awk
(variables and functions)</ALSO><ALSO>Visibility[AUX]Awk (variables and functions)</ALSO><ALSO>Visual Studio Code[AUX]VS Code</ALSO>
<ALSO>vMicro[AUX]Visual Micro and Debugging serial</ALSO><ALSO>Warnings[AUX]Awk (command line) and Perl (build)</ALSO><ALSO>Web site[AUX]Book's Web
site</ALSO><ALSO>while[AUX]Perl (arrays)</ALSO><ALSO>Wundef[AUX]AtmelStudio (import Arduino project)</ALSO><ALSO>Workspace font size[AUX]Arduino IDE
and AtmelStudio (editor)</ALSO>

Symbols
?
Regex lazy search 200
/*...*/
C-style comments collapse supported by AtmelStudio and by the Arduino IDE 57
\
pass by reference Perl operator 329
%
Perl provides three variable types: $scalars, %hashes, @arrays 331
+=
AddHive illustrates += operator overloading 186
=
= instead of == in an if or while, == instead of = in an - assignment 117
=~
Perl pattern matching operator 220, 223
$ (dollar sign)
$0 - Awk: record just read in, Perl: program name (also $PROGRAM_NAME) 218, 314,
329
Awk $ 1, etc. contains tokens of record just read in; Perl contains captures from regex 314,
330
Perl $. alternate $INPUT_LINE_NUMBER or $NR, line # of file being processed 329
Perl $, alternate $OUTPUT_FIELD_SEPARATOR, or $OFS separates fields from each
other 329
Perl $\ alternate $OUTPUT_RECORD_SEPARATOR (default newline) 329
Perl $; alternate $SUPSEP - index separator, comma by default 330
Perl $ARG - same as $ _, is the complete record (line of text) 329
Perl $ARGV array contains current file name when reading from <> array 329
Perl - $ _ contains the entire line that has just been read 326, 329
Perl $INPUT_RECORD_SEPARATOR, or $RS, or $/ - default \n 218, 324, 329
Perl $NR alternate $INPUT_LINE_NUMBER or $., line # of file being processed 329
Perl $OFS alternate $OUTPUT_FIELD_SEPARATOR or $, separates fields from each
other 329
Perl $RS alternate $INPUT_RECORD_SEPARATOR or $/ - input record separator, default
\n 329
 Perl provides three variable types: $scalars, %hashes, @arrays 222, 331
$File handle See Perl (file handles)
$ProjectDir, $TargetDir, $TargetName See AtmelStudio (build)
3D Printing
3D printing starts with CAD tool 228
Fused deposition modeling, Stereolithography, Selective layer sintering 229
I chose a Zortrax 200M 3D printer 228
Several 3D printing techniques available 228
Standardized file format .stl for 3D printing 228
Use Fusion 360 to design part 228
3D Printing See Fused deposition modeling FDM
@ARGV (Perl array)
Perl arg1 arg2... command line arguments stored in @ARGV array 320, 329
@array See Perl (arrays)
@ (at symbol)
Perl @_ alternate @ARG array passes arguments to a function 330
Perl @ARG or @_ - array passes arguments to a function 330
Perl provides three variable types: $scalars, %hashes, @arrays 331
__brkval
Address of begin heap instead of __flp when there are no holes 297
#define See Macros
__flp
Address of begin heap when there are holes 297
-flto flag
Code size depends on -flto flag; reduce by 40% 39, 62, 65, 232, 233
-fpermissive
AtmelStudio (import Arduino project) - add flag to C/C++ compiler options 43
Converts compiler errors into warnings, build completes 43
Included by default in Arduino IDE, PlatformIO, not AtmelStudio 43
%hash
Perl provides three variable types: $scalars, %hashes, @arrays 222
.h/.cpp files
Arduino loads all project directory .h/.cpp file into the editor 33
.hex files See bootloader
.h files See Header files (.h)
#ifdef...#endif
Collapsing supported by AtmelStudio 57
#if, #ifdef, #ifndef, #endif See Macros and Project Files Framework
#ifndef and #include See Macros and Header files
.ino See Arduino (build) - application entry file (sketch)
{ } See curly braces
:: See DOS batch files
# sign See Perl and Awk comment symbol, same as C++ style //
-Wlint flag
Awk - use -Wlint command line flag to display errors 216, 312, 313
-Wlint flag See Awk (command line)
-Wundef See AtmelStudio (build)
A
Aardvark
Aardvark() contains original setup code, key to interoperability; in Aardvark.h/.cpp 24, 26,
153
Aardvark See Arduino, AtmelStudio, Visual Studio/Visual Micro, PlatformIO(interoperability)
Action-blocks See Awk (rules)
Algorithm Test Framework
Define execution paths and controlling parameters, then fill in real code 174, 272
Formal procedure to test program logic; logic and work done are distinct concepts 174
Process: decision tree several levels deep 175
Start with flow diagrams to peg down algorithm's logic 174, 272
Two-pronged process: methodology (plan on paper and then code), and thinking (top down
how one thinks and bottom up fill-in the details) 2
Unravel add beehive logic with flow diagram 272
Application code size See Code size
Application entry files
Arduino: .ino file, AtmelStudio and Visual Studio/Visual Micro, PlatformIO: [Link] 152
Arduino (build)
Arduino, AtmelStudio, PlatformIO, Visual Micro use the GNU C++ compiler 8, 31
Arduino project's name is that of .ino file 32
avrdude uploads the executable into the microcontroller 31
Build command details 234
Compile/upload uses bootloader to upload program 45
-flto and -fpermissive flags included by default in Arduino IDE project properties 232
.ino file is Arduino's entry file (C++ source code) 32, 152
Installs basic and third-party libraries 31
Loads all project directory .h/.cpp file into the editor 33
main is generated behind the scenes, calls setup and loop 33
setup and loop - mandatory functions for Arduino programs; leave loop blank 33
Specify microcontroller and board, changing them transparent to programmer 76
Suggestion: do not use loop, do all inside setup 33
ArduinoBuilder See Code::Blocks
Arduino (caveats)
After a while, the Arduino IDE gives up, undo ctrl-Z messes up code; fragmentation fault
blocking 36
Compiler error reporting - must manually go to file/line # to access code 60
ArduinoDev See Code::Blocks
Arduino (editor)
Button compiles and uploads application 33
Collapses curly braces, not AtmelStudio 133
Ctrl-L takes you to line # in file 61
Edit menu item changes workspace font size 65
Indentation, code collapsing not cosmetic 129
Serial monitor interfaces with board 33
Arduino (general)
Arduino chips are AVR type 119
Arduino distribution seamless installation; IDE, libraries, toolchain, serial monitor 31, 32
 Arduino tools are user-friendly and free, components cheap, community huge 11, 30
Distribution includes: tools to create program, libraries, upload, serial terminal 30, 31
For new Arduino IDE version see Arduino (version 2) 30
IDE for Atmel 8-bit and ESP32 boards, no hardware-based debugging 30
IDEs: Arduino, AtmelStudio, Visual Studio, VS Code, PlatformIO, Visual Micro 30, 96
Two versions: 1.8.19 (legacy) and V2 (new features) 30, 34
Wraps tools inside a clean, practical interface 32
Arduino (interoperability)
ARDUINO_IDE macro enables Arduino IDE 26
Interoperability between Arduino IDE, AtmelStudio, PlatformIO, Visual Micro, hassle free
27
Arduino (upload)
Arduino build output 45
Full paths of [Link] and [Link] 45
Validate verbose checkbox in options to show details of avrdude upload command 44
Arduino (version 2)
Click on compilation error in output window does not take you to file/line 35
Copying text from serial terminal problematic 34
C-Style /*...*/ comments collapsing inoperant 35
Editor widows not undockable 35
Find/replace supports regular expressions 34
#ifdef…#endif pairs collapsible but renders function not collapsible 35
Look and feel similar to legacy version 34
Major advance over version 1, long way to go 36
New features and minor problems 34
New output window: serial plotter 34
Problem loading code into ATmega328P-Xmini 35
Provides name completion on defined items 34
Refactoring does not seem to be supported any more 35
Right click to visualize function declaration 34
Serial terminal is fixed bottom pane, not separate window 35
Slow to open 34
arg1 arg2, etc. See Perl (command line)
ARGC, ARGV, ARGIND See Awk (command line)
ARM (Advanced Risk Machine) See Microcontrollers
Arrays See Awk arrays and Perl arrays
Atest.h/.cpp See File organization
ATmega328P-Xmini, ATmega2560RFR2 Xplained Pro, Atmel ICE See Debugging
hardware-based
ATmega2560 See Hardware setup
AtmelStudio (build)
$ProjectDir, $TargetDir, $TargetName define .hex file location and name 46, 67
Add -fpermissive and -flto flags compiler flags 43
Add -Wundef to detect unused macros 43
Arduino, AtmelStudio, Visual Studio/Visual Micro, PlatformIOuse the GNU C++ compiler 8
AtmelStudio upload tool parameters: identify microcontroller, define COM port, baud rate
45
Build command details 234
 External tool for upload not needed with Visual Micro 41, 44, 46
Solution Explorer - add/choose existing Item 43
AtmelStudio (caveats)
Arduino core content seems locked into device first compiled for 75, 76
avrdude - cannot open input file error message hard to see 67
avrdude fails to upload code into chip 48, 63, 66
Build fail due to inexistant file in project not in error messages 64, 73
Careful with find/replace Opened Documents 64
Code changes no effect on runtime 67
Code remains obstinately collapsed 64, 75
Code size by default 40% too large, needs -flto flag 39, 62, 65, 232, 233
Does not support curly braces collapsing 50
Error and warning messages not displayed 71
Extra */ causes indentation failure 64, 68, 69
Failure to reckon with Debug vs. Release output causes confusion 63, 67
File has 20+ functions, only two showed up in the quick access selection box 63, 68
Importing Arduino project - directory navigation bug on searching .ino file 42, 62
Indenting of if...else statement depends on where opening curly brace is placed 63, 69
Make error from missing .cpp file drowned in reams of messages 64, 71, 72
Mysteriously imports second set of header files into ArduinoCore directory 63, 68
Occasional segmentation faults are benign; no clues on causes; do rebuild to fix 64, 70, 71
Select device board does not seem relevant when creating Arduino project 42
Workspace font size not documented; - '>' and '<' become ',' and ';' on French keyboard -
tied to physical key, not to key mapping 62, 65
AtmelStudio (documentation)
Atmel (Microchip) Web site two inline doc packages: Getting started and videos 39, 78
Inline help under the Help menu; VAssist help under top level pull-down VAssist menu 78
AtmelStudio (editor)
Arduino monitor displays output 41, 48
Bookmark to navigate back and forth 51, 61
Change signature changes the name and the type of a declaration and refactors 53
Clicking on function opens a goto multiple-choice window for declarations 53
Code collapsing in AtmelStudio is outlining; indenting is formatting 54, 57, 63
Collapses classes, functions, enums, ifs, whiles, etc. and #ifdef...#endif pairs 50, 57, 133
Color coding highly customizable 50
Compiler one-click access to faulty line 51, 60
Create declaration in corresponding .h file 53
Ctrl-G takes you to line # in file 51, 61
Documentation above a function call 53
Editor displays colors: functions red, variables blue, comments pale green, etc. 54
Establish Arduino IDE code collapsing #ifdef interoperability by adding curly braces 58
Extensive user assist editor features 8, 37, 38, 49
Find reference: all uses of variable/function 53
Format document indents code entire file 69
Indentation, code collapsing not cosmetic 129
Indenting in AtmelStudio called formatting 69
Inline tasks provide multiple-choice text box to rename, create, declare, etc. 38, 50
Insert snippet such as if, while, for, etc. 54
 Intelligent window scrollbars provide quick access to various tasks 49, 52, 55
Intellisense complementary to VAssist 49
Keyboard shortcuts tied to physical keys or to symbol mapping 65
Keyboard shortcut to quickly comment/uncomment highlighted sections 51, 59
Multiple-choice quick access to functions 49, 53
Name completion contextual popup opens 50, 56
Navigate to places recently worked on 51, 61
Refactoring - rename items throughout app 51, 54, 59, 73
Right-click on function for quick access 53
Spell checks strings and comments 51, 61
Supports one-click all code collapse 57
Surround selected code with a for, if, etc. 54
Syntax checking - color goes black when there is a programming error 38, 50, 54
Unnamed code blocks improve readability, decrease RAM 58
VAssist has many help mechanisms 49
Workspace contains two splitable unpinnable work windows and Solution Explorer 49, 51
AtmelStudio (file management)
$ProjectDir, $TargetDir, $TargetName define .hex file location and name 46
AtmelStudio directories key to interoperability 25
AtmelStudio target release: Debug or Release 46
Differentiate projects (managed individually) from solutions (managed as group of projects)
39, 40, 41
Solution Explorer access project components 51
AtmelStudio (find/replace)
Careful with find/replace Opened Documents 73
Little orange patches in vertical scrollbars show find location 55
Supports whole word, regular expressions, past searches 38, 50, 55, 191
To change application wise item name, consider refactoring instead of find/replace 73
AtmelStudio (general)
AtmelStudio Arduino compatible in two versions, without and with Visual Micro 37
AtmelStudio is Atmel specific Visual Studio 37, 79
Develop with Visual Studio 2022 + Visual Micro, hardware debug with AtmelStudio +
Xplained boards 80
Free, professional grade, short learning curve, easy import of Arduino projects 37
IDEs: Arduino, AtmelStudio, Visual Studio, VS Code, PlatformIO, Visual Micro 37
Improve productivity by orders of magnitude 8
MicrochipStudio is new name - continue referring to AtmelStudio in book 37
Supports hardware-based debugging 37
Visual Micro brings AtmelStudio missing features: upload tool, serial monitor, serial
debugging 39
Visual Micro enables serial debugging w/o dedicated hardware 125
AtmelStudio (hardware-based debugging)
Develop with Visual Studio 2022 + Visual Micro, hardware debug with AtmelStudio +
Xplained boards 80, 121
Hardware-based debugging with ATmega328P-Xmini successful, with Atmel ICE not so
54, 119
YouTube Atmel ICE tutorials 122
AtmelStudio (interoperability)
 Aardvark() contains original setup code, key to interoperability 24, 26
AtmelStudio Arduino project entry file [Link] contains setup and loop 25, 152
ATMEL_STUDIO macro to enable AtmelStudio 26
AtmelStudio's project name is Arduino project name 40
AtmelStudio upload tool parameters: identify microcontroller, define COM port, baud rate
45
Copy the Arduino sketch file (.ino file) and .h/.cpp files into the AtmelStudio project files 25
External custom avrdude upload tool needed, not with Visual Micro 44
External upload avrdude tool needed 41
Importing Arduino project procedure 22, 25, 40
Interoperability between Arduino IDE, AtmelStudio, PlatformIO, Visual Micro 27, 38
AtmelStudio (keyboard shortcuts)
AtmelStudio V7 User Guide - 14 pages which itemize keyboard shortcuts 77
Standard Windows shortcuts supported 77
Workspace font size not documented - '>' and '<' become ',' and ';' on French keyboard -
tied to physical key, not to key mapping 62, 65
AtmelStudio project and solution See AtmelStudio (file management)
Avoiding bugs See Bugs (avoiding them)
avrdude
Arduino compile/upload uses avrdude to send .hex file to board, bootloader uploads it 33
Arduino verbose details upload command 44
AtmelStudio requires external custom avrdude upload tool , not with Visual Micro 41, 44
Full path of [Link] and [Link] 45
avrdude See Build toolchain
AVR See Microcontrollers
Awk (arrays)
Array indices can be integer, float, strings, similar to key into database 215, 305
Arrays are associative, mix strings and numeric values 305, 308
Arrays defined when array items used 307, 308
delete removes array items 306
length function yields number of items in one-dimensional arrays 306, 307
Loop construct for (idx in array) facilitates array traversal 306
Multi-dimensional arrays supported 305
Two-dimensional array example 308
Using multidimensional arrays is complex 306
Awk (Awk vs. C)
Array indices can be integer, float, string 215
End of statement semicolon not mandatory 215
Origins trace back to Unix, hence based on the C language 214, 312
Print statement not quite literal, requires understanding its mechanics 215
Strings are 1-based 215
Supports regular expressions 215
The # sign signals a comment to end of line 215
Two variable types: floats and strings - they are not declared, they are used 215
Variables used in functions are global 215
Your C knowhow can be a false friend 214
Awk (Awk vs. Perl) See Perl (Awk vs. Perl)
Awk (built-in variables and functions)
 $ 1, $ 2, etc. - contain tokens of record just read in, $0 contains entire record just read in
314
Command line variables particularly useful: ARGC, ARGV, ARGIND 313
exit stops processing files; END function called 314
FILENAME is file being processed, same as (ARGV[ARGIND]) 311, 313
FNR is record number of file being processed; NR is overall input record number 313
FS - fields separator (default " "); RS input record separator (default "\n") 210, 314
next, getline interrupt rules processing 312, 314
OFS output filed separator controls whether space printed when comma encountered 316
Output record separator - default is newline (ORS = "\n") 210, 316
String functions: index, length, substr, match 315
Awk (caveats)
Careful with spaces in if, while, functions, etc. parens must be against if, etc. 216
Cycling through array in C-style for loop may create unwanted array items 306
Excessive simplicity cause of gotchas 216
Functions similar to C functions, may return value of any type 214
Function variables global, parameters local 217
Individual characters are strings 216
Program particularly sensitive to spelling errors, no warnings 313
Stray line of code silently interpreted as rule 216
Strings are 1-based 217
Variables typed when used - no declarations 216
Awk (command line)
Command line: Awk -Wlint -f [Link] file1 file2 ... > [Link] 301
Contains flags, program, files to process 313
-Wlint flag displays errors 205, 301
Awk (examples)
Find macros regex: #define, #ifdef, etc. 311
Generate documentation from code comment templates, list macros and enums 202
Generate function call reference - who calls who and who gets called by whom 202
List potential misdoings e.g. = instead of == 202
Print subset of a file 203
Regex finds macros: #define, #ifdef... 211
Short database example Awk program 303
Two-dimensional array example 308
Awk (general)
Download Awk, go to SourceForge 301
Edit Awk, Perl programs with Notepad++ 209
Extract info from text files via logical statement & regex 191, 201, 204
Learning curve is shallow - nothing to it 203
Program is sequence of condition/action-blocks (rules) and functions 204, 210
Provides trig, log, type conversion, random numbers math functions 316
Uses Awk may be good for 202
Why I started with Awk, then migrated to Perl 201, 204
Your C knowhow, a false friend when learning Awk 214
Awk (regex)
Find macros regex: #define, #ifdef, etc. 311
Parens-based captures not supported - Perl does support it 203
 Regex finds macros: #define, #ifdef... 211
Awk (rules)
Action-block similar to function - contains C-like statements enclosed in curly braces 209
After loading new line, Awk tokenizes it 204, 207
BEGIN and END blocks do work at start and end of program 207, 214, 312
Comment is # sign; down to end of line 208, 310
Concatenates files into set of continuous text processed one line at a time 207
Condition based on pattern search: simple logical statements or regexes 201, 207, 310
Condition only rule prints line if condition successful 212, 311
Default record separator ORS is newline (line of text); default field separator OFS is
spaces or tabs (word in line of text) 210
Files broken up into records, records into fields (lines of text/words) 210
getline, next stops processing line 209, 312, 314
Invoke Awk from DOS box via command: program name, -w option, file names 310
Logical statements for simple conditions, regexes for complex ones, or mix of the two 212,
213
Multiple file processing passes possible 302
No condition rule is always true - action-block invoked 212, 311
Print equivalent to C++'s [Link] 315
Process text files with sequence of condition/action-blocks (rules) 201, 206
Program is sequence of condition/action-blocks (rules) and functions 201, 204
Provides arithmetic, increment, assignment, unary, logical, text match operators and
conditional expression 314
Provides useful built-in functions and built-in variables 208
Rule could be condition only or action-block only 207
Rules use fields $ 1, $ 2, etc. from line tokenization 211, 311
Supports standard C program flow control operators if, while, for, etc. 314
Variables OFS, ORS control print's behavior 315
Awk (strings)
Similar to C++'s String class, char array 315
String functions: index, length, substr, match 315
Strings are 1-based 215
Awk (terminology)
Rules, conditions, action-blocks, statements, fields... 206
Awk (variables and functions)
Function declaration starts with 'function' 214
Functions similar to C functions, may return value of any type 208, 214, 312
Function variables global, parameters local 304
Variables created on the fly when first used 313
-Wlint flag warns nonexistent return 312
B
Batch files See DOS batch files
Beehive weighing system
Example of DataGroup Framework use 268
Frameworks required to improve beehive weighing system program 149
How the beehive weighing system got me into writing these two books 1, 341
 Monitor weights, temperature, humidity; based on RF communications and GSM board
341, 342
Test beehive weighing system execution paths with Algorithm Test Framework; unravel
add beehive logic with flow diagram 272
BEGIN See Awk and Perl (build)
Bitfield Storage Framework
BitfieldStorage class provides bitfield storage via inheritance 262
BitfieldStorage class requires info on variables stored in VarCharacteristics; uses bit-masks
to operate on bytes 168, 264
Bitfield storage example, 3 arrays: JobData, VarsMinMax, JobLabels data arrays 167, 168,
263
Job class inherits from the Bitfield class 167, 171
Optimize data storage via bitfield packed variables 166, 261
Put bitfields in a structure to access them easily via an offset from the structure's address
170
Variables accesses via enum driven generalized Get/Set functions 169, 261, 262
Bjarne Stroustrup
Suggests not using macros 143
Bookmarks See AtmelStudio (editor)
Book's Web site ([Link])
Events concerning book blogged in [Link] 338
For comments and info, please send email to cppArduino@[Link] 338
Go to [Link] for a link to download most of the code in
the two books 338
Bootloader See Build toolchain
Bottom of stack See Memory (use)
Bottom-up design See Object-oriented programming
Breadboards See Hardware setup
Bugs (avoiding them)
Apply good programming practices and adhere to Golden rules - check data, never
assume anything; do error handling 129
ATmega328P-Xmini - watch variables to find glitches 10
Careful with compiler function parameter type leniency and default initializations 140
Editor's auto indent detects bad curly braces 147
Inline comments clarify logic, fewer bugs 116
Ounce of prevention is worth pound of cure 114
switch default case missing or empty 295
Two-step approach: review code and apply systematic validity checks 116
Bugs (possible causes and cures)
Application displays gibberish, restarts, etc. 115
Apply methodology for hard to find bug, understand the logic 116
ATmega328P-Xmini - watch variables to find glitches 115, 116
Brute force approach to finding glitches; pare down application and rebuild progressively
115
First step to find glitch's cause, eliminate randomness 115
Glitches worse programming problems one can encounter; how does one fix them? 114
Strange behavior or crash - probably ran out of memory 179
Type checking leniency creates bugs due to undetected misplaced parameters 116
Build options See AtmelStudio and Arduino IDE (build)
Build toolchain
Bootloader transforms board into self ISP 15
C++ editor, make, preprocessor, compiler, linker, upload (avrdude) 13
Compiler --> source code to machine code 15
Intelligent C++ editor - AtmelStudio/Visual Studio best 14
Linker assembles machine code files + library components into executable file 15
Make - timestamp defines what to work on 14
Preprocessor enables text replacement, conditional inclusions, pasting files 15
Serial terminal - visualize program output 16
Uploader (avrdude) loads executable into microcontroller via an ISP 15
C
C++ Editor See Build toolchain
C++ exception handling See Exception handling (C++) and Pseudo exception handling
Capture See Awk and Perl (regex), Regex (captures/groupings)
catch, try, throw See Exception handling (C++) and Pseudo Exception Handling
Character class See Regex (general)
Chunks of data See Data Packets Framework
Class and Function Names Referencing Framework
Automate class and function IDs insertion 178, 278
class CandFnames does all the work 277
Framework used for event reporting 176
Identify classes/functions via IDs 176, 251
Instrumentation synchronizes class and function enum lists with names 178
MemAllocEvt uses function IDs to report with function names 183
Class Data Framework
Access data via Get/Set enum driven offsets 159, 254
CLion See PlatformIO
[Link] See DOS box
Code::Blocks
ArduinoBuilder - third-party plugin to Code::Blocks 111
Free, low learning curve C++ IDE; does not seem suited for Arduino dev 111
FreematicsBuilder - third-party plugin to Code::Blocks 111
Code collapsing See Arduino IDE and AtmelStudio (editor)
Code size
Are the IDEs as efficient code size wise 231
AtmelStudio code size by default 40% too large, needs -flto flag 39, 233
Code size depends on compiler options and on -flto flag; reduces size by 40% 232, 233,
234
Compile time code size corroborated by heap space at startup 233, 235
Develop with AtmelStudio, final release may be preferable with Arduino 233
Identical compile/link options yield different RAM/flash memory requirements 232
RAM used depends on IDE and use of PROGMEM and F() macro 232, 236, 295
Code skeletons
Start application with code skeletons 147
When creating functions, templates ensure items not forgotten 142
Code thrashing See Bugs
Collapsing code See Arduino IDE/AtmelStudio (editor)
Color coding See Arduino IDE/AtmelStudio (editor)
Command box See DOS box
Comments
Awk/Perl comment symbol is pound sign # 310
Comment closing curly braces and #endif 147
Comment! Comment! Comment! 136
Countless reasons for commenting code 9, 129, 136
Inline comments clarify logic, fewer bugs 116
Program Documentation Framework - use Awk or Perl regexes to extract comments 116,
136, 245
Comments See Program Documentation Framework
Common sense See Good programming practices
Compiler See Build toolchain, Arduino, AtmelStudio, Visual Studio/Visual Micro,
PlatformIO(build)
Conditional inclusions See Macros
Conditions See Awk (rules)
Constants
#define or const variables to define constants; centralize their location 131, 144, 145
Contiguous heap See Memory (heap contiguous)
Crash See Bugs
Critical available memory and critical reporting macros See Memory (monitoring functions),
Macros
Curly braces
AtmelStudio does not collapse curly braces; Arduino IDE does 133
Comment closing curly braces and #endif 131, 147
Extra curly braces facilitate code collapsing 133
Initialize using curly braces 131
Curly braces See Arduino IDE and AtmelStudio (editor), Good programming practices
D
DataGroup Framework
Beehive weighing system good example 268
DataGroup class does the work: header defines characteristics, body contains data 165,
266
Functions enable finding, inserting, removing records and defragment storage 165
GetRecord/WriteRecord/RemoveRecord get/set/remove data 267
Index/sequential fixed/variable length data 164, 165
Data Handling Frameworks
Bitfield Storage Framework stores data inside bitfields 261
Class Data Framework accesses class data via enums 254
DataGroup Framework supports index/sequential fixed/variable length data 265
Data Packets Framework breaks data into chunks for serial transmission 257
Event Storage Framework stores evens in contiguous byte arrays 269
Format Driven float to byte Conversion Framework reduces data size 258
Frameworks to handle data storage, data conversion, data compacting 150, 158
Linked List Framework enables any class to store values inside linked lists 270
Data Packets Framework
 Break up data into chunks/packets 161, 257
Packet has header (start code, packet size, ID, sequencing info), body (data), tail (end
code, checksum) 161
Packet is broken up into header (start code, packet size, ID, sequencing info); body (data);
tail (end code, checksum) 162
Send/receive packets via serial ports 161
Validity checks: data received and checksum on data 161, 162
Debugging hardware-based
Arduino does not support hardware-based debugging; Arduino V2 does 119
ATmega328P and ATmega2560 compatible with hardware debugging with debugWire and
JTAG respectively 119
ATmega328P-Xmini Uno pin compatible supported by AtmelStudio, success using it 10,
37, 120
ATmega328P-Xmini - watch variables to find glitches 10, 115, 116
ATmega2560RFR2 Xplained Pro - specialized hardware for RF communications, not
Arduino boards pin compatible, success using it 120, 121
Atmel ICE - Not successful debugging Uno or Mega 37, 113, 120, 122
AtmelStudio with specialized boards enable hardware-based debugging 113, 119
Bugs categorized as glitches, code thrashing, misdoings 113
Complementary tool relative to print-based debugging, requires special hardware and
compatible IDE 119, 188
Debugging hardware must be compatible with microcontroller architecture (AVR, ARM)
119, 120
Develop with Visual Studio 2022 + Visual Micro, hardware debug with AtmelStudio +
Xplained boards 121
Dragon Programmer, STK500/600 should enable fixing bad microcontroller using HVPP
124, 125
Olimex - hardware-based debugger for ESP32 119
PlatformIO supports Arduino, ESP32 hardware-based debugging, other 108
Watch corrupted variable to pause execution and find glitch's cause 115, 116, 119
Debugging print-based See Print-based Debugging Framework
Debugging serial
Visual Micro enables serial debugging w/o dedicated hardware 84, 85, 125
debugWire See Debugging hardware-based Atmega328P
default case in switch statement See Bugs, Good programming practices, Error Reporting
Framework, Misdoings
Defensive Programming
Good programming practices, plan and implement procedures and frameworks, use
professional tools 1, 4
delete See Perl (arrays)
Dependencies See Header files (.h)
Development tools See Arduino, AtmelStudio, Visual Studio, Visual Micro, PlatformIO, VS
Code, Code::Blocks, MPLAB
Documentation templates See Program Documentation Framework
DOS batch files
Batch file example 241
Batch files enable multiple DOS commands 241
Displays output or redirects to file (> and >>) 242
 ECHO prints a message in the DOS box 241
Launch Awk and Perl from batch files 241, 319
Manage batch file execution: START, TIMEOUT 240
May take parameters such as %1, %2, etc. 241
PAUSE interrupts processing, EXIT and ctrl-C closes DOS box 242
:: undocumented comment for batch files 240
DOS box
Awk invoked via DOS box - output screen or file 208
[Link] (DOS box) launched from screen or toolbar icon or run menu item 238
Copy/paste to/from other apps 242
Cryptic not found message 238, 240
DOS box in c:\Windows\System32\[Link] 238
Microsoft created DOS operating system for the IBM personal computer 237
Other details you ought to be familiar with 242
Prompt looks like this: D:\Dev\Atmel> 238
Recalcitrant app? TaskManager (ctrl+alt+del) 242
Text user interface (old Unix type console) 237
Windows DOS box for text interface programs 237
DOS commands
basic set: CD, DIR, DEL, EXIT, MKDIR, COPY, XCOPY, PAUSE, ECHO, REM 240
CD command to navigate to other directories 239
Do HELP for DOS commands 239
PATH displays or sets environment variables 240
..\ to backtrack one directory, .\ to proceed from current directory 239
Downloads. See Book's Web site to download code
E
Eagle software
Electronic schematic and Gerber files; now in Fusion 360 19, 227
Editor C++ See Arduino, AtmelStudio, Visual Studio, VS Code, PlatformIO (editor),
Notepad++, VS Code
EEPROM (Electrically Erasable Programmable Read Only Memory) See Memory (pools)
EGO trap
Avoid proving to yourself that you are an absolute genius 147
Exploit C++ features sparingly 131, 147
Psychological factors extensively covered in companion book Pragmatic C++ Arduino
Programming 147
END See Awk and Perl (build)
enums
class in enum declaration is scope specifier 145
enum lists begin with start, finish with end 145
enums better than #define macros 131, 145
enums synchronize multiple arrays 145, 160
Program Documentation Framework - enums are often an alternative to macros 156
Specifier defines size of individual enum items 145
Error handling See Validate data
Error Reporting Framework
enums synchronize events with array strings 186
 ErrorEvt and ErrorList inherit from LinkedItem and LinkedList respectively 285
ErrorEvt class stores info on event and links them as linked list (ErrorList) 185, 284
Handles error logging and reporting 185
switch default case missing or empty 185, 295
Error reporting macros See Macros and Print-based Debugging Framework
ESP32
ESP32 chips are ARM devices 119
PlatformIO supports Arduino, ESP32 hardware-based debugging, other 108
Supported by Arduino IDE and PlatformIO, not by AtmelStudio 30, 98
Event-based programming See Arduino (build - loop function)
Events See Memory Management Framework and Event Storage Framework
Event Storage Framework
256 events (8-bits), 244 event (7-bits) 171, 269
enum driven Get/Set functions get/set events packed in bytes 171, 269
Masks used to manage individual events 270
Predefined true/false conditions stored in 1 bit per event byte array, saves RAM 171, 269
Specific classes could be created to handle distinct event categories 172
Exception handling (C++)
Pseudo Exception Handling Framework - alternative to C++'s exception handling 183
try establishes landing point, throw returns back to landing point, catch handles the
problem 184
exit See Awk (built-in variables and functions)
External tools See AtmelStudio (upload)
F
False friends
Your C knowhow, a false friend when learning Awk 214
Your C knowhow, a false friend when learning Perl 218, 224
FDFBC class See Format Driven float to byte Conversion Framework
Fields See Awk (general)
File handles See Perl (file handles)
FILENAME See Awk (built-in variables and functions)
File organization
Aardvark() contains original setup code, key to interoperability 152
Class definitions in ClassSpecific.h/.cpp files 152
Do your testing in the ATest() function 153
Entry files - Arduino IDE: .ino file; AtmelStudio: [Link]; PlatformIO: [Link] 152
Functions.h/.cpp and FunctionsSKL.h/.cpp contain general and application specific
functions 152
Globals.h/.cpp contain #includes and global variables; macros in Macros.h file 152
HelperFunctions.h/.cpp: service functions 152
#includes in Globals.h, pivot around which all header files (.h) get included 154
Macros.h contains macro definitions 153
See Project Files Framework 152
Start development in Aardvark, called by setup, then expand into other files 152
Flash memory See Memory (pools)
Flow diagrams See Algorithm Test Framework
F() macro See Macros, Code size, Misdoings
foreach See Perl (build)
Format document See AtmelStudio (editor)
Format Driven float to byte Conversion Framework (FDFBC)
Beehive weighing system sensor values, FDFBC reduces data transfer from 24 to 9 bytes
260
Converts float values to packed bytes and vice versa, packing controlled by parameters,
FDFBC class does all the work 162, 258
float values stored in SensorFloatData[] 164
Parameters include packed byte size, decimals, divider, range, offset 163
fpermissive See -fpermissive (compiler options) (first page of index table)
Fragmented heap See Memory (heap fragmented)
Frameworks
Application independent tools designed to accomplish specific tasks 149
Do not reinvent the wheel - use frameworks 9
Frameworks source code available from [Link] 338
Three types of frameworks: organizational, data handling, specialized 149, 243
Frameworks See Organizational, Data handling, Specialized Frameworks
Free IDEs
Arduino, AtmelStudio, PlatformIO, Visual Studio, VS Code, Code::Blocks 111
Freematics See Code::Blocks
Function call nesting See Memory (use)
Function Creation Framework
Complete function skeleton example 250
Document functions exhaustively 157
Function template to create function 157, 249
Function level macros See Print-based debugging
Functions See Awk and Perl (variables and functions)
Function template See Function Creation Framework
Fused deposition modeling FDM
Cheapest most practical for 3D printing 229
Deposit molten plastic via nozzle 229
Each printer mfg. has own slicer tool 229
Fusion 360
Can import Tinkercad schematics 226
High end CAD design tool 227
Includes Eagle software (electronic schematic and Gerber files) 227
G
Gerber files See Hardware setup; Eagle (electronics design)
getline See Awk (built-in variables and functions)
Gibberish See Bugs (avoiding them, causes, cures)
Github See Why choose Arduino
Glitches See Bugs (avoiding them, causes, cures)
Golden rules
Comment! Comment! Comment! 136
Da Vinci: "Perfection lies in details, but perfection is not a detail" 140
Do not fall into the EGO trap 131, 147
 Do your homework, know your tools, update your C++ skills, do not reinvent the wheel
131, 148
Good mental condition crucial to good work 131
KISS principle - simple solutions often better than complex ones 129, 130, 140
The right tool for the right job crucial 8
Validate function parameters and return values; never assume anything 130, 137, 183
Good programming practices (GPP)
Adhere to the 'Golden rules' (see above) 130, 137
A hobbyist mindset leads to extensive debugging time 1
Be consistent, particularly when naming files, functions, etc. 129, 132
Comment closing curly braces and #endif 131, 147
Common sense improves productivity 8, 9
Countless reasons for commenting code 129, 136
Don't neglect validity checks and error handling - Check, Check, Check! 130, 138
Do your homework: know your tools, update your C++ skills, do not reinvent the wheel 148
Efficient programming starts with good programming practices 2
enums more flexible than #define macros 145
Exploit C++ features sparingly 131, 147
Indentation, code collapsing not cosmetic 129, 133
Initialize variables, whether local or global, preferably with curly braces 131, 146
Monitor memory - prevent stack overflow and lack of heap space 130, 139
Organize your code as separate files, .ino file should be small 130, 139
Plan work offline, think before typing; thinking hardest thing to do 129, 135, 137
Start application with code skeletons 131, 147
Use #define macros, const variables, enums instead of hard coding values 131, 144
Wrap-up means stay focused 'til the task is completed 131, 141
Greediness See Regex (greediness)
Groupings See Regex (captures/groupings)
H
Hardware-based debugging See Debugging hardware-based
Hardware See Build toolchain
Hardware setup
ATmega2560 for development, smaller board for final release 17
Breadboards enables rapid prototyping but has drawbacks (unreliable serial
communications, spaghetti error prone wiring) 17, 18
Breadboards, prototype board, PCB, SMT 17
Eagle (now in Fusion 360) for the schematics and Gerber files 19
PCB should have serial comm pins to interface board with computer 19
Soldered prototype boards atop ATmega2560 board replace breadboards 17, 19
Hashes See Perl (variables) identified by % sign
Header files (.h)
Arduino.h, Macros.h included in Globals.h file, #include <Arduino.h> in Macros.h 154, 244
AtmelStudio mysteriously imports second set of header files into ArduinoCore directory 68
#ifndef prevents multiple .h file inclusions 243
#includes in Globals.h, pivot around which all header files (.h) get included 154
Respect header file dependencies (top-down) and beware of interdependencies 154, 244
Heap See Memory use
High voltage reset (HVPP ) See Debugging hardware-based
Holes See Memory (fragmented heap)
I
IDEs See Arduino, AtmelStudio, Visual Studio, Visual Micro, VS Code, PlatformIO,
Code::Blocks, MPLAB
IDEs See Which IDE to work with?
Image See avrdude (.hex file)
Import Arduino project See Interoperability
Incremental programming See Think
index See Awk (strings)
Input record separator See Awk and Perl (built-in variables)
In-system programmer See avrdude (ISP)
Intelligent C++ Editor See Build toolchain
Intellisense See AtmelStudio (editor)
Interdependencies See Header files (.h)
Interoperability
Aardvark() contains original setup code, key to interoperability 24, 26, 153
Arduino project's name is that of .ino file 32
AtmelStudio import hassle free 38, 41
Code::Blocks not suited for Arduino development, does not create project 111
Common set of files independent of IDE 21
Create ATMEL_STUDIO, ARDUINO_IDE, PLATFORM_IO macros 23
Develop with Visual Studio 2022 + Visual Micro, hardware debug with AtmelStudio +
Xplained boards 121
How to establish interoperability 22
PlatformIO/AtmelStudio/Visual Micro interoperability requires AtmelStudio one solution per
project scenario 40
PlatformIO creates [Link] with empty setup and loop 21, 107
Seamlessly switch between AtmelStudio, Visual Studio, PlatformIO, Arduino IDE 21, 27,
38, 107
setup and loop located in application entry file 152
Interoperability See AtmelStudio (interoperability) and PlatformIO (interoperability)
ISP (in-system programmer) See avrdude
J
jmp_buf See Pseudo Exception Handling Framework
join See Perl (string)
json files See VS Code .json files
JTAG See Debugging (hardware-based ATmega2560)
K
KISS principle See Good programming practices
Know your tools See Good programming practices
L
Largest possible allocation See Memory (fragmented heap)
Laziness See Regex (greediness)
length See Awk (strings), Awk (arrays)
Libraries See Arduino IDE (build)
Linear thinking See Think
Line of text See Built-in variables (Perl is $ _ ; Awk is $0)
Linked List Framework
Based on LinkedItem and LinkedList 173
Individual items or entire list may be printed 271
LinkedItem::GetNext traverses linked list forward 'till desired item is attained 173
LinkedList class: GetFirst, GetLast... 173, 270
Linked list not constrained by item size and number of items; can grow and shrink 172
Memory management and error reporting 173
Memory Management Framework logs allocations and releases in linked list 182
Linker See Build toolchain, Arduino, AtmelStudio, Visual Studio, PlatformIO(build)
Link time optimization See Code size
Lint like inspect utility See PlatformIO (general), Misdoings (Perl program finds program
flaws)
Literal characters See Regex (metacharacters)
Log events See Memory Management Framework and Event Storage Framework
Logical statements See Awk (rules)
Logic testing See Algorithm Test Framework
longjmp See Pseudo Exception Handling Framework
Look ahead and look behind See Regex (look ahead/look behind)
loop See Arduino IDE (build)
M
Macros
AtmelStudio collapses #ifdef; extra curly braces facilitate cross platform code collapsing
133
Bjarne Stroustrup suggests no macros 143
Comment closing curly braces and #endif 131, 147
Critical reporting macros triggered when resource runs low (voltage, memory...) 188
Error reporting macros print highly visible text to describe errors 188, 189
Function entry and exit macros print details upon entering and exiting functions 189, 285
Is const instead of #define better choice? 144
Memory use macros inform on current available memory and heap fragmentation 188, 286
Milestone macros highlight program's location 188, 286
Misdoings: [Link] missing F() macro 117, 295
Parameterized name/value macro simplifies printing a parameter's value 290
Preprocessor does text/replace via #define and conditional inclusions via #ifdef 144
Print-based Debugging Framework defines macros hierarchy 188
Print-based debugging macros display total, contiguous, fragmented heap 189
[Link] See PlatformIO application entry file
main function See Arduino, AtmelStudio, Visual Studio, PlatformIO(build)
Make See Build toolchain
Manage data See Data Handling Frameworks
Manage files See Project Files Framework
Map mode See AtmelStudio (editor)
Masks See Bitfield Storage Framework and Event Storage Framework
match See Awk (strings)
MaximumPossibleAlloc See Memory (heap fragmented)
Memory (heap contiguous)
Contiguous heap between top of allocated memory and bottom of stack 179, 297
Essential component of Memory Management Framework 279
Function calls require contiguous memory space for stack frames 180, 183
Function nesting and recursion consume contiguous heap space 183
Functions: TotalHeap, FragmentedHeap, ContiguousHeap, MaximumPossibleAlloc 179
Memory (heap fragmented)
Essential component of Memory Management Framework 279
__flp (holes) and __brkval (no holes) address of start of heap 297
Fragmentation ratio: fragmented/total heap 299
FragmentedHeap function monitors extent of memory holes 182, 299
Functions: TotalHeap, FragmentedHeap, ContiguousHeap, MaximumPossibleAlloc 179
Has holes from random memory releases 179, 296
Largest possible allocation: either largest hole or contiguous memory 180, 298
Small holes may collectively yield a deceptively large heap 299
Memory (heap total)
Total heap = fragmented plus contiguous heap; reveals total available memory 180, 182,
298
Memory (leaks)
Could be a cause of lack of memory, depletes available memory slowly but surely, do not
manifest themselves during testing phase 139
Memory Management Framework discloses memory gluttons and memory leaks 179, 278
Memory Management Framework
Enough contiguous heap? Hole large enough? 180
Functions: TotalHeap, FragmentedHeap, ContiguousHeap, MaximumPossibleAlloc 180,
182
Heap has two parts: fragmented and contiguous heap 179
MemAllocEvt and MemAllocList - base classes for managing allocations 181, 279
MemMgt - main class for handling memory management events 279, 280
Memory allocations verify heap availability and reveal memory leaks 9, 179, 182, 278
Memory Management Framework discloses memory gluttons and memory leaks 182
Memory use macros provide a snapshot of available memory 296
Overloaded new: memory management, verify allocation success, log events 182, 186,
279
Memory (pools)
EEPROM writes limited to 100,000 164
PROGMEM and F() macro reduce code size 232
Memory (stack)
Bottom of stack found via programming trick 298
Function nesting and recursion consume too much contiguous heap space 179, 183, 297
Hard code function memory requirements to monitor stack needs 183
Memory allocation failure avoidable with memory monitoring functions 179
Stack frame structure and size 298
Stack overflow thrashes memory allocations, may go undetected a long time 139, 179
Memory (use)
Allocations add 2 bytes for allocation size 300
Bottom of RAM (system RAM) used for globals, static, system variables 179
 __flp (holes) and __brkval (no holes) address of start of heap 297
malloc and new need how much memory to allocate 300
Memory allocation failure avoidable with memory monitoring functions 130, 139, 179, 183,
279, 297
Memory used as the application runs 180
Print-based debugging macros display total, contiguous, fragmented heap 189
RAM memory and flash memory requirements 233
Total heap is fragmented plus contiguous 139, 297
Understand memory structure to optimize memory use 139, 296
Memory use macros See Macros
Mental condition See Think
Metacharacters See Regex (metacharacters)
MicrochipStudio See AtmelStudio (general)
Microcontrollers
Choosing microcontroller is compromise; why Arduino? 1
ESP32 chips are ARM devices 119
Microcontroller architectures: AVR, ARM, PIC 119
PIC (Programmable Intelligent Computer): both AVR and ARM chips 119
Microsoft Disk Operating System See DOS box
Milestone macros See Macros
Mindset See Good programming practices
Mind to keyboard See Think - incremental programming
Misdoings
= instead of == in an if or while or == instead of = in an assignment 117, 293
Dumb mistakes not detected by compiler; found by Perl [Link] program 9, 117,
293
Regex missing F() macro in [Link] 117, 295
switch default case missing or empty 117, 293, 295
MPLAB
May be overkill for Arduino dev 112
N
Naming conventions See Good programming practices
new (overloaded) See Operator overloading
next See Awk (built-in variables and functions) and Linked List Framework
Notepad++
Edit Awk and Perl programs - configure it for Perl: also useful for C++ editing 209, 219
O
Object-oriented programming (OOP)
Bottom-up design is fill in the details 2, 8
Design your system the way you think - outside-in, aka top-down 2, 8
Olimex hardware debugger See Debugging hardware-based
Operator overloading
AddHive += illustrates operator overloading 186
First create function to do the work 187
Overloaded new: memory management, verify allocation success, log events 182, 186,
279
Organizational Frameworks
Function Creation Framework - Use function creation template to create new functions 249
Program Documentation Framework - Document your code 244
Project Files Framework - Organize your project's files 243
Three frameworks help structure your files and document source code 149, 151, 243
Other IDEs See Code::Blocks and MPLAB
Outlining See AtmelStudio (editor)
P
Packages See Perl (build)
Parameter default initializations See Avoiding bugs
Parameter validation See Golden rules
Pattern matching See Awk, Perl, and Regular expressions
PCB (Printed Circuit Board) See Hardware setup; Eagle
Peripheral thinking See Think
Perl (arrays)
+ sign overloaded to enable merging arrays 327
Array indices must be integers - will convert non integer indices to integer 325
Array @_ is mechanism to pass arguments to a function 330
Convert $fileHandle to array via <...>, cycle with foreach 222, 325
Create array as subset of existing array, from a Perl list, or by tokenizing string 325, 327
Create individual array items on the fly 325
Create multidimensional arrays via nested square brackets initializations 325, 327
Cycle through the entire array, via a for loop, foreach loop, while loop 326
Getting dimension of subarray requires programming trick 329
Multidimensional arrays is cumbersome 327
Perl arrays are associative, can be mix of numeric (float) and string values 203, 319, 325
Perl arrays can be sorted via the sort function 327
Perl provides three variable types: $scalars, %hashes, @arrays 222, 325, 331
Process file via a shift function on array 222
push/pop add/remove last item in Perl array; shift/unshift does same on first item 326
scalar gets number of items in one-dimensional array 325
Perl (Awk vs. Perl)
Cycle through file names array with foreach, process files with file handles 320
Does not support parens-based captures - Perl does 203
Perl is better than Awk 201, 203, 223
Process same set of files several times - Awk limited to processing set once only 203, 223
Simulate Awk with Perl or how to transform Awk program to Perl 223, 318, 320
Perl (build)
Convert $fileHandle to array via <...>, cycle with foreach 222
C's switch/case/default are named given/when/default in Perl 324
Importing packages (use directive): strict, diagnostic, features 'switch' 218, 220, 319, 330
Lists enable doing tasks on multiple items in one line of code 203, 318, 323
Pass by value and by reference 218, 329
Perl handles records and fields 218
Perl provides three variable types: $scalars, %hashes, @arrays 218
Perl's program structure resembles C's 220
Process files: foreach, while on $FileHandle 222, 324
 Program flow control mechanisms are supported, namely for, if, switch (given), etc. 324
Scope resolution operator :: implements namespace concept 218, 331
The text match operator =~ enables using a regex in a statement 219
Perl (built-in variables and functions)
$ARG or $ _ is the line of text (record) 326, 329
$ARGV array contains current file name when reading from <> array 329
$INPUT_LINE_NUMBER, $NR, $. line number of file being processed 329
$INPUT_RECORD_SEPARATOR, or $RS, or $/ - default \n 324, 329
$OUTPUT_FIELD_SEPARATOR, or $OFS, or $, separates fields from each other 329
$OUTPUT_RECORD_SEPARATOR, or $\ - default newline 329
$PROGRAM_NAME, or $0, contains the Perl program launched 218, 329
$SUPSEP or $; - index separator 330
@ARG or @_: arguments to function 330
@ARGV array: command line arguments 329
English pragma: AND instead of &&, OR instead of || 329
Input field separator - Perl does not seem to have one 329
Perl supports the Awk like BEGIN and END 221
qq{}function enables embedding double quotes inside of string without escapes 330
split and qw// functions automatically tokenizes a line of text 330
Perl (command line - DOS box)
Invoke Perl directly or use batch file 319
[Link] arg1 arg2... > [Link] 319
Perl command line file/directory in arg1, arg2, etc.; in @ARGV array 219, 319, 320
Perl (file handles)
$filehandle created from file name string via the open statement 218, 222, 321
Convert $fileHandle to array via <...>, cycle with foreach 222, 322
Files in read, write, or update mode 218, 321
Process files: foreach, while on $FileHandle 222
Perl (general)
Edit Awk, Perl programs with Notepad++ or VS Code 92, 217, 219
Handles text files and binary files 219
Included in Unix, Linux, and MacOS but not Windows (download Strawberry Perl) 319
Perl details: download, use it, simulate Awk, constructs strange from C/C++ perspective
318
Perl is better than Awk 203
Perl program creates program documentation 116
Perl program looks for dumb mistakes: [Link] 9, 117
Perl supports regex parens-based captures - Awk does not 203, 219, 220
Powerful C-like programming language with regex support 191, 201, 217
Why I started with Awk, then migrated to Perl 201
Your C knowhow, a false friend when learning Perl 218, 224
Perl (strings)
Perl arrays to strings with list function 327
Perl string tokenization and concatenation functions: split, qw//, join 327, 330
Perl supports string concatenations 322
qq{} function enables embedding double quotes inside of string without using escapes 330
String interpolation differentiates a variable's name from its value in print statements 323
Supports literal strings (single quotes) and interpolated strings (double quotes) 224, 323
 Supports regex text match operator =~ 331
Perl (variables and functions)
BEGIN and END blocks executed at program start and finish 220
Functions can pass by value or by reference 220
Functions have a name and a body but do not have a parameter list 220, 224, 322
Functions may be called with parameters - unique way of handling (via array @_) 220,
224, 322
Perl types differentiated by special characters: $scalars, @arrays, %hashes 222, 331
Perl vs. C
Arrays are associative; index is integer based 224
Perl code resembles C but there are notable differences 224
Perl is a command line utility/interpreter 224
Pointers not supported 224
print statement items not necessarily inside parentheses 224
Regular expressions supported 224
# sign same as C++ comment - ignore up to end of line 224
Special character identifies variable's type: $ scalar, @ arrays, % hashes 224
Perl Web site
Documentation on variables, functions, operators 335
permissive See -fpermissive (first page of index table)
PIC See Microcontrollers
Plain text matches See Regex (general)
Plan your work offline See Think
PlatformIO (build)
Build command details 235
-flto and -fpermissive flags included by default in PlatformIO project properties 232
Uses Arduino distribution's GNU C++ compiler 8
PlatformIO (general)
Compatible with the Arduino project structure 101
Creates [Link] with empty setup and loop 103
Find/replace supports regular expressions 191
Free, professional grade, easy import of Arduino projects 27, 98
IDEs: Arduino, AtmelStudio, Visual Studio, VS Code, PlatformIO, Visual Micro 84, 98
IDE supports cross-platform, cross-architecture, multiple frameworks 27, 98
Installation - do it progressively, VS Code first 100
[Link] presents interesting tutorial on PlatformIO 108
Offers features beyond those offered by other IDEs 101
Plugin for Microsoft VS Code and CLion 27, 98, 101
Project configuration in [Link] file 106
Provides facilities for lint like inspect utility 101
Supports hardware-based debugging wide range microcontrollers 101
Supports multiple libraries packs 102
Supports version control, configuration management, continuous integration 102
Web site ([Link]) rich with info 108
PlatformIO (gotchas and caveats)
Build uses files in the project's source code directory and subdirectories 110
Contents of directory not updated after directory content changes 110
File editing window can be unpinned, editor reverts to being simple notepad like 110
 Has its share of gotchas and caveats 109
Import Arduino project location hard-coded 109
New com port not recognized 110
[Link] file variable in wrong section not reported 109
Removing file from project deletes it 110
The build/upload process works fine but no output from program 109
Uploading a program fails: access denied 109
PlatformIO (hardware-based debugging)
Requires special Arduino Nano 33 or MKR boards 108
Serial debugging not supported 108
Supports Arduino hardware-based debugging, ESP32, other 108
PlatformIO (interoperability)
Creates [Link] with empty setup and loop 21, 26, 107
Interoperability between Arduino IDE, AtmelStudio, PlatformIO, Visual Micro 27
[Link] is PlatformIO's Arduino app entry file (.ino replacement) 26
Operate in single- or two-level directory structure; code in src_dir [Link] variable 26
PlatformIO Arduino project located in AtmelStudio project base directory 22
Seamlessly switch between AtmelStudio, PlatformIO, Arduino IDE, Visual Studio 21, 107
PlatformIO ([Link] file - project config.)
Com port defined in upload_port in [Link] file 106
Define board with board = mega2560 or AtMega328 in [Link] 105
Operate in single- or two-level directory structure, code in subdirectory pointed to by
src_dir 103
[Link] file contains sections/variables 106
PlatformIO project is folder which contains [Link] file 103
Project configuration in [Link] file 106
Source code location defined in src_dir in [Link] file 106
PlatformIO (project management)
Creates [Link] with empty setup and loop 103
Open/Close project is Open/Close VS Code folder 102
PlatformIO projects are VS Code folders 102
Project configuration in [Link] file 106
pop See Perl (arrays)
Pragmatic C++ Arduino Programming book
Explains why C++ can lead to excessive debugging time and how to reduce it 1
Preprocessor See Build toolchain, Macros
Print-based Debugging Framework
Avoiding bugs - ounce of prevention is worth pound of cure 114
Complementary tool relative to hardware-based debugging 119, 188
Debug control macros turn debugging on/off global, file, and function level 188, 291
Debugging macro level: global, file, function 118
Display function specific data and errors 188
Entry/exit macros provide parameter and return values, and inform on selected items 285,
289
Macro categories: navigation, memory use, critical situation, data values 118, 188, 285
Printing strategic data values, where the program is at 118
Printed Circuit Board (PCB) See Hardware setup
PROGMEM (read-only data in flash memory)
 Web sites which describes PROGMEM 334
Program Documentation Framework
Class and file documentation template store class and file info 245, 247
Comment! Comment! Comment! 136
Create program documentation via Awk/Perl regex based comment extraction 136, 155,
245, 248
Example of comments extraction Aardvark 245
Finding functions using regexes complicated 252
Function template contains description, parameters, error checking, return values 247
PROGRAM KEYWORD START and END basis of generalized info extraction mechanism
157
Starts with inserting comments, organize them as templates (fill in the blanks) 244
Who calls who? Who gets called by whom? 248
Program logic See Algorithm Test Framework
Programming objectives
Enhance productivity, maintainability, robustness, compactness, and speed 4
Monitoring code size important since premium on RAM although speed might be a
requirement 4
Project directory See Interoperability
ProjectDir See AtmelStudio (build)
Project Files Framework
Aardvark() contains original setup code, key to interoperability 153
Atest.h/.cpp contain ATest() to do testing 153
Avoid interdependencies between .h files 244
ClassSpecific.h/.cpp files contain classes 153
Common sense dictates how to organize an application's source code 130, 139, 152, 243
Entry files: Arduino: .ino file; AtmelStudio and Visual Studio/Visual Micro: [Link];
PlatformIO: [Link] 152
Functions.h/.cpp and FunctionsSKL.h/.cpp contain general and application specific
functions 153
Globals.h/.cpp contain #includes and global variables; macros in Macros.h file 153, 244
HelperFunctions.h/.cpp: service functions 153
Macros.h contains macro definitions 153
setup and loop located in application entry file 152
Prototype board See Hardware setup
Pseudo Exception Handling Framework
Alternative to C++'s exception handling based on setjmp and longjmp 183, 281
Deep down the function call chain trigger longjmp back to setjmp landing point 282
EErrorID identifies longjmp 184
longjmp (throw) - return landing point 184, 282
setjmp (try) sets landing point 184, 282
push See Perl (arrays)
Q
qw// and qq{} See Perl (string)
R
Radio transmission constraints See Format Driven float to byte Conversion Framework
Random Access Memory (RAM) See Memory (use)
Redirection See DOS box
Refactoring See AtmelStudio (editor)
Regex (captures/groupings)
Grouping (capture) extracts of parens pair content 196, 198
Grouping with '|' (OR) lists alternatives 196
Regex (examples)
Extract a variable's name; define floating-point numbers 197
Find text enclosed in square brackets gotcha, greediness issue 200
Generate subset of a file 192
Greedy vs. lazy search anything 197
Look ahead to find all Serial.??? which are not prints 198
Perl and Awk program use regexes to extract program documentation from source code
191
Regex missing F() macro in [Link] 295
Regex to find = instead of == in an 'if', or == instead of = in an assignment 293
Regex used to list enums and #defines, find unused ones 192
Step-by-step example regex to find enums 197
Uncovers repeat words like the red red fox 192
ZIP code example gotcha or how laziness solved the problem 196, 199
Regex (general)
Anchor means look for something at the beginning ^ or at the end $ of the line of text 194
Character class [...] defines what characters to search for 193, 196
Many tools contain a regex engine - AtmelStudio and PlatformIO find/replace, Perl, Awk,
Word (wildcards) 38, 50, 55, 191
Match - return true if regex is successful, false otherwise 193
Regex engine is a generalized intelligent wildcard protocol to find/replace text 9, 191
Regexes use two types of characters: literal characters (what to search for) and
metacharacters (regex operators) 193, 195
Regex segments combine individual patterns to form elaborate ones 194
Regex (greediness)
Find text enclosed in square brackets gotcha, greediness issue 200
Greediness means grab all it can before relinquishing control (speed); laziness means
grab one character only and let the regex continue (functionality) 194, 199
Period/asterisk '.*?', period/plus '.+?' followed by question mark means lazy search
anything 196
ZIP code example gotcha or how laziness solved the problem 196, 199
Regex groupings See Regex (captures/groupings)
Regex (look ahead/behind)
Condition search based on existing item 198
Look ahead to find all Serial.??? which are not prints 198
Regex (metacharacters)
$ sign anchors search to end of text 196
Caret ^ used both as an anchor (beginning of text) and negation in character class 196
Context defines whether pattern matching character is metacharacter or literal 195
Escape (backslash \) transforms metacharacter into ordinary character 193, 196
Literal characters: characters searched for 195
 ? means preceding segment is optional, also means greedy or lazy when doing a search
195
Metacharacters are letters or symbols which become pattern matching operators 195
OR symbol '|' means one of several alternatives (choices) inside a parens group 196
Quantifier {1-x} means repeat 1 to x times 195
Regular expressions (regex) See Regex
Return values validation See Golden rules
Rules See Awk (rules)
S
say feature See Perl (build)
scalar See Perl (variables) identified by $ sign
Scope See Awk (variables and functions)
Scrollbars See AtmelStudio (editor)
Search and replace See AtmelStudio (find/replace)
Segmentation faults See Arduino IDE (caveats) and AtmelStudio (caveats)
Sensor data conversion See Format Driven float to byte Conversion Framework
Sensors
Thousands of Arduino compatible sensors, boards, devices available 12
Serial communications See Format Driven float to byte Conversion Framework
Serial terminal See Build toolchain, Arduino IDE (editor) and AtmelStudio (editor)
setjmp See Pseudo Exception Handling Framework
setup See Aardvark, Interoperability, and Arduino IDE, AtmelStudio and PlatformIO (build)
shift See Perl (arrays)
Simulate Awk See Perl (simulate Awk)
Size specifier See enum
Sketch
Name used to refer to Arduino's .ino file (program entry file) 32
[Link] See AtmelStudio (interoperability)
SMT (surface mount technique) See Hardware setup
Solution See AtmelStudio (file management)
Specialized Frameworks
Algorithm Test Framework plan and test multiple execution paths 272
Class and Function Names Referencing Framework - IDs identify classes/functions 276
Error Reporting Framework - log errors and warnings inside a linked list 284
Memory Management Framework informs on heap space, memory use, memory gluttons
278
Print-based Debugging Framework - selectively choose debugging print #defines 285
Pseudo Exception Handling Framework - alternative to C++'s exception handling 281
Seven framework toolkits to accomplish sundry tasks 150, 173, 272
Specialized macros See Macros
Spell-checker See AtmelStudio (editor)
split See Perl (string)
SRAM (RAM) See Memory (use)
Stack frame
Determining stack frame size 298
Functions require contiguous heap space for stack frame 183
Stack overflow See Memory use
Startup See avrdude (bootloader)
strict See Perl (build)
String interpolation See Perl strings
substr See Awk (strings)
Surface Mount Technology SMT See Hardware setup
switch feature See Perl (build)
switch statement default case See Bugs, Good programming practices, Error Reporting
Framework, Misdoings
Syntax checking See AtmelStudio (editor)
T
Tables
Awk - short database example Awk program 303
Bitfield Storage Framework - Table of bitfield based variables of Job class 167
RAM memory and flash memory requirements 233
Tab mode See AtmelStudio (editor)
TargetDir and TargetName See AtmelStudio (build)
Task creation See Code skeletons
Task wrap-up phase See Wrap-up phase
Text find/replace See Macros
Text matches See Regex (general)
Text match operator See Perl (text match operator =~)
Think
Easy to sit at the computer and code 137
Good mental condition crucial to good work; do not rush, take breaks 143
Good programming practices 135, 137
incremental programming, i.e. mind to keyboard can be a costly time wise trap 4
Linear thinking means focus and follow a path; peripheral thinking means let your mind
loose, let it dwell around a subject 141
Planning offline before coding can save loads of time 8, 130
Thinking hardest thing to do - requires effort 129, 135
Through-hole technology See Hardware setup
throw, try, catch See Exception handling (C++) and Pseudo Exception Handling
Tinkercad
Create visual breadboard wiring/schematics 226
Schematics exported to Eagle (Fusion 360) 226
Tokens See Awk (terminology)
Top-down design See Object-oriented programming
try, throw, catch See Exception handling (C++) and Pseudo Exception Handling
Type checking leniency See Bugs
U
Undefined references See AtmelStudio (caveats)
unshift See Perl (arrays)
Update your C++ skills See Good programming practices
Uploader See Build toolchain, avrdude, AtmelStudio build
use See Perl (build)
V
Validate data
Apply good programming practices and adhere to Golden rules - check data, never
assume anything; do error handling 130, 137, 138
Upon detecting error, undertake reporting and decide what to do next 184
Variables See Awk and Perl (variables and functions)
Variables See Initializations
VAssist See AtmelStudio editor and documentation
Verbose See avrdude and AtmelStudio (import Arduino project)
Version control See PlatformIO (general)
Visibility See Awk (variables and functions)
Visual Micro
AtmelStudio Arduino compatible in two versions, without and with Visual Micro 37
Develop applications for Arduino, ESP32, RaspberryPi, and others 84
Develop with Visual Studio 2022 + Visual Micro, hardware debug with AtmelStudio +
Xplained boards 80, 121
Improve productivity by orders of magnitude 8
Interoperability between Arduino IDE, AtmelStudio, PlatformIO, Visual Micro 27
Provides a serial monitor, compile/link/upload, serial debugging 84
Three identical Arduino plugins: for AtmelStudio and for Visual Studio (2019 and 2022) 39,
84
Visual Micro enables serial debugging w/o dedicated hardware 125
vMicro for AtmelStudio in top-level toolbar 84
vMicro for Visual Studio project location defined from Arduino IDE preferences 85
vMicro for Visual Studio tucked away in Extensions of top-level toolbar 85
Visual Studio
AtmelStudio is Atmel specific Visual Studio 37
Choose between two versions: 2019 and 2022 80
Create Arduino apps via Arduino project template or via Visual Micro 79, 82
Debugging - only Serial debugging via Visual Micro seems feasible 83
Develop with Visual Studio 2022 + Visual Micro, hardware debug with AtmelStudio +
Xplained boards 80, 121
Expand your horizons with Visual Studio (Python, Raspberry Pi... 80
Extensions for embedded development: Arduino, ESP32, RaspberryPi, ... 79
IDEs: Arduino, AtmelStudio, Visual Studio, VS Code, PlatformIO, Visual Micro 79
Improve productivity by orders of magnitude 8
Microsoft's flagship development tool - two versions, 2019 and 2022 79
Multilanguage: C++, C#, Python... and multi-platform: Windows, MacOS, Linux 79
Out-of-the-box Arduino Project Template solution creates both .ino file and command line
.exe program 82
Straightforward installation, two Arduino dev solutions 80
Visual Micro for Visual Studio identical with Visual Micro for AtmelStudio 82
Visual Studio Code
See VS Code 86
Visual Studio Code See VS Code
vMicro See Visual Micro and Debugging serial
VS Code
Awk extensions 96
C++ extension 94
 Edit Awk, Perl programs with Notepad++ or VS Code 92
Extensions can be installed/uninstalled and enabled/disabled 87, 91
File, folder, workspace used for project management 87, 91
Find/replace supports regular expressions 87
IDEs: Arduino, AtmelStudio, Visual Studio, VS Code, PlatformIO, Visual Micro 86
Microsoft's programming foundation platform 86
Multilanguage: there are extensions for C++, Python, Perl... 87
Perl extensions 96
Provides source control via Git repository 87
Regular expressions extensions 96
User interface appearance 89
User space vs. workspace 87
VS Code features 87
VS Code caveats
Arduino-cli failed to get installed 97
Unable to remove folder from active folders list 97
VS Code does not support the concept project content/disk content 97
VS Code .json files
Data files organized as text key-value pairs 96
Value is string, number, Boolean, array, other json object 96
W
Warnings See Awk (command line) and Perl (build)
Web site See Book's Web site
Which IDE to work with?
Arduino readily available, inexpensive, particularly easy to work with, required by other
IDEs 11, 27
AtmelStudio best: solid, professional tool, supports hardware-based debugging 27, 37
Code::Blocks does not seem suited for Arduino development 111
Develop with Visual Studio 2022 + Visual Micro, hardware debug with AtmelStudio +
Xplained boards 80
GitHub search on Arduino yields more than 100,000 results 11
Interoperability possible between Arduino IDE, AtmelStudio, Visual Micro, PlatformIO 27
MPLAB may be overkill for Arduino development 112
PlatformIO - free, professional grade, easy import of Arduino projects 27, 98
Short list: Arduino IDE, AtmelStudio, Visual Studio, PlatformIO, VS Code 27
Visual Micro - plugins for AtmelStudio and Visual Studio Arduino work 84
Visual Studio - highly professional multi-language development tool 79
VS Code - a foundation upon which many tools are created, including PlatformIO 86
while See Perl (arrays)
Workspace font size See Arduino IDE and AtmelStudio (editor)
Wrap-up phase
Adhere to good programming practices; stay focused 'til the task is completed 131, 141
Code skeletons help ensure items not forgotten 142
Wundef See AtmelStudio (import Arduino project)