Symmetric Multiprocessing (SMP) is a cost effective way of adding more processing power to your embedded designs. A single package can contain 2 or more cores giving you more processing power than ever before. Realtime operating system such as RTEMS (http://www.rtems.org) have been adding support for a while and SMP real-time applications are starting to appear.
SMP support has been in main stream operating systems for a while now and developers and users are experienced in using Windows, Linux, FreeBSD and other operating systems on SMP hardware. The developer of thread aware applications expects to have concurrent threads executing at the same time with the operating system balancing the load across the available processor cores. Realtime operating systems and applications designed to run on multicore devices present new challenges. The developers of these applications need to adjust some of the practices that work on single processor systems when moving to SMP applications. The applications need to be thread safe just like applications running on the main stream operating systems plus the developer needs to take into account things such as the executing priority or masking interrupts does not ensure exclusive access to data or hardware. Applications for single core systems may intentionally or even unintentionally be using these techniques to have exclusive access to data or hardware. The way the developer tests and debugs applications also changes. SMP gives you more power, it also gives you more ways for your code to fail and a more complex debugging experience.
Debugging with SMP is the topic of this post and an area of change. Many years ago embedded realtime debugging used a simple remote stub type application running on the hardware the developer connected to via a serial port. Often this was a single file of code and masking interrupts gave you exclusive control of the processor. Stop and consider what that simple stub would look like on a multicore system running in an SMP configuration. Gone is the simple easy to get working piece of code. These days we have On Chip Debugging (OCD) support that gets us directly into the device and this helps to some extent however it does not remove the underlying complexity SMP brings.
The GNU debugger has changed little on the main stream SMP operating systems. The thread model abstracts the SMP architecture for the developer and the core is really not an issue. The developer can ask GDB to show the state of a thread, the stack, local variables and even registers and the thread support in GDB combined with the operating system provides a suitable abstraction. When a developer steps the program other threads may or may not run and typically all threads stop when the debugger takes control. The developer often does not care which core the thread runs on.
Debugging with a realtime operating system close to the processor hardware with a single core assumes the processor's context is the running thread's context. The stack is the executing thread's stack and the register display command shows the registers for the processor and so the executing thread. Some realtime operating systems have thread support in GDB and these help however stepping an interrupt service routine is much more complex in these cases plus you need to scope breakpoints in terms of threads. When a developer stops the application it means stopping the processor at a low level and if using OCD it often means the device is frozen. When debugging on a multicore system the 1:1 relationship between the processor context and the running thread becomes a 1:n relationship when more than one core is present. This complicates the experience for the user. A back trace of the call stack is just the selected processor. GDB thread support would need to understand the cores and which threads are on the cores to access the correct data. Stepping and running becomes more complex. If a developer steps an instruction or a line of code should the other cores run, step or do nothing ? When the debugger halts should all the cores stop ? It is difficult to enforce a standard set of functionality here and I see different possible methods of operation being needed to help fix different types of bugs.
Developers of SMP realtime embedded applications need to consider the debugging side of the development process for themselves and their users. There are good tools available with advanced functionality including trace however these cost. Open source solutions are starting to appear but need more work. I also cannot find a standard approach a user can expect that lets them develop a tool chest of techniques.
