Файл: Embedded Systems Design - An Introduction to Processes Tools and Techniques (A. Berger, 2002).pdf

Remember that the logic analyzer is passively “sniffing” the processor’s I/O pins on every clock cycle. If every state that is visible on the bus corresponds to what the processor is doing with that instruction, the logic analyzer will accurately capture the instruction and data flow sequence of the processor.

However, if the activity on the bus does not have a 1:1 correspondence to the instructions being executed and the data being accessed, the logic analyzer’s usefulness begins to diminish. Most on-chip caches are generally refilled from main memory in bursts. Even if only one byte of data is not available in the cache, the bus control logic of the processor generally fetches anywhere from 4 to 64 bytes of data, called a refill line, from main memory and places the refill line in the cache. The logic analyzer records a burst of memory reads taking place. Where’s the data of interest?

If the caches are small, clever post-processing software can sometimes figure out what the processor is actually doing and display the real flow. You might still have a problem with triggering, but it’s possible to circumvent it. With small caches, branches in the code generally have at least one possible branch destination located outside the cache. Thus, clever software might be able to use basic block information from the compiler and the address of memory fetches to reconstruct what happened at various branch points. If the caches are large enough to normally hold both possible destinations of a branch, a logic analyzer, with only external bus information available to it, has no way to determine what instructions are being executed by the processor. Of course, most processors allow you to set a bit in a register and turn off the caches. However, as with the debug kernel discussed earlier, the performance degradation caused by the intrusion of the logic analyzer might cause your embedded system to fail or to behave differently than it would if the caches were enabled.

Trace Visibility

Most manufactures of embedded systems understand the importance of real-time traces and add on-chip circuitry to help logic analyzers or other tools decode what is actually going on inside of the processor. Over the years, several different approaches have been taken. National Semiconductor Corporation and Intel Corporation created special versions of their embedded processors called “bondouts” because additional signals that were available on the integrated circuit die were bonded out to extra I/O pins on these special packages. Thus, a normal package might have 100 pins, but the bond-out version has 140 pins. These extra pins provided additional information about the program counter and cache behavior. With this information, it becomes feasible to post-process the logic analyzer trace to uncover the processor program flow.

Traceable Cache™

AMD took another approach called Traceable Cache™. Rather than create a bondout version of the chip, certain AMD 29K family processors could be put into a slave mode through the JTAG port. The master processor and slave processor are mounted together on a preprocessor module, and both have their data lines connected to the system data bus. Only the master processor has its address lines connected to the address bus. The two microprocessors then run in lockstep. The unconnected address pins of the slave processor output the current value of the program counter on every instruction cycle. These additional states are captured by the logic analyzer, and, by post-processing the resulting trace, the actual instruction flow can be reconstructed, even though the processor is running out of the cache.

Today, the generally accepted method of providing trace information is to output the program counter value when a non-sequential fetch occurs. A non-sequential fetch occurs whenever the program counter is not incremented to point to the address of the next instruction in memory. A branch, a loop, a jump to subroutine, or an interrupt causes non-sequential fetches to occur. If the debug tools record all the non-sequential fetches, they should be able to reconstruct the instruction flow, but not the data flow, of the processor. Non-sequential fetch information can be output when the bus is idle, so it usually has a minimal impact on the processor’s performance; however, tight loops can often cause problems if the data is coming too fast. Some sort of an on-chip FIFO for the non-sequential fetch data usually helps here, but even that can get overrun if the branch destination is being output every few clock cycles.

Caches and Triggering

As noted earlier, triggering might still be a problem. Traditional triggering methods also fail when caches are present, so the semiconductor manufacturers place triggering resources on-chip, as part of their debug circuitry. Often, these resources are called breakpoint registers because they are also available to debuggers for setting breakpoints in the code. A breakpoint register might be set to cause a special code to be output on several status pins whenever the internal breakpoint conditions are met. The code is then interpreted by the logic analyzer as a trigger signal, and the logic analyzer takes the appropriate action, such as starting to capture a trace. Chapter 7 discusses the IEEE ISTO-5001 embedded debug standard, including the various dynamic debugging modes defined in the standard.

In these examples, you’ve seen that as the processor speed and complexity increases, the type of information that you must record to understand what the processor is doing necessarily changes as well. Today, attempting to capture the external bus states on every clock cycle is generally not possible or necessary. By effectively using the on-chip dynamic debug resources, such as the address information about non-sequential fetched and internal trigger resources, you usually can record enough information from the processor to reconstruct the realtime instruction flow of the processor.

Compiler Optimizations

With optimizations turned on, a good C or C++ compiler can generate code that, at the assembly language level, is almost impossible to relate to the original source code. Even with the C source statements interspersed with the assembly code, critical events might not show up in the trace where you expect them to be. Of course, turning off the optimizations can have the same effect as turning off the caches to gain visibility. The tool becomes intrusive, and performance is compromised, perhaps so much as to cause the embedded system to fail.

Cost Benefit

Even with all the limitations of logic analysis as a debugging tool, the negatives don’t outweigh the positives. Although high-performance logic analyzers can be expensive (over $20,000), excellent units are available below $10,000 that will meet your needs for a long time to come. Magazines, such as Electronic Design and EDN, regularly do feature articles surveying the latest in logic analyzers. One interesting new offering is a logic analyzer on a PCI card that can plug into your desktop PC. Although the performance is modest, compared with the standalone units, this type of a logic analyzer offers a reasonable amount of performance for a modest investment.

Other Uses

Up to now I’ve been considering the logic analyzer to be a specialized type of debugging tool. The developers define a trigger event that hopefully is related to the fault they are investigating, wait for the trigger event to occur, examine the resultant trace, and fix the problem. This seems to imply that all data is contained within a single trace. However, a logic analyzer also can be used as a key tool for processor-performance measuring and codequality testing. These topics are discussed in more detail in Chapter 9, so they are only introduced here.

Statistical Profiling

Suppose that instead of waiting for a trigger event, you tell the logic analyzer to capture one buffer full of data. If the logic analyzer is connected to a PC, you can write a fairly simple C program that randomly signals the logic analyzer to start immediately and capture one trace buffer full of data. You’re not looking for anything in particular, so you don’t need a trigger. You are interested in following the path of the microprocessor as it runs through its operation code under fairly normal conditions.

Each time you tell it to capture a trace buffer full of data, you wait for it to complete and then upload the buffer to a disk file on your PC. After uploading the trace buffer, you start a random time generator, and, when the timer times out, you repeat the process.

Note

The random time generator is necessary because the possibility exists that the time loop from taking and uploading a trace is somehow related to the time it takes to execute a fraction of code in the target system. If the logic analyzer were continuously restarted without the random time delay, you might never see blocks of code executing, or you might get the inverse situation so that all you ever see is the same block of code executing.

Each time you take a trace and upload it to the PC, you get a brief snapshot of what the program has been doing for the last 1 million or so bus cycles. The snapshot includes:

What instructions were executed and how often they were executed

What functions were being accessed and how long each function took

to run

What memory locations (instructions, data, stack, and heap) were being accessed

How big the stack grew

What global variables were being accessed and by which functions

Because the uploaded trace buffer is an ASCII text file, you can use the standard file-manipulation tools that come with the C or C++ library to gradually build statistics about your program. The longer you allow the embedded program to run with the logic analyzer attached and gathering data, the more accurate the information that you gather will be. From the linker map and symbol tables, you can relate the memory address to your variables, data structures, and functions. For example, you can easily build a graph that shows what percentage of the time is spent in each function.

Importance of Execution Profiling

Most software designers have no idea how long it takes for the various functions in their code to execute. For example, a company thought it would have to do a complete redesign of its hardware platform because the performance of the system wasn’t up to standard and the software team adamantly claimed the code had been finetuned as much as humanly possible. Fortunately, someone decided to make some performance measurements of the type discussed here and found that a large fraction of the time was spent in a function that shouldn’t have even been there. Somehow, the released code was built with the compiler switch that installs the debug support software because it was erroneously included in the final make file. The processor in the released product was spending half its time in the debug loops!

In “The Twenty-Five Most Common Mistakes with Real-Time Software Development,” David Stewart[6] notes that the number one mistake made by realtime software developers is the lack of measurements of execution time. Follow his steps to avoid the same trap:

First, design your system so that the code is measurable!

Measure execution time as part of your standard testing. Do not only test the functionality of the code!

Learn both coarse-grain and fine-grain techniques to measure execution time.

Use coarse-grain measurements for analyzing real-time properties.

Use fine-grain measurements for optimizing and fine-tuning.

One of the logic analyzer’s shortcomings is that it performs a sampling measurement. Because it must capture a discrete buffer each time and then stop recording while it is being uploaded, it might take a long time to gain accurate statistics because extremely short code sections, such as ISRs, might be missed. Chapter 9 discusses other methods of dealing with this problem. For now, it’s easy to see that because the logic analyzer often can operate non-intrusively, using it as a quality assurance tool makes good sense.

The logic analyzer can be used to show what memory locations are being accessed by the processor while it runs the embedded program code. If code quality is important to you, knowing how thoroughly your testing is actually exercising your code (i.e., code coverage) is valuable. Code-coverage measurements are universally accepted as one of the fundamental measurements that should be performed on embedded code before it is released. If your coverage results indicate that 35 percent of your code has been “touched” by your test suite, that means that 65 percent of the code you’ve written has not been accessed by your tests.

Experiment Design

I’m convinced that debugging is the lost art of software development. In the introductory C++ class that I teach, I actually devote class time to discussions and demonstrations on how to debug and use a debugger. What I see is that students (and many practicing engineers) have not a clue as to how you should approach the problem of finding a flaw in a system. Of course, sometimes a bug can be so obscure and infrequent as to be almost impossible to find, even for the best deductive minds. I can vividly remember a scene from my R&D lab days when a group of senior engineers were standing around for hours staring at an FPGA in a complex embedded system. It wasn’t working properly, and they could not conjure up an experiment that they could use to test the various hypotheses that they had as to why they were seeing this failure. What was so striking about this scene was

I saw them standing there at about 10 A.M., and, when I went by again at around 3:30 P.M. everyone was in exactly the same position with the same expressions on their faces. I assume they had gone to lunch, the restroom, etc., but you couldn’t tell from my before and after vignettes.

I think that part of the difficulty of debugging is the amount of time you need to commit to finding the problem. If faced with a choice between trying something “quick and dirty” or setting up a detailed sequence of experiments and observations, most engineers will opt for the quick and dirty approach. This isn’t meant to be a criticism, it’s just the way people are. Another popular debugging technique is to “shotgun” the problem. Shotgunning the problem means changing everything in sight with the hope that one of the things that you try will fix it. You do this even though you know from your high school geometry proofs that you should progress one step at a time. You should postulate a reason for the failure, based upon careful observations of your system under test. You then design an experiment to test your hypothesis, if it works, you are then able to explain what went wrong, and you have high confidence that you actually fixed the problem. We all do that. Right?

With my students, I often see the antithesis of any attempt at logical thinking. If it doesn’t work, they just write their code all over again! It is frustrating for the student, and for me, to try to single-step a timer ISR when the timer ticks every 150 microseconds. What is even more frustrating for me is that I even taught them how to use a logic analyzer.[1]

So what are we trying to accomplish here? The answer lies at the heart of what we are trying to do in debugging an embedded system, and I believe that in many ways it is fundamentally different from how we debug host-based programs. The embedded system is often a complex mix of external stimuli and system responses, controlled by one or more processors and dedicated hardware. Just getting an accurate picture of what’s going on is often the crux of the problem. A product marketing manager with whom I once worked summarized it quite succinctly. He referred to this problem of just trying to understand what is going on as, Time to Insight. In other words, how long will it take me to figure out what is going on with this !#%$*$#&* embedded system? The problem that you face is akin to the problem face by an electrical engineer trying to debug the hardware without an oscilloscope. You can measure some DC voltages with a digital voltmeter, but that won’t tell you if you have serious overshoot or undershoot problems with your bus signals.

Summary

The debug kernel is a powerful part of the embedded system designer’s toolkit. In fact, it’s arguably the most important tool of all. With a debug kernel, you have a degree of control and an observation window into the behavior of your system, with only moderate overhead demands on the target.

As the complexity (and cost) increases, these hardware tools are asked to address the issue of intrusiveness in their own particular way. With an embedded system, you need the run control feature set that the debugger provides because examining and modifying memory and registers, singlestepping, and running to breakpoints is fundamental to debugging software of any kind. You can use these

debug kernel features to trace program execution but not without intruding on the real-time behavior of the system. The logic analyzer, although more expensive, provides a less intrusive means of tracing program execution.

This chapter has considered the three fundamental tool requirements for integrating hardware and software:

A debug kernel for controlling the processor during code development

Substitution memory as a way to rapidly download and replace code images in non-volatile memory

A non-intrusive way to follow code execution in real time

In the next chapter, you’ll learn how to benefit even more by tightly coupling these three tools. In short, you’ll examine the in-circuit emulator.

Works Cited

1. Berger, Arnold S. “A New Perspective on Teaching Embedded Systems Design.” http://www.embedded.com/story/OEG20010319S0092, 20 March 2001.

Team-Fly®

Chapter 7: BDM, JTAG, and Nexus

Overview

Traditionally, the debug kernel has been implemented in firmware. Thus, for the kernel to execute correctly on new hardware, the new design must at least get the processor–memory interface correct. Unfortunately, as clock speeds increase and memory systems grow in size and complexity, this interface has become more and more demanding to engineer, which raises a question “how you can debug the system when you can’t rely on the system to execute even the debug kernel?”

Increasing levels of integration create a related problem: How do you modify firmware when it’s embedded on a chip in which you can’t use a ROM emulator?

To address these and other related issues, chip vendors are beginning to supply hardware implementations of the debug kernel as part of the chip circuitry. When the functionality of the debug kernel is part of the chip circuitry, debugging tools can continue to deliver run control and to monitor system resources even if the processor chip isn’t able to communicate with the rest of the board. This robustness makes it much easier to determine whether intermittent “glitches” are hardware or software problems.

Putting debug control directly in the processor solves other problems, too. In chips with sophisticated pipelines and complex caches, integral debug circuitry can report processor state without concern for the cache and pipeline visibility problems that limit logic analyzers. Well-designed debug interfaces can reduce the overall package pin count. Also, when implemented in silicon, the debug core can’t be accidentally destroyed by software that has run amok and has written over a debug kernel located in the target system. (Not only is this a nice convenience, it can be a major time-saver if the debug kernel has to be downloaded to the target system every time the system crashes.) As processors and embedded systems become faster and more complex, on-chip debug support becomes more critical.

Finally, when the debug kernel is implemented as a fixed, standard part of the processor, hosted tool vendors can’t communicate with the “debug kernel” via a proprietary protocol any longer. Thus, moving the debug kernel into hardware has contributed to the emergence of new standard interface protocols. Three major debug protocols are used today: BDM (Background Debug Mode), IEEE 1149.1 JTAG (Joint Test Action Group), and IEEE-5001 ISTO (Nexus).

Hardware Instability

In general, you will be integrating unstable hardware with software that has never run on the hardware. The instability of the hardware means that the interface between the processor and the memory might be faulty, so that any code, even a debugger, cannot possibly run reliably in the target system.

With today’s processors running at frequencies over 1GHz and bus speeds in excess of 200MHz, circuit designers must take into account the dreaded analog effects. A printed circuit board that checks out just fine at DC might fail completely at normal bus speeds.

An embedded system that has a marginal timing problem or a cross-talk problem can appear to work correctly for long stretches of time and then just die. When the right combination of 1s and 0s appears on the right bus at the right time, a glitch occurs, and a bit flips where it shouldn’t, taking the system down with it. Until recently, these kinds of problems could wreck a project. Unless the proces sor-to- memory system was stable, the system could not be turned on. The only tool that could overcome this problem was the ICE.

Background Debug Mode

BDM is Motorola’s proprietary debug interface. Motorola was the first embedded processor vendor to place special circuitry in the processor core with the sole function of processor debugging. Thus, BDM began the trend to on-chip debug resources. Today, embedded processors or microcontrollers are expected to have some kind of dedicated, on-chip debugging circuitry. The hardware design need only bring the processor’s debug pins out to a dedicated connector and the debug tool, called an n-wire or wiggler. Figure 7.1 is a schematic representation showing an n-wire tool connected to an embedded system.

Figure 7.1: n-Wire tool.

Embedded system connection to a host computer using an n-wire connection to the processor debug core.

Note

The hardware module that interfaces to the embedded system’s n-wire debug port is sometimes called a wiggler because it wiggles several pins on the processor to implement the protocol of the debug core being used.

Compared to the cost of a traditional ICE, a wiggler is an incredible bargain. For example, I purchased 10 wigglers for use with the Motorola MF5206e ColdFire processor for about $40 each (including an educational discount). The wiggler, from P&E Micro, connects through the parallel port of a PC and includes a basic debugger that runs on the PC and communicates with the BDM core in the processor. The wiggler is inexpensive because the complex portions of the functionality have been moved into the chip, where circuitry is cheap. The wiggler does little else other than implement the debug core’s timing and protocol interface to the CPU.

BDM was first implemented with the 683XX family and is used with the ColdFire processor family. BDM connects to a 26-pin connector that is mounted on the target PC board. Figure 7.2 shows the pinout for the BDM debug interface.