Verification Vertigo

Last week it was CDNlive down in San Jose, the Cadence captive conference where they showcase all of their new and exciting technologies and share their roadmaps with their customers. I was there for two days and presented a session on Wednesday afternoon about the application of mutation analysis to help improve the efficiency and effectiveness of verification. The session was well attended and by the number and depth of the questions afterwards it was clear that my message was hitting home and they really wanted to see how this could help them improve their existing methodologies. If you want me to perform a similar talk at your company, then please let me know and we can schedule a session – brian_bailey at acm dot org

Afterwards I was talking with a group of Cadence employees. They said that the total cost to put on such a show was significantly less than what they usually had spent on DAC. In addition they did not have to constantly look over their shoulders to see if someone was listening in to their conversations and the quality of the people who attended was so much higher than the leads they got from DAC. One person asked if they thought Cadence would ever go back to DAC. The consensus answer was – I don’t see why we would ever want to return to DAC.

NuSym, one of the companies vying for the emerging intelligent testbench space, has just managed to raise $8M in venture funding to help bring their technology to market. This is on top of the $8M they received back in 2005 when they were exploring the technology. As I wrote about in a previous blog, intelligent testbenches are potentially one of the game changing technologies that could put pseudo random generation into a decline, just as it is getting traction in the marketplace.

Back when everyone created directed test, teams of test developers hand wrote testcases to exercise the important aspects of a device. This was high in human capital and the methodologies associated with it were not extensively developed, such that these tests were not reusable, maintainable or many other able’s.

Then along came constrained random techniques that traded off the human capital with a smaller amount of more highly skilled labor and lots of computers to automatically generate and simulate huge quantities of tests. Productivity soared - or did it?

Now the third generation of verification solutions promises to pare back the large amount of wasted resources that constrained random injected into the equation. It is too early to tell yet if they will succeed, but we desperately need something to improve the efficiency and effectiveness of the verification process.

Good luck to NuSym and all of the other companies pushing into this space!!

An important piece of standards work is being undertaken within Accellera. It is the creation of a standard API related to coverage data and is being conducted under the UCIS committee. 18 months ago, a request for donations was made. All of the major EDA companies declined to accept the possibility. They stated at that time that there was no user demand for such a thing. The group continued their efforts driven by the very companies that the EDA companies said did not exist. They took the next year to produce a requirements document saying exactly what they wanted in such an API. That was in January of this year.

Since then, the EDA companies - specifically Mentor and Synopsys are falling over each other to make a donation and trying to exclude the other. In fact more time is being spent on the donation process and procedures for deciding on donations than any actual technical progress. One of the EDA companies today said that another was participating in stalling practices. Another EDA company says that they have technology to donate but it will take 2 months to produce documentation. It is not clear if that is legal clearance or to actually write the documentation.

Standards has become so political that it is not surprising that most user companies get disheartened and withdraw from helping to craft the standards.

Shame on you big EDA companies!

I have just returned from England, where I helped my parents celebrate their 60^th wedding anniversary – an event where even the Queen sends them a congratulatory card.

Just a little under a year ago, I helped them move from the house they had lived in for 50 years, the one in which I had gown up, into a retirement community half way across the country. They have both survived significant medical problems including cancer and diabetes and now at 86, they are fully enjoying life. Apart from wanting to write about my parents, I think it also leaves some lessons to us. They are not the same people they were when they first met. They have seen war, and many changes – including me – God bless them. During that time, they have learned what works and what doesn’t, but perhaps more than anything else they have adapted and changed.

Functional verification is still a baby in comparison, and I am not sure it has seen quite the same disturbances that one can expect in life. Still, there have been some large discontinuities and these are often the times when the real leaders distinguish themselves from the rest. They are the ones who are not scared to face the change, to take on the new challenges. I admit there have been times when I have wanted to continue on the safe path, even though deep down I know it is wrong, but I am glad that at the end of the day, I think I have usually done the right thing. I take courage and pride in my parents for making those same choices and I hope they can be an inspiration to all of us to continue to adapt and change.

With the recent announcement of nVidia purchasing rights to use the Transmeta power optimizing portfolio of patents I though it was time to talk about the problems of low power and verification. A recent book by Synopsys and ARM was disappointing in that it really did not address this issue at all, just saying that the addition of low power technology makes verification more difficult. Thanks guys - big help! [Keating, Flynn, Aitken, Gibbons and Shi. Low Power Methodology Manual for System-on-Chip Design. Springer 2007]. In a paper by Freescale and Cadence they discuss the problems of RTL verification and conclude:

“While it is possible to use ad-hoc methods to verify these design elements, the methods are often far from ideal and can be slow, cumbersome and incomplete. “

More recently, Synopsys announced a new product that brings voltage awareness to the verification problem. They say:

“MVSIM co-simulates with VCS. VCS performs functional simulation of the logic elements and MVSIM simulates the power-management functions within a design. The result is a completely verified power-managed design.”

They also have a multi-voltage extension to their formal rule checker.

Power management verification consists of a number of different areas, such as correct level shifters and isolation logic, that all design instances are connected to the correct supply, that the correct registers are preserving state during power down, and many other aspects. One of the things necessary is a way to specify power management functions so that tools can automate the process of including the right data in the simulations. I am not talking about just the gate level, but the higher levels of abstraction so that proper architectural tradeoffs can be considered and verified long before the actual implementation. The HDLs really do not include any constructs to describe power.

Synopsys also announced some low power assertion technology recently. While this does not verify the power management intent, it does check that an implementation implement what is necessary. For example, if a region is meant to be powered down, then none of the clocks in that region should be active

At this years DVCon – two sessions existed on the subject of low-power verification, a testament to how important this topic is becoming. With its awareness growing, I hope it is not long before someone comes up with a way to describe power-intent and then I would have more hope that this issue will receive some real solutions. Until then it will be – in the words of Cadence – slow, cumbersome and incomplete.

Most of the time, standards get created after the EDA industry knows what it wants. This is either because a defacto standard has already emerged, or the industry has enough knowledge to be confident in the solution it is creating. In other cases, such as the recent developments in the Unified Coverage Interoperability group within Accellera, it happens because users demand it. But there are times when a new technology is emerging and interested parties need to be brought together, with the view to formulating something that will serve the needs of many people, and create a foundation for the accelerated adoption of a technology. What I am referring to is the creation of a standard fault model for HDL models.

Let me digress for a while. It is important to understand the difference between an error model and a fault model. Think back to the days of manufacturing test, before SCAN and other such techniques. We needed a set of vectors that would ensure that all faults in the device would be activated and propagated to an output where a comparison could be made. If a particular chip or board produced results different from the reference, then the part was deemed bad. A typical problem in a chip or board would either be an open circuit or a short circuit between two points. However, a fault model called the stuck at model was adopted. It said that if every wire was tied first the ground rail and then the supply rail, and the vectors applied, that if we could detect all of those faults, then their was also a very good chance that we would be able to detect actual errors in the system. Thus the fault model was not the same as the error model, but a relationship had been established in terms of the quality of results.

OK, so back to the HDL world. There is a growing need to have a fault model for this higher level of abstraction. In a Design and Test article [1] this month, the authors called for a fault model to help them analyze transient faults in systems as the geometries get so small, that stray particles are likely to cause errors, or as parts age. In an interview with Hana Chockler conducted by Richard Goering[2], Chockler called for the creation of a fault model to help in the creation of closure metrics for formal methods. The third part concerned, is a company called Certess[3] that has been using mutation based analysis to find weaknesses in verification environments, and some of their customers are asking for the fault model to be standardized.

So it seems to me that there are enough people working in the same area and direction to warrant an industry discussion on this subject. I wish there were a way and a forum that such industry discussion could happen without it inevitably leading to a standard, but rather an industry position paper or statement. In some regards I always thought that VSIA served in this role, and while it did produce standards, it was the industry discussion that it fostered that was perhaps even more important. It allowed an industry to come together and talk about their common problems and potential solutions. That is what we need here.

[1] - Austin, Bertacco, Mahlke and Cao. Reliable systems on Unreliable Fabrics. Design and Test. July/August 2008

[2] - Coverage metrics gain ground in formal verification. SCDsource July 31^st 2008 http://www.scdsource.com/experts.php?id=282

[3] – Certess corporate website http://certess.com

In a previous blog, I talked about the differences between verification and validation. What can confuse those definitions even more is when we start looking at verification in a hierarchical manner. This hierarchical process also brings in another fundamental distinction, namely that of positive and negative verification. In the book ESL Design and Verification I defined those terms in this way.

You either demonstrate or prove that a design performs a necessary task—positive verification—or you demonstrate or prove that bugs do not exist—negative verification.

This would appear to be similar to the distinction between verification and validation, except that we have to remember the principle source of the second model against which we are comparing the design. Assume for a moment, that an executable specification for the system exists. When we verify the specification, we are doing validation. When we are comparing an implementation against the specification, we are doing verification. In order to do implementation, we first have to break the specification down into smaller parts, unless the system was small enough to start off with. In addition, partitioning and mapping may also have been performed. As we break the spec down, there is a chance that errors will creep in, especially since there is an almost total lack of tools or automation to help with this task. So by the time we get to the block level, several such decompositions steps may have taken place. There is thus a finite chance that errors will have crept into the block specification, or that elements of the specification have been omitted.

So now if we verify an implementation against that derived specification, we could prove that all bugs have been eliminated, but that does not mean that when put into the context of the system that it will work. In other words we need to either verify a block in the context of the system, or we have to validate that the sub-specification was correct. This is true for all levels of verification and integration back up to the full system, when the definitions for positive, negative, verification and validation fall back inline with each other.

Positive verification demonstrates that a design requirement is met by verifying it against the original specification. Negative verification exposes any flaws in the implementation of a requirement and thus compares it against the local specification. Most methodologies in use today focus on negative verification because of their bottom-up nature.

When we consider designs that include a large amount of design IP, be it from third party companies or design components that are reused internally, this situation is magnified. When IP is being verified, it is not known exactly how it will be used, so all behaviours are verified equally. But, some or many of those behaviours may not be used in the design, potentially wasting verification time. Consider for a moment a design that is to be based on a platform. That platform is likely to contain many components such as processors, memories, peripherals and the logic necessary to integrate them both at the hardware and software levels. The platform manufacturer has hopefully performed a lot of verification on this platform so let us assume that it does indeed contain no bugs. Does this mean that all designs placed on that platform will operate correctly? If the platform matches every need of the design and in a way that has been correctly understood by the user of that platform then the answer should be yes. But there likely were a lot of assumptions and loose statements that qualified its application, and the reality is that there will be mismatches in what the platform provides and what is required. In a verification flow, we would thus still have to show that each feature required in the final product is indeed properly implemented when the system has been assembled.

However, we cannot rely solely on positive verification either because it suffers from the problem that if a new behaviour is added or changed—perhaps due to a software fix—you have not verified that it will operate correctly in the hardware. Given the truth to the adage “if it hasn’t been verified, then it does not work,” this presents a problem. Hence, a balance is needed. Positive verification is mainly employed in the sections preceding implementation verification: post-partitioning analysis and verification. Implementation verification is the time to apply negative verification and to show that the positive verification has not been broken by the transformation. By thinking of verification from both the positive and negative perspectives it becomes possible to better schedule some of the verification tasks. Features that are most vital to a product can and should be verified before effort is expended on less important features. This ensures that if time pressures force the chip to be released before verification is finished, those features most likely to cause problems have been adequately verified.

Many things in our industry go in cycles. Perhaps the most famous of these is Makimoto’s wave which identifies the cyclic oscillation between standardized and customized semiconductors with a 10-year cycle.

An email exchange with one of my clients and an article in Mentor’s latest issue of Verification News reminded me of another wave in our industry. The subject matter is verification languages. In the early 80’s the waveform language was a separate and distinct language – at that time purely for defining stimulus sets by assigning values at specific times. More sophisticated languages emerged over the next few years that brought in behavioral programming constructs. Then along came Verilog and basically the design and verification languages merged into a single language. About 10 years later, with the introduction of Vera and Specman e, a new generation of separate languages emerged. Once these had reached their climax, they too were incorporated into the design language – SystemVerilog (Well sort of. They are still separate languages within a common syntactic shell).

We are now seeing the third generation of separate verification languages – those related to the new class of intelligent testbenches. From these graph based languages (that does not imply graphics, as one industry pundit believes), a condensed set of functional vectors can be generated that cover all possible behaviors. These languages are in essence the first form of executable specification that the industry has seen, and I can’t help thinking that these may someday also become the way in which we design systems – thus seeing the languages come together again.

As always your views and comments are welcomed.

Keeping you covered

Brian Bailey

For the EDA industry, standards are traditionally defined by the EDA companies. While end users often attend the first few meetings of a new standards venture, few of them have the time or inclination to spend the many hours it takes to work through the technical details or the politics. As a result, many of the standards end up being a compromise solution based on what the EDA companies know how to implement. There has recently been a big exception to this and the possibility of another coming. The current exception is the Unified Coverage group within Accellera. After the EDA companies first refused to donate technology to satisfy the need, the user community took it up and continued to push forward with it. Now, a year later, the EDA companies realize that this was important to users and are willing to come back to the table. One for the user community.

At DAC, I had the pleasure to moderate a panel titled “OVM, VMM or roll your own”. This lively discussion centered around the needs of a verification methodology and the infrastructure to support it, with a view to making it extensible enough to last 5 or 10 years. One of the big issues that came up is the waste created by having two such competing infrastructures. I recommended at the end of the panel, that this is where users can and should make their voices heard. The users have the power and ability to decide on the fate of this industry struggle. Let your voices be heard and the EDA companies will listen eventually.

Perhaps the most common question at DAC is “What did you find that was new and interesting?” I often wonder if this is because they had not found anything themselves and need some ideas, or that there is so much information flying around, that they have problems sorting out the wheat from the chaff. Of course, there is a third possibility that there was nothing that was new and interesting within your field of interest. I have to admit that I have found myself in the same position some years, especially when the theme was in an area that was not central to my interests.

This year was easy for me. While DAC continues to short change verification, with almost all of the related sessions being pushed to Thursday afternoon, when most people are heading for home, the show floor was vibrant in the area of functional verification. The most notable for this year was the emergence of intelligent testbench tools. First I should define what I think they are before I talk about the options available.

An intelligent testbench can either replace or enhance existing simulation based or formal verification methodologies. Constrained random generation techniques manage to create huge quantities of stimulus, but at the end of the day they have difficulties both with closure (achieving the desired verification goals) and secondly with efficiency (huge server farms required). An intelligent testbench can help either by determining efficient stimulus sets or by finding ways to reach difficult to reach coverage points.

There were four tools on show at DAC this year, three of which are being shown for the first time. This number shows the need and the maturation of the technology necessary to implement them. Within those four entrants, they fell into two distinct classes, each of which had two examples. The first pair are those that work from a user created graph of the functionality. The two entrants in this field and inFact from Mentor Graphics and Trek from Breker Verification Systems, the only one that has appeared at DAC in the past. These tools derive the graph from the specification and thus have the ability to identify missing functionality in the design. On the downside, it requires the user to build the graph, which contrary to some peoples view does not imply graphics! The other two entrants create the graph from the design basically by performing simulation on the design and keeping track of the information required to create efficient paths through the design. While the way they do this is somewhat different, the end result is similar. The two entrants are from NuSym and from Avery. Both of these tools are in Beta testing at the moment. While they are based on the structure of the design, they require very small changes in the user environment, being able to make use of the existing testbenches.

I look forward to seeing all of these tools develop in the market, as the need for them is very real - especially as constrained random techniques begin to struggle with more complex designs.